Product
Transforming natural language commands
into actual OS actions
AllvIa provides a fully automated NL pipeline from intent analysis to execution, verification, and approval, along with Shadow Mode that learns user behavior in the background. A production-ready agent featuring a Single Source of Truth (SSOT) core of 99+ Rust modules (111 Rust source files total) and 5-layer security based on Zero Trust principles.
NL Automation Pipeline
A 6-stage automated pipeline from natural language input to intention analysis, execution, verification, and approval.
Intent
intent_router.rs
Classify NL → IntentType
Slot
slot_filler.rs
Extract required parameters
Plan
plan_builder.rs
Generate execution steps
Execute
execution_controller.rs
Automated step-by-step execution
Verify
verification_engine.rs
Automated result verification
Approve
approval_gate.rs
Security gate for dangerous actions
OS-Level Control
Watch the screen, click, and manipulate apps directly
Visual Driver & Rust Collector
visual_driver.rs and collector_rs.rs work together to handle screen recognition and data collection. 5-minute aggregation and PII masking are handled at the core level.
Browser Automation
browser_automation.rs automatically performs web page navigation, form input, and data extraction.
Shadow Mode
analyzer.rs analyzes events collected in the background to identify repeating routines and recommend automation opportunities.
$ cargo run --bin local_os_agent
> surf "Open Notes, create a new note, and type Research Topic"
Environment Scan: Found 349 apps
Canonical: "Notes" → "Notes.app"
[OK] Notes.app execution (applescript.rs)
[OK] Shortcut: Cmd+N (Create new note)
[OK] Type: "Research Topic" (keyboard.type)
Verification: Note creation confirmed
Action Classification:
Safe: ui.find, screen.capture, app.list
Caution: ui.click, keyboard.type
Critical: file.delete, cmd.exec, submit.payment
Write Lock: LOCKED (default)
Tool Policy: TOOL_ALLOWLIST / TOOL_DENYLIST
PII Guard: [EMAIL], [CARD], [SSN] → masked
Kill Switch: Multi-press Esc → Instant Exit
Zero Trust Security
Control all executions with 5-layer security
1. PolicyEngine (policy.rs)
Classifies actions into 3 levels: Safe / Caution / Critical.
2. CommandClassifier (security.rs)
Automatically analyzes the risk of shell commands and blocks them.
3. ToolPolicy (tool_policy.rs)
Manages per-tool Allow/Denylists based on environment variables.
4. PrivacyGuard (privacy.rs)
Automatically masks PII such as emails and card numbers before saving to the DB.
5. API Auth (api_server.rs)
Applies middleware authentication based on STEER_API_KEY.
External Integrations
Automate everyday services with native integrations.
Gmail
gmail.rs (5KB)
- Email lookup/send
- Classification automation
- Gmail API integration
Google Calendar
calendar.rs (5KB)
- Schedule lookup/create
- Reminder management
- Google OAuth integration
Notion
notion.rs (6KB)
- Read/write pages
- Database query
- Notion API integration
Telegram
telegram.rs (12KB)
- Remote command execution
- Result report transmission
- PC control while away
Technical Specification
Full architecture and technology stack of the AllvIa agent.
| Component | Tech Stack | Detail |
|---|---|---|
| Core Engine | Rust (99+ core modules / 111 Rust files) | main.rs, core::lib.rs (Single Source of Truth), api_server.rs (Axum) |
| NL Pipeline | 6-stage Automation | intent_router → slot_filler → plan_builder → execution → verification → approval |
| Shadow Mode | Batch Pipeline | collector_rs → build_sessions → build_routines → build_handoff (Rust native) |
| OS Control | AppleScript + Visual Driver | visual_driver.rs, browser_automation.rs, screen_recorder.rs |
| Storage | SQLite (steer.db) | Event logs (v2), 5-min aggregation, daily summary, session/routine/handoff data |
| Orchestration | n8n (Docker / NPX) | n8n_api.rs, STEER_N8N_RUNTIME support (Docker/NPX/Manual) |
| Web UI | React/Vite + Tailwind | Dashboard, Chat, Launcher, Routines, Workflows, Settings |
| Desktop | Tauri | Native apps for macOS (.app) / Windows (.exe) |
| Integrations | Gmail, Calendar, Notion, Telegram | Google OAuth, Notion API, Telegram Bot API integration |
| Security | 5-layer Zero Trust | policy.rs, security.rs, tool_policy.rs, privacy.rs, api_server auth |
| Data Pipeline | Rust Collector + Sensors | build_sessions_rs, build_routines_rs, build_handoff_rs |
| Reliability | Self-Healing Guardian | Automatic process restarts, retry_logic.rs, singleton_lock.rs |
Check it out for yourself
Following the Getting Started guide, you can run your first automation in less than 5 minutes.