// OVERVIEW
About this project
A local-first agentic AI built at BloomKnights Hackathon 2026 with a team of four. Runs a full LLM (qwen3-coder:30b on local Ollama, zero cloud API cost) on a Windows 11 PC, and is controlled entirely from the phone via a Telegram bot and an Edge browser "J-badge" extension. Every submit/send/buy/delete action is gated behind a phone Approve/Deny card, so the agent can draft freely but can't act without consent.
// ARCHITECTURE
System architecture
Phone (Telegram) ↔ local PC (Windows 11). The bot bridge (aiohttp) receives commands from the phone; the agent core reasons over qwen3-coder:30b via a local Ollama server; a tool registry (TOOLS: {name: {schema, fn}}) resolves the request into a specific tool call. If the tool is marked destructive, the gate fires an Approve/Deny card back to the phone and only executes on the confirm callback. Optional side channel: the Edge J-badge extension tags interactive DOM elements so the browser agent can click through modern SPAs.
Agent Core
Module-level async functions, lean by design. A tool registry (TOOLS) maps tool names to {schema, fn}. Functions never raise — they return "Error: ..." strings so the agent loop can reason over failures instead of crashing.
Local Ollama Inference
qwen3-coder:30b on Ollama v0.31.2 (winget install, server on :11434). Python ollama lib hits the local server — no cloud API. First load ~64s, ~2 tok/s on 8GB VRAM with RAM offload; comfortable on the 16GB demo box.
Telegram Bot Bridge
aiohttp-based bridge routes /commands, plain text, voice notes, and approval callbacks. All three inbound paths (text, voice, scheduled) funnel through a single shared _dispatch_text so behavior stays consistent across entry points.
Safety Gate
confirm_cb wraps every destructive tool call. Sends an Approve/Deny card to the phone before executing send / trash / power / etc. Reading, drafting, and searching bypass the gate.
Scheduler
schedule.py + /schedule wizard. Recurrence: daily / weekdays / weekly HH:MM, every N minutes/hours, hourly, once. Due jobs enqueue via state.new_task with origin="internal" → same gate, same tool pipeline. Persists to schedules.json (gitignored).
Voice Pipeline
faster-whisper runs locally (STT model configurable via WHISPER_MODEL / DEVICE / COMPUTE env). Telegram voice notes are transcribed on-device, echoed back as "heard: ...", and routed through _dispatch_text so voice and text share one code path.
Browser Agent + J-badge Extension
Chromium extension tags interactive elements (piercing open shadow DOM, widened selector for role/tabindex/summary/contenteditable). Playwright drives clicks against the tagged badges. Vision overlay + click digest are decoupled so a modest 100-element digest cap doesn't hide 200 elements from the VLM candidate pass.
// FEATURES
Key features
Local-First Reasoning
All inference runs on-device against a local Ollama server (qwen3-coder:30b) — zero cloud API calls, zero API bill, and nothing leaves the machine. The phone talks to the PC over a Telegram bot bridge, so the LLM never sees the internet.
Phone-Gated Actions
Reading and drafting run autonomously, but any submit / send / buy / delete action fires an Approve/Deny card on the phone before it can execute. Draft-then-confirm is the default posture — the agent can prepare anything but can't act without a tap.
Gmail Send / Reply / Draft
On top of the existing read-only inbox scanner: send, reply-in-thread, draft, and trash. Scope bumped from gmail.readonly to gmail.modify (no permanent delete). Send/reply/trash gate through the phone; draft is ungated because it's reversible.
Scheduler
An interactive /schedule wizard on Telegram — walks the user through what → when → approval. Recurrence rules include daily, weekdays, weekly at HH:MM, every N minutes/hours, hourly, and one-off. Due jobs enqueue back through the normal agent pipeline and still hit the safety gate. Persists across restarts.
PC Remote Control
Volume + mute (pycaw), brightness (screen-brightness-control), media transport keys, screen lock, and gated power actions (sleep / shutdown / restart via confirm). Hardware libs are lazy-imported and platform-guarded so non-Windows dev boxes still install cleanly.
Voice Control
Send a Telegram voice note and faster-whisper transcribes it on-device, echoes "heard: ...", then routes through the same dispatcher as text so voice and text share one code path — no duplicate handling.
Edge "J-badge" Extension
A companion Chromium extension that pierces open shadow DOM and widens the interactive selector (role=link/menuitem/tab, [tabindex], summary, contenteditable, a[onclick]) so the agent can find and click through modern SPAs the vanilla Playwright selectors miss.
// ROADMAP
Development roadmap
- DONECore Agent Loop + Tool Registry
- DONESafety Gate (Approve/Deny Phone Cards)
- DONEGmail Send / Reply / Draft / Trash — PR #1
- DONEScheduler with Recurrence — PR #2
- DONEPC Remote Control (volume / brightness / media / power) — PR #3
- DONEVoice Control via Local Whisper — PR #4
- DONEVision + Browser Scraper Coverage Fix — PR #5
- DONEaiohttp Requirements Fix — PR #6
// CHALLENGES
Challenges and solutions
Full LLM Reasoning Without a Cloud API
PROBLEM
Most agentic hackathon projects lean on OpenAI or Claude — fast, but bills up and pins the demo to a network. The team wanted a real local model that could still reason well enough to plan multi-step actions.
SOLUTION
Ran qwen3-coder:30b on local Ollama (~18GB model), warmed on startup, kept the whole agent loop on-device. First load ~64s on an 8GB VRAM card with heavy RAM offload; the demo box (RTX 5080 / 16GB) ran it comfortably. Zero cloud API cost across the whole hackathon.
Safety Model for an Autonomous Agent
PROBLEM
An LLM that can send Gmail, control the PC, click through websites, and schedule tasks is one bad plan away from real damage. Naive autonomy is unshippable; gating everything makes it feel dead.
SOLUTION
Split every tool into 'reversible / autonomous' vs 'destructive / gated.' Drafts, reads, and searches run without asking. Submit / send / buy / delete / power actions fire an Approve/Deny card back to Telegram and only execute on the confirm callback. Later a `!` preauth prefix was added, then removed team-wide when a teammate landed unconditional gating — I merged origin/main into all four of my branches and conformed every feature to always-gate.
Shared Telegram Bot Token
PROBLEM
The team's shared @JerryAAHK_bot token was already polled on a teammate's machine. Telegram allows exactly one getUpdates poller per token, so starting the app locally threw telegram.error.Conflict and both instances flapped — disrupting the teammate's demo.
SOLUTION
Documented the constraint in CONTRIBUTING and switched every dev to their own @BotFather test bot for local runs. Verified the full pipeline (mock server + Ollama warm-up + bot bridge) came up clean on a personal bot before any shared-token change landed.
Browser Agent Missing Clicks
PROBLEM
The vision + click browser agent was 'not thorough, missed things, didn't click' on modern SPAs. The extractor JS fed both the click digest and the vision overlay, but only badged elements that were already tagged data-agent-id, so anything inside a closed shadow root or non-standard interactive element was invisible.
SOLUTION
Additive fix: pierce open shadow DOM, widen the interactive selector to include role=link/menuitem/tab/option/etc, [tabindex], summary, label, contenteditable, a[onclick], stop nested-dedup from swallowing buttons-in-cards, and raise digest / overlay / VLM candidate caps (100→200, 50→80). Validated with py_compile + node --check on the extractor JS.