Agent Desktop (Vercel Labs): Accessibility-Tree Computer Use for AI Agents

Give any coding agent reliable desktop computer use by reading and acting on real UI structure—not screenshots and coordinate guesses.

What it does

agent-desktop (Vercel Labs) is a cross-platform CLI and C library that turns OS accessibility APIs into a machine-readable control surface for AI agents. Instead of watching pixels, the agent takes a snapshot of an app’s accessibility tree, receives JSON with a stable ref for every actionable element, acts on that ref (click, type, set-value, and dozens more), then snapshots again to verify.

The loop stays in your agent: observe → act on a ref → observe again. Refs are bound to their snapshot; a bare @e8 is rejected, and stale UI returns codes such as STALE_REF or AMBIGUOUS_TARGET so the agent must re-snapshot rather than silently hitting the wrong control. On macOS, a private command host retains native accessibility objects so a removed row cannot silently resolve to another row with the same identifier.

Ship shape from the public README:

• Platforms: macOS, Windows, and Linux (AT-SPI), with an npm package that ships native binaries for x64 and ARM64; release archives and libagent_desktop_ffi for in-process use.
• Headless by default: Ref actions go through the accessibility API without moving the pointer or stealing focus; --headed allows physical input when needed.
• Command surface: On the order of 50+ operational commands across observe, ref actions, keyboard/mouse, apps/windows, notifications, clipboard, wait, sessions, and system helpers—plus version-matched agent skills via agent-desktop skills get desktop|platform.
• Agent integration: Installable as a skill (npx skills add … --skill agent-desktop) or documented in AGENTS.md; every command returns one JSON envelope with ok and either data or a structured error (code, disposition, suggestion). Retry only when error.disposition.retry is safe.

Homepage linked from the repo: agent-browser.dev/desktop.

Who it’s for

• Teams building coding agents, MCP tools, or RPA-style workflows that must drive real desktop apps (Finder, Slack, IDEs, internal tools)—not only browsers.
• Agent authors who want deterministic UI handles and retry-safe error codes instead of vision-only click paths.
• Developers on macOS / Windows / Linux who can grant accessibility (or AT-SPI) permissions and keep snapshots/clipboard/screenshots out of public logs—the README is explicit that those artifacts can contain private desktop content.

Stars / language / license

~8,267 stars · Rust · Apache-2.0
Data verified: 2026-11-02 ~22:16 Asia/Shanghai (GitHub API get_repository + README get_file_contents)

Trending note (selection ~22:14 Asia/Shanghai): ~+2,243 stars today on GitHub Trending daily

Category tags

AI · computer-use · desktop-automation · accessibility · agent-tools · MCP

How it differs from similar projects

Public positioning contrasts with pixel / screenshot computer-use stacks: agent-desktop’s contract is accessibility-tree structure + snapshot-scoped refs, not OCR or coordinate guessing. Compared with browser-only automation (Playwright/Puppeteer-style or browser computer-use agents), it targets native desktop apps across three OSes and ships both a CLI loop and a C FFI for embedding. Versus generic desktop RPA that often scripts coordinates or brittle selectors, the README emphasizes stale-ref detection, window disambiguation (--window-id), dense-app workflows (--skeleton then --root), and agent-facing skill docs version-locked to the binary.

This article does not claim latency or success-rate benchmarks beyond what the project publishes in its own docs. The Rising angle for overseas developers is straightforward: if your agents are stuck guessing from pixels on desktop apps, structure-first computer use is the clearest public alternative currently climbing Daily Trending.

Snapshot → ref → verify (from README)

agent-desktop snapshot --app Finder -i
agent-desktop click '@s8f3k2p9:e8'
agent-desktop snapshot --app Finder -i

Each response is one JSON envelope. Interactive elements carry ref_id values and available_actions. Dense UIs can be mapped with a skeleton snapshot, then drilled with --root.

Permissions and privacy

• macOS: Accessibility permission (agent-desktop permissions --guide).
• Windows: No extra grant for apps at the terminal’s integrity level (per README).
• Linux: User graphical session with at-spi2-core; Wayland/X11 notes live in install docs.

Snapshots, clipboard reads, screenshots, and debug artifacts may hold private content—keep them out of public uploads.

Repo link

https://github.com/vercel-labs/agent-desktop