[Late catch] Vercel agent-desktop 1.0: a CLI that lets agents read and drive desktop apps through the accessibility tree
What is it?
agent-desktop is a CLI and C library that lets AI agents read and operate desktop apps on macOS, Windows and Linux through the OS accessibility tree, the structured list of on-screen elements the operating system exposes. A snapshot returns JSON with a ref for every element an agent can act on. The agent clicks a ref, then takes another snapshot to check the result. No pixel matching is involved.
npm install -g agent-desktop
agent-desktop snapshot --app Finder -iWhy does it matter?
It's lighter than screenshot-and-click computer use, and the JSON output is easy to wire into coding agents like Claude Code or Codex as a tool. The loop stays inside your own agent, so you aren't tied to one model. Snapshots and clipboard reads can contain private screen content, so follow the README and keep them out of logs.
Maturity
GA1.0 is a stable release, and patches have continued through 1.0.2. Official Homebrew and Scoop packages are still under review.
How to track it
- Packaging: check the GitHub releases and README for Homebrew and Scoop availability.
- Integrations: watch the repo for skill or plugin integrations aimed at coding agents.