站内导读
定位:Vercel Labs 推出的 agent-desktop 是面向 macOS、Windows、Linux 的 CLI 与 C 库,让 AI Agent 通过操作系统无障碍树读写桌面应用,适合需要稳定操作界面的自动化场景。
核心能力
- 快照返回 JSON,每个可操作元素带 ref,Agent 按 ref 执行后再快照校验,不依赖像素匹配。
- 每次 CLI 调用独立进程,默认 headless,不抢焦点、不合成输入,全局 --headed 才允许物理输入。
- 提供 60 个命令名,其中 56 个可运行,key-down 等四个命令在守护进程支持前拒绝执行。
- 内置与版本匹配的 skills 文档,错误返回 code、disposition 与 suggestion,仅在 retry 为 safe 时重试。
适用人群:适合用 Node.js 或 Bun 编写桌面自动化 Agent 的开发者,需 macOS 辅助功能权限、Windows 终端同等完整性级别或 Linux 带 at-spi2-core 的图形会话。
注意:快照、剪贴板、截图可能含隐私内容,勿入公开日志。
位置づけ:Vercel Labs による macOS・Windows・Linux 向けの CLI と C ライブラリで、OS のアクセシビリティツリーを通じて AI エージェントがデスクトップアプリを読み取り・操作できます。
主な機能
- スナップショットは操作可能な各要素に ref を付けた JSON を返し、ref 実行後に再取得して検証します。ピクセル照合は行いません。
- CLI 呼び出しは毎回別プロセスで動作し、既定は headless。フォーカスを奪わず合成入力も使いません。--headed 指定時のみ物理入力が可能です。
- 60 のコマンド名のうち 56 が実行可能で、key-down など 4 つはデーモン対応まで実行を拒否します。
- バイナリにバージョン一致の skills ドキュメントを同梱し、エラーは code・disposition・suggestion を返します。
対象となる利用者:Node.js や Bun でデスクトップ自動化エージェントを組む開発者向け。macOS のアクセシビリティ権限、Windows はターミナルと同整合性レベル、Linux は at-spi2-core 入りのグラフィカルセッションが必要です。
注意:スナップショットやクリップボード、スクリーンショットには私的な内容が含まれ得ます。公開ログに残さないでください。
개요: Vercel Labs가 만든 macOS·Windows·Linux용 CLI 및 C 라이브러리로, OS 접근성 트리를 통해 AI 에이전트가 데스크톱 앱을 읽고 조작할 수 있게 합니다.
핵심 기능
- 스냅샷은 조작 가능한 각 요소에 ref를 붙인 JSON을 반환하며, ref 실행 후 다시 스냅샷해 검증합니다. 픽셀 매칭은 쓰지 않습니다.
- CLI 호출은 매번 별도 프로세스에서 실행되고 기본은 headless라 포커스를 빼앗지 않으며 합성 입력도 쓰지 않습니다. --headed일 때만 물리 입력이 가능합니다.
- 명령 이름 60개 중 56개가 동작하며, key-down 등 4개는 데몬 지원 전까지 실행을 거부합니다.
- 바이너리에 버전에 맞는 skills 문서를 포함하고, 오류는 code·disposition·suggestion을 반환합니다.
대상 사용자: Node.js나 Bun으로 데스크톱 자동화 에이전트를 만드는 개발자에게 적합합니다. macOS 접근성 권한, Windows는 터미널과 같은 무결성 수준, Linux는 at-spi2-core가 있는 그래픽 세션이 필요합니다.
주의: 스냅샷, 클립보드, 스크린샷에는 개인 정보가 포함될 수 있으니 공개 로그에 남기지 마세요.
What it is: A CLI and C library from Vercel Labs for macOS, Windows and Linux that lets AI agents read and operate desktop apps through the OS accessibility tree.
Key capabilities
- A snapshot returns JSON with a ref for every actionable element; the agent acts on a ref, then snapshots again to verify, with no pixel matching.
- Each CLI call runs in its own process and is headless by default, so ref actions never move the pointer or take focus; the global --headed flag allows physical input.
- It ships 60 command names, 56 operational; key-down, key-up, mouse-down and mouse-up refuse to run until a daemon can release held keys or buttons.
- The binary carries version-matched skills docs, and errors return a code, a disposition and usually a suggestion; retry only when error.disposition.retry is safe.
Who it is for: Developers building desktop automation agents in Node.js or Bun. macOS needs Accessibility permission, Windows needs apps at the terminal's integrity level, and Linux needs a graphical session with at-spi2-core.
Caveats: Snapshots, clipboard reads, screenshots and debug artifacts can contain private desktop content; keep them out of public logs and uploads.
README · 详细文档
来源 · 查看原文
agent-desktop
Computer use for AI agents: read and operate desktop apps through the OS accessibility tree.
A CLI and C library for macOS, Windows and Linux. A snapshot returns JSON with a ref for every element an agent can act on. The agent acts on a ref and snapshots again to check, with no pixel matching; the loop stays in your agent.
Every CLI call runs in its own process. Headless calls still prohibit foreground activation and synthesized input.
Snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content. Keep them out of public logs and uploads. The optional JEV helper sends screen descriptions to a configured model provider; it is separate from the native CLI. See privacy and data handling.
Installation
Install on macOS, Windows or Linux with npm:
npm install -g agent-desktop
agent-desktop version
Bun, Yarn and pnpm can install the same package. It includes native binaries and platform helpers for x64 and ARM64. For direct execution without a JavaScript launcher, download the matching native archive from GitHub Releases and keep its helper beside the executable.
Official Homebrew and Scoop packages are being submitted; they are not yet available in the default catalogs. The repository no longer maintains a custom Homebrew tap or Scoop bucket.
- macOS needs Accessibility permission:
agent-desktop permissions --guide. - Windows needs no grant for apps at your terminal's integrity level.
- Linux needs the user's graphical session with
at-spi2-core.
See installation and permissions for package-manager behavior, release archives, and building from source.
Quick start
Observe, act on a ref, observe again. Any app name from agent-desktop list-apps works; Finder is the macOS example. Each command prints one JSON envelope; a trimmed snapshot follows the commands.
agent-desktop snapshot --app Finder -i # interactive elements, each with a ref
agent-desktop click @s8f3k2p9:e8 # act on a ref from that snapshot
agent-desktop snapshot --app Finder -i # observe again to verify
{
"version": "2.6",
"ok": true,
"command": "snapshot",
"data": {
"app": "Finder",
"window": { "id": "w-204749", "title": "Downloads" },
"snapshot_id": "s8f3k2p9",
"ref_count": 59,
"complete": true,
"tree": {
"role": "window",
"name": "Downloads",
"children": [
{ "role": "treeitem", "name": "Documents", "ref_id": "@s8f3k2p9:e8", "available_actions": ["Click"] }
]
}
}
}
- Refs belong to their snapshot. A bare
@e8is rejected withINVALID_ARGS. When the UI changes, an action
returns STALE_REF or AMBIGUOUS_TARGET: take a new snapshot and use its refs.
- Several windows give
AMBIGUOUS_TARGET;snapshotlists the candidate window ids inerror.details. Pass one
with --window-id.
- Dense apps: map first with
snapshot --skeleton --app Slack -i --compact, then drill into one region with
snapshot --root @s8f3k2p9:e3 -i --compact.
- Headless by default. Ref actions go through the accessibility API without moving the pointer or taking focus.
The global --headed flag allows physical input.
- PowerShell. Quote every ref (
'@s8f3k2p9:e8'), because a bare@tokenis the splatting operator.
For AI agents
The binary carries its own docs, matched to its version:
agent-desktop skills get desktop # the loop, refs, verification and retry rules
agent-desktop skills get platform # what differs on the OS you run on
agent-desktop <command> --help # flags for one command
Install the skill into an agent harness from skills.sh:
npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop
Or add this to your AGENTS.md:
Use `agent-desktop` to see and operate desktop apps. Before the first use, run
`agent-desktop skills get desktop` and `agent-desktop skills get platform`, and follow them.
Every command prints one envelope with ok and either data or error. An error carries a code, a disposition and usually a suggestion; retry only when error.disposition.retry is safe. The contract, error codes and exit codes are in docs/json-output.md.
Commands
It ships 60 command names, 56 operational commands. The other four (key-down, key-up, mouse-down, mouse-up) refuse to run until a daemon can release a held key or button.
| Group | Commands |
|---|---|
| Observe | snapshot, find, get, is, screenshot, list-surfaces |
| Act on a ref | click, double-click, triple-click, right-click, type, set-value, clear, focus, select, toggle, check, uncheck, expand, collapse, scroll, scroll-to |
| Keyboard and mouse | press, hover, drag, mouse-move, mouse-click, mouse-wheel, key-down, key-up, mouse-down, mouse-up |
| Apps and windows | launch, close-app, list-apps, list-windows, list-displays, focus-window, resize-window, move-window, minimize, maximize, restore, open-system-surface |
| Notifications | list-notifications, dismiss-notification, dismiss-all-notifications, notification-action |
| Clipboard | clipboard-get, clipboard-set, clipboard-clear |
| Wait | wait |
| Sessions | session, trace, cursor-overlay |
| System | status, permissions, version, batch, skills |
Platform support
| macOS | Windows | Linux | |
|---|---|---|---|
| Accessibility tree | Yes | Yes | Yes (AT-SPI) |
| Click / type / keyboard | Yes | Yes | AT-SPI; X11 / GNOME keys |
| Mouse input | Yes | Yes | X11, --headed |
| Screenshot | Yes | Yes | X11 windows; Wayland portal |
| Clipboard | Yes | Yes | Yes; Wayland needs wl-clipboard |
| App & window management | Yes | Yes | Launch/list; X11 window control |
| Notifications | Yes | Yes | GNOME Shell |
Shell surfaces (open-system-surface) | Planned | Yes | GNOME tray, Action Center |
The Windows surface and launch notes and the Linux X11 and Wayland notes are in docs/install.md.
agent-desktop skills get platform prints the full capability table for the OS you run on.
Language bindings
Each release also ships libagent_desktop_ffi for macOS, Linux and Windows, so a program can call the tool in process instead of starting the CLI for each command. agent-desktop skills get ffi covers the entry points, memory ownership and threading rules.
import ctypes
lib = ctypes.CDLL("./lib/libagent_desktop_ffi.dylib")
lib.ad_adapter_create.restype = ctypes.c_void_p
lib.ad_adapter_destroy.argtypes = [ctypes.c_void_p]
assert lib.ad_init(7) == 0 # AD_RESULT_OK when the header's AD_ABI_VERSION_MAJOR matches the library
adapter = lib.ad_adapter_create()
lib.ad_adapter_destroy(adapter)
Development
Each platform adapter builds only on its own OS, so scope cargo to the host's packages. On macOS:
HOST_PKGS=(-p agent-desktop-core -p agent-desktop-macos -p agent-desktop -p agent-desktop-ffi)
cargo test "${HOST_PKGS[@]}"
cargo clippy "${HOST_PKGS[@]}" --all-targets -- -D warnings
On Windows, replace agent-desktop-macos in that list with agent-desktop-windows; on Linux, with agent-desktop-linux. CONTRIBUTING.md lists every gate. Build from source: docs/install.md.
Documentation
| I want to... | Read |
|---|---|
| Fix an install problem, grant permissions, verify a download or build from source | docs/install.md |
| Parse the JSON envelope, error codes and exit codes | docs/json-output.md |
| Drive the tool as an AI agent | agent-desktop skills get desktop (skills/agent-desktop/) |
| Learn what differs on one OS | macOS, Windows, Linux |
| Call the C library | skills/agent-desktop-ffi/ |
| Look up the vocabulary: refs, snapshots, sessions | CONCEPTS.md |
| Contribute and pass the gates | CONTRIBUTING.md |
| Read the engineering rules | AGENTS.md |
| Learn from past bugs and lessons | docs/solutions/ |
| Find answers to common questions | docs/faq.md |
| Report a vulnerability | SECURITY.md |
| Get help | SUPPORT.md |
License
Apache-2.0