站内导读

中文

定位:Vercel Labs 推出的 agent-desktop 是面向 macOS、Windows、Linux 的 CLI 与 C 库,让 AI Agent 通过操作系统无障碍树读写桌面应用,适合需要稳定操作界面的自动化场景。

核心能力

  • 快照返回 JSON,每个可操作元素带 ref,Agent 按 ref 执行后再快照校验,不依赖像素匹配。
  • 每次 CLI 调用独立进程,默认 headless,不抢焦点、不合成输入,全局 --headed 才允许物理输入。
  • 提供 60 个命令名,其中 56 个可运行,key-down 等四个命令在守护进程支持前拒绝执行。
  • 内置与版本匹配的 skills 文档,错误返回 code、disposition 与 suggestion,仅在 retry 为 safe 时重试。

适用人群:适合用 Node.js 或 Bun 编写桌面自动化 Agent 的开发者,需 macOS 辅助功能权限、Windows 终端同等完整性级别或 Linux 带 at-spi2-core 的图形会话。

注意:快照、剪贴板、截图可能含隐私内容,勿入公开日志。

README · 详细文档

来源 · 查看原文

agent-desktop

Computer use for AI agents: read and operate desktop apps through the OS accessibility tree.

Vercel Labs Product npm version: agent-desktop License: Apache-2.0 npm downloads per month: agent-desktop

A CLI and C library for macOS, Windows and Linux. A snapshot returns JSON with a ref for every element an agent can act on. The agent acts on a ref and snapshots again to check, with no pixel matching; the loop stays in your agent.

Every CLI call runs in its own process. Headless calls still prohibit foreground activation and synthesized input.

Snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content. Keep them out of public logs and uploads. The optional JEV helper sends screen descriptions to a configured model provider; it is separate from the native CLI. See privacy and data handling.

Installation

Install on macOS, Windows or Linux with npm:

npm install -g agent-desktop
agent-desktop version

Bun, Yarn and pnpm can install the same package. It includes native binaries and platform helpers for x64 and ARM64. For direct execution without a JavaScript launcher, download the matching native archive from GitHub Releases and keep its helper beside the executable.

Official Homebrew and Scoop packages are being submitted; they are not yet available in the default catalogs. The repository no longer maintains a custom Homebrew tap or Scoop bucket.

  • macOS needs Accessibility permission: agent-desktop permissions --guide.
  • Windows needs no grant for apps at your terminal's integrity level.
  • Linux needs the user's graphical session with at-spi2-core.

See installation and permissions for package-manager behavior, release archives, and building from source.

Quick start

Observe, act on a ref, observe again. Any app name from agent-desktop list-apps works; Finder is the macOS example. Each command prints one JSON envelope; a trimmed snapshot follows the commands.

agent-desktop snapshot --app Finder -i      # interactive elements, each with a ref
agent-desktop click @s8f3k2p9:e8            # act on a ref from that snapshot
agent-desktop snapshot --app Finder -i      # observe again to verify
{
  "version": "2.6",
  "ok": true,
  "command": "snapshot",
  "data": {
    "app": "Finder",
    "window": { "id": "w-204749", "title": "Downloads" },
    "snapshot_id": "s8f3k2p9",
    "ref_count": 59,
    "complete": true,
    "tree": {
      "role": "window",
      "name": "Downloads",
      "children": [
        { "role": "treeitem", "name": "Documents", "ref_id": "@s8f3k2p9:e8", "available_actions": ["Click"] }
      ]
    }
  }
}
  • Refs belong to their snapshot. A bare @e8 is rejected with INVALID_ARGS. When the UI changes, an action

returns STALE_REF or AMBIGUOUS_TARGET: take a new snapshot and use its refs.

  • Several windows give AMBIGUOUS_TARGET; snapshot lists the candidate window ids in error.details. Pass one

with --window-id.

  • Dense apps: map first with snapshot --skeleton --app Slack -i --compact, then drill into one region with

snapshot --root @s8f3k2p9:e3 -i --compact.

  • Headless by default. Ref actions go through the accessibility API without moving the pointer or taking focus.

The global --headed flag allows physical input.

  • PowerShell. Quote every ref ('@s8f3k2p9:e8'), because a bare @token is the splatting operator.

For AI agents

The binary carries its own docs, matched to its version:

agent-desktop skills get desktop      # the loop, refs, verification and retry rules
agent-desktop skills get platform     # what differs on the OS you run on
agent-desktop <command> --help        # flags for one command

Install the skill into an agent harness from skills.sh:

npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop

Or add this to your AGENTS.md:

Use `agent-desktop` to see and operate desktop apps. Before the first use, run
`agent-desktop skills get desktop` and `agent-desktop skills get platform`, and follow them.

Every command prints one envelope with ok and either data or error. An error carries a code, a disposition and usually a suggestion; retry only when error.disposition.retry is safe. The contract, error codes and exit codes are in docs/json-output.md.

Commands

It ships 60 command names, 56 operational commands. The other four (key-down, key-up, mouse-down, mouse-up) refuse to run until a daemon can release a held key or button.

GroupCommands
Observesnapshot, find, get, is, screenshot, list-surfaces
Act on a refclick, double-click, triple-click, right-click, type, set-value, clear, focus, select, toggle, check, uncheck, expand, collapse, scroll, scroll-to
Keyboard and mousepress, hover, drag, mouse-move, mouse-click, mouse-wheel, key-down, key-up, mouse-down, mouse-up
Apps and windowslaunch, close-app, list-apps, list-windows, list-displays, focus-window, resize-window, move-window, minimize, maximize, restore, open-system-surface
Notificationslist-notifications, dismiss-notification, dismiss-all-notifications, notification-action
Clipboardclipboard-get, clipboard-set, clipboard-clear
Waitwait
Sessionssession, trace, cursor-overlay
Systemstatus, permissions, version, batch, skills

Platform support

macOSWindowsLinux
Accessibility treeYesYesYes (AT-SPI)
Click / type / keyboardYesYesAT-SPI; X11 / GNOME keys
Mouse inputYesYesX11, --headed
ScreenshotYesYesX11 windows; Wayland portal
ClipboardYesYesYes; Wayland needs wl-clipboard
App & window managementYesYesLaunch/list; X11 window control
NotificationsYesYesGNOME Shell
Shell surfaces (open-system-surface)PlannedYesGNOME tray, Action Center

The Windows surface and launch notes and the Linux X11 and Wayland notes are in docs/install.md.

agent-desktop skills get platform prints the full capability table for the OS you run on.

Language bindings

Each release also ships libagent_desktop_ffi for macOS, Linux and Windows, so a program can call the tool in process instead of starting the CLI for each command. agent-desktop skills get ffi covers the entry points, memory ownership and threading rules.

import ctypes
lib = ctypes.CDLL("./lib/libagent_desktop_ffi.dylib")
lib.ad_adapter_create.restype = ctypes.c_void_p
lib.ad_adapter_destroy.argtypes = [ctypes.c_void_p]
assert lib.ad_init(7) == 0  # AD_RESULT_OK when the header's AD_ABI_VERSION_MAJOR matches the library
adapter = lib.ad_adapter_create()
lib.ad_adapter_destroy(adapter)

Development

Each platform adapter builds only on its own OS, so scope cargo to the host's packages. On macOS:

HOST_PKGS=(-p agent-desktop-core -p agent-desktop-macos -p agent-desktop -p agent-desktop-ffi)
cargo test "${HOST_PKGS[@]}"
cargo clippy "${HOST_PKGS[@]}" --all-targets -- -D warnings

On Windows, replace agent-desktop-macos in that list with agent-desktop-windows; on Linux, with agent-desktop-linux. CONTRIBUTING.md lists every gate. Build from source: docs/install.md.

Documentation

I want to...Read
Fix an install problem, grant permissions, verify a download or build from sourcedocs/install.md
Parse the JSON envelope, error codes and exit codesdocs/json-output.md
Drive the tool as an AI agentagent-desktop skills get desktop (skills/agent-desktop/)
Learn what differs on one OSmacOS, Windows, Linux
Call the C libraryskills/agent-desktop-ffi/
Look up the vocabulary: refs, snapshots, sessionsCONCEPTS.md
Contribute and pass the gatesCONTRIBUTING.md
Read the engineering rulesAGENTS.md
Learn from past bugs and lessonsdocs/solutions/
Find answers to common questionsdocs/faq.md
Report a vulnerabilitySECURITY.md
Get helpSUPPORT.md

License

Apache-2.0

相关项目 · GitHub 热门

查看全部「GitHub 热门」项目 →