Foreword¶

The philosophy of DSH is “everything is a plugin,” and the capabilities of client orchestration Agents are basically filled by plugins. On macOS, there are two common ways to let an Agent operate the local machine: one is to screenshot and rely solely on coordinates to guess the position, which is inaccurate and doesn’t self-correct after clicking wrong; the other is to wrap an external driver process, adding a runtime dependency during deployment.

dsh-computer-use is a client plugin released by maintainer lqq151510, categorized under clients for macOS. It synthesizes “read screen → operate → verify” into a set of computer_* tools: reading the screen uses the Accessibility Tree plus OCR, operating prioritizes semantic positioning over coordinates, and parts of the screen without accessible elements fall back to CoreGraphics pixel operations. The premise is that the local machine is macOS and that DeepSeek Harness must be granted Accessibility and Screen Recording permissions.

What is this¶

One-sentence positioning: Adding macOS desktop control capabilities to DeepSeek Harness, allowing the Agent to screenshot and see the desktop, and move the mouse, click, drag, scroll, type, and press shortcuts.

It registers eight tools:

computer_permissions
computer_screenshot
computer_windows
computer_mouse
computer_keyboard
computer_sequence
computer_read
computer_act

Among them, computer_read is the core: one call returns both the Accessibility Tree (element role, title, value, actions, screen coordinates, stable index) and Vision OCR text with its coordinates. Both channels share the same set of screen point coordinates, so the model doesn’t need to choose a channel first or perform pixel-to-coordinate conversion itself.

computer_act directly presses/focuses/sets_value/clicks based on index or visible title. Since it doesn’t move the real pointer during the process, it can operate background windows. Each operation carries an “element fingerprint” (role + title + position), which is used to re-locate the same control when interface changes between two calls cause path drift.

The Eight Registered Tools¶

Installation and Enablement¶

A note needs to be made about the installation command first: the README checked in this article only provides the uninstall command; the directory page is rendered by a script, and the scraped result does not contain verifiable installation command text. Therefore, the following does not splice commands like dsh plugin add ..., only gives verified clues.

Uninstall command (from README):

dsh plugin --profile desktop remove dsh-computer-use
rm -rf ~/.dsh/computer-use

By reverse inference, after installation you need to ensure the profile contains the plugin (README mentions the plugin is link:ed into the profile). Repo info is at the end directory page; before installing, confirm directory structure consistency according to the files list in the README:

index.js
lib/*.js
native/ComputerUse.swift
native/CUInput.swift
cordis.patch.yml
locale/*.json
icon.svg

Auto load dependency Node 22+ (the engines field in package.json), runtime platform only declares darwin. In cordis.patch.yml, high-risk items can be turned off, for example, setting allowInput to false makes it “screenshot only, disable mouse and keyboard”:

- id: computer-use
  name: dsh-computer-use
  config:
    allowInput: true       # false = 只截图，禁用鼠标键盘
    resolution: points     # points（推荐）| pixels
    defaultDisplay: 1
    blockedApps:           # 这些 app 在前台时拒绝发送输入
      - 1Password
      - Keychain Access
    typeMaxChars: 8000
    warmup: true           # 启动时后台预编译助手

blockedApps defaults to blocking 1Password and Keychain Access to avoid auto-typing in credential windows.

After changing lib/*.js, you need to reload the plugin:

# 在 DSH 里调 plugin_manager：
#   action=set_plugin target=include:computer-use enabled=false
#   action=set_plugin target=include:computer-use enabled=true

Native Swift changes don’t require reloading the plugin; lib/helper.js will automatically recompile based on the source code hash.

First Authorization¶

System Settings → Privacy & Security, check two items for DeepSeek Harness:

1. Accessibility — required to send mouse/keyboard events;
2. Screen Recording — required for screenshots.

Check them and restart Harness. Running computer_permissions once can show if they are complete. Passing openSettings: true will open the corresponding system panel directly.

Typical Workflow¶

The recommended order in the README is “read screen first, then operate,” four steps:

1. computer_read: Get elements (each with index and centerX/centerY) and text (OCR lines + coordinates);
2. If semantic operation is possible, computer_act action=press target=<index> (uses AX, can click background windows, not afraid of occlusion);
3. For controls not in AX, use computer_mouse by coordinate and computer_keyboard to type;
4. After changes, computer_read again to verify results (read input box value, read calculator display).

There are three reproducible examples in the README. The first is Calculator: Use computer_act to press visible titles 7, ×, 6, = in sequence, not by coordinates; then use computer_read to verify 7 × 6 and 42. This example also verifies the fingerprint mechanism — the Calculator’s clear button title changes between AC and C, relying on fingerprints to rebuild the path to continue clicking.

The second is background writing: When TextEdit is in background (Calculator is foreground), computer_act action=set_value writes text directly, then using osascript to read back confirms consistency.

The third is OCR channel: Open a PNG (using Preview), Accessibility Tree has no text nodes; computer_read uses Vision to recognize line by line and returns x / y / width / height for each line. Returned coordinates are screen points, can be passed directly to computer_mouse’s centerX / centerY, or used to determine what the line is before deciding to click.

Native Helper and No Runtime Dependency¶

Screenshot uses the system built-in /usr/sbin/screencapture; mouse, keyboard, and Accessibility operations are done by a Swift helper (native/ComputerUse.swift), compiled with xcrun swiftc on first use, cached in ~/.dsh/computer-use/bin/dsh-cu, cached by source code hash, auto-recompiled on upgrade. The repository does not package a DMG, does not pull a driver process, and has no precompiled binaries. It deliberately does not depend on pyautogui / cliclick / robotjs, a single ~120KB static executable, linking only AppKit/CoreGraphics, commands pass JSON via argv, JSON via stdout.

If a compiler is not found, the tool returns “run xcode-select --install” instead of failing silently.

There is also a lightweight helper native/CUInput.swift for mouse and keyboard, linking only CoreGraphics, running as a unix socket daemon (exits idle 15 min) after first call, saving the process startup overhead per call. README records local latency (M series, daemon warm):

game: true cuts event intervals (press-down-press-up 3ms, rapid-fire interval 40ms) to trade for lowest latency. README explicitly states these are means to “send input fast and stable,” not to bypass platform CAPTCHA or anti-cheat.

Coordinates and Known Limitations¶

Coordinates are unified as screen points, origin at the top-left of the main screen. Screenshot defaults downsampled to point size, so “pixels on image” equals “points clicked”; window or area screenshot returns origin and pointsPerPixel for conversion. Screenshot defaults to $TMPDIR/dsh-computer-use/, auto-cleans after 20 minutes.

Usage boundaries:

• Only macOS implemented, other platforms’ tools will explicitly state unsupported.
• Input is global synthesized events, sent to the current foreground window, the plugin does not switch windows itself. Use computer_windows first to confirm, activate if necessary.
• type uses Unicode events to insert text directly, bypassing IME candidates; may be invalid for games or remote desktops that only recognize physical keycodes.
• Self-drawn interfaces (some Electron, games, canvases) have few interactive elements, fallback to OCR + coordinate clicking.
• computer_read OCR defaults to zh-Hans + en-US, precise mode, original resolution; window level ~0.3–1 sec, full screen may take ten seconds. Use computer_read when text is present, don’t guess via screenshot. OCR language order matters, zh-Hans must be first, en-US first degrades Chinese recognition quality.
• Element not found returns element-not-found and suggests re-reading screen or using coordinates; missing permissions will specify which one and where to open in system panel.

Suitable Scenarios and Notes¶

Suitable for: Desktop automation on macOS, needing to read/write background windows, or scenarios where the Agent should be driven by interface semantics rather than coordinates. The built-in Swift helper means all native logic is auditable, no need to trust external binaries.

Notes: The plugin runs with the permissions of the current DSH process. Keyboard and mouse events are global synthesized events, so it can drive the current foreground window. Before installing, it is recommended to read the GitHub repo source code and check the license (package.json declares MIT), and confirm if allowInput and blockedApps in cordis.patch.yml match your security expectations.

The community directory is an independent site with no official affiliation to DeepSeek / Hypersphere, this article only organizes the public implementation details of this plugin.

Summary¶

The value of dsh-computer-use lies in converging “read screen” into one tool, prioritizing “operation” on semantic positioning, and then connecting the two paths (AX and pixels) with a set of point coordinates. The whole solution has no external runtime, just two system permissions on macOS to run.

• Directory Page: https://www.skillhub.cn/plugins/lqq151510/dsh-computer-use
• GitHub: https://github.com/lqq151510/dsh-computer-use