Preface¶

When doing Agent automation in DSH, a common problem encountered when dealing with Windows desktop programs is: the Agent can read/write files and run commands, but cannot retrieve content from within the window, nor can it simulate clicking and typing on behalf of humans. Writing scripts can solve part of the problem, but writing a separate automation script for every interface is brittle. This plugin provides this functionality as a DSH plugin: it adds a “Computer” page to the left sidebar, through which the Agent can observe windows and interact with them.

It is maintained by Kinfxhk, categorized under Workflow, licensed under MIT, and the repository address is https://github.com/Kinfxhk/dsh-computer-use . The content below is verified; commands and paths in the text are taken from the plugin README and package.json.

What is it¶

In one sentence: This is a Windows-specific DSH plugin that adds a “Computer” page to the left sidebar, allowing the Agent to observe and operate windows using the local Cua driver.

It solves the problem of: On Windows, enabling an Agent to see a specific window, locate clickable elements, click, type, press keys, and scroll, while being controlled by a human switch.

The current version is 1.0.1 (see package.json, name is @local/computer-use), and it requires DSH 0.2.1-rc.5 or higher.

Core Features¶

The sidebar “Computer” page contains a switch, driver status, and recent operation records. The Agent side exposes a computer_use tool with the following actions:

Several implementation details are worth noting:

• Every operation must specify pid + windowId; the target is always the specific window you specified.
• observe returns elementToken; prioritize using the token for positioning, coordinates are just a fallback.
• Background input is prioritized; foreground is required if background fails; foreground will briefly steal focus.
• The plugin starts its own private process, communication goes through the named pipe \\.\pipe\dsh-computer-use, uses --permission-mode standard, does not connect to other Agents’ drivers, and does not use unrestricted mode.
• Typed content is not written to page logs, only the length is recorded; the page retains the last 24 operations, with a maximum of 8 screenshot previews.

Explicitly unsupported capabilities: Clipboard, kill process, full desktop input, password/payment/permission dialogs.

Installation and Activation¶

First install dependencies, then install the plugin.

The plugin itself is Windows-specific and requires the Cua driver (cua-driver.exe). The official installer:

irm https://cua.ai/driver/install.ps1 | iex

After installation, the plugin defaults to looking for:

%LOCALAPPDATA%\Programs\Cua\cua-driver\bin\cua-driver.exe

If the driver is located elsewhere, simply set the environment variable CUA_DRIVER_BIN to point to that exe.

The repository does not host third-party cua-driver.exe (28 MB, not the copyright of this plugin), so you must use the official installer above, which also makes future updates convenient.

Plugin installation uses DSH’s plugin_manager, with target filled with the absolute path of the download directory:

plugin_manager(action: "install_bundle", target: "C:\\path\\to\\dsh-computer-use")

You can git clone the repository, or select Code → Download ZIP directly on GitHub. After installation, restart DSH, and “Computer” will appear in the left sidebar.

Typical Usage¶

After installing the driver, click “Start driver” on the “Computer” page, confirm the switch is on, and the Agent can call the tool. A common call sequence is as follows:

1. computer_use status: Confirm switch and driver status.
2. windows: Find the windowId of the target window.
3. observe with that pid + windowId: Get the screenshot and element tokens.
4. click the element with elementToken to locate; use type when input is needed, key for keys, and scroll for pagination.

Note that the observe query parameter can be used to filter elements; when unsure which element to click, add filtering first, then select a token from the returned clickable elements.

elementToken in click should take priority over coordinates; the coordinate method is a fallback, confirm the screenshot and window position are consistent at that time before passing.

Security and Applicable Scenarios¶

The switch is on by default. Click to turn it off: after closing, the Agent’s click/type/key/scroll will be rejected directly, and observe will only execute after being reopened. Only the “Computer” page can change the switch and start the driver; the Agent itself cannot.

The plugin will only operate on the one window you specified, no full desktop input, no clipboard, no process kill, and will not read password fields. The README clearly states it should not be used for payment or permission dialogs.

Applicable scenarios: Need to have the Agent stably complete window operations on Windows, and you want to be able to cut off with one click at any time. Not applicable: Scenarios requiring cross-window global input, clipboard copying, process management, or handling critical permission popups.

Also note that the plugin runs with the permissions of the current DSH process. It is recommended to read the source code (MIT license) once before installation to confirm the source; Cua driver is a third-party 28 MB binary from https://cua.ai/driver/install.ps1, and installation behavior follows that installer’s instructions and is not part of this plugin’s code.

Conclusion¶

The value of this plugin lies in: changing Windows window operations from “writing scripts” to a toggleable tool page in the DSH sidebar, giving the Agent a clear and auditable operation surface, and allowing humans to turn it off at any time. Version 1.0.1 fixed three issues: undefined causing the entire tool to error, failure to reconnect after the driver session ends, and incorrect status judgment when attaching to an already running daemon.

Index page: https://www.skillhub.cn/plugins/Kinfxhk/dsh-computer-use
GitHub: https://github.com/Kinfxhk/dsh-computer-use