Introduction¶

When running training tasks on Windows using DSH, you often encounter a situation where you press stop, the agent exits, but some python.exe in Task Manager is still holding GPU memory, and the GPU gets filled up again after a while. It looks like “stopped but started itself again,” but actually two things are layered together: Windows doesn’t cascade kill processes. Stopping the subagent only kills the shell it spawned directly. The bash -> conda run -> python -> worker underneath all become orphans, continuing to hold the CUDA context; meanwhile, the for loop or --retries N in the training script starts another round, so the memory gets filled repeatedly.

lvyue1314/dsh-gpu-guard is a DSH plugin written for this problem. It does two things: calculate who the “orphan root” is, and optionally let a watchdog keep an eye on it, ending the entire tree from the root after the orphan tree survives past a grace period.

Below is an introduction to how it works, how to install it, and typical usage.

Overview¶

A community-maintained DSH plugin, maintained by lvyue1314, classified as admin-security, under the MIT license. It targets Windows + Node >= 22 + DSH >= 0.2.1-rc.5 environments.

In one sentence: Scan processes holding GPU compute contexts, walk up the ancestor chain to find the “orphan root” where the initiating process no longer exists, and then use taskkill /T /F to end the entire tree starting from that root. Default is dry-run; true killing only happens with explicit confirmation or the watchdog enabled.

A typical leaking chain looks like this, with process names and hierarchy as tested by the plugin author:

<launcher shell>            <- This layer is killed when the agent stops
 └ bash run_train.sh        <- Everything from here becomes orphaned
   └ conda run -n myenv python -m pkg.train --retries 2
     └ cmd.exe /d /c %TEMP%\tmpXXXXXXXX.bat
       └ python -m pkg.train --worker
         └ python train.py  <- Holds CUDA context, memory and utilization are occupied

At the command line level, taskkill /T /F /PID <root> can solve this, but the difficulty is that you have to know who the root is first, and you have to act before the script launches the next round. This plugin automates these two steps.

Detection Rules¶

For every PID holding a GPU compute context, the plugin walks up the ancestor chain and classifies it according to the following rules.

1. A parent PID on the chain no longer exists in the process table → Classified as an orphan; root is the topmost living process on the chain.
2. Parent process exists, but its start time is later than the child’s → Indicates PID reuse; the original parent is actually dead → Classified as an orphan.
3. Reaches 0 or self (System, service host) → Considered normal, leave it alone.
4. Cycle in chain or length exceeds reasonable depth → Unclear ownership, do not kill.
5. Any hit on protect regex (default jupyter, nohup, code.exe, explorer.exe, services.exe, svchost.exe, case-insensitive, matches process name and full command line) → Skip.

The true signal is “the process that initiated it is gone,” not “it has been running for a long time.” So daemons launched via nohup or scheduled tasks are not affected, unless you remove nohup from the protection list.

It kills the orphan root at the very top of the tree, not the deepest worker. This way, retry loops in scripts and supervisors are taken out together, avoiding situations where “worker is killed but parent starts another.”

Core Features¶

The plugin exposes three usage interfaces, sharing the same scanning and detection implementation (lib/core.js is a pure function layer shared by the plugin and command line, no second logic).

The command-line tool gpu-guard provides several subcommands:

node bin/gpu-guard.mjs status              # Who is occupying GPU, who are the orphans
node bin/gpu-guard.mjs reap                # Preview: which trees will be killed
node bin/gpu-guard.mjs reap --apply        # Real kill
node bin/gpu-guard.mjs watch --apply       # Persistent watchdog (default 60s scan, only acts if orphan > 60s old)
node bin/gpu-guard.mjs status --json       # For scripts

The JSON output structure is { at, gpu: { gpus, holders }, holders, roots, warnings }. The CLI version doesn’t depend on DSH running; it’s ready to use anytime after installation.

The plugin provides three tools in the chat:

• gpu_status: Memory/Utilization/Temperature per card + process holding GPU + ancestor chain + which are orphans.
• gpu_reap: End orphan trees, default dry-run, confirm: true needed for real kill; can process specific pids.
• gpu_guard: Watchdog status, start, stop, configure (interval, grace period, protect regex).

The watchdog is enabled via a config file:

node bin/gpu-guard.mjs config --auto     # Writes to ~/.dsh/gpu-guard.json: { "autoReap": true }

The timer starts when the plugin loads. When autoReap=false, the watchdog only logs orphan trees to the log and stderr, taking no action. An orphan tree must survive continuously for graceSeconds (default 60s) before being processed; brief orphans (e.g., a command just finishing) won’t be mistakenly killed.

Installation and Enabling¶

Link the plugin into the profile:

cd ~/.dsh/profiles/desktop
pnpm add link:/path/to/dsh-gpu-guard

Then choose one of the two mounting methods.

Method A, writing into the profile’s package.json:

{
  "dependencies": { "dsh-gpu-guard": "link:/path/to/dsh-gpu-guard" },
  "dsh": { "profile": { "bundles": ["@deepseek-ai/dsh-base", "@deepseek-ai/dsh-web-app", "dsh-gpu-guard"] } }
}

Method B, writing into the profile’s cordis.patch.yml:

- insert:
    - id: dsh-gpu-guard
      name: file:///path/to/dsh-gpu-guard/lib/entry.js

If the profile is configured with patchReload: live, the three tools will be available on the next model call; otherwise, restart DSH. The README also provides an example examples/scheduled-task.ps1 of using a scheduled task instead of a daemon process to run the watchdog.

Configuration¶

The config file is located at ~/.dsh/gpu-guard.json, and the path can be changed via the environment variable DSH_GPU_GUARD_CONFIG:

Logs are appended to ~/.dsh/gpu-guard.log (environment variable DSH_GPU_GUARD_LOG). Plugin status can also be checked directly: GET http://127.0.0.1:<port>/dsh-gpu-guard/state.

Applicable Scenarios and Notes¶

Suitable scenarios: Running CUDA training or inference tasks on Windows via DSH, where GPU memory remains occupied after the agent stops, or the GPU gets filled up every few minutes. The plugin only handles orphan trees holding GPU compute contexts; the node is Windows, and the CLI version works without DSH.

A few things to note:

1. The plugin runs with the current DSH process privileges and uses taskkill /T /F to kill processes. You should read the source code before installing to confirm behavior and verify the license is MIT.
2. By default, interfaces don’t kill; gpu_reap requires explicit confirm: true, and the CLI reap requires --apply.
3. Under WDDM, nvidia-smi’s per-process memory shows as N/A and only lists compute contexts. The plugin reads from Windows Performance Counters (\GPU Process Memory(*)\Dedicated Usage) to include graphics contexts; on rare server SKUs without these counters, it issues a warning and falls back to N/A.
4. Graphics contexts (hardware acceleration in browsers/chat apps, desktop composition) are marked as G, compute contexts as C, both as C+G. If the sum of performance counters matches the total reported by nvidia-smi, it indicates no invisible debris remains, and the remaining memory is usually occupied by other persistent programs.
5. PID reuse is blocked by startup time validation; in extreme cases (PID reused within the same second), there is still a window, killTree will re-read the process table before killing.
6. Does not handle DSH background jobs: If a process is started with run_in_background: true, it should survive the subagent by design. The plugin doesn’t judge semantics, it acts based on the hard fact of “whether the ancestor chain is broken”.

The author also states in the README that this is a temporary fix; the root solution is to use Job Objects (JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE) in the dsh-subprocess-local layer to take the whole tree down when the parent dies. Related issues point to the existing spawnCurrentTokenJobProcess primitive in @deepseek-ai/dsh-win32-process.

Conclusion¶

The issue of GPU daemons that stop but don’t die is common in the Windows + DSH combination. The value of dsh-gpu-guard lies in turning “finding the orphan root” from manual troubleshooting into a single command or a tool call, and it doesn’t mistakenly kill things under default configuration. For such tasks, it is worth trying the dry-run mode first to see if the judgment is accurate, and then deciding whether to enable the watchdog.

Plugin Directory Page: https://www.skillhub.cn/plugins/lvyue1314/dsh-gpu-guard
GitHub Repository: https://github.com/lvyue1314/dsh-gpu-guard