Preface¶

In DSH plugin development, after tools or plugins are updated, it’s often necessary to answer: does the same input still produce the same behavior, and are the execution time and output stability comparable? Looking at a single run result is insufficient; typically, you need fixed test cases, a fixed measurement methodology, and reports that can be compared repeatedly. dsh-benchmark is a DSH plugin built around these needs to provide deterministic benchmark evidence.

What This Is¶

dsh-benchmark is a benchmarking plugin for DeepSeek Harness (DSH) tools and plugins, positioned as “reproducible, deterministic benchmark evidence.” The account appearing in the repository path and installation commands is dongsheng123132. It provides inspection, execution, and comparison capabilities around fixed test cases, and generates content-addressed reports for baseline regression comparison.

Core Features¶

Below are the main capabilities provided:

• dsh_benchmark_inspect: Inspects protocol metadata and fingerprints without executing the target.
• dsh_benchmark_run: Runs fixed test cases and writes content-addressed reports.
• dsh_benchmark_compare: Compares the current report against a baseline report, combining manifest thresholds for judgment.
• CLI: Provides three command types—inspect, run, and compare.
• MCP: Exposes two interfaces—benchmark_manifest_lint and benchmark_report_address—as a standalone stdio MCP server.
• Reporting and scoring: Uses versioned deterministic scoring, content-addressed reports, and baseline regression comparison.

Installation and Enablement¶

First, install the plugin in DSH:

dsh plugin --profile benchmark add github:dongsheng123132/dsh-benchmark

After installation, it registers the dsh_benchmark_inspect, dsh_benchmark_run, and dsh_benchmark_compare tools in DSH. The runtime requires Node.js 22+. Apart from the optional DSH tools SDK peer dependency, it uses no additional runtime dependencies or install lifecycle scripts.

Typical Usage¶

First, inspect the manifest and target fingerprint—this step does not execute the target:

dsh-benchmark inspect --root /workspace --manifest benchmark.json

Then run the fixed test cases and write the outputs to the specified artifact directory:

dsh-benchmark run --root /workspace --manifest benchmark.json --artifact-dir benchmark-artifacts

When you already have both baseline and current reports, use the following command to compare:

dsh-benchmark compare --root /workspace --manifest benchmark.json --baseline benchmark-artifacts/baseline.json --current benchmark-artifacts/current.json --artifact-dir benchmark-comparisons

The repository also includes example manifests. To run an example from the repository directory:

node bin/dsh-benchmark.mjs run --root . --manifest examples/benchmark.example.json --artifact-dir artifacts

Exit codes have fixed meanings: 0 means pass; 2 means reports or comparison results were written but the scorer determined failure; 1 means a manifest or operation error.

Use Cases and Considerations¶

This is suitable for fixed benchmark validation of DSH tools or plugins: use the same manifest to pin test case inputs, run to obtain reviewable reports, and compare against baselines in subsequent versions.

Things to note before use:

• Only run trusted benchmark executables. Its security isolation is not an OS sandbox for malicious code.
• Use shell: false, avoiding command strings or shell interpolation.
• Workspace paths must not escape workspaceRoot through path traversal or symbolic links.
• Child processes receive a minimal deterministic environment, not inherited environment secrets.
• Manifest fields containing secrets are rejected, including tokens, cookies, authorization, credentials, and custom environment secrets.
• Reports contain hashes and measurements, not command input or output bodies.
• The MCP interface accepts bounded inline JSON, does not execute commands, and does not read from or write to the file system.
• The plugin runs with the same permissions as the current DSH process. Review the source code and license before installation. The license is MIT.

Conclusion¶

The value of dsh-benchmark lies in transforming benchmark results from a single run into reviewable, comparable evidence: first inspect, then run, and finally compare. GitHub repository: https://github.com/dongsheng123132/dsh-benchmark. The community directory page was not included in the currently verified materials, so only the verifiable GitHub repository link is listed here.