dsh-benchmark: Generating Reproducible Benchmark Evidence for DSH Tools and Plugins

The dsh-benchmark is a benchmarking plugin for tools and plugins targeting the DeepSeek Harness (DSH), designed to provide reproducible and deterministic benchmark evidence. Centered around fixed use cases, it leverages three core commands—`inspect`, `run`, and `compare`—to enable protocol metadata inspection, fixed use case execution, and baseline regression comparison, while generating content-addressed reports. After installation, it registers corresponding tools within DSH and supports interaction via both the CLI and a standalone stdio MCP server interface. The workflow is as follows: first, inspect the manifest and target fingerprint; then, run fixed use cases to produce artifacts; finally, compare the current report against the baseline to evaluate performance or behavioral stability. Exit codes carry clear meanings, respectively indicating pass, scoring failure, or operational errors. This plugin is applicable

Read More