dsh-eval-harness: Making every number in Agent application evaluation reproducible
dsh-eval-harness is a DSH skill package maintained by LTJ002 (a non-host functional plugin) that provides reproducible evaluation methodologies and runnable scripts for Agent applications. The criteria rely solely on external facts such as audit tables and structured logs, employing positive and negative bidirectional assertions, with baselines compared by absolute count. Among the three strictness levels (offline / live-strict / live), security categories are subject to zero tolerance, and mutation testing is used to verify that the evaluator itself is functioning correctly. It covers RAG retrieval, RAG generation, tool calling, prompt security, and multi-turn dialogue. Being framework-agnostic, it only requires programmatic access to the dialogue and queryable side effects. The installation command is `dsh plugin add https://github.com/LTJ002/dsh-eval-harness`. Datasets must be replaced entirely, and the three interfaces in `eval_online.py` need to be rewritten according to the specific project. An exit code of 2 indicates that the result is unavailable rather than a failure in quality. The documentation is in Chinese, licensed under MIT, and version 1.0.0.
Read More