Preface¶

When evaluating LLM/Agent applications, the common problem isn’t the lack of numbers, but the reluctance to use them: criteria read the model’s own answer wording rather than server logs; negative assertions are trivially true on empty answers, so “no leakage” is misjudged as PASS; evaluation scripts may contain bugs but report all green. After fixing RAG, tools, or prompts, we want to know “whether there is degradation,” but current methods mostly just give a “looks correct” total score.

The ltj002/dsh-eval-harness introduced below is a set of Agent application evaluation skills for DeepSeek Harness (DSH), maintained by LTJ002, under the MIT license, and categorized as admin-security in the community directory SkillHub. It provides methodologies and runnable scripts for the problems mentioned above: criteria only use external facts, bidirectional assertions, baseline comparison by count, zero tolerance for security cases, and use mutation testing to verify the evaluator itself isn’t broken.

What is it¶

One-sentence positioning: dsh-eval-harness is a methodology (readable by the model via SKILL.md) combined with runnable or editable evaluation implementations (Python scripts in assets/, JUnit disk writer in Java), packaged as a DSH skill pack.

It is not a host feature plugin. The package.json in the repository only uses dsh.bundle to register the skill into ctx.skills, does not provide dsh.client, does not modify the host UI or commands, and does not inject runtime code. After installation, the skill appears as a bundled provider in ctx.skills (rank 600).

It is also not an evaluation platform: it does not connect to your retrieval service or model gateway; it only handles “calculating metrics and judging degradation based on external facts you export.”

Framework-agnostic is its design premise. Agent applications built with LangChain, LangChain4j, Spring AI, AutoGen, CrewAI, Dify, etc., can use the same criteria as long as they meet two premises:

1. Programmable access to conversations — expose callable conversation interfaces (HTTP/SSE both okay);
2. Queryable side effects — tool calls, write operations leave queryable records (audit tables, structured logs, or DB).

Criteria read from these external facts, not the model’s self-statements. Switching frameworks only requires changing the adapter layer, not the criteria.

Core Capabilities¶

Look at it in five parts.

1. Three-layer evaluation: Offline assertions + Live model execution + Baseline comparison.

• eval_online.py: Online evaluation skeleton, four suites tool / inject / answer / multi-turn, corresponding to criteria on the real call chain;
• eval_rag.py: Retrieval layer metrics Recall@K, MRR, nDCG@K, Precision@K;
• eval_rag_answer.py: Generation layer Faithfulness, citation accuracy, completeness, contradiction detection, based on human annotation pairs, no model self-evaluation;
• eval_baseline.py: Compare with baseline by use case, red if any item drops.

2. Bidirectional criteria, avoiding “false green”.

Criteria are either positive (answered what should be answered) or negative (didn’t answer what shouldn’t, didn’t appear what shouldn’t). Negative criteria for security cases are especially important: answering sensitive content is a failure, not “didn’t answer correctly.” eval_online.py hardcodes five anti-false-positive rules into code:

# ① 工具是否被调用 → 查审计日志，不看回答措辞
called = tool_calls_of(conversation_id)

# ② 降级跳过：负向断言在空回答上恒真，不能算 PASS
if is_degraded(text, cost):
    continue

# ③ PII 判"未打码"，不判宽禁词
UNMASKED_PHONE = re.compile(r"(?<!\d)1[3-9]\d{9}(?!\d)")

# ④ 观测不到 → None，不进分母也不判失败
called = tool_calls_of(cid)
if called is None and needs_tool:
    unobservable += 1
    continue

# ⑤ 401/402 基础设施异常单独计 errors：不因重试恢复，退出码升 2
except urllib.error.HTTPError as exc:
    if exc.code in (401, 402):
        errors += 1

The second rule addresses the empty answer trap: the model didn’t answer, so the negative assertion naturally holds, looking like “no leakage,” but actually nothing ran. The fourth rule is denominator discipline: unobservable data is not included in the denominator, avoiding passing “because it wasn’t tested.”

3. Three strictness levels, zero tolerance for security.

The trade-off for live-strict is clear: the randomness in model output can explain capability fluctuations but cannot explain leakage.

4. Mutation testing, verifying the evaluator itself isn’t broken.

assets/mutation_check.py corrupts the implementation on a temporary copy to verify that assertions will indeed turn red. This is a direct response to the risk of “the evaluation script lying to itself.” assets/selftest.py is a regression self-test for criteria and denominator discipline; it must be run after changing any script. npm test also includes compiling and running smoke tests for EvalResults.java (explicitly skipped if no JDK is present).

5. Governance and Traceability.

eval_governance.py provides a three-in-one set for root cause stratification, dataset governance, and Judge calibration — banning entry if Judge consistency and kappa are not up to standard. eval_manifest.py creates sha256 fingerprints for prompts and datasets and records config diffs; eval_report.py generates a summary report but is read-only and has no gatekeeping. Experimental parameters are jointly constrained by eval_manifest.py and ci-eval.yml.example.

Installation & Enablement¶

The installation command comes from the repository README, in the form of a repository URL, do not concatenate it as dsh plugin add github:owner/repo:

dsh plugin add https://github.com/LTJ002/dsh-eval-harness

After installation, you don’t need to copy the skill to the project’s .agents/skills; you can reference it within the session via the skill directory or @ai-eval-harness.

It is worth noting that package.json intentionally does not write dsh.compatibility.dsh (host version range). The reason given in the README is that this package only depends on the public skill registration interface and has not been tested against multiple versions of the DSH host, so it does not declare a compatibility range. engines.node >= 20 is a build-time requirement for this repository’s development and CI, not a requirement for the DSH host. Current version 1.0.0.

Typical Usage¶

Step 1, build the dataset. assets/datasets/*.example.jsonl is a format template, not a test question bank. The question phrasing comes from the example project used during writing (photo orders) and is irrelevant to your business. Replace it with your own following these steps:

1. Copy *.example.jsonl to a real dataset under eval/datasets/, replacing questions, expected documents, expected answers, and forbidden words entirely with your business content;
2. Expected values (expected / mustContain / expected tools) must have a real source (knowledge base files, API docs), do not write them based on feeling;
3. Cover positive, negative, multi-turn, and retrieval annotation pairs; format remains unchanged.

What doesn’t need rewriting is the criteria engine: the metric math in eval_rag.py, the strictness in eval_baseline.py, the five rules in eval_online.py, and the gatekeeping mechanism in mutation_check.py are unrelated to the business, so zero changes when switching projects.

Step 2, connect the adapter layer. eval_online.py is the skeleton; login() / chat() / tool_calls_of() must be rewritten according to your project’s contract. assets/adapters.py points to the project-side module via EVAL_ADAPTER, with unimplemented functions falling back to the built-in base. The vocabulary is also externalized: markers.py plus markers.default.json (7 groups) must be copied together with eval_online.py / eval_governance.py, otherwise imports will cause actionable errors; use EVAL_MARKERS to override for a different business vocabulary.

Step 3, run baseline comparison and self-tests:

python skills/ai-eval-harness/assets/selftest.py          # 判据与分母纪律回归自检
python skills/ai-eval-harness/assets/mutation_check.py    # 变异测试：断言必须翻红
npm test                                                   # bundle 结构冒烟，含 Java 落盘器编译运行

Exit code conventions:

Exit code 2 is a point easily overlooked in this design: separating “result unavailable” from “quality failure” avoids misjudging “poorly configured environment” as “application degradation.”

Scenarios & Precautions¶

Suitable for teams that: have LLM/Agent applications in continuous iteration (fixing RAG, tools, prompts), have queryable server-side side effect records, and want to use reproducible criteria instead of model self-evaluation for gatekeeping.

Several boundaries must be clear before getting started:

• eval_online.py is the skeleton; interfaces and datasets must be rewritten for the project, assets/datasets/*.example.jsonl must be replaced entirely;
• eval_rag.py / eval_rag_answer.py do not connect to any retrieval or generation service; annotations like retrieved, supportFacts, citations, requiredPoints are exported by your system or manually annotated;
• The regression self-test for judgment logic monkeypatches module-level functions, verifying judgment discipline, not the real network path; real connectivity still requires running an online evaluation once;
• Judgments like whole-sentence semantic similarity or whether an answer is “natural” are not in the tool; they require model self-evaluation or manual review and are not gatekept according to the first principle; eval_governance.py judge is only responsible for calibrating whether the Judge is qualified to act as an auxiliary score;
• The criterion design for the multi-turn suite and tool five stages has been implemented (run_multi_turn_suite / check_tool_stages), but the corresponding datasets need to be rewritten for the business, and the complete side effect recording interface tool_call_records() needs to be connected to the project’s audit table.

This skill pack runs with the permissions of the current DSH process, registering methodologies and scripts, not host capabilities. Before installing, you should read the repository source code yourself to confirm that the criteria design in SKILL.md and the implementation in assets/ are consistent with project compliance requirements, and that the MIT license is acceptable. Additionally, the repository currently has 0 stars and no forks; it is a new project. The four local validations claimed in the README (structure smoke test, criteria self-test, mutation test, Java smoke test) are subject to the repository’s CI and local execution output.

Related Links¶

• Directory page (community directory, independent site, no official affiliation with DeepSeek / Hypi): https://www.skillhub.cn/plugins/LTJ002/dsh-eval-harness
• Source code and full documentation: https://github.com/LTJ002/dsh-eval-harness
• English README: README.en.md inside the repository