Preface¶

In the DSH plugin ecosystem, one type of demand is very practical: letting others see a glimpse before the assistant finishes speaking. dsh-quality-guard, maintained and categorized as a workflow by wdj20050927, does exactly this—the assistant’s reply is judged by a second model before actually being sent to you; if it fails the criteria, it is sent back for rewriting.

The difference from existing approaches lies in where the criteria come from. Mechanical criteria (word count, regex, format) cannot control off-topicness, fluff, or term stacking: a completely irrelevant reply can still be compliant as long as it is short enough and contains a link. This plugin delegates the criteria to an independent model invocation. The basis provided in the README is “Self-correction within the same context is largely ineffective” (Huang et al., Large Language Models Cannot Self-Correct Reasoning Yet, arXiv:2310.01798), while verification in independent rounds is effective (CoVe, arXiv:2309.11495). The cost is one additional model call per round.

Introducing the plugin below. It consists of two parts: engine/ (the hook itself) and panel/ (the visualization panel, optional). Current version 1.6.1, license MIT, repository at https://github.com/wdj20050927/dsh-quality-guard.

What Are the Two Parts¶

Changes to the panel take effect immediately without relying on hot reload: the engine re-runs stat + reads settings.json before every judgment, using mtime:size as the cache key; once the file changes, the next round uses the new value.

How a Check Works¶

1. When a round is about to end, the engine reads turn-stopping and first reads settings.json.
2. The judge is sent the criteria + original question + draft, returning per-item 0-10 scores (pressed onto the first line of output) and FIX: suggestions.
3. The engine converts the scores to 0-100: the denominator is always ‘count × 10’, and unanswered items are forced to 0; if any item is < 6 points, the total score ceiling is forced to 59, and it is marked as fail.
4. If it fails and the number of reworks has not exceeded maxReworks, it steers the model to rewrite according to FIX; otherwise, it passes through.

If the judge cannot access the LLM, times out, errors out, or has no route, it always takes the pass-through branch (fail-open). However, scores that have already started outputting are not discarded: interrupted flow, max-tokens, or partial JSON first attempt to rescue the score; if no score is obtained, a supplementary judgment is made; only if neither is obtained does it enter the unknown exit, at which point onUnknown decides whether to pass or rework.

Any single criterion below 6 points is considered failed (since v1.5.0). Items below 6 convert the full score to 59, which is definitely lower than any passing line of ≥ 60; a score of 6 itself does not trigger a veto (6 points means halfway done), only 0/2/4 are converted. In other words, each criterion has a veto power.

Installation and Activation¶

Install the panel first, then the engine. The panel can be installed with a single command:

dsh plugin --profile web add github:wdj20050927/dsh-quality-guard

The package.json in the repository root declares dsh.bundle.patch: ./panel/cordis.patch.yml, so after installing dsh, it automatically inserts the quality-guard line into the profile layer. To confirm it’s inserted, run:

dsh --profile web --dump-config

The output should show:

# == dsh-quality-guard
- id: quality-guard
  name: dsh-quality-guard

The engine cannot be installed like this; it needs to be mounted via an absolute path and specify its own configuration path. Copy the entire engine/ to the plugins directory:

~/.dsh/plugins/selfcheck-guard/

Then append an insert block to the end of ~/.dsh/profiles/<profile>/cordis.patch.yml, replacing the path with your own absolute path (use forward slashes for Windows):

- insert:
    - id: selfcheck-guard
      name: 'C:/Users/YOUR_NAME/.dsh/plugins/selfcheck-guard/index.mjs'
      config:
        settingsFile: 'C:/Users/YOUR_NAME/.dsh/quality-guard/settings.json'
        criteriaFile: 'C:/Users/YOUR_NAME/.dsh/plugins/selfcheck-guard/criteria.example.md'
        auditFile: 'C:/Users/YOUR_NAME/.dsh/quality-guard/audit.jsonl'
        dryRun: true
        maxReworks: 3
        timeoutMs: 60000

After changing, restart dsh (or reload the patch). It’s okay if settings.json doesn’t exist; the engine will start with the static config in the patch plus criteriaFile. Note: Do not fill in maxTokens in the patch; the output budget is calculated automatically by the engine based on the tier; filling in a number disables the automatic budget.

The panel can also be installed manually: copy it to ~/.dsh/plugins/dsh-quality-guard/, create a directory link in the profile’s node_modules (Windows), and add a line to the patch with - id: quality-guard / name: dsh-quality-guard. The panel requires react ^18.2 and @deepseek-ai/dsh-client-locale and @deepseek-ai/dsh-client-ui-layout; the latter two are bundled with dsh. You can run it with just the panel installed, but nothing will read the settings it writes.

Configuration¶

Configuration has two sources, live settings take priority:

1. settingsFile (~/.dsh/quality-guard/settings.json) — what the panel writes, manually editable by humans, read every round by the engine.
2. Static config in the patch — serves as a fallback only when ① cannot be read or understood.

Common fields (full table in README):

Values out of range will be clamped back to default values (on the engine side readLive()), while the panel side will report human-readable errors. You don’t need to fill in the judge’s tier and output budget: the engine first uses the most economical tier (low; off is rejected by upstream), and only upgrades one tier to re-judge if it cannot judge; the budget rises with the tier. You can still run with old values stuffed into a manually written settings.json, but as soon as the panel saves, it normalizes everything to auto.

How to Write Criteria¶

Choose one of two writing methods. The text file (criteriaFile) is fed to the judge as-is, not parsed; in the panel, use the criteria array in settings.json, each item {id, label, enabled, text}, skipped if unchecked, exceptions one line per item. Both ultimately form the same block of text:

Criteria (any single item not satisfied means failure):
1. ……
2. ……

Exceptions:
- ……

If all are unchecked or completely empty, it reverts to the code’s built-in criteria—better to judge strictly than to silently stop checking.

Typical Usage¶

First, run a round with dryRun: true; only judge and log, do not reject, to see if the scores and worst / fix provided by the judge are reasonable, then decide whether to enable rejection. After confirming the criteria are effective, change dryRun to false, set maxReworks between 1-3 according to actual needs, and scoreThreshold defaults to 70 in patch and 80 in the panel.

Self-test without internet, without spending money, using a fake LLM:

node engine/selftest.mjs
# → 220 passed, 0 failed

Covers criteria assembly, timeout/error pass-through, maxReworks limit, dryRun, external criteria files, live settings overriding patch, and rescue paths for judge output in reasoning, aborted, and max-tokens exceptions.

Applicable Scenarios and Notes¶

Suitable for scenarios that need to turn “reply quality” into an executable gate: issues that mechanical criteria cannot control, such as off-topicness, fluff, and term stacking, which can only be caught with an independent round of semantic judgment. For scenarios sensitive to latency where you don’t want an extra model call per round, consider this cost first.

A few usage notes: Rejection uses agent.steer(). The old reply has already been appended to disk, so it will remain as an extra item in the UI; this is a design trade-off to facilitate comparison between long and short versions. The judge being strict doesn’t necessarily mean the judge is wrong; the README recorded a real test case where the judge judged “no” based on “no digression” and “values must have sources,” while the actual reply gave a location and values without sources. Confirm the corpus itself is unambiguous before testing the judge. The system prompt “Prefer to judge as failed” in the judge has been removed since v1.2.0; if false positives are found, adjust the criteria instead of adjusting the strictness.

Regarding privacy: each line of audit.jsonl is a retelling of real conversations, and worst / fix are the judge’s original words, essentially leaving a conversation summary on disk. The repository’s .gitignore already blocks *.jsonl, settings.json, and backups; do not forcibly git add -f. Before sharing screenshots, check if private settings are mixed into the criteria. To leave no trace, simply set auditFile to empty, and quality checking continues to work.

This plugin runs with the permissions of the current dsh process and can read your conversations and local configuration. It is recommended to read the source code and license (MIT) thoroughly before enabling, especially regarding which model the judge routes to.

Directory Page: https://www.skillhub.cn/plugins/wdj20050927/dsh-quality-guard
GitHub: https://github.com/wdj20050927/dsh-quality-guard