dsh-llm-verifier: Configurable LLM Verification Plugin for DSH

dsh-llm-verifier is a native DSH plugin designed to introduce an independent review mechanism to address challenges in selecting among multiple candidate solutions, tracking progress in long-running tasks, and validating session outcomes. Based on a configurable judge model, it performs evaluations using explicit tools—verifier_compare, verifier_select, verifier_track, and verifier_current_session—employing an A-T fine-grained scoring scale that is normalized to a 0-1 score. The plugin supports probabilistic expectation scoring, mitigation of position bias, and persistent caching, automatically scheduling the validation process at agent/turn-stopping boundaries. After installation, DSH automatically aligns the configuration, allowing users to set up the provider, model, and concurrency strategies via a web interface, or integrate it as an npm

Read More