Introduction¶

In LLVM/MLIR related development, running tests usually isn’t just a single lit command. What is truly valuable is the next layer of information after failure: which CHECK line failed, which line of actual output broke it, and how to only rerun those tests that just failed. When doing this loop manually, people switch back and forth between lit output, test files, and actual IR. Fragmented steps are well-suited for tools in the session.

dsh-lit-filecheck is a DSH plugin for this scenario. It splits the lit + FileCheck testing loop into three tools, allowing agents in the session to sequentially execute “run once, rerun only the failed set, separately validate a pair of files,” and structuring failure information to the CHECK line level. Below introduces its capabilities, installation method, and typical usage.

What is this¶

dsh-lit-filecheck is a DSH plugin, maintained by Rcyc-1025, npm package name dsh-lit-filecheck@0.1.0, plugin id is lit-filecheck. The problem it solves is: allowing agents in the DSH session to execute lit test suites and extract actionable CHECK diagnostics from failure output, rather than just getting a block of raw lit output.

It belongs to the “Workflow” category, written in ESM, does not start background tasks, and does not persist state—the rerun target only exists in the current plugin instance’s memory. Simply put, when the session ends and the plugin instance is destroyed, the failures run last time disappear with it.

What do the three tools do¶

The plugin provides three tools corresponding to three things in the testing loop.

lit_run: Runs llvm-lit in the given suite root directory, reports total/passed/failed/skipped, and splits each block of failure output into individual CHECK diagnostics—pointing out which line in the test file is the CHECK and which line of the actual IR broke it.

lit_rerun_failed: Only reruns the tests that failed in the last lit_run of this plugin instance, without a full rescan. No --filter, the command line only carries the paths of previously failed tests.

filecheck: Runs FileCheck once on a specific pair of “test file with CHECK lines + actual IR file,” reporting matches and mismatches as well as error lines.

Installation and enabling¶

The plugin is provided for installation as a DSH bundle. The cordis.patch.yml in the repository is responsible for inserting the plugin entry into the session configuration, and the plugin’s own package.json also declares the bundle patch location:

"dsh": {
    "bundle": {
      "patch": "./cordis.patch.yml"
    }
}

The entry structure of cordis.patch.yml is as follows, config contains the plugin’s default configuration:

- insert:
    - id: lit-filecheck
      name: 'dsh-lit-filecheck'
      config:
        timeoutMs: 120000
        maxOutputBytes: 1048576
        maxChecks: 50
        logDir: .lit-logs
        requireFileCheck: true

engines requires Node version and DSH version to satisfy:

"engines": {
  "node": "^22.19.0 || >=24.0.0",
  "dsh": ">=0.2.1-rc.5 <0.3.0-0"
}

A complete npm installation method is to add the package as a dependency and ensure the bundle patch is read. This depends on the DSH bundle mechanism you are currently using. The README only provides the cordis.patch.yml entry itself and not a command in the form of dsh plugin add, so I won’t stitch it together here.

Configuration items¶

The plugin’s configuration items are centralized in a config object. toolchainPath points to toolchain.json, lookup order is: that key > MLIR_TOOLCHAIN_JSON > <session cwd>/mlir-lang/toolchain.json.

litCommand is the complete lit argv array, highest priority, e.g., ["D:/Code/python/python.exe","D:/Code/llvm-build/bin/llvm-lit.py"]; after setting, the plugin stops probing. litPath points to llvm-lit.py or an executable file; .py files will automatically have pythonPath prepended, .cmd / .bat will be rejected (cmd forwarding on Windows breaks arguments). pythonPath is the interpreter for .py lit, if not set it tries D:/Code/python/python.exe and PATH in sequence. filecheckPath points to FileCheck.exe; if not set, it looks in llvmRoot/bin, D:/Code/llvm-build/bin, D:/Code/llvm/bin, and PATH.

Keys related to runtime behavior: timeoutMs defaults to 120000, the total budget for a single lit run; maxOutputBytes defaults to 1048576, the upper limit for retaining stdout/stderr respectively; maxChecks defaults to 50, the upper limit of CHECK diagnostics parsed per run; logDir defaults to .lit-logs, relative to session cwd; requireFileCheck defaults to true, if true and FileCheck is missing it reports tool-missing, if false it only warns and continues running lit.

Numbers out of range do not throw an error, they only issue a warning and fall back to the default value.

Typical usage: The trio of tools in sequence¶

The typical session flow is to run the suite once first, then isolate against failures.

Step one, lit_run. Parameter suiteDir is required, also filter, jobs (1-64), timeoutMs. The plugin executes in the suiteDir directory:

<litCommand...> -v --no-progress-bar --show-unsupported [--filter=<filter>] [-j <jobs>] .

The return value includes ok, total, passed, failed, skipped, and the failures array. Each failure contains test, status, reason, runLine, and a checks array; each checks entry gives checkFile, line, column, prefix, pattern, expected, matched, actualLine, actualText, message. Also logPath, truncated, timedOut, durationMs, preflight, warnings, error.

ok is true if preflight passes, exit code is 0, failed === 0, and no timeout. A failed run still returns the complete failures / checks / logPath—the truly informative part is this red result. reason takes one of the following: filecheck-mismatch, command-failed, timeout, no-filecheck-diagnostic, unknown.

Step two, lit_rerun_failed. No parameters. The command it executes only carries the previously failed paths, and has no --filter:

<litCommand...> -v --no-progress-bar <failed test paths...>

Several edge cases: if there were no failures last time, then ok:true, noop:true, total:0, and no process is started for lit; if lit_run was never run, it returns no-previous-run; if a failed test file has been deleted, it will appear in missingTests with a warning; if all are missing, it returns path-not-found. The return structure is consistent with lit_run, with additional onlyFailed, noop, source{suiteDir,startedAt}, missingTests.

Step three, filecheck. Parameters testFile and irFile are both required and must be existing regular files, also extraArgs and timeoutMs (default 30000). It executes:

<FileCheck.exe> --color=false <extraArgs...> <testFile> --input-file=<irFile>

extraArgs containing --input-file or - will be rejected because it would override irFile. Returns ok, mismatches[], stdout, stderr, truncated, logPath, durationMs, error, hint; ok is only true when the exit code is 0. Returns no-check-lines when there are no CHECK directives in the test file at all, it won’t be guessed as passing.

How CHECK diagnostics are parsed¶

The standard error line of FileCheck looks like:

<check-file>:<line>:<column>: error: CHECK: expected string not found in input

The plugin reads the <line> line as pattern and the text after CHECK: as expected. actualLine / actualText come from the first available note: prioritizing note: possible intended match located ..., then note: scanning from here. When both do not exist, they remain null, no guessing is done.

Crash information (PLEASE submit a bug report / Stack dump) will not be treated as diagnostics, only entering logs and warnings.

Logging and error handling¶

Every run writes the complete output to <logDir>/lit-<yyyyMMdd-HHmmss>-<rand>.log, including argv, cwd, start/end times, exit code, counts, failure details, and complete stdout/stderr. The stdout/stderr in the tool’s return value will be truncated by maxOutputBytes (preserving head and tail). If the log cannot be written, logPath is null with a log-write-failed warning, but the result itself is still returned.

Failures do not throw exceptions, but return ok:false + error{code,message} + an actionable Chinese hint. Error codes covered: invalid-input, path-not-found, not-a-directory, tool-missing, spawn-failed, timeout, lit-internal-error, no-previous-run, invalid-extra-args, no-check-lines, log-write-failed, parse-failed.

Testing and verification status¶

The repository comes with tests. npm test runs 68 test cases, 66 passed, 2 real lit integration test cases skipped by default; npm run smoke will actually run a plain suite and print the results of the three tools. To run real integration tests:

$env:LIT_FILECHECK_REAL='1'; node --test --test-isolation=none test/integration.test.mjs

The verification status recorded in the README is the result on the local machine with LLVM 23.1.4 on 2026-11-05. The lit side has been verified: test/fixtures/plain/ has 5 tests, 3 passed 2 failed, lit_run reports total=5/passed=3/failed=2, lit_rerun_failed exactly reruns those 2; failure block parsing is based on the captured real output test/golden/plain-lit-output.txt. The FileCheck side has not been verified against real binaries: the Windows LLVM 23.1.4 package on this machine does not include FileCheck.exe, nor is there an MSVC toolchain available to compile; CHECK-level parsing (line corresponding to CHECK line, actualLine corresponding to actual IR line) is covered by manually written test/golden/filecheck-stderr.txt and test/golden/lit-filecheck-output.txt. Need to rerun real CHECK-level acceptance once FileCheck is available.

Known limitations¶

Three limitations worth confirming before adoption.

lit.cfg.py must be pure ASCII. On Windows, lit uses the platform locale (GBK) to read config files, and non-ASCII comments will cause it to crash with UnicodeDecodeError.

The .cmd / .bat form of llvm-lit is intentionally rejected, please use llvm-lit.py and specify pythonPath.

In the restricted DSH sandbox, Node spawns with pipes are rejected (spawn EPERM), lit’s multi-process pool will hit PermissionError, and the tool returns spawn-failed; it works normally when running on the host machine.

Use cases and notes¶

Suitable scenarios: processing MLIR/LLVM lit test suites in a DSH session, especially when needing to pinpoint failures to CHECK lines, needing to rerun only failed sets, or needing to run FileCheck separately on a pair of “test file + actual IR”. Classification is workflow, the positioning is to streamline the testing loop process, not to replace lit itself.

A few points to note. The plugin runs with the permissions of the current dsh process, will execute external lit/FileCheck processes, and read the suite directory and log directory. Before installation, check the source code and license (this plugin is MIT licensed, see repository LICENSE). Rerun targets only exist in plugin instance memory, lit_rerun_failed has no memory after session restart. On Windows, lit with .cmd / .bat is unavailable, lit.cfg.py needs to use ASCII. The README clearly states that the FileCheck side has not yet been verified against real binaries; when first integrating, it is recommended to run the real integration test once with the command above before deciding.

Summary¶

The value of dsh-lit-filecheck lies in turning the “run lit -> split CHECK diagnostics -> rerun only failed set -> separately validate a pair of files” series of actions into three callable tools in the session, and structuring failure information to the line level. It suits repetitive iterations in MLIR/LLVM test scenarios, at the cost of needing to confirm lit paths and FileCheck availability under Windows, and accepting that the FileCheck side currently only has manual golden coverage.

Plugin Directory: https://www.skillhub.cn/plugins/Rcyc-1025/dsh-lit-filecheck
GitHub: https://github.com/Rcyc-1025/dsh-lit-filecheck