Introduction¶
The MLIR/LLVM AOT backend (like toyc --emit=exe) and the JIT path (like toyc --emit=run) are expected to produce the same results. After backend changes, differences often appear only on a specific type of input, and might be the 3rd line of stdout, the exit code, or content written to a file. Manual comparison requires launching a process on both sides, copying outputs, and comparing logs; this is slow and prone to missing issues.
dsh-aotjit-compare is a DSH plugin built for this purpose: it runs the same test case for both the AOT executable and the JIT command, compares stdout / exit code / stderr / file side effects field by field, and directly tells you “which line, which side, what is expected, what is actual”. Below is an introduction to its capabilities, installation, and usage.
What is this¶
Plugin ID: aotjit-compare, Tool Name: aotjit_compare, Version: 0.1.0, Maintainer: Rcyc-1025, Category: Workflow, License: MIT.
It does one thing: run both sides, compare, locate, and log. Finding a difference is a normal return, not an error throw—the tool structures the difference for you, letting you decide which side to fix next.
Core Features¶
- Run the same set of test cases for both the AOT executable and the JIT command, supporting per-test-case overrides for arguments, stdin, env, and cwd.
- Compare fields one by one; by default, it compares
stdoutandexit, capturesstderrby default but does not compare it, and disables file side effects by default. - Differences are located by field, line number, expected value, and actual value;
diffs[].expectedis always the AOT side, andactualis the JIT side. - Normalization strategy is configurable: CRLF, ANSI colors, trailing newline, and path prefix replacement.
- File side effects support three modes: hash / content / tree, and provide isolated directories to prevent pollution between the two sides.
- Logs are written on every call, including both sides’ argv, exit codes, duration, truncation and encoding status, and every difference; log failures only go into
warnings[]and do not affect the conclusion. - Illegal configuration does not stop loading; it falls back to default values and is recorded in
warnings[]; call arguments take priority over configuration, and environment variables can perform process-level overrides.
Installation and Usage¶
The plugin directory needs to be referenced by the desktop profile. The README provides two ways. The first uses the DSH CLI to install the local directory into the profile:
D:\exe\DSH\resources\runtime\cli\bin\dsh.cmd plugin --profile desktop add D:\workplace\aotjit-compare
The second is to register it manually in <profile>/package.json; the profile directory is usually %DSH_HOME%\data\profiles\desktop:
{
"dependencies": { "dsh-aotjit-compare": "file:D:/workplace/aotjit-compare" },
"dsh": { "profile": { "bundles": ["...", "dsh-aotjit-compare"] } }
}
cordis.patch.yml registers this plugin using the insert mode, name: 'dsh-aotjit-compare'. The package declaration is dsh: { bundle: { patch: "./cordis.patch.yml" } }. The Node environment requires ^22.19.0 || >=24.0.0, and DSH requires >=0.2.1-rc.5 <0.3.0-0.
Typical Usage¶
The minimal call requires jitSpec and cases. The example below runs the AOT executable and JIT command on the same test case:
aotjit_compare({
aotExe: "build/s01/add.exe",
jitSpec: { cmd: "build/toyc.exe", args: ["examples/add.mul.mlir", "--emit=run"] },
cases: [{ name: "add.mul" }]
})
When both sides output 7\n and the exit code is 9, it returns allMatch: true.
The argv assembly rules need to be remembered clearly, as this is the easiest place to configure incorrectly:
AOT: [aotExe, ...aotArgs, ...case.args]
JIT: [cmd, ...(case.jitArgs ?? jitSpec.args), ...case.args]
case.jitArgs overrides jitSpec.args, it does not append. This allows 10 test cases to share the same JIT entry point, changing only the input file. case.args is appended to both sides, used for runtime switches (e.g., --seed=1). argv is passed directly without going through a shell, so quotes are not needed, and pipes or redirections cannot be written. If jitSpec.cmd points to .cmd / .bat, it will be rejected, returning JIT_CMD_SHELL_WRAPPER.
Output and Determination¶
The most confusing part of the return structure is ok vs allMatch: ok indicates execution is complete, not that it is consistent; allMatch means all test cases are consistent. When a difference is found, ok is still true.
{
"ok": true,
"allMatch": false,
"firstMismatchCase": "case-07",
"firstMismatchField": "stdout",
"total": 10, "matched": 9, "mismatched": 1,
"rows": [
{
"index": 6, "case": "case-07", "match": false, "firstMismatch": "stdout",
"diffs": [
{ "field": "stdout", "kind": "text", "aotLine": 1, "jitLine": 1,
"expected": "7", "actual": "8", "expectedLength": 2, "actualLength": 2 }
],
"aot": { "cmd": ["D:/p/build/parity/case-07.exe"], "cwd": "D:/p", "exitCode": 0,
"exitCodeHex": "0x0", "signal": null, "timedOut": false, "durationMs": 12,
"stdout": "7\n", "stderr": "", "stdoutTruncated": false, "stderrTruncated": false,
"spawnError": null },
"jit": { "...": "同上" }
}
],
"logPath": "D:/p/.aotjit-logs/20261106-103000-a1b2-case07.log",
"durationMs": 183, "warnings": [], "hint": null
}
The field priority is spawnError > timedOut > exit > stdout > stderr > sideEffects. Several boundary behaviors are worth noting: when either side’s output is truncated, the truncation prefix is not compared, and firstMismatch: "truncated" is set directly, avoiding false conclusions from incomplete output; when the output is not valid UTF-8, byte SHA-256 is compared instead; Windows negative exit codes keep the original value and provide exitCodeHex (e.g., 0xc0000005).
Configuration¶
Configuration is written in cordis.patch.yml or the host config. The keys and default values listed in the README are as follows:
| Key | Default | Description |
|---|---|---|
timeoutMs |
120000 |
Single process timeout; after timeout, SIGTERM is sent first, then SIGKILL after graceMs |
maxOutputBytes |
1048576 |
Byte limit retained per stdout / stderr stream |
maxCases |
100 |
Upper limit on the number of test cases allowed per call |
maxSideEffectBytes |
16777216 |
Total bytes allowed for side effect snapshots |
logDir |
.aotjit-logs |
Log directory (relative to session cwd) |
compare |
{stdout:true, exit:true, stderr:"off", sideEffects:false} |
Default comparison items |
normalize |
{crlf:true, ansi:false, trailingNewline:"exact", replacePaths:[]} |
Default normalization |
saveArtifacts |
false |
Whether to save complete stdout/stderr of each side to artifacts/ |
Normalization rules: crlf unifies CRLF and isolated CR into LF; ansi must be set to true to strip color sequences; trailingNewline is "exact" to preserve the trailing newline, or "ignore" to remove all trailing newlines; replacePaths replaces specified prefixes with <path>, used to smooth directory diffs in stderr. compare.stderr supports "off" (default, no comparison), "normalized" (compare based on normalize), and "strict" (only unify CRLF).
Environment variables AOTJIT_TIMEOUT_MS / AOTJIT_MAX_OUTPUT_BYTES can override at process level, call arguments have the highest priority.
File Side Effects¶
When sideEffects is enabled, the plugin snapshots and compares specified paths:
compare: {
sideEffects: { paths: ["out/result.txt"], mode: "content", isolate: true, seed: ["input.bin"] }
}
paths is required, relative to the run directory, absolute paths and .. escapes are forbidden, otherwise SIDE_EFFECT_ESCAPE is returned. mode has three types: hash compares SHA-256, content compares content and gives the first different line, tree recursive dir tree.
isolate: true when AOT and JIT each run in <workDir>/.aotjit/<runId>/<caseIndex>-aot|jit, seed files or dirs are copied into both sides, ensuring AOT written files JIT cannot see. A call ends (success, fail or break) whole <workDir>/.aotjit/<runId>/ will be cleared, clean failed only warnings[]. seed byte count is expanded after counted, AOT/JIT sides sum is under maxSideEffectBytes constraint: limit bytes: max bytes side limit: max bytes effect bytes limit: bytes max side bytes effect limit: bytes max side bytes effect limit: bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes side effect limit bytes limit side effect limit v bytes side effect limit bytes side effect limit side bytes effect side limit bytes side effect limit side effect limit side effect limit side effect limit side effect limit side effect limit side effect limit side effect limit side effect limit side effect limit side effect limit side effect limit side effect limit side effect side effect limit side effect limit side effect limit side effect limit side effect limit side effect limit side effect limit side 7sideeffectside7sideeffectside7sideeffectside7sideeffectside7sideeffectside7sideside7side7side77side777777777777777777777777777777 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 2 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 OT 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 OT 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 1 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 777777777777 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 6 6 6 7 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 7 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 a6 666 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 66666666666 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 666 66 6 6 6 6 6 6 6 6 6 6 6 666666 6 6 6 6 6 6 6 6 6 6 6 6 6666666666666666 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 61 6 6 6 6 6 6 6 6 6 6 666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666 66666 6 6 666666 6 6 6 6 6666666666666666666666OT6666666666666666666676666666666666666666666666661666666666666666666666666 6 6 6 6 5 6 6 6 6 6 6 6 6 6 6 6 6 5 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 1 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 0 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 666 6 6 6 6 6 6 6 6 5 6 6 6 6 6 6 6 6 6 6 6 6666666666666666666 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 66666666 6 6 6 6 6 6 6 6 6 666666666666666 6 6 6 6 6 1 6 6 6 6 6 6 6 6 6 6 6 6 666666666 6 6 6666666 6 6 6 6 6 66666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666dsh-aotjit-compare6666666666666666666666666666666666666666666666666666666666666666666666666666666666666616666666666666666666666666666DSH 666666666666666666666666666666666666666666SH 666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666661666666666666OT6666666666666666666666666666666666Output truncated`
```
Conclusion¶
This plugin has a narrow focus: it does not do fuzzy matching, nor multi-scenario orchestration; it simply turns “run both sides + compare + locate + log” into a single call, and the field priority and boundary behaviors are all documented in the README. If your work involves MLIR/LLVM AOT and JIT consistency regression, you can first verify the toolchain via the npm run integration:parity chain, then run with real test cases.
Directory page: https://www.skillhub.cn/plugins/Rcyc-1025/dsh-aotjit-compare
GitHub: https://github.com/Rcyc-1025/dsh-aotjit-compare
