Corpus Benchmark

A reproducible benchmark for academic validation: cfgdrift mines configuration change pairs from real project git history, standardizes them into a JSONL corpus, and supports dual annotation with Cohen's kappa — so research claims about detection accuracy are verifiable.

Toolchain

cfgdrift corpus init --workspace <dir>          # corpus.yaml + state.json + repos/
cfgdrift corpus fetch --workspace <dir>          # git history -> change pairs -> instances.jsonl
cfgdrift corpus export --workspace <dir>         # idempotent full rewrite (deterministic)
cfgdrift corpus validate --workspace <dir>       # JSONL schema check + stats (corrupt → exit 2)

• Repositories: corpus.yaml lists owner+repo (clone with --filter=blob:none --no-checkout + incremental fetch) or local_path for an existing local git repo (offline / CI-safe).
• Incremental: state.json records last_commit / stars / instance_count; re-fetch only handles new commits.
• Instance schema (instances.jsonl): schema_version / instance_id / metadata / file / before / after (semantic tree + parse_ok + present) / diff (items + summary + constraint_violations + feature) / labels. Raw text is not persisted.

Dual annotation + kappa (v0.8.0)

cfgdrift corpus annotate --workspace <dir> --annotator alice            # interactive
cfgdrift corpus annotate --workspace <dir> --annotator alice --batch labels.yaml   # CI-friendly
cfgdrift corpus kappa --workspace <dir> [--annotator-a A --annotator-b B] [--weighted linear|quadratic] [--json]
cfgdrift corpus stats --workspace <dir> [--json]

• Annotations are stored separately in annotations.jsonl (3-class ordinal severe|minor|normal) so repeated exports never lose them.
• kappa computes Cohen's kappa + agreement rate + confusion matrix; without arguments it auto-pairs the two annotators with the most overlap.
• export merges each instance's latest annotation into labels.
• kappa --export (v0.10.0) exports a paper appendix Markdown / UTF-8 BOM CSV.

Zero-noise regression: the corpus feeds the diff and constraint engines, and the full test suite asserts byte-identical output on legal changes — see docs/ system design docs and README.en.md.