Skip to content

Know what changed between a model and anything built from it.

Modeldiffr runs a base model and its fine tune, quant, merge or edit on identical items, reports every change with a paired interval, looks inside for the features that moved, and writes a report anyone can verify against the exact weights.

pip install "modeldiffr[hf]"
Run your first audit How it works
base Qwen2.5 1.5B Instruct derivative a 2 bit quant of it
-0.10 0 +0.10 mmlu_accuracy-0.012 gsm8k_accuracy-0.071 ifeval_prompt_accuracy-0.034 truthfulqa_mc1_accuracy+0.004 xstest_safe_over_refusal_rate+0.004 mgsm_accuracy-0.063
Illustration of a report. Each line is a 95% interval for derivative minus base on the same items. Red lines exclude zero after correction.

One report, five questions

Each layer answers a question the one before it raises. Run as many as the decision needs; every layer writes a file bound to the same weights.

  1. Proof
    Which exact models, items and settings were compared, and can anyone check that?
    modeldiffr verify modeldiffr sign Fingerprints and proof
  2. Behavior
    What changed on thirteen benchmarks, and is it larger than noise?
    modeldiffr audit Statistics
  3. Representation
    Which layers moved, and which features are new, lost or reshaped?
    modeldiffr layers modeldiffr features Representational layer
  4. Cause
    How much of the behavioral change do those layers and features carry?
    modeldiffr causal Causal layer
  5. Anomaly
    Is there an input where the derivative behaves unlike anything in its base?
    modeldiffr scan Anomaly scan

Nobody has to take your word for it

An audit id is the hash of everything that defines the comparison, computed before anything runs. Every item and result is committed to in a Merkle tree, and the report can be signed with Sigstore and timestamped on Bitcoin or an L2. Anyone can recompute it from the file, or from the real weights with --models.

Verify a report Sign and verify

$ modeldiffr verify report.json --models --key acme.pub
audit id mdr1:71e1c0a4b2d39f10...
PASS  schema                     valid against report schema 0.3
PASS  audit_id                   mdr1:71e1c0a4b2d39f10...
PASS  results_root               sha256:4be1...
PASS  metrics                    17 metrics recomputed from the items
PASS  base_model_files           weights match the report
PASS  derivative_model_files     weights match the report
PASS  signature                  signed with acme.pub

verified

Where to start

Quickstart
Install, run a first audit on a laptop, read the report.

Gate a checkpoint in CI
Fail a pull request when a metric changes, and upload a signed report.

Audit a quant
Compare a GGUF build with its original through llama.cpp.

Find changed features
Train a crosscoder and read what is new in the derivative.

Publish audits
Batch many pairs and build a registry that verifies in the browser.

Python API
Run audits from your own code with any backend.

Version 0.9: every layer has shipped. Version 1.0 waits for the published study and zoo results described in the roadmap, and the backdoor scan makes no claim about real models until then.