Installation

Install Modeldiffr from PyPI or from source.

Requirements

• Python 3.10, 3.11, or 3.12
• A GPU for anything larger than a small model. Both models are held in memory at once.

Install from PyPI

With model supportStandard benchmarksvLLM or llama.cppEverythingSigning and anchorsVerify reports onlyCore only

pip install "modeldiffr[hf]"

The hf extra installs torch>=2.1, transformers>=4.40, accelerate>=0.30 and huggingface_hub. Use this to run audits.

pip install "modeldiffr[hf,tasks]"

Adds datasets for the lite, standard and full suites.

pip install "modeldiffr[vllm]"
pip install "modeldiffr[hf,llamacpp]"

See Backends.

pip install "modeldiffr[all]"

pip install "modeldiffr[proof,sign,anchor]"

sign adds cryptography and sigstore; anchor adds the OpenTimestamps client and web3. See Sign and verify.

pip install "modeldiffr[proof]"

No GPU and no PyTorch. Enough to run modeldiffr verify report.json with the full schema check.

pip install modeldiffr

The core package has no dependencies. It is enough for CI jobs that only read report.json, and for the statistics in modeldiffr.stats.

Warning

Running modeldiffr audit without the hf extra exits with code 2 and prints:

error: Hugging Face support is not installed. Run: pip install "modeldiffr[hf]"

Install from source

git clone https://github.com/amareshhebbar/modeldiffr.git
cd modeldiffr
pip install -e ".[hf,dev]"

The dev extra adds pytest, ruff, build, twine, jsonschema, numpy, safetensors and gguf.

Verify

modeldiffr --version
modeldiffr suites

Expected output:

modeldiffr 0.4.0
quick: 24 capability items, 20 over refusal prompts, 12 divergence texts

You can also run the package as a module:

python -m modeldiffr --version

Device support

--device auto picks the first available option in this order:

1. cuda
2. mps (Apple Silicon)
3. cpu

Pass an explicit value such as cuda:0 to override.