veri-safe/SWD

Sparse Weight Decomposition Checkpoints

This repository contains factor-only Sparse Weight Decomposition (SWD) checkpoints used in our replacement-fidelity and circuit-extraction experiments. It does not redistribute any base model. Load the corresponding base model first, then apply one checkpoint with swd_loader.py.

Each replaced matrix is represented as

output = input @ read @ write + bias

The intermediate coordinates are the SWD bottleneck units used for circuit scoring and ablation. s=0.5 and s=0.75 mean that 50% and 75% of all entries across the two factors are zero, respectively.

Included Checkpoints

Base model	Replacement	Setting	Data used	CE delta vs dense
GPT-2 Small	Layer 8 mlp.c_proj	s=0.5	16,384 tokens	0.000889
GPT-2 Small	Layer 8 mlp.c_proj	s=0.75	16,384 tokens	0.008292
Qwen2.5-0.5B	Layer 12 mlp.down_proj	s=0.5	1,024 tokens	0.001222
Qwen2.5-0.5B	Layer 12 mlp.down_proj	s=0.75	1,048,576 tokens	0.005010
Qwen2.5-1.5B	Layer 14 mlp.down_proj	s=0.5	1,024 tokens	0.000733
Qwen2.5-1.5B	Layer 14 mlp.down_proj	s=0.75	1,048,576 tokens	0.001222
Qwen3.5-27B	Layer 31 mlp.down_proj	s=0.5	2,048 tokens	-0.000427
Qwen3.5-27B	Layer 31 mlp.down_proj	s=0.75	1,048,576 tokens	-0.000448
GPT-2 Small	Layer 8 complete MLP	s=0.5	16,384 tokens	0.004003
GPT-2 Small	Layer 8 complete MLP	s=0.75	1,048,576 tokens	0.015521
GPT-2 Small	All 48 transformer-block linear projections	fixed-support SWD-FT	20,578,304 tokens	0.151263*

* The full-model value uses its full-model stress evaluation and should not be numerically compared with the single-matrix unified CE rows.

The Qwen2.5-3B checkpoints are distributed separately at veri-safe/SWD-Qwen2.5-3B because the upstream model uses the Qwen Research License.

Usage

Install the lightweight loader dependencies:

pip install torch safetensors transformers huggingface_hub

After downloading this repository, load a base model and apply a checkpoint:

from transformers import AutoModelForCausalLM
from swd_loader import apply_swd_checkpoint

model = AutoModelForCausalLM.from_pretrained("gpt2")
apply_swd_checkpoint(
    model,
    "checkpoints/gpt2-small/layer8-cproj/s0p5-tokens16384",
    mode="factorized",
)

mode="factorized" installs SWDLinear, exposing component_activations(inputs). Use mode="folded" to write read @ write back into the original dense module for conventional inference.

The base model must be fully materialized before applying a checkpoint. For large models loaded with a device map, each replacement is moved to the device and dtype of its target module.

Format

Every checkpoint directory contains:

model.safetensors  # factor tensors only; no pickle
config.json        # base model, module paths, shapes, sparsity, and token exposure
provenance.json    # source hashes, conversion rule, and validation result

All public factors follow [input, rank] @ [rank, output], independent of the source framework's dense-weight layout. Qwen2.5 source factors are transposed into this convention; Qwen3.5 feature shards are concatenated along the rank dimension without changing dtype or values. Full-model GPT-2 checkpoint biases are included because they belong to the fixed-support fine-tuned checkpoint.

RELEASE_MANIFEST.csv is the machine-readable checkpoint index. Each checkpoint's provenance.json records its source hashes and conversion validation.

Validation

Before release, every checkpoint was checked for source identity, finite tensors, shape compatibility, factor nonzero counts, and exact tensor equality after the safetensors round-trip.

The release intentionally excludes base-model weights, activation Grams, dense target/reconstructed matrices, optimizer state, data caches, remote-transfer archives, credentials, and cluster-local paths.

Links

• Code: https://github.com/Veri-Safe/SWD
• Models: https://huggingface.co/veri-safe/SWD

License

The SWD release code and the checkpoints in this repository are distributed under the Apache License 2.0. The GPT-2-derived checkpoints also retain the upstream Modified MIT notice in THIRD_PARTY_LICENSES/GPT2-MODIFIED-MIT.txt. See NOTICE for attribution.