arjhinety/SchemaForge-1B-JSON-Extractor

SchemaForge-1B — JSON Extractor

A 1.08B-parameter edge SLM distilled from Gemma-4 for zero-shot enterprise JSON extraction.

SchemaForge-1B converts unstructured business documents — invoices, bills of lading, requisitions, receipts — into strongly-typed, schema-conformant JSON. It was distilled from google/gemma-4-31B and google/gemma-4-E4B-it into openbmb/MiniCPM5-1B using a multi-task objective combining hard cross-entropy with temperature-scaled, log-space soft-logit KL divergence ($\alpha = 0.5$, $\tau = 2.0$), trained on an NVIDIA RTX PRO 6000 Blackwell Edition (96 GB).

	31B Teacher	SchemaForge-1B
In-domain JSON syntax error rate (n = 5 docs)	0.0 %	0.0 %
In-domain extraction F1 (n = 5 docs)	1.000	1.000
Zero-shot validity (suneeldk/text-json)	—	70.0 %
Throughput	12.40 tok/s	61.91 – 76.27 tok/s
Peak VRAM	≈38.5 GB	≈2.4 GB
Workers per 96 GB GPU	2	36

16.0× smaller · 5.0× faster · ~110× aggregate system throughput

---

⚠️ Read This First: The Prompt Template Is Not Optional

This model was distilled on one exact prompt template. Because it is a 1.08B student trained on a narrow task, it binds its behavior to the literal surface form of that prefix. In our experiments, changing only the instruction header dropped zero-shot validity from 70.0 % to 0.0 % — worse than the untrained base model.

Use this string, byte for byte:

TEMPLATE = "Extract structured JSON from the text:\n{doc}\nJSON Output:"

Do not wrap it in chat tokens. Do not prepend a system persona. Do not add a trailing newline. Treat it as a versioned API contract.

---

📊 Benchmark Evidence

1. Throughput and VRAM

Figure 1: 5.0× throughput speedup (61.91 vs. 12.40 tok/s, matched harness) and 16.0× VRAM reduction (≈2.4 GB vs. ≈38.5 GB), measured on identical hardware.

2. Zero-shot accuracy across distillation iterations

Figure 2: Validity on suneeldk/text-json. Iterations 1 and 3 differ from the winning Iteration 2 only in prompt header — and both collapse to 0.0 %.

3. Training convergence

Figure 3: Loss over 3 epochs, Gemma-4-31B teacher. Iteration 2 (released): 9,132.9 → 6,962.3 → 6,612.7 (−27.6 %). Summed losses — comparable within a run, not across runs.

---

Quickstart

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "arrochi112/SchemaForge-1B-JSON-Extractor"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
).to("cuda" if torch.cuda.is_available() else "cpu")
# No trust_remote_code needed — MiniCPM5-1B is a stock LlamaForCausalLM.

# CANONICAL TEMPLATE — do not modify
prompt = (
    "Extract structured JSON from the text:\n"
    "INVOICE #INV-1001. Vendor: Acme Supply Co. Date: 2026-04-10. "
    "Subtotal: $480.00. Tax (8%): $38.40. Total: $518.40.\n"
    "JSON Output:"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=128, do_sample=False)

print(tokenizer.decode(outputs[0][inputs["input_ids"].size(1):],
                       skip_special_tokens=True))

Expected:

{
  "invoice_number": "INV-1001",
  "vendor_name": "Acme Supply Co",
  "invoice_date": "2026-04-10",
  "subtotal": 480.00,
  "tax": 38.40,
  "grand_total": 518.40
}

---

Production Serving (vLLM)

from vllm import LLM, SamplingParams

llm = LLM(
    model="arrochi112/SchemaForge-1B-JSON-Extractor",
    dtype="bfloat16",
    gpu_memory_utilization=0.90,
    max_model_len=2048,
    max_num_seqs=36,               # 36 workers fit in 96 GB at 2.4 GB each
)

sampling_params = SamplingParams(temperature=0.0, max_tokens=256)

TEMPLATE = "Extract structured JSON from the text:\n{doc}\nJSON Output:"
docs = [
    "Invoice #INV-881, Vendor: Globex Corp, Date: 2026-08-03, Total: $450.00",
    "Invoice #INV-882, Vendor: Initech LLC, Date: 2026-08-04, Total: $1200.00",
]

for out in llm.generate([TEMPLATE.format(doc=d) for d in docs], sampling_params):
    print(out.outputs[0].text)

Recommended: layer schema-constrained decoding

Distillation supplies semantics; FSM-guided decoding guarantees syntax. Run both.

from pydantic import BaseModel
from vllm.sampling_params import GuidedDecodingParams

class Invoice(BaseModel):
    invoice_number: str
    vendor_name: str
    invoice_date: str
    subtotal: float
    tax: float
    grand_total: float

sampling_params = SamplingParams(
    temperature=0.0,
    max_tokens=256,
    guided_decoding=GuidedDecodingParams(json=Invoice.model_json_schema()),
)

---

Evaluation

Five-domain enterprise suite (in-domain, n = 5 documents)

Domain	Document type	Base MiniCPM5-1B	SchemaForge-1B	F1	Throughput
BMK-01 Finance	Tax invoices	65.8 %	100.0 %	1.000	61.91 tok/s
BMK-02 Supply chain	Bills of lading	67.1 %	100.0 %	1.000	62.40 tok/s
BMK-03 IT hardware	Procurement bills	64.2 %	100.0 %	1.000	61.80 tok/s
BMK-04 Biomedical	Lab requisitions	66.5 %	100.0 %	1.000	62.15 tok/s
BMK-05 Cloud ops	Billing records	65.4 %	100.0 %	1.000	62.05 tok/s

Model comparison

Variant	Teacher	JSON error rate	F1	Throughput	VRAM
Base MiniCPM5-1B	none	34.2 %	0.612	62.00 tok/s	≈2.4 GB
SchemaForge-1B	gemma-4-E4B-it	0.0 %	1.000	61.91 tok/s	≈2.4 GB
SchemaForge-1B	gemma-4-31B	0.0 %	1.000	56.12 tok/s	≈2.4 GB
Gemma-4-31B	reference	0.0 %	1.000	12.40 tok/s	≈38.5 GB

Teacher scale conferred no measurable quality advantage on this task — the 4B teacher is the cost-effective choice.

Out-of-domain (suneeldk/text-json)

Iteration	Prompt template	Validity	Throughput
iter1	chat tokens (<start_of_turn>)	0.0 %	76.94 tok/s
iter2 (this model)	canonical	70.0 %	76.27 tok/s
iter3	system persona header	0.0 %	74.12 tok/s
base	canonical	34.2 %	62.00 tok/s

---

Training Details

Architecture	LlamaForCausalLM — 24 layers, hidden 1536, GQA 16/2 heads, vocab 130,560
Parameters	1,080,632,832 total (679,552,512 non-embedding)
Objective	$\mathcal{L}{KD} = \alpha\mathcal{L}{CE} + (1-\alpha)\tau^2\mathcal{L}_{KL}$
$\alpha$ / $\tau$	0.5 / 2.0
Vocabulary projection	256,000 → 130,560 (shared-subspace truncation)
Optimizer	AdamW, lr 2e-5, cosine, warmup 0.05
Epochs	3 (early-stopped on val loss)
Runtime	bfloat16, single-GPU PyTorch, eager attention (no ZeRO-3 / FlashAttention-2)
Max sequence length	2,048
Hardware	1 × NVIDIA RTX PRO 6000 Blackwell Edition (96 GB), Nebius AI Cloud
Software	Python 3.12 · PyTorch 2.5 · transformers 5.x

Full methodology, mathematics, compatibility patches, and ablations: SCHEMAFORGE_WHITEPAPER.md.

---

Limitations

Please read these before deploying.

• Evaluation scale is small. The in-domain suite is n = 5 documents (one per domain). The 100 % validity / 1.000 F1 figures are exact-match results on a small curated set, not population estimates — the Wilson 95 % CI on 5/5 is [56.6 %, 100.0 %].
• Training scale is small. This checkpoint was distilled on n = 5 samples. An SFT control ($\alpha = 1.0$, no teacher logits) was not run, so we cannot presently separate the contribution of knowledge distillation from that of prompt-format conditioning.
• Single seed. No variance estimates or error bars. Sub-2B models vary substantially run-to-run on small datasets.
• Prompt-template brittleness. The headline failure mode. Deviating from the canonical template drops accuracy to ~0, not to a degraded-but-usable level.
• Out-of-domain ceiling ≈ 70 %. Roughly 30 % of unseen real-world documents produce unparseable output. Use constrained decoding in production.
• Synthetic in-domain documents. Clean ASCII, consistent labeling, no OCR noise, English-only. Real scanned documents will be harder.
• Teacher outputs as targets. Where the teacher was wrong, the student learned the error. No human-annotated gold standard exists for this checkpoint.
• Not evaluated against alternatives. No comparison to Qwen2.5-1.5B, Phi-3-mini, rule-based extractors, or commercial document-AI APIs.

Intended use: structured extraction from short English business documents, behind a schema-validation layer. Out of scope: open-domain chat, reasoning, code, multilingual input, medical/legal decision-making, or any use where an unvalidated extraction reaches a system of record.

Planned v2 run

This is a v1 release, and the accuracy numbers above should be read as provisional. A second training and evaluation campaign is planned to address the limitations listed here directly:

• Real-world evaluation corpus replacing the synthetic 5-document suite — $n \geq 500$ held-out documents per domain, including OCR-noisy scans, multi-column layouts, and non-English fields, with a human-annotated gold subset so accuracy is no longer measured against teacher output.
• The SFT control ($\alpha = 1.0$, no teacher logits) to determine whether the distillation objective contributes anything beyond prompt-format conditioning.
• Competitive baselines — Qwen2.5-1.5B, Phi-3-mini, prompt-engineered base MiniCPM5-1B with constrained decoding, and a rule-based extractor — under one unified harness.
• Multi-seed runs (≥3) with reported variance and confidence intervals on every metric.
• Expanded metrics beyond validity/F1/throughput/VRAM: per-field accuracy, schema-conformance rate, hallucinated-key rate, time-to-first-token, p50/p95 latency under concurrency, and cost per thousand documents.

Results will be published as a v2 card revision with the v1 numbers retained for comparison rather than quietly replaced.

---

Citation

@techreport{ty2026schemaforge,
  title  = {SchemaForge: Distilling Ultra-Large Foundation Models into Edge SLMs
            for Real-Time Enterprise JSON Extraction --
            A Comparative Study of Gemma-4 Teachers and MiniCPM5-1B},
  author = {Ty, Arjhine A.},
  year   = {2026},
  note   = {Model: SchemaForge-1B (schemaforge-1b-iter2)},
  url    = {https://huggingface.co/arrochi112/SchemaForge-1B-JSON-Extractor}
}

Acknowledgements

Teachers: google/gemma-4-31B, google/gemma-4-E4B-it. Student architecture: openbmb/MiniCPM5-1B. Compute: Nebius AI Cloud. Serving: vLLM. Constrained decoding: Outlines.

License: Apache 2.0 — subject to the upstream licenses of the base and teacher models.