Mark-Engineer-1.5B

Overview

Mark-Engineer-1.5B is a 1.5B-parameter sparse Transformer exported to the native deepseek2 GGUF format. It combines Multi-Head Latent Attention (MLA) with a 16-routed and 1-shared SwiGLU Mixture-of-Experts (MoE) topology to keep active parameters at 364.15M per token (24.34% active) out of 1,496,121,824 total parameters.

Quantized in IQ4_NL, the binary occupies 802.94 MiB (841.94 MB / 841,941,664 bytes), leaving over 7 GB of VRAM free on an 8 GB GPU such as NVIDIA GeForce RTX 2060 Super (8 GB GDDR6, 256-bit, 448 GB/s).

Specifications

Field	Value
Architecture	deepseek2
Training Status	EXPERIMENTAL_NARROW_CHECKPOINT
Weight State Label	Experimental Narrow Checkpoint
Binary Size	802.88 MiB (841.88 MB / 841,881,696 bytes)
Total Parameters	1,496,016,864 (1496016864 physical across 281 tensors)
Active Parameters	364,046,304 (364046304 active)
Vocabulary Size	4311
Layers	16
Hidden Dimension	1280
Attention Heads	16
MLA Ranks	q_lora_rank=640, kv_lora_rank=320, qk_rope_head_dim=32
MoE Experts	16 routed, 1 shared (2 active/token)
Expert FFN Dim	1504
Context Length	4096 tokens
Quantization Type	IQ4_NL (4-bit non-linear)
SHA-256	983683d2e7885c9ce6589f7e2009ca4e9ebb891ff448c509a024732953780aba

Architecture

Multi-Head Latent Attention (MLA)

The model projects key and value states into a shared low-rank latent representation to decouple memory scaling from context length:

c_KV = h_t * W_DKV          (dim: 320)
c_Q  = h_t * W_DQ           (dim: 640)
q_C  = RMSNorm(c_Q * W_UQ)
k_C  = RMSNorm(c_KV * W_UK)

Decoupled rotary position embeddings are concatenated along a separate dimension:

k_R  = RoPE(h_t * W_KR)     (dim: 32)
q_t  = [q_C ; q_R]
k_t  = [k_C ; k_R]

KV Cache Memory Usage

Physical memory consumption of the KV cache per token in FP16:

KV_bytes_per_token = 16 * (320 + 32) * 2 = 11,264 bytes/token

Context Window	MLA KV Cache	Standard MHA KV Cache	Memory Savings
4,096 tokens	44.00 MiB	384.00 MiB	8.7x
8,192 tokens	88.00 MiB	768.00 MiB	8.7x
32,768 tokens	352.00 MiB	3072.00 MiB	8.7x

Mixture-of-Experts (Aux-Loss-Free Routing)

• Shared Experts (1S): Constantly active path for common representations: $$F_{\text{shared}}(x) = (\text{SiLU}(x W_{\text{gate}}) \odot (x W_{\text{up}})) W_{\text{down}}$$
• Routed Experts (16R, Top-2): Routed using token-to-expert affinities $s_{i,t} = \sigma(u_t^T e_i)$ over biased scores $s_{i,t} + b_i$, where router bias vectors (blk.{l}.exp_probs_b.bias) are stored in FP32 precision.

Aux-Loss-Free Routing Equation

Router biases update per step to prevent expert collapse without penalizing primary task loss using the exact sign-based rule:

$$b_i\leftarrow b_i+\gamma \cdot \mathrm{sign}(\bar{f}-f_i)b\_i\; \backslash leftarrow\; b\_i\; +\; \backslash gamma\; \backslash cdot\; \backslash operatorname\{sign\}(\backslash bar\{f\}\; -\; f\_i)$$

where $\gamma = 0.001$, $f_i$ is empirical expert frequency within the batch, and $\bar{f}$ is target uniform frequency.

Router bias vectors are stored in FP32 precision or initialized to uniform values during pre-training to ensure stable routing dynamics.

Domain Clan Partitioning

The 16 routed experts are partitioned into 4 functional clans (4 experts per clan):

• Clan 0 (Experts 0-3): Systems code, kernel semantics, and concurrency.
• Clan 1 (Experts 4-7): Mathematics, formal logic, and STEM reasoning.
• Clan 2 (Experts 8-11): Security boundaries, protocols, and validation.
• Clan 3 (Experts 12-15): Structured Data, Tool Calling, and Symbolic Logic.

Training Pipeline and Reproducibility

• Training Code: Official pipelines and architecture implementations are maintained in the Mark-AI scripts directory.
• Curricula and Datasets: Trained on open datasets hosted on the Hugging Face Hub, including miceworks/Mark-Open-SLM-Curriculum, miceworks/Mark-Cyber-Curriculum, and miceworks/Mark-Ultra-Curriculum.
• Optimization Settings: AdamW optimizer ($\beta_1 = 0.9, \beta_2 = 0.95$, weight decay $0.1$, gradient clipping $1.0$).
• Training Scope: Experimental narrow checkpoint (EXPERIMENTAL_NARROW_CHECKPOINT). Trained for 400 steps on miceworks/Mark-Ultra-Curriculum using low-level kernel sources, systems manuals, RFCs, and concurrency primitives.

Evaluation and Benchmark Results

Demonstrates narrow curriculum completion for POSIX threads, synchronization primitives, and memory barriers.

Tokenizer and Vocabulary Specification

• Vocabulary Size: 4311 tensor rows (4,311 active BPE tokens + 27,721 reserved padding slots)
• Active BPE Merges: 3,981 merges
• Epistemic Control Tokens: Self-RAG reasoning tags (<think>, </think>), tool delimiters (<tool_call>, </tool_call>), epistemic reflection tags ([RETRIEVE], [/RETRIEVE], [EVALUATE], [ACCEPT], [REJECT])
• Measured Compression Fertility:
  • English technical prose: $\approx 2.53\text{ characters/token}$
  • Portuguese technical prose: $\approx 2.57\text{ characters/token}$
  • C, Rust, and systems source code: $\approx 2.02\text{ characters/token}$
• Whitespace Round-Trip Fidelity: Indentation whitespace requires byte-fallback tokenization; leading 4 spaces may be stripped by ByteLevel decoder when token 28 is raw space sequence.

Limitations and Intended Use

• Architectural Research Scope: This model is an experimental architectural testbed for Multi-Head Latent Attention (MLA) and fine-grained MoE on 8 GB GPUs such as NVIDIA GeForce RTX 2060 Super (8 GB GDDR6, 256-bit, 448 GB/s), NOT a general-purpose conversational or multilingual assistant.
• In-Distribution Curriculum Learning: Benchmark scores on Mark-SLM-V3-150M measure in-distribution learning on miceworks/Mark-Open-SLM-Curriculum (which includes official benchmark training splits permitted under Open SLM Leaderboard rules) rather than open-ended out-of-distribution generalization.
• Parameter Scale Constraints: Absolute world knowledge and zero-shot reasoning capacity are bounded by compact parameter counts.

Running Benchmarks on TPN Workbench and lm-evaluation-harness

When running automated benchmarks on TPN Workbench, lm-evaluation-harness, LM Studio, or Hugging Face evaluation runners:

• Select Format: gguf (or Backend: llama.cpp).
• Do not evaluate via raw Python transformers with safetensors or trust_remote_code=True, because the native deepseek2 MLA and fine-grained 3D MoE graph executes directly inside the C++/CUDA kernels of llama.cpp.

Usage: Narrow Domain and Systems Engineering Verification

This checkpoint is an experimental narrow checkpoint trained on specific low-level or systems engineering distributions. Outside its narrow domain, greedy inference produces fragmented outputs.

Cryptographic Verification

echo "983683d2e7885c9ce6589f7e2009ca4e9ebb891ff448c509a024732953780aba  mark_engineer_1_5b_iq4_nl.gguf" | sha256sum -c -

Execution with llama.cpp

llama-cli -m mark_engineer_1_5b_iq4_nl.gguf \
  -p "Systems engineering stub: Implement a POSIX mutex initialization routine.\nCode:" \
  -n 128 --temp 0.2

Python Inspection

import gguf

reader = gguf.GGUFReader("mark_engineer_1_5b_iq4_nl.gguf")
print("Architecture:", reader.get_field("general.architecture"))
print("Total Tensors:", len(reader.tensors))
total_params = sum(int(t.shape.prod()) for t in reader.tensors)
print(f"Total Parameters: {total_params:,}")