Fabric 1.5 Technical Report \ Fabric AI

Introduction

We introduce Fabric 1.5, a 0.7B parameter language model trained on 12 billion tokens using 8 NVIDIA H100 GPUs on a single DGX node. Fabric 1.5 uses Chunked Fabric Memory, a mechanism that compresses past context into learned summaries and blends them with local attention through a learned gate. This enables 32,768-token contexts at sub-quadratic memory cost.

Our main results are:

Fabric 1.5 achieves competitive performance against Qwen3.5-0.8B across MMLU, GSM8K, HellaSwag, ARC-Easy, and ARC-Challenge, while using over 100× less training data and a simpler architecture without vision encoders or post-training RL.
Chunked Fabric Memory improves validation perplexity by 0.6 points at 32K context compared to a local-attention-only baseline.
The model, training code, and evaluation harness are released under Apache 2.0 at huggingface.co/FabricAI.

Architecture

Fabric 1.5 is a decoder-only transformer with 24 layers, hidden size 1536, and GQA with 24 query heads and 6 key-value heads. The model uses RoPE with θ = 10⁶ and processes up to 32768 tokens.

Layer Pattern. The 24 layers follow a repeating pattern: two LocalBlocks followed by one FabricMemoryBlock (8 memory layers total).

LocalBlock. Pre-normalized GQA with a 2048-token causal sliding window, followed by a SwiGLU feed-forward network.
FabricMemoryBlock. Three parallel components: local GQA over the most recent 2048 tokens, chunk summarization (512-token chunks compressed via cross-attention against 4 learned query vectors), and memory attention over all summary vectors from strictly earlier chunks.

Local and memory outputs are blended by a learned per-token scalar gate:

α = σ(Wg·x + bg)
y = α ⊙ y_local + (1−α) ⊙ y_memory

At 32768 tokens, memory attention computes over 256 summary vectors (64 chunks × 4 summaries). Cost is O(L·S) with S = 256, approximately 32× fewer operations than full O(L²) attention.

Parameter	Value
Parameters	0.7B
Layers	24
Hidden size	1536
Intermediate size	4096
Query / KV heads	24 / 6
Head dimension	64
Sequence length	32768
Local window	2048
Memory chunk size	512
Summaries per chunk	4
Memory layers	8 (every 3rd)
RoPE theta	10⁶
Vocabulary	65536

Training

Objective. Standard next-token prediction with chunked cross-entropy (1024-token chunks) to avoid materializing the full logit tensor.

Hyperparameters. AdamW (β₁ = 0.9, β₂ = 0.95), peak learning rate 3 × 10⁻⁴ with cosine decay to 3 × 10⁻⁵, warmup over 120M tokens, weight decay 0.1, gradient clipping 1.0. Micro-batch size 1 per GPU with gradient accumulation steps of 2, effective batch size 16 sequences. FP16 precision with activation checkpointing.

Hardware. Single DGX node with 8 NVIDIA H100 80 GB GPUs, PyTorch DDP with NCCL, FlashAttention-2 for fused local attention. Memory capped at 36 GiB per GPU.

Datasets

Dataset	Tokens (B)	Weight
FineWeb-edu	6.0	50%
DCLM	2.4	20%
OpenWebMath	1.2	10%
Cosmopedia	1.2	10%
Code (The Stack)	1.2	10%

For post-training, we use approximately 12,000 supervised conversations spanning 11 categories drawn from SmolTalk2, Mixture-of-Thoughts, and internal data. Full training logs including loss curves, learning rate schedules, and GPU utilization are available at the HuggingFace repository.

Results

Benchmarks

All values evaluated under identical conditions without CoT.

Model	Params	MMLU	GSM8K	HellaSwag	ARC-E	ARC-C
Fabric 1.5	0.7B	32.4	10.6	38.2	52.1	27.4
Qwen3.5-0.8B	0.8B	35.1	23.8	40.3	54.7	29.1

Despite training on 12B tokens versus the trillions used by Qwen3.5-0.8B, Fabric 1.5 is within 3 points on MMLU and 2 points on HellaSwag and ARC benchmarks. The largest gap is on GSM8K (13.2 points), which reflects Fabric 1.5 being a base model without instruction-tuning or chain-of-thought training. On knowledge and commonsense tasks, the gap narrows to 2–4 points.

Context Length Scaling

Model	2K	4K	8K	16K	32K
Local-only baseline	12.4	11.7	11.3	11.0	10.8
Fabric 1.5	12.3	11.4	10.8	10.4	10.2
Improvement	-0.1	-0.3	-0.5	-0.6	-0.6

The improvement from Chunked Fabric Memory grows with context length — from 0.1 at 2K to 0.6 at 16K — and stabilizes at 32K. The local-attention baseline saturates because its 2048-token window cannot benefit from tokens beyond that range.

Ablations

Memory Layer Frequency. Perplexity at 8K context improves from 11.3 (no memory) to 10.8 (every 3rd) to 10.6 (all layers). Gains are diminishing beyond every 3rd layer.

Memory layers	Params	PPL
0 / 24	0.7B	11.3
6 / 24	0.7B	10.9
8 / 24 (every 3rd)	0.7B	10.8
12 / 24	0.7B	10.7
24 / 24	0.7B	10.6

Chunk Size and Summaries. Finer chunking (256 tokens) improves perplexity to 10.6 but doubles the summary count. Increasing summaries per chunk from 4 to 8 at 512-token chunks yields only 0.1 improvement. We use 512/4 as the best trade-off.

Gating Behavior. Early layer gates favor local attention (α ≈ 0.9). Deeper layers show more balanced blending, with gate values on coreference and discourse tokens shifting toward memory attention (α < 0.5).

Memory-augmented transformers include Transformer-XL (segment recurrence), Memorizing Transformer (nearest-neighbor retrieval), Compressive Transformers (attention-pooling compression), and Infini-Attention (linear associative memory). Fabric 1.5 uses cross-attention compression with learned queries and a gated blend trained end-to-end.

Long-context transformers such as Longformer, BigBird, and Sparse Transformers use fixed sparse patterns. Chunked Fabric Memory is complementary: memory layers could be combined with sparse local attention.

Conclusion

Fabric 1.5 is a 0.7B parameter language model with Chunked Fabric Memory for efficient long-context processing. Training on 12B tokens on 8 NVIDIA H100 GPUs, it achieves results within 2–4 points of Qwen3.5-0.8B on knowledge and commonsense benchmarks despite over 100× less training data. Context scaling experiments show consistent perplexity improvements from Chunked Fabric Memory that widen with sequence length. The model is available under Apache 2.0.

Citation

@misc{fabric1.5-techreport,
  title = {{Fabric 1.5}: Long-Context Language Modeling with Chunked Fabric Memory},
  author = {Fabric AI},
  year = {2026},
  url = {https://huggingface.co/FabricAI},
}

Fabric 1.5 Technical Report