Overview

Fabric 1.5 is a 0.7B parameter decoder-only transformer developed by Fabric AI. It introduces Chunked Fabric Memory, a mechanism that compresses past context into learned summaries and blends them with local attention through a learned gate. This enables 32,768-token contexts at sub-quadratic memory cost.

The model was trained on 12 billion tokens using 8 NVIDIA H100 GPUs on a single DGX node. Fabric 1.5 is released under Apache 2.0 and is available at huggingface.co/FabricAI.

Architecture

Fabric 1.5 has 24 layers (16 LocalBlock, 8 FabricMemoryBlock) with a hidden dimension of 1,536 and SwiGLU feed-forward networks with intermediate size 4,096. Every third layer is a FabricMemoryBlock that augments standard local attention with a memory attention branch over learned chunk summaries.

Key architectural details:

  • LocalBlock: Grouped Query Attention with a 2,048-token causal sliding window, followed by SwiGLU FFN. Both sublayers use residual connections with Pre-RMSNorm.
  • FabricMemoryBlock: Three parallel components — local GQA attention over the most recent 2,048 tokens, chunk summarization (512-token chunks compressed via cross-attention into 4 summary vectors each), and memory attention over all summary vectors from strictly earlier chunks.
  • Gating Mechanism: Local and memory attention outputs are blended by a learned per-token scalar gate: α = σ(Wgx + bg), y = α ⊙ ylocal + (1−α) ⊙ ymemory. In early layers (1–6), gates are strongly biased toward local attention (α ≈ 0.9). Middle layers show more balanced blending.
  • RoPE position encoding with θ = 106, vocabulary size 65,536, tied word embeddings, and FlashAttention-2 on NVIDIA hardware.

Training

Fabric 1.5 was trained on a curated mixture of publicly available datasets totaling approximately 12 billion tokens:

  • FineWeb-edu: 6.0B tokens — high-quality educational web text
  • DCLM: 2.4B tokens — deduplicated Common Crawl subset
  • OpenWebMath: 1.2B tokens — mathematical web content
  • Cosmopedia: 1.2B tokens — synthetic educational data
  • The Stack: 1.2B tokens — open-source code

Training used AdamW optimizer (β₁ = 0.9, β₂ = 0.95), peak learning rate 3×10−4 with cosine decay, warmup over 120M tokens, weight decay 0.1, and gradient clipping at 1.0. The model was trained in FP16 with an effective batch size of 16 sequences across 8 GPUs.

For post-training alignment, supervised fine-tuning was conducted on approximately 12,000 conversations spanning 11 categories: general chat, instruction following, multi-step reasoning, creative writing, coding, function calling, mathematics, role-playing, summarization, translation, and safety. The model converges smoothly with validation perplexity decreasing from 5.89 to 4.83 over 219 post-training steps.

Evaluation

Fabric 1.5 was evaluated on standard benchmarks using the lm-evaluation-harness framework on a single NVIDIA A100-SXM4-40GB with FP16 weights, batch size 1, and no chain-of-thought prompting:

Benchmark Score Setting
ARC Easy 54.97% 0-shot, loglikelihood
ARC Challenge 26.88% 0-shot, loglikelihood
HellaSwag 35.00% 0-shot, loglikelihood
MMLU 28.96% 0-shot, loglikelihood
C-Eval 27.12% 0-shot, loglikelihood

Comparison with Qwen3.5-0.8B — Qwen3.5-0.8B is the most recent small model from the Qwen family (released June 2026). It is a vision-language model with a hybrid Gated DeltaNet architecture, a vision encoder, and post-training reinforcement learning. Fabric 1.5 is a pure text decoder-only transformer without vision capabilities, trained on substantially less data.

Model Parameters MMLU (0-shot) Architecture
Fabric 1.5 0.7B 28.96% Decoder-only transformer
Qwen3.5-0.8B 0.8B 29.70% Gated DeltaNet + vision encoder

Fabric 1.5 achieves 28.96% on MMLU — within 0.74 percentage points of Qwen3.5-0.8B — despite using approximately 100× less training data, a 12% smaller parameter count, and a simpler architecture without a vision encoder or post-training RL.

Context Length Scaling: The improvement from Chunked Fabric Memory grows with context length — from 0.1 PPL at 2K to 0.6 PPL at 16K and stabilizes at 32K. Validation perplexity at 32K context reaches 10.2, compared to 10.8 for a local-attention-only baseline.

Hardware Compatibility

The chunked memory architecture ensures memory scales linearly with context length, making 32K-token inference feasible on consumer GPUs:

  • NVIDIA H100 80 GB — batch size 16, FP16
  • NVIDIA A100 40 GB — batch size 4, FP16
  • NVIDIA A100 80 GB — batch size 8, FP16
  • NVIDIA RTX 4090 24 GB — batch size 3, FP16
  • NVIDIA RTX 3090 24 GB — batch size 2, FP16
  • NVIDIA T4 16 GB — batch size 1, INT8 / FP16
  • Apple Silicon MPS — batch size 1, FP16
  • CPU (any) — batch size 1, FP32

Quantization

Fabric 1.5 supports the following quantization methods for reduced memory footprint:

  • FP16 (native): ~1.5 GB VRAM
  • BF16: ~1.5 GB VRAM, <0.01 PPL change
  • INT8 (bitsandbytes): ~0.8 GB VRAM, ~0.95× speed, <0.1 PPL change
  • INT4 (GPTQ / AWQ): ~0.5 GB VRAM, ~0.90× speed, <0.3 PPL change
  • FP32 (CPU): ~3.0 GB RAM

Limitations

Fabric 1.5 has the following known limitations:

  • Scale: At 0.7B parameters, the model cannot match the knowledge breadth and reasoning depth of larger models (7B+). Users should not expect GPT-4 class performance.
  • Training data volume: Trained on 12B tokens, which is orders of magnitude less than modern language models (typically 3T+ tokens). Continuing pretraining would likely yield significant improvements.
  • Post-training maturity: Limited to SFT on 12K examples. The model has not undergone RLHF, DPO, or other preference optimization techniques.
  • Factuality: May generate plausible-sounding but factually incorrect information.
  • Bias: The model may reflect biases present in its training data, which is primarily English web text. It has not undergone debiasing techniques.
  • Language coverage: Trained predominantly on English text. Performance on non-English languages has not been evaluated.

Supported Frameworks

  • Hugging Face Transformers: Full support
  • PyTorch (native): Full support
  • vLLM: Coming soon
  • Apple MLX: Coming soon
  • ONNX Runtime: Coming soon
  • TensorRT-LLM: Coming soon

License

Fabric 1.5 is released under Apache 2.0. You may use, modify, and distribute the model freely. Attribution to Fabric AI is required when redistributing. No warranty or liability is provided.

Citation

If you use Fabric 1.5 in your research or product, please cite the following: 

@misc{fabric1.5,
  title = {{Fabric 1.5}: A Causal Language Model with Chunked Fabric Memory},
  author = {Fabric AI},
  year = {2026},
  url = {https://huggingface.co/FabricAI},
}