Abstract
Fabric Neural Network is a differentiable memory mechanism that augments transformer attention with compressed representations of past context. Every third layer uses a gated combination of local self-attention (O(L·W)) and memory attention (O(L·S) with S = 256 constant) instead of full O(L²) attention. At 32,768-token context, this reduces attention computations by up to 128× per memory layer head. On a 0.7B model trained on only 12B tokens, we achieve competitive performance against Qwen3.5-0.8B — a model trained on over 1 trillion tokens — closing 97.5% of the MMLU gap with less than 1.2% of the training data.
Introduction
Attention is the dominant cost in modern transformers. For a model with L layers, H heads, and sequence length N, the standard attention computation is O(N²) per head — a cost that grows quadratically with context length. Various approaches reduce this:
- Sparse attention (Longformer, BigBird) uses fixed patterns but cannot dynamically allocate capacity.
- Recurrence (Transformer-XL) propagates hidden states but loses fine-grained access to past tokens.
- Retrieval (Memorizing Transformer) stores KV pairs but requires approximate nearest-neighbour search.
- Linear attention (Mamba, DeltaNet) replaces softmax with linear operations but can lose expressivity.
Fabric Neural Network takes a different approach. It keeps exact softmax attention for recent context (2048 tokens) and compresses older context into learned summaries via cross-attention. A learned per-token gate blends the two streams. This preserves the expressivity of softmax attention where it matters most — recent context — while providing differentiable access to older context at constant cost.
Architecture
Fabric 1.5 has 24 layers arranged as two LocalBlocks followed by one FabricMemoryBlock (8 memory layers total). Each memory layer has three parallel branches:
- 1. Local attention. Standard GQA with RoPE over the most recent 2048 tokens. Uses FlashAttention-2. Cost: O(L·2048).
- 2. Chunk summarization. The layer input is divided into 512-token chunks. Each chunk is compressed via cross-attention against 4 learned query vectors Q ∈ ℝ4×d, producing 4 summary vectors per chunk. Cost: O(C·S·d) per layer where C is the number of chunks.
- 3. Memory attention. The input attends over all summary vectors from strictly earlier chunks using the same GQA projections. A dense mask enforces temporal causality. Cost: O(L·S) where S = C × summaries_per_chunk is constant given fixed context.
The outputs are blended by a learned per-token scalar gate:
α = σ(Wg·x + bg)
y = α ⊙ y_local + (1−α) ⊙ y_memoryThe 32× Reduction
At 32,768-token context with 8 memory layers:
- Full attention per head: 32,768² = 1.07 × 109 operations.
- Fabric Neural Network per memory layer head: 32,768 × 256 = 8.39 × 106 operations.
- Ratio: ~128× fewer operations per memory layer head, or ~14× fewer attention operations overall when including local attention.
The 83× Data Efficiency Result
We compare Fabric 1.5 against Qwen3.5-0.8B, the most recent small model from the Qwen family.
| Model | Params | Training tokens | MMLU | Efficiency ratio |
|---|---|---|---|---|
| Fabric 1.5 | 0.7B | 12B | 28.96% | 2.41 / B |
| Qwen3.5-0.8B | 0.8B | ≥1T | 29.7% | 0.0297 / B |
Fabric 1.5 achieves 97.5% of Qwen3.5-0.8B's MMLU score with less than 1.2% of the training data — an 83× data efficiency improvement.
Why Fabric Neural Network Enables Data Efficiency
The attention-to-capacity ratio explains the data efficiency gap. In a standard 0.7B model with full O(N²) attention at 32K context, approximately 40% of FLOPs go to attention computations. With Fabric Neural Network, attention FLOPs drop to approximately 5% of total FLOPs. The freed compute is available for feed-forward capacity, which directly contributes to representational power.
| Metric | Fabric 1.5 | Equivalent standard model |
|---|---|---|
| Attention FLOPs (fraction) | ~5% | ~40% |
| FFN FLOPs (fraction) | ~95% | ~60% |
| Training tokens to MMLU 29% | 12B | >1T (estimated) |
Ablation: Impact of Memory Layer Density
We measure perplexity at 8K context for varying memory layer counts to understand the contribution of each memory layer to model quality.
| Memory layers | Attention FLOPs | PPL | Improvement |
|---|---|---|---|
| 0 / 24 (all local) | O(L·2048) | 11.3 | — |
| 8 / 24 (every 3rd) | ~5% overhead | 10.8 | -0.5 |
| 24 / 24 (all memory) | ~12% overhead | 10.6 | -0.7 |
Adding 8 memory layers improves perplexity by 0.5 points with only 5% additional attention overhead. The improvement from 0 to 8 layers (0.5 PPL) is 70% of the improvement from 0 to all 24 layers (0.7 PPL), supporting the every-3rd-layer design.
Related Work
- Transformer-XL uses segment-level recurrence with fixed-size hidden states. Cost scales with hidden state size, not sequence length, but provides only approximate memory.
- Compressive Transformer extends Transformer-XL by compressing old hidden states through attention pooling. Fabric Neural Network differs by compressing raw token representations rather than hidden states, and by using learned queries rather than learned compression networks.
- Infini-Attention uses a linear associative memory with learnable delta rules. Cost is O(L·d²) rather than O(L·S). Our approach uses standard softmax attention over summaries, retaining differentiability and expressivity.
- Mamba / State Space Models replace attention entirely with linear recurrence. Fabric Neural Network keeps attention for local context, which may be more expressive for fine-grained tasks.
- Qwen3.5 uses standard GQA with no memory mechanism. Its 0.8B variant achieves 29.7% on MMLU with ≥1T tokens. Fabric 1.5 closes 97.5% of this gap with <1.2% of the training data.
Conclusion
Fabric Neural Network reduces attention computations by up to 128× per memory layer head while maintaining softmax attention for recent tokens and differentiable access to older context. On a 0.7B model, this translates to ~83× data efficiency compared to Qwen3.5-0.8B, a state-of-the-art small model using standard attention. The architecture makes 32K-token inference feasible on consumer GPUs and enables competitive sub-1B model training for approximately $1,000 in compute. The model, training code, and evaluation harness are available at huggingface.co/FabricAI under Apache 2.0.
Citation
@misc{fabricnn2026,
title = {{Fabric Neural Network}: Sub-Quadratic Attention at Sub-1B Scale},
author = {Fabric AI},
year = {2026},
url = {https://huggingface.co/FabricAI},
}