Research · language architectures
Dalton
A Mixture of Experts architecture with multi-head latent attention, aimed at more efficient inference.
Problem
Scaling language models with dense attention gets expensive in memory and compute. It is worth exploring architectures that activate experts selectively.
Approach
DaltonMoE: Mixture of Experts combined with Multi-Head Latent Attention, RMSNorm, and RoPE — architecture research, not a packaged product.
Outcome
Research code for the architecture (MoE + latent attention). It is an experimental base; we do not report invented public benchmarks.
Stack
PyTorch · MoE · Multi-Head Latent Attention · RoPE