Link preview
Next-Latent Prediction Transformers [R]
Microsoft Research Preprint Next-token prediction is myopic. What if transformers learn to predict their own next latent state? Microsoft Research present Next-Latent Prediction (NextLat): a self-supervised learning method that teaches transformers to form compact world models for reasoning and planning. It also unlocks up to 3.3x faster inference via self-speculative decoding! On top of next-token prediction, NextLat trains the transformer to predict its own next latent state given the current latent state and next token. NextLat has a few key benefits: Representation Learning: NextLat encourages transformers to compress history into compact belief states. Better Data Efficiency: predicting in latent space provides denser supervision than predicting one-hot tokens. Faster Inference: via recursive multi-step lookahead. I'm super excited about this work. Please do check it out below: 💬 Blog: https://jaydenteoh.github.io/blog/2026/nextlat 💻 Code: https://github.com/JaydenTeoh 📝 Paper: https://arxiv.org/abs/2511.05963 submitted by /u/jayden_teoh_ [link] [Kommentare] reddit.com · reddit.com
Microsoft Research Preprint Next-token prediction is myopic. What if transformers learn to predict their own next latent state? Microsoft Research present Next-Latent Prediction (NextLat): a self-supervised learning method that teaches transformers to form compact world models for reasoning and planning. It also unlocks up to 3.3x faster inference via self-speculative decoding! On top of next-token prediction, NextLat trains the transformer to predict its own next latent state given the current latent state and next token. NextLat has a few key benefits: Representation Learning: NextLat encourages transformers to compress history into compact belief states. Better Data Efficiency: predicting in latent space provides denser supervision than predicting one-hot tokens. Faster Inference: via recursive multi-step lookahead. I'm super excited about this work. Please do check it out below: 💬 Blog: https://jaydenteoh.github.io/blog/2026/nextlat 💻 Code: https://github.com/JaydenTeoh 📝 Paper: https://arxiv.org/abs/2511.05963 submitted by /u/jayden_teoh_ [link] [Kommentare]
Comments