it’s very interesting to see where next-token transformers still fall short (like the path-star graph problems) while crushing problems that look even more daunting.
I had the pleasure of chatting with jayden teoh from MIT who walks us through next-latent prediction (NextLat), a simple fix in the training objective he worked on when he was still an undergrad!
the change is super tiny:
-> bolt a small MLP onto a GPT and make it predict the transformer’s own next hidden state.
to pull that off, the internal representation of the transformer needs to become a belief state, a compact summary of everything that matters for the future.
cool thing is that you can throw the MLP away at inference and the model runs exactly like a normal GPT, just with a way better internal world!
very solid lecture and we even have some great slides in there from jayden!
👉 Jayden Twitter: https://x.com/jayden_teoh_
👉 NextLat (thread):
👉 NextLat (paper): https://arxiv.org/abs/2511.05963
👉 NextLat (blog): https://jaydenteoh.github.io/blog/2026/nextlat/
👉 NextLat (GitHub): https://github.com/JaydenTeoh/NextLat
Check out Arcee (American open research lab and sponsor of this interview):
♥️ https://www.arcee.ai/
Also trinity large thinking is available on open router:
♥️ https://openrouter.ai/arcee-ai/trinity-large-thinking#providers
♥️ and check out the NAC harness https://www.arcee.ai/blog/nac
Table of content:
- 00:00:00 - what’s next latent prediction?
- 00:04:20 - walkthrough of the next latent prediction paper with your boy
- 00:14:46 - “hey jayden how are you man”
- 00:15:32 - introduction and background of jayden
- 00:19:25 - how did you find and cultivate your research field of interest?
- 00:23:43 - is failed experiment worth publishing?
- 00:25:57 - book recommendation for diving deeper into deep learning
- 00:28:49 - theoretical or empirical research design?
- 00:30:57 - where did the intuition for NextLat came from?
- 00:34:20 - [presentation] Next-Latent Prediction Transformers Learn Compact World Models
- 00:39:27 - methodology of NextLat
- 00:42:17 - why stop gradient?
- 00:43:50 - did stop gradient was used for the MLP?
- 00:45:17 - how do you think about this lack of generalization with the scale of transformer?
- 00:49:47 - how is deepseek MTP different from NextLat?
- 00:51:19 - better internal representation (of newyork)
- 00:55:11 - why is MTP representation much more compressed than GTPs?
- 00:57:58 - Planning (the path-star graph)
- 01:01:25 - Reasoning (with the countdown game!)
- 01:03:34 - stronger representation unlocked
- 01:07:43 - how is NextLat different from JEPA or LLM-JEPA?
- 01:10:13 - how would that loss work for other modalities?
- 01:12:36 - how would you extend this to predict earlier layers or several layers at the same time?
- 01:15:40 - language modeling and MTP
- 01:19:10 - speculative decoding directly!
- 01:23:13 - data-efficiency is super important
- 01:25:09 - RNN without recurrence?????
- 01:28:20 - where does the trained MLP length generalization coming from?
- 01:31:03 - why not have more MLP at different layers?
- 01:34:38 - what’s next for NextLat?
- 01:37:40 - how about SEGreg instead of SG?
- 01:40:45 - have you tried to visualize NextLat hidden state after training vs something like GPT?
- 01:46:22 - do you think there are other inductive biases we should explore forcing the transformer to have?
- 01:56:53 - how to prove things are scalable?
- 02:02:34 - did you engineer the speculative decoding gain while designing NextLat?
- 02:06:04 - what’s next for jayden and this research direction?
enjoy! 🌹
