usually post-training with GRPO will straight up collapse the model onto a tight set of potential answers (i.e. the spooky sharpening of the distribution).
which in some cases might be exactly what we’re looking to do.
however, in the grand 2026, we are increasingly using these models in long running harnesses that requires the LLM to think about different solutions and act on them compositionally to stumble on a discovery.
having the model being so cooked it can’t produce diverse set of solution anymore isn’t very optimal in these situation.
this is why ryan bahlous-boldi and co. from MIT took a deeper look at the machinery behind post-training to develop an objective that elicit these diverse set of solution.
Vector Policy Optimization (VPO) is what came out of this intense cogitation and leverage a few structural elements of post-training:
1. rewards are often vector valued so let’s keep that up?
2. in-context learning is pretty good already so why not keep in-context multiple draft solution?
3. let’s use up all the possibly weighting across rewards and make a vector that maximize all dimensions?
taking the mean of this bad boy give us the post-training objective that VPO optimize for and voila, we have a model that can do a whole bunch of cool stuff like clear up hard problem that GRPO can’t even fathom.
it was a very fun session and I learned a whole bunch from ryan highly recommend folks to keep an eye out for that scientist.
👉 Ryan Website: https://ryanboldi.github.io
👉 Twitter: https://x.com/RyanBoldi
👉 Vector Policy Optimization (main paper): https://arxiv.org/abs/2605.22817
👉 Vector Policy Optimization (main website): https://vpo-rl.github.io
Check out Arcee (American open research lab and sponsor of this interview):
♥️ https://www.arcee.ai/
Also trinity large thinking is available on open router:
♥️ https://openrouter.ai/arcee-ai/trinity-large-thinking#providers
♥️ and check out the NAC harness https://www.arcee.ai/blog/nac
in this 2h04 journey we have:
- 00:00:00 - diversity in RL finetuning
- 00:03:49 - quick explanation of VPO by your boy
- 00:09:07 - what make a good research paper idea feat. VPO + AlphaEvolve
- 00:16:59 - Heeyyyyyyy Ryan Bahlous-Boldi how you doing man
- 00:22:29 - core intuition behind VPO
- 00:31:50 - test-time search requires training for diversity
- 00:35:08 - is hindsight replay used to create new reward signals?
- 00:36:55 - will harness-level fine-tuning update heuristics directly?
- 00:38:38 - post-training (currently) optimizes mainly for one answer
- 00:40:01 - inference is no longer just one forward pass
- 00:40:44 - mismatch between exploration and exploitation
- 00:42:25 - how do you shape the reward for exploration?
- 00:45:40 - a magic trick 🪄
- 00:47:25 - multiple answer in your bag is needed
- 00:48:06 - you gotta have a diverse set of solution
- 00:49:30 - solution should be a vector and rewards are vector-valued.
- 00:51:35 - walkthrough of one beautiful figure godamn
- 01:02:37 - how do reward weighting, solution diversity, and sample count affect training when performance varies significantly?
- 01:05:35 - the different kind of dimensions to manipulate (number of solution / different type of weighting)?
- 01:12:39 - can mean based weighting overlook exceptional solutions? like is there better weighting method?
- 01:16:14 - why do the solution have a shared reasoning prefix?
- 01:19:23 - VPO covers the frontier of solution
- 01:20:53 - VPO results + ablations
- 01:29:15 - how does VPO compare with dense rewards methods that group each reward dimension separately?
- 01:30:19 - VPO ablations continued
- 01:32:32 - VPO on hard problems!!!
- 01:34:35 - VPO on even harder problems!!!
- 01:39:25 - diversity of VPO during training
- 01:40:35 - why this thing works?
- 01:44:30 - yacine ramble about solutions collapse with VPO and how defining space of solutions
- 01:46:30 - limitations of VPO
- 01:50:11 - the high level vision of VPO
- 01:51:42 - does uniform group scaling make optimization equivalent to max-at-N with nonlinear rewards?
- 01:52:49 - how are non-committal post-training exploration and continual learning connected?
- 02:00:53 - what’s next on this research direction?
