continual learning is one hell of a hot topic in 2026 and it’s beautiful because (like consciousness) barely anyone agree on what are we even talking about.
this is why earlier this year I sat down with Ilija from the prime intellect bunch to discuss what he thinks about the subject while bathing in his radiating enthusiasm!
in this 1h30min discussion we are going to talk a whole bunch on how he thinks about the components of continual learning while being exposed to a barrage of exquisitely curated literature.
amazing guy and lovely mind to exchange with highly recommend.
👉 Ilija Website: https://ilijalichkovski.github.io
👉 Twitter: https://x.com/carnot_cyclist
👉 Defining Continual Learning (main article): article
Table Of Content:
- 00:00:00 - what even is continual learning?
- 00:01:45 - ilija background in physics, ai safety, machine learning
- 00:08:05 - continual learning from first principles presentation
- 00:12:08 - time component in machine learning vs organic learning
- 00:17:30 - three flavours of incremental learning
- 00:20:10 - five desirable aspect of continual learning
- 00:23:10 - general capacity that should stay the fixed in continual learning?
- 00:32:40 - replay method in ML
- 00:36:10 - regularization family of methods
- 00:38:00 - meta learning methods
- 00:43:50 - hypernetworks method
- 00:46:25 - experiment with hypernetwork
- 00:48:45 - “what is a hypernetwork btw?”
- 00:49:35 - hypernetwork on split-mnist
- 01:02:18 - compositionally generalization experiments
- 01:10:45 - continual learning benchmark should be method agnostic
- 01:13:25 - on the generalization of LLM from in-context learning to finetuning
- 01:17:07 - more experiments with self-study
- 01:19:18 - OPSD vs GRPO experiments with Qwen3-8B
- 01:21:55 - learning different topics that are connected (or not connected)
- 01:28:20 - discovery shaped problem
- 01:30:40 - what’s next for ilija
enjoy! 🌹
big rafale of papers and blogs:
- Overcoming classic challenges for artificial neural networks by providing incentives and practice | https://arxiv.org/abs/2410.10596
- Three types of incremental learning | https://www.nature.com/articles/s42256-022-00568-3
- In vitro neurons learn and exhibit sentience when embodied in a simulated game-world (Cortical Labs, Pong) | https://doi.org/10.1016/j.neuron.2022.09.001
- Human brain cells play Doom (Cortical Labs CL1) | https://github.com/SeanCole02/doom-neuron
- Self-Distillation Enables Continual Learning | https://arxiv.org/abs/2601.19897
- Overcoming catastrophic forgetting in neural networks | https://arxiv.org/abs/1612.00796
- Metalearning Continual Learning Algorithms | https://arxiv.org/abs/2312.00276
- Text-to-LoRA: Instant Transformer Adaption | https://arxiv.org/abs/2506.06105
- Doc-to-LoRA: Learning to Instantly Internalize Contexts | https://arxiv.org/abs/2602.15902
- Human-like systematic generalization through a meta-learning neural network | https://www.nature.com/articles/s41586-023-06668-3
- Continual Learning Bench (website) | https://continual-learning-bench.com/
- Continual Learning Bench (repo) | https://github.com/pgasawa/continual-learning-bench
- Continual Learning Bench (paper) | https://arxiv.org/abs/2606.05661
- Negation Neglect: When models fail to learn negations in training | https://arxiv.org/abs/2605.13829
- On the generalization of language models from in-context learning and finetuning: a controlled study | https://arxiv.org/abs/2505.00661
- Learning to Discover at Test Time | https://arxiv.org/abs/2601.16175
- Learning, Fast and Slow: Towards LLMs That Adapt Continually | https://arxiv.org/abs/2605.12484
- A Theory of Usable Information Under Computational Constraints | https://arxiv.org/abs/2002.10689
pheeeeeeeeew!
