this very promising post-training paradigm where the model acts as its own teacher by conditioning on environment feedback or demonstrations.

we cover the SDPO algo for reinforcement learning with rich feedback and SDFT for continual learning without forgetting along with many applications.

we dig into how it works, why it’s simpler and faster than GRPO, and where this is already showing up in production systems.

table of content:

0:00 - what is self distillation

2:50 - idan (MIT) and jonas (ETH Zurich) introduction and motivation

18:40 - different perspective of on-policy self-distillation (presentation)

36:00 - metacognition and specificity in self-distillation

37:24 - very long hard task and self-distillation

42:00 - continual learning with self-distillation (presentation)

1:16:50 - what is next in this research direction?

1:20:00 - is there any experience with subjective feedbacks?

1:22:50 - quality vs number of feedbacks?

1:26:40 - what setting would self-distillation struggle vs GRPO?

my random thoughts on the paradigm

I think this is it. I think this will have the same impact as CoT had when it was first introduced and I would be very surprised given it’s strong performance against SFT that it is not already weaved into the main closed source models.

It also is becoming clearer to me that the traditional very rigid boundary between pre-mid-post training are starting to collapse a bit and that the reality is more of a mix that is very dependent on the expected performance of the model.

on a more meta note, I think having the model being it’s own teacher make just so much sense. like I was looking at my kids and I realized that each one of them has this bias of looking at the next in term of age for clues about how to do things.

they have this innate fascination about HOW the one that is a bit older is doing things, not even the eldest.

and I don’t think it’s just a fun quirk of nature.

the “policy” of the next in line kid is technically very close to whatever the base kid is at the moment. yes they can learn from me or my wife, but we are so much advanced that it’s a bit hard for them to understand how we are doing things (even though the movement are more precise).

it’s much easier to copy the one that is a bit older even with a slightly flawed policy and listen to them if they give feedback because they can just understand them better.

it reminded me of the example idan gave where if a smaller model was just listening to the big teacher model in robotics they will just hit point in the data they can’t recover from because they are going to make mistakes that are just not possible for the larger model.

anyway, this paradigm has a beautiful kernel of truth in term that is fundamental in my view and is a very exciting angle to get up to speed with!

papers

the slides were super crisp really cool of them to share!

if you are interested in digging more into the literature here is a few papers that are worth checking out:

📌 Reinforcement Learning via Self-Distillation (SDPO): :https://arxiv.org/abs/2601.20802

📌 Self-Distillation Enables Continual Learning (SDFT): https://arxiv.org/abs/2601.19897

📌 Aligning Language Models from User Interactions: https://arxiv.org/abs/2603.12273

📌 RL’s Razor: Why Online Reinforcement Learning Forgets Less: https://arxiv.org/abs/2509.04259

📌 Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs: https://arxiv.org/abs/2410.08020

enjoy my guys 🌹

now that I think of it the format does make sense since me and the interviewee are mostly chatting and exchanging ideas!

will be posting more of these in the future!

in this episode I had the pleasure of spending not 1 but 2 days interviewing the learning legend justin skycak.

we've talked about his quite impressive self-learning journey (3000h of math in high school) all the way to how he hand curated the initial knowledge graph for math academy to make that process more efficient.

I like justin because I understand his inner desire for learning.

I had the exact same “awakening” let’s say in my tiny bedroom when I was a teenager. I realized while doing some homework that I loved learning and found an immense sense of peace within it.

this realization kind of got lost in the midst of winter depression for a while until it got awoken again with a burning flare that never died down since (more trauma dump here).

this understanding that there is a meta-learning-skill is so fun because then the whole world kind of open up for you.

you encounter a hard problem you don’t know how to even get started? → great let’s use “learning”.

you got an opportunity that you are under prepared for? → great let’s use “learning”.

you want to make something seemingly impossible happen? → great let’s use “learning”.

what I like about justin’s story is that it is very concrete and touch math which has far reaching usefulness in life.

so here is the very lively 3h discussion where we touch these topics:
0:00:00 - intro:
0:02:10 - justin background
0:05:45 - 3000h math self study in high school
0:11:45 - what a day looked like for that 3000h stretch
0:16:10 - meta-learning vs pure math learning
0:21:50 - when did you get into cognitive neuro?
0:29:55 - how did the fundamental math helped in your research projects
0:43:10 - what does the math academy learning system looks like
0:47:34 - how did you guys build the 2000 topic knowledge graph
1:01:15 - would LLM be useful as an interface to that knowledge graph for the students?
1:10:46 - how does the FIRe spaced repetition algorithm works?
1:17:34 - does the same knowledge graph structure would work for physics? or other topic?:
1:34:05 - how do you understand the subject vs the curiculum
1:35:50 - is there a connection between studying math and learning a sport?
1:42:00 - do you think in math doing and teaching requires different skills?
1:56:25 - could you get understanding without automaticy?
2:05:35 - do you see any upside of confusion in learning?
2:14:11 - learning math as an adult?
2:19:20 - how to fill the motivation gap after learning the fundamental?
2:24:10 - how should teaching math for kids and adults balance fundamentals and creativity?
2:33:55 - is it ever too late to learn math seriously?
2:46:00 - mastery learning vs ultra learning
2:51:30 - top-down vs bottom-up
2:53:40 - mastery learning for domain without a structured hierarchical structure?
2:56:30 - neurodivergence / adhd for structured math learning?
3:06:20 - amateur mathematician augmented with technology will be able to contribute to research?
3:14:37 - what are you most excited about right now in term of learning

enjoy my guys! :)

I was a bit surprised how calm the city was on a wednesday morning. I always pictured san francisco as being extremely agitated.

the street of the headquarters was actually even calmer, like a really nice residential area. as I walked in I was welcomed by a researcher that just recently joined the team. he told me that they were moving offices soon as the place was getting a bit cramped.

I understood what he meant after moving out of the small inner courtyard and looking at the packed computer room where a team of absolutely focused engineers and researchers stood in various configurations. I was looking at a few rows of people I can only describe as not actually being there; they were in some of the purest forms of flow there is.

now. I’m a researcher at heart. I understand this particular state where you feel like mentally crawling over a shape thinly tangible but which terrain you start to map out. there is hardly anything in life I like more than to drag my metaphysical body in these spaces.

to not disturb these fine folks in their sacred trance I made my way upstairs. I realized too late that the situation wasn’t much better. a few designers were discussing and exchanging short sentences right in between making iterations on some kind of launch video (which in my humble paleolithic opinion looked pretty lit).

I sat down quietly at the dinner table for a few minutes trying to collect my initial feelings about this team. one that I’ve been interacting with cybernetically for quite some time now.

from where I was I could see the bay. a large boat was dragging its immense hull through the water. the contrast between this large expense and this crowded monastery made me feel that I was in a cocoon, a chrysalis of sorts.

as I was meditating about what the hell that even meant, a meeting suddenly started led by sami who also leads the research effort at prime.

the rumbling of all these people suddenly snapping out of their flow subspace and congregating around me was interesting. not that I mattered much. It’s just that I was on a kind of stool right in front of the screen (which I didn’t even notice).

getting right into it, sami showed us how their new intellect-3 model (100B+ MoE which was almost done training on 512 gpus) was generating some interesting SoTA results.

results which I usually only hear about in blog posts or tech reports online to my great excitement.

I zoned out for about 5 min thinking about how weird all of this was. here we were all excited about something we couldn’t touch, see or even understand entirely.

artificial intelligence has this interesting property that the substrate these researchers were working on isn’t the end artifacts (like the final model weights).

It’s not like software engineering or woodworking where the artist is sculpting, chipping away at its canvas until the vision is realized .

as I was looking at these guys looking intently at sami I realized that they looked like chemists or physicists building the machinery that creates the environment the desired result needs to grow into. and we were all looking at the result of this machinery producing state-of-the-art results in the realm of intelligence.

something about seeing the team that made it possible in a physical space was also pretty odd. because none of these researchers was entirely responsible for building this end system, but they all kind of were in each of their own ways.

as I snapped out of wherever I was I realized that surprisingly the presentation wasn’t a victory reel built to showcase how cool their results were. the team already switched gears to discuss with great excitement the shapes of things to come.

a disproportionate amount of time was now being spent on thinking and talking and hypothesizing about what they want to do for the next intelligence they wanted to grow.

I have a lifelong fascination with learning in all its facets and the last 10-minute discussion triggered by kalomaze about how it could be possible to push continual learning stayed with me well throughout the night.

It reminded me of all these evenings I’ve spent singularly focused on detecting consciousness in my little lab at the very top of the montreal general hospital. knowing that whatever blockers I was facing I just needed to talk it out with a colleague and figure out the shape of it and that at some point one of these ideas would allow me to hold onto something in that darkness.

sometime I wonder why I haven’t just continued chipping away in that trance-like way of living forever and I then remember I decided to build a startup and have three kids.

I love startups btw. the energy in a healthy one is very palpable especially when the team is holding one of these rare gems of value they know they can actively refine. yet, it’s usually hard to keep the flow of value coming out without dropping said gem. the metaphorical pressure is just too much.

I always had this dream of doing both research and startup in some shape or form. finding such a gem with value pouring out in waterfalls all around me and pooling in such a way I can meditate until I die on some of these hidden mysteries of life.

I got a glimpse of that dream and how to make it a reality at prime intellect. that team is actually a mashup of researchers and engineers which are chipping away at superintelligence with the energy of a startup and the stillness of research. all while shaping their organization to grow and capture these streams of value.

yes folks we’re talking business now. I know I know. money sucks, but superintelligence isn’t exactly a cheap endeavor.

I was zoning out again as I was looking at these words on a slide labelled mission in the applied research meeting. I don’t even remember which day it was. my stay was such a blur.

but I remember thinking “what an interesting combination of concepts to strive for...”

I quickly saw by the discussion that ensued on that slide that they were actually serious about the whole lot:

• bringing about superintelligence using tools they built.

• providing these tools to everyone.

• doing that work in the open.

It was getting hard not to love these guys honestly.

I was sitting later that night in an overcrowded in-and-out out on fisherman wharf with my double-double thinking that there was something about the current era of machine learning that felt electric to the practically minded soul (which I’m a proud member of).

the empirical nature of machine learning shines with such a gleam.

you can just try things and get feedback pretty rapidly at a scale and breadth that I’ve never thought we would get to. back in the prehistoric machine learning time of 2019 all the ai projects were such a miserable slog of issues and problems and blockers.

now problems still exist but it feels as if you are running fast enough you can just leap past a whole bunch of them. these fine prime folks were running a whole lot and preparing to accelerate some more it seemed.

this pattern of intense focus on that multifaceted mission wasn’t isolated within the cocoon of prime intellect; the second meeting I got invited to was for the RL residency where a few researchers distributed all across the world got together to chat up about various projects they were working on.

I love student researchers man. one of the coolest slices of our society.

they are usually such interesting individuals; each one I met was a world. all have extreme uniqueness in how they go about things. very hard for them to be anything but themselves because they are thinking so deeply about doing the seemingly most random technical feat at the boundary of what is known.

they barely have any brain power left to fake or conform much.

one of them was showing us his 20 page draft notion page with the most radiant smile I’ve seen in a while. another researcher spoke about 42 words on a topic that was at the fringe of RL finetuning yet everyone understood. another was making a whole book environment and his neighbor in the google meet was making a universal verifier jesus man I was pumped.

they were all thinking and working and tinkering on the problems littering the path to this superintelligence mission with such a zeal. It was beautiful.

each project stretching the possible a tiny bit while being grounded in empirical findings these guys have been bulldozing in the past weeks. the interesting bit is that these jolly souls weren’t even full time employees. they were working on pushing reality in between their PhD works or right after they’ve clocked out of their dev jobs or at night when the stillness is conducive to these oversized ideas making contact.

one of these researchers found a hilarious result that appeared when there was a visual moral dilemma on some closed source models. the personality of all the models was surprisingly on-point glowing through their weights when confronted in the visual spaces with two bad moral outcomes.

and here we were all looking at this quirk of intelligence and thinking about ways of bending it to our will some more using anime filters.

I’ll be honest I felt good. not only because I’ve escaped the most random winter storm to hit montreal in november no no (a bit yes), but because I was surrounded with such an interesting breed of people that all are pushing in the same beautiful direction.

the last time I felt so intellectually good was in 2018 when I was at a conference about transformers at the Mila with the one and only yoshua bengio. he was being asked in a hundred different ways if attention was truly all we needed to scale intelligence by a whole crowd of very ecstatic academics.

funnily enough the man had one of the weirdest prosthetic casts on his leg and at first I thought his whole leg was robotic or something. yet no one cared.

people were eagerly grasping and crawling over what he had to say, anything really that he could show or hint towards the path forward.

in the days that passed in the san francisco bay I had many other conversations with the denizens of prime intellect all ablaze with desire to make things move.creet_z who comes from their OSS effort (check his blog post on GRPO btw very sharp),ash with a superb operational mind and that gave me one of the best compliments I ever had (almost got a nosebleed from it),will brown with a mystical third eye for RL,vincent and johannes who have such a clear vision of what prime intellect shape should be and the many others I took out of flow states with my left-field questions.

as someone who dislikes groups in general; I kinda liked these unique people and wished I could spend a whole week with each of them. they all felt so unique yet all came together in these microstructures that blended their skill sets and perspectives into a coherent whole.

this whole setup felt reminiscent of some words from karpathy that I read from a medium blog post about modules melting into an optimal whole (waaaaaaay back in the software 2.0 days).

It felt like these folks were fusing their intellect into a strange and intense logical machinery to melt orbs of intelligence to create a goddamn whole lot of babelic trowels.

pretty rad if you ask me.

while I was there I decided to sample a bit more of the city and its surroundings by extending my tiny digital tentacles. I dined with big tech researchers (some breaking NDAs), chatted with people building in ai labs (big and small), guys doing startups late at night and academic researchers at stanford on a secret mission to decode the brain (they hiring btw).

through each of these individual personalities, I could see they were reaching in a similar subspace in various ways. somewhere they all felt something gigantic slowly rumbling over.

each one of them interpreting its shape through the lens of its own ideals and thinking about harnessing it for their own purpose.

as my plane took to the beautiful san francisco sky I was still thinking about the shape of what we were all building.

not only at prime intellect but at all the different labs around this city and the world. thousands upon thousands of minds iterating on the ideal of intelligence to usher in its metamorphosis.

maybe the agitation that I pictured in my head about this place was real after all. happening not in the street but in the curvatures of this new idea looking to take form. all these offices, apartments, co-working places focused inward in rearranging how intelligence is done.

whether open, closed, decentralized or monolithic, something was moving within the chrysalis and about to take flight.

I received quite a lot of distressed queries from a pretty funny population of engineers which is the curious kind.

those with such an insatiable desire to learn that they end up in a peculiar situation where they feel completly lost.

it starts to dawn on them that this beautiful universe is infinite in the way it’s shaped and that their mortal shell will not hold long enough to understand any sizeable portion of it.

so then they feel stuck and they end up either in constant FOMO or jumping from one thing to another like a puppy in a buffet.

what I usually tell these precious rare souls is that they haven’t found their thread yet and that they should look closely.

and that’s what I’m going to show you how to do right about now with a rather convoluted story.

short trauma dump about my seasonal affective disorder

I get SAD during winter.

like real SAD.

it’s better now that I’m literally pilled up all winter on high doses of vitamin D.

but there was a time right in between high school and university that it was pretty bad.

I was walking the beautiful winter soggy path between the subway station and this very inspiring and colorful pre-university school (CEGEP for those that speak quebecois).

god damn.

everything was gray.

the trees were gray the clouds were gray the road was full of all kind of greyish grime the road was gray the metro was gray I believe I had a gray shirt and when I got inside the cafeteria I saw different tones of grayish brown.

I was looking forward to jackshit.

not even to be done with my day because the road home will be darkish gray with the everlasting winter dome hovering above my head.

I proceeded to move FSD-style on the third floor where I found an empty spot to just open up my books.

now that day I came in a bit earlier than I usually do because in my grayish luck I didn’t realize that my morning class was canceled.

so I mechanically opened up my books.

and this is where I caught a glimpse of my thread.

a sparkling glimmer that looked almost blinding to my very SAD eyes.

it was a physic book if I remember correctly about electricity and magnetism. nothing cutting edge that would blow fullfilled people boots away.

but for me it was a glimmer of something I truly enjoyed at a cellular level.

I was reading doing some exercise reading some more correcting my little mistakes and I was having a jolly time.

I felt a true hue of something passing through my little parched and wrinkled gray soul.

and suddenly as I was reaching inside the depth of the universe through the easiest magnetic fields questions you could think of I latched onto my thread

I couldn’t even tell at that time what fabric it was made of. but I felt so so good holding it.

THEN something even more remarkable happened.

some of my friends saw me and came to sit with me. smile in hand they all were surrounding me like a warm and earnest social hug.

now I love these guys. truly great people with most likely a much more well balanced mind than I ever had.

but at that time it was like a whole fat gray persian rug was thrown on my head and was suddenly unrolling it’s beautiful gray motifs over my eyes.

that particular group of friend didn’t give two shit about physics let alone any particular subject. they were purely social beings of the highest degree. they were into exchanging and having fun and making amazing social plans.

now. I’m not a recluse or an antisocial mole dweller.

I like people I like to chat.

but at that particular moment in time I would have spelunked and wiggled my body down to a comfy hole if I could just open that godamn book and study just a few more minutes.

I then did something that was totally out of character which was to excuse myself because I had to go to a non-existent class at the fourth floor.

I got up took the escalator to the fourth floor then another floor then I walked down a corridor then I climbed a whole bunch of stairs into this secluded tower that hosted the math classes.

I walked some more and I would have continued to walk through the wall if I wasn’t at the literal furthest end of this beautiful gray building.

I opened my books studied some more and fucking loved every minute of it. I didn’t stop at the physics stuff god no. I did my math problems I read that novel I had to write a report on I read and worked and studied aahhhhhhhhhhh man.

it was during this stretch of time and the weeks that followed while I was sitting in my gray puddle of SADness that I realize that I love learning.

I loved it to a ridiculous degree.

not just the physics or the math or any subject really.

It was this act of learning this meta concept of learning that brought me boundless joy.

felt like a 3 yo davinci discovering painting existed with hands and feet completly smudged with a joyful mess of colors.

I got better over the years to allow myself space to enjoy this learning act while not ending up like a crazen oakenshield alone in his room filled with gold.

but that thread that I grasped that day I never ever ever let it go.

what can you do with this thread?

great question buddy.

nothing, the thread is useless.

hope it was useful 🌹

best,

yacine mahdid

for more learning topic do consider subscribing haha no man I’m joking.

but for real this thread that I clasped with my rattling hands from the jaws of the universe has no purpose.

a bit like the color purple or a triangular shape has no purpose.

these little threads that compose our existence are higher level concepts and construct that are layered in our slice of the world.

if you look closely in different situations you can see them being there holding up our human conceptualization of what’s going.

remember that group of friend? boy did they find their thread. I don’t even know what the name of the thread they found was but it must have been something like “social connection” or “friendship through sharing” or whatever.

I’m very happy for them. my thread was something else.

with that thread firmly in my hands I started to make my moves but always by looking for its very distinct shape and color in whatever I was doing.

sometime it was vibrant. layering a whole room in thick woolly texture.

other time it was there thin and barely visible (you had to squint hard).

and a few time it was just not there severed by accident or sabotaged.

that thread helped me finally decide what I wanted to do with my life which was to learn about learning.

I went into cell and molecular biology in uni to really understand how cells worked. I was in particular very interested about neurons and how they manage to create this concept of learning I loved so much.

I was voraciously learning anything I could put my hands on about the brain and its functions.

I pushed my learning skills academically and outside of a classroom for the pure thrill of it.

there wasn’t a single subject I didn’t find the thread right there for me to hold. even things I couldn’t give two lints about (like cooking) I found it solidly anchored on the oven and the pans.

there were situations though where this thread was so weak that it was waste of time to look for it.

so I cut myself out of these setups pretty quickly.

from the outside it looked sudden but from my perspective thread in hands it was nonsensical to continue wasting my days while I could just go where the thread was so much more there.

what I did with this thread

I did a whole lot of research in different labs

I published scientific articles ran some experiments that tested the limit of how I understood how to learn

I taught to hundreds of people in person online 1-1 in groups video blogs power point via dms anything goes

I got into artificial intelligence where the blueprint of this thread is dissected to its first principle and the learning stream can be controlled and oriented towards useful tasks

I also found this thread with such a gleam in entrepreneurship (albeit totally by accident) which led me to build a cool ai company in the industrial supply chain space that solves a real problem

I also became a father and found that thread with all it’s primal radiance within these babbling learning masters

more generally it gave me these little secret superpowers:

• even when I go into seemingly absolute random tangents I never feel I’m off track because I see that thread as clear as day.

• I rarely feel any sort of FOMO because what matters to me is if this thread is there in the situation I’m in. there is massive advance in thermodynamic computing? that’s very cool man I’m looking at latent reasoner right now.

• It gives me a silver lining whenever I work on the most boring tasks known to man (grant writing) because I can always bring it back to this higher level concept that is dear to my heart.

what now? what do I do yacine!!!

get depression.

go into debt if you have to.

(note: do maybe delete this dumb joke before publishing)

it’s rather simple. you just need to look and peer and look at all the stuff that truly brings you a sizeable amount of joy in life.

shed for a minute all the other’s expectations and look deep. shed also the convention about topics which are very arbitrary delimitation made by others.

these thread lingers and appear across subjects.

maybe it’s a thread called “creation” maybe it’s something about “community” maybe it’s “power” or maybe it’s also this “meta-learning” thingy I got going?

I can’t know for you and these threads have this wondrous quality of not having clear-cut names.

but once you have it in your hands you would find it ridiculous to let go.

hope it helps buddy god speed

hello toddlers today we are assembling the hierarchical reasoning models by sapient intelligence step by step.

it’s a very fun kit you’ll see

you will be able to use this architecture on all your favorite puzzles like:

• your grandma sudoku

• your dad sunday newspaper maze

• your uncle francois chollet arc-agi 1 test sets

on the last puzzles set you can even get about ~37% of them correct in like 12h.

It’s great value for the price, but watch out kids some steps require adult supervision, so grab one of those.

also please don’t eat the puzzle embeddings at the end.

they are small and absolutely important (you can also choke on them so you are legally warned here).

⚠️ disclaimer for adults:

for a complete assessment of the HRM usefulness, do check out the full video by our collaborator Julia Turc:

step 0: get a 27M encoder block

first block you should reach out for is the encoder-only block which is about 27M parameters in size.

the block looks like the one on the left:

remember kids we are building a puzzle solving model not a language model (yuck).

those sucks kids, you hate them because they are ugly and pompous and really wasteful when you think of it.

if a language model was a thing, it would be a dictionary of the effective functions to recconstruct the humans representation of the world.

BOOOOOOORING bleh.

anyway, you are going to feed the full puzzle all at once and return it transformed with the proper solution added so no need for a decoder.

like this:

step 1: plug the 27M block to input and output source

now carefully plug the encoder with the input block and the output block like so:

the input block will turn your puzzles into fun embeddings and the output block will spit out a solution for the puzzle the model is working on.

you can turn the power on and test it.

do notice how much the output puzzle kinda suck.

it’s because this 27M encoder model is fixed depth which is poopoo when dealing with puzzles that requires making mistakes and backtracking.

there is an easy fix for this.

step 2: add a latent recurrence wire

now carefully unplug the input and output block and add the thin latent recurrence wire.

like so, very simple modification:

at first glance the wire might look like a chain-of-thought wire from these disgusting large-language-models. yuck no.

those are slimy and no good at all.

no no no my sweet child we are dealing with latent reasoning here.

this is pure, sweet, like the hug from your mom after coming back from school.

as a legal disclaimer we have to say that the HRM didn’t invent this latent reasoning and we can see the exact same pattern in the universal transformer.

adding this latent reasoning loop changed the architecture for one with fixed depth to one with variable depth.

the model can now technically loop however long it wants.

now test it. it’s better isn’t it? but wait…

oh no! this poor model got lost.

poor thing doesn’t remember what to do.

we can’t leave it like this, let’s help it out.

step 3: add a recall wire

models like this need to be told to always keep an eye on their original input.

so take the recall wire and connect it back to the model.

now this is called a recurrent network with recall and it’s a good thing.

but your mom didn’t buy you this kit to build a good model, she wants you to do great things ™️.

let’s look at rats.

step 4: rats and funny pranks

now listen close because this is important.

big important adult, which we call neuroscientists, had a job for rats.

they gave the rats the task to find a funny surprise in two bucket in two different room.

the trick for rats was that the funny surprise was switched in the two rooms (what a funny little prank).

now, the neuroscientist weren’t just pulling the legs of the rat for fun, they had an important job to do.

they were secretly looking inside parts of the rat brain (called the hippocampus, looks like a seahorse yehaaaw) to see how the rat was learning about the prank.

what they saw was that some parts of the brain were working and talking to each other to learn the trick!!!!:

• some part were working rush rush rush because they were very busy and had lots of rapid information to handle.

• other were working sloooooooowly like an old adult and helping the busy part get the work done.

this great teamwork made it possible for these poor little rat to not get too confused and understand the jolly-ol’ prank from these funny neuroscientist.

now we too love collaboration and working with friends.

let’s add collaboration to our model!

step 5: hierarchical reasoning

now, I will need you to take a deep breath and be big a big boy/girl ok?

yacine was pulling a little fun prank on you since the beginning (just like with the rats!).

we are going to remove some of the puzzle pieces we put in because they weren’t the right one to put in the first place! (oh no!)

don’t get mad, you will see it’s a good thing.

you will need to take two smaller identical encoder-only blocks and place them as follows:

• one of the blocks needs to be wired in the input (with recall and recurrence wires); we’ll call this one low-level block

• and the other block needs to be wired to the low-level block (with recall and recurrence) and to the output; we’ll call this one the high-level block.

you might be a bit confused here, which one is my low level block and which is my high level block???

“IS THIS ANOTHER PRANK YACINE” I hear you say fist full of yacinego™️ block.

no my dear child, the blocks become a high or low level block because of the wiring!

the l module will be the one that will talk a lot, it will be recurring with itself T times before giving it’s work to the h module which will review it. the h module will then give gentle guidance to the l modules and it will do this N times.

in total now your model will talk and work in friendship on your puzzle N*T times.

good news also for your mom and dad credit card, you don’t need to run back-propogation-through-time with this model!

you can just use a one-step gradient at the very end of a N*T cycle!

now, if you test this model on sudoku and mazes you might be a bit disappointed with the result.

we are missing an important element, let’s add it.

step 6: add outer loop wire

now carefully take two recurring wires from the kit and plug them as so:

• plug one end on the h module end state to it’s starting state

• and another one on the l module end state to it’s starting state

the model is now able to do something very important which is spend more time thinking about a problem.

kind of like when someone asks you a difficult riddle.

you don’t like to be rushed don’t you?

well the model is the same, he can now spend more time thinking on a puzzle.

but wait a second…

how long should we let the model think?

that’s where the road lights come in, let’s add them.

step 7: add road lights

take the tiny road lights, the one with the green and red lights, and plug it at the end of the model like so:

with this road light the model will be able to know when it’s time to stop thinking and when to continue for another cycle.

this way if it hits an easy puzzle it can do it quick quick.

if it hits a very difficult puzzle it can sloooooow doooooown and do a lot more thinking.

It’s great! It’s a good thing!

with the magic of reinforcement learning, it can even learn to use the road light on its own (just make sure to wire it to the loss properly).

now you have a very good model for your grandma's sudoku and your dad's sunday mazes.

but uncle chollet arc-agi puzzles are still too hard!

how come?

if you look carefully in uncle chollet puzzles set you see that he only gives you two example for each puzzle.

“JUST TWO??? NOT FAIR” I hear you say.

worry not my dear child, this trusty kit has got you covered.

step 8: data-augmentation-framework™️

thanks to the data-augmentation-framework™️ you can turn any puzzles with limited instructions into a full dataset to train your model!

the only thing you need to do is to carefully lift and insert your HRM model into the provided case, like so:

the data-augmentation-framework™️ works by taking your few examples and doing all sort of funny changes (300-1000 of them) like wiggling a bit the puzzles, rotating them, flipping and even changing the color.

how fun for the dataset!

after all these silly augmentations, the model will do the same but in reverse to clean up the mess a bit.

because at the end of the day, even if we had a lot of fun, we need to be serious and get the job done.

so it will take all of the possible puzzle answers and count those that look the same.

the two answers the model will have counted the most will be what we are going to give to uncle chollet.

legal disclaimer: the HRM didn’t invent the data augmentation methodology for arc-agi it was already tried in a previous network that didn’t use pre-training called compressARC

we are almost there kiddo, almost we just need one tinyyyyyy piece to make everything work.

step 9: the last piece of the puzzle

look into your kit, you should see one tiny tiny piece which is the puzzle embedding layer.

take this piece, resist the urge to taste it, and plug it right at the beginning where your input block is, like so:

this tiny piece will take each transformation the data-augmentation-framework™️ has done on the arc-agi puzzles and will create a super duper secret name for it.

this secret name is very useful for the HRM model to understand what it needs to do, because remember this isn’t a bulbous disgusting hunk of language model junk (bleh bad bad).

this is a small and humble latent reasoning model; it needs all the help we can provide to do a good job.

nothing more nothing less, as our lord and savior intended.

and voila now with this kit you are able to solve all these puzzle!

how fun!!!!!!!!

don’t forget to ask your parent to get you the next yacinego kit, where we will add a moral judgment circuit into these disgusting language models!

the school year is soon starting back and I got reminded by a master student in germany how brutal it is for some.

so this sem, I doubled down, stopped all outdoor activities, spent every hour in learning the subjects and practicing math to the point I got saturated staying in my room (subject is Signal Processing) and I still got failed (I panicked in the exam of concepts I was familiar with and forgot them totally).

This is really bothering me as I thought I will get done with subjects and start writing papers as I want to become a Research Engineer.

my heart goes out to these poor students for whom failing a class is a likely stressor throughout the year.

I tutored all throughout my undergrad and grad time and I’ve seen firsthand how it can turn people into molten stress balls.

so here’s my simple mindshift steps to make sure there is 0% chance you will fail a class ever again.

disclaimer:

1. please don’t turn into an asshole

2. always anchor your action in fairness

3. this is mostly applicable to adversarial grading setup doesn’t work for research or practical projects.

step 1: burn your current grades mindset down

let’s get one thing straight; you are currently paying for this education (you and your dads and moms).

hard earned money.

not only that but with actual time your non-refundable life essence.

the way you are likely conceptualizing this transaction is the following:

1. you pay the school money and time

2. in exchange there is a teacher who will teach you about the subject

3. if you do a good job at learning the topics the teacher will award you points

4. if you get enough points you pass the class and can move on to other class

5. you can even get enough points to get these golden A grades that are such a pride for you and your genetic dynasty

my sweet summer child, what is this.

this isn’t 1943, this isn’t how any of this work.

you take all of these nice ideas and bury them deep deep deep.

step 2: flip this mindset up

it’s the year 2025 of our lord jesus guys.

everything is available online to an extent that the only gap between you understanding something is your drive to learn the topic.

you aren’t paying to learn.

you are paying to have a stamp of approval from a university that you have a validated baseline understanding of a set of topics.

the way they can give you that stamp without dilluting the value of the stamp is to grade you *objectively.

you are giving money and time to get the stamp, not to get the learning.

the deal is actually the following:

1. you pay the stamp provider money and time.

2. in exchange they assign a whole roster of teachers to GATE access to that final stamp.

3. if you do a good job at following the teacher's expectation they will remove less points in the class.

4. if you have enough points left at the end of the class you can move on to the next gate.

5. if you do a very good defensive job you can even get these A-grade which give you access to even better stamps!

*now now I know what you are thinking.

what is this bastardization of the learning institution? where are these golden value of learning and curiosity that should be instilled into every puppil heart?

hold that thought because it’s going to get worse.

step 3: you start with an A grade and your job is to defend that grade

with this mindset in your noodle basket next thing you need to understand is that every class start with an A grade.

your only job throughout the semester is to guard this grade from the teachers that have the mandate from the power-that-be to chip away at your beautiful and perfect grade.

now, they are only doing their job and 99% of teacher are doing it fairly.

they will tell you flat out how they are going to remove you points, like in excruciating details.

any modifier like curving down or up they usually also mention it during the very first classes.

you can even drop out of the class in most uni without any penalty during the first few weeks if you don’t like the deal.

this chipping away at your sweet beautiful grade is anchored in fairness.

it’s your job to understand how the chipping is done.

step 4: plan your defense

this process usually starts way before the semester start during the selection of the class.

you got options and agency here.

you can decide if this semester will be a confusing mess or an uphill negotiation battle or a topic focused season or an embarassingly dumb-easy ones.

it’s your life guys.

there’s many permutations you can do but overall here’s what principles I think you should adopt:

• avoid at all cost courses that are thought by a difficult teacher. if you like the topic they teach, well nice just self-learn it.

• select courses that make sense for your stamp (aka avoid courses on lunar cycles just because they are easy).

• try to pick topics that overlap in main subjects to get some transfer learning happening during the study sessions.

• if you can’t avoid a difficult teacher, make sure to spread them out so that you don’t get a whole semester in hell.

don’t forget that all of the above is subjective.

some people click better with a teacher than others or maybe you really fucking love lunar cycles.

just make sure you feel comfortable with your choices here.

step 5: understand how the points are removed

understand this for each of these beautiful hand-picked organically sourced classes you took ownership of going through.

it’s absurdly simple to figure this part out:

1. read that syllabus and study it.

2. figure out when points are up for deduction and to which proportion.

3. figure out for each point deduction events (i.e. assignment, midterm, take-home, finals, bonus exercise, quizz, attendance, lab) how they are deducted. sometimes you can’t have all the details too early.

4. whenever applicable try to get examplar of these points deduction event (legally) like an old exam, prior assignment with correction or quizzes.

5. whenever in doubt you go knock on these teacher doors and you flat out ask them.

I can’t emphasize enough how important this is folks.

you don’t have to yolo this.

it’s not a mystery how points are removed from your grade; there is a direct reasoning trace behind the grading scheme.

just get this information and document it for each one of your class.

step 6: fortify your fortification

you study for an assignment or an exam to not have your previous points deducted.

repeat this every time you open these pricey books.

this means a few things:

• the bulk of your study session should be aimed directly at defending your points. don’t go on to read a random chapter for the sake of it. is this in the grading scheme in any sort of way??? NO??? WHY YOU DO THIS THEN!

• sorry, but it needs to be emphasized folks, your time is short before the grading chipping start and the stress start growing so make sure that all your study actions are aligned with the point removal ritual.

• whenever you don’t know if something is going to be on the exam, just ask man. worst case you are mildly annoying and they are going to say ‘maybe’. you take that as a yes and you move on.

• move most of your learning to exercises that tightly mirror how you are getting graded.

  • multichoice knowledge exam? → flash card

  • open-ended question? → outcome-based project learning

  • short objective question? → drill all the exercises multiple times.

if there is a topic that resonate with your toddler soul with a vibrant note use it to decompress. but make a clear separation between that type of learning and the stamp type.

btw as soon as you know that your A is defended 100%, move on to the less defended class.

there’s no point in reinforcing the murud-janjira with taller brick walls.

step 7: keep your eye on fairness

now come the touchy subject which can be culturally very difficult to understand for some.

here we go:

your teacher is an human being and can make mistake.

oh boy, yes and it happens much more than you think.

the situation can get very very very sticky very fast if you don’t have your eye on fairness.

which means that for every point that is deducted it is your god given imperative to understand why.

this is one of the few spots in life where you are literally owed an explanation.

AND BY THE GREAT MONAD UP ABOVE YOU ARE GOING TO GET IT.

this is the portion of the article where I will emphasize two conflicting things:

1. you got to stand up for yourself

2. don’t be an asshole

since the situation can get emotional and very adversarial fast just try your best ok.

here’s some guidance:

1. when you get your corrected assignment/exam look at each of your deductions.

2. try to understand why you got points deducted.

3. if you do understand don’t snivel take the loss and fortify your defense.

4. if you do not understand you go see whoever corrected it for an explanation.

5. if the explanation has a logical flaw you will need to give your argument as to why that is and get your points back.

6. public forum are great for this if you have a knack for insightful and polite argumentative writing, but it is more emotionally charged for the teacher.

7. whenever a TAs commited a mistake and doesn’t want to budge, you NEED to escalate to the teacher and get to the bottom of it. doesn’t matter if it’s just 2 points these are yours.

this is the hardest realization for a polite and shy student.

you sometime have to stand the hell up and point to a situation which is unfair and sometime it’s not a fun discussion to have.

some ways to alleviate these tricky situations are the following:

1. whenever you find a potentially ambiguous question in an exam, even if it’s multiple-choice, always write why you think the question is ambiguous and detail your thought process for the ambiguity.

2. network with your peers to ensure that the fairness was evenly applied and to garner some group power modifier when dealing with a very difficult teacher with a penchant for unfairness.

3. build a relationship with the teacher by not only showing up for your points but also for your questions. they know they can’t just shoo you away because guess who’s coming back next week for office hour?

this pay-back crusade in the holy name of fairness is an integral part of the university experience.

final step: make use of your reclaimed time for real learning

now that you stopped wasting your time on bumbling around for your grade you can invest these newfound hours into something that has a high learning potential like:

1. get invovled in labs: you most likely have access to labs in your uni go volunteer in there to witness the limit of human knowledge.

2. get an internship: the stamp is for getting a job, try to secure that job beforehand my guy.

3. sign up in a student club: you will be able to network with driven folks, work on cool cv worthy projects and get proximity with the industry.

4. build your own cool projects: you have access to much more than just gates and stamps in uni. make use of the facility and all your perks in there to make something you are proud of.

anyway hope this useful folks have a fantastic 2025-2026 school year!

these past few weeks I’ve received a massive amount of dms from CS students being absolutely lost.

like to a level of lost I’m feeling their stress and anxiety in my own chest.

on top of that some of you are going through the absolute worst time of your life I don’t know how you guys are still standing up but I’m in awe.

I found myself reiterating the same advice to at least 7 people now, so I’ll write it all here for those who are too crippled to ask for help.

hope it’s useful folks and keep it up.

never surrender.

Hey I’m in {{UNIVERSITY or JOB}} and I feel lost can you help me learn {{AI/ML/MATH}} so I can get {{DREAM JOB}}

I’m receiving a variation of this, but the common element is always that the person feels completly lost.

lost because AI will take their job, lost because it seems there is too much to learn, lost because they don’t know if they actually like this whole tech thing, etc.

hold on.

hold it right there.

what do you want to be in like 5 years? who are you and what do you do?

most had no idea.

they had the vaguest picture of what they would be doing in 5 years.

and trying to figure it out by looking at technology is the wrong way to look at it.

it would be like trying to obsess intensely over hammers, nails, saw to build a house. tech is the same; it’s just a set of tools to do something.

the wafflehouse method ™️

enter my favorite method to get your life untangled, which I call the wafflehouse method (bear with me, I swear this blog post is useful).

I’ll detail here how to use it, but remember that untangling your life goals is a deeply personal process.

take inspiration from this, but feel free to modify so that it feels good.

step 0: take two days off in the middle of the week

call in sick, cancel your meetings and plans for two days.

straight in the middle of the week.

don’t tell your kids, wife, moms or friends that you are taking off.

you are going to go on a deeply personal introspective adventure and the only one invited is you.

48h to find yourself in the whole year is the least you can do.

step 1: [day 1] it’s time to vomit everything at your favorite spot

sorry for the image but yes that’s what you gonna do.

pick your favorite spot, the one you feel very good to be at, and can spend a whole day by yourself.

can’t give you advice on picking a spot just make sure that wherever you are going has deep significance for yourself.

mine was this bench.

now sit alone with your notebook and be quiet for a while.

try to imagine this one image: yourself 5 years from now.

not as of how you are going to get there from the current trajectory you are following, but how you would feel proud to be.

imagine it vividly, it may be still very abstract, but try to dive a bit in your mind what is going on in this vision.

does it have to be tech related?

no absolutely not, on the contrary.

imagine yourself at the pinacle where you really feel proudness bursting within your chest. this looks different for everyone.

try to remove anything attached to other expectations of yourself. doesn’t matter what your moms and dads want of you if you feel like a husk inside.

now that you have something to latch upon, write and write and write. write everything, how you feel now, how you feel then, what is different from now, what you regret, the things you think are not realistic, that one hunk of shame you have been carrying over forever, all these fears and hope lay them all down.

write and write until there is nothing more to be said.

write it all up until you have colors and shape of things you want to come to pass.

it doesn’t have to be clear for now, it just needs to be vivid.

when you are satisfied with this painting, it’s time to head home for a good day rest you are going to need it.

step 2 : [day 2] enter the wafflehouse

now is time to enter the wafflehouse (or whatever regional equivalent you have mine is called cora).

beautiful architecture

make sure that wherever you are going can tolerate you for a whole day because you are going to be squatting there opening until closing.

bring you computer, your notepad and this image of yourself.

step 3 : [day 2] refine the image into a 5-year goal

now that you are comfortably sitting with your beverage of choice it’s time to refine the image you painted into 2-3 clear goals that happen in 5 years time.

the goals need to be very specific and detailed.

if you feel absolutely ridiculous writing them down, it’s maybe too much, but don’t hold back. the goals need to match the image.

write as much detail as you can about each of these goals and make sure that there are as few as possible while still retaining the essence of the image.

step 4 : [day 2] time for interpolation on 3-year goals

now that you have these 2-3 5-year goals you interpolate what you need to achieve in 3 years time to be on track to hit them.

it’s ok if your interpolation is a bit crooked, you just need to trace a course here.

if you feel the 3 years are ridiculous, feel free to change the 5 years and adjust the knob of your dream self.

step 4 [day 2] bring it down down still on 1-year goals

now you do the same on the 1-year goal.

you are for sure going to do some back and forth between the 1-3-5 years adjusting things as you go.

try to keep the number of goals as few as possible.

if they sprawl around, it might be best for you to reduce the scope of your image a bit. for instance trying to be a nobel prize winner marathon karateka black belt might be too much.

focus on one aspect if you see your goals proliferating, the one that is more important to you.

I know it’s tough to make a call here, but you have to be somewhat pragmatic at some point.

btw the goals aren’t starting in like january 1st, they start now now.

step 5 [day 2] 100s of nodes in a directed acyclic graph

now you are going into the details for real. you’re going to make a DAG.

they look like this usually notice that there are no cycle (for the rest of the blog view them as the node on the left being at the top and the node on the right being at the bottom).

at the top it’s where you are at. put some numbers in there about your current situation (e.g. maybe you are broke).

at the bottom put your 1-year goals (e.g. maybe you are not broke anymore)

now in between write as many steps as possible you think you need to take in a year's time to go from your current situation to the end goals (remember there is more than one goal usually) based on what you have right now and the fact that luck is against you.

this last bit is important. your whole plan is based on the fact that every damn day you will wake up will be a bad day.

sun is shining exactly 0 times for 5 years.

create a 100s of these steps that flow logically from your current node to the end node.

some will be more vague than others. these are your knot points.

step 6 [day 2] : unknot the knot points with knotledge

for each of your knot points node you should research to the best of your current ability how to unknot them.

sometime it’s because it’s actually 5 nodes mushed together and you don’t know it’s 5 steps.

sometime it’s because it’s the wrong step and you are acting on false prior.

doesn’t matter try to untangle them as much as possible so that you have 100s of very highly defined step from start to finish.

as highly super duper detailed as possible.

step 7 [day 2] : make your monthly goals

this month.

not the other month which will start in 3 weeks.

now. this month now now.

take whatever first steps there is in this directed acyclic graph and put it as stuff to do in the month you have left.

here you have to be realistic about what you can chew, but remember that these tasks are what bring you closer to what you want to be.

that image that made you feel good remember?

yeah that’s the path you are rolling in now so take as much tasks as you can from the list and put it in the month.

cancel other stuff if you can to have more time to work on these.

step 8 [day 2] : make your weekly goals

you thought we would stop at monthly!

NO!

we’re going down still!

take whatever you picked for the month and bring them into your weekly goals for this week.

remember when we are now at the wafflehouse? in the middle of the week? well good news you got 3 days left to get these tasks done.

and before I see any of you start to spend 3 weeks making the perfect notion dashboard dingy NO!

you take whatever you have right NOW I don’t care if it’s on a notebook.

you spend 0 minutes optimizing your project management here, you take all the time into actually doing the tasks towards a future you are proud of.

step 9 [day 2] : make your daily goal

you should be getting dangerously close to getting evicted from the wafflehouse so take the little time that is left to look at your weekly goals and pick a few tasks to work on tomorrow.

the waitress should be looking at you like this.

at this point thing should be so concrete that even if you were to not remember your yearly goals it wouldn’t matter at all.

every day from now on you will have two category of things to do:

1. stuff that actually brings you closer to where you want to be in life

2. and the other stuff.

things that your mom wants you to do, the stuff that this one friends always put on you or that your boss decided you were responsible for.

I’m not saying to throw all of this in the garbage (. . .).

but you have to make conscious decisions that you have a plan that is yours and that it is up to you to get it done.

good day bad day.

step 10 : step out of the wafflehouse

now you’re done, it’s time to work and this work is your life's work.

you don’t need to parade this mega plan to your relatives it’s a deeply personal one.

it really doesn’t matter because they should start to see the difference anyway since this plan is one where you will transform yourself into what you truly want to be.

every day before going to bed take some of the stuff from this week and put it in your bucket for the next day.

when you wake up look at the task your past self is imploring you to get done and oblige.

every week on a quiet sunday afternoon, tend to this plan like a garden.

look out at the work done in the week, check what need to be done in the month, revise a bit the directed acyclic graph or the 1-3-5 years goals.

make the small tweaks based on your learning from the week that passed.

and when you still feel good about the trajectory and that this image still bring a smile to your soul pick a few tasks for next week.

disclaimer:

you have to be careful though.

if you actually do this you get a very stressful side effect:

• there is no one to blame but yourself

it’s not your mom, it’s not your teacher, it’s not your friends, it’s you.

from that point going forward you have your life in your hands and it’s time to act like it.

if this was useful to you in anyway hit me up periodically with the update.

you got this.

adamw is momentum based though

empirically momentum based optimizer tend to favor few update directions over the rest

update matrix look suspiciously low-rank

it’s like trying to roll a boat down the cost plane

muon’s contributor thinks that if we could dampen these directions and boost rarer ones it would lead to better convergence

kinda bringing the boat back to ball shape

muon’s contributor says that if we run svd on momentum update matrix and throw away the singular value then we get roughly unit vector length in all directions

can’t run a svd on each update step though too costly

we don’t have to be that precise so we can look at faster iterative algorithms

newtonschulz is one such algorithm

basically if you take an odd-matrix polynomial and apply it to a matrix it’s like applying it straight to the singular values while U and V.T are left unchanged

would be neat to apply the sign function to the singular value because then we would just have 1s on that diagonal

cant use sign because it’s not a polynomial even though it’s odd

we can approximate the sign function though with cubic/quintic polynomial if we apply the function multiple times on itself (not well but close enough around [-1,1])

5 iterations (or steps) are good enough

if we normalize the update matrix between [0,1] we are always sure that the approximation is sign-like and gives us a 1 out

actually, empirically, we don’t even need to be at 1 straight, if we are at [0.7,1.3], that’s good enough

the cursed quintic gives us one ugly-looking sign function but has all the right characteristics + it’s fast

it’s parameter are `a, b, c = (3.4445, -4.7750, 2.0315)`

that’s muon

ps:

• nesterov better than sgd-momentum

• muon only work on 2d hidden dense linear layer

• so no bias and no embedding

• convolutional filters are good you just need to put them back in 2d

• also empirically first and last layers should use adamw

32 min step-by-step breakdown here:

big shout out to SJ and Adrian for supporting the content!

also

more paper for deepdive:

📌 overview: https://rixhabh.notion.site/Muon-2302d1eca8f6801c88e3f80e43c316da?pvs=74

📌 Keller blog: https://kellerjordan.github.io/posts/muon/

📌 Jeremy blog: https://jeremybernste.in/writing/deriving-muon

📌 newtonschulz overview: https://docs.modula.systems/algorithms/newton-schulz/

1. Code generation via LLM is very valuable.

2. Coding benefits from improved long context.

3. Long context is still hard to get right.

It’s also not just about having a very long context LLM because you can theoretically get it right just by having an aggressive positional embedding interpolation and a hybrid architecture that makes it possible (like Scout).

There are multiple reasons why long context is hard:

1. It’s difficult to find quality and coherent long context data that is relevant for users. The benchmarks on this front are still in their early days.

2. The time complexity of self-attention does make training with large input cumbersome. Using linear attention mechanisms can remove that bottleneck, but it has a performance drawback.

3. Training on smaller average input lengths and testing on a larger one has a sharp performance decline in traditional RoPE-based systems. Positional interpolation helps, but there are limits.

More about this on my recent review of the subject here:

From RAG-based system to coding in mature repositories, the clear business case for LLM is in being able to ingest a large amount of context without breaking and produce quality output. The best models at the moment struggle beyond a certain point.

From what I’ve seen and read, the field will most likely figure out the recipe to create models with high-quality 1M- 10M context length in 2025. Yes, Scout can theoretically already do 10M, but I would qualify that as low-quality context.

The solution will most likely involve heavy use of hybridization of the layers, both for positional embedding (like Scout mix of RoPE and NoPE) and attention layers (like in Minimax mix of regular and linear attention).

It’s a hard problem, but after watching that full interview with Nikolay Savinov from DeepMind, I think the path is somewhat clear:

The emphasis on having solid inference work being done to make the model work makes me think that Google might still use a non-linear attention mechanism, and that a breakthrough is currently being researched on that front to unlock useful 10M context length.

In all cases, I think this concerted effort to increase context length will have a tremendous impact in multiple AI use cases like RAG-based systems, reasoning models, and coding agents. It will most likely fuel the next big phase of intelligence growth in LLMs, but will for sure require new ideas.

Interested in AI Engineering? Do check out the AI courses at Scrimba; these are pretty solid for beginners!

Have a great week 👋

They should theoretically be better since they allow for larger context length.

However, FlashAttention managed to get some more juice out of the quadratic softmax attention and extend the context length by sheer GPU optimization (which is great).

This is where I got into Lightning Attention, which cleverly mixes both the FlashAttention type of optimization and the linear attention type of algorithm.

A large-scale productionalization of lightning attention was recently published by Minimax as an MIT-licensed model (open-weight)!

With some pretty neat results!

Let’s go through the high-level theory here and figure out how the authors put together a large-scale linear attention model.

TLDR: The Main Findings

The Minimax-01 model integrates Lightning Attention with a Mixture of Experts that has it’s GPU usage optimized. Lightning Attention is hybridized with a 8:1 ratio with softmax attention.

This combination allows for:

• Big Context Lengths: Handling sequences of up to 4 million tokens.

• Efficient Scaling: Combining linear attention with MoE to achieve massive scale without compromising performance. A whole lot of GPU optimization had to go behind the scenes to make it happen (chapter 3 of the paper)

• On-par with closed source: The model managed to be on-par with most benchmark closed-source models using softmax and excel at large context length tasks.

The model is huge, with 460 billion parameters, and they even managed to slap a vision module on top to make it multimodal.

The Problem with Softmax Attention

Most leading large transformer-based models are making use of softmax attention.

This type of attention has great performance. However, it has quadratic computational complexity with respect to the sequence length. This means that as the input sequence grows, the computational and memory requirements increase dramatically.

Cool illustration taken from this blog post:

The AiEdge Newsletter

Understanding The Self-Attention Mechanism

Read more

2 years ago · 15 likes · Damien Benveniste

This quadratic complexity becomes a severe bottleneck when dealing with long sequences, like those found in document-level tasks, long-form question answering, or high-resolution images.

Addressing the Bottleneck with FlashAttention

To tackle the softmax attention bottleneck, Dr. Tria Dao introduced FlashAttention in 2022, a method that aims to make attention mechanisms more efficient.

The core realization was that most of the inefficiency in softmax attention was in reading and writing to memory. Not calculating attention per se.

This means that by just doing a better job at managing the GPU, massive gains could be had with noting changing on the model side.

FlashAttention did that by:

1. IO-Aware Computation: It optimizes the attention computation by minimizing data movement between the GPU’s high-bandwidth memory (HBM) and on-chip SRAM, leveraging tiling techniques to reuse data efficiently.

2. Exact Attention Calculation: Unlike some approximate methods, FlashAttention computes the exact attention, ensuring no loss in model quality.Which was a pretty big deal, effectively a free lunch!

When compared to even the linear attention mechanisms, FlashAttention was able to manage larger sequence lengths with less memory footprint by being just more efficient with handling the GPU.

Main Impact:

• Speed Improvements: FlashAttention significantly speeds up the attention computation, making it feasible to train models with longer sequences.

• Memory Efficiency: It reduces the memory requirements, allowing for larger batch sizes and models.

This was then integrated in large models and the methods for improving attention GPU utilization were researched some more. We even got FlashAttention2.

Where Linear Attention Falls Short

Difference between softmax attention and linear attention.

While linear attention methods like Linformer and Performer offer theoretical computational advantages, they often faced challenges in practice:

1. Performance Gap: Linear attention models struggled to match the performance of softmax-based Transformers, particularly in tasks requiring complex reasoning or long-range dependencies. Especially after the optimization from FlashAttention!

2. Training Inefficiency: Some linear attention methods, especially those relying on cumulative summation (cumsum), suffer from inefficient training due to sequential dependencies, which hampers their scalability. Making their theoretical advantage essentially useless when practical GPU implementation is needed.

Combining Both with Lightning Attention

Lightning Attention addresses the shortcomings of both traditional and linear attention mechanisms.

It achieves this by doing 3 things:

1. Hybrid Approach: Lightning Attention divides the attention computation into intra-block and inter-block operations. It uses conventional attention for intra-blocks and linear attention for inter-blocks, effectively combining the strengths of both approaches. It’s leveraging a divide-and-conquer methodology.

2. Tiling Technique: By adopting a tiling strategy similar to FlashAttention, Lightning Attention minimizes the need for cumsum operations, overcoming the training inefficiency of traditional linear attention. That’s where the intra and inter blocks come in. Do check this nice video for a refresher on that methodology.

3. IO-Friendly Implementation: The method is designed to be hardware-efficient, leveraging GPU parallelism and minimizing memory access overhead.

Figure 4: Lightning Attention splits the attention computation into intra-block and inter-block operations, using different strategies for each.

The Core Results of the Minimax-01 Model

The Minimax-01 paper introduces a model architecture that integrates Lightning Attention with a Mixture of Experts (MoE) to achieve scalability and good performance.

Interestingly, it’s not a full lightning attention model but rather a hybrid one. In a proportion of 1 softmax attention block for every 7 lightning attention blocks!

When only lightning attention is being used, the model falls short in retrieval tasks like needle-in-a-haystack.

Here are the key highlights:

1. Context Window Expansion: The model supports context windows of up to 1 million tokens during training and can extrapolate to 4 million tokens during inference.

2. Scalability: The model scales efficiently, with 456 billion total parameters and 45.9 billion activated parameters per token. However, it requires quite a recipe to get it right. The full recipe is given in sections 4 and 5 for the pre-post-training and wow.

3. Performance: The model matches the performance of state-of-the-art models like GPT-4o and Claude-3.5-Sonnet while having a much larger context window (20-32 times longer).

3 Key Takeaways for Transformer-Based Deep Learning

1. Attention Mechanism Evolution: These results demonstrate that hybrid approaches can overcome the limitations of traditional and linear attention mechanisms. Expect to see more innovations that blend different attention strategies.

2. Scalability and Efficiency: As models grow in size and complexity, efficient scaling solutions like MoE and linear attention will become increasingly important. These techniques will be crucial for developing models that are both powerful and sustainable. It also seems like hybridization of different techniques makes a lot of sense to strike the right scale.

3. Long-Context Understanding: The ability to process longer sequences might change how some applications that are making use of RAG systems are setup.

Figure 6: This NIAH figure was pretty funny, haha.

Overall, pretty cool methodology.

There seems to be something fundamental about softmax that is still required to make large language models perform, but linear attention mechanisms bring a new perspective to these models.

I wouldn’t be surprised if the new Gemini 2.5 integrated linear attention in some way!

Two elements are surprising in the paper:

The first surprising information, is obviously the claim that you shouldn’t build a fully autonomous agent. These are systems that can create and subsequently use their own code in order to achieve their own task.

Here we aren’t talking about coding agents, these are still fine. A fully autonomous agent isn’t bounded by a set of tools and can use coding agents in order to build more tools to use.

Huggingface researchers even did a full grid analysis on multiple dimensions. No matter which dimension you want to remove, the risk-reward ratio for a fully autonomous agent isn’t looking too good.

Like all things engineering, the sweet spot seems to always start with the simplest possible solution and move up the complexity later if needs be.

This sentiment from HuggingFace is surprisingly also echoed in an interview from Anthropic:

I don’t know if these two engineers have been briefed before the interview, but their candidness about the fact that agent are not really working in production is hilarious. Peak AI comedy, highly recommend watching the full thing.

The second surprising thing is more semantic, there is no clear definition of what an agent is.

You might be saying “Yes, it’s because of these pesky AI companies hyping up the AI field with non-sense” and you would be partially right.

The issue runs much deeper than that, with this quote being the eye opener:

The question what is an agent? is embarrassing for the agent- based computing community in just the same way that the question what is intelligence? is embarrassing for the mainstream AI community.

The problem is that although the term is widely used, by many people working in closely related areas, it defies attempts to produce a single universally accepted definition

This quote is from Wooldridge & Jennings in their book from 1995.

1995, folks.

This term is at the same level of murkiness as AGI, intelligence or consciousness at this point.

Which I believe requires engineers working in that field to use much more precise words about what they are building (workflows, prompt-chaining, orchestrator-evaluators, etc.).

A good place to start learning this semantic is this blog post by Anthropic, which is very clear and devoid of hype (it’s related to the interview mentioned earlier):

The team has been gracious enough to provide a lot example (text and coding) of what each of the patterns looks like in what they call workflows.

btw if you are new to agent, workflow, or agentic systems, I’ve made a tutorial on it last week with the resources above:

ps: if you know a thumbnail guy, send me a DM mines suck.

All in all, there isn’t anything too new here on the software engineering side with the statement from Hugginface.

The old adage still holds in the stochastic realm of AI engineering:

Build the least complex solution as possible that meets the requirements.

If you want to test new tech, like a swarm of AI agents, do it in a hobby project, not in production.

There is very very little in common between the two fields.

If we were to do the concentric circle diagram it would be the red X:

I’ll explain what I mean by that and why this confusion happens.

btw, if you are interested in learning more about AI engineering, like in a hands-on manner, do check out this great course by Scrimba!

Why are they not related?

They should be, at a glance, because we just showed that generative AI is within the concentric circle right under deep learning.

However, an AI engineer isn’t touching something fundamental in machine learning which is the weights of the model.

Generally, an AI engineer will modulate the input to a trained foundational model to gather the output they need.

In some rare occurrences, the AI engineer will dip their toe in the machine learning world by finetuning the model.

But the models are so big now and so complicated to train thanks to being mostly a mixture of experts that a smart AI engineer avoids this like the plague.

By the time you are done finetuning, there is a new model out there that would make your use case easier to build.

Truth is, with one or multiple foundational models you can stitch up solutions that are highly intricate and useful.

Why are they confused together?

They are confused together for three main reasons:

1. People who don’t know how to program see this as, rightfully, the same bucket as ML magic.

2. The techniques for AI engineering look from the outside so complicated that they are intuitively put in the same bucket as the rest of ML.

3. There is some element of “randomness” in classical machine learning systems that is equated to the tool the AI engineers are putting together.

But, when you look at the differences in the workflow the AI engineer goes through versus the machine learning engineer/scientist it’s very clear that they are at opposite ends of a research/engineering spectrum.

Machine learning starts with data, involves the its study, transformation, training of a model to generate the desired output.

It finish with a touch of engineering with the productionalization of the trained model to do the same flow with the customer.

AI engineering starts with a trained foundational model (aka big parameters), they then use (or not) the data from the customer to orchestrate a solution to generate something of value. This system is then deployed into production.

AI Engineering is Software Development of the Stochastic sort.

This is a very weird thing to say, but it does feel like that.

You are dealing with the equivalent of a black box API, in which the documentation is being studied actively and where the output changes for no apparent reason (even if the inputs stay the same).

This black box API however is very good at doing a whole lot of things since it has internet-scale knowledge of all things human.

It acts surprisingly, accepts any input, and always outputs something (no matter how ridiculous the request is).

Managing this stochasticity, this uncertainty, while wiring the input/output in a way that generates value is the core skill of an AI engineer.

Taming this stochasticity while balancing automation is currently a very difficult thing to do. Should you automate a whole workflow using these foundational AI systems or use them more as a co-pilot?

Being able to balance the value, with the risks of generation is at the core of AI engineering.

How to get good at AI engineering?

Build AI solutions. That’s it.

It would be like asking how to get good at Frontend development.

Should you start by learning the deep computer science theory underlying the DOM, browser, and networking?

Yes, at some point. But to start you should build a bunch of frontend solutions!

Good luck! 👺

Should you learn to program in 2025?

hell no - Jensen Huang CEO of NVIDIA

Especially given two market shifts:

1. Increase adoption of AI to write code.

2. It is difficult to find a job as a programmer.

Are the two things linked? Maybe a bit, but I see them as two separate events intertwined by narratives.

Here are my thoughts…

spoiler: you should and if you want to get started with the coding aspect check out this Scrimba course: learn Python (5.6h and free).

First off… What’s coding and what’s programming?

Coding is the process of writing instructions for a program, while programming is the process of designing and developing a program.

You take any experienced enough programmer and they can feel this difference in their bone.

The junior programmer usually has no idea it’s two separate things. Coding and programming are interchangeable term for sitting in front of the computer and writing lines of code (which may or may not work).

A truly senior programmer will be able to look at all the context fed through their senses and start formalizing and refining a solution.

This solution will be reworked and improved while they pluck context and add it to the requirements and constraints.

Once this flow starts, there is a moment where everything clicks in the senior programmer's head. They feel satisfaction and they “know” in their gut that this system is “right”.

The number of lines of code written could be 0, yet they feel they achieved their purpose because they understand that the coding aspect will be a deterministic process.

Yes, there is back and forth between that planning and the coding, but in general it follows an overarching blueprint.

What LLMs are good at?

LLMs right now are great at producing code in constrained environments. You still need to gear them in the right direction, sometime they do hallucinate, but if you have a good enough vision of what you want to build they are helpful co-pilots.

The more precise the context, the more generally solvable the tas the better they are at generating a solid working piece of code.

What they aren’t good at?

LLMs aren’t going to figure out for you what you should be building.

They won’t figure out by themselves the unseen context that is business critical and add it to the requirements.

They won’t understand what context has become suddenly irrelevant.

They won’t plan and formalize the solution in a way that makes sense for the specific stakeholders in your business.

You need a programmer for that.

Programming is formalized problem-solving.

There is a whole art that is applied to make sure that whatever information problem we need to solve as humans can be deterministically achieved by a computer.

The part that is complex here, is the understanding of the problem in all it’s constraints and requirements.

Being able to articulate that clearly and succinctly is what a good programmer does.

Do you remember what’s needed for an LLM to generate accurate code?

Context.

So logically yes, you should 100% become good at programming. Do use LLM co-pilot for coding as you develop your programming skills.

But put your big brain on focusing on the user needs and on having stellar communication skills with other humans.

That’s what good programmers already do.

Something about it is just joyful from a research perspective, especially the level of transparency displayed.

One element that caught my eye is their implementation of KL divergence as a penalty term in their GRPO formula:

For those who don’t know, KL Divergence is simply a statistic that measures the difference between two probability distributions. It’s tightly related to cross-entropy and entropy.

It’s used in R1 to “anchor” the model that is learning to reason with a “reference” model (Deepseek V3 in that case).

This means that if the policy model starts to behave very differently than the reference model it needs to have a strong performance to justify it.

This term is used differently in PPO vs GRPO:

The first thing to notice is that they shifted KL away from the reward term compared to PPO and directly into the policy model training.

In PPO the formula where KL divergence is being applied looks like this:

In the GRPO the KL portion looks like this:

Widely different implementation!

You might just nod and say “Fair enough, that’s KL divergence”. But it looks nothing like it… How did they come up with that penalty term?

If you go down the rabbit hole, like I did, you find an interesting truth that is at the root of the deep learning mindset. More on that at the end.

You can check out my video of the subject if you want a walkthrough:

The TLDR is that it’s empirically a very good Monte-Carlo approximation of KL divergence, with less variance than the version from the PPO algorithm while still being unbiased.

The proof of that isn’t even another research paper, it’s just a blog post that one of the co-founders of OpenAI, Schulman, posted in 2020:

Yet, the flow is elegant and I suggest you take a look at it (slowly): http://joschu.net/blog/kl-approx.html

In a nutshell here is the flow:

1. We can’t use KL divergence directly as it’s impractical for RL purposes and we don’t need to be too correct. The approximation is good enough for the training procedure.

2. The first approximation k1 is log(q/p) which is what PPO is using. This is an unbiased approximation but has high variance since k1 can be negative while the true KL cannot (bounded by [0,inf[)

3. The second approximation Schulman proposes k2 is 0.5*log(p/q)^2 which is a biased estimator, but with lower variance than k1. It has lower variance because it’s an f-divergence like KL and one of the properties of f-divergences is that when the probability distribution p and q are very similar, all f-divergences look roughly like KL.

4. The third approximation k3 is (p/q -1) - log(p/q) which is what we find in GRPO (a bit rearranged). This is an unbiased estimator because it is k1 + a control variate (p/q -1). That control variate has 0 expectation so it doesn’t change the biasesness of k1.

When Schulman ran the simulation he got the following results:

This shows that k3 seems to be strictly better, while k2 is not performing too badly either except when the true KL is getting larger.

What’s the takeaway?

A good portion of deep learning is very empirical. Is PPO wrong for using k1 instead of k3? Maybe, maybe not.

It might not even impact the training that much and that high variance penalty term might be just good enough.

Should k2 be used in GRPO or PPO? Maybe, maybe not.

It’s all about tradeoffs and validating empirically what yields at the end of the day the best model.

When you dig deep enough into the internals of deep learning systems, you realize the thousands of tiny design decision that have been made to get to the result.

Beautiful to gaze upon as a researcher.

Hope you are doing fantastic today!

I’ve been reading DeepSeek R1 last week.
I found the method, results, and just the general ideas within that paper super interesting.

I’ve made a walk-through over here if you are interested in going through it step by step:

Here is the TLDR and a map to follow along:

The end output of the R1 research is three family of reasoning models (all open source):

• DeepSeek R1-Zero

• DeepSeek R1

• DeepSeek R1 distilled into Llama and Qwen

All model creation pipelines have a root based on DeepSeek V3, a huge 600+B parameter model (also open source).

The research is doing post-training to induce reasoning into DeepSeek V3. Aka, try to make the model talk to itself in order to be better at complex tasks like coding, engineering, and problem-solving.

As seen above, DeepSeek R1 performance is on par with Open AI o1 which is closed source. Arguably replicating their methodology.

The interesting first result is that DeepSeek R1-Zero pipeline involves Reinforcement Learning with no human in the loop.

To prompt the model, they used this very simple prompt:

By adding a proper reward function on top of that prompt, the model was able learn to think longer and induce reasoning over time.

This increase in reasoning in the model is correlated directly with an increase in performance over time for DeepSeek-R1-Zero.

The magic of DeepSeek R1-Zero not requiring human feedback during the reinforcement learning process hinges on the Groupe Relative Policy Optimization (GRPO) and their rewards function.

For GRPO we have the following:

Which basically says:

• use current log probabilities for the policy across the group (pi) scaled by the old policy times the normalized reward (A) as an optimization step.

• don’t take too big a step compared to the old policy though (clip part).

• don’t diverge too much from DeepSeek V3 to parameters (KL divergence part).

For the reward:

The reward function at this point in the research was all deterministic and made sure the formatting of the reasoning stayed within the think tag:

This is a big deal as it breaks with the idea that we require reinforcement learning from human feedback (RLHF) to induce reasoning in models.

However, DeepSeek R1-Zero reasoning was difficult to decode, so the researcher posited that finetuning DeepSeek V3 prior to doing reinforcement learning would help the performance.

To do so, they made a whole complicated pipeline that consisted of generating reasoning and non-reasoning data in a multistep training process (see map - I got in painstaking detail in the video).

In the end they get a solid model, on par with o1, and with an intelligible reasoning context:

To make the models broadly useable they used distillation using R1 into smaller models like Qwen and Llama. The results are also very performant and MUCH smaller:

Finally, when they tried to run their reinforcement learning pipeline on a smaller model vs using distillation, they realized that distillation was better:

The bottom line for the distillation part is that it seems like reinforcement learning “heightens” the reasoning core that is already present in the very big models vs not really sharp in the smaller ones natively.

You can find all these models on Ollama and you can play with them easily on your GPUs:

That’s it!

Hope you enjoyed the TLDR, for more detailed information I suggest you check out the tutorial or the paper directly:

📌 https://arxiv.org/pdf/2501.12948

Have a great week everyone! 👋

I hope you are enjoying the last remnant of summer (if you are in the northern hemisphere that is).

In this edition I wanted to share a bunch of advice I would give to my former young undergraduate self or any undergraduate interested in research.

This thought was prompted by this message I’ve received:

Hi Yacine!

My name's [BLANKED OUT], and I'm a 19 year old 2nd year Undergraduate at University of [BLANKED OUT] studying computer engineering. I recently joined one of my campus' research labs as an intern, it focuses on ML applications within healthcare and medicine.

I'm still unfamiliar with many of the concepts we utilize, but I luckily have some coding experience.

I was wondering if you have any advice for a beginner in this field?

How did you become interested and so familiar with the machine learning concepts you teach in your videos?

I'm eager to make meaningful additions to my colleagues work but I currently feel like I'm taking up more of their time having them explain certain things to me-- rather than helping them with their research.

By the way, he’s talking about these tutorials I post on machine-learning topics that interest me.

My answers seemed useful to this bright student so I thought about sharing them here more broadly!

The Answers

Hey man,

First of all big props for reaching out, this is the right mindset to have especially if you are doing something as difficult as contributing to research while doing a bachelor's degree.

I’ll separate this answer in two, just so that you have the right context in which I’m answering it.

1. I’ll first tell you how I got involved in research as an undergraduate because I was in the same exact situation as you.

2. Then I’ll tell you what I would do if I could do it again.

A1: What did I do to contribute early on?

I got into a lab doing lots of physiological signal analysis and I was responsible for a portion of the coding.

The first thing I did was try to learn as much as possible about how everything worked.

I was mainly trying to build my knowledge of the lab bottom up and try to help in whatever way I could in whatever project came up.

I didn’t always have the full context of a project I was helping out, I was mostly glad to have a specific project I could help with.

The very first project I was the full owner of was a Java project which involved lots of signal acquisition and processing with the Android SDK.

These types of signals.👆

I got kind of into a rabbit hole in some of the technical aspects I had to help with which was very cool.

It allowed me to get a very practical grasp on how everything was analyzed, collected, or processed in some of the lab projects.

As I gained more knowledge about the technical, I started proposing ways into which we could get interesting result by modifying some analysis.

I also made sure to optimize the code I was building so that the throughput of the lab was better. Like for instance by making heavy use of parallelization later on (we had access to a supercluster for free that was awesome).

Anyway, at the end of it, I was able to publish quite a few papers which were mostly “assist” papers where I was a co-author. I was the main contributor to the general codebase of the lab and I became the go-to researcher for all the tricky technical problems.

A2: What I would do to contribute in retrospect?

Okay so that’s what I did, in retrospect I would do it a bit differently.

I would absorb the context the lab is operating in wayyyyyy earlier.

Looking back there was a lot of wasted technical effort in the process I followed. I learned a lot, but I could have been more impactful faster.

To get the context of the lab faster I would do the following:

• Take one of the papers the lab already published (like the best recent one) and then piece it down into the technical component.

• I would ask very targeted questions about why some stuff was done in a certain and how I could replicate the results.

• Once I have a good idea about that blueprint, I would have a very good idea about how the lab is structured to generate science-worthy results:

  • Who did they collaborate with?

  • Who made this code?

  • Why is it set up like this (sometimes it’s silly like “the intern only knew how to do it in R”)?

  • Why did they choose this analysis over another one?

  • What have they tried that failed and what changed?

All of that meta-information is absorbable in under a month and you will get up to speed with how the lab functions fast.

THEN with that context, I would help whoever is most collaborative in the lab with their experiment and focus on two areas:

• the mundane repetitive tasks that are annoying to do: in our case, it was cleaning up the EEG data we collected.

• the coding tasks to push analysis forward: given your background, the coding part is where you can bring some assistance. Since you already understand the setup you should be able to contribute quickly.

After that, I would check what I’m most interested in and double down on that area. This way, it will provide me with growth and allow the lab to benefit from my learning experience.

btw have fun during the process too, that’s the most important part.

A3: How I Became Interested in Machine Learning?

To be frank, I’m not directly interested in machine learning.

I’m interested in learning in general, as a meta-concept. I’ve started my bachelor's in cell biology and I was fascinated by neurons.

So fascinated that I interned in a neurobiology lab during my summer studying memory in Aplysia (it’s a sea slug).

beautiful!

I was doing my best to cut out a protein with multiple ubiquitin-binding domain to check how this would affect it. This was all part of a greater study about mice, genes, and memory.

That’s my protein! 👆

I was fascinated, and I went on to learn about memory in humans in a psychiatric lab. I had a project where I was trying to teach the memory palace technique to patients with schizophrenia.

Then I took lots and lots of computer science classes which included signal processing and machine learning.

I’ve also took online classes on the subject, but where I learned the most was by directly trying to apply the concept in the research project I was working on.

These projects directly benefited my research lab and I had great motivation to dig deeper into both the data that my lab was generating and how to use machine learning properly.

Don’t be afraid to self-learn machine learning, this topic is so vast and multidisciplinary that you will never find one resource that will show you everything you need to know.

It’s the kind of field that you learn in various way and that you try to apply ml to your problems in creative fashion.

Let me know if you want to voice-chat later this week, might be easier to answer your questions!

END OF THE MESSAGE.

Hope that was useful everyone 🌹

Have a great rest of the week!

Best

Yacine Mahdid

PS: feel free to shoot me questions by replying to this email, I love them.

In-Context Learning.

If you never heard the term this issue is for you.

Basically if you are using LLM like Gemma or GPT-4 chances are you already have used in-context learning.

Something like the following prompt would qualify as asking a model to do in-context learning:

Circulation revenue has increased by 5% in Finland. \n Neutral

Panostaja did not disclose the purchase price. \n Negative

Paying off the national debt will be extremely painful. \n Positive

The company anticipated its operating profit to improve. \n __________ ← to complete

In that particular case, the output for something like GPT-4 looks like this:

The sentiment for "The company anticipated its operating profit to improve" is Positive.

How the heck did the model understood that:

1. This was a sentiment analysis task?

2. That it needs to respond Positive or Negative?

We’ll explore that in this issue!

btw: for the visual folks out there, there is a video + deck version available on my Youtube channel.👇

Definition

In a nutshell, In-Context learning is loosely defined as showing a few examples of input and output to a large language model and just by conditioning on the prompt the model can learn how to do a new task.

In-context learning like many things in machine learning is a bit of a confusing terminology. There isn’t any learning happening in the traditional machine-learning sense (i.e. the weights of the models aren’t changing).

However, something is happening within the models that feel like it understands the tasks that are requested.

The term in-context learning originated from GPT-3 in the paper that introduced the model title “Language Models are Few-Shot Learners”.

Few-Shot in this context is giving a model a few examples to train on a new task

Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches.

[…]

For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model.

OpenAI

The sentence above is absolutely bonkers from a machine-learning standpoint.

The gist of it is that by showing one or a few examples of the input-output pairing of what the task is, the model can drastically improve its performance as seen below.

The interesting point on the graph is:

1. This ability emerge at a rate proportional to the size of the models (175B curve is steeper than 13B). Yet even in the 13B, there is in-context happening, except at a rate that requires more examples to be given in context.

2. There is a very sharp improvement in the larger 175B model when 1 example is given and the performance keep increasing as more natural language examples are showcased to the model.

To leverage in-context learning in practice the last output pairing needs to be left blank so that the LLM can fill it out as in the example above.

There are many ways to use in-context learning as it is versatile in a wide variety of tasks and can even be used to make external calls to APIs in the case of toolformers architecture.

Why is it Surprising?

Being able to learn on the fly with a few examples isn’t something new, meta-learning that allows for optimizing a model for few-shot learning already existed well before GPT-3.

The reason why in-context is leaving researchers quite puzzled is that it wasn’t part of the training procedure at all.

Remember, in-context learning originally emerged in big transformer models trained on next token prediction from internet scale data.

There is nothing in this training regiment that explicitly conditioned the models to develop that capability.

Yet, it happens consistently and allows for LLMs to be a massively versatile interface if conditioned properly with input and output (like in this GPT-4 example prompt):

There are two connected leading hypotheses of how this emerging ability is working, let’s explore them.

Hypothesis 1: Bayesian Inference Framework

The first prominent hypothesis for how in-context learning could work comes from Xie/Min et al, where the authors proposed a Bayesian framework explaining this emergence.

In a nutshell, the LLM uses the in-context learning prompt with input/output examples as a guide to locate the previously learned latent concept within its weights.

As you can see in the picture above, the LLM could be infering and retrieve the right concepts depending on the content of the prompt.

In this framework, the LLM is learning and inferring the concept of two things:

1. the pre-training concepts distribution while it was trained to do the next token prediction.

2. the prompt examples concepts distribution which usually share a common theme.

In this framework, the prompt examples are used to “sharpen the posterior distribution over concepts”. As we add more examples in the prompt we are helping the model select the right concept to sample within the “p(concept | prompt)” part of the formula.

Here the prompt information can be separated into two:

• Signal: which are the training examples and the information contained within.

• Noise: which are the transition between examples or even the correctness of the input-output pairing.

The basis here is that if the signal-to-noise ratio is high in-context learning can emerge. Interestingly, some noise can be added to an in-context learning setting that would make no sense for a supervised learning algorithm but would leave an LLM unbothered (e.g. like having all the output labels being randomized).

As we see here, by having enough signals the model learned:

1. the input and output pairing structure for the task.

2. the type of task: sentiment analysis.

Even though there is a tremendous amount of noise in the output (i.e. all the bottom examples are random!) it kind of doesn’t matter much for the model in such a task setting.

If we were to break down the content of an in-context learning prompt, it would look like this:

• Input distribution (very important)

• Input-Output mapping (not so important)

• The format used (important)

• Output space (very important)

This idea was explored concurrently from another angle by the smart folks at Anthropic as they found a circuit they call an induction head in transformer architectures.

This type of sequence gets its name from inductive reasoning because:

In inductive reasoning, we might infer that if A is followed by B earlier in the context, A is more likely to be followed by B again later in the same context.

Anthropic

Even in the toy transformer network they studied, the researchers at Anthropic quickly realized that the presence of such a circuit in the network meant the architecture was implementing an algorithm within.

Notice that induction heads are implementing a simple algorithm, and are not memorizing a fixed table of n-gram statistics. The rule [A][B] … [A] → [B] applies regardless of what A and B are.

Which bridges the gap to the second hypothesis.

Hypothesis 2: LLMs are Implementing Learning Algorithm Implicitly

Now the second hypothesis of how in-context emerge is that the transformer-based in-context learners are literally implementing learning algorithm implicitly somewhere within their weights structure.

It has been shown that transformers architecture even of small size can implement gradient descent in the paper “Transformers learn in-context by gradient descent”.

One of the most compelling arguments that the big LLMs are implementing a learning algorithm to help with in-context learning tasks comes from a paper by the Google Research team titled “Larger Language Model do In-Context Learning Differently“

It was shown by the team at Google Research that if models are large enough they gain the ability to override their prior pre-training distribution and learn a new erroneous input-output mapping.

What we see in the figure is that we are feeding increasingly more flipped label percentages for an in-context learning task to various models. As we increase the number of wrong examples what we witness is that the bigger models like PALM-540B are overriding their typical internal in-context behavior of not listening to input-output mapping.

Note that it still requires enough flipped labels for this “failure” at the task to appear in these large models since the hypothesis is that the models are quite literally using function mapping from input to output during an in-context task.

Something similar is happening with what is called “Semantically-unrelated targets in-context learning” which is a fancy way of saying “replace the output with random words like Apple or Oranges.

Small models struggle throughout, while larger models get better and learn the right mapping for the task.

Where Are We Now?

So, for both hypotheses highlighted in this newsletter, the mechanism seems to be a contextual activation of the right embeddings or knowledge domains within the model and this activation seems to be producing the right output.

The current research direction is pushing the boundaries towards that second hypothesis quite heavily with articles such as:

• Transformers Learn In-Context by Gradient Descent

• Trained Transformers Learn Linear Models In-Context

• Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression

• What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning

If this research direction of big language models yields positive results, there is a very good case to be made to just “train larger models and see what emerges”.What other meta-learning ability will appear as the number of parameters increases?

Conclusion

Anyway, this is a fascinating topic and I believe that a solid understanding of the mechanism of in-context learning will help us develop the next generation of AI models.

If you want to dig even deeper than this first introduction, check out these awesome resources:

• The Gradient: In-Context Learning in Context

• The Stanford AI Lab Blog: How does in-context learning work? A framework for understanding the differences between traditional supervised learning

Also, for a more detail-oriented explanation of in-context learning check out Sang Michael Xie's contribution.

If you have questions or additional thoughts, let’s continue this conversation in the comment section!

Hope it was useful! 🌹

Best,

Yacine Mahdid

How’s your summer going?

It’s been quite a long time since I haven’t published in this newsletter, but recently I’ve been receiving lots of DMs from machine learning practitioners.

I feel it will be much more useful for everyone if I summarize my practical tips in here.

One of the questions I recently received was about selecting the proper cross-validation scheme in a machine-learning experiment.

For those that don’t know, cross-validation is defined as:

Cross-validation is a model validation technique for assessing how the results of a statistical analysis will generalize to an independent data set.

It is used during the training of a machine learning model to figure out how the model will behave once deployed in real life (aka how it will generalize).

The two main issues I’m seeing with practitioners getting bogged down with cross-validation are:

1. They don’t do cross-validation at all.

2. They do not choose the right cross-validation scheme that fits their use case.

This is problematic because it leads to the dreaded overfitting issue where the model seems to be doing fine during training, but completely collapses in production.

In this issue, I’ll show you a trick to always select the right cross-validation scheme for your use-case!

btw: there is a video version over here if you are more of a visual learner.

What’s the trick?

There are a lot of cross-validation options to choose from:

Luckily for us, it’s fairly straightforward to figure out which fits best a use case.

The trick is as follows:

Think about how your model will be used and interact with data in a deployed setting. Then recreate this environment for the validation of the model.

It’s as simple as that.

There are usually three components interacting together when a model is in production:

• There is a data source that generates the input for the model.

• There is the trained model per se.

• There is the output of the model which should be solving a task.

To get satisfactory output during the production phase, the training of the model and the way the input data is generated need to fit together.

Let’s check an example to illustrate:

Assuming we want to detect the presence of a disease in samples from an unseen patient. How would the model interact with the data in production?

The important part of the interaction above is that the model never saw the patient during its training.

This piece of information is crucially important because it means that during the training of the model, we cannot have data from a single patient in both the training and validation split.

This effectively pinpoints exactly which of the different kinds of cross-validation techniques in the tree of possible schemes to use we should be using.

Let’s look at that tree!

Decision Tree of Cross-Validation Techniques

At a glance, the selection process looks like this:

We’ll walk through each of the branches in a minute, but one thing to note is that each of the leaf nodes contains variants that you can use to ensure the cross-validation score match the generalization.

Lots of Data? - True

If you have a lot of data, your cross-validation usually is very simple.

It will consist most of the time of doing a simple hold-out cross-validation, where about 80% of your data is used for training and 20% is used for testing.

The reason for that is that both the training and test sets are so large that they are equally representative of the data in general.

Let’s use our production trick to illustrate this:

If you were training a large language model to do autocomplete based on user generated prompt, the setup would look like this:

The data generator is an independent human that wrote a bit of text with a missing word. The model is GPT and the output is the same prompt, but with the most likely word added at the end.

In that particular scenario, if you train with enough data like common crawl billions of internet pages or Reddit discussions, you get enough data generators (i.e. independent humans) to have a similar distribution however you cut the data during testing.

Therefore, when your model is faced with the showcased input in production, it isn’t that much different from how it was trained.

Disclaimer: This might not hold all the time, for instance when you have very very strong dependency between data points or when the distribution is completely skewed during testing (i.e. someone prompt in a language the model never saw before).
But in most cases it generally does.

Lots of Data? - False

Now, when you don’t have lots of data it means that you have to be very careful in how you do the cutting and validation during training.

The first question to ask yourself is if your observations are independent of each other or not.

Assuming that some data is Independent and Identically Distributed (i.i.d.) is making the assumption that all samples stem from the same generative process and that the generative process is assumed to have no memory of past generated samples.

This assumption is common in machine learning but it very rarely holds 100%. What you want to check here is if the independent and identically distributed assumption kinda hold.

Observation Independent? - True

So let’s say it does hold.

Here it means that whatever is generating the data is generating each of them independently without a strong correlation between data points.

Let’s consider the following example:

Given a newly created product on a store, predict whether its first review will be positive or negative.

In this case, the data generator is the user base which are creating a single review for a new product. The new product meta information is the input to the system. The model was trained with these reviews in some ways and the label is binary.

In that particular case, if we assume that no one group of users is dominating the review process for the first review of a new product, we can say it’s IID.

This would have been perfect for a hold-out split if we had Amazon level of data, but if we don’t we need to use Kfold to generate a proper estimate of the model generalization performance.

KFold is super simple.

Chunk your data into equally sized blocks, train on one bunch of blocks, and test on the other. Then you alternate which block is the testing block and which ones are the training blocks.

Usually, 5 to 10 folds are good enough to have a good estimate of the performance of the model.

There are other variation of KFold like stratified and shufflesplit, depending if your classes are severely imbalanced or if you don’t have much data.

Observation Independent? - False

But, let's say that the IID assumption doesn’t hold at all.

The first thing to figure out is if the observations have time dependencies.

Observation Time Dependent? - True

Time-dependent data points are fairly straightforward.

You can think of them as being generated by a process that over time will generate points sequentially and with full knowledge of the previous data points.

A good example of time-dependent data generation is with stock prices.

One of the most important pieces of information for stock prices is knowledge about historical stock performance.

So in production, if you have the following task:

Given previous information about the stock market, predict the next stock price of a given stock.

You have to take into consideration the time aspect whenever cross-validating. Namely, you cannot train on future stock prices (i.e. t = 3,4,5) and past stock prices (t = 0, 1) to predict a stock price (t = 2).

This will not be possible in production to see the future, therefore when cross-validating you need to “take a walk through your data”.

Which is what a TimeSeriesSplit does.

The first fold will have the least data to train on and you will progressively gather more training data. You will always test on “future” data in this particular case.

Observation Group Dependent? - True

Now, your data points might not be dependent on the time domain.

Maybe it’s something else that make them correlated, like in our earlier example with disease diagnosis.

Consider the following situation:

Given labelled EEG data about patients' emotional states predict what an unseen patient's emotional state will be.

If you didn’t take enough time to explore your data, you might not realize that the data generators (i.e. the brain of the participant) are creating very correlated data points.

The process kind of looks like the above.

If you were to run a clustering algorithm on the data point, you will usually have a very easy time to find back which data points belong to which participant.

Therefore, if we use our production trick we will have something like this:

The data generator is a new participant brain that is generating EEG signals. The model was trained on previous participants and had never seen this participant before. The output labels are the emotional states of this new participant across time.

The perfect methodology for this is GroupFold

It’s the same idea as KFold except that the groups always stay together in either the training or the testing set. In our example, a group is a participant generating multiple data points. Meaning that during testing you can be 100% sure the model never saw that group before.

Conclusion

All in all, don’t forget the simple trick to think hard about how your model will be used in production. Because at the end of the day, it’s kind of the only thing that matters.

If you want to explore the subject a bit more I highly recommend the following resources:

• A Gentle Introduction to K-fold Cross-Validation (Machine Learning Mastery)

• 3.1. Cross-validation: evaluating estimator performance (sklearn docs)

I hope this was useful, if you have any questions don’t hesitate to shoot me a DM.

Have a great rest of the week everyone! 🌹

Best,
Yacine Mahdid

Subscribe now