Share from 0:00
0:00
/

How to Get Into AI Research in 2026?

with Andrew Drozdov Research Scientist at Databricks

should you do a PhD? should you leave your software engineering job for grad school? how do you get into AI research coming from industry?

I get these questions all the time, so I sat down with Andrew Drozdov, senior research scientist and one of the retrieval leads at Databricks, who went through every single one of these steps.

andrew wrapped up his undergrad at Michigan a semester early without even planning it, then filled a whole semester with grad-level courses like J. Alex Halderman's security class (two research papers per lecture, reviews graded by your peers, and live demos like freezing RAM with an upside-down aerosol can to read the password straight out of memory!). after that he worked at Okta and Datadog, did Andrew Ng's lectures on the side, and made the tough call to leave a job he liked for a master's at NYU right as attention was taking off, then a PhD at UMass.

we talk about how to know if grad school is actually right for you, why "basically no one should do a PhD" (unless you are really passionate), why you should tie your success to more than papers when ICLR is getting 60,000 submissions, his front row seat at NYU's CILVR lab (attention in RNNs, parallel neural architectures, latent syntax), and how to pick work that lasts by anchoring yourself on problems rather than methods.

I really enjoyed this interview, a great starting point if you are thinking about grad school or a research career.

part 2 on KARL and AutoIndex is coming soon!

Also, check out over here a few useful links from andrew:
👉 his website: https://mrdrozdov.github.io/
👉 his advice for PhD students: https://mrdrozdov.github.io/blog/2024/advice/
👉 check his twitter too: https://x.com/mrdrozdov
👉 his PhD thesis on adaptive retrieval-based methods!!!! : https://scholarworks.umass.edu/entities/publication/04520d37-6227-4441-9ec8-e91495271ce8
👉 KARL (for part 2): https://arxiv.org/abs/2603.05218
👉 AutoIndex (for part 2): https://arxiv.org/abs/2607.18603

btw also check out more of Deep-ML I love you moe man keep doing what you are doing:
📌 premium: https://www.deep-ml.com/premium
📌 the Anthropic track to get started: https://www.deep-ml.com/interview-prep/anthropic


Table of Content:
- 00:00:00 - should you do a PhD? (and all the other questions I keep getting)
- 00:03:34 - Andrew Drozdov!!!!
- 00:06:40 - why take grad-level classes during your undergrad?
- 00:10:18 - Halderman's security class (two papers per lecture + peer graded reviews)
- 00:14:39 - a cool demo about cold booting demo🪄
- 00:16:22 - is this where your research interest was seeded?
- 00:18:32 - what advice would you give your undergrad self getting into AI research?
- 00:21:50 - just get in! (help out in a lab even with zero qualification)
- 00:23:10 - how did working at Okta and Datadog help you afterward?
- 00:27:17 - what made you ready to leave a job for grad school?
- 00:29:50 - basically no one should do a PhD, unless...
- 00:30:27 - ICLR at 20,000 plus submissions: tie your success to more than papers
- 00:32:53 - you don't have to finish it (dropping out to go to Big Co is ok)
- 00:34:23 - how was it to study during the 2015-2024 big boom?
- 00:35:10 - front row seat at NYU's CILVR lab (attention in RNNs, QRNN, latent syntax)
- 00:39:38 - is attention really all we need?
- 00:40:33 - how do you know what you work on will still be useful in the future?
- 00:47:18 - conclusion: nobody is going to hand you the path into research man seize it

enjoy and do follow andrew on twitter, and read his PhD advice post if you are thinking about grad school!

Andrew Drozdov — transcript

00:00:00 – Introduction

Yacine Mahdid

Should you do a PhD? What should you major in? Should you take a break right now and go to industry? How do you get into research from a software engineering background? I get variations of all of these questions all the time, so I figured I’d bring in someone who is uniquely positioned to answer them better than I can: Andrew Drozdov, research scientist at Databricks.

A crowd asking career questions
The career questions that keep coming up: PhD or not, what to major in, industry first, and how to get into research from software engineering.

He’s one of the retrieval leads on their research team, and he recently co-authored KARL, a reinforcement learning paper on knowledge agents, among other things. That paper is how I first came across his very elegant work. By the way, we’re going to have a part two with him on this topic sometime this month, if I manage to get my stuff together to edit it.

His main focus is finding ways to give models the context they need so that they can actually answer the questions people care about, especially in enterprise settings.

Give the model the *context* it needs
Andrew’s research focus: getting the model the context it needs to answer real questions.

He studied computer science at the University of Michigan, and he was cranking through requirements so fast that he finished a semester early, which is so funny. Instead of just coasting, he took graduate courses that actually sparked his research interest.

After that, he went into industry at Okta and Datadog as an early employee, and he started working through Andrew Ng’s machine learning lectures on the side. Eventually he made the tough call of leaving a job he liked to go after AI, and this was way before the LLM takeoff. He did his master’s at NYU right as attention and the early transformer wave were starting to form. Then he did his PhD at UMass.

Undergrad, industry, then *AI*
Andrew’s path: undergrad at Michigan, industry at Okta and Datadog, then AI with a master’s at NYU and a PhD at UMass.

During this interview, I asked him a whole bunch of questions fueled by the kind I’ve been receiving. It’s pretty all-encompassing, with the grad school stuff, the industry stuff and the research stuff all blended together. Hope you enjoy.

00:03:34 – Pulling in context the model never saw

Andrew Drozdov

All right. So, Yacine, thanks for having me on. You reached out to me, I feel like, more than six months ago, shortly after the KARL paper came out, so I’m glad we’re getting this together.

I’m Andrew Drozdov. I’m a research scientist at Databricks, I work on retrieval research, and I’m one of the retrieval leads on the research team. I like to work on problems where we pull in important context that was not necessarily in the model’s training data, so that it can actually get the answer to your questions correct.

Basically, there are a lot of questions that people care about, that are really high value, that are based on data from their daily interactions, or from business information, or from real news events that are coming out all the time. And there’s just no way the model can answer those questions without pulling in that information.

I think by now it’s become kind of standard to do these types of things, but when I was doing my PhD, it was not as clear. So I’m glad people realize this is important, and I really enjoy working on this problem. I did my PhD on core NLP, on retrieval and on language models, and I’m basically still working on all those topics day to day.

Yacine Mahdid

Yeah, and it’s still very important in various ways, not just for the big LLMs. I have a project right now with one company I’m working with, and we’re almost done writing the paper, but we had to finetune the cross-encoder to help do the retrieval. All the techniques behind it are extremely important, especially when you actually have a whole bunch of data, you’re live in production, and you have latency and all these concerns. I’d say it became even more important than before.

Andrew Drozdov

It’s a big space. I’m happy to talk about all the little corners.

00:05:52 – The grad class that blew his mind

Yacine Mahdid

But I think it would be cool if we could wind it back to your undergrad. The whole arc of your career has all the facets people usually ask about: industry, academia, what to research, whether you should do a master’s. All of it is there.

I want to take it back first to the undergrad portion, because you did something I also did when I was doing my undergrad: you made it your own and took harder classes, grad-level classes, like computer and network security, right? So first off, why did you take these classes during your undergrad? And what exactly did they teach you that the normal stuff usually didn’t?

Andrew Drozdov

To this day, I think it was the best class I’ve taken. Professor J. Alex Halderman taught this graduate-level security course at the University of Michigan, where I did my undergrad, and it was so interesting. I think about it all the time. It’s also what got me excited about research. It was many years later that I started working in grad school more seriously, publishing papers and working in AI, but if you track my history, it was really that course that made the biggest difference.

To further set the stage: I studied computer science at Michigan. I didn’t totally know what I wanted to do, so it took me a little bit of time to settle on computer science. The side effect of that is I spent my middle years, sophomore and junior year, just making sure I could crank out all of the requirements.

It was honestly a bit of an overwhelming experience. I went to a pretty small high school, like 100 people per class. I think the computer science department at the University of Michigan is something like 2,500 students. And that’s just computer science, not even the whole university.

You would be taking calculus, or intro to programming, or an intro to algorithms course, and you’d be sitting in a lecture with a few hundred other people. And yours was just one of the blocks; there were probably two or three other sections of that course the same day. So I was just cranking away in this pretty rigid, exam-focused environment, where I learned a lot about the fundamentals of computer science. But I didn’t spend a lot of time understanding how computer science impacts society, or what I actually wanted to pursue in the field.

Another side effect of jamming in as many courses as I possibly could is that I inadvertently completed my requirements one semester early. I didn’t even plan for it to happen. I don’t remember exactly when, but somewhere in my first semester of senior year, I realized I was done with credits. I was planning on just graduating and hanging out on campus for one semester, finally getting a break.

I think my parents finally convinced me I should get a job. So I got a job at Menlo Innovations, a very well-respected software development firm in Ann Arbor. That way I could still be on campus and get some work in. But I was one week into that job when I had this epiphany, I think just from talking with classmates, that I could still take more courses if I wanted to.

It hadn’t even occurred to me, and maybe this happened to you too, that you could just take more classes. I thought you take all the requirements you need to graduate and then you’re done. Why would someone take more? Within 24 hours of that realization, I had signed up for basically another full semester of graduate-level courses, because I had hit enough requirements that I could actually take them.

One of those was this class taught by Professor J. Alex Halderman, and it totally blew my mind. I was so not ready. In algorithms and computer theory, you’re basically reading from the textbook, or maybe some notes the professor has compiled, you’re doing psets, and there’s a final exam. If there’s discussion or office hours, it’s just to understand why you can reduce this NP-complete problem to that other one, or why this algorithm was designed this way.

Reducing independent set to clique
The kind of exercise undergrad theory classes revolved around: reducing independent set to clique.

This class, just to set the stage a little bit, was all research-paper driven. Every lecture, you would have two research papers assigned, and you were expected to have read them before the lecture. The way they actually measured whether you’d read them is that you would submit paper reviews.

They had this bespoke software platform built just for this class by the TA. I looked it up: I think it’s Zakir Durumeric. Zakir was fantastic. I think he’s now a professor in security at Stanford, but at the time he was a PhD student. We would submit our reviews on this platform, and as soon as you submitted yours, you got assigned reviews from your peers, and you would grade your peers’ reviews.

Zakir Durumeric
Zakir Durumeric, the TA who built the class’s peer-review platform, now a security professor at Stanford. (source)

Yacine Mahdid

Oh, wow.

Andrew Drozdov

This was just so strange to me. I had not read any research papers up until that semester. I didn’t really know what a research paper even was,

To be honest, I just thought textbooks were the end of the knowledge I was looking into. Eventually I had a realization: oh yeah, these textbooks are all derived from research papers.

So I was doing this research paper reading, and it was really fun. We would come in, talk about the paper, critique it, talk about what the paper means. Because it was about security, a lot of the papers were about building complex systems to be more secure, to defend an existing system or to attack a system. Some of them were more mathy, about cryptography.

On top of these discussions, we would sometimes have guest speakers, and the guest speakers took it to the next level. The one I remember the most: there was this paper at the time called Mining Your Ps and Qs. They basically saw that a lot of routers had a manufacturing issue. Pseudo-random number generators still need some initialization, so when you manufacture routers with built-in encryption, the key is to make sure the random number generator is initialized differently on every router. But they found a defect where the majority of the routers actually had the same initialization.

So you could do a programmatic sweep and see how long some SSH connection or some website connection had been live. Based on how old it was, you could really narrow down what the pseudo-random number generator was initialized to. Then you could run a decryption attack on their keys, and take something that would take a million years, something that would take forever, and do it in a few minutes or even milliseconds.

Small correction by Andrew: The vulnerability was caused by insufficient randomness during cryptographic key generation on embedded devices, resulting in some RSA keys sharing prime factors. The explanation involving connection age was inaccurate.

People would make Chuck Norris jokes about the lead author on that, because she could decrypt your RSA key by hand. She didn’t even need software. If she had enough of the metadata, she could just do this demo.

Yacine Mahdid

She could derive it.

Andrew Drozdov

Yeah, yeah. Sorry, it’s a really long description, but just to wrap up a couple of other things from that class, because I had such a nice time there. One other demo that I thought was really cool: the professor would enter his credentials into his laptop, then immediately turn it off, pull out the RAM and use an aerosol can that he would invert.

If you invert an aerosol can, it sprays out much colder. By making his RAM really cold, he could slow down the physics of the memory being wiped, and if he was fast enough, he could plug this RAM into another laptop he had and read from it. The way your password was being saved at the time, it was somewhere in memory, and he could just read the memory. Things I’d never thought about were being demoed live in class.

The professor's cold boot demo
The cold boot demo: the RAM is chilled with an inverted aerosol can, moved to a second laptop, and the password read from its memory.

Yacine Mahdid

Oh, this is so cool, man.

Andrew Drozdov

It basically expanded my world. It expanded how I thought about computer science. Plus it had this research paper reading aspect, and it was a project-based course in the end. So we had to take all the lessons we were learning throughout the semester and try to actually build something of almost paper quality, something you’d be proud to publish. You would meet with the professor and talk with the TA, so it was kind of my first research advisor experience, working with him that semester.

Yacine Mahdid

Do you think this is where your research interest was seeded, so that you could do research later on?

Andrew Drozdov

Yeah, yeah. That was my first exposure to the research world: understanding that these textbooks are all derived from papers, and that if you spend enough effort, you can understand and decompose these papers and critique them, the actual reviewing of the papers. In the beginning I was probably more in awe, but as the semester progressed, I got progressively more critical. By the end, I felt like, oh yeah, maybe if I put enough energy into this, I could write some of these papers myself.

00:17:09 – Advice for undergrads eyeing AI research

Yacine Mahdid

This is so cool, because I did a similar thing. I think I graduated with 12 or 16 credits too many. I was taking too many courses; even in the summer I was taking classes. And I realized two things. You can take as many as you want; they don’t check. And I took some graduate-level courses without the prerequisites, and they didn’t check any of that either. So you could just take the class and get a good grade.

When I was graduating, a lady found me and said, “Oh, but you don’t have the prerequisite for this class.” And I was like, “Yeah, I got an A in it. So what are you going to do about it? It’s fine, we can move on.”

What I realized is that these graduate-level courses were actually what I thought university would be like when I was in high school. In the undergrad courses, you’re in a gigantic classroom with 500 other people, barely able to talk to the teacher, everything is on a timer, and you could basically have watched it online and it would have been the same thing.

If you think about yourself back in that undergrad era, what advice would you give your younger self, if your undergrad self wanted to do the stuff you now see happening on the AI research side?

Andrew Drozdov

Yeah, it’s really hard. I’m pretty happy with how my trajectory panned out, but at the same time, I think a lot about the advice I want to give to the next generation of researchers, or to colleagues who are maybe deciding to go down the research path if they haven’t already done graduate school. I also have a daughter now, so I’m thinking about her future. I want her path to be smoother than mine, for sure.

I think part of it is choices I made, but also very much environment, and maybe it’s gotten better over time. When I was going through undergrad, I had minimal exposure. I probably knew a few people who were in graduate school. I was reading a little bit of the news, so I would sometimes read about new innovations that were coming out, but I hadn’t connected the dots that it was something I could do.

Nowadays there’s all the advancement in open source, and AI discussion is much more mainstream. People were already doing a lot of AI when I was an undergrad. Some of the most ubiquitously used technologies, like search, have been using pretty advanced AI models for a very long time. But I don’t think it was very accessible; you pretty much had to be an expert in that area to even have a sense of what they were using. Now everyone knows what machine learning is, and you can prototype things really quickly in your browser with one of the API-based models.

So to make this more concretely an advice: if you are early in your career, just try to talk with people, read a lot about what’s happening, and feel empowered. A lot of the people working on these technologies were doing their undergrad just a year or two ago, or they just completed their PhD. If you measure things in discrete time, it’s just a few months, or a few dozen months, of focused study to get to that stage. Of course, it helps a lot if you’re already building on a foundation of fundamental education across reading, writing and mathematics. But if you just get excited about a topic, you can decide: I’m going to pursue that now.

Yacine Mahdid

Yeah, I agree. In my view, the action part is pretty important. If you’re interested in it, just try to get in. If you’re in undergrad, you could technically be helping out in whatever lab is available, and they’re pretty open to it. I personally did that with a bunch of neuroscience labs I was interested in, and I had no qualifications whatsoever. You demonstrate that you have agency and that you can learn things. Most researchers are very open to teaching you some stuff, and then they give you a chunk of the work to do. And right now it’s easier than ever to get at least some understanding of things and show that you can be of minimal use, at least.

Andrew Drozdov

Yeah, I think everyone should try to do that, basically. And not only people who want to pursue something a little different from what they’ve already been working on. Especially in this field, everything in AI changes so quickly that you kind of need this self-starter DNA just to keep up.

00:23:09 – From single sign-on to computer use

Yacine Mahdid

True. So you did your undergrad, and then you worked at Okta and Datadog as a software engineer before you decided to do research. Can you explain that part to us a bit, and how it helped you or not afterward?

Andrew Drozdov

Yeah. My first job out of undergrad was at a startup at the time called Okta. I was something like employee 200-something. At that time I wasn’t really doing AI. I was on a small team focused on improving the robustness of the single sign-on capabilities.

Okta logo in frame
Okta, the single sign-on startup where Andrew took his first job out of undergrad.

One of the big projects I worked on ended up edging closer and closer to AI. We had a browser extension that needed to figure out how to correctly identify your username and password fields, and the related buttons, for every single website. We initially just treated it as regression testing with heuristics, and building that system was really cool. It’s weird, everything feels like a time loop to me, because it was basically large-scale computer use.

And now that seems to be a really trendy topic again. So I would have hundreds of Selenium browsers, a bunch of headless Chromes, on my laptop, all clicking things in parallel and logging the results. I worked on that for a little bit, but at the time AI wasn’t really a big focus for the company.

Simultaneously, I had started doing the Andrew Ng lectures, and it was just too much to do both. So I made the tough call to try to get into graduate school. I had really decided that AI was something I wanted to pursue more, and it was going to take me a long time to do that at my current job.

So I made that call and applied for grad school, and while I was waiting to get in, I worked at this other company called Datadog. That was a little bit closer to AI: I was doing more large-scale data engineering. Datadog is a monitoring company, so they ingest huge amounts of data and do large-scale computation on top of it. They really need an efficient, scalable MapReduce system, and that was the team I worked on. We were building the in-house version of that.

Datadog logo in frame
Datadog, the monitoring company where Andrew worked on large-scale data engineering while waiting to start grad school.

They were doing some early time series analysis that used AI, but I had already decided I wanted to do grad school at that point, so I left Datadog after a short amount of time.

Both companies were really cool for different reasons. A lot of people know Okta now, because if you work at a company, it’s very popular. Okta was really cool because it was right in the heart of Silicon Valley. At one point we had this force push problem: someone had force pushed to some branch and we couldn’t figure out how to fix it. We were familiar with Git, but not pros. GitHub was a few blocks away, so we just walked over to GitHub and asked them for advice.

Yacine Mahdid

Oh wow, this is so cool, man.

Andrew Drozdov

That was really fun. Datadog was a super hardcore team. You’d walk into the office and it was dead silence; everyone was totally focused. I would sometimes try to chat with my mentor, but he would often direct me to Slack, because he preferred for all of our conversations to be indexed so we could really easily refer to them later.

00:27:06 – No one should do a PhD, unless

Yacine Mahdid

Oh, this is good, man. The very interesting part is that at some point you were like, yes, I want to go into grad school. But what was the element that made you ready to commit to going there and literally leave a job? I have a whole bunch of people who have a software engineering job, but they want to learn more of this stuff, and their job is not about that. So they try to learn internally, they talk to the right people, but it’s kind of hard for them.

They have this yearning for doing more research, so they’re thinking about academia, but then they wonder how they know it’s actually going to be good for them. Any advice here, or the element that triggered it for you?

Andrew Drozdov

Absolutely. By the way, it was a really hard decision, because I had really liked my job in both places, and I saw myself thriving in those environments and getting better. But there was just this one thing about AI that I was really excited about. And it wasn’t even a career move in a sense. I wasn’t doing it because I thought it was a lucrative way to move up the ranks or improve my TC. I was just getting really excited and passionate about this space.

So I think graduate school is a big investment of time, energy and opportunity cost, and you should probably think about it pretty seriously. I think there are a lot of really good, viable alternatives, especially these days. What you learn in graduate school has gotten more and more accessible.

And especially now, all the companies are really excited about AI. So if you’re excited about AI, you might even have a better chance just convincing your manager to get you some GPUs and start experimenting in that company. You could become the AI lead in your company if you’re one of the first people getting into it. It’s really hard to say that doing 5 or 6 years or more in graduate school is competitive with becoming the AI lead at a company where you’ve already established yourself.

So I don’t think there’s an easy threshold where you know it’s time to move. You’ve got to think about it really deeply, and it’s a personal decision. To reiterate something my friend likes to say, which was the advice he gave me when I was looking into graduate schools: basically, no one should do a PhD. But if you are really passionate about a topic and you’re willing to put in the time and eat the opportunity cost, then maybe you should do a PhD. Even at that stage, it’s still a little bit unclear and personalized.

00:30:13 – Success beyond paper counts

Andrew Drozdov

The flip side of that is that graduate school is really hard for a lot of reasons, and I think it’s gotten progressively harder in some areas in the last few years. There’s this story about ICLR getting 60,000 paper submissions. It’s totally wild. And it does seem like a lot of paper submissions are tied to people’s progression through graduate school, so I could imagine that being a really stressful environment for the current round of graduate students.

Small clarification by Andrew: The figure of 60,000+ refers to preliminary abstract submissions for ICLR 2027, not finalized paper submissions.

Yacine Mahdid

Yeah, it’s crazy.

Andrew Drozdov

For people who are already getting started or already in the program, I would say try to tie your success to things outside of paper publishing as well, like publishing open source libraries where people can directly use the things you’re building. Working at Databricks especially has instilled that more into my thinking: if you’re going to invest all this effort to deeply understand a topic and build something useful that you yourself can use, you might as well find a way to let other people use it too.

Yacine Mahdid

Yeah. And to come back to the point about companies: almost all of them are entering this age of research, and they’re very happy to do AI and have somebody in-house doing that stuff. I’ve seen it even in literal manufacturing companies. I’m not even talking about, let’s say, Airbus, which already has an R&D department. Literally 200 or 300 people, manufacturing companies that make sub-sub-sub-components of airplanes, have research folks focused on implementing AI in some of their parts.

I think the only thing you don’t get when you’re doing this is the same kind of networking opportunities, the closeness to people sharing one or a few problems that are very closely related to yours. You get people in a similar context from the industry, but if you’re the AI lead, it’s going to be a bit harder to get that stuff by osmosis or discussion. So I think grad school is still worth it, at least for these links you can create with people who are as excited as you about a similar problem.

And to be very honest, you don’t have to finish it. It’s not a hard requirement, and it’s not a failure if you don’t finish. I know two people who actually dropped out of their undergrad to go work at OpenAI, and both of them are on the research side. Same thing with a master’s and a PhD: as long as you get what you had to get from the experience, there shouldn’t be a downside. If it’s becoming toxic and there’s nothing you’re learning, you can just drop out and come back to industry, and you should be all right.

Andrew Drozdov

Yeah, those are the two things to disentangle. If you’re already in the headspace where you know what you want to do with your life, you’re ready to start a business or you know what job you want, then preparing directly for that is probably going to increase your chances of making it happen more than the average experience in university.

The other side is that universities, graduate programs and conferences all need to keep iterating, and to understand that it’s hard and that we can do better. The nice thing is that there are a lot of great people in these programs, and sometimes really amazing ideas and changes. I’ve had a lot of positive experiences throughout my career of study, and I would love for more and more programs to adapt as well.

00:34:23 – A front-row seat at NYU

Yacine Mahdid

You did your master’s and PhD between 2015 and 2024. I don’t want to say it was the golden age of AI, but this is where there was this crazy acceleration. Before the transformer there was a series of big leaps, but as soon as the transformer was there, it just kept going, and now it’s even faster and faster. The weight of the gigantic discoveries has shifted toward these industrial labs, but there was still this rapid progression. How was it to literally be doing your studies during that time?

Andrew Drozdov

Man, it was really cool to have one of the front row seats during this period. I did my master’s at NYU, in the CILVR Lab. NYU has a bunch of different AI professors; a lot of them are affiliated with the Center for Data Science, but also with the Courant Institute. Yann LeCun was one of the well-known names in that group. I also worked a lot with Kyunghyun Cho and Sam Bowman.

CILVR Lab at NYU
The CILVR Lab at NYU, where Andrew did his master’s alongside Yann LeCun, Kyunghyun Cho and Sam Bowman. (source)

I could see everything happening in slow motion. When I was just starting, attention in recurrent neural networks was a very hot topic. There was this New York Times article about how Google was switching its translation system to a large recurrent neural net that uses attention.

I think Kyunghyun Cho was one of the key people credited with inventing that mechanism. The other well-known name is Ilya Sutskever, who was one of the early authors on attention. I think there were three attention papers at the time: Ilya, Kyunghyun, and, oh man, I’m going to be kicking myself later for not remembering the third paper.

Neural Machine Translation by Jointly Learning to Align and Translate
Figure from Neural Machine Translation by Jointly Learning to Align and Translate, the attention paper co-authored by Kyunghyun Cho: look at how the decoder attends over the source words. (source)

But that was the first ingredient, where people realized we could really push these things further with this one architectural change. And then there were so many intermediate things that kind of worked. I think there was one paper that explored 100 different variants of the LSTM.

...was both GRUs and LSTMs, with slightly different properties. And eventually people were moving more towards these parallel architectures. There were these convolutional neural nets for sequence models that seemed to be a little bit faster and still worked really well. There were these hybrid architectures; I think the QRNN from Salesforce was one of these, all around that time. A lot of them came out all at once, and we were constantly reading and talking about these papers in reading groups.

I was working on my own version, something with Sam Bowman, where we tried to use RL to decide what you’re going to attend to, and to have this syntactic bias. Sam has a really deep background in linguistics, and one of the key ideas is that people understand language in a compositional way, but machines don’t really do that. So if we could somehow instill that in machines, they would be able to learn from data much more efficiently and handle much longer sequences more efficiently.

Of course, the question was also whether we could get machines to do this compositional thing, and it was surprisingly hard. People were claiming, “Oh yeah, my RNN with attention has this syntactic latent structure.” But we looked really closely: we extracted as many syntactic structures as we could, looked really closely at them, and they really weren’t that interesting.

There was even a whole challenge task at the time, which segues into some of the stuff I was working on. There was this task about doing unsupervised syntactic parsing, and it was this kind of detached leaderboard, because it was a really interesting problem, but no one had yet tied it to anything economically meaningful, not that I know of. As a graduate student, when I was doing my master’s, I hadn’t quite completed the circle of how the work I was doing was going to power future AI systems. I had inklings. I was just pumped to be climbing this leaderboard.

So maybe another piece of advice is: don’t just climb the leaderboard. Really think deeply about what you’re working on and how it ties into things that are going to make direct impact.

Yacine Mahdid

Or if you’re climbing a leaderboard, just make sure it’s the right one and focus on that. Then just climb it and you should be good to go.

00:39:38 – Anchor to problems, not methods

Yacine Mahdid

Another element: in retrospect we’re like, “Oh wow, the transformer, of course.” But when you were in it, it was not like that at all. I remember I was at a conference at Mila, attending this big talk with Yoshua Bengio, and the big question was: is attention really the only thing we need, or is there something else missing? I was half listening, because I was doing my neuro stuff, so it was not related, but I just wanted to attend. And I remember vividly how messy and unsure people were at the time. We didn’t really know if this was the thing or if we needed something else.

So how do you figure out that what you’re working on will be at least marginally useful in the future? Do you have any tips on that from your experience?

Andrew Drozdov

I was very method-driven at the time. I was very excited about building these compositionally biased models. It was really fun to build them because they were always a little off the beaten path of the most efficient models. We always had to do a bunch of bespoke things, which was fun hands-on experience for me. Over time, I’ve drifted away from tying myself too much to individual methods and think more deeply about the downstream tasks and the problems.

At that time, I just had this idea of unsupervised syntactic parsing, or syntactic parsing, and I could just get lost in that single task. But it really tends to be more of an intermediate task towards things like translation or search or text generation. The advice I have for people in their graduate programs, about making sure what you’re working on has staying power, is to think deeply about the problems the method you’re working on is trying to tackle. I also think it’s good to diversify across problems, and to tackle big ones and invent new ones.

I remember one EMNLP I attended where Kristina Toutanova was giving a keynote. She was one of my heroes, because she had a lot of really impressive parsing and structured prediction papers; I was constantly reading her work. It was shortly after BERT had come out, so this was already later in my PhD. Her message to the audience was: just be more ambitious. We have these more powerful transformers that have been trained on massive amounts of data, and we can still use them for things like part-of-speech tagging and sentence-to-sentence translation. But there are many, many categories of tasks people haven’t even really tried to use AI for, because they had never had technology in their hands that was powerful enough to try those things.

Kristina Toutanova
Kristina Toutanova, whose parsing and structured prediction papers Andrew read constantly. (source)

I think that is really fun. Of course, it’s really hard, though. I don’t know that there’s an easy way to do these things. You just have to read and talk to people at the end of the day, and get some hands-on experience. What do you think about that? Are you more of a problems or a methods person these days?

Yacine Mahdid

I think the problem dictates the method, and the methods are almost irrelevant. We can see it especially right now: okay, there’s this new method, this variant of this thing being developed, and then we get to this point. But it’s not about the thing. It’s about something a bit more meta that it represents. The method per se is not that important. So even if you’re working on the method and it takes you a bit of time to get it right, and now you get it right but it’s not the most beautiful thing, it doesn’t matter if you’re actually hitting an important problem in an interesting way. I think that’s what matters: being less attached to the method, more about what the method means with respect to the problem, and thinking back about its interaction with the problem.

I really like how Ilya looks at it, because it’s more vibe: the overall vibe, the implication of what you’re doing right now with respect to the problem, what it should and should not be. It’s almost your gut feeling about the shape of it. It’s very, very high in latent space. But if you’re thinking at this level, then you go down on the ground and do whatever groundwork you need to cobble the stuff together and make it work. The capacity to have these two views is important, and I’d just say anchor toward the problem.

Because when you get to industry, or you’re actually trying to do something useful, the way you’re doing it almost doesn’t matter, whether you’re doing it this way or this other way. What matters more is: what can you do, and is this pushing what we were supposed to push? That’s what I’ve also learned. I’ve worked on many different things, and that came back again and again. Every time I tried to get anchored to the problem, whatever I was doing at the method level, at the very micro level, was never wasted. Maybe I’ll never reuse it in any shape or form afterward, but the components of it, and my understanding of how to try stuff and connect it to the problem, that stays, and you can reapply it everywhere.

Andrew Drozdov

Really well said. The way you’re describing it, I feel like everything actually ends up being a little bit connected. You need to have deep familiarity with the methods so you can actually pursue the interesting problems.

Yacine Mahdid

Exactly. As Hal said, there’s no just vibing yourself with the problem and then being done with it. You have to do both. You have to look at the problem, but then go down and actually understand what’s going on and how these things work. It’s these two views. You cannot just do the methods and not understand what you’re working toward, because you’re going to miss a very obvious thing you could try.

And I think this restriction around the problem actually gives you a lot of freedom to try stuff out, because now the method doesn’t matter. It’s: are we able to get there? Yes. And then you switch things up and try to be more creative around what you’re actually working on.

00:47:00 – Conclusion

Yacine Mahdid

All right, folks. I hope you got as much out of this discussion as I did. If I had to boil down the core nuggets in there, the first is that nobody’s going to hand you the path into research. You’ve got to take it and embrace the messiness of it. Which is kind of on brand, because if you’re doing research, you realize how messy everything is.

Andrew didn’t wait for permission to do stuff. He just took it. He signed up for grad classes nobody told him he could take. And when he was ready, when he’d formed his idea about really doing research, he left a good job to do a master’s.

The second thing: once you’re into research, anchor yourself more on the problem you really like than on the actual methods to solve it. Methods come and go and come back all the time, and very fast, in this field. But if you understand the problem deeply and you care about it, none of the work you put in there is going to go to waste.

If you want a good next step, I recommend checking out the advice he wrote for PhD students. It’s a blog post on his website, and it’s very useful well beyond grad school.

Advice for Early PhD Students
Andrew’s blog post “Advice for Early PhD Students: Prioritize Problems” on his website. (source)

And if you want to go deeper into his research, I recommend reading the KARL paper and also his newest paper on AutoIndex, and hopefully the interview on that will follow soon. Have a great week, everyone, and see you in the next one.

Discussion about this video

User's avatar

Ready for more?