9
Nov

Research in Computer Science: lessons from MIT and EPFL | Aleksander Mądry (MIT) | ML in PL


The best return investment that the university
can get is to produce outstanding people, who go abroad, become successful and then
come back. Hi Aleksander. Hi. You did Master’s here in Poland and then you
move to the USA. Why did you decide to continue your scientific
career there? Yeah, I don’t think this was like a very deliberate
decision. In some way, they way it happened was when
I started my computer science studies I was not even thinking about being a researcher,
but then during my studies I really discovered that I’ve enjoyed doing this and then somehow
like there was one of these days that you kind of read some paper that you really liked
and somehow in contrast to Physics which was another subject I also studied you kind of
realise that people who wrote this paper they actually are alive and they are not only alive,
they are actually teaching at a university in the US and then you look online you realise
that there actually is the way for you to study under their supervision. So it was like, more or less like that that
kind of this idea came to my head. And then I kind of get intrigued by this and
my mentor said that the university actually supported it, so kind of I just did it. It wasn’t like much of thinking like should
I stay or should I go, it was more of a kind of interesting idea and cool idea to try out. Also, I did not expect that this would succeed,
so it was something I just tried, and then once I tried it, well, then everything was
suddenly happening one thing after the other. Okay, so you just say that this was just quite
coincidence that you decided to do it, but you could have another actually you could
go for example to the industry. Apart from your postdoc at Microsoft Research,
you didn’t spend much time at the industry. Why did you decide to continue as a researcher? Sure, so just like one point is that even
though my decision to go study abroad was not very thoughtful, it was more of a spure
of the moment, incidental decision. I think it was a good decision in the retrospect
and we can come back to this point later, but now about this kind of industry versus
academia, never ending question. So first of all, beyond being a post doc at
Microsoft, which was more like a pure research position there, I spent some time at Google,
Google Brain and so it’s not like I really didn’t spend time at the industry, it was
more of a question of priorities at the given moment. In the US, unlike Poland, you know kind of
this difference industry vs academia is much more spectrum than like either or. This kind of connections are much closer and
for me what was the reason I didn’t spend that much time in the industry yet was just
because of the kind of research I was excited about was kind of the better environment for
this was in academia than in the industry. There is companies like Facebook, Google,
you know, they really have people doing like basic research like very interesting research
it’s just the type of research they are doing the kind of this the style of the research
they are doing was not something to be coherent for what I wanted to do at this moment, but
you know it could change at any moment. I can easily imagine spending more time in
industry. Okay, I see. Do you think that PhD was important moment
in your scientific career path or could a person contribute to the field for example
without doing a PhD? Okay, so there are two questions. The first one, okay, I don’t think that the
PhD per se was anything that important in my life. Okay, this is a major point in your scientific
career and where your career really starts, but like, but this I really didn’t care too
much about. What was, though, important was pursuing a
PhD degree. So essentially like when I went for the doctoral
studies at MIT this is where essentially I really learned how to be a researcher and
all of the most important things about being a researcher where all of […]. So essentially
this was like an important period for me. That kind of formed me for the rest of my
career, but the PhD itself is just a degree. It doesn’t really mean much by itself. Now, going to your second question, I don’t
really think that, you know, having a PhD degree or Master’s degree is a requirement
or important prerequisite for having impact in any field, not only Deep Learning. What I think is important though is to get
certain maturity and sensitivity and certain training that essentially pursuing a PhD is
the most natural way to get, but I know plenty of people who actually never, actually, some
of them didn’t even get a Bachelor’s degree. They actually ended up being quite successful
researchers that did a lot of interesting work. So I wouldn’t say that it’s about getting
a PhD or Master’s degree and it’s not even about getting a PhD or Master’s degree in
Deep Learning. I think actually I am also a bit of an outsider
to the field and I think that it’s actually a benefit. As long as you have the right attitude and
a bit of humility essentially like you first learn the field, before you say that you’d
do it better than everyone else. As long as you understand that you are an
outsider and may not be aware of all the important developments, you can actually a lot by coming
with a different perspective. In the end, science is about mixing different
perspectives into new ideas. Okay, so after your PhD at MIT, then you went
to do PostDoc in Microsoft Research and then you became, correct me if I am wrong, the
associate professor at École polytechnique. EPFL yes. Why did you decide to go there? Well, it was an interesting moment right,
so there are like these important moments in an academic career of a faculty is well
your first faculty job. Right, and somehow among the options I had
this was the one that was the least expected, but actually was the most intriguing one. So essentially coming from MIT, you have a
very US centric view and then there was a place in Europe that kind of once I started
asking, I heard a lot of good things about it. It intrigued me that it really conducts research
in US style. It’s a very focused, very dedicated institution
to do the top level research and then I went there, I visited and I really like what I’ve
seen and decided to drove there and give a try. I definitely don’t regret it. This a very, very good experience. I felt like being at the university that for
some reason is next to the Lausanne in Switzerland. It has beautiful mountains […]. It was again
a little bit a spure of the moment and something I didn’t know how it would work out, but yeah,
It worked out. So you are happy about these moment in your
life, but you decided to go back to United States. What made you do that? Yeah, that’s the interesting question , I
really, genuinely enjoyed my time at EPFL, definitely it was better than I hoped for. However, at some point I realised my training
and my most formative stage was at MIT. There was something about atmosphere at MIT
in particular that was really compelling to me, really resonating with me and essentially
MIT is a place with a lot of energy, a lot of […] people working on myriad things […]. This
was something that always felt very motivating and I really enjoyed energy there. This was a little bit less in case of EPFL. Of course, every university is different in
its vibe. I was missing the vibe of places like MIT
and also I was missing… In the end, even though it’s not that hard
to travel between EPFL and US, many of the things I was interested in were happening
in the US […]. Okay, I see. Do you think it is necessary to go abroad
in order to do research in Deep Learning? You mean abroad from Poland? Abroad from Poland. Yes I think so, and it’s that much about Deep
Learning, and it’s not that much about learning Deep Learning. I think the reason that it is extremely important
to go abroad is to get exposed to different culture of work and different environment. So I think the most important thing about
doing research is the environment in which you are. There is a saying that you are an average
of five people you interact the most with and you should always make sure people that
these 5 people are the best people that you could have. So, that’s one thing – going to the top place
in the US that exposes you how good can it be. How does the top research look like. And the other thing is what you also learn
what is very important about being a successful researcher is learning the know how of science. This is a little bit not about domain knowledge,
but about general knowledge how to be scientist, how to give talks, how to choose research
questions, how to publicise your research questions. There is myriad of things that they are not
kind of soft skills, but they are very important soft skills that essentially help you be an
effective researcher. And unfortunately in Poland, especially in
the domain of machine learning, we still don’t have the critical mass to get the right practices. I think we are learning that and hopefully
we have some promising faculties that hopefully will become leaders, but they are not there
yet. Usually, the way these people become leaders
they also went abroad. They learnt some new ways of doing things
and came back to seed it in Poland. That’s why I think in general science is about
being exposed to many ideas and combining many ideas that you have seen and if you just
stay in one place or in one country, it is very constraining, especially when you are
young. That’s the time when you should explore and
learn what are all the cool things that one could do. So do you have any recommendations for Polish
universities to become as good places for research as the ones in the US. Yes, again it’s a very complex question that
I much really like thinking about it quite a bit. There is a couple of recommendations again
they are not easy to follow. One thing which is very important for the
universities to understand is that as counterintuitive as it may seem actually pressuring your top
students to go abroad is a good thing. There is unfortunately many of the places
in Poland that they have this culture how can we get rid off our top students. I think this is because they may not come
back or we are like losing them. I think this is really wrong kind of thinking,
like the best kind of return on investment that the universities can get is to produce
outstanding people who go abroad, become successful and then come back. So that’s one thing, the other thing, again
this is a complex problem that both sides are at fault here is I think it would be very
important to understand, especially in the field like machine learning, that the industry
can be a powerful partner. And again, this is not like that currently,
it’s rather you do science or you do the industry. It’s viewed mostly as a completely separate
paths and that’s very damaging to both sides. Essentially, what US academia is […] disconnection
to the industry is very vivid and very close, because that’s where all the cycle of the
development of the idea: testing them out and […] hundred people go back and forth. That’s what it essentially enables people
to do and this is not really existent in Poland. […] The reasons for that are both on the
industry and on the universities side. Okay, so the question I think is a follow
up for this one – how to make this people actually come back to the Polish universities
and I think this is the same for the industry – how to make these people not lean in the
direction of the industry, so, for example, they would stay there, how to make them come
back? So I think the fact that the part of the problem
is exactly that you are asking this question this way. You don’t want to make them come back. You don’t want to force people to do something
they don’t want. The whole idea is that the talented people
it’s very hard to make them do anything and the moment you want to make them do anything
they lose the talent, you are destroying more than you are gaining. If they feel like slaves, they will not perform
like the top of their ability. So the whole point is just playing the game
of numbers. Meaning, you know tens will go abroad. Maybe only one will come back, because, maybe
family reasons or whatever or maybe they really actually want to come back. That one person that will come back will be
worth all of the investment of these ten people that went away. That is why it is counterintuitive and kind
of feels, kind of not the right thing to do. Just, you should not compel them to come back,
but you just welcome them if they want to come back. So there is this kind of homing grand like
the foundation Polish Science was doing, it was a great idea, just kind of make them feel
welcome and, in general, creating positions, creating kind of an environment. People who want to come back from abroad and
essentially like a start a group. It should be easy for them. Just a funny thing, you know, actually when
I came back to Poland, just for, you know, a short period of time, you know, and I wanted
to be haired at the university. It turned out that I don’t have a PhD, because,
you know, Ph.D. from MIT is not recognized as an official PhD degree in Poland. And again, there was a procedure that fixed
this, but like these small things, that kind of, you know, that all to be like, kind of
prototyped and these should be like more streamlined. Decisions, should be a… We should not force people to come back. Just say: “Listen, if you ever want to come
back, let us know, here is how you can do it, it will be easy, you will have a lot of
impact…” and so on. The same is with industry, I don’t think you
shouldn’t stop people being in the industry. You know, like the most professors I do now,
especially more applied sciences, they wend out, they founded the company, they were very
successful, they were completely emerging into it, immersing to it for, you know, for
two, three, four years. And then they came back and because now they
felt like “Okay I really love academia for different reasons. Now I’m ready to come back, I kind of did
what I wanted to prove to myself in industry. And now I’m ready to kind of be again more
of a professor.” And in the US, you can really be like again
you can be on leave and , you know, and work at the company and then come back. Essentially, they understand this is a part
of the carrier, the carieria of the researcher is forty years. You can’t just commit yourself to either or. This sounds like a good point. I have another question: on your website,
it is said that you don’t accept student internships unless they are from MIT. My question is if there is a student, for
example from Poland, came to you with a research idea, contacted you through email. Would you guide him? And if yes, then what kind of characteristics
this student should have? So… ok, no… there are two points here…
yeah… first of all the answer is no, meaning… it’s very simple, like essentially, I’m a
person I think, I definitely allocate everyone and embrace this student, when I do something
I want to do it really really well and I don’t really believe it is possible to be an effective
advisor to, you know, to someone if they are abroad. I don’t think it really works, so essentially
the only… If someone from Poland want to works with
me, the only way to do it is via applying to PhD program, because then if, you know,
if they, if they come, then I will be able to guide him/her, you know, very well, I really
can kind of take a responsibility of their development and really work with them closely. I don’t believe in kind of half measures,
that you can kind of do it over email, actually I try it with a cupule of people and it never
worked out, it just… you know. So I have an attitude that either student
can be physcaly at MIT and work there with me for an extended period of time or they
should… if they want to work with me, they should pursue a way to get there. And, you know, you can get to PhD program
at MIT from Poland, there is not… it is of course very selective procedure, but there
is no prior reson no to do so. If you have a great research project and you
lay it out in you application that would be something different that will catch our eye
if this is something interesting. Ok, I see. You’ve actually done a lot of research on
algorithms, but recently your research is mainly on deep learning. Why did you decide to switch from algorithms
to deep learning? Yeah, so I wouldn’t call it a switch. I would say it’s a change of focus and, you
know, so internal like I embraced the idea that, you know, the beauty of being in academia
is that you have a freedom of pursuing what fascinates you and, indeed, recently, you
know, deep learning, things I will talk about in doing a lecture are things that I find
quite fascinating. So I definitely credit towards it. But now, how did it happen? Well it was pretty interesting, right, so,
as many of the good things that happens in research, it was actually prompted by students,
my students. Sort of lucky. We did keep shooting about this deep learning. And kind of hear that first and says okay
this is another wave it will like just pass away, like why would I pay too much to it
but then you know hear about this and kind of my students got curious about this and
they came to “oh you know like we are really interested in that so how about we kind of
start the reading group, to just try to see what’s going on there”. So that’s what we did, and we started like
reading about this and after that and after a couple of months of studying this subject,
we realized, well, you know, this is actually quite fascinating
because A) it really works, you implemented this thing and some of the basic things actually
work B) it’s clear that no one has an idea why it works, like there’s clearly a lot of
challenges that kind of people don’t know how to tackle
and then realize that we actually have some ideas you know how to push through these challenges
and then a distracting the moment I said “okay guys we can do it but if you do it, we do
it for real, I buy GPU machines, I kind of… you know, we learn TensorFlow” at that time
and essentially we kind of do it for real or we don’t do it. And they said, “ok, let’s do it”. And we started doing this. We had quite a lot of success with our first
papers and then, kind of, you know, success, biggest success. Yes, like once we understood some piece of
it and we contribute something. There’s another piece that you can
contribute to and again. This way we kind got more and more excited
about what we are doing and kind of that’s how it evolved into a research agenda. That just honestly like all the science happens. It’s always a random walk. It’s a random walk with momentum. But it’s a random walk nonetheless. Ok, so. Basically, the reason why you actually produce
so many works on deep learning recently was basically excitement, right? Yes Or there were other reasons? It’s excitement but also no. The culture of ML is… So this is the mistake we did with our first
work where like produce a long paper with a lot of appendices. We just wanted to look into everything very
very kind of… Explain everything right away. Then we realize that actually like the way
this field works is like it prefers like chop everything in smaller pieces and you know
that’s how this field communicates. So we kind of and we had a lot of say, but
we had to chop it into smaller pieces and that’s the kind of the result of material. We are definitely very excited about this. And we think we have a lot of fun pursuing
it and we are quite productive. Ok and last question: do you think deep learning
is overhyped? Of course, it is. It’s how could it be not overhyped? The real question is: is it overhyped for
good or bad reasons right? Like in some ways every new exciting idea
that has impact, gets overhyped. That’s kind of human nature and… That’s totally fine. My worry is that, you know, this overhype
might lead to bad outcomes. I generally believe there is
something new and very interesting about deep learning we did not yet zoom in on what this
thing is, but it is there. But now my worry number one is that in this
excitement and hype over deep learning we will kind of start deploying the solution
too early on things that matter and that might lead to a catastrophe. Because the reliability is not there, the
robustness is not there. Any kind of… and there might be some real
negative consequences for doing it too early. And the second connected point is that, you
know, after every overhype comes underhyping. People get like
it is promising that it’s cancer and do everything overnight and now it is not, so people say
“oh this is maybe not worth anything and maybe we should completely
abandon it”. And I would like it to happen. Essentially I think again. There is genuine, genuine value in a like
in some of the insights discovered there and I would really hate it if you had another
AI winter where this all of this is discarded. because now we just don’t like it, because it was overhyped. so it is overhyped. My only worry is though that this will not lead to these negative outcomes. Okay, thank you for today’s talk and see you next time. Yeah, thank you.

Tags: , , , , , , , , , , , , , , , , ,

There are no comments yet

Why not be the first

Leave a Reply

Your email address will not be published. Required fields are marked *