Chat about Teaching AI at Stanford with Abigail See
52:39 Share

Hello and welcome to the Last Week in AI podcast, where AI researchers discuss what's going on with AI. This is a special episode where we will not be talking about news from the Last Week in AI, but instead talking about our experiences in AI with one of our friends.

In particular, we'll be talking about our experiences being teachers in computer science and AI, especially as we were PhD students at Stanford and took part in teaching the AI curriculum there. I'm one of your hosts, Andrey Karenkov. I'm currently a PhD student at the Stanford Vision and Learning Lab and have co-taught a few classes in my time at Stanford.

And I am Dr. Sharon Jo, and I am currently teaching the GANS class at Stanford as well as on Coursera and have also taught computer science courses throughout my undergrad as well as graduate career.

Hi, I'm Abigail C. Oh, Dr. Abigail C. I think that's the first time I've said that out loud. And I am a recently graduated PhD student from Stanford NLP group. During my PhD, I was the head TA of the deep learning and NLP course.

Yeah, so we all have some mixed experience teaching and we just thought it would be fun to kind of chat about what it has been like, what we got out of it, what we, you know, thought about it and so on. And to start with, we can go in a bit of a chronological order. I was thinking where we could go into our first experience of teaching versus our most recent and presumably more kind of involved experiences where we had CAs and, you know,

co-created classes and crazy stuff like things. So, yeah, I guess I can go first. I started teaching pretty early on. I was at Georgia Tech. And Georgia Tech is interesting because a lot of the CS classes, most of the first and second year CS classes, all of the teaching assistants are undergrads. Most of them. There's really no grad students until you get to a graduate level.

So for classes like intro to object oriented programming, intro to data structures, it's all people who took it, you know, within half a year or a year and did well. So, yeah, I got into it in my first year. You know, I applied after my first semester. I did not pass the interviews to get to get a TAship, but I did start tutoring people.

And then my first summer, after my first year, I started teaching intro to object-oriented programming. And I can talk a bit more about how that was. What was your first experience, Sharon?

That's very cool. My first experience would probably characterize it also as being in college. I went to Harvard and actually, I think it was a bit different. It was typically not undergrads teaching. It was mainly graduate students. But the intro CS course took

a very different stance on it. And I think it was really smart of David Malin, this is for CS50, to get very excited undergrads who had just taken the course to then be TAs. Because when I had taken the course first, my TAs stayed up to 2 or 3 a.m., understood the undergrad experience, were undergrads for the most part, and just...

It inspired me to want to give back the next year to that experience and also stay up to 2 or 3 a.m. working on P-sets just to help each student get to the next stage where I was just the year before. And so I think that was really smart. And they largely, when they recruited, you had to do well in the course, but you also had to be very enthusiastic. And I think that pairs really nicely with teaching.

Um, and I, I love that experience. I ended up teaching another class at, uh, at Harvard, um,

user experience design. And that was really, really fun as well. And I think most maybe memorable experience teaching at Harvard was actually an extracurricular where I had to teach people who were youth in prison. So youth that was incarcerated. And one person that I was paired with used to be in the gang Bloods.

and had potentially killed people before. And yet he couldn't read. And something that just like has really stuck with me was just that he couldn't, you know, he couldn't read, couldn't read. He would act out like people would have to restrain him. Like he did almost hit me a few times and it was scary. But then one day he just like yelled at me and said, you know, why are you even here? Like how much they pay you?

And I said, no one pays me. I just volunteer. I just come here every week. And after that, he started to learn to read because because someone believed in him. And I think honestly, like that really relates to my experience learning. Like I needed people to believe in me for me to really get get through a step and also to become very intellectually curious. And so that's something that stuck with me.

Wow. Yeah, that's quite the story. I had no idea you volunteered. No idea, I forgot. So that's cool. That's okay. That sounds like a really extreme kind of setting to be teaching in. And it reminds me of a much less extreme, but some elements in similar elements. Oh, fuck. Let's get rid of that. Nope, that's what we're keeping. All right. Starting our senses again.

Sharon, that's a really interesting story. And it reminds me of an experience I had that was much less extreme, but had some similar experiences.

elements, which was I was tutoring some high school students to try to help them pass their essentially high school maths exams. And honestly, I don't think at that time I did a great job because I didn't understand where they were coming from. So I think especially in maths, if you get behind the class, then it becomes so impossible to catch up. So I

in the curriculum for these exams they were taking with things like quadratic equations. But what I didn't realize is that these students hadn't mastered fractions. They didn't really understand fractions. And then once I realized that, then I kind of went back and we ended up spending

the whole period on fractions, actually, because it seemed like we couldn't really progress if they couldn't do fractions. And in the end, I don't know if we made that much progress because, again, coming from such a different background, I wasn't understanding their struggles. So, for example, I gave them some books and some homework to do, but they never really did it. And then sometimes the reason would be...

that they had left the books with one parent, but then they'd been staying with another parent, but they hadn't had, I think, enough stability or even knowing where they would be each day to plan ahead and do the homework. So they were facing such different struggles that I think I was not well enough prepared as a volunteer tutor trying to help, but really in the end, I think not helping them very much.

Wow. So I'm really admiring of you, Sharon, that you managed to make a difference with someone who was in such tough circumstances, because honestly, I didn't know how to make a difference there with the fractions. Well, I'm impressed that you were able to just adapt really quickly to a different subject, because I feel like I often...

including especially or rather, especially when I was first teaching, I needed to prepare like crazy. So I would prepare so much stuff and being able to adapt is, is, is hard because you're like, I've prepared so much for this lesson. I even prepared like when to hand out candy, you know, like baked cookies. Yeah.

every possible element of it. But of course you need to be able to explain every single thing and be able to be like, okay, actually we're going to scrap this whole thing and go back to the drawing board. And I think my best teachers in, in high school, elementary school were, were from, were people who could do that really well and just say like, Oh, okay, no, let's scrap that. Let's focus on this when they realize the class didn't understand something, you know? So I think that's really important.

Yeah, I can relate not too much. I mean, I haven't gone outside college and teaching really, but I do think there's a similar element. And one thing I really liked about being a teaching assistant is not only did you have these recitations, like lectures, where you kind of talked about elements and had some questions, but for me, we had office hours, right? And back in Georgia Tech,

I remember pretty vividly my first experiences. We had this group room and that was like in big classes. So we had five, six TAs and a lot of people came by. And I think in that respect, it was similar in that people of different backgrounds came in all the time, like people who had done programming and were comfortable with it or people who were sort of just getting started.

you know, there were people like me who were doing it from high school and were like, you know, kind of ahead and people who weren't as comfortable. And you're right in that, you know, you do have to sort of understand where they're at and adjust how you communicate and sort of, yeah, really try to be on their level. And I remember, you know, I don't, I can't really remember why I first got into it. You know, thinking back, it seemed like just another, you know,

thing to do you know for my overachieving kind of personality and to earn some cash but once I got into it you know especially in those office hours and when you were talking through a concept right and then that clicked in their mind right and you really see it yeah I got I really started enjoying it especially for that aspect of sort of just walking people through and then

having them get, uh, what they're struggling with. Um, yeah, I really enjoy it. So for me, after my first year, uh, I TA'd, I think the following three years at Georgia Tech. So I TA'd pretty much for all of my undergrad, which was really fun. And then, yeah, later on I did, um, intro to AI for two years as well in my last two years. And, uh,

That was pretty fun as well. I got to be at TA as an undergrad and change the curriculum a bit and kind of foreshadow my experiences at Stanford. I think the system that they have in the US of TAs to hold office hours and help students, maybe especially with the kind of lower level difficulties when they just haven't grasped the basics of what's on the curriculum.

I think it's really valuable. My experience going to a British university was that there was nothing like that. It was very kind of fend for yourself in a way that I didn't find an effective learning environment. So you'd go to lectures, the professor would give the lecture, then you'd be given a problem sheet and you'd work on the problem sheet. And there was a kind of unspoken culture that you wouldn't collaborate with others. Again, I think the US culture of collaborating study groups

is a lot better in terms of learning efficiency. So yeah, it was kind of struggle alone and then you'd hand it in. And at least where I was studying, which is at Cambridge University,

Then we had supervisors who would usually be graduate students or even faculty who would then grade your work. And then you'd have these kind of two on one supervisions where you and another student would go through the work you did with that supervisor who would kind of explain the solutions for the ones that you couldn't figure out or discuss anything else you had to discuss. So I guess, you know, that was a big privilege to have such kind of support.

intimate access to discussing these things with very highly qualified people who knew the subject at a research level.

So that is good if you're at the level where you're flying through material and you want to ask like really, really high level advanced questions that go beyond the material. But if you're struggling and you just didn't really understand the basics of what was given in lectures or any other reason you might be struggling, which, you know, most students are at some time or another, then actually that wasn't the most useful thing.

I think having access to study groups and office hours with undergrads who did the course last term would have been a lot more useful. And honestly, I was more often in the struggling with the material than the flying ahead group because, you know, it was pretty hard. So, yeah, I think if you've experienced this kind of US style, then you should definitely appreciate it. Good to know. Good to know.

I really like the US style of like the struggling with the material, but then you can opt in to office hours with the professor, you know, and talk about things more at a high level if you want to. It's not always structured exactly that way. Like maybe office hours is not always the high level, but having that as an opt-in, I think it's great because after you struggle, maybe you then do start thinking,

You start thinking of new things and then you can go and opt into that and you always have access to that in a way. So I don't know. I do find that really nice as well because I think we always...

I don't think anyone goes through without ever landing themselves in the struggle camp. You know, it's just inevitable because it's not even a function of like your intelligence. It's a function of like whether like something was explained well to you, you know, everything. Like it's not, you know. Absolutely. Yeah. Yeah. That makes you think actually from the learning side, I had this experience where, you know,

You know, these were the first two years of college. I was very much of a lone wolf approach. I didn't go to study hours or office hours. I just kind of worked through things and it usually worked out. But then when I got to higher level classes, and especially at Stanford, when it got pretty fast paced and I had other stuff going on, I had to learn to actually go to office hours and ask people for help and be in study groups.

And in some ways I felt disadvantaged because I was not good at that. But then, yeah, I appreciate that being something I could do and should do to, you know, be able to learn more effectively. And I think this sometimes comes down to a fundamental tension in education, which is learning versus learning.

evaluation, like the need to learn, which is obviously the primary one, and then the secondary need to evaluate. And sometimes the need to evaluate can overshadow the need to learn and leads to less efficiency learning. So I think the need to evaluate leads to a bunch of anxiety in students. And then you can feel anxiety

shame even at this need to collaborate or be in a study group or have things explained that you didn't understand the first time around and then that can get in the way of actually learning and I certainly don't know the solution to this because people who have whole PhDs in education are working on these kinds of problems but that was something I found as a student and also found as a teacher. Yeah

I know that when I, the moment I think about grades, my grades tank. Like it's very obvious in like finals, grades, everything. Like if I care too much about the grade, then I start like,

over like optimizing for that and being really anxious about that as opposed to, um, actually loving learning. And that's like, that's so consistent across because everything I've done in education, it's yeah. Love it. Gotta love the journey. Yeah. I can definitely say that when I've been in environments that have had a kind of toxic, um,

emphasis on on grades and the grades representing your worth or your intelligence then that has just been an absolutely disastrous learning environment for me in terms of just efficient learning and actually understanding things and taking them away with me in a usable form and one thing again that I appreciate about the US system is the greater emphasis on students choosing what to learn based on what is useful for them in their world and their future careers

And I know it's definitely not perfect. And there was certainly still a lot of stressing about grades among Stanford undergrads. But again, I think that's a better system. Good to know. I didn't realize we had so good. Well, with our first experiences discussed, and it sounds like all of us kind of enjoy teaching and got a lot out of it early on. We can, I guess, jump ahead to Stanford, where we all

I ever have been in your cases or I still am for the time being.

Hopefully not for too long in time. But yeah, I would say each of us has done pretty, like we've covered different areas. So I have taught, I've co-taught intro to AI first, then machine learning. And then most recently I was, we had CA of the CS2401A of the computer vision program.

geometry and something classically like classical non-deep learning computer vision class that was actually getting revamped. And I think in your cases, you had also kind of head CA or pretty major roles in your respective classes. Yeah, that's right. I was head TA of the NLP and deep learning class for two years. And the class is taught by Chris Manning.

It was actually originally just an NLP class, but then it evolved into this NLP and deep learning class once deep learning became the dominant approach in NLP. So, yeah, I really appreciated taking this class that had the input of so many great people in the NLP group over the years. Obviously, Chris, Richard Socha and then lots of different students involved.

over the years. So it was great to kind of inherit all of that material and then evolve it for the coming years. But that was also really difficult because I think people...

People can contribute these really excellent kind of bricks in the whole wall of the class. Right. So they they contribute an excellent assignment or lecture or sequence of lectures. But then sometimes when you kind of put together all those bricks from different years together, then from the student's point of view, even though the bricks are good, like they're getting a bit confused in the slight disjointedness moving between them.

And I think that that was one of the hard things about the necessity to replace some of the bricks each year because the field is moving so fast. It meant that sometimes just things would be unexplained and kind of fall through the gaps. Right. So some new assignment would be talking about this or that or a lecture. And then the students like, wait, I'm confused. We didn't talk about this last time. Or maybe we're using kind of different terminology to what we used in the previous lecture. So honestly, that was hard.

a big problem that I could see was a problem from the student's point of view. And something I tried to work on when I was working on the course as a whole was to make the, the kind of the arc of all the sequences of lectures, a more coherent one.

But that is so much work, honestly. It's a lot. It's a lot to take on. And I really do empathize and understand how people who are running these kinds of courses can sometimes end up with things that are a little bit disjointed because you just don't have the bandwidth usually to smooth out the whole thing, right? And you just have to deal with what you've got. Yeah. And I think going into PhD, maybe that's something you would not expect as much because

You know, you would know you would be doing research, but I think many people do end up getting kind of higher level roles in teaching a class. And then you have to, as you did, kind of try to take part in revising material. And I also had to do that at Georgia Tech for Intro to AI. We like revamped a lot of our homeworks. When I had CAA'd the computer vision class this last year,

Actually, it went through a major change where half of the material was revised to deal with more deep learning based techniques and things like that. So it's really hard. And then there's a lot of blunders that happen. I think in my experience of when you put in your assignments, there's always things that go wrong and you have to scram and fix things real quick.

There's a lot of confusion, as you said, you know, problems with the slides. And yeah, and especially in AI, I guess, as you said, you know, a lot of these classes like, you know, I don't know, data structures or algorithms are not necessarily evolving as fast, but NLP and computer vision, all of these things, you have to revise pretty much on a yearly basis. So it's an interesting kind of set of material to,

to work on.

Yeah, I definitely feel that as well. So I teach the GANS class at Stanford and have been teaching that now for the second year and also on Coursera. And it's a slightly different dynamic in terms of, you know, on Coursera, we have a very specific type of structure such that it's supposed to help the learner get through things. It's not that much, you know, per week.

But if you're going to have to what we call refresh it, do a refresher. It's a huge process. It's like another, you know, six months, a year process of refreshing the whole course to make sure everything is consistent to make sure you know, like if you're going to get rid of one section, you want to

Um, or maybe you want to shrink one section a lot, then you want to like merge it with something else. And then you probably have to rerecord and everything, um, is, is time. Like it takes just a lot of time and editing. It's like redoing an entire course with a few resources in place, but, um, not, not everything. You can't just...

continue adding lectures and assume that you can keep going. So yeah, definitely, definitely feel that. And I assume actually with the GANS course and also the NLP course, you know, with being Stanford, these are big classes. There's like at least a hundred people usually, if not hundreds, which I assume is true in some of these classes you taught. Yeah, I think we had about 450. Yeah. Yeah.

And then, yeah, when I taught intro to AI machine learning, those are like 600, 700, something crazy, you know, 30 teaching assistants. Computer vision was a little less intense. It was more like 150. But still, you know, it takes a team of people and there's so many different people getting so many different kind of questions that adds a whole new dimension for sure. So we would...

So we were just talking about how everything is time and you probably don't have that much time if you're a PhD student. So I think this is one of the kind of fundamental tensions that makes being a PhD student hard, right? It's like, are you studying or are you researching or are you teaching or all of them maybe?

Or maybe are you mentoring other students? So I certainly found it extremely difficult when I was teaching because it was so all consuming and I genuinely enjoyed it and cared about doing a good job and making the course as good as it could be. So, you know, I quite willingly put...

all of my working time into it but then um it was tough because uh I kind of put all my research on hold and it didn't feel like that was a good idea in terms of like the rest of my PhD making progress uh and not to mention actually the time which we spent working on the course in the quarter before it launched so did you guys struggle with that when you were teaching yeah

Yeah, I mean, I think that's kind of one of the unfortunate things with a PhD is the metric that really ultimately for real matters is the published work, right? And your citations and so on. And when people, you know, care what you have taught, it's on your CV and so on. But

it's not as ideal than publishing papers. And for me, yeah, I would say not just in terms of teaching, but also in other forms of service. Like I've,

done various things that have been outside of research that were time consuming like starting with Stanford AI Lab blog and co-running it for like two years that took some consistent effort and

I thought it was important and really valuable, and I would consider it part of what I would do as my PhD and work as part of a PhD. But it did feel sometimes like I'm slowing down my research in favor of something that really won't count in my favor professionally, even if I think it's a good use of time.

So it's, yeah, it's a challenge. And I guess you just have to prioritize what you think you should do versus what, you know, how much you want to succeed in the end. Well, I do...

So, yes, you have to prioritize. But I do think there is a slightly unrealistic expectation on PhD students, but of course, also research faculty who are expected to teach as well. It's a slightly unrealistic expectation that you're expected to continue to output research at a certain rate, but then also deliver excellent teaching that the students review well and understand.

I have a lot of sympathy with all of the people who struggle with that. And I think it can be to the detriment sometimes of the students who are learning when the professor was not given enough time or incentives to work on it. I know that sometimes as a student, I really struggle to comprehend the material in lecture courses.

because it just wasn't really accessible for me where I was at as a student at that time. Because I think obviously one of the biggest difficulties in teaching is

You have to, as an expert, remember what it was like to be a beginner. And I think that, you know, that doesn't really come naturally to anyone. It's really tough. But, you know, it's definitely not going to be any easier if you're not being incentivized to work hard at it. So, yeah, as a student, sometimes I felt really lost when I was watching these lectures by lecturers who,

I think you'd really kind of lost touch with what that field might look like to a beginner. Yeah.

Yep, that is definitely true. I think I came about teaching at Stanford during a time when I stopped caring about my PhD. So I think I have made sense. Pandemic just hit and I was actually just teaching my sister about my research. And it was actually really great because we, Abby just mentioned this earlier,

expert blind spot, essentially, of the expert doesn't really understand why a beginner doesn't understand things anymore. But she would be willing to say that makes no sense to me, you know, and so that kind of turned into a course because then I could say, okay, that doesn't make any sense. How do I make this work?

sensical to this person who is probably like my target student because she had taken intro AI, you know, but didn't know stuff about GANs and probably, and she admitted, she's like, I took intro AI, but I probably like forgot everything in it. I don't know

okay so let's really like let's really figure out what this person needs um so that was super helpful and I also at that point didn't really care so it was means to hang out with her um yeah that sounds like a good precursor to actually making the course and you you like co-developed the course right for Gans yeah yeah she was so helpful that's awesome

By being just willing to say that doesn't make sense. Because, yeah. Yeah, it's interesting. I think looking back, also another aspect of teaching I appreciate is I do feel that, you know, not only did I enjoy it for the sake of helping people and getting out, but I do think it

taught me a lot obviously so like we all could agree i assume that teaching material is the best way to learn it fully like yeah and i think that's another great reason why the u.s system of having undergrads as tas is a really great opportunity for those undergrads yeah yeah undergrads can be way better teaching assistants than grad students sadly but uh

Yeah, so you learn material real well and you get really comfortable with it and understand it at a deeper level. But also, one thing that I guess is surprising in retrospect is I think I got really comfortable and started enjoying public speaking as a result of teaching. So doing these recitations in undergrad, early on I was...

I would say probably pretty awkward and didn't necessarily enjoy it. But after you do it for, you know, one semester, two semesters, three semesters, standing in front of a board, you know, talking people through stuff, you kind of, at least in my case, you get sort of the confidence. And then when you need to deliver talks for papers, it comes kind of naturally in some sense, at least for me. So I always...

I think I got a lot of teaching in terms of kind of both knowledge and skill set. So question for both of you as great presenters of research or teaching material. What are your main kind of findings personally for how to make a good engaging lecture or presentation? I think there's a few things. I mean, you need to prepare a lot. I think both of you would agree you need...

It's almost like writing an article, right? You need this structure and flow of ideas to make sense and to kind of build on each other and not become a mess and not, you know, make jumps. You need a coherent storyline for that lecture or presentation. And that's hard. And that actually also goes into research where you need the same in a paper, right? To convey a research concept, you need to tell a story, right?

And the same could be said of classes. I don't know what you think of that, Sharon. I think a lot of conveying the idea comes down to...

connecting it with past concepts that the student already knows about. So making sure that you don't like jump into something immediately. And I actually find something really ironic or weird is that class descriptions make no sense because for the user, essentially, the person coming in looks at the class description and it's just like, I have no idea what any of these words mean. I guess I'm taking this class. And it's like,

It's like, you will be learning blah, blah, blah, blah, blah. And these are all keywords. And, you know, it's great for, um, it's given. And now I know, cause I've been on the other side, it's actually given to the department to verify that they are teaching all of these things. Right. But they are not for the student learning. And so I feel like a lot of things are not thinking about the student and what they know already. Um,

And I think that is a really good example of that. I'm probably an extreme example. Hopefully courses do think about, you know, where the student, the student is coming from.

So I think it's that plus I really capitalize on engagement because I am someone who basically never went to class because I was so bored and would fall asleep or I would go to class. Actually, sometimes I would sit in classes, just get other work done because I'm like really good at not paying attention in classes. So it's like how good I am at not paying attention. It makes me focus on other things. So yeah,

I try to throw in, you know, memes, humor, everything, um, but make it fun, light, and also make examples that make sense of the person and, you know, draw from, um, their experiences. Um, yeah. I totally agree. I think sometimes people draw a false dichotomy between, uh, fun and lightness and accessibility and, you know, rigor or content. And I don't think that's true. Um,

One bugbear of mine is introducing a new concept or a new definition with no intuition. This happens a lot in pure math. It's just like, blam, definition. Here's a thing, loads of symbols. Why? I don't know. Just here it is. And I found that so hard when I was learning math. So I think it's so much better if you can give a kind of

even in kind of like everyday speak, uh, intuition or definition of like, what's the point of this and why is it? Then you can now give the very precise formal definition and like everyone wins, right? Like maybe some people will only get the intuitive one for now, but they're going to go back and look at those lecture slides. Cause I think,

almost everyone who's taking the class and needs to work on this stuff, the assignments is probably going to go back and look at those lecture slides again. So it's all right to kind of have some people get through the initial watching the lecture on the intuitions and then they can go back later and that's fine. I think people who are teaching or giving presentations need to be really realistic about how many people you're going to lose during the presentation itself. Um, you know, depending on the material you're presenting, it's, it,

could very well be most of the audience or be lost in some way or another on the details of what you're telling them. And I think if you're realistic about that, then it can change your perspective quite a lot because that will highlight like the importance of a story that people can latch onto. If you even have like sometimes a slide

about like, you know, here's the story of like, we're going to find out what is this and then we're going to find out why it's a problem and then we're going to find out an interesting solution or whatever. And then you keep showing that again and saying, okay, now we're in part two. And sometimes I find it even useful to like directly acknowledge I might have lost you if I've lost you, that's okay. And I think firstly, if the teacher normalizes that and says it's okay to be lost and that gets rid of some of that shame,

so you know I sometimes say if I've lost you that's okay right now is a good time to jump back in because even if you didn't understand all the details of what just happened you will be able to follow this next bit so I find that's useful sometimes um and uh

Another one, I think, for kind of normalizing being lost or not knowing everything is sometimes if the students put their hands up and ask you a question and you don't really know the answer, I think it's quite good. It takes a bit of bravery, but it's quite good to say, I don't know. Instead of just trying to style it out by giving a related piece of information or, I don't know, pretending that you need to go look it up or something. I think it's quite good to say, I don't really know, but reflect thoughtfully on the question and say, yeah.

you know, why it's a good question or whatever. Yeah, I agree also with Sharon on that, with you, Abby. Like, I'm as a learner, I usually spaced out during lectures and just like watch out for work. Yeah, all the time. I mean, I think it's just kind of not really natural for humans to pay attention completely for 80 minutes. Yeah, and because of that, when I do present, I try...

You know, yeah. Be engaging, you know, use what's the word, you know, kind of the same as you would be in a talk again. Like you want to have some energy. You want to use your voice to get people's attention and not just, you know, drive, deliver the whole thing end to end. And I also, one of my classes that I did intro to robotics is

incorporated a really fun thing of like life quizzes where the professor incorporated three or four stops in a lecture to ask, you know,

here's this thing, what do you think the outcome is? And there's multiple choice, and there's something else. And then that was a great way for her to gauge, are people following it? Do they understand what's going on? And if they don't, she could go into actually explaining and reiterating some of the key concepts. So yeah, I think I'm a big fan of more interactive things. And also in that sense, I also kind of

I'm tempted to think reverse classroom stuff is a good idea, but... I don't think I've ever experienced reverse classroom. Yeah. Yeah, it's interesting. I think now, also in my experience, like, at Stanford, we have recorded lectures for a lot of stuff, right? For a lot of classes, especially in CS.

So people don't go to lecture and then just watch the recording later. And at that point during COVID, one of my classes that you also took, Sharon, the computer architecture class, did this where he just published the recorded lectures. And then during the lecture, you could just like, he went through the slides again and you could ask follow-up questions and clarifications. And I found that a lot. Yeah. I didn't know. He never went through.

Serious point about flipped classroom, right? I've heard a little about that. I haven't experienced it as a student or done it as a teacher. I mean, apparently some people say that's great and much better, but then it's at times like these when I feel really underqualified and

to have done these big teaching jobs that I did do because I have no formal training in being a good and effective teacher. And essentially everything that I did that was good or that was bad was just born out of my own personal experiences, being a student and what I thought would be a good idea. And I think, you know, that some good stuff came out of that. But then at other times I felt like I just didn't,

really know what I was doing. And one of the areas in which I struggled the most was evaluation. So I found it fairly natural to do lecturing and explain concepts in the way we just spoke about, but kind of designing the evaluation rubric for assignments and all of that was an absolute nightmare. Personally, I did not like doing that at all because it was, um,

I don't know, it feels punitive, right? Because if you've got to define like what's a good grade, then you're also saying like, who's going to get not such a good grade. And that really matters to students. And, um, I,

I really didn't like that very much because I felt like I wasn't really formally trained in education and how to do that kind of thing correctly. And I was just really a person doing my best. But, you know, these students, they were trying really hard and their grades mattered to them a lot and their grades would have an effect on later things. So, yeah.

when I read about stuff like flipped classroom or other kind of initiatives that people who know so much about education and have done these studies to find out what's most effective for students, you know, I feel kind of bad. Like, I wish I knew more about that, but I didn't really have the bandwidth on top of what we were already trying to do to understand those things and kind of innovate in those areas as well.

I wonder if you guys had any thoughts on this about like being sufficiently well-trained to do what you were doing. Yeah, it's an interesting point that, you know, at the grad school level, you go into teaching without taking a class on teaching anything.

as far as I know, like in almost all cases. So to be fair to Stanford, they do have this center for teaching and learning, something like that. And you can opt in to get help from them to be more effective. But it's really an opt-in thing that not very many people do. And again, I just didn't really feel like I had the bandwidth. So I was just going to acknowledge that. But go ahead with what you were saying, Andre. Yeah, yeah. I think it's an interesting point. It seems like...

As with research as well, you're supposed to sort of, there's this informal mentorship system, you know, where there's more senior PhDs and professors and you're supposed to sort of get their skill set by osmosis as you're like doing intro level TAing or taking the classes and stuff like that. And yeah, certainly I think for giant like intro level classes, it's

it would make more sense to have kind of more informed or more, yeah, more educated approaches. Cause when you're teaching 800, 900 people, it's not the same as a grad school seminar or something, you know, I think it would be of benefit, but I guess we don't do that. I did read some books about teaching and teaching methods that were, you know, tried and true or studied or researched or stuff. And I think, yeah,

It's really easy to read about those points, but harder to implement them because I'm sure every teacher I've had knows what those points are. But sometimes it's just hard to enact that and make sure that does actually happen in the classroom. And regardless of formal training, I feel like I've had teachers that were really good and the teachers that are really bad, but I'm sure all of them have looked into learning theory at some point. And it's,

And yeah, it's, it's, I think it's really hard. And I wonder about, you know, having formal training, whether that would actually be useful, useful or not. And I, I do know that, like, I think people discuss, you know, people wish that, you know, professors had management training before being in charge of PhD students, you know, and so I think, yeah.

how effective, like how effective I wonder are these training programs and would these training programs be? And like how, cause I feel like it would have a huge impact if they were really effective. Um, but I feel like I'm slightly cynical that it's going to be like one of those like things where you're trying to click through as fast as possible. Um, yeah, I guess I feel at the end of the day, very,

That would help, but what helps most of all is just caring and putting in the effort. And so, yeah, I guess looking at myself, I think certainly I could improve in some ways if I, you know, learn some more of a theory. But at the same time, having been teaching now for years, you know, having co-taught

like six, seven classes. I can draw a lot of my experience and just, you know, wanting to do well. And in that sense, I think I'm less kind of anxious where maybe some things I could be doing better, but I have a lot of intuition built up from just having done it enough, maybe. Yeah.

One thing I found kind of tricky with teaching AI in particular is that at least the current era of deep learning AI has a lot of kind of empirically motivated, but we don't really know why it works stuff that you're teaching.

So often with students, we describe like, here is the deep learning recipe for how we currently do this. We used to do it like this, but this thing seems to be more effective. We don't really know why. Some people have this kind of intuition story for how this might be working, but honestly, not sure if that's right or not. So one thing I found a bit tough as a lecturer, let's say, was how to present that.

right? So you can choose to maybe simplify it a bit and just give the intuition is like, that's why it works. But then, you know, maybe in a few years you look back on that and it turns out you were kind of wrong or like that thing didn't work so well actually after all. Uh, but you can kind of acknowledge the complexity and basically say what I just said about here's how it works, but we don't know why. Um, but then I think that can be a bit tough as well for students. It's like a lot of information to take on. Um,

I suppose I often went with the kind of acknowledging the uncertainty one sometimes. But did you guys struggle with that at all? Yes, definitely. Because students always ask why. And if you can't give them a why, it's really frustrating. And as someone reading a paper, I'm also asking why. But there's no reason. And it is really frustrating. And it just becomes something you have to maybe memorize if it's on a test. It just doesn't make any sense to have something like that on it, you know?

Yeah, exactly. So I think it's one thing to kind of just have a discussion in the lectures, but the idea of having that in a test or assignment to explain why, that was also a really tricky area, right? Because

having students answer the question of why and look within themselves of what they've read to try to come up with some ideas why, I think is probably a valuable exercise for them. But then to then grade that and say what's right or wrong was basically kind of impossible. Yeah, I think for me, I haven't taught as much cutting edge deep learning. I taught a lot of intro AI, intro machine learning, intro robotics. So in that sense, yeah,

It wasn't like the ideas were more established, but that also meant I had to understand them in a deeper way to be able to explain why, you know, beyond just like the surface level, I needed to understand well enough to convey things. So, yeah, that's another challenge for sure. Another big area that I think is relevant to deep learning right now is the

I think it's really helpful to have students be able to run things, you know, and run like little examples or toy, like a concept. And maybe it's a toy version, but they can implement it. Right. And that's a huge challenge when compute costs are really high now. And so it's really hard for a student to run a lot of these things. And what I'm trying to do is like fit everything into a collab, you know, if they can run on collab, great. But then a lot of these things,

cutting edge things don't fit on a collab. So what do you do? You know, how do you slice and dice it so that they can actually see the results of something and feel, feel it tangibly and feel like they are actually, they can build it. They're learning this. They can connect it, connect the dots with the concepts with the code. So yeah,

I find that as a huge challenge and I only see that getting basically worse with models getting larger and larger. So and I think there are ways, you know, maybe they can play with the API or something, but it's not as exciting. And then you just have to request so many compute credits for a class. It's it's almost prohibitive to a lot of different places. And I think we're really fortunate to be at Stanford and Stanford.

I remember being on call with AWS and they're like, yeah, how many credits do you want? Like whatever. And they're like, if you want a hundred thousand, like you want whatever you want, we just can add it there. And, um, but I don't think that's the case at other universities at all. And so, um, there's going to be this big disparity in how tangible and what kind of projects people can, can do. So.

Yeah, I think obviously there's an increasing gulf between the kinds of results you talk about when talking about current research and what the students can feasibly do in their assignments, because sometimes you have something that honestly kind of barely functions, which is what they trained in the assignment, which isn't great. But on the plus side, I find that, as you said, having students run things themselves can be really useful, especially to deflate hype.

So it's very easy in the lecture to say word vectors, man minus woman equals king minus queen. But then I think it's a good thing to then let the students play around with some word vectors and see how actually very few of those actually work. Yeah, exactly. I feel...

I really love an aspect of CS and AI where you do get to get your hands dirty with concepts and build neural nets and, you know, run reinforcement learning. And you get to understand these things so much better, both like an intuitive level. You know, you kind of get how these things work, not at a theoretical level, but

and also know all the bolts and nuts and bolts, you know, where you kind of get the equations maybe in a rel, but when you actually run something, you're like, oh, okay, that's a Q function, that's a reward, and so on. And you're right that it's challenging now. In the computer vision class, I tried to fit this segmentation model training on a colab and had this issue where it was running out of memory half the time, which was a whole nightmare. So...

Yeah, it's a challenge, especially as deep learning is eating AI. But I guess that's another thing we have to adjust for. Okay. Yeah, I think that was a good conversation. Really good to look back on our teaching experiences, having taught AI a bunch at Stanford.

And with that, we're going to close out. I think we do have some plugs to make as to where you can find these classes and actually check them out yourselves. I think, Abby, you have your one? Yeah. Thank you so much for having me on the podcast. So CS224N, which is NLP with deep learning, is on YouTube. And in fact, the 2021 version, which is now way more up to date than the 29 version I was involved in,

2019 version I was in. You can see that new one 2021 on YouTube now. Wow. And Sharon, yours is on Coursera, right? Yeah, get on Coursera, people. Yeah, you can watch the entire Gantz class there. Is it just the Gantz class? It is actually just the Gantz class.

And with that, thank you so much for listening to this special holiday episode. Maybe not holiday, but very special episode on teaching AI with Abby, Andre and myself, Sharon. Please. I can do it. Smash that like button.

Subscribe to us wherever you get your podcasts. And don't forget to leave us a rating and review if you like the show. Be sure to tune in next week. Damn, I haven't memorized. I didn't think I did.