Three-Toed Sloth

The "Quality Control" Interview for Big Classes

Attention conservation notice: Advice on teaching, which I no longer follow myself.

I teach a lot of big classes --- the undergraduate advanced data analysis class passed 100 students many years ago, and this fall is over 230 --- which has some predictable consequences. I don't get to talk much to many of the students. They're mostly evaluated by how they do on weekly problem sets (a few of which, in some classes, I call "take-home exams"), and I don't even grade most of their homework, my teaching assistants do. While I try to craft problem sets which make sure the students practice the skills and material I want them to learn, and lead them to understand the ideas I want them to grasp, just looking at their scores doesn't give me a lot of information about how well the homework is actually working for those purposes. Even looking at a sample of what they turn in doesn't get me very far. If I talk to students, though, I can get a much better sense of what they do and do not understand fairly quickly. But there really isn't time to talk to 100 students, or 200.

About ten years ago, now, I decided to apply some of the tools of my discipline to get out of this dilemma, by means of random sampling. Every week, I would randomly select a fixed number of students for interviews. These interviews took no more than 30 minutes each, usually more like 20, and were one-on-one meetings, distinct from regular open office hours. They always opened by me asking them to explain what they did in such-and-such a problem on last week's homework, and went on from there, either through the problem set, or on to other topics as those suggested themselves.

In every class I did this in, it gave me a much better sense of what was working in the problems I was assigning and what wasn't, which topics were actually getting through to students and which were going over their heads, or where they learned to repeat examples mechanically without grasping the principle. There were some things which made the interviews themselves work better:

• Reading each students' homework, before the meeting. (Obvious in retrospect!)
• Handing the student a copy of what they turned in the week before. (Though, as the years went on, many brought their laptops and preferred to bring up their copy of the document there.)
• Putting a firm promise in the syllabus that nothing students said in the interview would hurt their grade. (Too many students were very nervous about it otherwise.)
• Putting an equally firm promise in the syllabus that not coming in to the interview, or blowing it off / being uncooperative, would get them a zero on that homework. (Obvious in retrospect.)
• Offering snacks at the beginning of the interview.

Setting aside a fixed block of time for these interviews didn't actually help me, because students' schedules are too all-over-the-place for that to be useful. (This may differ at other schools.)

Choosing the number of students each week to interview has an obvious trade-off of instructor time vs. information. I used to adjust it so that each student could expect to be picked once per semester, but I always did sampling-with-replacement. In a 15-week semester with 100 students, that comes out to about 3.5 hours of interviews every week, which, back then, I thought well worthwhile.

I gave this up during the pandemic, because trying to do a good interview like this over Zoom is beyond my abilities. I haven't resumed it since we went back to in-person teaching, because I don't have the flexibility in my schedule in any more to make it work. But I think my teaching is worse for not doing this.

Corrupting the Young

The Presentation Exchange for Workshops and Classes

Attention conservation notice: Advice for running an academic workshop, which I've only followed myself a few times.

Some years ago, Henry Farrell and I ran a series of workshops about cooperative problem-solving and collective cognition where we wanted to get people with very different disciplinary backgrounds --- political theorists, computer scientists, physicists, statisticians, cognitive psychologists --- talking to each other productively. We hit upon an idea which worked much better than we had any right to hope. (Whether it's ultimately due to him, or me, or to one of us tossing it out as obviously dumb and the other saying "Actually...", neither of us can now recall.) We've both used it separately a few times in other settings, also with good results. Since we both found ourselves explaining it recently, I thought I'd describe it in a brief note.

1. Every participant in the workshop writes a brief presentation, with enough lead time for the organizers to read them all.
   In the context of an inter-disciplinary workshop, what often works best is to describe an outstanding problem in the field.
2. The workshop organizers semi-randomly assign each participant's presentation to someone else, with enough lead time that the assignee can study the presentation.
   Again, in the interdisciplinary context, the organizers try to make sure that there's some hope of comprehension.
   (While I called this the "presentation exchange", it needn't be a strict swap, where A gets assignd B's presentation and vice versa.)
3. Everyone gives the presentation they were assigned, followed by their own comments on what they found interesting / cool / provocative and what they found incomprehensible. No one gives the presentation they wrote.
   In some contexts, I have found it helpful to institute the rule that the author don't get to speak until after the presentation is finished...

Doing this at the beginning of the workshop helps make sure that everyone has some comprehension of what everyone else is talking about, or at least that mis-apprehensions or failures to communicate are laid bare. It can help break up the inevitable disciplinary/personal cliques. It can, and has, spark actual collaborations across disciplines. And, finally, many people report that knowing their presentation is going to be given by someone else forces them to write with unusual clarity and awareness of their own expert blind-spots.

As I said, Henry and I hit on this for interdisciplinary workshops, but I've also used it for disciplinary workshops --- because every discipline is a fractal (or lattice) of sub-sub-...-sub-disciplinary specialization. I've also used it for student project classes, at both the undergrad and graduate level. That requires more hand-holding and/or pastoral care on the part of the teacher than a research workshop, and I've never tried to make it the way I start a class.

Learned Folly; The Collective Use and Evolution of Concepts; Corrupting the Young