Three-Toed Sloth

36-402, Spring 2012: Self-Assessment and Lessons Learned (Advanced Data Analysis from an Elementary Point of View)

Attention conservation notice: Complacent navel-gazing about teaching statistics to undergraduates at a very un-representative university. Written back in May, not posted due to laziness.

Overall, I think this went tolerably but not perfectly. The content was improved and is near (but not at) a local optimum. I've also got a better sense than last year of how much they're learning, and I think the homework is helping them learn. But the course could definitely do more to move them towards doing data analysis, as opposed to answering homework questions.

It was a substantially bigger class than last time (88 students vs. 63), and this led to some real issues. The lecture room was fine, but office hours were over-crowded, the stream of e-mail from student seemed unending, and, worst of all, there simply wasn't enough TA time available for grading. (88 weekly assignments of serious data analysis is a lot to grade.) Everything did get graded eventually, but not as fast as it should have been. From the number of students registered for the fall course in regression which is ADA's pre-requisite, I can expect about as many again in 2013, or a few less. This is about twice as many students as I'd like to see in it.

Speaking of grading, I had to completely re-do the assignments, since solutions from last year were circulating¹ --- and I did not put the new solutions on-line. This was a big chunk of work I was not anticipating, but at least now I have two sets of problems for the future.

In terms of actual content, I cut nothing major, but, by eliminating exam review sessions, etc., squeezed in some extra stuff: a reminder lecture on multivariate distributions, smooth tests and relative densities, time series, and more on estimating causal effects. I am relatively happy with the over-all content, though less so with the organization. In particular I wonder about devoting the very first part of the course to regression with one input variable, as a really elementary way to introduce fundamental concepts (smoothing, kernels, splines, generalization, cross-validation, bootstrap, specification checking), and only then go to regression with multiple inputs. I worry, however, that this would be coddling them.

The "quality control samples" were eye-opening. Some students found them nerve-wracking, which was counterproductive; but lots of them also gave every sign of being candid. What I found most surprising was how many of them (a third? more?) were taking ADA because they had been funneled into it by major other than statistics, often one of CMU's several larval-quant programs. (I'm not sure how those took my plugging Red Plenty in class while explaining Lagrange multipliers.) Their self-de-motivating question was not "Is this going to be on the exam?", but rather, "Is this going to come up in interviews?". If I were a better teacher I'd work more at reaching these students.

Week-to-week, the quality control samples were very helpful for figuring out what students actually grasped, and, even more, how their thinking went astray (when it did). In particular, there turned out to be much more variation than I'd anticipated in readiness to apply material they had already learned in a new context — the "but that's from chapter 7 and this is chapter 12" problem, or even the "but that's from introduction to statistical inference, or matrices and linear transformations, which I took a year ago" problem. I am not sure if I should provide more cues, or if, again, that would be coddling.

These facts about motivation and preparation don't please me, exactly, but at least for right now these seem like constraints I have to (and can) work within.

I will certainly use the quality-control samples again in 2013, but I may take fewer each week, and make it clearer to the students that they're not being graded.

Some specifics:

• I need to do more to set expectations at the beginning, especially in light of why the students are taking the class. Namely: ADA is a course in statistical methodology; it will cover models (e.g., kernel regression), methods (e.g., bootstrap), and knowing when to use which tools. It is not a research-projects course. (That's 36-490.) It is not, as one (good) student said he'd expected, "linear models plus programming". It is not a "get a job in a data startup" or "get a job as a quant" course. (It may be a data science course, since I continue to think that "data science" means "computational applied statistics", pace Cathy.)
• There's a definite trade-off between having the students gain understanding by building their own tools, and letting them do more data analysis by using pre-built tools. I think most of them now get bootstrapping intuitvely, because I didn't let them use boot, and kept drilling them on it. I am less confident as to whether they grasp, say, factor analysis, or even additive models, at the same level.
• Relatedly, and very importantly, all the assignments probably give the students too much "scaffolding", that is, guide them far too much as to exactly what analysis to do, with what steps in what order. But deciding on these things is a key part of doing good data analysis. As time allows, re-do the assignments so that they leave more and more to student initiative as the class goes on.
• Making The R Cookbook required was a good idea. Things would have been better still if everyone had taken some programming, if not necessarily statistical computing. (I suppress here my usual rant about how, if you are taking a class called "Advanced Data Analysis" in 2012, it is really not unreasonable to expect you to write some code.) Look, down the road, into making this a pre-requisite.
• pcalg proved too hard for many students to install, or rather Rgraphviz did. If I had unlimited time I'd write my own R interface to graphviz, which wouldn't choke if you installed the latter with a different version number.
  (If I had really unlimited time, I'd re-write pcalg so that, as Peter and Clark suggested originally, you could give it a partial ordering of the variables, and force the graph to be compatible with that. [Currently you can say that some undirected edges must be present, and others must be absent, which is not at all the same.] This might make a suitable project for one group of The Kids in statistical computing in the fall, whereas hacking at Rgraphviz is probably too much. [In the event, I did not give them that project.])
• Having a required textbook is no longer useful. Make it clear that the notes take the place of a textbook, and that they're expected to study them — and not just the figure captions. (But "police the reading" homework wastes time they could spend doing more important stuff.) Consider keeping Faraway as a recommended but optional text, or perhaps replacing it with Wood's Generalized Additive Models.
• Introduce some simple simulations very early on, before the main chapters on simulation and, especially, before the chapter on the bootstrap. Move the main simulation chapter back before bootstrapping?
• Do not call a test statistic t unless it is really a t-test. (This led to a lot of confusion when doing specification testing.)
• More practice in finding confidence intervals for differences directly. ("The difference between 'significant' and 'not significant' is not statistically significant.")
• More practice in answering "what if?" questions from non-linear regression models, including error bars on the what-ifs, before getting to causal inference.
• Say more about measurement models in the context of graphical models and especially causality.
• Enough students have taken econometrics that more explicit de-programming about instrumental variables is called for.
• I gave a smaller proportion of A's than last year, but also failed absolutely fewer students. Some who had struggled during the semester managed to pull it together for the final, which was very gratifying.
• Shorten the final and make it due earlier; the rush to get in grades in time for graduating seniors was Not Good.
• Coursekit proved to be vastly less annoying to use than Blackboard. That is, it was not actively painful to perform such obscure operations as "enter grades" or "send e-mail to all students" or "link to lecture notes". It's a free service, so I am sure that it's all part of some nefarious plan ("if you're not paying for something, you're not the customer; you're the product being sold"), but at this point I really don't care, and intend to keep using it. [Update, November 2012: Coursekit, now Lore, got "upgraded" over the summer to a level of dysfunctionality that equals or even exceeds Blackboard's. I will thus switch back to the user-hostile software which doesn't obviously have exploitative intellectual-property or social-marketing designs on my students.]

Finally, if students again complain that the class makes it harder for them to take seriously what they hear in economics or psychology or the like, remember not to laugh maniacally, or use expressions like "Victory is mine!" Rather, maintain a sober, thoughtful, benevolent, and above all sane expression, and talk solemnly about the great prospects for using good statistical methods to solve real-world problems.

[1]: One advantage to putting bad jokes in your solutions is that it becomes pretty obvious when someone is parroting them back to you.

Advanced Data Analysis from an Elementary Point of View