36-402, Spring 2012: Self-Assessment and Lessons Learned (Advanced Data Analysis from an Elementary Point of View)
Attention conservation notice:
Complacent navel-gazing about teaching statistics to undergraduates at a very
un-representative university. Written back in May, not posted due to
Overall, I think this went tolerably but not perfectly. The content was
improved and is near (but not at) a local optimum. I've also got a better
sense than last year of how much they're learning, and I think the homework is
helping them learn. But the course could definitely do more to move them
towards doing data analysis, as opposed to answering homework questions.
It was a substantially bigger class than last time (88 students vs. 63), and
this led to some real issues. The lecture room was fine, but office hours were
over-crowded, the stream of e-mail from student seemed unending, and, worst of
all, there simply wasn't enough TA time available for grading. (88 weekly
assignments of serious data analysis is a lot to grade.) Everything did get
graded eventually, but not as fast as it should have been. From the number of
students registered for the fall course in regression which is ADA's
pre-requisite, I can expect about as many again in 2013, or a few less. This
is about twice as many students as I'd like to see in it.
Speaking of grading, I had to completely re-do the assignments, since
solutions from last year were circulating1 --- and I did not put the
new solutions on-line. This was a big chunk of work I was not anticipating,
but at least now I have two sets of problems for the future.
In terms of actual content, I cut nothing major, but, by eliminating exam
review sessions, etc., squeezed in some extra stuff:
a reminder lecture on multivariate distributions,
smooth tests and relative
densities, time series, and more
on estimating causal effects. I am relatively happy with the over-all
content, though less so with the organization. In particular I wonder about
devoting the very first part of the course to regression with one input
variable, as a really elementary way to introduce fundamental concepts
(smoothing, kernels, splines, generalization, cross-validation, bootstrap,
specification checking), and only then go to regression with multiple inputs.
I worry, however, that this would be coddling them.
The "quality control samples" were eye-opening. Some students found them
nerve-wracking, which was counterproductive; but lots of them also gave every
sign of being candid. What I found most surprising was how many of them (a
third? more?) were taking ADA because they had been funneled into it by
major other than statistics, often one of CMU's several larval-quant
programs. (I'm not sure how those took my
plugging Red Plenty in class while
explaining Lagrange multipliers.) Their self-de-motivating question was not
"Is this going to be on the exam?", but rather, "Is this going to come up in
interviews?". If I were a better teacher I'd work more at reaching these
Week-to-week, the quality control samples were very helpful for figuring out
what students actually grasped, and, even more, how their thinking went astray
(when it did). In particular, there turned out to be much more variation than
I'd anticipated in readiness to apply material they had already learned in a
new context — the "but that's from chapter 7 and this is chapter 12"
problem, or even the "but that's from introduction to statistical inference, or
matrices and linear transformations, which I took a year ago" problem.
I am not sure if I should provide more cues, or if, again, that would be
These facts about motivation and preparation don't please me, exactly, but
at least for right now these seem like constraints I have to (and can) work
I will certainly use the quality-control samples again in 2013, but I may
take fewer each week, and make it clearer to the students that they're not
- I need to do more to set expectations at the beginning, especially in
light of why the students are taking the class. Namely: ADA is a course
in statistical methodology; it will cover models (e.g., kernel
regression), methods (e.g., bootstrap), and knowing when to use which tools.
It is not a research-projects course.
(That's 36-490.) It is
not, as one (good) student said he'd expected, "linear models plus
programming". It is not a "get a job in a data startup" or "get a job as a
quant" course. (It may be a data science course, since
I continue to think that "data
science" means "computational applied
- There's a definite trade-off between having the students gain
understanding by building their own tools, and letting them do more data
analysis by using pre-built tools. I think most of them now get
bootstrapping intuitvely, because I didn't let them use boot, and kept
drilling them on it. I am less confident as to whether they grasp, say, factor
analysis, or even additive models, at the same level.
- Relatedly, and very importantly, all the assignments probably give the
students too much "scaffolding", that is, guide them far too much as to exactly
what analysis to do, with what steps in what order. But deciding on these
things is a key part of doing good data analysis. As time allows, re-do the
assignments so that they leave more and more to student initiative as the class
- Making The R Cookbook required was a good idea. Things would
have been better still if everyone had taken some programming, if not
necessarily statistical computing. (I suppress
here my usual rant about how, if you are taking a class called "Advanced Data
Analysis" in 2012, it is really not unreasonable to expect you to write some
code.) Look, down the road, into making this a pre-requisite.
- pcalg proved too hard for many students to
did. If I had unlimited time I'd write my own R interface
to graphviz, which wouldn't
choke if you installed the latter with a different version number.
(If I had really unlimited time, I'd
re-write pcalg so
suggested originally, you could give it a partial ordering of the variables,
and force the graph to be compatible with that. [Currently you can say that
some undirected edges must be present, and others must be absent, which is not
at all the same.] This might make a suitable project for one group of The Kids
in statistical computing in the fall, whereas
hacking at Rgraphviz is probably too much. [In the event, I did not
give them that project.])
- Having a required textbook is no longer useful. Make it clear
that the notes take the
place of a textbook, and that they're expected to study them — and
not just the figure captions. (But "police the reading" homework wastes time
they could spend doing more important stuff.) Consider
keeping Faraway as a
recommended but optional text, or perhaps replacing it with
- Introduce some simple simulations very early on, before the main chapters
on simulation and, especially, before the chapter on the bootstrap. Move
the main simulation chapter back before bootstrapping?
- Do not call a test statistic t unless it is really a t-test.
(This led to a lot of confusion when doing specification
- More practice in finding confidence intervals for differences directly.
difference between 'significant' and 'not significant' is not statistically
- More practice in answering "what if?" questions from non-linear regression
models, including error bars on the what-ifs, before getting to causal
- Say more about measurement models in the context of graphical models and
- Enough students have taken econometrics that more explicit de-programming
about instrumental variables is called for.
- I gave a smaller proportion of A's than last year, but also failed
absolutely fewer students. Some who had struggled during the semester managed
to pull it together for the final, which was very gratifying.
- Shorten the final and make it due earlier; the rush to get in grades in
time for graduating seniors was Not Good.
- Coursekit proved to be vastly less annoying to use than
Blackboard. That is, it was not actively painful to perform such obscure
operations as "enter grades" or "send e-mail to all students" or "link to
lecture notes". It's a free service, so I am sure that it's all part of some
you're not paying for something, you're not the customer; you're the product
being sold"), but at this point I really don't care, and intend to keep
using it. [Update, November 2012: Coursekit,
got "upgraded" over the summer to a level of dysfunctionality that equals or
even exceeds Blackboard's. I will thus switch back to the user-hostile
software which doesn't obviously have exploitative
intellectual-property or social-marketing designs on my students.]
Finally, if students again complain that the class makes it harder for them
to take seriously what they hear in economics or psychology or the like,
remember not to laugh maniacally, or use expressions like "Victory is mine!"
Rather, maintain a sober, thoughtful, benevolent, and above all sane
expression, and talk solemnly about the great prospects for using good
statistical methods to solve real-world problems.
: One advantage to putting bad jokes in your
solutions is that it becomes pretty obvious when someone is parroting them back
Advanced Data Analysis from an Elementary Point of View