Friday, July 16, 2010

RIP: James (Jim) Penning

Political scientists have lost one of their own this week with the passing of Professor Jim Penning of Calvin College. I had known Jim for 11 years. We first met at the American Political Science Association's Annual Conference in 1999. With our common research interests (religion and politics, American politics, and elections), we often found ourselves attending the same panels. Our common interests brought our paths together many times over the years. For a short sample of his work, read Jim's September, 2008 blog post for Religion and Ethics Newsweekly on the Democratic Party's religious outreach efforts. 

The last time I saw Jim was also at the APSAs Annual Conference last August in Toronto. I chaired a panel for which Jim was presenting his research on the God gap in the 2008 elections with several of his co-authors. This research is also in a recent book published by Oxford University Press, The Disappearing God Gap? It was a pleasure to know Jim. He always greeted me with a warm smile and, with genuineness, would inquire about the goings-on in my life. He will be missed.

Monday, July 5, 2010

Are student evaluations an appropriate measure of teaching performance?

This question has been raised (again) by two recent news stories. The first was a study published in the Journal of Political Economy, by Scott E. Carrell and James E. West. Carrell and West used data from the USAF Academy, where students are randomly assigned to required courses. To measure how well professors taught students in those required courses, the study looked at student performance in higher level classes that required those courses as a prerequisite. They refer to this as a "value-added measure." (How much value was added to the students ability to learn, in this case, mathematics?) The value-added measure was compared to student evaluations. If student evaluations are a good way to measure teaching performance, then students would give high marks to the teachers who prepared them well for the higher level courses.

Carrell and West discovered, however, that just the opposite is true. Students thought that the best teachers were the ones who did the worst at preparing them for the higher level classes. Hence, Carrell and West conclude, "students appear to reward higher grades in the introductory course,  but punish professors who increase deep learning."

In the second story, the Texas educational system has taken the notion that students are effective evaluators of teacher performance to a new extreme by letting students decide which professors get teaching rewards of up to $10,000. Dr. Stanley Fish railed against this scheme in a recent column for The New York Times. Fish writes,
Once this gets going (and Texas A&M is already pushing it), you can expect professors to advertise: “Come to my college, sign up for my class, and I can guarantee you a fun-filled time and you won’t have to break a sweat.” If there ever was a recipe for non-risk-taking, entirely formulaic, dumbed-down teaching, this is it.
Part of the problem of using students to evaluate teaching, Fish argues, has to do with "deferred judgment."  Time, sometimes years, if often required to fully understand what you have learned from a class, especially a class well taught.
And that is why student evaluations (against which I have inveighed since I first saw them in the ’60s) are all wrong as a way of assessing teaching performance: they measure present satisfaction in relation to a set of expectations that may have little to do with the deep efficacy of learning. Students tend to like everything neatly laid out; they want to know exactly where they are; they don’t welcome the introduction of multiple perspectives, especially when no master perspective reconciles them; they want the answers.

But sometimes (although not always) effective teaching involves the deliberate inducing of confusion, the withholding of clarity, the refusal to provide answers; sometimes a class or an entire semester is spent being taken down various garden paths leading to dead ends that require inquiry to begin all over again, with the same discombobulating result; sometimes your expectations have been systematically disappointed. And sometimes that disappointment, while extremely annoying at the moment, is the sign that you’ve just been the beneficiary of a great course, although you may not realize it for decades.
Needless to say, that kind of teaching is unlikely to receive high marks on a questionnaire that rewards the linear delivery of information and penalizes a pedagogy that probes, discomforts and fails to provide closure. Student evaluations, by their very nature, can only recognize, and by recognizing encourage, assembly-line teaching that delivers a nicely packaged product that can be assessed as easily and immediately as one assesses the quality of a hamburger.
The problems associated with student evaluations have been know for decades. In the many studies I have seen on the topic, not one has shown student evaluations to be an effective measure of teaching performance. (If you find one, please let me know.) Nonetheless, student evaluations are used as the main source, or the only source, of information to evaluate teaching performance. This incentive structure is known to lead to teaching that tries to be entertaining, lightening of the student's work load, and grade inflation (which has continued to rise since the introduction of student evaluations in the 1960s).

In one now well known experiment, two people were asked to teach the same class on the same topic. Both lectured on the topic and had time at the end for some Q&A with the students. One was a professor and expert on the topic. The other was an actor. The actor was charming, charismatic, funny, and bluffed his way through the whole lecture and Q&A. (You can probably guess where this is going.) The students by far thought the actor was the better teacher, and (here's the kicker) more knowledgeable on the topic.
 
The obvious question then becomes, if we know that student evaluations are poor measures of teaching performance, and have known this for decades, why do colleges continue to use them to measure teaching performance? The answer, I believe, has to do with a problem common in the social sciences--the data that is most likely to be used is the data that is easiest to collect. Deans and hiring and promotion committees rely upon this data because that is the data they have. They look at all those means and standard errors nicely laid out in those neat little tables and it becomes easy to assume those numbers mean something. The situation reminds me of a joke told by economist Ken Rogoff to explain why economists failed to foresee the current economic crisis. 
A drunk on his way home from a bar one night realizes that he has dropped his keys. He gets down on his hands and knees and starts groping around beneath a lamppost. A policeman asks what he’s doing.
“I lost my keys in the park,” says the drunk.
“Then why are you looking for them under the lamppost?” asks the puzzled cop.
“Because,” says the drunk, “that’s where the light is.”
In the same way that economists failed to understand the economic collapse because they failed to collect the right data, college administrators fail to effectively evaluate teaching because they are collecting the wrong data. 

Related Articles:

Full version of the Carrell and West study in PDF.
Cowen, Tyler. "Does professor quality matter?"
Fish, Stanley. "Student Evaluations, Part Two"
Douthat, Ross. "In Defense of Student Evaluations."
Douthat, Ross. "Now, The Case Against Student Evaluations."
Jacobs, Alan. "Stanley Fish is right again."

Sunday, July 4, 2010

Independence Day: Independence From Whom?


Marist College recently conducted a poll of 1,004 US residents appropriate for this Independence Day. They asked, from which nation did we declare our independence in 1776? The results were disturbing. About 1 in 4 (23-29%) of Americans either do not know, or will give the wrong answer. Even more disturbing, among those age 18-29, the results are significantly higher: 37-43%. 

Why is this happening? I don't know much about what is being taught in high schools these days, so I don't have an answer to that. David Brooks and Mark Shields, however, responded in their weekly analysis on The Newshour this Friday. They said,
DAVID BROOKS: To me, the substance of it is that we have traded history for social studies in schools, that we don't do the ABC, here is what happened when.
And I notice this when I talk to kids, including sometimes my own kids. They just don't get the dates. They don't get the dates. They don't have the scaffolding of history. And they do a lot more social structure. They do cultures. They do this. They do that.
But they don't have the basic -- the facts and lineage of what happened when. And, so, those basic facts if, you don't have the scaffolding, you are not going to remember. You're not going to know how to organize it and put it all together into some sort of theory.
MARK SHIELDS: We're a lot more sensitive, but we're a lot less informed.
What is rather terrifying is the figure you cited about 40 percent of the people under the age of 29. And 80 percent, close to, over -- those over 45 do know. There was something going on in schools. The people, the older people are less likely to have gone to college than are the younger ones.
And the idea that somebody is going through college and graduating and not knowing a fundamental fact like that is terrifying and it's depressing.
For more opinion on the poll, check out Steven L. Taylor's post at Outside the Beltway.

What do you think?  Are you disturbed by these results?  Why have we done a poor job teaching fundamental facts of our nation's history?

Saturday, July 3, 2010

SCOTUSblog: Everything you read about the Supreme Court is wrong

Tom Goldstein, writing for SCOTUSblog, makes a compelling argument that most of the conventional wisdom we here about the Supreme Court is wrong. The common assumptions Goldstein addresses are:
  • The Court is easily categorized along liberal (Breyer, Stevens, Sotomayor, Ginsburg) and conservative (Roberts, Alito, Scalia, Thomas) lines, with one swing vote (Kennedy).
  • Big cases are decided by 5-4 majorities along ideological lines.
  • The Court is moving in a decidedly conservative direction.
  • The Court is pro-business.
  • Scalia and Thomas are the "arch-conservatives," providing extremely right-wing opinions, especially against criminal defendants. 
  • The liberal wing of the court is "activist," showing little deference to the elected branches and overturning precedent. 
  • The liberal wing "coddles criminals."
Goldstein provides examples of cases from this term where each of these assumptions held true, but the Court decided about 90 cases this term and one case does not make a trend. Fewer than 20% were decided by a 5-4 vote and slightly more than 10% were decided along ideological lines. About half the decisions were unanimous (9-0).

Goldstein takes apart each of the other assumptions as well, citing a myriad of cases. His reflections on Scalia and Thomas are particularly interesting. While conservatives are often perceived as being more supportive of criminal prosecutors and less supportive of defendants, Goldstein argues that Scalia and Thomas have taken the side of defendants more than any of the other seven justices.

Also, the conservative wing has been more "activist," or willing to overturn Congress and precedent than the liberal wing. This makes sense, according to Goldstein, because as the court changes it will want to undo the decisions of prior courts, which would be viewed as bad decisions. When the court became more liberal in the 1960s-70s, for instance, it was more willing to overturn prior decisions made my more conservative courts. Now that the court has moved in a more conservative direction, it is more willing to overturn those decisions made by prior liberal courts.

Read the whole article here.