Assistant Professor of Government Charles Crabtree teaches a course on experiments in the Program in Quantitative Social Science at Dartmouth. Along with other faculty members at the college, he was recently asked to provide some advice to the graduating members of the Class of 2021. Here is an interview with Prof. Crabtree, describing his course, his pandemic year, and the origins of his words of advice.
QSS: Before we talk about your six words of advice, please tell us a bit about yourself. You have been at Dartmouth for one year, and it turned out to be a pandemic year. In brief, how did things go? What did you learn?
CC: I think that things went very well. While I missed the serendipitous, unscheduled interactions that so enrich university life, I enjoyed the many opportunities I had to meet over Zoom with Dartmouth students and colleagues. This was my first year teaching undergraduates, and I really enjoyed it - though I wish that the classes would have been in-person.
QSS: In Winter 2021, you taught a section of QSS 30.03, Experiments in Politics. What is this class about and what sort of experiment did this class conduct?
CC: I love this course, which I modeled after Professor Mia Costa and Brendan Nyhan's awesome classes. The idea behind the course is that students and I work together to design, implement, and analyze and write up the results from a novel survey experiment. The long-term goals is to eventually publish the paper we write together in class as an article in a paper-reviewed journal.
My Winter 2021 class focused on the important topic of anti-Asian discrimination. Our research question was: To what extent do Americans racially discriminate against healthcare workers? While a large literature shows that racial biases pervade the American healthcare system, there has been no systematic examination of these biases in terms of who patients select for medical treatment. To examine this, we conducted a type of survey experiment, called a conjoint experiment, with a national sample of 1,498 Americans. In the experiment, survey respondents were asked to evaluate several pairs of doctors. For each pair, we asked them to select the doctor that they would prefer to see. We randomized several attributes of these doctors, including their race/ethnicity.
QSS: What did you find? Are you and the students in your QSS course working on a paper?
CC: Surprisingly, we found that while racial inequalities mark nearly every aspect of the American healthcare system, our respondents did not, on average, discriminate against Asian or other racial minority doctors. We also find no consistent evidence of treatment effect heterogeneity; Americans of all types appear not to care about the racial identity of their doctor in this context. This finding has broad implications for the potential limits of American prejudice, we think.
QSS: Let's now turn to your six words of advice. To start, what were they?
CC: Don't select on the dependent variable.
QSS: Thanks. Can you tell us what that means? What does "select" mean and what is a "dependent variable?"
CC: A key decision that scientists face is what cases, or observations, to look at, or select. Before selecting their cases, scientists normally have some research question in mind. Most research questions take the form of 'What effect does X have on Y?' Y is the dependent variable, or the outcome, that scientists want to understand.
Putting that all together, the quote I offered means that scientists - and people more generally - shouldn't try to understand variation in some outcome, such as how well students do on tests, by only looking at cases that have a certain type of outcome, such as students that get As.
QSS: Thanks for that explanation. Let's talk about your advice a bit more. Why did you choose your six words? Did you have a particular lesson in mind? Have you seen people select on the dependent variable and run into problems because of that?
CC: I choose those six words because, when taken seriously, they suggest that we all should be somewhat wary of only following advice from successful people. We don't know if unsuccessful people did the same things or not.
People often select on the select on the dependent variable and it can cause all sorts of problems. For example, most sports analysis tries to understand what players and teams will succeed in the future by only looking at players and teams that have succeeded in the past.
QSS: That's really intersting to hear. It sounds like you took your "six words of advice" very seriously and we hope that students will your advice. Thank you for spending on this with us, and I hope you have a nice start of the summer.