For regular attendees of our monthly sHAI meetings, my criticism of the original studies that formed the CASA Theory is well-known. And I am therefore really proud to finally put some data where my mouth is, and have published a direct replication of the 1994 politeness study. And in Nature Scientific Reports to boost! However, getting this publication out there has been a much longer journey than most people realize and, I think, includes some academic-life lessons that I wanted to share.
Where it all began
I originally came across CASA and Clifford Nass in the first year of my PhD (2013). I did my PhD at a psycholinguistics research institute (not a university) surrounded by people doing fundamental language learning / development / evolution studies. My PhD was open-ended: They had a VR lab that no one used and I should use it. So my first experiment was to see if we could study language not by inviting two participants (logistical nightmare) but by having one participant interact with a computer. The pros are obvious: you can have the computer talk however you want.
Are you interested in the role that ‘uhms’ have in conveying meaning? Systematically manipulate that with a computer.
Are you curious what a longer pause would do to the flow of a conversation? No problem, the computer can do that. You can experiment with specific pause lengths; 250 ms? 300ms? Another human can't.
And, of course, CASA suggested that conclusions we draw from human-human interaction should be applicable to human-computer interaction as well. My first couple of experiments showed this (comparing HHI and HCI).
But in preparation for my PhD defense 3 years later, I dove further into the studies that underly CASA, looking at the methodology and I realized that it was a bit fishy. So much attention for a chronically underpowered study (10 participants per condition; power of 0.4). There were no direct replication studies, only indirect, looking at websites or voice assistants. Not desktop computers.
Without money it's 'just' a (very good) idea
I came across the NWO Replication Grant and I decided to go for it. I was invited for an interview. The day itself fell in the week that I discovered I was pregnant. The day of the interview, I had horrible morning sickness and would need to run to the bathroom every 10 – 15 minutes. I wasn't comfortable, I wasn't reactive, my interview did not leave an impression and I wasn't awarded the grant. I had learned from my mistakes and kept my eyes out for a new round, but it wasn't announced.
In preparation for the grant, I had read more about CASA. I got my hands on the Media Equation book (Reeves & Nass, 1997) and read it, covered it in post-it notes, took photos, and did specific searches in the book. I did a literature review on all 200+ articles that cite the original CASA study, specifically looking for replication studies. What were they like? How did they do them?
I had so many opinions that I decided to put my discoveries into an opinion paper. If I can't find the money to do the replication, maybe I can make some noise. I played around with different approaches to why we need to replicate the CASA studies. People did not respond well to the strategy of highlighting how underpowered the original studies were; in some cases, how badly designed.
Okay, then let's focus on why we should care: What would it mean if CASA didn't replicate? Then, I come across all the opinion papers that were published about CASA recently: Lombard, Fortunati, Gambino, etc. What set my issue apart? Was it part of the same puzzle they were writing about, or was it different? How could I express that in a way that people understood?
I had colleagues read and review my opinion paper and I submitted it to Human-Machine Communication (the same journal that published Lombard, Fortunati, and Gambino). I got a very long review back, which helped identify which arguments were not well received, but it boiled down to: without data we can't help you. A strong opinion, supported by a thorough literature review, wasn't going to get me anywhere.
I adjusted my arguments and practiced it in lecture halls, at workshops, at summer schools. Which arguments got the students excited? Which arguments did the students understand, get discussions going? Those were the arguments I needed. That, and money.
I tried a VENI application, that didn't even get past the first round. So, I decided to teach a summer school course. One intense week for €5,000. Not a lot, but significantly more than I had. I had pitched an idea to the Radboud Summer School course repeatedly since 2019 (Introduction to Human-Computer Interaction), but I never had enough students enroll for the course to happen. In 2021, I finally did.
And so, in the first week of my pregnancy leave (not the same baby as the replication grant interview – different baby) I gave an online course. It was August 2021 so everything was 'thankfully' online. And on the last day, after giving my fine-tuned arguments about CASA, I proudly told the students that I was going to use their money to replicate it.
Believing so much in your idea that you make it work no matter what
Now I could finally do it. I used the money to fund a team, because we needed A LOT of participants. My theory had always been that it would not replicate, so I needed a lot of participants to prevent the “under-powered” argument. This is how I met Leo Block Santos, whose story will be told a different time.
The actual data-collecting part of the study, the analysis, the writing up, all went as they normally do. I wrote the paper during a Writing Week, and submitted it to Science, to Nature, to Nature Robotics, Nature Human Behaviour. I got really nice emails back, telling me how relevant and timely my article was, but they just had so many submissions that this one wasn't special enough. But I was clearly on the right track. If I wanted this paper to make noise (and I believe it will), it needed to be in a big journal. Arguments against it be damned. Nature Scientific Reports was my last attempt before I was going to lower my standards. And here it is: https://doi.org/10.1038/s41598-023-46527-9
This journey was the beginning of the beginning, because I have many more things up my sleeve. But it was a journey of commitment, of perseverance, of believing so much in your idea that you make it work no matter what.