"I ain't tellin' white folks nuthin": A quantitative exploration of the race-related problem of candour in the WPA slave narratives

From 1936-38, the Works Progress Administration interviewed thousands of former slaves about their life experiences. While these interviews are crucial to understanding the "peculiar institution" from the standpoint of the slave himself, issues relating to bias cloud analyses of these interviews. The problem I investigate is the problem of candour in the WPA slave narratives: it is widely held in the historical community that the strict racial caste system of the Deep South compelled black ex-slaves to tell white interviewers what they thought they wanted to hear, suggesting that there was a significant difference candour depending on whether their interviewer was white or black. In this work, I attempt to quantitatively characterise this race-related problem of candour. Prior work has either been of an impressionistic, qualitative nature, or utilised exceedingly simple quantitative methodology. In contrast, I use more sophisticated statistical methods: in particular word frequency and sentiment analysis and comparative topic modelling with LDA to try and identify differences in the content and sentiment expressed by ex-slaves in front of white interviewers versus black interviewers. While my sentiment analysis methodology was ultimately unsuccessful due to the complexity of the task, my word frequency analysis and comparative topic modelling methods both showed strong evidence that the content expressed in front of white interviewers was different from that of black interviewers. In particular, I found that the ex-slaves spoke much more about unfavourable aspects of slavery like whipping and slave patrollers in front of interviewers of their own race. I hope that my more-sophisticated statistical methodology helps improve the robustness of the argument for the existence of this problem of candour in the slave narratives, which some would seek to deny for revisionist purposes.