Quantitative Methodologies and Perpetuating Inequity: The Struggle for Equitable Empirical Research
Introduction
In his 2022 speech at Princeton University, professor of Computer Science Narayanan asserts that: “currently quantitative methods are primarily used to justify the status quo. I would argue that they do more harm than good” (Narayanan 2022, 25)
At the heart of Naryanan’s position is his belief “that baked into the practice of quantitative methods is a worldview that sees the status quo as unproblematic” (2022, 3). He explains that qualitative methods tend to be overly simplistic and subjective and describes several ways in which qualitative research is limited in its ability to study discrimination (2022, 3, 4, 16). The following paper will outline Narayanan’s argument and contextualize quantitative research within a larger body of literature. Ultimately, it will provide an alternative framework to consider quantitative research, and suggest more equitable approaches to its methodology.
The Quantitative Approach and Discrimination
Limits of the Quantitative Process
As Narayanan describes, one of the most basic and important ways that quantitative methods are limited is that the null hypothesis for quantitative studies is almost always the absence of discrimination (2022, 7). Beginning a study by assuming the absence of discrimination in a given system is dangerous, as it creates a perpetual uphill battle for activists or people claiming discrimination’s presence, giving them “the burden of proving again and again and again that it exists” (Narayanan 2022, 8).
Narayanan also outlines other ways that quantitative methods can mislead audiences and hide inequalities. Quantitative researchers are limited by the datasets they rely on, which are often collected at a single point in time by a single, potentially biased source. Narayanan explains that the numbers we rely on are not neutral: in fact, data is political (2022, 8). Quantitative researchers also tend to blame discrimination on a suite of other neural-presenting factors that together simply indicate race. Instead of acknowledging structural inequalities, many “clever” researchers will search for some “omitted variable” (Narayanan 2022, 13).
Scholarly Support
A wide range of literature supports Narayanan’s critiques of quantitative methods and contextualizes his claims.
Dr. Arellano, for example, examines the history of quantitative methods and the ways they are integrated in higher education Arellano (2022). She explains that the origin of quantitative methods was to justify the eugenics movement, formalizing White supremacy over other races using numeric evidence -Arellano (2022). Quantitative methods originated as a way to entrench power structures, and this goal, while less explicit, pervades today. Many studies, for example, create models trained on powerful groups (typically wealthy white males), and declare that disparities between this group and minorities is a deficit or a measure of poor performance, which then perpetuates a White normative perspective Arellano (2022).
Despite its tendency to reinforce social hierarchies, research methodology is ubiquitous in education, especially at the collegiate level. Students carry the methods they learn in college with them to their own research, PHDs, and jobs Arellano (2022). Unfortunately, quantitative research in higher education is often presented without critical consideration of the data’s origin. It builds upon the dominant narrative and further marginalizes disadvantaged communities. Even the process of publishing peer-reviewed journal articles upholds hegemonic discourses, inequity, oppression and White supremacy -Arellano (2022). Arellano maintains that quantitative research in higher education is “riddled with subjectivity and biases,” and the whole higher education system must be redeveloped, this time with a focus on inclusion, equity, and anti-rascist critique of the status quo -Arellano (2022).
Ahmad et all and Buolamwini&Gebru discuss the danger of homogeneity in quantitative research. A lack of diversity in the group of people collecting data and testing models is a primary barrier to fair and unbiased methodology. In an article in Frontiers in Psychology, Ahmad et al argue that in order to foster diversity in higher education, “undergraduates from different backgrounds must engage in high quality research” -Ahmad et al. (2019). The authors describe how institutional biases can prevent minority students from participating in research within universities. On a national scale, this results in research continually being produced, approved, and reproduced by white students.
Further, Buolamwini&Gebru illustrate the consequences of a lack of diverse representation in quantitative research. The authors create a model and use it to evaluate three commercial gender classification systems, finding that darker-skinned females were the most misclassified group (2018, 1). This finding points to exclusion and bias in the training methodology, as well as within the institutions that create, approve, and adopt such models (2018, 12). Their study highlights underlying power structures, the importance of diverse representation and intersectionality in quantitative analyses.
A More Just Quantitative Approach
Benefits of Quantitative Methods
While Narayanan critiques quantitative methods, he avoids renouncing them altogether. He calls on his audience to use empirical methods differently, in such a way as to minimize their potential to mislead (2022, 3). He highlights the importance of choosing datasets with discretion and critical thought, suggesting following an intentional process of curating and evaluating data (2022, 21). He argues for the incorporation of descriptive elements as a means to describe and support quantitative results, and for highlighting individual experiences (2022, 22, 23). Most importantly, he urges his audience to acknowledge and think about systemic racism and inequity, and to view statistics as symptoms of these underlying structures (2022, 24).
In his speech, Narayanan mentions Ann Oakley, a British sociologist and feminist, as a defender of the quantitative process (2022, 4). In her 1998 article, Oakley argues for “rehabilitating quantitative methods and integrating a range of methods in the task of creating an emancipatory social science” (1998, 707). She writes that quantitative research is widely perceived to defend the masculinist status quo, and qualitative research instead furthers a more feminist perspective (1998, 707). Ultimately, Oakley problematizes these conceptions. Oakley explains that on the one hand, quantitative analysis is inherently anti-feminine. It is a means to predict and control, “ideologically linked with men’s desire to dominate” (1998, 711). However, she ultimately praises the quantitative method, noting that society would have no understanding of women’s oppression without quantitative analysis that situates their situation “vis-a-vis” that of men (1998, 723). Oakley invests power in the ability of statistics to illustrate intersectionality, and the “ways in which gender, class and ethnicity intersect as axes of discrimination,” citing efforts such as those of The Support Stockings, a group of women in Sweden who utilized gendered statistics to draw attention to women’s (disadvantaged) position (1998, 732). Similarly to Narayanan, Oakley ultimately argues that the quantitative method can be “altered without abandoning the basic methodological techniques themselves” (1998, 723). Although scholars such as Narayanan and Oakley provide substantial evidence illustrating the ways quantitative methods and results can uphold the hegemonic (white, male) powers in American society, they argue that if done correctly, these methods can be powerful and empowering.
A Framework for Improving Empirical Methodologies
While Narayanan and Oakley argue for the adoption of a “better” quantitative approach, they provide only an idealistic means to do so. Scholars Gillborn, Warmington and Demack provide a concrete framework for decreasing discrimination in quantitative methods. Building on the pillars of Critical Race Theory (CRT), the authors propose a “QuantCrit” framework with a number of central tenets: 1. Racism is both deeply rooted in society and not easily quantified. 2. Statistics are nonneutral, and should be critiqued in the ways they serve the White agenda. 3. Critical evaluation and thought is imperative to evaluate units and methods. 4. Human perspectives, especially those of marginalized groups, are critical to informing data. 5. Quantitative analysis is inherently meaningless but can contribute to the struggle for social equity and justice (Gillborn, Warmington, and Demack 2018, 158). The authors both recognize the power of statistics to highlight social structures, they also disguise racial inequities (2018, 160). Building on Critical Race Theory, they critique the U.S.’s “uncritical use of quantitative data” (2018, 175), providing a new way to conceptualize empirical methods. Following the QuantCrit framework is a path to using quantitative statistics for social good, instead of for reproducing the status quo.
Conclusion
Ultimately, I agree with Narayan’s statement that quantitative methods as researchers use them today tend to do more harm than good. While there are a number of limitations of quantitative methodologies that make them susceptible to discrimination, I believe the downfall of these methodologies is primarily their failure to consider systemic inequities and racism. Without a critical consideration of the underlying power dynamics in the U.S., models and statistics end up compound inequalities, intentionally or not. Even operating under the assumption/null hypothesis that a system lacks discrimination is harmful to communities marginalized daily on large and small scales. At the same time, I admire Narayan’s and Oakley’s optimistic attitude towards quantitative studies, and am impressed by Gillborn et al’s QuantCrit framework, which positions quantitative statistics both as a tool to maintain White hegemonic power as well as a means to enact social justice. Empirical data harbors immense power; it can marginalize already marginalized groups, or, it can expose injustice and foster empathy. While quantitative studies can have immense impact, such as informing policy or exposing and quantifying trends, “numbers by themselves have no meaning” (Narayanan 2022, 26). To create a just society, we must think critically about data and the power structures it supports.
To view the source code, pelase follow this link: https://github.com/madgallop/madgallop.github.io/blob/main/posts/quat_bias_blog/quantBiasBlog.ipynb