It was, as a matter of fact, a sunny day in 2014. I probably should remember this with greater clarity—such days are rarities in late-winter London—but my memory has imposed a drizzle that Google denies. Apparently, I was too captivated, too distracted, by what unfolded as we gathered in the newly remodeled common room for our routine departmental meeting, the smell of recently installed linoleum still fresh in the air.
At the time, I was a lecturer [1] in the Department of Sociology of the London School of Economics and Political Science. As in most other institutions, being a full-time academic at LSE involved occasional meetings with colleagues. We would discuss matters relevant to our teaching programs and internal administration, the hiring of new faculty, the intellectual direction of our department, and our views on our institution’s constantly changing initiatives and policies. Bureaucracy, some would grumble, or, more generously, community-building, such conclaves are among the indisputable ceremonial practices of academic participation. Anyhow, as we did every now and then, we undertook that strange mix of scholarly ritual and managerial intervention, talking shop on an unusually sunny late-winter afternoon. We read and approved the minutes from the previous term’s meeting; we heard from our head of department; and we listened to faculty committees’ reports concerning students, teaching, and research. Having ticked off all the items on our agenda, we had reached the apparent end of this organizational rite. Just then, in the twilight closing moment when one begins thinking ahead to the day’s demands (office hours, a meeting here, an unattended inbox there) yet remains aware of the parting pleasantries, my mind snapped to attention.
“The school has requested us,” began the head of department, “to come up with a list of journals that we consider prestigious in our fields of expertise—that define us as a department. They want to use this for our next evaluation, to have a better sense of our own standards of excellence.” The chatter vanished, replaced by this infinitely more intriguing gambit. Silence followed, heavy with a combination of both incredulity and resignation. “This is an opportunity for us to decide how we are evaluated,” the head of department nudged.
Several moments passed before an intrepid colleague ventured the first contribution: “the British Journal of Sociology, I guess.” This sacrificial offer made sense: in addition to being one of UK sociology’s flagship journals, the BJS was currently stewarded by our department. “Sociology,” followed another, naturally forwarding the scholarly publication of our discipline’s professional association. No argument there. “City & Community,” said someone else, reflecting our department’s investments and interests in urban sociology. “Certainly Theory, Culture and Society,” said a fourth. “Work, Employment and Society,” called out another, and I heard “Antipode” from a voice at the back of the room.
All these journals were (and remain) sensible suggestions. They were, after all, close to the topics, scholarly genres, and intellectual traditions followed by academics in our department. They contained the voices of our community, the traditions of our craft. But these suggestions were not, perhaps, as prestigious as they might have been, at least not in the eyes of the upper administration. They wanted to see “top journals,” the kind that dominate rankings, rack up citations, and confer scholarly esteem to their contributors and affiliated institutions.
“Think big,” urged the head of department. “The list has to be credible; it needs to convey we are ambitious and want to publish in the very top.”
“Well then,” someone in the room responded, “it’s the American Journal of Sociology and the American Sociological Review, even if we rarely publish there.” At the time, “rarely” was quite the understatement.
This seemingly banal exercise, at precisely this moment, became for me the sight that eclipsed the sun. There we were, a room full of sociologists anxiously fashioning chimeras, lists that combined tradition with aspiration, practice with expectations, and, in doing so, forging the very chains that would bind our knowledge, link by link, word by word.
Repeated across universities in the United Kingdom and often with far less participation from staff,2 this exercise is a direct response to the cultures of assessment and evaluation that have proliferated throughout the British higher education sector in the past four decades.3 Since 1986, when the British government first mandated that publicly funded universities submit regular, standardized assessments of research quality, scholars and managers have faced the vexing problem of evaluating the intellectual worth of articles, books, and other creative products of career academics, with the aim of rewarding “excellence.”4 With furrowed brows, we ask: When do we know that a paper is outstanding? How do we know if a book made a substantive contribution to knowledge? And we wonder whether the tallies we create answer the government’s implicit questions: Is the public expenditure on science worth it? Is state funding being used efficiently, going to the best possible researchers in the most effective centers of knowledge production?
In this book, I am concerned with a pair of naturally extending questions: Do universities foster a form of scholarly excellence and selectivity that is, in fact, visible, measurable, accountable to the public? And how do the quantification and ranking of scholars and their work, through lists, assessment exercises, and other devices, affect the scholarship itself? The list my LSE colleagues were asked to produce was intended as an instrument that allowed managers and academic peers from other disciplines within our organization to make sense of our work, to assign us value. Being invited to craft this list certainly granted us a sense of buy-in, yet each scholar in the room understood it would be used as a measuring tape, of sorts—one more way we might be compared, quantified, and ranked, with consequences on the knowledge we produced.
Knowledge is difficult to quantify; still, we try. In the following, I look at a particular instance of how trying to quantify the value and excellence of knowledge—specifically, the British evaluations known as the Research Assessment Exercises and, more recently, the Research Excellence Framework—changes the nature of scholarship and academic lives. In The Quantified Scholar, I argue that measuring value and excellence in science fosters specific forms of what sociologists Wendy Espeland and Michael Sauder call “reactivity”5: it introduces novel incentives for managers and shifts the goalposts for scholars, changing the way they think about and experience their careers and their craft. Quantification makes visible specific and highly arbitrary hierarchies of worth that, when tied to the long-standing cultures of repute in scholarly fields, change notions of who and what is valuable.
Studying the British case, this book shows how the adoption of standardized research evaluations changed the way social scientific knowledge about the world was produced. It did so in two ways. At one level, it perturbated local labor markets for academics, changing the structure of careers in a way that produced more homogeneous institutions within the fields of anthropology, economics, political science, and sociology. At another level, these evaluations changed the way academics understood and organized their own worth, echoing in their everyday lives the hierarchies that were implicit in the practices of these formal assessments of research excellence. Slowly but surely, these vast and intrusive evaluation exercises made the conceptual schemas of scientists increasingly similar, ever more homogeneous, across the four disciplines that I study.
In a book about the impact of quantification on knowledge, however, the argument I make here is ultimately about academic vocations. If quantification holds a strong grip over the work of British academics—and those elsewhere exposed to the countless metrics of modern scholarly work—it is because of how scholars collectively come to accept and reproduce cultures of repute, overwork, and sacrifice connected to the ideals of research in science. Academics are often trained to hold research as the dearest of their obligations, striving to produce forms of knowledge that will be recognized by their peers and future generations. Even more, we are habituated to see ourselves in the research we produce. When we think of scholars celebrated for their contributions to the understanding of culture, politics, economy, and society—the types of names peppered throughout most introductory classes in high school and college—we casually equate their works with the lives of their authors. We talk of Max Weber as we do of his works; we conflate Hannah Arendt and her essays; we shorthand Adam Smith for his foundational books. They are one and the same: the carefully considered, curated, crafted, edited words on paper, and the messy, complicated, and contradictory lives of their authors, their bodies, and their careers. This is what we are trained to think, both as audience and as performers.
But in this training and vocation, scholars often reproduce a view of the world that dissolves bonds of solidarity in our workplace and profession. This is not a story of algorithmic or organizational inevitability but of choice—of recognizing our role in deciding how we value peers and their various forms of work. In the above vignette, what mattered was certainly how my LSE colleagues collectively populated the list; yet in accepting without dispute the role of the list as an instrument of value, we begrudgingly accepted the logic of standardized evaluations and what they expected about knowledge. Compiling a list of journals involved playing the game; but for a brief second, we had the upper hand to set the ground rules of our evaluation. The list we chose reflected our wants and aspirations rather than our community and strife. This is ultimately the challenge of the quantified scholar: choosing solidarity over the politics of prestige in a profession that sees repute as its prime currency.
Counting Knowledge
It is little wonder the measuring tapes would come for the sciences, given a vast and impressive literature that attests to measuring the world as an essential, centuries-old feature of scientific practices.6 The social sciences especially count on counting to explain the intangibles that animate our ever-changing settings: economic growth, social class, political attitudes, religious conviction, unseen psychological dispositions, interpersonal trust, human suffering, commitment, taste, and so on. Quantifying these objects allows scientists to produce knowledge, making claims about various complex processes of the social world. But, what happens on those occasions when science itself is quantified, particularly with a clear managerial intent?7
Metrics have long been used to organize, reward, and shape the course of science throughout the world. I still have vivid memories of my parents, both professors of biochemistry working at public institutions in Mexico, assiduously undertaking their annual rituals of verification. The national funding agency expected them to report the impact factors8 of the journals in which they published their research as a way of guaranteeing the quality of their contributions. This onerous task meant securing print versions (often second- and thirdhand photocopies) of the Journal Citation Reports, in which these impact factors were organized alphabetically. When the internet made these measures more readily available, scholars began being asked to submit individual citation counts for their papers; these, too, were diligently gathered (regardless of expensive paywalls) and provided in order to demonstrate value and sustain state support. However often quantification is decried as “a cheap and ineffective method of assessing the productivity of individual scientists,”9 metrics like these are routinely employed by universities, funding agencies, government bodies, and international organizations interested in learning what they get for their money, tallying their returns on investment.
Scientists are not uninterested in valuations of their work. Although largely an instrument of external management today, counting science arguably started within modern science itself, as a means of measuring quality and excellence in research. As Paul Wouters reminds us in his now classic dissertation, for example, research librarians have “systematically applied citation analysis” since the early twentieth century, ostensibly to measure the usefulness of costly subscriptions.10 The interest in counting was not limited to librarians but extended to the broader scholarly community. The notorious statistician Alfred Lotka, for example, sought to identify patterns in the distribution of publications as early as 1926 to “determine the part which men [sic] of different caliber contribute to the progress of science.”11 While motivated by intellectual concerns about the structure of scientific disciplines, the fields of bibliometrics and scientometrics are now established and frequently marshalled to determine the worth of scholars and their contributions to those disciplines.12
We might think of metrics, scales, and rulers as merely convenient devices for representing our world. In practice, however, most forms of measurement are tied to purposeful interventions. We measure to do things, whether to establish property lines with strings and trigonometry, understand the fundamental relations between subatomic particles, or cut a sheet of plywood into the components of a new drawer. But unlike tracing imaginary lines on the land beneath our feet, estimating the arc of a particle in a cloud chamber, or measuring bits of lumber, quantifying the productivity and quality of scholars is a particularly interventionist act. Measuring these “social kinds,” to borrow from philosopher Ian Hacking,13 alters the qualities of the objects under assessment—be they scholars, institutions, disciplines, or knowledge at large. When quantification is public and constantly visible, affording comparisons and competition, as Jelena Brancović and colleagues note, it triggers reactivity—that is, changes in the interests, practical strategies, and intellectual approaches of scholars in response to the incentives of rankings, numerical assessments, and metrics of work.14
How do quantification and its associated reactivity affect scholars and their scholarship? Does this process produce, over time, “better” knowledge? Or does its lead to “worse” accounts of the world? I’ll be as straightforward and honest as possible: quality is relative. I cannot give you, reader, a definitive ruling on the future, long-term effects of quantification on scientific knowledge. (Being in the fortunate though entirely hypothetical position of knowing what is optimal for science would mean having cracked a quandary at the core of philosophic inquiry—what, precisely, separates science from other forms of knowledge—which I can’t say I have.) Throughout this book, however, I will provide extensive evidence suggesting that efforts to quantify the value of science have affected knowledge production as well as the organization of scientific disciplines, fields, and academic units and the progression of individual careers. Whereas counting helps social scientists make sense of our world, being counted helps account for the questions we ask (and those we don’t ask). By studying the mechanics of research evaluations in the United Kingdom, I show that knowledge produced by social scientists has become increasingly homogeneous within and across institutions. The actuarial demands of austerity, in which impact scores, journal rankings, and periodic evaluations dictate funding allotments and justify intellectual investments, circumscribe the types of topics explored by social scientists and the meanings associated with their concepts and theories of the world.
The consequences are stark: the uptake of quantification as a means for managing British science has resulted in ever more disciplinary logic and organization in academic fields. This occurs through a process I call “epistemic sorting”: the cultures of evaluation fostered by quantification create incentives for scholars to sort themselves out across the institutional space of British higher education in ways that funnel their disciplines toward homogeneity. Epistemic sorting alters what scientists know—and seek to know—about the world in potentially fundamental ways. While I cannot say with absolute foresight that this is a negative outcome, my findings are shot through with an underlying value claim. For those who find worth in intellectual diversity and scientific serendipity, the effects of quantification that I document in this book are likely pernicious: scientists at the UK’s leading institutions are, over time, conducting less risky, less innovative work.
Attentive readers will note my careful references to Britain and the UK. They are purposeful. The British case is particularly useful for examining the way quantification changes scientific work, given how it combines patterns of fiscal austerity, government-sponsored quantification, and the internationalization of academic work since the 1980s. While populated by institutions old and new, the British higher education sector is relatively standardized, providing a sort of natural experimental control. As explored in chapter 2, the various types of British universities—from the medieval institutions that served aristocratic elites to the newer ex-polytechnics designed to train the postwar professional masses—have converged through mimesis into a single institutional type.15 Prestige and financial resources remain unequally distributed across the sector and teaching loads and research expectations vary, yet, by and large, most British universities operate under the same academic model: scholars are expected to publish, teach, and provide service to the institution and their profession, with the first of these taking symbolic prominence. Unlike the U.S. system, in which wealthy private institutions compete with flagship state colleges for both students and faculty, while also sharing space with a panoply of institutional types and forms, higher education in the UK is largely a public enterprise. It is regulated by the national governments of England, Scotland, and Wales and organized across a single workers’ trade union, the University and College Union. This relative uniformity in organizational design allows for analyzing the effects of research evaluations with due care for teasing out extraneous factors that might otherwise create “noisy” data and inconclusive causality.
In addition to having a long-standing and established higher education system, Britain has periodically assessed academics employed in its public universities through vast peer-review-like exercises that are meant to determine the quality of research produced across institutions and their disciplinary units. These evaluations matter doubly: besides affording status and prestige (a form of currency that is central to science), they are tied to state financial resources. Pools of “quality-related” research funding are disbursed only to the best performing institutions in each field, as determined by the assessment exercises. Over time, these assessments have taken various forms and names—initially Research Selectivity, then Research Assessment Exercise, and since 2014 Research Excellence Framework—but in all cases they have involved evaluations of academic units performed by panels of disciplinary peers. These differ substantively from the more individualized forms of assessment that exist in other countries, which focus on each scholar’s productivity or visibility as a measure of worth. This is not because individual scholars are not evaluated in the UK assessments, but simply because their individual results are never made public. So, in every iteration since the first, in 1986, scholars of all British public institutions have been assessed, with their work read and scored by peers across a scale of quality from “unclassified” outputs that fall “below the standard of nationally recognized work” to four-star research, “world-leading in terms of originality, significance and rigor.”16 The scores of individual research products become not a mark borne by each scientist (no one really knows how their work was evaluated, with individual scores shrouded in secrecy and destroyed shortly after the assessments) but, rather, as cumulative calculations of their institutions’ quality and disciplinary value.
In this, the forms of state-sponsored scholarly quantification observed in the United Kingdom are structurally closer to the ways U.S. institutions are ranked (the emblematic case being law school rankings, as revealed by the aforementioned Espeland and Sauder17). Though the evaluations are ostensibly high-level—department A produces more excellent research than department B—the various ways administrators and peers place value on, and react to, the outcomes of these evaluations have consequences for individual careers and the disciplines that scholars inhabit. The effects of these exercises are moderated by distinct organizational cultures. In some institutional settings, the evaluations are hardly felt; in others, they are objects of constant anxiety, directing hiring and promotion decisions, resource allocations, and other consequential processes. Even the administration of assessments can become a tacit game, with institutions attempting to sway scoring (and risking mistakes like submitting scholars’ records to the wrong disciplinary peer review panels, leading to poor results and negative consequences for capable employees).
To study quantification, I have departed from some of the traditional approaches taken by scholars in the sociology, anthropology, and philosophy of science. For decades, these colleagues have studied knowledge as constituted not through some universal method of discovery but, rather, through piecewise processes of enrolment, delegation, representation, intervention, looping, controversy, falsification, refutation, contestation, and closure. Scientific knowledge is “socially constructed,” insofar as it is created within specific communities of experts who, on the basis of ongoing conversations and interventions, revise their claims about the world.18 These approaches clearly foreground the epistemic dimensions of science, tackling the question of how practices, communities, and institutions come together to assemble scientific knowledge. In what follows, I am informed by these approaches, though I focus not on the ways knowledge is made (how it encodes politics and interests, how it depends on complex alliances between humans and instruments, or how it produces or forestalls social action, for instance) but on the conditions experienced by those who are in the business of its production. That is, I consider the work world of academic science. Laboratories are certainly sites for epistemic practices, but they are, too, invariably sites of work, of paid employment, of managerial intervention.19 By considering knowledge as a distinct product of labor, The Quantified Scholar finds that quantification matters for how knowledge is produced because it alters how the knowledge-makers experience their crafts.
Trajectories of Devotion
The objects of this study are embedded workers whose intellectual labor is invariably shaped by the affordances, incentives, biases, and barriers creating and curtailing the shop floors of the modern university. I am inspired by a large sociological literature on work and occupations, while emphasizing a particular facet of academic employment relations and the ways scholar-employees’ experiences of evaluation shape the course of their labor over longer periods of time. The processes I trace in this book—the forms of epistemic sorting and linguistic change that are tied to the implementation of quantified research evaluations—are not punctual but processual, forming slow shifts in the register of scholarly conversations and the organization of disciplinary fields. These changes are connected to, yet overflow, the contractual relations between employers and managers, because academics are both workers, bound by contract to the universities that employ them, and professionals bound by the practices, traditions, and evaluation cultures of their individual fields.
Careers are fascinating meso-level phenomena to social scientists. Foundational scholar Erwin Goffman characterized the concept of the career as one that allows us to connect micro- and macro-level social processes by moving “back and forth between the personal and the public, between the self and its significant society.”20 In the present study, a scholarly career is what unites the individual, rational, isolated epistemic worker to the social agent who actively navigates institutional structures over the course of a field-based intellectual career.
Considering the tensions between the personal and the public provides a novel understanding of how knowledge and scientific fields change over time. Like others, barring exceptional luck or nepotism, scholars find establishing “successful” careers virtually impossible without expending some degree of individual effort—without putting in work and investments that translate into an intellectual contribution that helps maintain an institutional affiliation that provides a site for research. At the same time, careers are shaped by factors beyond the control of individuals: gender and racial biases saturate formal evaluations, peer networks amplify some work through frequent citation (and ignore similar work performed by out-group members), and life circumstances draw greater penalties to particular, often minoritized, groups.21 Many of these well-documented biases—reflected in productivity gaps, promotion gaps, salary gaps, and citation gaps—surely reflect larger, economywide structures of discrimination and inequality (academia is not unique in this sense). Within academic science, however, these inequalities are additionally bound to the noncontractual expectations that shape our sense of commitment to academic disciplines.
Who we decide to include in our syllabi or cite in our works is rarely controlled by our employers—but is often policed by our disciplinary peers. Academia, in other words, is unexceptional as a form of employment in which the formal structures of our employing organizations impinge on our careers, yet exceptional in that our work is also associated with a form of vocation, evaluated and shaped by the invisible, weighty traditions to which we belong.
My strategy in The Quantified Scholar is to move back and forth between the individual scientist, with their personal experiences as managed workers, and the public, collective disciplinary settings in which their work is read, used, and assigned worth. These distinct but interrelated domains of the workplace (ruled by specific managerial expectations and contractual arrangements) and the discipline (where more tacit notions of value are produced and enacted through peer training and habits) are linked by the cooperation of research evaluations across both. Neither entirely “bureaucratic” interventions nor simply “intellectual” exercises, research evaluations establish expectations of productivity and scholarship that tie faculty work to institutional interests (with career consequences) and, by virtue of their connections to notions of scholarly quality, status, and prestige, reify disciplinary norms around relative value that, in turn, shape scholars’ everyday intellectual decisions. More pointedly, because research evaluations are grounded in disciplinary peer review—with sociologists evaluating sociologists and so on—they lead to increasingly homogeneous scholarly fields. The diminishment of scope, which may be connected to less risky, recombinant, innovative forms of research,22 is not then a direct product of scholars’ individual choices and inclinations; it is a synthetic consequence of their conditions of work. Thus, to understand how research evaluations lead to the production of more homogeneous, paradigmatic forms of knowledge, I focus on their effects on scientific careers—the scaffolds on which we iteratively build our fields of academic practice.
These scaffolds are admittedly peculiar. A notable feature of scientific careers is the degree to which they are framed by the idea of a vocation, a “calling” to produce knowledge for its own sake, a devotion to the discipline, its logics, and its practices. This point was famously raised by Max Weber in his lecture on Science as Vocation, wherein he eloquently captured many scholars’ personal enthrallment with and passionate dedication to scholarship in general as well as to their highly specialized objects of study.23 This vocation is not practical—as sociologist Steven Shapin notes, the scientist’s orientation does not encompass, in Weber’s conception, “commercial goals and entrepreneurial means.”24 It is concerned solely with the production of facts and knowledge—finding truths that, however fickle and ultimately falsifiable, populate our shared fields of scholarship and pay the entry fee for a shared sense of academic, collegial, intellectual integrity. Although written almost a century ago, when performance management and scholarly evaluation were still incipient, Weber’s account still resonates (decade upon decade, The Scholar’s Work warms the printing press). Despite the professionalization of scientists and the structural changes to higher education, the transformation of scholarship from a calling into a “mere” job is necessarily incomplete. Shapin reminds us that, to this day, the tension between employment and vocation, paycheck and devotion, alters the identity of scientists and how they value their own contributions to scientific advancement.25 After decades of professional change, we remain vocational in a Weberian sense. Our work is a “mode of life” that encompasses our minds, our bodies, and the very fabric of our souls.
The survival of this vocational spirit in the sciences has concrete implications for peer evaluation. Yes, Weber was correct that scientists, carving out our niches, produce increasingly specialized knowledge claims on ever more particular fractions of our world. But even in today’s hyperspecialized scholarship, where no single person can feasibly know their entire field, the larger structures of disciplines loom large. The tremendous rise in “interdisciplinary” research has not dislodged professional identities nor how organized performance evaluations hew to universities’ departmental and divisional organization. Time marches on, and we continue to frame intellectual value and scholarly contributions in relation to the identifiable disciplines and subfields that anchor the objects of our vocation. Within these, loosely institutionalized forms of prestige become yardsticks, as sociologist Richard Whitley argues: repute sits at the base of many of our organizational forms, a convenient means for assigning confidence to knowledge claims in an otherwise messy ecology.26
Our vocation is, in this way, bound to the numerous hierarchies—of institutions, of scholars, of traditions, of theories, of concepts—into which we are habituated. Quantification has surely ossified many. In Grading the College, for example, the historian of higher education Scott Gelber relates very early efforts to rank colleges in terms of how well they “prepared” their graduates for work in business and government (in 1912), as well as by increasingly intensive pushes to evaluate teaching quality starting in the 1920s.27 That these rankings largely mirrored existing perceptions of institutional prestige should not come as a shock.28 This finds echoes in the work of Jelena Branković and Stephan Wilbers, who identify both the long historical roots of academic rankings at the beginning of the twentieth century and the mechanisms through which they acquired postwar dominance. Moving away from their originally peripheral position to their place as primary managerial instruments required a shift in the logic of institutions of higher education toward framing excellence as a form of performance that could (perhaps had to) be constantly evaluated. Being an assiduous scholar was not enough: true devotion was only seen in ongoing, assessable, measurable actions and contributions.
We can return here to Espeland and Sauder’s now canonical account of how such public evaluations transformed organizations and, in the process, our collective comfort with quantified hierarchies.29 In their study, external rankings of law schools by U.S. News & World Report began as public instruments but soon became environments requiring organizational attention—the rising metrics came to change institutional strategies and priorities as well as the self-conceptions of their managers and workers (today, U.S. News ranks everything from high schools to mutual funds, ostensibly guiding consumers in a variety of crowded “marketplaces”). In part, we accept hierarchies like these because they are readily observable, patently material. They are “facts” that speak to the logic of our work and travel across organizational settings, making comparison of otherwise distinct objects possible and allowing for common conversations about worth.
The use of metrics in the evaluation of scholarly work is certainly fascinating, yet what I find interesting about rankings, ratings, and scores is not that they exist and have performative effects (that, by design, they are self-fulfilling prophecies), but that we readily accept them and the forms of worth they imply. Our discussions of quantification seem to lack a link between the historical circumstances that made counting research possible and the way scientists continue to frame their work in vocational terms. What we have learned from the literature on self-tracking is that quantification is seductive, allowing individuals to evaluate their own worth and efforts, then aspire to selves prefigured by the devices and arrangements that measure them. The quantification of scholars is no different. While it depends on certain historical conditions of possibility, it is maintained by practices of status, prestige, and repute that hold affinities with our vocational ideals.31 The quantified scholar is not merely a professional demand but a way to fulfill our desire to truly belong within our rationalized, modern, scholarly vocations. It is as if the numbers demonstrate that we have earned our place.
This explains, perhaps, why devices like the h-index, barely fifteen years old at the time of writing, so quickly came to embed the global infrastructures of science metrics and research evaluation. The h-index measures the cumulative performance of a scholar’s career by counting the number of publications for which they have been cited by others at least that same number of times. An author with ten publications only four of which have been cited four or more times would have, hence, an h-index of 4. The uncanny history of this metric shows that, rather than zealous administrators seeking to extract ever more from their scientist workers, the metric rose on the backs of scholars actively adopting it to better “judge the performance of researchers.” “Most scholars,” read an editorial in Nature in 2005, “prefer an explicit peer assessment of their work. Yet those same researchers know how time-consuming peer assessment can be.”32 In other words, if the gold-standard peer evaluations were too time-consuming, and measures like citations, journal factors, and institutional pedigree were regarded as less than adequate measures of quality, the h-index provided an appealing enough measure. It shorthanded a scholar’s lifetime output against a discipline-specific proxy of impact. The h-index was soon adopted by scholars and refined as a frenzy of academic articles in physics, biology, sociology, computer science, and elsewhere tested its performance against previous metrics. Note that this was not about rejection, but calibration and acceptance. The success of the h-index did not stem solely from some powerful autonomous assemblage that, marketlike, sought to economize intellectual value, as sociologist Roger Burrows suggests;33 to the extent this form of quantification succeeded at all, it was at least as much because, deep within our modern vocation, within our training, habituation, and disposition, scholars have clear affinities for measuring their own prestige.
This is even more complicated because, as happens with artists and other creative workers, scholars have difficulty separating their personal and professional lives. Vocations are not 9-to-5 jobs, and they are not put on hold.34 Academic careers are fuzzy cominglings of personal and professional selves (recalling Goffman, above), and this encourages us scientists to elide the personal and professional in our own careers. No lesser a sociologist than C. Wright Mills noted his early realization that most of the “thinkers and writers whom [he] admired never split their work from their lives.”35 We are what we do. Ours is a “strange intoxication”, as Weber wrote, that renders the hierarchies of our fields and the criticisms and exultations of our work synonymous with our individual efforts or failures. Our identities and our careers, amplified, refracted, and modulated by quantification, alter what we see and think of as objects worthy of our individual passions or unjustifiably risky for our prestige.
Reflexive Knowledge
Sitting around the green and orange Formica tables of our departmental common room, the LSE faculty in the opening vignette were knowingly tracing the contours of our promised selves, committing to aspirational personas that would reflect back onto our work and our sense of scholarly vocation. This peculiarity is what erased my memory of the sunshine: my colleagues and I so quickly accepted—hell, created—a daily target of ten thousand epistemic steps though we knew little about the terrain ahead.36 The looping effects of rankings, lists, and quantification were not alien to us: paper after paper, study after study in our very own discipline suggested that counting things matter in markets, organizations, and employment. We understood quantification as a fact of the social world, a matter of life and death. And yet there we were, trying to balance control and bureaucratic dictum, internal consistency and external legitimacy, forging our own chains of knowledge, creating our own boundaries, establishing our own measures of value. That they were required at all seemed beyond question.
Admittedly, this practice of setting goals and parameters of excellence is common to other academic disciplines, although it is put in practice in different ways. British sociologists have adamantly stressed the importance of evaluating books and papers by their contents rather than the status of their publishers or the affiliation of their authors, but this ethos is in no way shared across disciplinary lines. Fields like economics have readily accepted hierarchies of value, with clear tiers of journal quality (books are exceedingly rare as a research output in this field) mapping neatly onto broader evaluations of intellectual quality. In other domains, like political science (in Britain, more appropriately understood as politics and international relations), there are hierarchies, but these are noisy, defined by multidirectional tensions between European and American traditions of political thought as well as varying methodological approaches. In a smaller field like anthropology, the onus is traditionally placed on the peer-established quality of texts, though unofficial conversations and references regarding publisher rank and the discipline’s institutional “golden triangle” (formed by elite departments in London, Cambridge, and Oxford) assert themselves throughout scholarly valuations. These different disciplinary cultures, as sociologist Michèle Lamont argues, vary in their definitions of excellence, as in how willingly they accept certain forms of self-quantification.37
These four disciplines share a sense of reflexivity—that is, a willingness to accept that the knowledge they produce about the world is inflected by the experiences and institutional situations of their scholars. Even economics, which seems to hold the most naturalized and individualistic view of the social, operates under the assumption that knowledge can be used to optimize or nudge the object of study in particular directions, including economists themselves. Part of this reflexivity involves a widespread recognition that quantification is a descriptor of research quality but that it can nevertheless be used with specific aims. What matters is not so much whether quantification is “actually accurate” as how it is made sense of by those it counts. Such self-awareness arguably leads to stronger forms of reactivity than in other fields like, say, high-energy physics or biotechnology. That is partly why I have chosen these four fields—anthropology, economics, politics, and sociology—for study in The Quantified Scholar. In addition to lacking the fixed investments associated with disciplines in which laboratories and other forms of equipment make careers stickier (moving a lab is obviously a costly and fraught endeavor)38 and displaying higher rates of single-authored works39 (both of which explain my exclusion of fields like psychology), the vocation of the social scientist is modulated by a form of reflexivity that gives a different texture to discussions about excellence, quality, and performance evaluations.
The social sciences are also arguably more “flexible” toward their institutional settings, making them ostensibly better targets for studying how quantification changes their practices of knowledge-making. Sociologist Marion Fourcade’s exceptional work on the development of economics, for example, shows that the variation in the organization and contents of this discipline—visible in its intellectual interests, theoretical approaches, and forms of institutionalization—observed in France, the United Kingdom, and the United States, is traceable to economists’ position with respect to the state and industry in each country. A similar variation would be hard to find in the natural sciences—not because it cannot exist (national varieties of the “hard” sciences have been well documented by historians) but because these have experienced greater levels of global standardization than have the bulk of the social sciences.40
Within these four disciplines, I focus on academic careers—primarily, on movements of scholars between institutions (what the social scientific literature terms labor mobility)—to demonstrate how disciplines change over time in relation to the uptake of research evaluations. Knowledge is the product of communal efforts, yet it is ultimately tied to the bodies that make it possible. Academic social scientists’ relative independence from physical research infrastructures like laboratories allows me to assess how quantification shapes their organizational strategies as they attempt to govern and make sense of their immediate environments, while their employment contracts and disciplinary norms help them retain a degree of control over their labor. How they use this control is key: they can, in a sense, “play the game” of quantification or attempt to extract themselves from it, either through collective action or through changes to disciplinary norms. Though indirect, this control is practiced through internal organizational processes (for example, in scholars’ approaches to assessing colleagues, serving on external panels, or reviewing peers’ work). At the same time, the reliance of social scientists on shared knowledge infrastructures—things like data sets resulting from large-scale, high-quality surveys used by (and to train) large numbers of scholars—makes them subjects of distinct forms of disciplinary control. This tension between organizational and disciplinary control matters centrally for this book: even in settings with greater relative autonomy, quantification comes to matter in the way knowledge is made, cutting across the reflexive dispositions of scholars, the managerial logic of their institutions, and the prestige-based hierarchies of their fields.
The forms of reflexivity, autonomy, and institutional situatedness that attach to the social sciences point to opportunities to challenge some of the most pernicious effects of quantification. The central lesson of The Quantified Scholar is not that quantification is necessarily bad, but that when it becomes part of our way of making sense of the value of others and their knowledge, it leads to less hospitable, dynamic, and innovative disciplinary fields. The power contained in numbers, rankings, lists, and other measurement tapes hinges on their embedding organizations, with their sites of application and use potentially attenuating their effects. In looking at the spectrum of experiences under quantification in Britain, I will key in on cases in which scholars have actively resisted the disadvantages of quantification, fostering deeper consideration of the forms of solidarity that may serve as a balm against such corrosive outcomes. Reflexivity—of our shared condition, our common vocation, our collective knowledge—can be leveraged to produce more equitable, humane conditions of work and to tamp down the reactivity of our disciplines to bureaucratic demands.
Structure of the Book
To answer the question of how quantification changes scientific knowledge, I adopt a multipronged approach that combines various computational techniques of text analysis, quantitative models of career mobility, and interviews with British scholars active in anthropology, economics, politics, or sociology, as well as union representatives. Each tactic provides evidence that, jointly, suggests a process of increased homogeneity in the British social sciences driven by quantification’s effects on careers. (A fuller description of my methods and data analysis, which echo the logic of the extended case method, are available in the appendix.)
I begin, in the next chapter, by exploring the origins of research assessments and quantification in the context of key transformations of the British higher education system. This story connects the drive to quantify scholarly excellence to the implementation of austerity measures and their attendant “audit cultures” (a term coined by anthropologist Marylin Strathern) across UK universities in the 1980s.41 After additionally explaining how research evaluations quantify “excellence” in practice, I turn, in chapter 3, to the effects of quantification on academic careers, using evidence that links research evaluations with changes in the structure and organization of academic departments across time. Chapter 4 analyzes different fields’ linguistic (and, ostensibly, knowledge) shifts in response to disciplinary pressures toward epistemic conformity. Together, chapters 3 and 4 present the concept and mechanism of epistemic sorting and its centrality to the effects of quantification on academic labor and scholarly careers.
In chapter 5, I take a different perspective, looking at how quantification has been experienced by academics in their workplaces. In particular, this chapter stresses the importance of local managerial implementation in understanding how scholars rethink their vocations under quantification regimes. One key observation is that quantification is moderated by hierarchies: individuals at the top of their field and their institution are less swayed by the pressures of quantification than are those in the “upward-oriented” middle of the hierarchy and peers at resource-strapped, teaching-intensive institutions that aspire to climb the league tables by expecting more “excellent” research from already overstretched staff. The interplay between quantification and prestige also offers an opportunity to discuss disconfirming cases, in which scholars were insulated from the negative effects of research evaluations by the support of their peers.
The Quantified Scholar closes, in chapter 6, by pitting quantification against the scholar’s vocation. In this adjudication, I argue that the problem posed by quantification is fundamentally the way it triggers reactivity: it is not the quantification per se, but the way disciplines collectively deal with the individualization of scholars’ professional worth. Having studied the practice and its implications for social science, I insist on the importance of rethinking our vocation, moving beyond devotion to scholarship as a calling toward devotion centered on scholarship as a lived, shared, multidimensional form of labor.
[1] British lecturers are roughly the equivalent of tenure-track assistant professors in the United States.
[2] Unlike the United States, where faculty is a distinct group, academics in the United Kingdom are referred to as ‘academic staff’. I will use this terminology throughout the book.
[3] Strathern, Audit Cultures.
[4] Wilsdon, The Metric Tide.
[5] Latour and Callon, “‘Thou Shall Not Calculate!’Or How to Symmetricalize Gift and Capital.”
[6] Espeland and Stevens, “A Sociology of Quantification”; Porter, Trust in Numbers.
[7] Lepenies, The Power of a Single Number; Özgöde, “Institutionalism in Action”; Bland, “Measuring” Social Class” A Discussion of the Registrar-General’s Classification”; Goldthorpe and McKnight, “The Economic Basis of Social Class”; Bukodi and Goldthorpe, “Decomposing ‘Social Origins’”; Verhulst, Eaves, and Hatemi, “Correlation Not Causation”; Norpoth and Lodge, “The Difference between Attitudes and Nonattitudes in the Mass Public”; Vaisey, “The ‘Attitudinal Fallacy’ Is a Fallacy”; Vaisey and Lizardo, “Cultural Fragmentation or Acquired Dispositions?”; Jerolmack and Khan, “Toward an Understanding of the Relationship between Accounts and Action”; Himmelfarb, “Measuring Religious Involvement.”
[8] The Journal Impact Factor is a proprietary metric developed by the Institute for Scientific Information (now Clarivate Analytics) that tries to approximate the visibility of publications by calculating the ‘average’ frequency of citations of papers published in peer-reviewed journals.
[9] Lotka, “The Frequency Distribution of Scientific Productivity.”
[10] Godin, “On the Origins of Bibliometrics”; Godin, Measurement and Statistics on Science and Technology; Abramo, D’Angelo, and Caprasecca, “Allocative Efficiency in Public Research Funding”; Cronin and Sugimoto, Beyond Bibliometrics; Gingras, Bibliometrics and Research Evaluation.
[11] “How to Improve the Use of Metrics.”
[12] “Assessment Criteria and Level Definitions : REF 2014.”
[13] Hacking and Hacking, Representing and Intervening; Hacking and Hacking, The Social Construction of What?; Barnes, Bloor, and Henry, Scientific Knowledge; Latour, Science in Action; Latour, The Pasteurization of France; Kuhn, The Structure of Scientific Revolutions; Lakatos, Proofs and Refutations; Feyerabend, Against Method; Popper, The Logic of Scientific Discovery.
[14] I owe this observation to Judy Wajcman and her pioneering work.
[15] Goffman, Asylums, 127.
[16] Long, “Productivity and Academic Position in the Scientific Career.”
[17] Weber, “Science as a Vocation.”
[18] Shapin, The Scientific Life.
[19] Gelber, Grading the College.
[20] Espeland, Sauder, and Espeland, Engines of Anxiety; Espeland and Sauder, “Rankings and Reactivity.”
[21] Berman and Hirschman, The Sociology of Quantification.
[22] “Ratings Games.”
[23] Burrows, “Living with the H-Index?”
[24] Mills, “On Intellectual Craftsmanship (1952).”
[25] Lupton, The Quantified Self.
[26] Much of The Quantified Scholar is inspired by Michèle Lamont’s, How Professors Think., which highlights the tensions in interdisciplinary spaces (like grant and award review panels) where evaluations of quality and excellence are made across distinct and relatively insular academic fields. Fortunately, what I study here are mostly contained disciplinary struggles, which allows approximating in a slightly more idealized way the disciplinary cultures that characterize different fields. The reader should understand that these are always typifications of a much messier reality, and that while some commonalities may be present within fields, they are not by consequence defining.
[27] Brito and Rodríguez-Navarro, “Evaluating Research and Researchers by the Journal Impact Factor.”
[28] Fourcade, Economists and Societies; Fourcade, “The Construction of a Global Profession.”
[29] Strathern, Audit Cultures.