1The idea of “impact factors” is to provide a measure of the quality of journals—as Clarivate says, they are used in “the process of academic evaluation” by providing a “gross approximation of the prestige of journals in which individuals have been published”.1 It was probably not really “prestige” that was intended. It must be apparent that the prestige of a journal is more likely to arise from its having a high impact factor, and related characteristics, than it is that prestige is a pre-existing characteristic that is miraculously measured by arithmetical operations on the citation counts of its contents within two-year windows.2
2The choice of that word is forgivable because of the inconvenience of providing a quick verbalization of the broad sense of the scientific or scholarly worth of a highly-cited paper, and hence of a journal containing papers of such worth. That worth might arise from, for example, a paper’s scientific insight; some other aspect of novelty even or perhaps from its being one that proved persuasive on some point, or containing an idea’s best exposition, even if it was not original in stating it. If frequent citation properly reflected such characteristics as these, then surely the counting of citations—and the classification of papers by citation counts as compiled from citation indexes such as Clarivate’s Web of Science (from which the Journal Impact Factor is derived)—might provide information about the quality of papers and the journals in which they are published.
- 3 The numbers were taken from the ISI Web of Science database on 28 August 2020. The most recent six (...)
3It is my objective in what follows, however, to cast doubt on the view that highly-cited papers can be reliably presumed to be worthy ones, whatever reasonable sense of “worth” one might prefer. I seek to do this principally by using the counter-example of Friedman (1968)—the publication of his Presidential Address given the previous year to the American Economic Association. It is a very highly cited paper, yet it is, so I have argued, a poor one of negligible intellectual value on any basis. The weaknesses of the paper should make it apparent that no prestige can attach to it, but as it happens this is thrown into very sharp relief by comparison with Friedman (1972)—an Address to the American Philosophical Society in 1971. That paper pursues much the same objectives as the earlier one, but is a much higher quality presentation. Yet, while Friedman (1968) has been cited exactly 2000 times, his 1972 paper scores a mere 12, and of those, the most recent are in historical treatments, so that its recognition in economics more broadly is practically nil.3
4In making my case, I draw heavily on aspects of my own previous work. The comparison of Friedman’s two papers arises from Forder (2018a). The point that there is nothing important and original in Friedman (1968) comes from the analysis of the paper in Forder (2018b) and of the prevailing theory of the time in Forder (2014). And my comments on the actual persuasiveness of Friedman’s paper are derived from Forder and Sømme (2019).
5As I argued in Forder (2018a), Friedman (1968) and Friedman (1972) argued essentially the same case—that monetary policy is better conducted according to rules rather than discretion. The latter paper however, was in every way I could think of, the superior piece of writing.
6The earlier paper contains no ideas that Friedman had not expressed before, it is badly organized, with arguments jumbled up, whole paragraphs apparently presented in the wrong order, and points relevant to one issue made in the middle of discussion of other issues. The paper contains very little evidence for any of Friedman’s claims; its citation of the literature is generally lacking, and specifically deficient, not just when other authors are credited by name with some idea, but when they are accused of error as well. It is chock-full of little mistakes, and five or six distinct failures of argumentation. None of the little mistakes, it is true, is fatal to Friedman’s wider argument, but they clearly contribute to making the paper an inferior work.
7Friedman (1972) presented some new analysis arising from his ongoing work with Anna Schwartz. All the major mistakes and very nearly all of the lesser ones from the earlier paper were eliminated. Some were corrected; others, being on points which were not of much relevance in the argument anyway, were simply dropped. Proper citations were introduced, and evidence presented for the significant claims Friedman was making, and the whole argument was straightened out so that points were made in a logical order. The combination of general similarity and specific differences suggests that when he came to write the second paper, Friedman himself could see that the first one was poor, and that being the case, there is really no reason for anyone else to deny it. Certainly, any later author with knowledge of his writings, and wanting a respectable source for Friedman’s views on these issues would ignore Friedman (1968) to cite Friedman (1972).
8People do sometimes write poor papers. Indeed, there are even poorly-written papers which present original and important ideas. So it is natural to suppose that Friedman (1968) has some striking characteristic that makes it worthy of frequent citation despite its numerous flaws.
- 4 My claim is impressionistic. There are many shades of meaning, but most of the time, when Friedman (...)
9Most likely it would be said that it was this paper that first made the point that ongoing inflation would change expectations and hence that any tradeoff between inflation and unemployment would be only temporary. Certainly this seems to be the most common ostensible reason for citing it.4 There are various responses to that. In Forder (2018b) I showed that pursuing such a point was certainly no part of Friedman’s objective. The paper is clearly framed as being about rules and discretion; the discussion of the Phillips curve is brief, receiving no special emphasis, and going unmentioned in both introduction and conclusion. Friedman’s later writings suggest the same conclusion—for some years he made nothing of the 1968 paper, and only started saying he had brought a great insight in about 1975, after others were already doing so.
10In any case, whatever might be said of Friedman’s intentions, it was a principal point of Forder (2014a, ch. 4, part 1) to put it beyond doubt that the expectations argument was a very old one, and a very routine one. Furthermore, Friedman ([1958] 1969) was one of those who had stated it. If he were to be given credit for devising it, there would be no case for citing anything as late as 1968.
- 5 After 1968 there was some debate about it, though more about whether the expectations effect was ac (...)
11It might then be argued that, these things notwithstanding, it was Friedman (1968) that brought the expectations argument to wide attention. Again, Friedman showed no sign of thinking the argument had not been recognized. But it is in any case thoroughly implausible that an argument that, as I showed, had been advanced by, amongst others, Bronfenbrenner, Christopher Dow, Haberler, Hayek, Hicks, Lerner, Morton, Karl Polanyi, Reder, Robbins, Robertson, Samuelson and Solow, Vickrey and Wallich had somehow not been noticed. The point here, it probably needs to be emphasized, is not that Friedman had “precursors”, but that there is no reasonable basis for saying that the argument in question was not simply commonplace, ordinary, and well-known to every economist. Related ideas, such as that although the argument was known, it was not accepted, and it was Friedman who persuaded economists of it, must fail for related reasons. Those others who had advanced the argument surely were persuaded by it, and if that is the case, it would be interesting to know who it might be who, as of 1968 or so, was doubting it.5
12The common sense of these points is, furthermore, supported by detailed analysis in Forder and Sømme (2019), of those who cited Friedman (1968) in the first few years after its publication. A striking feature of the literature is that it seems that before 1972, there was no one who attributed originality in the expectations argument to it, whilst there were several who noted it as a recent statement of a known argument. Clearly, those first reading Friedman knew the expectations argument perfectly well.
- 6 Except that as a referee has pointed out that Phelps (1967) and other papers by the same author are (...)
13So, I believe we are left, by default, with a different conclusion. That is that Friedman (1968) is so much cited for some reason other than that it has particular intellectual merit. The clearly apparent specific possibility is that it is cited because so many economists incorrectly believe it is the source of the expectations argument. The close reading of the literature in Forder and Sømme (2019) supports this view in that it suggests that the origins of this misperception can be dated to 1972. The point can hardly be said to be proven, but it seems likely that once this misperception emerged, no one saw a reason to question it.6
14The difficulty, I take it, is apparent. The explanation of the paper being highly cited has lost contact with what is supposed to be the reason for being interested in how often it is cited. It is not that Friedman introduced a new argument; it is not even that he brought the argument to professional attention. The general line of the paper is merely a rehash of views Friedman had previously presented, and in detail, it is not even a rehash, but merely a hash. It is very clear that citation counts are not measuring quality in any reasonable sense. Indeed, since the paper is so poor, its constant recognition suggests that the citation counts are not even measuring the kind of acquaintance that might arise from other authors having read it. And meanwhile there is a fine paper there that could be cited, but in fact is noted only by historians of economics and that, of course, is not influence of any kind.
15If the case of Friedman (1968) were unique, then that would make it an even more interesting case to study, but one would be reassured about the value of citation counts. It seems, however, that there are a number of strong candidates to be other, comparable examples.
- 7 My argument on those points is made in Forder (2019a, ch. 10)
16One would be another famous paper by Friedman—Friedman (1953)—his essay on the methodology of positive economics. It is often thought to be strikingly original for the idea that economic hypotheses or theories are to be assessed on the basis that the world works “as if” their assumptions were true. That again was not original to Friedman—it is in Knight (1922), who was one of Friedman’s teachers. There are other points that might be thought to be original in Friedman’s essay, but in fact they too turn out to be old ones, and again there is next to nothing any intellectual merit in the paper.7 The possibility that it might have achieved fame as the best exposition of those ideas can certainly be dismissed. The paper is full of all sorts of little errors and inconsistencies, and is indeed just the “dog’s breakfast” as which Dennis (1986, 367) identified it.
- 8 As described by Forder (2021), the “Phillips curve” came to mean all sorts of things so that discus (...)
- 9 Forder (2014) provides the basis for using the expression “great historical myth”, and Forder (2019 (...)
17Another case of a paper widely cited for making a supposedly original point which was in fact well known would be that of Phillips (1958). The idea that wages tend to rise more quickly when unemployment is low was certainly not one that broke suddenly into economics a good decade and a half into the age of demand management. The claim made by Lipsey (2020) that the paper caused excitement by offering a numerical value of the stable inflation rate of unemployment finds no support in the historic literature. Phillips’ paper picks up citations in connection with a range of other related questions,8 but there is surely no denying that the development of a great historical myth around his story has led to the expression “Phillips (1958)” appearing in the text of works where it had no authentic role and hence to that paper being “highly cited”.9
- 10 These points arise from analysis of the literature in Forder (1998) where the various arguments are (...)
18Developments with some resemblance to these can found arising from the 1980s literature on the idea of “policy credibility”. Rogoff (1985) is frequently cited in ways which overplay his role in advocating (or inventing the idea of) central bank independentce. Barro and Gordon (1983) are often cited in connection with credibility being analysed by means of the supposition that there are distinct “types” of policymaker. They were writing in that general area, but the specific idea comes from Backus and Driffill (1985). It is the idea of those authors that underlies many discussions of “Barro and Gordon (1983)”.10
- 11 The first was suggested to me as a possibility by Roger Backhouse, the second by Christina Laskarid (...)
19Further instances of similar phenomena are surely there—the citation of Stiglitz and Weiss (1981) for ideas that are no later than Jaffee and Russell (1976) would be one. Other possibilities that could be investigated are those of a confusion between the ideas of Solow (1956) and Solow (1957); and the tendency to cite specifically Epstein (2005) for what was at that time a routine definition.11 A difficulty about being sure just how widespread the problem is, though, evidently arises from the fact that only those with detailed knowledge of the relevant material are in a position to detect it. Otherwise, the character of the problem is that a great weight of citations appears to offer confirmation of whatever it is that it being claimed about a work and it might well seem foolish to doubt it—and certainly, it is much quicker just to include the same citation oneself and thereby, in a certain sense, display one’s familiarity with the literature.
20It is to be emphasized that there is no suggestion that in general citation counts cannot be explained. A referee has suggested that part of the explanation in the case of the two Friedman papers may be that the earlier one was initially presented to a large group of economists, whereas the later one had a more mixed audience. That is very reasonable. There are numerous such possibilities. One that would be interesting to have tested is that papers are more highly cited if their authors have many friends. Likewise, it is not as if there is no explanation of Beethoven’s Violin Sonata No 9 being known as the “Kreutzer Sonata”—it is just that the explanation is not that Kreutzer had anything to do with its composition, performance, and it is not even that he thought well of it. What is being suggested is that the case of Friedman (1968) shows that there can be no confidence that it is the intellectual worth of a paper that explains the frequency of its citation.
21I realise, though, that the argument I have made about Friedman will strike many as implausible. One response is that hope it will lead to a reading of the papers making the components of the case more fully. But another is to consider what it is that makes my conclusion seem implausible? The straightforward answer to that is that it is because Friedman’s paper is so widely regarded as so important. But that is substantially so not because every economist has made a detailed study of it and found the great merits that have so completely eluded me, but because since it is so highly cited, it is presumed to be an excellent paper. And there we are: Whether it be because Friedman had many friends, or whatever, the paper is highly regarded because it is highly cited. It is supposed to be the other way round.
- 12 This point was suggested by Christina Laskaridis.
- 13 The research of Salganik and Watts (2008) offers some parallels. There, listeners to music appear t (...)
- 14 The family is described in Forder (2021).
22In the development of such situations, there may just be a process something like the “Matthew effect” described by Merton (1968).12 He saw the most famous author amongst collaborators acquiring disproportionate credit for their joint discoveries. The initial position of being famous thereby led to greater fame. Similarly, all the papers I have discussed are surely cited by some because they were cited by others. Particular explanations of why that happened can certainly be offered. In the case of Friedman (1968), mention of the paper serves to label certain lines of thinking. For some, it is a badge of allegiance, showing their approval of the post-1980 consensus macroeconomics. Others, of course, are citing it on the same kind of basis, but because it represents a line of thinking which they oppose.13 Similarly, “the Phillips curve” became a label for a whole family of concepts and so appears in all kinds of discussions,14 with an obvious temptation to cite Phillips (1958) when it does. But in these cases, like “Matthew”, the citations are merely labels. The names and dates appear not as acknowledgement of any recognized intellectual debt, but more as a social obligation, or tribal ritual. It is as if, round the camp fire, when one shouts “stagflation” the others must chant in return “Friedman nineteen-sixtyeight; Friedman nineteen-sixtyeight”.
23The harm in this—the harm considered here—is not the misattribution or mislabelling itself. The harm comes with the construction of impact factors from the resulting literature. Impact factors are supposed to measure something loosely characterized as the “prestige” of journals. Clarivate’s Web of Science Journal Impact Factor in particular arises from a calculation in which citations over a two-year period play the critical role. “Prestige” that is measurable in this way is equivalent to prominence in citation indexes. Yet the case of Friedman (1968) shows that there is no reasonable sense in which the citation counts that yield such prominence connotes the appropriateness of scholarly acclaim. At least some high citation counts arise neither from the quality of the work nor the actual impact on anyone’s thinking.
24There is perhaps something to be said for impact factors, as compared to citation counts. It might be said that, whereas I have considered the failings of citation counts over periods long enough for papers to come to be cited more for reasons of habit or ritual than intellectual debt, an impact factor that considers citations over, say, a 2-year period may be free of that problem. Up to a point, that may be true.
25But we face other problems that diminish that point. In a world where, for most authors, the difference between two or three citations, or a dozen or twenty seems very large, to have a few undeserving cases scoring in the thousands distorts the whole picture provided by science citation indexes such as Clarivate’s Web of Science. Adopting the perspective offered here, one begins to see how numerous the undeserving cases may be. But there is more to it than that because there is also the difficulty that in social science, it should be the long-term influence of works that makes them laudable. The power of Keynes (1936) was not spent before the War. The importance of Adam Smith is not that he changed everything in 1777 and 1778, but rather that much later he is read with more intellectual profit than so many of his successors—including those with four-figure citation counts for some of their worst papers.
This paper is a revision of one presented at the History of Economics Society in New York, June 2019. I am grateful for the comments and ideas of Roger Backhouse, Mike Beggs, Pedro Duarte, José Edwards, Christina Laskaridis, Steve Meardon, Seamus Perry, conference participants, and anonymous referees.
Flux RSS
