Can SSRN be gamed?

Gordon is down on SSRN as a measure of scholarly performance because of the risk faculty and schools would game the system.  He fears that authors could download from multiple addresses, or have friends, students, alumni do the downloading.

I’m not buying Gordon’s argument. The basic problem is that the argument doesn’t address at all norms and reputational sanctions. As Black and Caron argue, “even a skilled and determined gamer will be cautious, for fear of reputational harm if his actions become known.”   Reputational harm is not merely the loss of the reputational credit gained from gaming the system, but the shame and disgrace for violating a community norm that the members of the community have internalized. 

The norm here is clear – don’t take academic credit you don’t deserve.  It’s well enforced among the group because each group member has a strong reputational incentive to make sure the system is accurate.

Of course there’s no penalty without detection.  But detection over the long haul – which is the period over which reputations and careers are made – is very likely. Even given the current technology, which is not strongly geared to detection, an author himself can only create a limited number of downloads because of the need to use multiple ip addresses. Big numbers would require outsourcing, thereby significantly increasing the risk of detection.  Gordon even assumes the possibility of advertising for downloaders, which busts the game wide open.

Moreover, there’s no reason to assume current technology.  SSRN is contemplating logins for downloads.  If SSRN starts gaining significant network rents, it could hire enforcers.  Like the SEC and insider trading, it could investigate download spikes like the one Gordon proposes for his own works.  To evade that sort of detection, the download scam would have to occur gradually over a long period, thereby increasing the risk of whistleblowing. Meanwhile, scammers would be exposed to blackmail and extortion by all of those who are in on the scam.

Putting all this in a formula, one would game SSRN if the probability of success x amount of payoff > probability of detection x harm if detected, both discounted to present value.  The probability of success and payoff depend on the credibility of the SSRN system.  So there would be a high probability and high payoff only if we assume that SSRN is not commonly gamed. Moreover, the discount depends on the time frame, which is likely to be long on the left side of the equation.  On the right side, if we’re assuming the system is working, then we’re assuming that norms and monitoring are in place.  For the reasons given above, both the penalty for being caught before payoff and the penalty if caught are pretty high.

The bottom line is that I see zero risk that the kind of gaming Gordon fears would seriously undermine the system.

The problem with using SSRN, as with any single statistic, is not that it will be gamed, that it doesn’t measure everything. For example, I see little correlation between the downloads my papers get and my own estimation of their scholarly merit.  Downloads measure topical interest and extraneous factors like brevity. Citations are a better measure of scholarly impact, but they’re not perfect either for reasons that many have written.

So why not use a combination of citations and downloads?

Comments

[Kate Litvak]

Larry is my long-lost twin brother...

[Larry Ribstei]

Wow! Who knew?