linking back to

My lab:
For the uninitiated, much of what follows must seem like a bizarre debate: many scientists really want to know if a scientific article is worth reading before they decide to read it. This in itself is not bizarre, yet, as we all want to know if we should go to that new restaurant, listen to that new band or watch that new movie. However, scientists don't ask people who have already read the article they ask the company which published the article. Even worse, some scientists even ask a company which has nothing to do with either the publisher, or the journal, or the article in question. Now that's really bizarre!

In science, journals publishing scholarly articles are ranked by a multi-billion dollar corporation (Thomson Reuters, TR) according to very opaque, unscientific methods. Of course, this corporation sells their ranking to scientific institutions for huge profits. My own university just told us yesterday that they are proud to now be able to afford this service starting next year.

But back to the scientists asking publishers or TR which scientific articles they should read. I would tend to believe that even these scientists, like all ordinary people, would either ask friends or consult newspapers or magazines, or even, gasp, the internet, for advice on where to eat, which concert is worth attending or if the new blockbuster is going to be worth spending a date on. Most likely, they would not take statements by the restaurant owner, or the band manager or the movie producer very seriously. These persons would have a rather obvious conflict of interest. However, it seems perfectly fine for these scientists to take professional editors at high ranking journals seriously, when they tell them they should read the articles they are selling in their own journal. How bizarre is that?

People use Guide Michelin because their staff actually taste the food in the restaurants. People use movie or concert reviews because the authors have actually seen the movie, been to the concert. It would seem very bizarre indeed if one would choose the concert exclusively because that particular venue had great concerts before. It would also seem bizarre if one would choose a restaurant because that particular street happens to have a good restaurant a few blocks down. Most people would probably also find it bizarre if people chose to see a particular movie only because of the movie studio it was shot in.

And yet, this is precisely what happens in science.

One reason it happens is that it is rare to find independent reviews of recently published papers. Most of these reviews are commissioned by the publisher and will understandably be fluff pieces. If anything, reading such a news piece on a scientific publication would often enough do the opposite and prevent scientists from actually reading the paper itself: they now already know its content, by and large. However, even if these pieces weren't commissioned and at least technically 'independent', you would still be hard pressed to find candid negative reviews as it's usually scientists writing about other scientists. F1000 Prime provides such an independent platform (in which I participate as a volunteer faculty member), but it is only set up for positive evaluations. Another thing that might seem quite bizarre from a non-scientist perspective: a culture of critical, investigative science journalism is only just now evolving.

However, the vast, vast majority of articles never receive any such coverage at all. Thus, in the absence of even fluff coverage, what is a researcher to do? In this case, he or she is in the same boat as someone looking for music, movies or restaurants: just like there isn't an objective measure for the best music, the best movie or the best food, there isn't an objective measure for the best science. So, since TR provides the veneer of something objective, scientists routinely use something that has been shown time and again to be unscientific, to the embarrassment of all other scientists. Just yesterday, at our faculty council meeting, one colleague mentioned the journals where a candidate had published as credentials of his/her excellence. Just as if the street where a restaurant is located, would tell anything about the quality of their food, or the record label would tell anything about the 'quality' of the music.

Some organizations have recently sprung up to remedy this, though: they collect data on various aspects of article impact irrespective of journal. One common criticism of such article level metrics is that the metrics have to accrue over time, while journal rank tells you immediately which articles are worth reading. Which is the last bizarre thing I'll write about today: how do you know which is the best music out of the music nobody has listened to, yet? How can you tell which is the best movie of those nobody has seen, yet? How do you figure out which is the best restaurant, of those where nobody has eaten, yet? For anything but science, the answer is obvious.
Posted on Thursday 15 November 2012 - 13:22:42 comment: 0

You must be logged in to make comments on this site - please log in, or if you are not registered click here to signup
Render time: 0.0746 sec, 0.0047 of that for queries.