Friday, June 13, 2008

Do citations in supplementary data not count in ISI?

I came across something troubling today in the supplementary information that accompanied a paper I was going over. Now I have a lot of misgivings about the whole concept of supplementary data, yet I hadn't realized this one until now.

That is, in the Thomson ISI database, citations made in supplementary data are not counted in the "Times Cited" count. That's worrying to me, because one of the most prevalent ways for people who are not experts in a given field (or subfield, or subsubfield...) to assess the influence of a scientist or paper is to use the number of citations the paper garners. For example, these citation counts are used to create the impact factor to rank journals (and at times erroneously to rank individual papers), and the Hirsch, or h-, index used to evaluate scientist output.

Meaning, if these citations are not included in ISI, then for all intents and purposes they do not exist.

So now, not only can supplementary info be used as a dumping ground for your inconclusive or crappy data, but you can also stick references to your competitors in there and shaft them their citations.

But hey, at least you've got your plausible deniability!

Anyone else troubled by this? Anyone else have opinions on supplementary data?


juniorprof said...

ISI is a royal pain in the ass, no other way around it. Their monopolization of the process of citation counts and IFs coupled with their inability to put together meaningful search function makes them more or less useless in my opinion. I much prefer SCOPUS but my institution doesn't want to pay for ISI and SCOPUS at the same time. I'm trying to convince people that we should switch. Interestingly, the library admin people are on board with the idea. The Dean's office doesn't think its such a grand idea though.

BTW, nice to see someone working on TRP channels starting up a blog. I got my start in TRP channels and cannabinoids (former Hargreaves' lab member) and I dabble in them from time to time now. You're in a great lab, I look forward to seeing what you have to say about TRP channels, could make for some lively conversation.

Dave Bridges said...

I love online supplementary data. No longer can people just toss something out there and suggest (data not shown). On that front i find it really increases the number of relevant control experiments, and information on methods, especially in "limited space" journals like nature/science.

On the other hand, just throwing in tons of inconclusive or irrelevant experiments seems to becoming problematic as well (we dont really understand this, but at least our supplementary figure has thirty five panels).

As for citations, they absolutely should be included, and probably in the main paper reference section (even if numbers are skipped in the main text)

Nat Blair said...

@jp - I'll have to check out SCOPUS. I think I've heard about it, but haven't used it. And looking through the Harvard library resources, I don't immediately see it. Maybe tomorrow I'll email a librarian there, and see what they have to say.

There are certainly lots of interesting TRP channel stuff going on, but many of them are also turning out to be difficult to hack away at, esp. those with little to no useful pharmacology. But we're making progress. Still, wider and faster discussion would be worthwhile.

@dave bridges - That is one good point of supplementary data. As a replacement for "data not shown" is good, but for me "data not shown" was easier. I just ignored it, as though it never existed. Now when it's in supplementary data I feel obligated to check it out.

And as for limited space journals, that's true as well. But I have always been of the opinion that stuff published in limited space journals (e.g. Nature, Science esp) are really just the tip of the iceberg, and an advertisement of the general finding. I've always liked it when the same lab group follows it up with a more detailed article in a different journal. But I kinda get the feeling that this is happening less and less these days (likely resulting from the rise in supplementary data and just less inclination to spend time on a lower profile publication). Given the inherent contigency of scientific knowledge, I am kinda dismayed by that.

As for citations, maybe I'll email folks at Thomson. We'll see if I get any response.

Anonymous said...

I agree that being to have online supplementary data is better than not having it. I can't tell you how many times I have seen "(results available from authors upon request)" and emailed the authors for the data, only to be denied. Typically the denial is plausible, eg "my computer crashed", "I changed computers and the data are on my old computer", etc. But still inexcusable.

Nat Blair said...

That's actually one good point in favor of supplemental data: you rarely see the "Data Not Shown" nonsense of yesteryear.

Supplemental data has its place. But, beyond possibly unindexed citations, it is the cramming of entire papers-worth of data into the supplement that is problematic. That information doesn't get thoroughly reviewed, doesn't get it's own complete treatment and analysis, and doesn't result in enough credit the trainees who actually did the work.