Academic Karma: a case study in how not to use open data

Update: in response to my feedback, auto-generated profiles without accounts are no longer displayed at Academic Karma. Well done and thanks to them for the rapid response.

A news story in Nature last year caused considerable mirth and consternation in my social networks by claiming that ResearchGate, a “Facebook for scientists”, is widely-used and visited by scientists. Since this is true of nobody that we know, we can only assume that there is a whole “other” sub-network of scientists defined by both usage of ResearchGate and willingness to take Nature surveys seriously.

You might be forgiven, however, for assuming that I have a profile at ResearchGate because here it is. Except: it is not. That page was generated automatically by ResearchGate, using what they could glean about me from bits of public data on the Web. Since they have only discovered about one-third of my professional publications, it’s a gross misrepresentation of my achievements and activity. I could claim the profile, log in and improve the data, but I don’t want to expose myself and everyone I know to marketing spam until the end of time.

One issue with providing open data about yourself online is that you can’t predict how it might be used. Which brings me to Academic Karma.

Academic Karma came to my attention on Twitter via Chris Gunter.

To which they replied:

Everyone with an ORCID? I have one of those. Sure enough, appending my ORCID ID to their URL reveals that I have a profile.

You’ll note that my profile states “no review information shared” and that the data are sourced from ORCID. These are recent changes, brought about by one of my less polite tweets.

Karma, apparently, according to someone

Karma, apparently, according to someone

Previously, profiles looked like the one shown in the image, right. In my case, as I have not included any reviewing or editorial activity in my ORCID profile, this resulted in a large, prominent “NA” for so-called “karma earnt”. This gave the misleading impression that I am a bad “corporate citizen”.

To their credit, the people behind Academic Karma made changes to profile views very quickly, based on my feedback. That said, they seemed genuinely bemused by my criticism at times.

So let me try to spell it out as best I can.

  1. I object to the automated generation of public profiles, without my knowledge or consent, which could be construed as having been generated by me
  2. I especially object when those profiles convey an impression of my character, such as “someone who publishes but does not review”, based on incomplete and misleading data

I’m sure that the Academic Karma team mean well and believe that what they’re doing can improve the research process. However, it seems to me that this is a classic case of enthusiasm for technological solutions without due consideration of the human and social aspects.

7 thoughts on “Academic Karma: a case study in how not to use open data

  1. Titus Brown

    I have some collaborators that use and value ResearchGate; I actually buy the survey results… the rest of what you say is spot on :)

      1. aeon

        Seems to me that RG has mainly European and Asian “active users”; and some African.

        Whole other sub-network indeed. Like, “not on Facebook”, perhaps? (Can’t check, I’m not on Facebook. xD )

  2. Lachlan Coin

    Hi Neil,
    It was not really my intention to try to build another research gate. The profiles were really only meant to support the peer reviewing platform we built – i.e. as a tool for users to see when they have done their fair share of peer review, not to make anyone look bad. I did try to make it clear that the profile was based on data from ORCID (this was always the case). Nevertheless I accept your criticism that it might look like you had created the profile. Anyway, given the purpose is as a tool for users to keep a track of their reviewing against publishing activity we are no longer displaying profiles of people who haven’t signed up.

    Cheers for the feedback

    Lachlan

  3. Pingback: Academic Karma: a case study in how not to use open data « Another Word For It

  4. John G

    I agree with your summary at the end; context is essential and content re-publishers such as ResearchGate and AcademicKarma are well-advised to mind how they go.

    That said, services like these, that aggregate public content and present it in new ways, can be extremely valuable. (And in fact, such reuse is one of the points of having open data and open identifiers.) I happen to be a heavy ResearchGate user, and appreciate very much the ability to see what it can tell me of my colleagues’ publications (among many other value added services). Similarly, I would be happy to be able to look up a researcher by ORCID and see what other information is associated with that ORCID.

    As members of RG appreciate, the content is highly dependent on its users’ contributions, as well as harvesting of other public information (always known to be incomplete and possibly containing errors), and it is indeed obvious if the user is not managing their account. So I don’t see any misrepresentation there, sorry. Possibly most of those in the “whole “other” sub-network of scientists” that actively use ResearchGate would say the same.

    1. nsaunders Post author

      Some good points. I’d point out that as an expert RG user you can distinguish user-curated profiles and are aware of potential errors and deficiencies. Probably not the case for many casual users who discover profiles via search engines.

Comments are closed.