The ACM.org website published the work of a team at Carnegie Mellon (#CMU) which was said to include source code. Then the code was omitted from the attached ZIP file, which only contained another copy of the paper. I asked the lead researcher (a prof) for the code and was ignored. Also asked the other researchers (apparently students), who also ignored the request. The code would have made it possible to reproduce the research and verify it. ACM also ignored my request and also neglected to fix the misinfo (the claim on the page that source code is available). Correction: ACM replied and tried to find the missing code but then just gave up.

It seems like this should taint the research in some way. Why don’t they want people reproducing the research? If the idea is that scientific research is “peer reviewed” for integrity, it seems like a façade if reviewers don’t have a voice. Or is there some kind of 3rd party who would call this out?

  • themoonisacheese@sh.itjust.works
    link
    fedilink
    arrow-up
    18
    ·
    2 months ago

    I have a contact at cmu’s student journal (for what that’s worth). They might be interested in this, maybe if they start prodding around the researchers will realize the mistake. What’s the publication?

    • plantteacherOP
      link
      fedilink
      arrow-up
      5
      ·
      edit-2
      2 months ago

      Oh, wow… I wasn’t expecting that reply. I was actually looking to discuss in general how to address this variety of issue. It was a few years ago but the code would still be interesting to see. I dug this up:

      https://dl.acm.org/doi/10.1145/2911988

      And now that I dug back into this, I must make a correction. ACM replied to say they are looking for the missing material… then they never found it and they dropped the ball at that point and also neglected correct the description. AFAIK, ACM did not try to reach the researchers, who ignored my inquiries.

      (Irrelevant trivia: ACM used to be in Cloudflare’s access-restricted walled garden, making it difficult to access research. They are still in that shitty place but at least they are now whitelisting Tor which slightly reduces their exclusivity.)

        • plantteacherOP
          link
          fedilink
          arrow-up
          2
          ·
          2 months ago

          They could try to say that but I doubt people would believe it.

          Who throws away their own code particularly when it’s not junky commercial code but code their heart and soul was behind on a non-profit project? I keep my old code around if anything just to be able to search it to re-teach myself coding and design tips I forgot about. This code backs their research which they may need to refer to when a prospective employer asks for detail on how they executed the study.

          • jatone@lemmy.dbzer0.com
            link
            fedilink
            English
            arrow-up
            2
            ·
            2 months ago

            Thats a you decision. Many people dont actually care about their phd or research studies. And this particular one looks very throw away.

  • drspod@lemmy.ml
    link
    fedilink
    arrow-up
    8
    ·
    2 months ago

    Who funded the research? They probably have the source code and want to keep it proprietary.

    • plantteacherOP
      link
      fedilink
      arrow-up
      7
      ·
      edit-2
      2 months ago

      Maybe the acknowledgments gives a hint?

      ACKNOWLEDGMENTS We would like to thank Kelly Idouchi, Manya Sleeper, James T. Graves, and Celine Berger for their contributions to this project. Similarly, we thank Chris Hoofnagle, Daniel Solove, and the attendees of the 2014 Privacy Law Scholars Conference (PLSC) for valuable feedback on an earlier version of this work.

      (edit) there is also this about page and perhaps this lab was involved.