As quoted from the linked post.

It looks like you’re part of one of our experiments. The logged-in mobile web experience is currently unavailable for a portion of users. To access the site you can log on via desktop, the mobile apps, or wait for the experiment to conclude.

This is separate from the API issue. This will actually BLOCK you from even viewing reddit on your phone without using the official app.

Archive.org link in case the post is removed.

https://web.archive.org/save/https%3A%2F%2Fold.reddit.com%2Fr%2Fhelp%2Fcomments%2F135tly1%2Fhelpdid_reddit_just_destroy_mobile_browser_access%2Fjim40zg%2F

  • Laxaria@beehaw.org
    link
    fedilink
    arrow-up
    3
    ·
    edit-2
    2 years ago

    Reddit violates ALL of these example rules.

    No.

    minors use reddit and there is no indication that reddit experiments exclude them. Minors are not prohibited on the site and there is no tracking of age other than the vague “show me NSFW results” checkbox

    Strictly speaking, COPPA prevents Reddit from collecting information from users under the age of 13. While there are no explicit guarantees that a person on the site isn’t 13 or older, and also recognizing that age of majority is typically 18, then in a general literal sense yes there are minors involved, in so far as the activity discussed is research.

    intervention is not brief, has lasted a week or more

    A research intervention is an intervention in so far as it intersects with the participant. A drug trial that lasts for 10 weeks total but only gives doses to participants for 2 weeks (and then results monitoring during, 2-weeks post, and 4-weeks post) does not mean that the “intervention” lasts 10 weeks.

    In most practical terms, running an experiment for 2-3 weeks is very common to collect sufficient data. However, the intervention itself may be quite brief (for example, a short 45-minute interview with a participant would be the “intervention” for a 2-3 week long study interviewing physicians on their concerns about organizational capacity for change).

    For a repeated measures experiment, the intervention usually involves the actual experiment encounter and maybe some additional time between them.

    For the case of A/B testing, usually the “intervention” in this case is the A/B test as it applies to the user at the moment, and not the entire duration of when the testing is taking place for all participants.

    interventions having harm

    Once again, to iterate, you are equating the inability to view the content of a website without logging into an account with such substantial emotional and psychological harm that is comparable to being verbally derided in public, for a week, shamed on a public channel, and/or comparable situations. You are not going to convince an IRB that being able to view the content of a website without logging in, then subsequently going to a different community to ask for help, and then hypothetically ruminate about the matter for weeks, is going to exceed the kind of everyday ordinary harm to qualify a risk level above minimal risk.

    the subject was deceived and was definitely not informed prospectively that deception may take place, neither has agreed to it; subject was not informed even retrospectively other than some random admin suggesting they were part of an experiment after they complained online; for that matter the subject was not informed that an experiment would be taking place at all and has never agreed to anything, other than possibly in the ToS.

    Deception goes beyond simply “lying” to or “not informing” the participant. Duke University gives some good considerations here:

    • If, in order to counter the demand effect, researchers cannot disclose their research hypotheses, the failure to disclose is not considered deception.
    • General statements about the purpose of the research, as well as a full description of the research tasks and activities, should be provided in the consent form. (emphasis, should, not must).

    Additionally, a waiver for utilizing deception in research has to:

    1. The risk must be no more than minimal.
    2. The rights and welfare of the subjects will not be adversely affected.
    3. The research could not practicably be carried out without the waiver. This does not mean that it would be inconvenient to conduct the study without the waiver. It means that deception is necessary to accomplish the goals of the research.

    Satisfying #1 and #2 in an UI/UX A/B testing regime that Reddit used here is pretty easy. You are specifically hung up on the implicit harms involved but in reality they are of no particular serious concern.

    #3 is particularly interesting, because effectively what that means is you need to demonstrate deception is necessary in experimental design for the experiment to actually work. If you are seeing whether a person will interact more with a site or not if they are blocked from seeing content without logging into an account, telling them ahead of time could already bias the outcome. This is in very specific consideration that #1 and #2 are already met: being unable to view a website without logging into an account is not anything more than minimal risk. And even then, it is important to emphasize that the failure to disclose the research hypothesis to counter the demand effect is NOT deception.

    The kind of UI/UX A/B testing Reddit employed in this specific instance is absolutely not equivocal to the Asch conformity studies.

    To be very clear here, when YOU operate under the belief that being unable to view the content of the website, then posting about it elsewhere and potentially being ridiculed for it is sufficient of a bar to meet beyond minimal risk, then we have very different definitions of what “minimal risk” entails. Since we cannot come to consensus on this particular topic, and instead you’ve gone so far as to associate this kind of activity’s harm of equivalent to the Asch conformity studies is frankly ludicrous.

    If we cannot agree on this, then so be it. However, I will repeat (with added finality): YOU running the same A/B experiment Reddit is doing on an university-sanctioned website will NOT get you run out of the university by the IRB. In a real-world scenario. you would likely discuss this experiment with an administrator or similar at your department, then maybe send an e-mail off to the IRB for any clarification, would likely have some back-and-forth, and then would ultimately receive a determination that it is exempt (at worst), or not considered human subjects research at all. I can see a few circumstances where such an effort might merit an expedited review, but to do this would involve some torturous twisting of the situation that could be easily avoided (for example, running this testing on a page that has instructions for performing the Heimlich maneuver).

    The Asch conformity study experiments are absolutely not equivalent to doing A/B testing on a wide scale with regards to how users interact with a website when presented with a pop-up preventing further interaction without logging in.

    • TauZero
      link
      fedilink
      arrow-up
      1
      ·
      2 years ago

      Thank you for agreeing that Asch conformity experiment falls under human experiment ethics considerations and informed consent requirements! I am surprised though that you consider the reddit experiment unlike the Asch experiment (I personally see reddit as the worse one actually!). Could you explain how in your mind, Asch does carry a risk of causing harm in a way that reddit does not? Asch is just looking at a bunch of lines on a page after all! How can a bunch of lines cause harm? I also find it odd your cavalier attitude towards the word “should”.

      General statements about the purpose of the research, as well as a full description of the research tasks and activities, should be provided in the consent form. (emphasis, should, not must).

      If “should” isn’t prescriptive, why even have any “should” statements in our guidelines if you are just going to ignore them all? And yet again your link disagrees with you:

      • The risk must be no more than minimal. “Minimal risk means that the probability and magnitude of harm or discomfort anticipated in the research are not greater in and of themselves than those ordinarily encountered in daily life or during the performance of routine physical or psychological examinations or tests.”
      • The rights and welfare of the subjects will not be adversely affected.
      • The research could not practicably be carried out without the waiver. This does not mean that it would be inconvenient to conduct the study without the waiver. It means that deception is necessary to accomplish the goals of the research.

      Ameliorating Deception

      Protocols must include procedures for ameliorating possible negative effects of deception. In addition to thorough debriefing that explains the need for deception, emphasis should be placed on correcting any false feedback given to participants about their performance, competency, or other personal characteristics.

      Participants whose behavior was recorded without their knowledge, such as during a fake “break” in study, should be given the opportunity to request that the recording be destroyed.

      If a study was designed to provoke negative behaviors, participants should be told that most people react the way they reacted and that their behavior was a normal response.

      Debriefing

      Debriefing for participants who were deceived includes a description of the deception and an explanation about why it was necessary. The discussion should presented in lay language and should be sufficiently detailed that participants will understand how and why they were deceived. If the study included multiple deceptions, each should be addressed. If participants were filmed without their knowledge, they must be given the option to ask that the researchers do not use the film

      Reddit never had any intention to ameliorate the deception, to debrief the participants, or to give them an opportunity to delete their experiment records after the fact. Reddit never implemented alternative practical research methods like opt-in studies. Are you seriously arguing that because the page says “should” and not “must”, reddit was perfectly ethical simply not doing any of this at all? This isn’t some RFC, this is normal people language!

      Or if you are saying this wasn’t deception, then why link to the entire Duke deception page at all? The only relevant sentence here to you is:

      If, in order to counter the demand effect, researchers cannot disclose their research hypotheses, the failure to disclose is not considered deception.

      And it only refers to the disclosing the research hypothesis itself, not the very fact that you are taking part in some experiment! And you agree that it is not impossible to perform usage experiments without informing participants in advance (I brought up Firefox as a better example alternative), it is just more laborious. Moreover, reddit did engage in actual deception beyond simply keeping the fact of the experiment secret:

      being unable to view a website without logging into an account is not anything more than minimal risk. And even then, it is important to emphasize that the failure to disclose the research hypothesis to counter the demand effect is NOT deception.

      What happened went beyond that. If you read the reddit OP:

      I’m logged in on my phone (iOS) but I use a browser, not the app. As of an hour ago, the mobile view is showing that I’m logged out, with no option to log in and a permanent “this looks better in the app” banner on the page.

      This isn’t some simple A/B testing of things like text size or link color. This isn’t like Facebook or Instagram blocking everyone equally from seeing communities without logging in. OP was logged in. Reddit lied to them saying they were not logged in when they were. Reddit lied to them saying there is no way to log in. Reddit lied to them saying the only way to see the content was to download the app. This is the deception part. This is the part that’s similar to Asch and the people in the room with you lying that they are participants like you. You think you are in a normal situation but you are not. You’ve been singled out and no one believes you.