I have a fair amount of experience with data visualization, analysis etc and thought it would be a fun project to try to visualize the Lemmy network, specifically which instances have strong links to one another via subscriptions from users in one to communities in the other.

How/where can I get that data?

EDIT: It sounds like many people would find this a violation of their data privacy and I simply shouldn’t do it. I had thought this kind of data was intended to be entirely accessible by design, but I learned something new!

  • gabe [he/him]
    link
    fedilink
    19
    edit-2
    7 months ago

    Likely within the API, but I would say be cautious as people on the fediverse really do not like having their data scraped, especially for projects like this.

    • alex [they, il]
      link
      fedilink
      127 months ago

      I think this is fine - people I’ve seen who objected to this kind of project were more about their account being indexed. Projects like respective size of instances were always fine

    • @mookulatorOP
      link
      97 months ago

      Interesting. Isn’t it entirely public though?

      • gabe [he/him]
        link
        fedilink
        97 months ago

        Yes, still doesn’t make people fine with it. People defederate over stuff like that on mastodon. It’s often seen as violating peoples privacy.

        • @mookulatorOP
          link
          -37 months ago

          lol I won’t make you defend these people’s logic. Thanks for the warning!

            • @mookulatorOP
              link
              2
              edit-2
              7 months ago

              What do you mean? I thought it was an explicit feature of this place that literally all of it was public and nobody owns any of the data. Isn’t it just sitting there in the public domain?

              If people know that’s how it works, they can’t get mad if someone does access the data. Especially for innocuous curiosities like this.

              • @Dave@lemmy.nz
                link
                fedilink
                107 months ago

                The content of lemmy is in public, but is not in the public domain. They are different concepts.

                • @mookulatorOP
                  link
                  37 months ago

                  Thanks for the correction. I guess is see the point

              • @unexpectedteapot@lemmy.ml
                link
                fedilink
                67 months ago

                Your public domain assumption doesn’t have to apply to others, legally or ideologically.

                Data ownership does exist in the Fediverse, in fact it is one of its selling points that you can set up your server and own the data instead of using a surveillance capitalist SaaS that stores, manipulates and imposes legal rights over your data. Applications like Mastodon do send a federation request to other instances to delete data if submitters want to. Additionally, some users put licenses on their profile that might have restrictions (i.e: CC non-commerical, etc.) on what you are legally allowed to do with the data.

                So no, accessing the data is not the same as using or processing it for many people, legally too in several parts of the world. Also, “innocuous curiosities” label is entirely subjective.

              • @PseudoSpock@lemmy.dbzer0.com
                link
                fedilink
                47 months ago

                I think the more commonly held belief is that the data, while unfortunately available right now, was going to become more secure in the near future; but the exodus from reddit happened too soon. So now there is a lot of that data, and the better management and protection of it hasn’t had time enough to happen.

                In comes you, seeing the opportunity, and you seek to exploit it.

                • @mookulatorOP
                  link
                  17 months ago

                  I didn’t know that about the plans. I thought it was by design. Thanks I won’t do this project then

  • spectre [he/him]
    link
    fedilink
    English
    37 months ago

    I mean the fediverse isn’t exactly super mature; from what I know you should expect that you’ll have to generate that data yourself some way or another.

  • @Danterious@lemmy.world
    link
    fedilink
    17 months ago

    Personally I would love to see this kind of project done. But for it to work most people on the servers data that you are working with would have to be informed and given time to answer. I hope you reach out to the administrators of each instance and ask them if they would be ok with this and give them time to ask their users. Knowledge is power and if the visualization was public I think it could be helpful.

    • @mookulatorOP
      link
      27 months ago

      Thanks for the support! I thought it would be a useful tool too. People could use it to find instances they didn’t know about, but which are popular in their network, or to find under appreciated instances to build bridges.

      Anyway, sounds like database access is a must, and I’m not trying to take on a massive data wrangling exercise. I had it in my mind that the data were just sitting their for download, which was wishful thinking!

      • @Danterious@lemmy.world
        link
        fedilink
        27 months ago

        If you are interested there is a professor named Damon Centola that might be interested in dedicating time to gathering that data and might help with the visualization as well. If you are in the sociologist space or just interested you could reach out to him.

      • @Danterious@lemmy.world
        link
        fedilink
        16 months ago

        Sorry for commenting again but I think there is a way for you to do this in a completely open, easy, and privacy-preserving way. You don’t need to access their database.

        1. Get a list of instances that you want to look at the subscription patterns for. (All the instances here + Lemmy.world)

        2. Go to that instance’s website, click the “Communities” tab at the top, and then click “All” It shows how many users from that instance are subscribed to that community (both communities from that instance and outside of that instance)

        If you find a way to automatically (or manually) scrape this data from all of those websites you can create the visualization that you were talking about.

        So you were right, the data is open source it is just specific to each website.

  • BlueÆther
    link
    fedilink
    17 months ago

    I don’t think you could do it without DB access, but not 100% sure