The instance has been updated to version 0.19.4!

You can see the official release post and notes here: https://lemmy.ml/post/16585359?scrollToComments=true

Since this update required some manual intervention anyway to update the database version, I took the opportunity to perform some overdue server maintenance and upgrades.

I have migrated the server to a more powerful configuration, and changed the way image files are being stored and served (object storage as opposed to filesystem storage).

I am also aware of some federation issues that have affected this and other instances. I am not sure whether this upgrade will deal with these problems automatically. I will be looking into it and hopefully will implement some solution soon.

Sorry for the downtime experienced in the past few days, but hopefully with the more powerful server the experience will be smoother and will remain so for a while.

For other admins and users interested, this is a bit of information about the server

The previous configuration:

Hosting provider: serverspace.io

8 GB RAM

4 Core CPU

50 Mbps network

400 GB SSD

Cost: $55 / month

The 400 GB SSD was being used to store both the server’s database and the image files, and this component in isolation had a monthly cost of $36. I have migrated the images to an object storage provider ($6/month for 500 GB), and used the resources previously used for disk space to increase the other parameters. The new configuration is a lot better and less expensive.

New configuration:

Hosting provider: serverspace.io

Object storage provider: contabo.com

16 GB RAM

6 core CPU

50 Mbps network

100 GB SSD

500 GB Object storage

Cost: $47/month

Thanks a lot to the lemmy and pict-rs devs for the hard work they put into developing the infrastructure!

  • SalamanderOPMA
    6·
    5 months ago

    (@Shdwdrgn@mander.xyz ) I think it is related with the processing of so many activities per second while syncing with Lemmy.World.

    I have tried changing some rate limits and the number of maximum database connections, but I get similar errors while moving around the site. The server is running well below 100% on all metrics, and the logs don’t produce an obvious error. I will wait until we catch up with Lemmy.World and see if that fixes the issue. If it doesn’t, then I will trouble shoot more in-depth.

      • SalamanderOPMA
        3·
        5 months ago

        Thanks! The database is currently fetching lemmy world data as fast as possible, so this might be saturating the database. Once the syncronization is tested, I will try to tune postgres if it still happens.

        • Lvxferre
          3·
          5 months ago

          The issue seems to be gone for me. And the instance “feels” considerably faster than it did before :-D

          • SalamanderOPMA
            3·
            5 months ago

            Great, thanks for letting me know!! I did a bit of research and I think that the next time I need to upgrade the instance I will move it to a dedicated server instead of a virtual private server. But hopefully it runs smoothly for some time! :)

        • RobotToaster
          3·
          5 months ago

          I just resubbed to the communities that were causing issues and it looks like everything is working well now, thanks.

        • ShdwdrgnEnglish
          2·
          5 months ago

          Ah! It looks like it JUST fixed it, I had checked about 15 minutes ago and still couldn’t get my subscribed list to load, but it does appears to be loading up now. I’ll reset my preferences and keep an eye on it through the day – hoping for the best!

          • SalamanderOPMA
            1·
            5 months ago

            It should have been fixed about 13 minutes ago! Thanks for checking! If problems persist I will try to fine-tune the database settings as suggested above.

            • ShdwdrgnEnglish
              3·
              5 months ago

              There was a brief outage soon after I left that last reply, assume you were doing another round of maintenance at the time, but otherwise it’s still looking good from here.

              • SalamanderOPMA
                2·
                5 months ago

                Ah, thanks. I think that was when I reboot the instance to change a setting that controls how post thumbnails are generated, as the new setting was failing to generate thumbnails for many posts.