We have been having a lot of server issues today where I have had to reboot the server every hour or so.
I have seen there is a lot of reports of scaling issues with the federation that they say may cause issues.
To test if this is the problem we are experiencing, I have turned federation off for a few hours, maybe overnight.
I will let you know as soon as I know more.
In the meantime, I apologize for the inconvenience.
I have made a few tweaks and turned federation back on.
Images will be down for a few days while I keep working on this
Thanks for the update
awww, hope things get better soon.
Me too.
V16 was very very stable for us.
V17 has been a disaster. This could in part be caused by active Lemmyverse users increasing by a factor of 20 in the last month. It could also be some of the new instances we federated with trying some funky stuff.
In any case I know the devs are working very hard. I have never seen do many different people contributing to the development effort.
So despite these scaling problems, the future looks bright.
Bring on V18
Good chance it’s just increased usage from all the new federated users. Sounds like the lemmy devs have a plan to optimize things. There may be some low hanging fruit optimization we can look into doing with the postgres config. I’ll try to find some time into reading the current default configs.
“You can use the PGTune tool to tune your postgres to meet your server memory and CPU.”
from https://github.com/LemmyNet/lemmy-ansible
It was configured to a much larger machine. So I plugged in the specs for my linode and updated it.
I also upgraded to V17.4 even though that breaks images for some reason.
I figured better to be up and running smoothly with no images, then up and down all the time.
I will watch the server today and see if it performs better and let you know.
Fingers crossed.
Good news, these tweaks have fixed the cpu spikes and the Disk i/o spikes.
You should be seeing better performance.
Still researching what is causing the images not to display.
I did notice the website going in and out all night.
Also the number of federated posts just jumped big time. We were at something like Ninety thousand total just a couple weeks ago and now we’re at over 125,000 new OPs.
Yeah I have been up working on it since 2am this morning.
Not much to show for my efforts yet sadly.
Onwards and upwards we go.
My posting had slowed down because of it but hey it’s a marathon, not a sprint, and it seems to be back on track now.
The server is running a lot smoother today compared to yesterday. I have been doing a lot of small things tonight.
I do truly appreciate all of this hard work considering how difficult the situation must be with the sudden influx in the Fediverse and all the moving parts that must go with this.
Persistence conquers all :-)
