Are there any Discord servers or somewhere in the Matrix to chat about hosting a Lemmy instance? I’ve got Lemmy running, but I think there are several of us in the same boat struggling with federation performance issues and it might be good to have some place to chat real time.
Your original link is from 9 days ago, before the massive surge hit.
https://lemmy.world/post/56228 Came 8 days ago, with reports of it being pretty well saturated.
Remember- the big surge, is in the last 3-4 days.
Fediverse stats: https://fediverse.observer/dailystats
In the last 4 days, they have went up over 400% in size.
I don’t know if you’re totally missing it… here is the CPU usage from 3 hours ago: https://lemmy.world/comment/377946
Even if you 4x the usage from the alleged 400% growth, the spec of the server itself is not the bottleneck. They’ve also significantly increased the federation workers to 10000 based on my private chat… so something is not scaling to the fullest potential.
I think the point of focus should be more on why it is not using all the resources available, rather than ‘that server is weak’. We’re about to see a much larger influx up comes July 1st that’s going to make the 400% growth look like a joke, and if current larger instances aren’t able to handle the federation now, the current smaller instances will buck hard up comes the big move.
Ok- sorry- I 100% missed that.
I am onboard with you now.
Hopefully the upcoming 0.18 release I keep hearing about helps compensate for a few of these issues…
Quick skim through the commits on the master branch, I don’t see many changes pertaining to federation. This one looks interesting/related, but I think in itself only tells server admins when to increase worker counts: https://github.com/LemmyNet/lemmy/commit/25275b79eed0fb1fe90d27c197725f510f9965bb
Perhaps- https://github.com/LemmyNet/activitypub-federation-rust is the place we need to be focused on.
Perhaps… The main branch commits doesn’t have much either at this time either though: https://github.com/LemmyNet/activitypub-federation-rust/commits/main
So- poking around a bit tonight, and I did find this in the main lemmy repo.
https://github.com/LemmyNet/lemmy/blob/main/docker/federation
Interesting, as it actually appears to be a testing setup for testing federation.
Don’t know if they changed it or not-
But, I am busy making a few kubernetes manifests for deploying lemmy- and I am noticing a ton of extra debugging / logging that doesn’t need to be there for production use.
Seriously doubt it’s going to fix the issue- but, reducing some of the debugging enabled by default, wouldn’t hurt.
100% with you. A lot of the current deployment are very development centric… Let’s pretend the default
docker-compose.ymlisn’t opening up the postgres DB to the internet with a generic password…The pace the entire system must mature to enable the platform to handle the hopeful upcoming growth is… interesting, to say the least…
Well, if I can knock out a decent helm chart for these manifests, it might actually help a bit-
Most of the components can scale quite easily on k8s. The only piece I am unsure of currently, is lemmy itself.
The lemmy-ui scales. Appears mostly stateless. pictrs scales. postgres scales.