Are there any Discord servers or somewhere in the Matrix to chat about hosting a Lemmy instance? I’ve got Lemmy running, but I think there are several of us in the same boat struggling with federation performance issues and it might be good to have some place to chat real time.
That is a VERY small server…
MY server, has 32 cores, 64 threads, 256G of ram, and 130T of storage (4T of which is NVMe)
Sheesh, that is prob why that instance is dragging!!
https://lemmy.world/post/56228
They’ve bumped the server much more than the original posted VM. I was pointing to the zabbix charts and actual usage. Notice CPU is sub 20%, and the network usage being sub 200Mbits. There’s plenty of headroom.
I found the newest link- https://lemmy.world/comment/379405
Ok, that is a pretty sizable chunk of hardware.
I care less about what it is running on, but what is consumed. At sub 20% usage, it really doesn’t matter what the hardware is, because the overall spec is not the bottle neck.
Your original link is from 9 days ago, before the massive surge hit.
https://lemmy.world/post/56228 Came 8 days ago, with reports of it being pretty well saturated.
Remember- the big surge, is in the last 3-4 days.
Fediverse stats: https://fediverse.observer/dailystats
In the last 4 days, they have went up over 400% in size.
I don’t know if you’re totally missing it… here is the CPU usage from 3 hours ago: https://lemmy.world/comment/377946
Even if you 4x the usage from the alleged 400% growth, the spec of the server itself is not the bottleneck. They’ve also significantly increased the federation workers to 10000 based on my private chat… so something is not scaling to the fullest potential.
I think the point of focus should be more on why it is not using all the resources available, rather than ‘that server is weak’. We’re about to see a much larger influx up comes July 1st that’s going to make the 400% growth look like a joke, and if current larger instances aren’t able to handle the federation now, the current smaller instances will buck hard up comes the big move.
Ok- sorry- I 100% missed that.
I am onboard with you now.
Hopefully the upcoming 0.18 release I keep hearing about helps compensate for a few of these issues…
Quick skim through the commits on the master branch, I don’t see many changes pertaining to federation. This one looks interesting/related, but I think in itself only tells server admins when to increase worker counts: https://github.com/LemmyNet/lemmy/commit/25275b79eed0fb1fe90d27c197725f510f9965bb
Don’t know if they changed it or not-
But, I am busy making a few kubernetes manifests for deploying lemmy- and I am noticing a ton of extra debugging / logging that doesn’t need to be there for production use.
Seriously doubt it’s going to fix the issue- but, reducing some of the debugging enabled by default, wouldn’t hurt.
Perhaps- https://github.com/LemmyNet/activitypub-federation-rust is the place we need to be focused on.