this post was submitted on 01 Jul 2023
3729 points (97.1% liked)

Lemmy.World Announcements

29185 readers
2 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages πŸ”₯

https://status.lemmy.world/

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations πŸ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 2 years ago
MODERATORS
 

Looks like it works.

Edit still see some performance issues. Needs more troubleshooting

Update: Registrations re-opened We encountered a bug where people could not log in, see https://github.com/LemmyNet/lemmy/issues/3422#issuecomment-1616112264 . As a workaround we opened registrations.

Thanks

First of all, I would like to thank the Lemmy.world team and the 2 admins of other servers @stanford@discuss.as200950.com and @sunaurus@lemm.ee for their help! We did some thorough troubleshooting to get this working!

The upgrade

The upgrade itself isn't too hard. Create a backup, and then change the image names in the docker-compose.yml and restart.

But, like the first 2 tries, after a few minutes the site started getting slow until it stopped responding. Then the troubleshooting started.

The solutions

What I had noticed previously, is that the lemmy container could reach around 1500% CPU usage, above that the site got slow. Which is weird, because the server has 64 threads, so 6400% should be the max. So we tried what @sunaurus@lemm.ee had suggested before: we created extra lemmy containers to spread the load. (And extra lemmy-ui containers). And used nginx to load balance between them.

Et voilΓ . That seems to work.

Also, as suggested by him, we start the lemmy containers with the scheduler disabled, and have 1 extra lemmy running with the scheduler enabled, unused for other stuff.

There will be room for improvement, and probably new bugs, but we're very happy lemmy.world is now at 0.18.1-rc. This fixes a lot of bugs.

(page 5) 50 comments
sorted by: hot top controversial new old
[–] kazerniel@lemmy.world 4 points 2 years ago

It keeps logging me out, plus when I log in, alarmingly other users' profile names flash in the top right corner for a second.

[–] plumbercraic@lemmy.world 4 points 2 years ago

Excellent. Now I can login with jerboa πŸŽ‰

[–] darkbaron202@lemmy.world 4 points 2 years ago

thank you for letting us know behind-the-scene stuffs.

me myself is a sysadmin and really like story of successfully scaling up servers. very satisfy to hear.

once again, thank you!

[–] nico@r.dcotta.eu 4 points 2 years ago* (last edited 2 years ago)

Have you considered running your Lemmy instance on more than a single machine? If it is possible to run two lemmy containers anyway (ie, lemmy is not a singleton), why not run them on separate machines? With load balancing you could achieve a more stable experience. It might be cheaper to have many mediocre machines rather than a single powerful one too, as well as more sustainable long-term (vertical vs horizontal scaling).

The downside would be that the set-up would be less obvious than with Docker compose and you would probably need to get into k8s/k3s/nomad territory in order to orchestrate a proper fleet.

[–] Rooki@lemmy.world 4 points 2 years ago (1 children)

Is there a issue with the api? ( Because the api wrapper lemmy-js-client doesnt work on login. ) I tried it yesterday but not today yet. I will test it when i can :)

load more comments (1 replies)
[–] Netrunner@lemmy.world 4 points 2 years ago

I can finally login!

[–] danielfh@lemmy.world 4 points 2 years ago

Thank you!!

[–] stickyfranklin@lemmy.world 4 points 2 years ago
[–] nucleative@lemmy.world 4 points 2 years ago

Working well here and can use Jerboa again. Although wefwef is really growing on me!

Edit: couldn't post from Jerboa, got network error. But wefwef worked.

[–] cute_cat@lemmy.world 4 points 2 years ago

I really enjoy how lemmy is growing!

[–] feitingen@lemmy.world 3 points 2 years ago

It seems there is still some performance issues.

Maybe consider a webcache like varnish to take some of the load off?

[–] DelvianSeek@lemmy.world 3 points 2 years ago

Just want to say thank you. Your hard work is very much appreciated.

[–] Guster@lemmy.world 3 points 2 years ago

Nice work :)

[–] cute_cat@lemmy.world 3 points 2 years ago

I really enjoy how lemmy is growing!

[–] ChriskiV@lemmy.world 3 points 2 years ago

Signed into Jerboa! Thanks devs!

[–] yuki2501@lemmy.world 3 points 2 years ago

Thanks! Now I can use Connectβ„’ for Lemmy without problems. πŸ‘πŸ™‚

[–] Olap@lemmy.world 3 points 2 years ago

Let us know where donations can go, suspect a stacked docker-compose will reach limits very quickly

[–] dasnet@mastodon.social 3 points 2 years ago
[–] ubergeek77@lemmy.ubergeek77.chat 3 points 2 years ago* (last edited 2 years ago)

Running so many Lemmy instances against the same database doesn't cause race conditions? I wonder why that "just worked" so easily, usually load balancing DB-backed apps is a whole beast on its own.

[–] cute_cat@lemmy.world 3 points 2 years ago

I really enjoy how lemmy is growing!

[–] DigitalPortkey@lemmy.world 3 points 2 years ago

Absolutely loving the new UI changes. Thanks for being so transparent with us as well!

[–] WigglingWalrus@lemmy.world 3 points 2 years ago

Logging in works now! Also got 2FA enabled without issues.

[–] Calzoner@lemmy.world 3 points 2 years ago

This is awesome. Was a fun read too. Super cool to see what was going on behind the scenes.

[–] Bushwhack@lemmy.world 3 points 2 years ago

I survived!

[–] mtnwolf@lemmy.world 3 points 2 years ago* (last edited 2 years ago) (1 children)

Half the time when I comment, it just spins. :( Edit: Apparently when I comment it posts, but just shows spinning until I manually refresh. Must be on my end.

load more comments (1 replies)
[–] BitOneZero@lemmy.world 3 points 2 years ago (1 children)

we created extra lemmy containers to spread the load. (And extra lemmy-ui containers).

Is Rust HTTP server running into thread limits? database connection pooling? All kinds of internal questions bout that solution.

[–] ruud@lemmy.world 3 points 2 years ago

I don't know rust. But there were 150 database connections setup by lemmy, but only about 15 of them were used, the rest idle.

[–] mtnwolf@lemmy.world 3 points 2 years ago

Looking good πŸ‘

load more comments
view more: β€Ή prev next β€Ί