this post was submitted on 04 Jul 2023
2245 points (99.1% liked)

Lemmy.World Announcements

29164 readers
42 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news ๐Ÿ˜

Outages ๐Ÿ”ฅ

https://status.lemmy.world/

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations ๐Ÿ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 2 years ago
MODERATORS
 

Status update July 4th

Just wanted to let you know where we are with Lemmy.world.

Issues

As you might have noticed, things still won't work as desired.. we see several issues:

Performance

  • Loading is mostly OK, but sometimes things take forever
  • We (and you) see many 502 errors, resulting in empty pages etc.
  • System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)

Bugs

  • Replying to a DM doesn't seem to work. When hitting reply, you get a box with the original message which you can edit and save (which does nothing)
  • 2FA seems to be a problem for many people. It doesn't always work as expected.

Troubleshooting

We have many people helping us, with (site) moderation, sysadmin, troubleshooting, advise etc. There currently are 25 people in our Discord, including admins of other servers. In the Sysadmin channel we are with 8 people. We do troubleshooting sessions with these, and sometimes others. One of the Lemmy devs, @nutomic@lemmy.ml is also helping with current issues.

So, all is not yet running smoothly as we hoped, but with all this help we'll surely get there! Also thank you all for the donations, this helps giving the possibility to use the hardware and tools needed to keep Lemmy.world running!

you are viewing a single comment's thread
view the rest of the comments
[โ€“] DrSanta@lemmy.world 6 points 2 years ago (1 children)

Does Lemmy have the ability to replace default links?

Basically, replace signup link with one that redirects to a page that gives a very simple as possible explanation what's going on, what fediverse is and gives s list of other instances to try.

Reinforce "All are viable and can browse lemmy.world subs"... Or communities or whatever term we use here for lemmy equivalent of subreddits.

[โ€“] CoderKat@lemm.ee 3 points 2 years ago (2 children)

I was personally thinking more along the lines of if we could have a load balancer whose sole job is to route users to a random set of possible instances (which can all be administered by the same person, so that you're still joining the instance "group" that you want). The load balancer would route someone the first time they land on the page and also handle logins. That's it.

I'm assuming that the servers we're talking about are single servers, because that's how things sound. I'm personally used to only developing servers that use the "many servers behind a load balancer" approach. While distributed databases can certainly make those easier, in the absence of support for that, you could always run the backends as entirely separate servers, with the load balancer just serving to pick the backend. So you'd have a lemmy1.world and so on.

Of course, for all I know, maybe this is already the architecture of a Lemmy instance. I've never checked. Even with a good architecture, scaling can be difficult. An unnoticeable performance issue in a dev environment can be a massive bottleneck when you have tens of thousands of concurrent users.

[โ€“] yaaaaayPancakes@lemmy.world 4 points 2 years ago (1 children)

There was a post about it. They're running a number of instances of the frontend and backend containers, behind nginx which they're using as a load balancer.

[โ€“] CoderKat@lemm.ee 2 points 2 years ago (1 children)

Do you (or another commenter) have a link to that post?

[โ€“] yaaaaayPancakes@lemmy.world 5 points 2 years ago

https://lemmy.world/post/920294

Talked about in the solutions section

[โ€“] DrSanta@lemm.ee 2 points 2 years ago (1 children)

I'm not sure I'm following.

Wouldn't this load balancer be swapping the user's current instance user420@lemmy.world may suddenly become user420@lemm.ee?

Or more like multiple servers within the same umbrella instance? User420@lemmy.world, User420@lemmy.world1, User420@lemmy.world2, User420@lemmy.world3.

Apologies, while I think myself fairly tech savvy, development and networking is still a bit out of reach.

[โ€“] CoderKat@lemm.ee 2 points 2 years ago (1 children)

Or more like multiple servers within the same umbrella instance? User420@lemmy.world, User420@lemmy.world1, User420@lemmy.world2, User420@lemmy.world3.

This is what I was originally picturing. So that logged in users would be browsing on pretty much entirely separate instances (avoiding them having to reuse as much).

I hadn't really decided on how I best liked the idea of handling logins, since there's so many possible options. It could be that users would just have to either know their server (so you'd have to sign in as User420@lemmy1.world). Or the load balancer could maintain a simple store of users/emails to instances to avoid that. Or at the cost of extra complexity (yay), you could replicate the user across all the instances but only make a single instance active for that user at a time (that's a pretty common technique with systems I work with, with servers being strongly coupled to some range of resources to maximize efficiency).

[โ€“] DrSanta@lemm.ee 2 points 2 years ago

I noticed you talked about the load balancer being a person. Sounds like it'd be better if it was a bot. They just see which pool is currently the emptiest and put them there, right?

Although you seem to be suggesting live instance swapping. Which might be possible in the future. Right now appears to be tied to registration.