this post was submitted on 19 Jun 2023
3 points (100.0% liked)

Announcements

241 readers
4 users here now

Community updates and announcements.

Admins will post any updates here so be sure to follow!

Important updates will be pinned to the local feed.

founded 2 years ago
MODERATORS
 

On June 19, 2023, I setup scaling for the environment. My experience doing this is with AWS; however, I host this service with DigitalOcean. The good news is I now understand how DigitalOcean Load Balancers work. The bad news is I had intermittent outages throughout the day.

Sorry, I'll do better.

  • jgrim
top 3 comments
sorted by: hot top controversial new old
[–] jgrim 2 points 2 years ago (1 children)

I created a new webhead image that was slimmed down and only had the front end. I set up scaling with DigitalOcean to create new instances from a snapshot. I didn't know that when a snapshot creates an instance, it reset the /etc/hosts file, breaking private network traffic. I tried to use DigitalOcean's tools to persist the host file. However, none of these options worked. I have to find a new way to map traffic.

I put the site behind a load balancer with 2 web heads. The rate limiting by Lemmy kicked and started blocking traffic. I tried raising the limits too high to see if that'd fix it. It didn't work. That's when I realized that the Nginx configuration for Lemmy has additional rate-limiting configured. After I reconfigured the Nginx configuration, I was able to get traffic flowing again.

This took a big part of my day. Sorry for those that missed out.

[–] james_baxter 2 points 2 years ago (1 children)
[–] jgrim 2 points 2 years ago

Yes, it did. Today was the day I posted this site on LinkedIn too. So there's that. 🤷‍♂️