On June 19, 2023, I setup scaling for the environment. My experience doing this is with AWS; however, I host this service with DigitalOcean. The good news is I now understand how DigitalOcean Load Balancers work. The bad news is I had intermittent outages throughout the day.
Sorry, I'll do better.
- jgrim
I created a new webhead image that was slimmed down and only had the front end. I set up scaling with DigitalOcean to create new instances from a snapshot. I didn't know that when a snapshot creates an instance, it reset the
/etc/hostsfile, breaking private network traffic. I tried to use DigitalOcean's tools to persist the host file. However, none of these options worked. I have to find a new way to map traffic.I put the site behind a load balancer with 2 web heads. The rate limiting by Lemmy kicked and started blocking traffic. I tried raising the limits too high to see if that'd fix it. It didn't work. That's when I realized that the Nginx configuration for Lemmy has additional rate-limiting configured. After I reconfigured the Nginx configuration, I was able to get traffic flowing again.
This took a big part of my day. Sorry for those that missed out.
that sucks
Yes, it did. Today was the day I posted this site on LinkedIn too. So there's that. 🤷♂️