[-] supakaity@lemmy.blahaj.zone 7 points 1 month ago

Poor li'l blighter got attacked by a bin chick'n and was limping away last we saw em... after goin' the bin juice drinkin' irrit of course.

[-] supakaity@lemmy.blahaj.zone 22 points 1 month ago

Apparently I was wrong, it's decided it's done and working again now.

[-] supakaity@lemmy.blahaj.zone 28 points 1 month ago* (last edited 1 month ago)

The pict-rs upgrade is ongoing.

From what I can tell it'll be about another 5 hours. I'm going to have to go to bed and check on it in the morning.

Unfortunately the stock-standard lemmy-ui doesn't like it that pict-rs is migrating to a new version of database and not serving images, so it's stubbornly just not working at all.

100
Lemmy updated to v0.19.5 (lemmy.blahaj.zone)

Hey all!

Our lemmy.blahaj.zone has been updated to v0.19.5.

Let us know if you notice any issues with the upgrade!

[-] supakaity@lemmy.blahaj.zone 24 points 3 months ago

Our best haj, Shonky (they/them) is available now, over at their own Github repository for use when referring to Blåhaj Lemmy.

I'm guessing we'll need them to make an appearance for the Canvas template.

50
Test upload image (lemmy.blahaj.zone)

Just testing image upload.

[-] supakaity@lemmy.blahaj.zone 3 points 4 months ago

So this was at the very start of the s02e10 episode - "Pegasus".

1
Alternative frontends (lemmy.blahaj.zone)

Hi all our lovely users,

Just a quick post to let you all know that along-side the upgrade to 0.19.3, we've also added a couple of alternate UIs to the Blåhaj lemmy for you.

Obviously the default lemmy-UI at https://lemmy.blahaj.zone still exists and has been updated to 0.19.3 alongside the lemmy server update.

There's also now an Alexandrite UI at https://alx.lemmy.blahaj.zone which is a more modern, smoother UI, written in svelte, by sheodox.

And then for those who are nostalgic for reddit days of yore, and memories of when PHP websites last ruled the earth, there's MLMYM (courtesy of rystaf) at https://mlmym.lemmy.blahaj.zone.

Please enjoy, and I hope the upgrades work well for you.

2
¡La mariposa, muy bonita! (lemmy.blahaj.zone)

Esto es una prueba

1
Testing image upload (lemmy.blahaj.zone)

Test

[-] supakaity@lemmy.blahaj.zone 9 points 10 months ago

I also agree.

While I couldn't care less (it's their body and bodily autonomy is super important to me), I'd be a little hurt if they didn't tell me about it.

In that case I'd be more interested in talking to my partner to try and understand why they felt that they couldn't talk to me about it and needed to keep it a secret.

1
submitted 1 year ago* (last edited 1 year ago) by supakaity@lemmy.blahaj.zone to c/main@lemmy.blahaj.zone

We're currently in the process of migrating our pict-rs service (the thing responsible for storing media/images/uploads etc) to the new infrastructure.

This involves an additional step of moving our existing file-based storage to object storage, so this process will take a little time.

New images/uploads may not work properly during this migration, however existing images should continue to load. We expect this migration to take about an hour.

[EDIT]

Migration has completed.

685,271 files / 153.38 GB were migrated. Copying to object storage took about 1.5 hours. Starting service back up on new server and debugging took another 30 minutes.

Timeline:

  • Migration started at 2023-10-01 22:43 UTC.
  • [+1h32m] Objects finished uploading to object storage at 2023-10-02T00:15 UTC.
  • [+2h06m] Migration was completed at 2023-10-02 00:46 UTC.
1

The server will be briefly down while we install a new updated version of lemmy and restart it.

The maintenance window is 15 minutes, but should be much shorter.

[-] supakaity@lemmy.blahaj.zone 8 points 1 year ago

Oh goodness, why is this so funny? lol.

blobhaj, ani, hearts, animated

1
State of the shork! (lemmy.blahaj.zone)

So it's been a few days, where are we now?

I also thought given the technical inclination of a lot of our users that you all might be somewhat interested in the what, how and why of our decisions here, so I've included a bit of the more techy side of things in my update.

Bandwidth

So one of the big issues we had was the heavy bandwidth caused by a massive amount of downloaded content (not in terms of storage space, but multiple people downloading the same content).

In terms of bandwidth, we were seeing the top 10 single images resulting in around 600GB+ of downloads in a 24 hour period.

This has been resolved by setting up a frontline caching server at pictrs.blahaj.zone, which is sitting on a small, unlimited 400Mbps connection, running a tiny Caddy cache that is reverse proxying to the actual lemmy server and locally caching the images in a file store on its 10TB drive. The nginx in front of lemmy is 301 redirecting internet facing static image requests to the new caching server.

This one step alone is saving over $1,500/month.

Alternate hosting

The second step is to get away from RDS and our current fixed instance hosting to a stand-alone and self-healing infrastructure. This has been what I've been doing over the last few days, setting up the new servers and configuring the new cluster.

We could be doing this cheaper with a lower cost hosting provider and a less resiliant configuration, but I'm pretty risk averse and I'm comfortable that this will be a safe configuration.

I woudn't normally recommend this setup to anyone hosting a small or single user instance, as it's a bit overkill for us at this stage, but in this case, I have decided to spin up a full production grade kubernetes cluster with a stacked etcd inside a dedicated HA control plane.

We have rented two bigger dedicated servers (64GB, 8 CPU, 2TB RAID 1, 1 GBPS bandwidth) to run our 2 databases (main/standby), redis, etc on. Then a the control plane is running on 3 smaller instances (2GB, 2 CPU each).

All up this new infrastructure will cost around $9.20/day ($275/m).

Current infrastructure

The current AWS infrastructure is still running at full spec and (minus the excess bandwidth charges) is still costing around $50/day ($1500/m).

Migration

Apart from setting up kubernetes, nothing has been migrated yet. This will be next.

The first step will be to get the databases off the AWS infrastucture first, which will be the biggest bang for buck as the RDS is costing around $34/day ($1,000/m)

The second step will be the next biggest machine which is our Hajkey instance at Blåhaj zone, currently costing around $8/day ($240/m).

Then the pictrs installation, and lemmy itself.

And finally everything else will come off and we'll shut the AWS account down.

[-] supakaity@lemmy.blahaj.zone 37 points 1 year ago

So, one thing I'd mention is the systems and admin work involved in running an instance.

This is on top of the community moderation, and involves networking with other instance admins, maintaining good relations, deciding who to defeferate from, dealing with unhappy users, etc.

Then there's the setup and maintenance of the servers, security, hacks, DDoSing, backups, redundancy, monitoring, downtime, diagnosis, fixing performance issues, patching, coding, upgrades etc.

I wouldn't be here doing this without @ada. We make a formidable team, and without any self effacement, we are both at the top of our respective roles with decades of experience.

Big communities also magnify the amount of work involved. We're almost at the point where we are starting to consider getting additional people involved.

Moreover we're both here for the long haul, with the willingness and ability to personally cover the shortfall in hosting costs.

I'm not trying to convince you to stay here. But in addition to free hardware, you're going to need a small staff to do these things for you, so my advice is to work out if you have reliable AND trustworthy people (because these people will have access to confidential user data) who are committed to do this work long term with you. Where will you be in 3 years, 5, 10?

[-] supakaity@lemmy.blahaj.zone 8 points 1 year ago

We have already begun moving, and this announcement was to warn our users about future downtime as we migrate more critical services like databases etc.

[-] supakaity@lemmy.blahaj.zone 77 points 1 year ago

To be clear, $3k is an accurate, but unacceptable amount.

As in that's what it's actually costing us, but it's not what it should be costing. I'd imagine more like $250 is what we should be paying if I wasn't using AWS in the silly way I am.

I'm admitting up front that I've been more focused on developing rather than optimising operating costs because I could afford to be a little frivolous with the cost in exchange for not having to worry about doing server stuff.

Even when the Reddit thing happened I was wilfully ignoring it, trying to solve the scaling issues instead of focusing on the increased costs.

And so I didn't notice when Lemmy was pushing a terabyte of data out of the ELB a day. And that's what got me.

About half that $3k is just data transfer costs.

Anyhow the notice was just to let our users know what is going on and that there'll be some maintenance windows in their future so it doesn't surprise anyone.

We have a plan and it will all work out.

Don't panic or have any kneejerk reactions, it's just an FYI.

[-] supakaity@lemmy.blahaj.zone 47 points 1 year ago

Just want to say, I don't blame anyone else but myself.

I certainly don't blame anyone at 196.

I hope I'm really clear about that. It's one of the reasons I specifically didn't name 196 in my announcement.

We've got a solution planned, we've already started to implement it and have the image transfer issue solved already.

We can afford to cover this ridiculous AWS bill, I just need to do some maintenance work so this doesn't continue because I can't continue to line Jeff Bezos' pockets like this indefinitely.

2

Discussion of the current situation with the Blåhaj instances, and upcoming maintenance.

[-] supakaity@lemmy.blahaj.zone 5 points 1 year ago* (last edited 1 year ago)

You are actually a little bit correct.

Most uses of the misnomer "transphobia" are actually incorrect, because the people involved aren't actually suffering from an irrational medical or psychological fear of transgender people.

What they actually are doing is more accurately called "transmisia", because they are transmisic. This means they are heavily prejudiced against transgender women and believe that they shouldn't be involved in the competitions.

Their reason for this are not because the transgender women don't meet the beauty standards, or aren't passing the judging categories, but because they just don't want them there purely due to their trans-mysogynistic preconceptions.

1
Lemmy updated to v0.18.2 (lemmy.blahaj.zone)

Our lemmy is now running the 0.18.2 release version, which should fix some lingering issues we've been having.

Let @ada or myself know if there's any issues!

1

Hi everyone, I'll begin migrating the lemmy blåhaj database to the new server this morning in about 20 minutes.

Expected duration is about 1 hour for this migration.

There will be a maintenance page up during the migration and I will be updating the status as we go to keep you updated on the process.

Later today I'll also be upgrading the software to the latest release as well.

0
Blåhaj Lemmy hacked (lemmy.blahaj.zone)
submitted 1 year ago* (last edited 1 year ago) by supakaity@lemmy.blahaj.zone to c/main@lemmy.blahaj.zone

Hi everyone.

Lemmy had an XSS vulnerability and we were one of the instances attacked through this vulnerability. We had our homepage replaced by a youtube video.

The lemmy devs were quick to create a PR for this issue and we have patched our instance to fix it,

Also while I do not believe any login tokens were compromised, we have taken the additional measure of rotating our secret key, so everyone will have to login to the site again as your existing credentials will no longer be valid.

If login fails to work properly, you may have to clear your cookies for the site... it seems to get confused about whether you're logged in or not.

Sorry that this happened, if you have any questions please contact @ada@lemmy.blahaj.zone or myself.

blobcat, heart

view more: next ›

supakaity

joined 2 years ago
MODERATOR OF