this post was submitted on 05 Nov 2024
225 points (99.6% liked)

Fediverse

28224 readers
394 users here now

A community to talk about the Fediverse and all it's related services using ActivityPub (Mastodon, Lemmy, KBin, etc).

If you wanted to get help with moderating your own community then head over to !moderators@lemmy.world!

Rules

Learn more at these websites: Join The Fediverse Wiki, Fediverse.info, Wikipedia Page, The Federation Info (Stats), FediDB (Stats), Sub Rehab (Reddit Migration), Search Lemmy

founded 1 year ago
MODERATORS
 

Disclaimer: I wrote this article and made this website.

There was some talk of this issue in the recent fediverse inefficiencies thread. I'm hopeful that in the future we'll have a decentralized solution for file hosting but for now I deeply believe that users should pay for their own file hosting.

top 50 comments
sorted by: hot top controversial new old
[–] moon@lemmy.cafe 1 points 16 minutes ago* (last edited 13 minutes ago)

Personally I'm in the camp that I want history to be lost. That's part of the appeal to me. In fact my favorite feature in the fedi is Mastodon's option to enable auto-deleting posts of a certain age.

Only content that is explicitly pinned or reaches a certain amount of interactions should be saved imo. Since that's the stuff you'd actually want to preserve rather than the 99% of forgettable content, and it would also drastically cut down on file hosting.

Another thing is that a federation should only act as the exchange between users on ActivityPub. It should only cache relevant information and not be expected to store everything, like I wrote before. The user should be a portable account that is stored on a device. The federation server would sync your account between your devices, but not store it. You send your content to the federation, and then the federation sends it out into the world where they choose to do what they want with it. The federation shouldn't hoard it indefinitely.

Also this makes sense from a privacy perspective. If you care about privacy, why would you also want all your data indefinitely stored? Unless certain things are relevant and explicitly kept, it should be expected to expire and be lost by default. Where did we get this expectation that data should be stored forever? Also you expect it to be stored forever and not be trained on by AI?

This comment for example, after about a week or two most of the visibility and interaction of it will drop to zero. At that point, this comment should expire and no longer exist. I wrote this comment, it reached some people, and served it's purpose and should expire. I'm not going to pretend like this comment is some kind of historic document that should be indefinitely preserved, nor do I expect or want it to be.

[–] M0oP0o@mander.xyz 6 points 1 hour ago (1 children)

Too wet for server racks in the forest.

[–] sosodev@lemmy.world 4 points 57 minutes ago (1 children)
[–] M0oP0o@mander.xyz 2 points 52 minutes ago

Look I know there called "farms" but like I told the last forest gnome, the dank woods is no place to host data.

[–] TORFdot0@lemmy.world 3 points 3 hours ago (1 children)

Is a p2p system for media with the instances just hosting magnet links too slow for fediverse purposes? To me this seems like the most resilient way to handle media in a decentralized system

[–] ayyy@sh.itjust.works 2 points 2 hours ago (1 children)

If a social network is to take off, it must be accessible from mobile devices behind CGNAT (carrier grade network address translation).

[–] pinkystew@reddthat.com 1 points 2 hours ago (1 children)
[–] ayyy@sh.itjust.works 4 points 2 hours ago

Because that’s where all the users are. The “social” aspect of a “social” network. Anyone can host a forum but it’s useless without users.

[–] tehn00bi@lemmy.world 8 points 4 hours ago (2 children)

Is file hosting really a must? I mean Reddit and feddit are basically forums. And not many forums allow file uploads. Also, we should have retention limits. Low value posts are allowed to fade away. High value posts that have some level of interaction stay alive longer.

[–] Azzu@lemm.ee 1 points 1 hour ago

Reddit is basically entirely image or video posts, all hosted by reddit directly.

[–] hendrik@palaver.p3x.de 19 points 4 hours ago (1 children)

A lot of pictures and memes get posted here. And every other post shows a thumbnail picture. These images are all files.

[–] tehn00bi@lemmy.world 8 points 4 hours ago (2 children)

Not denying that. But maybe we should accept that photos and memes and whatnot aren’t that valuable and limit their size or the volume allowed per user. Just a thought.

[–] nasi_goreng@lemmy.zip 1 points 1 hour ago

limit their size or the volume allowed per user

Fedi software like Misskey already did exactly that. Each user have limited "drive" which can be upgraded/customized per user. People even reuse image they already have on their drive, so it won't be any duplicate files.

[–] hendrik@palaver.p3x.de 6 points 3 hours ago* (last edited 3 hours ago)

Yeah, I wonder if that would fly with the users. I just scrolled through my timeline and nearly every post has some colorful image to it. (except in Ask Lemmy and No Stupid Questions.) I'm not sure if users would accept this platform if it were mostly textual. And putting restrictions in place would certainly reduce the number of images. Scrolling through Lemmy would feel like Hackernews, not any modern social media platform. I doubt mainstream people appreciate that.

But yeah, that'd be possible. We could just close the meme communities for example. Or exclude them from individual instances to save some space there.

[–] NuXCOM_90Percent@lemmy.zip 64 points 6 hours ago (2 children)

We totally need sustainable file hosting. Freedom!

Wait... the fuck did you just upload? Oh god. Oh god no. Do I have to call the cops on you? Oh no. Wait, does this count as possession? FUCK!!!!!!

We need someone else to handle the totally sustainable file hosting. Freedom!

[–] grue@lemmy.world 3 points 4 hours ago (1 children)

Seems to me that this is a use-case ~~Freenet~~ Hyphanet would be good for, both because it distributes the problem of file storage load and because it eliminates responsibility for each host to police his node by making it impossible for anyone to know which file chunks said node is hosting.

[–] NuXCOM_90Percent@lemmy.zip 5 points 3 hours ago (1 children)

Nothing solves the problem of CSAM quite like... making everyone partially culpable in the storage and distribution of CSAM.

You can't prove I was hosting child porn. Statistically, we all only had a 70% probability of having it on our computers

[–] grue@lemmy.world 3 points 3 hours ago (1 children)

Stuff that isn't accessed eventually gets deleted. If the Lemmy instances (which are clearnet, of course) delete the references to it, it would go away.

[–] NuXCOM_90Percent@lemmy.zip 2 points 3 hours ago

Which gets back to volunteers going through and moderating it. And the ethical and moral question of whether people who upload it are reported.

And... honestly? if there is even a 20% chance that running a file sharing node (because I just love to give away both bandwidth and storage...) is being used to store CSAM? I ain't doing that shit and most people will similarly run screaming and call the cops.

[–] Kalkaline@leminal.space 18 points 6 hours ago (2 children)

Yep, there needs to be moderation tools that can be quickly deployed to stop the illegal/immoral/evil stuff from spreading and taking over self-hosted servers.

[–] NuXCOM_90Percent@lemmy.zip 20 points 6 hours ago (2 children)

And moderation of this kind of content almost always sounds like torture when you hear about what facebook and the like are outsourcing.

Theoretically, this is a good problem for computer vision/machine learning. But there are a LOT of false positives (I think it was Aftermath who did an article on a study of when a nipple becomes female?). And.. what ethical responsibility do you have to report on the fiftieth time that SheIsReallyAnEightThousandYearOldDragon_6969 uploaded CSAM? And how quick do you think people are going to lose faith in you and start wondering if you'll also report on the rampant piracy?

And... there are also false negatives. At which point you find out you have been hosting something truly heinous for the past few months... possibly when local law enforcement tells you.

Like a lot of things: it sounds great. But nobody in their right mind is going to host this for free. And once you start accepting money you start opening yourself up to a LOT of regulations.

[–] rigatti@lemmy.world 5 points 6 hours ago (1 children)

A nipple becomes female precisely when it wants to!

[–] grue@lemmy.world 1 points 4 hours ago* (last edited 4 hours ago)

Nipples are wizards, confirmed.

[–] Kalkaline@leminal.space 3 points 5 hours ago

It doesn't even have to be to that extent, just being able to slow/stop that awful content from being uploaded by a bunch of malicious bots. Even if it's not malicious content, you could still have people uploading spam that eats up the server space.

[–] Lost_My_Mind@lemmy.world 5 points 5 hours ago (2 children)

Illegal I can begrudgingly agree with. Even though I am a proponent for piracy, I will conceed that for growth's sake, the tools need a clear well defined path to moderation.

That being said, who's to say what IS immoral and evil?

In the republicans minds, porn is evil and should be banned. Trans rights are evil and should be banned. Abortion is evil and should be banned.

I disagree with all those claims. I do not think any of them are immoral, or evil.

I think pineapple on pizza is wrong, and evil. Some agree, others don't. If I had my way, promoting of pineapple on pizza would be banned.

Now, who's to say what is, and what isn't evil? I think the only clear line to a moderation approach is to have a clear, unquestionable set of rules. These rules are to be based on public laws.

Everything else, I feel you should have the freedom to do as you wish. But also, I believe other people that you don't agree with should be free to do as they wish.

You may never know how someone feels, or understand their perspective, but as long as they aren't breaking laws, I feel they should have the ability to feel that way consequence free.

I may not like that you put pineapples on your pizza, but I feel that you should have the right to enjoy it. Even if it goes against MY views as to what constitutes a REAL pizza! Much to my surprise, pineapple on pizza ISN'T illegal. So you should have the right to enjoy it......

And yes. I did take the most pandtentic example I could think of, in order to display the absurdity of the concept of how easy it is to accept others rights in this world that don't affect you.

Now just apply that same concept to every other example in the world. Then take into consideration that by using vague undefined terms to define your rules, you create grey area that's easy to exploit. Who's to say what IS evil? Adults told their teenagers in the 1950s that Elvis was evil. Parents in the 1920s told their teenagers that jazz was evil.

We need to define the terms that define our rules.

[–] Kalkaline@leminal.space 1 points 3 hours ago

I would say you go to the extremes and work back from there for what's immoral/evil.

load more comments (1 replies)
[–] hperrin@lemmy.world 26 points 6 hours ago* (last edited 6 hours ago) (3 children)

Ok, hear me out.

We find the users with the slowest internet and start sending them all the data. They don’t have to keep anything on disk. Then they send it all back and forth between each other. Any time a user makes a request, we just wait for one of the slow nodes to come across the data and send it out.

We use the slowest wires for all the storage. It’s fool proof.

[–] sosodev@lemmy.world 27 points 6 hours ago (2 children)

Somebody actually did make this as a joke years ago haha https://github.com/yarrick/pingfs

[–] dojan@lemmy.world 4 points 4 hours ago

I was brushing my teeth when reading this comment and inadvertently ended up swallowing all my toothpaste.

load more comments (1 replies)
[–] NaibofTabr@infosec.pub 11 points 6 hours ago
[–] poVoq@slrpnk.net 14 points 7 hours ago* (last edited 7 hours ago) (1 children)

Have you considered providing something like this: https://jortage.com/ and maybe contribute to their efforts to develop a specific API for that? Source code is here: https://github.com/jortage

[–] sosodev@lemmy.world 14 points 7 hours ago (2 children)

Jortage is a really interesting approach. It definitely helps reduce the impact of the file hosting problem but it doesn’t fully address the underlying cost issue. The cost of storing files grows every month indefinitely while donations typically don’t.

I would like to see a file hosting pool come to lemmy though. So I will look into it. :)

[–] Deebster@lemmy.ml 2 points 4 hours ago (1 children)

I think the major advantage is the deduplication - when an image goes viral across Mastodon (or Lemmy) it's currently stored hundreds or thousands of times, each with its own cost. Do you dedupe (for either your customers' benefit or your own)?

[–] sosodev@lemmy.world 1 points 3 hours ago* (last edited 3 hours ago) (1 children)

Are the images duplicated when shared? My understanding is that only a link to the file is replicated across servers and duplication comes from users manually uploading the same file to another server.

My website does not do any deduplication at this time.

[–] Deebster@lemmy.ml 4 points 3 hours ago* (last edited 3 hours ago) (1 children)

Yes, for example go to https://infosec.exchange/explore

I see the top post as https://infosec.exchange/@nocontexttrek@mastodon.social/113433063621462027 and the image is https://media.infosec.exchange/infosec.exchange/cache/media_attachments/files/113/433/063/582/671/258/original/71da3801e4e4f08c.png

The link is to the original on https://files.mastodon.social/media_attachments/files/113/433/062/676/773/993/original/f828afef5cc7ed1c.png but when you click image the javascript loads a modal with the local cached version (same image as the thumbnail that infosec.exchange loads.

There's lots of different codebases across the fediverse so perhaps some hotlink, but local copies is the default.

[–] sosodev@lemmy.world 3 points 2 hours ago* (last edited 2 hours ago) (1 children)

The Lemmy server config indicates that is an optional setting to improve user privacy so requests don't ever hit the original server from the client. Those cached files are only temporary and will be deleted after some time. So it's not really full blown duplication.

The default setting is to only generate the thumbnails and store those locally (indefinitely?) but even that can be turned off. I checked and it appears that lemmy.world has the thumbnail generation disabled so all images from other instances just link to the original on that instance.

[–] Deebster@lemmy.ml 1 points 2 hours ago (1 children)

Ok, so Lemmy doesn't cause the same amount of duplication, but I'd still argue that dedupe is valuable: it saves on hosting costs (your costs, in this case) and users will get a small advantage in having slightly higher cache hits.

[–] sosodev@lemmy.world 1 points 2 hours ago

For sure, I'll add it to the list. :)

[–] poVoq@slrpnk.net 7 points 6 hours ago* (last edited 1 hour ago)

Pict-rs that is used by Lemmy to store images already supports S3 type storage, so in theory it should work with Jortage, but I don't think anybody has tested that yet. The people behind Feddit.org might have experimented with it as they expressed interest a while back.

[–] Valmond@lemmy.world 6 points 6 hours ago

I think Tenfingers could be an interesting option as hosters do not know what they host, the data can be modified, and it's 100% decentralised.

[–] muntedcrocodile@lemm.ee 3 points 4 hours ago

I found this https://github.com/Marie673/Torrent_Proxy

Its in chinese so idk if its whats needed but if each instance also acted as a torrent proxy then thats decentralised domain agnostic file hosting that doesnt break frontends but also allows clients to update to do resource resolution themselves while helping to serve the fediverses files as a whole.

[–] fishcharlie@eventfrontier.com 5 points 5 hours ago (1 children)

I know I'm not necessarily the target audience for this. But it feels too expensive. 6x the price of Cloudflare R2, almost 13x the price of Wasabi. Even iCloud storage is $0.99 for 50 GB with a 5 GB free tier. But again, I know I'm not necessarily the target audience as I have a lot of technical skills that maybe average users don't have.

If you ever get around to building an API, and are interested in partnerships, let me know. Maybe there is a possibility for integration into !echo@eventfrontier.com 😉.

load more comments (1 replies)
[–] abff08f4813c@j4vcdedmiokf56h3ho4t62mlku.srv.us 9 points 6 hours ago (1 children)

You're not the first to think about this.

See https://aumetra.xyz/posts/the-fedi-ddos-problem - there an embed server is proposed, to be shared by multiple instances (ideally a great many would use just the one), which can host things like image files and previews.

[–] moon@lemmy.cafe 1 points 5 minutes ago

There's a big issue with this.

If malicious content like CP gets uploaded on to a server, obviously other servers do not want this to be replicated to their servers. So how would you solve this problem? Well they could give all moderation power to the original server they're replicating, but that could be far too slow or they could even miss malicious content like this. Or maybe they even disagree about taking down certain things.

Another solution is that any server participating in the content mirroring could take it down for just themselves or for all the other members as well. The issue here is now you're expanding moderation abilities, while also giving the other servers much more responsibilities.

It's not as simple as wanting to replicate content. If you host it, you are responsible for any illegal content a user may upload to it. Not to mention laws vary by country as well. Ignoring the technical challenges here, it's also mandatory that the other servers replicate the other servers data to also choose to be responsible for what gets uploaded. And that is a really big ask. The law doesn't care about the technical reasons, they'll just see illegal content uploaded to your server.

[–] bulwark@lemmy.world 6 points 6 hours ago

I wish there was some version of PBS for Lemmy, like public funds for hosting. I'll admit I haven't really thought this through, so there's probably some problems with my idea.

[–] jaggedrobotpubes@lemmy.world 6 points 7 hours ago

This feels like something the Fediverse is ultimately going to build for itself. I know jack squat about the details, but it's gonna have to be a thing eventually, I think.

[–] tofuwabohu@slrpnk.net 6 points 7 hours ago (1 children)

Interesting approach, good luck! Admittedly I'm not sure if many users want to take their media uploading in their own hands and pay for it but maybe I'm wrong. Where are the images stored? Do you have your own hardware? Backups etc?

Also since you're interested in Fediverse media storage, I recently read about https://jortage.com/ It's a third party storage for your instance with deduplication, pretty interesting idea. Takes away a bit of the federated part though

load more comments (1 replies)
load more comments
view more: next ›