A nice place to discuss rumors, happenings, innovations, and challenges in the technology sphere. We also welcome discussions on the intersections of technology and society. If it’s technological news or discussion of technology, it probably belongs here.
Remember the overriding ethos on Beehaw: Be(e) Nice. Each user you encounter here is a person, and should be treated with kindness (even if they’re wrong, or use a Linux distro you don’t like). Personal attacks will not be tolerated.
Subcommunities on Beehaw:
This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.
It's neat from a research and proof-of-concept perspective but practically speaking I'd like to see the CPU cycles required for the LLM compression compared to PNG or FLAC compression. We've always known we can increase compression by throwing more computing power at the problem but we settle on a happy medium at the intersection of "good enough" for compression and performance.
Bad. The time performance is bad. Even if it was O(n) there's billions of parameters involved in an LLM.
While that's generally true we might want to look into utilizing available cores more - but I guess with LLM it might be harder to scale that while keeping file size the same.
A lot of the current compression programs only use one thread properly - which was still perfectly fine a few years ago, but thanks to AMD cores have become cheap. Few years ago most notebooks would come with two cores, and either two or four threads, with higher end models with 4c/4t. Something bigger pretty much didn't exist for notebooks, and was expensive for desktops.
Nowadays you can get 16 cores in a reasonably priced notebook, and if it benefits your work you won't think much about spending a bit extra for a 32 or 64 core CPU in your workstation - where just 6 years ago you'd have had no option for such a notebook, and paid the equivalent of a not too shabby car for the workstation.
With a neural network, you wouldn't be able to mathematically prove that the signal is perfectly recovered 100% of the time for all possible inputs. That is the case with PNG and FLAC. If you're just listening to music and need a good compression ratio, then sure, it won't be a big deal if a couple of bits are wrong. But that's also why we have lossy compression. If the goal is to make signal degradation imperceptible to a human, then you could get a much better compression ratio using neural networks. If it's truly critical that the signal isn't corrupted, it would probably be better to just use the original method.
That's not what lossless data compression schemes do:
In lossless compression the general idea is to create a codebook of commonly occuring patterns and use those as shorthand.
For example, one of the simplest and now ancient algorithms LZW does the following:
However, once this is done, you now need to find an encoding that takes your characterset (the original characters+the new dictionary references) and turns it into bits.
It turns out that we can do this optimally: Using an algorithm called Arithmetic coding we can align the length of a bitstring to the amount of information it contains.
"Information" here meaning the statistical concept of information, which depends on the inverse likelihood a certain character is observed.
Logically this makes sense:
Let's say you have a system that measures earthquakes. As one would expect, most of the time, let's say 99% of the time, you will see "no earthquake", while in 1% of the cases you will observe "earthquake".
Since "no earthquake" is a lot more common, the information gain is relatively small (if I told you "the system said no earthquake", you could have guessed that with 99% confidence: not very surprising).
However if I tell you "there is an earthquake" this is much more important and therefore is worth more information.
From information theory (a branch of mathematics), we know that if we want to maximize the efficiency of our codec, we have to match the length of every character to its information content. Arithmetic coding now gives us a general way of doing this.
However, we can do even better:
Instead of just considering individual characters, we can also add in character pairs!
Of course, it doesn't make sense to add in every possible character pair, but for some of them it makes a ton of sense:
For example, if we want to compress english text, we could give a separate codebook entry to the entire sequence "the" and save a ton of bits!
To do this for pairs of characters in the english alphabet, we have to consider 26*26=676 combinations.
We can still do that: just scan the text 600 times.
With 3 character combinations it becomes a lot harder 26*26*26=17576 combinations.
But with 4 characters its impossible: you already have half a million combinations!
In reality, this is even worse, since you have way more than 26 characters: you have things like ", . ? ! and your codebook ids which blow up the size even more!
So, how are we supposed to figure out which character pairs to combine and how many bits we should give them?
We can try to predict it!
This technique, called [PPM](Prediction by partial matching) is already very old (~1980s), but still used in many compression algorithms.
The important trick is now that with deep learning, we can train even more efficient estimators, without loosing the lossless property:
Remember, we only predict what things we want to combine, and how many bits we want to assign to them!
The worst-case scenario is that your compression gets worse because the model predicts nonsensical character-combinations to store, but that never changes the actual information you store, just how close you can get to the optimal compression.
The state-of-the-art in text compression already uses this for a long time (see Hutter Prize) it's just now getting to a stage where systems become fast and accurate enough to also make the compression useful for other domains/general purpose compression.
That’s not what lossless data compression schemes do:
Yes it is.
You went into a lot of detail about how they do it, but it's still what they do.
I think the main point they're disagreeing with is this:
you wouldn’t be able to mathematically prove that the signal is perfectly recovered 100% of the time for all possible inputs
They explain why you don't need 100% accuracy - most compression codecs would only use the network for a prediction, which doesn't actually have to be correct. It just has to be "more likely to be correct" than existing algorithms.
If you want to read up more on the context of these prediction functions, the general class of compression algorithms you'd use for this are called prediction wavelet codecs. FLAC and arguably PNG are both prediction wavelet codecs.
Just wanted to give props to this super informative comment. Thanks for the write up and relevant links!
Seems like another “hey, what if we used LLMs for this” scenarios. It might be more effective, but exactly how many more resources are being used to make it do the same work as current compression algorithms? Effective doesn’t mean efficient and I think for lossless applications efficient is truly more important.
A LOT. You can barely run 13b parameter models on a 24gb gfx card and outputs are like a page or so of text. Translate that over to audio and it would have to be broken down into discrete chunks that the model could use as "prompts" to output a section of audio that fit into the models available output. It might compress better, but it would be exceedingly painful and slow to extract even on AI focused cards. And it would use OODLES of watts to get just a little bit better than flac.
13b parameters works out to about 9GB. You need a bit more than that since it needs more than just the model in memory, but at 24GB I'd expect at least half of it to go unused. And memory doesn't use much power at all by the way. LPDDR4 uses something like 0.3 watts while actively reading/writing to it.
The actual computations use more, obviously, but GFX cards are not designed for this task and while they're fast most of them are also horribly inefficient.
I run 13b parameter models on my ultra portable laptop (which has a small battery, no active cooling (fanless) and no discrete GPU). It has 16GB of RAM not GPU memory - RAM, and I'm running a full operating system, web browsers, etc a the same time. Models like llama2, stable diffusion, etc get perfectly usable performance without using much battery at all (at a guess, single digit watts while performing the calculations).
There is efficient hardware now and there will be even more efficient hardware in the future. My laptop definitely isn't designed to run these models and on top of that the models aren't designed to run on a laptop either. There's plenty more optimisation work to be done in the years to come.
Ok, it's been a while since I tried running a language model so I might have been thinking of the 30b models that were showing up at the time. The point remains though that this thing they were running would be well beyond hardware generally available and completely impractical for realtime use. Like.. why would you do all that when flac and png are good enough. It is far cheaper and uses less power to accommodate the slightly less compressed files.
Ok but what if we used LLM AND blockchain for this
Our company has been looking for a brilliant innovator like you, how would you like to apply for a new position called professional cool sounding tech peddler, I mean director of creative technology?
I want 200k and 30% of the company
Best I can do is $125k and $300k in company stock over 4 years
Im sure we can squeeze an nft in there somewhere
S m a r t c o n t r a c t s
That isn't really the case; while many neural network implementations make nondeterministic optimizations, floating point arithmetic is in principle entirely deterministic, and it isn't too hard to get a neural network to run deterministically if needed. They are perfectly applicable for lossless compression, which is what is done in this article.
It sounds like the actual compression isn't taking place within the neural net here, though.
I think this model has billions of weights. So I believe that means the model itself is quite large. Since the receiver needs to already have this model, I’d suggest that rather than compressing the data, we have instead pre encoded it, embedded it in the model weights, and thus the “compression” is just basically passing a primary key that points to the data to be compressed in the model.
It’s like, if you already have a copy of a book, I can “compress” any text in that book into 2 numbers: a page offset, and a word offset on that page. But that’s cheating because, at some point, we had to transfer to book too!
Yeah, it's like saying I can "compress" a png of the Mona Lisa to just the string "Mona Lisa" because I have a database of art.
I feel it's somewhere in the middle. Like your book example only works if you already have the book. If this is a model that is a few gigabytes of data, but it works for every movie or audio file it can still be useable. In that case it's not that you have to send the book first, but you do need to have the same dictionary.
🤖 I'm a bot that provides automatic summaries for articles:
Click here to see the summary
When an algorithm or model can accurately guess the next piece of data in a sequence, it shows it's good at spotting these patterns.
The study's results suggest that even though Chinchilla 70B was mainly trained to deal with text, it's surprisingly effective at compressing other types of data as well, often better than algorithms specifically designed for those tasks.
This opens the door for thinking about machine learning models as not just tools for text prediction and writing but also as effective ways to shrink the size of various types of data.
Over the past two decades, some computer scientists have proposed that the ability to compress data effectively is akin to a form of general intelligence.
The idea is rooted in the notion that understanding the world often involves identifying patterns and making sense of complexity, which, as mentioned above, is similar to what good data compression does.
The relationship between compression and intelligence is a matter of ongoing debate and research, so we'll likely see more papers on the topic emerge soon.
Saved 75% of original text.
I think this is a legitimate use case. It shouldn't have any security vulnerabilities beyond regular compression-related vulnerabilities.
The core to compression is prediction. Most compression algorithms work sort of like this:
If your guess is good it doesn't take much data to encode the difference. So the data stream is smaller.
AI image generation can be used to guess the data quite effectively, and it can use context that is hard to encode in classic algorithms (such as what a car looks like). This is basically the next step of shared dictionary compression (like what makes Brotli quite effective) where instead of building a dictionary as a simple Huffman table you compress the dictionary into the model weights. Since the model can do a pretty good job at creating "Image of a girl with brown hair looking right" you "just" need to encode the difference.
IIUC neither PNG or FLAC use pre-shared data, so sending a massive set of neural weights can be an advantage (and presumably you only need to send these weights occasionally).
An example of a compression algorithm that does support tuning parameters before hand is zstd.
Even if something isn't in a pre-shared dataset, I wonder if a sufficiently advanced LLM might be able to do well at compressing predictable but non-repeating data, such as "abc, bcd, cde, [...]".
Once we figure out how to get data out consistently 1:1 without hallucinations, the floodgates will open IMO.
And i'll be all over it personally, especially with FLAC files that range anywhere from 20MB to 70MB, any savings to rein these in closer to a typical MP3 will be much appreciated by myself. I don't mind long compression times, as 7zip and the other formats give us long waiting times already.
If AI accelerator hardware is able to speed up the data compression process, this is where I'd maybe start to get a bit suspicious, as these accelerators at the moment are included in various in IoT and camera SoCs. A single exploit is all that would be needed to theoretically allow the user's personal data to be siphoned off quickly, without noticing a change in the volume of network traffic, or negative impact to the performance of the IoT device
I never really had issues with flac in the last years, since 128+ gb micro sd cards got dirt cheap. But maybe my music taste isn't general/wide enough to fill up such a card fast.
If someone wants to know what music I listen to... they could just ask. Or point a directional microphone at the window from a distance (technology that has existed for almost a century now, and modern systems can likely work from miles away).
Either would be a lot easier than exploiting a zero day side channel attack which would surely be detected and closed.
Also, it can take years for a music track to go from an idea in the artist's head to being available for download by a consumer. If it takes a week to compress the file... that's no biggie at all. Only the decompression side needs to be power efficient.