Image

Compress Links For Optimal QR Codes

Link shorteners have been a staple of the online world for over two decades now, but they’ve got some issues– for one thing, it’s totally non-transparent where the link actually goes, leaving you open to all sorts of shenanigans, of which RickRolling is probably the best case. For two, your traffic is going through an external service who may have their own nefarious intent. [PortalRunner] had an idea: don’t shorten the link, but compress it.

You see, a traditional URL shortener like tinyurl just generates a random code and associates that with your original link in its database. That’s fine, but you’re relying on a third party database. The alternative is to take the URL, encode it in some way, and apply some compression algorithm to the data. If the encoding and compression are open-source– which [Portal ]’s absolutely are— then you can check yourself before following the link, and/or self-host the whole thing for piece of mind. As a bonus [Portal]’s Ha.mr– that’s pronounced Hammer– also gives you a QR code optimized for easy scanning. QR codes have a specific alphanumeric character set built in, and it isn’t the full UTF-8– if you naively use random text, you’re in byte mode, which needs a lot more QR real estate. Or inverting that, the fewer bits it has to store, the easier a qr code is to scan at the same size. The text version of the compressed links can use UTF-8– including emoticons– but they don’t have to.

The whole project has a “why isn’t everyone doing it this way” vibe about it. We’d probably want to self-host this if we were using it seriously– [Portal] put this together on a lark and makes no promises it will be online forever–but again, this is open source, so we can. [Portal] is using normal compression algorithms here, but if you really want to squeeze text, use a neural net. 

Continue reading “Compress Links For Optimal QR Codes”

Image

AI Can Now Compress Text

There are many claims in the air about the capabilities of AI systems, as the technology continues to ascend the dizzy heights of the hype cycle. Some of them are true, others stretch definitions a little, while yet more cross the line into the definitely bogus. [J] has one that is backed up by real code though, a compression scheme for text using an AI, and while there may be limitations in its approach, it demonstrates an interesting feature of large language models.

The compression works by assuming that for a sufficiently large model, it’s likely that many source texts will exist somewhere in the training. Using llama.cpp it’s possible to extract the tokenization information of a piece of text contained in its training data and store that as the compressed output. The decompressor can then use that tokenization data as a series of keys to reassemble the original from its training. We’re not AI experts but we are guessing that a source text which has little in common with any training text would fare badly, and we expect that the same model would have to be used on both compression and decompression. It remains a worthy technique though, and no doubt because it has AI pixie dust, somewhere there’s a hype-blinded venture capitalist who would pay millions for it. What a world we live in!

Oddly this isn’t the first time we’ve looked at AI text compression.

Image

Text Compression Gets Weirdly Efficient With LLMs

It used to be that memory and storage space were so precious and so limited of a resource that handling nontrivial amounts of text was a serious problem. Text compression was a highly practical application of computing power.

Today it might be a solved problem, but that doesn’t mean it doesn’t attract new or unusual solutions. [Fabrice Bellard] released ts_zip which uses Large Language Models (LLM) to attain text compression ratios higher than any other tool can offer.

LLMs are the technology behind natural language AIs, and applying them in this way seems effective. The tradeoff? Unlike typical compression tools, the lossless decompression part isn’t exactly guaranteed when an LLM is involved. Lossy compression methods are in fact quite useful. JPEG compression, for example, is a good example of discarding data that isn’t readily perceived by humans to make a smaller file, but that isn’t usually applied to text. If you absolutely require lossless compression, [Fabrice] has that covered with NNCP, a neural-network powered lossless data compressor.

Do neural networks and LLMs sound far too serious and complicated for your text compression needs? As long as you don’t mind a mild amount of definitely noticeable data loss, check out [Greg Kennedy]’s Lossy Text Compression which simply, brilliantly, and amusingly uses a thesaurus instead of some fancy algorithms. Yep, it just swaps longer words for shorter ones. Perhaps not the best solution for every need, but between that and [Fabrice]’s brilliant work we’re confident there’s something for everyone who craves some novelty with their text compression.

[Photo by Matthew Henry from Burst]