(v04 12-18-22)
TO LOAD THIS PICO-8 CART, in immediate mode, type:
load #newcompress
Hello.
I have need of a data compressor for the 2023 project I'm working on now so I wrote one. And it's better than a 6-bit or 7-bit because once again, this is an 8-bit compressor.
Now this is different from the picture compressor I wrote which you can find HERE:
https://www.lexaloffle.com/bbs/?tid=45335
That compressor is looking for Pico-8 pixels specifically.
No, THIS compressor is looking for raw data. While I am demonstrating it with pictures it is in fact geared for byte data, maps, PCM sound, and what have you.
Usage:
There are 2-functions all squeezed down to a single programming line each.
The first is the compressor. It can be used 2-ways.
string=compres8(m,e,s,f)
compres8(m,e,s,f)
(251 tokens in size)
It uses 4-arguments and the string in the beginning is optional. By default it will save your compressed data to the clipboard for immediate use. If you want to save it to a string then put that variable in front.
The first is "M" the memory location to start where you want to begin your compression. You can also use 0x8000 high memory if you so choose.
The second is "E" the length to read. For instance the screen at 0x0000 is 0x2000 or 8192-bytes in length. You can also use a length greater than 0x2000 if you so choose.
The next is "S" for shift. Now by default this is zero. With some fiddling of it you can shave off a few bytes from your string compression. Basically it SHIFTS the value so for instance if you had a bunch of "\0" in your code, by shifting it 128 you would then get █ instead which registers as a single character whereas "\0" registers as 2-characters from your 64k available.
However you should never find \0\0\0 beside itself ever in the final compressed data. Everything gets compressed when a pattern is found.
If you don't want to mess with it, don't enter a value or if you need the next argument, leave it at zero.
The last argument is "F" for flag. If it is anything except NIL then it will not convert to a clipboard friendly string, instead it will save the image to a true 8-bit string that can only be output but not pasted in your code. You will need the starting variable for this.
It is for debugging and you shouldn't need it unless you come across some data that is not compressing properly and you need to show this to me so I can isolate the problem.
However if you are not debugging and just using the compressor normally, after you have called the compressor it is ready for use !
In a line of your code, press CTRL+P, CTRL+V, then CTRL+P again followed by the ENTER key to confirm. That will save your data to the string in your code, and of course, you can rename the variable data to whatever you want.
It will look something like this:
data="\0 ¥たみまたみまたみまたみまたみまたみまたみまたみまたみま█ ... etc " |
Then you are ready to decompress it as described in the next function to follow.
decompres8(string,o,s)
(126 tokens in size)
The 1st argument is for the string that you created with compres8(). It will be a series of odd characters within 2-quote marks. See code sample above.
The "O" (oh) variable is for the memory you wish to start writing it to. You are not limited to 0x0000 or 0x6000, you can also write to 0x8000 if you so choose.
The "S" is exactly as above. It SHIFTS the value. So if you recorded using compres8() with a shift of 64, then you need to use decompres8() also with the value shift of 64.
And that's it ! Please see the sourcecode for the sample cartridge above for further information, especially on usage of the 2-functions.
Now I wrote these functions for myself as I need them for the main project I'm working on. However if you find them of use yourself or have any questions to this point, please let me know.
Thanks !
Love your carts and nice comments, nice to be the first one seeing one of your posts!
Also, really cool how you got this all to fit together, nice work!
Well thank you so much, @CrownDeluxe ! That is very kind of you ...
I have tested it rather thoroughly, not just for images, but SFX, maps, and extra types of data that I need for my project.
Hopefully this will be of use to others in their programming endeavors ...
What is the goal compared to px9? much smaller compression/decompression code?
Zep px9 code uses more tokens but yields far better compression ratio (including conversion to string) - for example, first screen produces a 4748 char string (vs. yours at 8184).
P#123162 [under spam review]
Why me ... I'll delete this once that spam above is gone.
@freds72, Yeah, I think that's the goal, I know it is with the compression systems I've made (my latest one being just about 50 tokens). In general, though, I think looking at the character count in the strings isn't terribly helpful, as Pico-8's built-in compression usually cuts the byte count down quite a bit (except for bit-packed data like PX9's string output), and compressed bytes is what really counts if you're storing data in the code area.
@JadeLombax, guys, please, talk about Px9 on the Px9 thread.
As I have already stated, this is for compressing data, not pixels.
Thanks !
@dw817,
I wasn't really talking about PX9, I was just stating that for judging the effectiveness of a string-based compression system, you need to look at how many compressed bytes the strings take up.
My code is especially for cutting down the code size of the limited 65535-chars. I am aware that compression takes place for the 15616-bytes otherwise. If it was just P8's own internal compressor I wouldn't have written this utility.
But we are bound to the 65535-chars of coding, is why I wrote it.
And I choose not to argue, @JadeLombax. @freds72 has been negative on me in other carts I've posted throughout the years. Stopping previous pleasant conversations in their tracks. Always.
Only him. It just gets tiring for me after a-while.
Fair enough. Just wondering, though, have you made carts where you actually ran into the character limit? I guess besides possibly a slideshow of images with a lot of black in them and therefore a lot of "\0" sequences, I'm curious as to what kind of data would do that, as I generally run out of compressed capacity first.
Hi @JadeLombax. Thanks for your interest !
You have the option to SHIFT. I was pretty excited when I wrote that. :)
This allows you to convert "\0" and "\48" to "\57" to a higher character set that does not use 2- or 3-characters for code.
The easiest way would be to add 32 so a compression field would be:
compres8(m,e,s,f) |
Where "s" is the SHIFT value. In this case, 32. To decompress you must use the same shift value as well.
decompres8(string,o,s) |
Where once again "s" is for shift. In this example, 32.
So instead of \0 in your compression you would get char #32, space, and the characters \48 to \57 would change to p q r s t u v w x y.
Char #255 would become char #31, etc.
As for running out of space, I never have. I imagine my carts have never been incredibly ambitious - except for Mildew's Manor, that game I wrote that many years ago.
https://www.lexaloffle.com/bbs/?tid=31730
I suspect a lot of things killing that 64k in coders are long variable names. I came originally from an age where you could only have 2-characters maximum for a variable. While I do give important variables names up to 4- 5-characters, I seldom exceed that total.
Hmm, interesting idea there in the shift feature. Never thought to make the value offset user-defineable, though in my latest compressor all the values are reversed (255-v) when converted to and from string form, as this usually gets rid of lots of escape sequences for zero and low-value pixels or bytes, and only adds a couple tokens to the decompressor code.
Inversion is a good idea at that, @JadeLombax. Something I hadn't considered. Making it user-definable though might be better.
You could for instance have a whole mess of 255's and that would change back to zeroes \0 . Also these values are getting compressed so you will never have 3-numbers precisely the same unless the COUNT matches also the number you are compressing. Even then it's still smaller than no compression at all.
Hello @dw817, I was wondering if the idea of creating a compressor it's still relevant to this day for PICO-8?
Based on your experiments, I tried to create something that can crunch more data, so I implemented an algorithm but to compare with yours, I would need to test with the same images you used for your cartridge.
May you help me find the images to test and post the results?
Thanks
[Please log in to post a comment]
