I'm pleased to announce picotool, a set of tools and Python libraries for manipulating Pico-8 game files. picotool can read .p8.png and .p8 files, and can write .p8 files.
To get picotool and learn how to use it, visit the picotool Github repo.
The p8tool command is a multi-tool with multiple functions. You can run it like this:
% p8tool stats helloworld.p8.png hello world (helloworld.p8.png) by zep version: 0 lines: 48 chars: 419 tokens: 134 |
picotool currently includes the following tools:
- build: builds a cart from pieces of other carts
- stats: a statistics reporting tool
- listlua: prints the Lua region of a cart
- writep8: converts a .p8.png cart to a .p8 cart
- luafmt: writes a new .p8 cart with Lua rewritten with regular formatting, to make it easier to read
- luamin: writes a new .p8 cart with Lua rewritten to use a minimal number of characters, to make it difficult to read
- luafind: searches for lines of Lua code containing a string or matching a pattern
- listtokens: prints a visualization of Lua tokens
- printast: prints a visualization of the Lua parser abstract syntax tree (AST)
The Python library includes a hand-written Lua parser with support for Pico-8 features, with API access to the token stream and abstract syntax tree. Additional modules provide access to the graphics, map, sound, and music data. All game data can be transformed or created with Python code and written to a .p8 or .p8.png cart file.
I've included the Lua minifier (luamin) as an example tool, though I don't actually think it's a good idea to use it. Statistically, you'll run out of tokens before you run out of characters. The Pico-8 community benefits more from published carts with code that is easy to read and well commented. All luamin does is make the code difficult to read, without much benefit.
Here is a chart of the character count, minified character count, and token count of 496 carts that have been posted to the BBS, relative to the Pico-8 maximums:
As shown, even un-minified code tends to stay below the token count, percentage-wise. Minifying reduces the (uncompressed) character count to 65% of the original size on average.
The inspirational use case for this library is actually luafmt, which re-formats the Lua code of a cart to be easier to read. The current version of this tool is simple and only adjusts indentation. There is much more luafmt can do by analyzing the AST, but this is not yet implemented.
As of today, this is very much a v0.1 early release. There are known issues regarding parsing and token counting, and the library APIs are incomplete and messy. This project has already gotten away from me a bit, so I'm not sure how far I'll take it. But if you do want to build against it, patience with non-backwards compatible changes will be appreciated. :)
Let me know what you think! Thanks!
Update 2015-10-29: Added luafind tool. Refactored AST walking code to make it easier to build tools that read or transform the parser tree.
Update 2016-10-17: Added build tool. Added support for updating and creating .p8.png cartridge files.
Do you know if compacting the code saves on tokens? Sort of like javascript compression, where the code becomes basically unreadable/usable but saves on space?
hseiken: By definition, minifying the code as luamin and JavaScript minifiers do does not change the token count. They only reduce the number of characters used to represent long tokens, such as by renaming variables. Pico-8's token count already excludes comments and whitespace, and only counts each name as a single token, regardless of its length. (Try the listtokens tool for an illustration of how tokens are defined and counted.)
There may be a few token optimizations that a tool could do automatically, but they'd be pretty smart compiler-style optimizations that actually rewrite the semantic structure of the code to a shorter equivalent. I'm thinking about building a tool that does dead code elimination, for example. (No promises, I might not have time, but it's the kind of thing you can do with access to the AST.)
I'm pleased to announce that picotool now supports writing .p8.png files, including code compression.
I've also started a new build tool. Its features are modest for now: it can replace sections of a cart (Lua, spritesheet, sfx, etc.) with sections from other carts, and it can also read Lua code from a .lua file and add it to a cart. It can also create carts from scratch and erase sections.
I have plans for more compelling features for the build tool (require() support, dead code elimination). My summer sabbatical is coming to an end so I might not get to it right away, but I thought I'd push what I have so far.
The new .p8.png code includes the code compression routine, the dual of the decompression routine that we've had for a while. Thanks to zep for providing the last few missing pieces!
Enjoy!
[Update: fixed] Oof, just noticed picotool's compressor is compatible but not efficient, and is not producing the same result as Pico-8's compressor. It's bad enough that I don't think this is quite ready for the build workflow of a real (large) game. I'll update again once I've fixed it, but I'm mentioning it in case I don't get around to it right away. (My vacation is over! :( )
Tracking bug: https://github.com/dansanderson/picotool/issues/7
I made a quick fix by temporarily replacing my compression code with a direct port of the Pico-8 version. It'll be worth going back and looking for differences, if only to speed it up. But this'll do for now.
Dear dddaaannn.
I am trying to use your picotool to minify my game. Currently I have:
65302/65536 char count and 8136/8192 token count.
When I am trying to use luamin on my cart I get:
C:\Python34>python p8tool luamin saveyourself.p8
saveyourself.p8 -> saveyourself_fmt.p8
warning: token count 29022 exceeds the Pico-8 limit of 32768Traceback (most rece
nt call last):
File "p8tool", line 8, in <module>
sys.exit(tool.main(sys.argv[1:]))
File "C:\Python34\pico8\tool.py", line 512, in main
return args.func(args)
File "C:\Python34\pico8\tool.py", line 349, in do_luamin
return process_game_files(args.filename, luamin, args=args)
File "C:\Python34\pico8\tool.py", line 210, in process_game_files
procfunc(g, out_fname, args=args)
File "C:\Python34\pico8\tool.py", line 242, in luamin
lua_writer_cls=lua.LuaMinifyTokenWriter)
File "C:\Python34\pico8\game\game.py", line 746, in to_file
self.to_p8_file(outfh, *args, **kwargs)
File "C:\Python34\pico8\game\game.py", line 659, in to_p8_file
for l in self.gfx.to_lines():
File "C:\Python34\pico8\gfx\gfx.py", line 95, in to_lines
yield bytes(newdata).hex() + '\n'
AttributeError: 'bytes' object has no attribute 'hex'
|
I have ton of text packed into loooong strings if that helps.
Please help me, as I can't even export my game to html at this point.
Thanks in advance!
@dagondev: Can you share the cart with me so I can troubleshoot?
It's likely that the cart just doesn't compress under the limit and there's not much to be done, but I'd like to clean up the error messages at least. If we're lucky, minification will get it just under the wire, but no guarantees. :)
Hey, I wrote followup at issue page in github: https://github.com/dansanderson/picotool/issues/9
Here is the code: https://bitbucket.org/dagondev/save-yourself/src
I ended up minifing my code by hand, so no pressure now... But if you get this working, that would mean I could fix bugs on non minified version - so I would be grateful regardless. :)
Replied on the bug but will summarize here for reference: I need to add support for glyphs and labels before I can properly parse your cart. These are likely unrelated to the symptom you're seeing but I can't repro otherwise, and they're notable shortcomings regardless. I'll keep the bug open and revisit once I've had a chance to make these changes. Thanks for the report!
@dagondev:
what os are you using?
Bacause I have the same problem on andriod's qpython and I solve it.
Maybe you need to use new python 3 version.
I'm having a strange problem; I'm attempting to use writep8 on Mac OS Sierra/Python 3.5 to convert p8.png carts to .p8 carts (to be edited in Sublime Text). However, every time I've attempted to do so, p8tool reports:
../carts/cartname.p8.png -> ../carts/cartname_fmt.p8.png |
and sure enough, when I check the resulting output file, it's still a *.p8.png cart, and attempting to read it with cat or Sublime only yields byte data. Any ideas what could be wrong?
EDIT: The most recent cart this failed on was the TinyTV Jam base cart, found here: https://www.lexaloffle.com/bbs/?tid=28480
It also failed on the Emily Quest cart found here: https://www.lexaloffle.com/bbs/?tid=2294
I'm looking forward to trying out luamin over the weekend. I must be one of the rarer cases that have hit the character limit before the token limit. I am parsing long strings in order to reduce token count which is where most of it would be going.
Hi Dan, I understand if you aren't interested in supporting this. But just on the off chance;
I have just run p8tool luamin on my pico quest card #37305# (found here: https://www.lexaloffle.com/bbs/?tid=28574) and got a syntax error when running (on startup):
syntax error line 99
....
'then' expected near 'cg'
radcore: Good find! I filed this as a bug: https://github.com/dansanderson/picotool/issues/11
I'm going to blame Pico-8 for this one because it appears to be a shortcoming in its short-form "if" statement that luamin doesn't take into account. luamin will collapse space between a non-word non-space character and a word character. This is valid for Lua, but breaks Pico-8's short-if, which is implemented as a preprocessor replacement.
x=true y=(0) if (x) y=1 print(y) |
minifies to
x=true y=(0)if (x) y=1 print(y) |
Pico-8 doesn't recognize the "if" as a short-if and fails to insert the "then" and "end" keywords before parsing the Lua. (picotool's own parser doesn't see the problem because I implemented short-if as a formal part of the grammar before I realized that Pico-8 was using a preprocessor replacement.)
I see a similar issue for a space before a "local" keyword being collapsed and causing problems. I'm less sure of the cause within Pico-8 but the solution is similar: it needs leading spaces preserved.
You can work around the short-if issue by avoiding short-if (using if-then-end) in problem locations. Unfortunately the workaround for "local" would be to make sure the previous line ends with a word character (and not in a comment), which is not a good workaround. As a cheap hack, you could write a little script that replaces "local" with " local" after the minification.
If you want to experiment, I found the first problematic short-if on line 454 of your original cart. Let me know if you find other cases beyond short-if and local. Sorry for the inconvenience.
Awesome, thanks for getting back to me :) I will try replacing my short if's and let you know how I go. I'm hoping I ultimately can get by without the luamin tool. But it is a great backup plan.
Cheers!
A few recent changes to picotool:
BREAKING CHANGE: All Lua code is now handled in the API as bytestrings and not text strings. Pico-8 uses a custom text encoding equivalent to lower ASCII plus glyphs as high bytes. Instead of trying to manage conversion between these high bytes and Python text strings via an encoder, I just converted the whole thing to keep Lua code in bytestring form. This is a breaking change for the parts of the API that accept or return strings for/from the Lua code (token data, cart title, etc.). I probably busted stuff in the process so let me know if you see bugs.
With thanks to juanitogan, the minifier knows about more Pico-8 built-ins (and so doesn't munge their names). Also, a few bits were using an API only available in Python 3.5. I replaced those bits to use juanitogan's suggested Python 3.4-compatible equivalent, so Cygwin users can play. (Cygwin is currently still using Python 3.4 as its 3.x package.)
I added support for loading and saving .p8 files that have a label. This does not yet know how to convert labels between .p8 and .p8.png carts. (That'd be cool, especially since the same code could also support importing and exporting PNG images as spritesheet or label data.)
Hallo,
I just tried luamin with the latest Version. The generated _fmt.p8 gives me a Syntax error:
SYNTAX ERROR LINE 67 ... SYNTAX ERROR NEAR 'END' |
Tested with this cartridge:
Also 'stop' was replaced with an undefined function..
I would be so happy if this would work
OK... probably I changed something while testing. Thanks alot
@Sascha217: Thanks for the report. I submitted a few fixes to get your cart to go through luamin, including adding "stop" to the keyword list. There's still a syntax error in the result, and I believe this is related to issue #11 regarding luamin being too aggressive with eliminating space between non-word chars and keywords. I can edit the result and insert spaces to resolve some syntax errors (though this needs to be done throughout). I'll look at this issue more this evening (PT).
One more thread bump to say that I've committed another batch of changes and small improvements. This includes support for a few popular but unofficial or accidental quirks in Pico-8's Lua syntax. if-do is now supported, as are C++-style // line comments. If you pulled since Feb 22, please pull again.
I'm pleased to say that with this last set of improvements, picotool successfully recognizes as valid all valid carts published to the BBS up to cart ID 28280. In this set there are four carts with errors that I've jailed for obscure reasons where Pico-8 accepts them but I don't see a good reason. I'll do a fresh crawl and test again on more recent carts. (This testing methodology only shows that picotool accepts these carts as valid. it does not prove that it produces correct parse trees. :) )
just tested the latest version... My minified cart has still systax errors
https://mega.nz/#!qN1RnLIC!DLN9Kq2-KI7W8EdR6IEENBY-ulp6HRZVxrvjwch424M
needed to change, amongst other things
bs[jp][jq]=jn[jo]jo+=1 |
TO
bs[jp][jq]=jn[jo] jo+=1 |
also added a whitespace before each 'end' and 'local'. Probably that are way to much whitespaces. But anyway, this reduced the compressed code size from 15 to 10kb.. Thank You
Sascha217: I submitted a fix, please try again. I can minify your cart and it runs successfully. mario005.p8 goes from 41096 uncompressed to 233353 uncompressed, and from 15743 compressed to 10389 compressed.
It's a quick fix and may or may not be a complete solution to the problem. (In theory it might actually add chars in trivial extreme cases, but I'm not worried.)
Pretty significant compressed chars savings that I desperately need. Nice work. I did run across a bug.
Example:
print("Sum: "..A+1 .."!")
|
A whitespace is required after the 1 so that the following period doesn't get misconstrued for a decimal point. Right now, the script creates a "malformed number" error when the program is run.
FunFetched: Fixed, please try again (without the space). I broke that case to support "4." as a numeric literal, but I can support both with a lookahead assertion. :)
Again.. thank you very much. Works now!
Following warning is outputted:
| warning: token count 27725 exceeds the Pico-8 limit of 32768 |
Works anyway... The token count was probably increased to 65536
Sascha217: Can you send me the cart that produced that warning? 27725 < 32768. :) Thanks!
Thanks. The issue was just in the error message. The actual token limit is 8192, so 27797 definitely exceeds that. Also I can't save your cart as a .p8.png because the compressed code size is too large. You may need to consider a multi-cart solution.
strange.. it's OK here
| tokens: 7763/8192 program chars: 27911/65536 compressed: 12656/15360 |
Sorry, it was still reporting chars as tokens as of my previous message. You don't have 27797 tokens. :)
picotool does still appear to have a significant disparity from Pico-8 in token counting. Other than doing rigorous black box testing (with a lot of manual effort) through various parts of Lua syntax, there's not much I can do to nail down the last few differences. I'd like picotool to be accurate so it can report when it's building a cart that won't fit, but maybe that's an impractical feature for a weekend hobby project.
p8tool stats mario010.p8 reports 8560 tokens, 47866 chars, 18710 compressed chars. Pico-8 reports 7719 tokens, 47865 chars, 18720 compressed chars. So picotool erroneously overcounts and warns that it might be over the limit. (Then on top of that I had a broken message, which I'm about to fix.) Shrug emoji.
Found something else. menuitem() is a function in the PICO-8 API, but it gets clobbered by the script. I went ahead and fixed it on my end by adding it to PICO8_BUILTINS in lua.py.
about luamin,
I get a lot of these (the dreaded short-if):
if(a) then if(b) c=1 end |
gives this
if(a) then if(b) c=1 end |
which doesn't work in pico ('then' expected). I replaced all 'if' by '\nif' afterwards as a workaround.
now I have function "gu" set to nil or something... (*)
luamin does both variable renaming and separator optimizing, a way to disable the one or the other would be nice!
(*)edit: needed __index in PICO8_BUILTINS, works a treat!
thanks a lot!
Added these to the issue tracker. Thanks!
This is a cool tool. I do have a minor issue to report: it was failing to load my game because I was using an alternate form of the print command that's smaller with fewer tokens.
PRINT("HELLO",1,1)
becomes
? "HELLO",1,1
Picotool was hanging at the question mark.
I don't believe this is a bug or deprecated form or anything like that, it's been mentioned in Pico-8 changelogs before. In fact you could probably incorporate it into the minifier to replace PRINT.
(Mentioned in Discord but posting the link here for posterity:) Bug tracking link for short-print support: https://github.com/dansanderson/picotool/issues/39
Thanks for the report!
Love this tool! But can't seem to get luamin to work on my cart. Keep getting syntax errors about nil values. I assume it's messes with my table references? Any idea what might cause that?
Can you post or send me a sample? Feel free to just post it as an issue in Github. https://github.com/dansanderson/picotool/issues
Will do, thanks! And thanks again for developing and supporting such a useful set of utils.
Nice tool! I want to use it to modularize my project, splitting code into different files and separating code from data. However, the PICO-8 editor directly puts sprite and music data into the game file (p8 or p8.png). I need a method to extract those data chunks and export them into separate files. Then, I can call "p8tool build" with the options to import the data files back into the final game file.
This would allow me to version control each data file alongside the code, and to exclude the built game file from version control. Currently, I would need to track the main game file because it's the only file containing my data, which means I'm tracking the code twice: in the source lua files and in the game file.
Without a data export script, I would need to copy-paste the data chunks to data files manually before each commit.
From what I can see, exporting data is feasible with the current API. list_lua uses g.lua.to_lines() to access the lua chunk, and we can similarly access other chunks.
Ex: for l in g.gfx.to_lines(): print(l)
We could output these lines in the correct format in some game.gfx file. Same for other data chunks.
For now I'll just make a simple data export script for my own game, but since you offer a tool to build a game from both code and data, I think such a feature would be a nice addition to the core tool package.
@huulong: The "p8tool build" command already does exactly what you describe. You would keep your graphics data in a Pico-8 cart file of its own, then pull it in at build time:
p8tool build mygame_final.p8.png --gfx mygame_gfx.p8.png --lua mygame.lua |
If "mygame_final.p8.png" does not exist, it'll create it, using empty sections for any section you don't specify. If it does exist, it'll preserve any section you don't specify.
It sounds like you want to track source files in version control, and not track the final built output. In that case, you could either 1) specify all options to p8tool build to create a fresh cart during the build, or 2) keep everything but Lua in a source cart, copy that cart to the build location, then run p8tool build to add the Lua to that cart.
Yes you can write your own tool using the library if you'd like. You can look at the source for p8tool build for ideas on how to use the API. But it sounds like your use case is covered by the existing tool.
Hm, I hadn't considered working on a cartridge made purely of data. That would allow me to work in the P8 editor, save the data cartridge then build the final game, without even the need to extract data. I will try this today, thanks!
And it works! I just started my project, you can see the project structure with sublime text build commands here: https://github.com/hsandt/what-happened/tree/644c4a21a777f187da87feb4b7b7c9b1d736a5ed
I can open the data to edit it and save it, build and run the game, or just build it if I prefer reloading the game with Ctrl+R to avoid a full restart.
I have another problem now. I'm trying to unit test my modules with pico-test (https://github.com/jozanza/pico-test). As you explained in the README, we should be able to create lua scripts containing their own tests. Except it doesn't work if you use the "local module = {} / return module" pattern, because building a cartridge directly from the script will preserve the "return" which causes a parsing error (you're not supposed to do that in a normal game script).
So either you need to use one of the 2 other export methods (global function or global module), or you need to build from a main testmodule.lua script which will require the said module. In this case, you can either keep the test on the module side and just call them, or move the tests to the testmodule script. But this method requires an extra test runner script for each module.
What would do recommend, and do you have other workarounds?
[Please log in to post a comment]
