Hi All,

I've a suggestion that I believe has some value to people making tooling chains is the ability to store metadata within the text version of a cart cart.

This would allow for saving anything that an external editor of a p8 file would want to save and restore upon loading a cart.

For example storing custom pallets or even palette transitions means that they could draw their sprites and see how they look in a fade/palette swap. If they had a fade that they liked they could work in the sprites in an external tool. Anyone else using the same tool could see the settings that were applied.

Pico8 could also store its own metadata there, like the undo history. so if someone reloaded or switched platform they could pick up where they left off.

"save the .p8 on your desktop, move it over to your handheld while you travel, and then back to desktop, and there's no need to get back in the groove manually each time you load up the file" - this is what someone suggested on discord, I'm not naming them in case they don't want to have their name associated with this post.

Even something as simple as linter settings would be useful for people who work together.

How I imagine it would work.

the p8 file is already made of blocks like __lua__ __sfx__ etc. I suggest that another block is put in that pico8 doesn't read but does keep on save. You would be free to specify any limit in size that tools have to agree too.

This could be in any format you like. Raw text, json, a hex byte array similar to the binary blocks we have at the moment.

In this way a tool could use the extra space, but I wouldn't expect this data to be retained when in PNG or any of the other binary formats. It can't be abused to get extra space in the cart as it would not be visible to the user code running within pico8.

I just thought you may appreciate the suggestion as it seems pico8 and picotron are both going down the external tooling path.

I think this could be useful.

Have a chunk type in the file that's considered metadata, which is data PICO-8 or an external tool knows is there and can pass along, but doesn't have to parse unless it wants to and knows how.

Maybe let there be any number of these chunks, and have them identify themselves so that interested tools can read them if desired.

The first line of the chunk would be a type identifier, e.g. "editor state", that any tool, including PICO-8, could choose to recognize or not, if it knows the format of the identified block. If it doesn't, it only needs to preserve the contents when saving the file again.

Here's an example that would allow the PICO-8 IDE to remember some basic stuff, so that when you save the file and open it again, you're right back where you were, even if you open it on a different device:

pico-8 cartridge // http://www.pico-8.com
version 35
__lua__
print("hello world")
__gfx__
0000 ... etc.
__meta__
p8 source editor state
view_tab=0
view_y=11
view_x=0
caret_offset=287
select_offset=293
...etc. - *anything* except a line that looks like __xyzabc__
__meta__
p8 sprite editor stamp
0808
00777700
07777770
77077077
77777777
70000007
70000007
07000070
00777700

And, again, if some tool is stupid and doesn't preserve the __meta__ chunks, then that's okay. It's just slightly inconvenient that the next time the file is opened, the editor state will be all defaults, but none of the actual content would be lost.

(Heck, you could even save the undo chain! No edit history lost on exit/reload! Probably not a great idea, though, could see it causing problems with source control and merging changes, etc. But still, a cool thought!)

I do like the idea of a general metadata section, but in the past felt YAGNI. The two pieces of metadata (title and author) are optional and so it made sense to store them as comments in the code to get rendering and editing for free. I guess also the tab 'labels' are similar. However, this approach only really makes sense for short human-editable/readable snippets of text.

I've added __meta__ to 0.2.4c which may or may not be utilized by future PICO-8 versions :)

It is a single block of text to make preserving that section easier for tools that otherwise don't care about the contents. To avoid collisions between tools, I would suggest instead using sub-section markers:

__meta__
__meta:super_palette_editor__
  // arbitrary multi-line data that super_palette_editor
  // knows how to handle and where it ends.
__meta:bobs_custom_preprocessor__
  // arbitrary multi-line data that bobs_custom_preprocessor 
  // knows how to handle and where it ends.

.. but that's for the first tool writers who modify the meta section to decide! PICO-8 doesn't need to know about it, yet.

PICO-8 would need to 1) ignore a section with that heading, and still find subsequent sections, and 2) preserve a section with that heading untouched when saving .p8 files. Sounds good for 0.2.4c!

We might want to document some simple rules to keep first-party and third-party parsing demands to a minimum, so PICO-8 and others know how to avoid corrupting the metadata section while preserving it. For example, PICO-8 can declare /^__.*__\s*$/ a reserved line pattern for all lines, such that if a section body contains such a line the behavior is undefined.

I notice that PICO-8 already isn't happy about some such lines in the Lua body. Typing the following in the code editor, saving as a .p8, then reloading fails in a predictable way (as of 0.2.4b). Whether this is an important bug or not I don't know, but it's an example of how third-party tools can fail without some guidance as to how to parse. If the file format spec were to allow the following (and PICO-8 is currently non-compliant ;) ), then all .p8 handlers would need partial Lua tokenizers to know when to ignore something that looks like a section separator.

print("hi")
--[[
__gfx__
--]]
print("bye")

> I've added meta to 0.2.4c which may or may not be utilized by future PICO-8 versions :)

> It is a single block of text to make preserving that section easier for tools that otherwise don't cart about the contents. To avoid collisions between tools, I would suggest instead using sub-section markers:

The subsection headings are exactly the format I had in mind and @Felice had as an initial idea.

However from your code block I want to be clear.

__meta:super_palette_editor__ This isn't a block header is it? Its just a line that looks like a header within the meta block?

@zep do the order of the blocks matter in file handling. If so I suggest this is always the last block as its least likely to get in your way reading.

I've no idea how you save the files if you use random access or you treat the file as a stream or you load all the contents of the p8 file into memory. If you do load the file contents into memory I strongly suggest that you define a limit to how much data can be stored in __meta__ and that conveyed to tooling authors. (In my head I thought 8k but that's completely a completely arbitrary amount of data)

Also I wanted to point out that @merwok thought that this would be a problem as the meta would be lost between changing between .p8 and .p8.png formats. I disagree with that being a problem but I thought I'd mention it for completeness sake especially if it makes you change your mind.

Hooray! Thanks, zep! 😀

Edit: BTW, as for the metadata being stripped from .p8.png, I think it's unfortunate but it's also understandable. It's a bit like losing all of your local settings and stuff when you commit your changes to git for other people to see. You only distribute the content files, not the local workspace stuff.

Good point @dddaaannn -- The __meta__ section indeed can not contain arbitrary data as the section headings are reserved.

To make parsing easier, and because the .p8 format won't need to be / shouldn't be extended in the future (I can use __meta__ if needed), the rule can simply be that the 9 reserved section headings can only start a line when they are acting as section headings:

__lua__
__gfx__
__label__
__gff__
__map__
__sfx__
__music__
__change_mask__
__meta__

This is what PICO-8 currently assumes when parsing, and as noted doesn't bother differentiating section headings appearing in code even inside multi-line comments.

When serializing a blob of data in base64 or something that might produce a section heading by chance, tool authors can just make sure it won't be at the start of a line:

base64_blob=hIue78a__music__3jI-ASz12

@SquidLight That's right: __meta:super_palette_editor__ is part of the content of the __meta__ block, it just looks similar. Perhaps better to choose a different subsection heading format to make that clearer.

An additional requirement for tools parsing / writing __meta__ blocks is that they also need to handle the subsection headings.

This is assuming a need for subsections at all -- maybe just a free-for-all key:value collection would work just as well and be easier to parse. I don't have strong enough feelings about it to make that call ~ anyone please feel free to jump in to suggest a __meta__ format!

On order: Any section can legally appear anywhere in a .p8, but __meta__ should normally be written at the end for human-readability.

On stripping for pngs: I think it feels right because I regard .p8 as the 'development project' format and .p8.png as the distributable.

On size: I've set the limit for now at 256k -- I can live with the existence of a 300k .p8 file and want it to cover some unlikely but interesting heavy cases. For example, someone might want to store Moonscript or verbose code in __meta__ that is transpiled to Lua or minified to produce the code in __lua__. Again, feels ok to me as part of 'dev project' data.

Okay this is my attempt at defining a meta format.

__meta__
#palette-editor:v1
key=value

#pico8
view_tab=0
view_y=11
view_x=0
caret_offset=287
select_offset=293

//END OF META BLOCK

__gfx__

So really simple blocks start with #tool-name with an optional :{version} at the end. The last line of the block should be a empty line followed by \n as its easier to see separation this way.

Allowable names are [a-z0-9\-]+ followed by an optional : and version info which can have [a-z0-9\.\-]

Upper case characters should be discouraged to keep in line with pico-8 source code (which confusing is all lower case)

Followed by 0 or more lines of data.

I think the data format inside the block should be up to the tool. I'd probably try to use
key=value format. But I could see other people wanting json/yaml or base64 encoded blocks. or even hex bytes in a similar format to __map__

comments should start with -- to keep in line with lua,

I don't think we need to define a new format for the meta body itself. There are plenty of existing formats to choose from that would be compatible with the minimal spec of "be UTF-8 and don't have body lines that look like block delimiters." It could be TOML, JSON, base64, or even PICO-8's own ASCII-hex format if appropriate.

Emphasizing the encoding bit for the spec: .p8 files are UTF-8 encoded Unicode. Meta blocks must follow suit to avoid confusing UTF-8 readers. Meta block bodies cannot be raw binary data. (Not that anyone has suggested otherwise, just thinking through the spec.)

I'm not entirely sure we need to spell it "meta" specifically, for that matter. Maybe you want to namespace section names to keep everything else reserved for PICO-8's own use. We can just say __data:(.*)__ is a .p8 section heading that is preserved by PICO-8 and ought to be preserved by third-party tools that handle .p8 files, and group(1) is a tool ID up to the tool builder's discretion. I'd bet money that we don't need to reserve tool IDs or anything like that.

Idea for .p8.png: revisit the use of tEXt blocks outside of the steganographic region to preserve metadata across format conversions. Use the delimiter as the "key". (Dunno if sections can repeat.)

@dddaaannn Zep states one metadata block for all usages. He also states that the ____meta:super_palette_editor__ is not a block title.

It looks to me like a block title which is why he said:
> anyone please feel free to jump in to suggest a meta format!

For this reason I am suggesting #something as a sub block title. It looks like a markdown heading it also looks like a comment if you read c style (Actually I'm not sure C is what uses this)

Given the fact that he uses 1 block and asked for comments on meta format is why I put that here.

I agree with Dan that we might be better off using an existing format, since it makes writing tools easier (well-understood, road-tested, lots of existing libs).

JSON is good, because it comes with sub-(sub-(sub-))sections automatically. Mind you I find reading or editing JSON rather cumbersome. Maybe it'd be more appropriate to have a similarly-stripped-down Lua-Table-Object-Notation format. LuTON or something.

At one point I was actually writing a parser for such a thing, but specific to PICO-8's extended grammar, and I was gonna call it P8ON (pronounced like "peyton"). Alas I never finished it, like pretty much everything I work on.

The other option would be something like .INI format, which is a lot simpler to parse and also has existing code out there to use.

@dddaaannn I'm sorry I completely misread (at least part of) your comment. I think the piece I posted about its not __anythingarbitary___ is still correct. But yeah I'm happy with any standard.

I do think key value (like ini) would be more readable.

But I'm happy with sub block begin sub block end however thats represented. I don't think the internal metadata needs to be consistent format between tools so long as they know the beginning and ending of their content.

@Felice JSON I would think would be the only obvious interchange format. Again though I think Its not necessary to define this. which I think @dddaaannn is saying. But like you I find JSON pretty unreadable.

For what I intend to use it for I'm likely to have lots of numbers, Storing palettes, storing arrays of individual sprite locations. Storing the map format (internal or tile) and it feels in keeping with pico8 to keep the size on disk minimal.

Json isn't the best format for number/byte arrays, though its certainly possible and easier to use.

Yup, all good. I've gravitated to TOML in my other projects because it's both INI-like and YAML-like with more robust and featureful syntax. (And it's trendy in the scripting world rn.)

The general point about __xxx__ section dividers is that any given application can do whatever it wants with its meta section as long as it complies with the basic rules of whole-file encoding and simple parsing of dividers. Whether we're subdividing a single meta section or just letting tools populate the top level (perhaps corralled in a namespace) is a minor detail easy to leave at zep's discretion. :)

pico-8 cartridge // http://www.pico-8.com
version 35
__lua__
print("hi")

__gfx__
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

__meta:tiled__
H4sICMNqPmIAA3QucDgA7ZFBCgIxDEX3PUWYlS6cFkRGvEwptU4LtS2djPX4xoILL2BcTJJFwufzH6QEmw9nsKZiDdfZgZTgEctFytbaWLo82nwXD1eXkBMcT0LruBqtRakh4W7wYdh/9uZiBPQGwaSlkaUfpJNpvj3JpJhrA9gACGDqwwigJsb8vwDgfwFrvQGcMpwAFP6V76h/CfACgIozvH4GAAA=

__meta:aseprite__
{"k": ["a", "b", "c"]}

__meta:docs__
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook V5.0//EN"
               "http://www.oasis-open.org/docbook/xml/5.0/docbook.dtd">
<book xmlns="http://docbook.org/ns/docbook" version="5.0">
<chapter xml:id="ch1"><title>Chapter One: Using the PICO-8 Library</title>
<para>When Lexaloffle, Inc. first pre-announced PICO-8 at E3 2012 in Las Vegas,
  it was the talk of the town...</para>
</book>

A point in favor of @dddaaannn's __meta:my_tool_name__ format immediately above is that @zep was concerned about having to save many disparate blocks, but if it's assumed that everything from the first __meta:xyz__ is guaranteed to be more meta chunks up to EOF, then a disinterested tool can simply read from there to EOF and write it back out as a single block.

Actually, I think the only real differences from zep's implementation are that A) it's guaranteed to go at the end and B) there's no __meta__ line before the __meta:blah__ chunks.

I mean, I like the idea of unifying it under something like JSON, but at the same time, most tools have their own text config formats, and as long as their format doesn't have something that looks like __meta:blahblah__ alone on a line, they can simply write & read the sort of thing they're accustomed to writing, resulting in minimal integration friction.

Also worth noting is that, if the first __meta:blorg__ line means that it's now metas all the way down, it's basically a total state change for the .p8 parser. After that, the only forbidden line in a private data format becomes __meta:*__.

BTW, may I just say that I really, really like the idea of PICO-8, whose community is receptive, friendly, and collaborative, having a form of open, collaborative tool data interchange built into its workspace format. :)

Hrm -- yeah, ok I'm sold on __meta:*__ section headings. That's a compelling example @dddaaannn, and I hope that the Las Vegas pre-announcement accidentally ends up as accepted history, haha.

Thanks for the proposed subsection format @SquidLight -- I was imagining something similar, but it seems it won't make parsing / multi-tool cooperation much easier by standardizing it. I was partly tempted by the prospect of offloading the contents of __meta__ as a userland problem and being done with it! But this way, a section contents format like this can be adopted or not as each tool author wishes.

So, current working solution for 0.2.4c is now to accept and preserve sections with a heading: __meta:*__.

Let's say also that __meta:*__ sections MUST appear after all other (reserved) sections, and that including __meta:*__ in the code section has undefined behaviour.

So now the laziest (and thus best) way for a tool to modify its section:

1. search for the line: __meta:mytoolname__
2. search for the next line matching __meta:*__ (or EOF) to find the end of that section. No need to check for reserved section headings.
3. rewrite everything in the file as-is except for that [modified] section

.. and the only requirement for section data is that it does not include \n__meta:*__

@Felice Agreed! Even if it is not heavily used, I think the feature merely being available adds a certain welcoming element to making p8 tools. I also like it that a little standards committee can spring into existence like this :)

This looks great, should be a good replacement for the

--@conf
--[[ ... ]]

configuration section we're using for our map editor :)

Using a standardized format like JSON wouldn't satisfy everyone - for example, we just use Lua since our editor is also in Lua which makes it super easy to parse.