picotron_pod.txt
author: zep
updated: 2023-05-07 (WIP!)
A POD ("Picotron Object Data") is a string that encodes Lua values: tables, userdata, strings, numbers booleans, and nested tables containing those types.
PODs form the basis of all data transfer and storage in Picotron. Every file is a single POD on disk, the contents of the clipboard is a POD, images embedded in documents are PODs, and messages sent between processes are PODs.
PODs can be created with pod(val, flags). Flags indicates which encoding(s) to use:
0x1 pxu: encode userdata in a compressed (RLE-style) form
0x2 lz4: binary compression pass (dictionary matching)
0x4 base64 text encoding
Any output from pod() can be converted back to the original value using unpod(pod_str)
The simplest form a POD can take is a Lua snippet that constructs the value being encoded:
?pod({a=1,b=2})
{a=1,b=2}
This form is useful when manually constructing POD strings, or for debugging.
Plaintext PODs can get quite large if they contain images or map data. A compressed binary encoding can be generated using flags 0x1 and 0x2, which are normally used together as the pxu format aims to produce output that can be further compressed by lz4.
The resulting string contains non-printable characters and starts with the header "lz4\0", so only the first 3 characters are printed here:
?pod({a=1,b=2}, 0x3)
lz4
store() uses this format by default, which gives compression ratios similar to png / px9, but is faster to encode and decode.
The output from the binary compression pass can be returned to a text-friendly format with 0x4.
This form is still relatively compact but can be safely shared via channels like email, forums, and pasted into regular text documents. It is also used for embedding images in picotron documents and source code. Like the other plaintext format, it is also valid lua code that returns the value being encoded.
> ?pod("the quick brown fox", 0x7)
unpod("b64:bHo0ABcAAAAVAAAA8AYidGhlIHF1aWNrIGJyb3duIGZveCI=")
> ?unpod("b64:bHo0ABcAAAAVAAAA8AYidGhlIHF1aWNrIGJyb3duIGZveCI=")
the quick brown fox
// For small chunks of data like this one, the base64 encoding is larger than the text it encodes because of the overhead of bookkeeping information stored at each pass.
POD Metadata is arbitrary information about a POD such as its author, the time it was created, or hints about what to do with the pod when it is pasted or embedded somewhere.
POD metadata (if it exists) is stored at the start of a pod string in the form --[[pod...]]
This format doubles as an identifying header, so that POD files and POD strings inserted into text documents can be identified as such. It also allows the metadata part of a pod string to be ignored by the lua interpreter, and used as a regular value in .lua source code.
The metadata part of the POD string can be generated by pod() by giving a table of metadata as the third argument.
> str = ?pod({x=1,y=2}, 0x0, {description="an uninteresting vector"})
> ?str
--[[pod,description=\"an uninteresting vector\"]]{x=1,y=2}
unpod() returns any decoded metadata as a second result:
> c,m = unpod(str) -- returns content and metadata
Technical notes if generating metadata strings by hand: Metadata is always a table value at the top level, but the enclosing {} are not written in the pod string. When pod_type is a member of the metadata table, it is written first (unlike the usual alphabetical ordering). Otherwise a dummy value "pod" is written first. This allows the metadata to double as an indentifying header for POD data, as in both cases the pod string starts with the string: "--[[pod".
A POD can be written to disk using store(filename, val), and retrieved using fetch(filename):
> ?store("dat.pod",{1,2,"foo"})
> tbl = fetch("dat.pod")
> foreach(tbl,print)
1
2
foo
All files in Picotron are considered to be PODs, and are stored and fetched atomically. There is no stream-like way of interacting with files, for example to read half-way into a file. A typical way to save data is to construct a temporary table containing all data to be saved, and then to store() that table.
Like unpod(), fetch() returns any metadata as the second return value. Metadata can also be read from disk separately using fetch_metadata().
Individual metadata values can be written either using the third parameter of store(), or store_metadata(). In both cases, a table of values to write is given, and any existing metadata is preserved.
> store_metadata("dat.pod", {author="zep", foo=3})
> store_metadata("dat.pod", {foo=4}) -- author is not clobbered
> meta = fetch_metadata("dat.pod")
>
> for k,v in pairs(meta) do print(k..": "..v) end
modified: 2023-05-07 04:23:46
author: zep
foo: 4
fetch_metadata() and store_metadata() can also be used to handle metadata about folders, including .p64 folders. Internally, the metadata is stored in a hidden file called ".info.pod".
> mkdir foo
> store_metadata("foo", {sort_by="filename"})
> ?fetch_metadata("foo").sort_by
filename
When the value given to store() is a single string, pod_format="raw" is recorded in the metadata, and the contents of the string are dumped to the file. This allows it to be edited in a standard text editor outside of Picotron if desired. Conversely, when a text or binary file is loaded that has a pod_format="raw", or no header at all, the contents of the file are read as a single (possibly binary) string.
The output of pod() is deterministic; keys are sorted to ensure that they are written in a predictable order. Two POD strings are identical when and only when the two values they encode are identical.
This makes POD strings a useful target for create_diff() and apply_diff(), to implement undo/redo stacks:
a = {1,2,3}
b = {1, "banana", 2, 3}
d = create_diff(pod(a), pod(b))
-- reconstruct b using only a and the delta (d)
s = apply_diff(pod(a), d)
foreach(unpod(s), print)
1
banana
2
3
