So this is probably going to be a long, weird post, and is inspired by this post by @Mot a few weeks ago. I kind of dislike the way that Pico-8's cartridge RAM and Lua RAM exist in different worlds, and how everyone eventually uses huge strings to store data on the Lua side, when they run out of token space. It's fine to do that, and in fact I love that the option exists, but is it really a good default way for people to bypass the Pico-8 limitations? I have an idea I want to float to the community about this.

My proposal is to add two functions to the API, that convert between a Lua table and a cartridge bitstream. Using them to get your token count down would be a pre-publishing task, but it could be automated so that you only have to comment out the table to publish your game:

table = { true, x=15.5, [0]="yes", {"why?"}={"because","lua","supports","this","too"} }
len = luastore(0x0,table[,max_length]) --stores at 0x0 in cartridge RAM (not ROM), returns length of stored data
--cstore(0x0,0x0,len) --ROM is much slower and limits what these two functions can do, so keep it optional
table = luaextract(0x0[,len]) --use length if provided, otherwise it's at the start of the bitstream

In addition to giving people a standard way to store things without wasting a large number of tokens decoding the bitstream, this also does a lot to connect the Lua and cartridge worlds, and makes things like saving games trivial:

luastore(0x5e00,save_data_table,256)
save_data_table = luaextract(0x5e00) --length is explicitly encoded at the start of the bitstream

But in addition to those advantages, it also gives people an opportunity to store data in a compressed bitstream that is standardized across cartridges, and gives a real use case for the 0x4300-0x5dff user data section of memory as temporary Lua storage! Writing your own encoders and decoders can be fun, but a fast, internal system that doesn't use your limited token space would open up this kind of development for a lot more newcomers. I think it caters to the idea of a cozy development environment, since it makes hitting that token limit barrier easily fixable for many creators, and less demoralizing.

Demo implementation

The rest of this post is a sample implementation of how Lua data could be serialized quickly, but with a decent amount of compression so that it works for the intended purpose. I did this more for fun than anything, and I'm sure there are people here that could do a better job of this, but hopefully this explains what I was thinking:

compressed bitstream storage format, for types: NUMBER, STRING, BOOLEAN, TABLE
(do not store types: NIL, FUNCTION, CFUNCTION, USERDATA, THREAD)

All bitstreams start with an implicit 16-bit header, representing the total number of bytes stored. This is more a safety measure than anything else, and can be bypassed with a clever destination offset of 2.

All other LENGTHs are stored as packed ints: [2 bits: coded length in nibbles][4-16 bits: value]

STRINGs are stored as: [LENGTH][x bytes: one per character]

DATA is stored as: [4 bits: number flags][2 bits: type code, if non-number][type-specific data...]
Number flags prioritize compressing numerical data first, since it is assumed most common.

NUMBER DATA is stored as: [non-zero in 0xff00][non-zero in 0xff][non-zero in 0x.ff][non-zero in 0x.00ff][1-4 bytes: value]
0000 in number flags = non-number type follows

Two-bit type codes define less common types (6 total bits to declare these data types):
00 = FALSE DATA (end of entry)
01 = TRUE DATA (end of entry)
10 = STRING DATA (STRING storage follows)
11 = TABLE DATA (TABLE storage follows)

TABLEs are stored as: [2 bits: key flags][string keys+data][number keys+data][non-number/string keys+data]
Key flags optimize what follows in the table:
00 = empty table (end of entry)
10 = string keys only
01 = number keys only
11 = all keys, including BOOLEAN and TABLE keys (for full Lua support, more than for usefulness)

If STRING key flag is active, that section is stored as: [LENGTH of string keys][STRING][DATA][STRING][DATA]...

If NUMBER key flag is active, that section is stored as: [LENGTH of number keys][NUMBER][DATA][NUMBER][DATA]...
Since all keys in this section are numbers, [0000] in number flags is shorthand meaning "previous key + 1". If [0000] is the number flags of the first entry, it is assumed to be the first default key in a table, 1.

If both key flags are active, the final section is stored as: [[2 bits: type code][type DATA...][DATA]]...10
Since all keys in this section are not numbers, 4-bit number flags of DATA type are always omitted for keys.
Since type code 10 (STRING) is impossible in this section, those two bits signify the end of the table structure.
Therefore, in the false-positive case of only NUMBER and STRING keys, this section is only two bits: 10

demonstration encoding:
luastore(0x5e00,{1,[0]=0,str=5.25,[true]=false,{"what"}="why?"}) becomes this bitstream:
[0000|11|11]: [00|0001]:[00|0011][24 bits:"str"]=[0110|00000101|01000000];
              [00|0010]:[0000]=[0100|00000001],[0100|00000000]=[0100|00000000];
              [01]=[0000|00]
              [11|01]:{[00|0001][0000]=[000010][00|0100][32 bits:"what"]}=
                      [000010][00|0100][32 bits:"why?"]
              [10] - end of typed key table, and end of table overall
[table w/all]:[1 str key]:[3 chars:"str"]=[num type:5.5];
              [2 num keys]:[default key]=[num:1],[num:0]=[num:0];
              [bool:true]=[bool type:false],
              [table w/num]:{[1 num key]:[default key]=[str type|4 chars:"what"]}=
                      [str type|4 chars:"why?"]
              ; = 27.75 bytes, store as 30 total bytes in 2-byte integer at bitstream start

I'm interested to hear if people like this idea, and if @zep thinks it would add something to the PICO-8 API. I know it just hit Beta and this is unlikely to be implemented, but I do think it would add a lot of value for many people in the community, myself included.