It was suggested I post this here, so I'm just going to copy/paste. This is a write-up on how I jumped some hurdles to make my rpg WaGa (having issues uploading the cart to here on my phone, I'll put it on soon)

PICO-8 and limitations

So, indie games are usually made to emulate older games, and follow similar restrictions. That's cool, but older games, at the time, were all trying to break past those restrictions, this gets a bit lost. PICO-8 is a platform given to us that attempts to force us to follow some restrictions, so now we get to make games exactly like older games: pushing those limits.

So recently I made an RPG for PICO-8, WaGa. I hit the token wall pretty quick. I had a lot of data, 50 some items, 8 enemies, enemy groups, stair exits, 4 characters with stats, and a bag that could hold items, and so on. PICO-8 follows a subset of lua, so I had to get fairly creative.

The biggest thing I found I could exploit, by far, is strings. One string is just a variable, and follows the same one token count as any other variable, this really opens things up. The only string function PICO-8 gives you is sub(), which is, as I found, all I needed to get done what I had to get done.

I did need a few other functions though. I needed explode, and I needed a string->number function. I think to myself, if I have these, I can have my entire set of items only take up one token! Had I really wanted to, I could have all of this a step farther, and had one huge database string itself, which i explode into datasets, which i explode into arrays.

Heres an example, my item dataset
--item[n] = "id|itemtype|strength|uses|weapon|special"
ids = "1|0|0|0|0|0|_2|2|10|50|1|0|_3|2|15|40|1|0|_4|2|25|25|1|0|_5|3|10|50|1|0|_6|3|15|40|1|0|_7|3|25|25|1|0|_8|4|10|50|1|0|_9|4|15|40|1|0|_10|4|25|25|1|0|_11|5|10|50|1|0|_12|5|15|40|1|0|_13|5|25|25|1|0|_14|6|6|50|1|0|_15|6|15|40|1|0|_16|6|25|25|1|0|_17|7|10|50|1|0|_18|7|15|40|1|0|_19|7|25|25|1|0|_20|10|25|0|0|0|_21|10|50|0|0|0|_22|10|99|0|0|0|_23|11|25|0|0|0|_24|11|50|0|0|0|_25|11|99|0|0|0|_26|12|25|0|0|0|_27|12|50|0|0|0|_28|12|99|0|0|0|_29|13|25|0|0|0|_30|13|50|0|0|0|_31|13|99|0|0|0|_32|1|50|5|1|0|_33|1|250|4|1|0|_34|1|999|3|1|0|_35|8|0|1|0|1|_36|8|0|1|0|2|_37|8|0|1|0|3|_38|8|0|1|0|4|_39|8|0|1|0|5|_40|9|0|0|0|0|_41|9|2|20|1|0|_42|9|5|20|1|0|_43|9|10|20|1|0|_44|9|25|5|1|0|_45|9|100|1|1|0|_46|9|10|5|1|1|_47|9|0|5|1|2|_48|9|0|5|1|3|_49|9|0|1|0|4|50|8|0|0|0|6|"

I first explode the dataset by _, into an array, go through there and explode each one of those arrays into my item variable. I also did this for monsters, npcs, coordinates, text blocks, pretty much everything. I'd estimate it saved me somewhere from 500-1000 tokens altogether, but I'm really not sure since I went adding more and more after implementing this.

I didn't have internet at the time when I was working on this, so it was pretty fun trying to figure out how to make some of these functions. Lets get to them then

function substring_to_number(st)
digits = {"0", "1", "2", "3", "4", "5", "6", "7", "8", "9"}
for i=0,9 do
if st == digits[i+1] then
return i
end
end
return 0
end

this is a simple one, if theres only 1 digit, its really easy to go from string to number!

function string_to_number(s)
if #s > 1 then
rval = substring_to_number(sub(s, #s, #s))
for i=1, (#s - 1) do
rval += substring_to_number(sub(s, i, i)) * 10 ^ (#s-(i))
end
return rval
else
return substring_to_number(sub(s, 1, 1))
end
end

This is how I tackled multiple digits. I'm not sure if its the best way, but my game works perfectly fine, so it seems to work!

At this point, I can convert strings into numbers, all I need to be able to do is explode strings, which turned out to be much easier to write.

function explode_internal(s, delimiter)
retval = {}
lastpos = 1
for i=1,#s do
if sub(s,i,i) == delimiter then
add(retval, sub(s, lastpos, i-1))
i += 1
lastpos = i
end
end
return retval
end

and thats pretty much all you need, you can store near limitless amounts of data in a single variable with these few functions. I wrote some more to manage loading datasets.

function load_dataset(dataset)
local retval = {}
local a = explodeinternal(dataset, "")
for i=1,#a do
local b = explode_internal(a[i], "|")
retval[i] = {}
for k,v in pairs(b) do
retval[i][k] = v
end
end
return retval
end

so all I need to do now to load that mindless jumble of numbers and symbols is
item = load_dataset(ids)
and so-on for other datasets. This is all really great stuff, and without writing it, WaGa would've been a game without NPCs to talk to, and I would've had to possibly remove features just to be able to put in the calls to music(), or not have music at all! It wouldn't have abilities that did things other than damage, a title screen, and I was even considering having a fixed party.

You can also use this to store map data, and write a function to write that map data via mset(), you can store artwork, and draw to the screen with pset(), draw to the spritesheet with sset() and fset(), song and sfx data is stored at 0x3100 and 0x3200, so I'd assume you could poke() some new music in there as well! The issue you're going to run into is the character limit, and the compressed code limit, and at that point, all you can do is crunch. I'd imagine if you had enough data, it would be worth it to even write compression/decompression routines. If you're crazy, you could store things in multiple carts, and there'd be no reason you couldn't have full video.

Hopefully this information helps someone, since I don't have internet at home, I don't know if similar things have been published or suggested, so if this is already out there, have fun with this:

poke(0x5f2c, 5)

Nice! This is a great write-up with a clever way of extending the limitations of pico-8.

Just like classic RPGs; Walk 2 steps. Enter a battle. Win. Rinse and repeat. lol :-D
Very well done game and write-up.

Nicely done! If you're interested in refinements:

You can coerce a string value that represents a number into a number value by adding 0:

x = '12' + 0
rectfill(x, x, x+10, x+10, 7)

This raises an exception if the string is not parseable as a number. This works with integers and fractional decimals, within the limits of Pico-8's fixed point number type. (If the string value overflows the number type, the result is -32768, not an error.)

If you're packing a large collection of byte data, base64 encoding is a good choice. Overkill posted some routines a few months ago:

https://www.lexaloffle.com/bbs/?tid=3738

Keep in mind that while packed strings (of any format) save on tokens, they consume characters. More importantly, they tend to consume the "compressed" limit more quickly than regular code. It's easy to end up with a program that can be saved as a .p8 file but not as a .p8.png (and therefore not publishable to the BBS) because you're under the char limit but over the compressed limit. It's still a useful technique, just be mindful of the tradeoffs.

Well, that certainly would have made things easier! That's a lot of great info, I won't be developing WaGa further, but if I start another game it'll certainly be helpful

Also to reply, string datasets definitely make compressed code the issue rather than tokens or characters.

Oh hey, I just recently implemented something very similar for my circuits puzzle game. Here's the string definition for all the entities in one screen:

parse_room[[1,1
|energydoor/20/84/facing=east/cfacing=south/coffs=v{1,3/doorway={1,0,3,0
button/10/116/flipx=true/reset=6
energydoor/92/52/facing=south/cfacing=west/doorway={0,1,0,2/coffs=v{-4,1
button/10/76/flipx=true
button/18/28/cshow=false/flipx=true/reset=2/color=6/norobot=true
robot_spawner/16/16/cfacing=west/robot_id=0/coffs=v{-2,0
button/118/16/cfacing=south/cshow=false/reset=5
energydoor/60/44/doorway={0,-4,0,-1/facing=north/cfacing=east/coffs=v{3,-2
key/113/73/id=1
keydoor/20/44/id=1/doorway={1,0,3,0
relay/7/108/facing=east
relay/7/68/facing=east
arrow/20/113/facing=west/text=1
arrow/20/66/facing=west/text=2
arrow/110/30/facing=south/text=3
|{11,1,2,1
{11,2,1,1
{12,1,4,1
{6,1,5,1
{7,1,8,1
]]

Pipes "|" separate sections, newlines "\n" separate entities, and slashes "/" separate key-value attributes in each entity. The second section of just arrays of numbers defines the pre-wired connections between entities.

The parsing code is about 275 tokens, and supports building strings, numbers, key-value tables, array tables, and my custom "actor" and "vector" types. The current format is very specific to my game but I've been pondering whether I could make a more generic table-parser in a similar number of tokens -- probably not JSON exactly, but something like JSON but more optimized for bytes used.

I haven't really looked into doing compression on the strings yet at all. I'm not sure how much of a win it would be with pico-8's built-in compression.

I've been running into size limits before I run into token limits, though if I minified then I'd definitely run into token limits first.

This could probably be taken a step further by encoding the data into the sfx, map, and sprite slots. sfx, map, sprites, and map/sprite shared space all together is around 16KB which is slightly more than the amount of space you're allowed for code, so you'd be more than doubling your allowed space if all your sprites, sound, and sprite placement was procedural.

Would be cool if someone made a general utility that could extract an array of integers/strings/etc out of those databanks or something like that. Maybe I'll try playing around with it if I have time.