Yep saving I get, but I don't understand how I load my map. Surely there must be a way to load a map you have already saved?
I guess I'm not quite understanding. Are you asking how to load a map into the PicoMap editor, or how to load it into Pico-8's built-in map editor?
Sorry. Yes I would like to be able to load a map into the PicoMap editor to make more changes and things.
Hmm, I thought that part was pretty straightforward, but maybe I skipped explaining something.
So, when you've made a map, you need to exit by pressing esc, then go to the data strings tab and press ctrl+v to paste the contents of the clipboard there. You need to be sure and paste over the existing strings, otherwise it could read the wrong ones, and be sure to save the cart after doing this. You can actually save, load, and edit many levels, since the level strings are stored in a table, and this whole table is output to the clipboard.
I'm not seeing it. This is what I have, but you'll see that the strings aren't similar and if I just paste them at the bottom nothing seems to be happening.
Okay, I know what's happening.
Looks like you're using the export feature to output the level maps, but that's only supposed to be used later, to send the levels to the cart where they'll actually be used. The strings containing level data are constantly being output by the program as you edit your levels, so all you need to do is press escape, and paste the contents of the clipboard over the existing strings.
I did things this way because it lets you freely edit the spritesheet and semi-compressed level data in the editor before further compressing them in the final cartridge so much more can fit. I tried to explain everything, along with illustrations, in the system description and tutorial, as it's a bit complex and data has to be stored in an unconventional way to fit so much in a cart.
Ohhh I thought the copied text and the export was the same thing. I'm a bit dense. Thanks!
I understand the confusion, I had to break things up to make the fullest use of space. In the final cart, the spritesheet is used for raw binary level data, since that gives a full 12KB of contiguous possible level space, and the data's pretty much incompressible. Your sprites will be compressed into a string, since that does compress okay.
In the next update, I'm planning to incorporate a version of zep's PX9 compression system as an option, as that should allow storing a whole 2nd spritesheet in map memory if you don't need all 12KB for levels, which most projects probably shouldn't. Haven't quite settled on a standard solution for spritesheet switching, though.
The Most Impressive Tool Made to Date
This is now the number one most impressive and useful tool ever created for Pico-8, while keeping the aesthetics of Pico-8.
I am more impressed by this work than I am by PicoCAD or Memsplore, which are both outstanding in their own right, but as far as I'm concerned, you're in a league of your own. Mainly because this tool will have SIGNIFICANT impacts on what users can or are willing to make.
Having been writing a book on PICO-8 for the past 2 years now, thinking I had seen most everything, this has come along and made me reconsider just how complete the book is....
Thank you so much for your contribution to the community and also for the documentation. If all software did it like you have, there's no telling how many more people would be drawn into their untapped abilities. Seriously, as the runner of the unofficial api docs , I feel like there will be some massive changes made to the presentation of it all, having seen what you did here.
Thank you for your kind words,
I think the biggest thing this program does is show that Pico-8 is more capable than people would expect, basically by using a new spin on old compression techniques that people don't bother with anymore due to plentiful storage and memory.
To be fair, I'm not the first or only person to work at expanding the system's capabilities by going beyond the standard map system, for example:
-
@paranoidcactus has used a custom metatile-like map system for multiple action-platformer games
-
@2Tie appeared to be using a custom metatile system in PICOROID, their WIP Metroid demake
-
@Saffith developed a metatile-type system heavily based on autotiling and used it in their game Barp the Balldragon, and even released an editor for the game that gave me some design ideas (though I barely understood the code when I tried reading a few bits of it).
- @pancelor made the Bigmap Editor, which takes advantage of Pico-8's new HiMem features and, when combined with zep's PX9 compression, allows a few times more map space than previously available.
My goal with PicoMap was to not only prove that Pico-8 was more capable than people thought, but to make exploiting that capability relatively easy, so people could make, say, an expansive platformer without doubling their effort by having to start from scratch in making an expanded map system. I think for a moderate expansion in map size the bigmap editor will probably be more intuitive to use, but if somebody wants to make a game with a big 'ol world to run around in, It should make things a lot easier for them than they otherwise would have been. I've also learned a lot in making it, so that's cool too.
PICOROID just replicated the room and metatile system from famicom metroid, with some minor improvements. I had also written about metatiles (using Pokemon Green as an example) in one of the pico zines a year or two back; actually, i wonder if that one ever got published?
But yeah, pico-8's been very capable of this sort of thing for a while, but this tool definitely makes more things accessible. I'm suprised people are still noticing picoroid all this time later! (i did manage to fit the whole game map into memory back then, but ran out of tokens to implement enemy logic)
Yeah, some time after getting the idea for what would become PicoMap I looked through the forum to see if similar things had been done, and came across Picoroid. That's pretty cool that you managed to implement a similar tile setup to the NES, even using monochrome tiles with definable color attributes. Just curious, do you have any idea around how many tokens your metatile system was using?
Also, I would be very interested in checking out your article on metatiles, if you still have that around somewhere. ;)
Hey @JadeLombax, I'm trying to export a small map just to take the editor for a test drive and figure things out, but whenever I try to export Picomap crashes with this...
I made sure to change the export cart in the user settings, and placed a matching file in the Pico 8 folder. Am I doing something wrong?
I don't know that I've come across that exact glitch before, so it's hard to say. If you could post the cart you're trying to export from to the thread, though, I could try and pinpoint the issue.
Hello, I can reproduce in 0.2.5e this error by just loading up #picotron and then exporting.
Hmm, okay. I'll check this out after work today, see if I can figure out what's up.
A small update : I took some time to binary search the last version that worked on 0.2.5e and turns out the first version of Picomap that boots is the ninth (so the update that moved the do..end out of a single-line if) and from this one to the most recent version, all versions fail on export.
EDIT : I'm trying to pry the cart open to see what happened too. It looks like it breaks on trying to extract a number from an empty string (looks like it's caused by not having objects in the cart, according to tab 2?). Still digging.
EDIT 3 : Looks like
local start_screen if #c > 0 then start_screen = val(c,1,3)\16 else start_screen = 0 end |
instead of just local start_screen = val(c,1,3)\16 (tab 2 line 42) allows the export to visibly do something. I don't know if it works though.
I was wondering if it was something like that, but I didn't think I used that kind of shortcut in here. Will have to check this out some more, sorry, been busy w/other stuff.
It's alright, between the moment I reported the bug and the moment I tried to debug it I was also busy with some other stuff, life happens, right!
Yikes, Pico-8 0.2.5.d (or e) seems to have done a number on this. After some testing, it seems the string error you guys are getting might be due to not saving the level strings before exporting, as attempting to export a header-only level string with no objects in it gave me the same error. The problem is that when I try exporting with objects in the string, the screen just goes black and the editor doesn't come up again. Also, CPU usage is way up for some reason, like 75% with just a few objects in a map, which is crazy.
I'd like to understand why all this is happening, but at this point I think it might just be easier to upload a version of the build I've been using for my Mega Man port. It's a prototype of v0.50 with some WIP features and a little bug or two, but it works and exports okay. I think I'll comment out a few things, do a bit more debugging, and upload that.
Version 0.42 is up. Debugging took longer than I planned, but it seems to be working well with no significant bugs, and since I based it on a prototype of 0.50, there are some architectural improvements. Details in the changelog.
Edit: Reuploaded to fix a small glitch with the mask object tool.
So I tried to use PicoMap on a real project (pico sonic which currently uses 4x2=8 data carts dedicated to tilemaps), but I'm not sure about the end-usage process:
- Exporting will create a p8 cart named following
export_cart, which contains encoded binary data in gfx: this looks very efficient and my huge map only took one line of pixels of gfx. However, the rest of the instructions tell us to use the sprites (and other) unicode strings in the clipboard, which is even more convenient to use in code, however it blows up my characters count on this project (where I have limited character budget left - we're talking 43553 to 73125 chars, so above 65535 and it won't even calculate compressed chars). Are the two methods, loading from unicode string, and loading from cart, two alternative valid methods? And in this case, where is the code that can load the full tilemap from exported cart GFX? - I thought the advantage of using metatiles was that you would only fill data where there is something (so if map contains a lot of holes, you spare data) but it seems that PicoMap is using a flat, incompressible array of data anyway, so even though I just added a few metatiles for testing, it's already taking a lot of space (at least in unicode string). The advantage is that I know the max storage I need in the end, after I complete the map - the disadvantage is that I'll be wasting space even if I have a few platforms and big autotiling-filled rectangular parts in my level
I need to make sure that I can get around these technical limitations before I switch to PicoMap - one advantage is that the PicoMap runtime code is smaller than my own multi-cart streaming region code. Another advantage is that loading the whole map in memory means no multi-cart reload, which adds a lag every time you change region unless you patch the PICO-8 binary for "fast reload" (both multi-cart and the patch preventing players to play on their native PICO-8). But I'd need to make sure I can load the whole data from compressed GFX info (if it's super compressed I could even place it at the end of an existing spritesheet - although not sure I'll be able to go under 3 carts in the end anyway).
Otherwise, I may just get inspiration from PicoMap, namely on the part that preloads the whole map in runtime memory (or local RAM in compressed form), just to avoid cart memory reloading while streaming the whole level.
Since you're having issues with the data strings taking too much space/too many characters, sounds like you either have a whole lot of data in the map memory "scratchpad" area used to build metatiles, a very dense spritesheet, or both, as those are the strings that take up the bulk of space.
If your use case makes storing this data in strings unfeasible, the current build of PicoMap I'm using for my port of Super Mario Bros. stores it to cart memory using zep's PX9 compression instead, freeing up a good deal of code space, the tradeoff being a somewhat larger decompressor that takes an extra 150 or so tokens.
If you could post a copy of the editor cart you're using, I could see how it might be streamlined, and maybe upload an updated version of PicoMap using PX9 if it would help. Was planning to wait to incorporate some other features used in my Mega Man port, but I won't be finishing that real soon.
@JadeLombax I see, I didn't use much of the "scratchpad" though, just defined a few meta tiles, but I have a full spritesheet coming from a real project.
I see, PX9 could definitely help. I was also thinking that if BBS doesn't seem to allow multi-cart, at least multi-cart are still playable with native PICO-8 by downloading all the carts in the same folder, making a native build for realistic (I still need to work on my tokens usage).
Hm, I'm not sure how to share my cartridge without publishing an ugly cart with ID along the way (I don't want to make it an "official" upload on lexaloffle). But I can share a link to the public GitHub repo which contains the carts:
picomap_stage1.p8 (the editor cart)
https://github.com/hsandt/sonic-pico8/blob/6bf37c831569134d3fc14e3869c533d205f03eca/data/picomap_stage1.p8
picomap_stage1_assets.p8 (the cart containing tile assets to import)
https://github.com/hsandt/sonic-pico8/blob/6bf37c831569134d3fc14e3869c533d205f03eca/data/picomap_stage1_assets.p8
picomap_data.lua (the generated unicode strings)
https://github.com/hsandt/sonic-pico8/blob/6bf37c831569134d3fc14e3869c533d205f03eca/src/data/picomap_data.lua
Note: to be fair, I already use a similar unicode string cart for PCM audio (https://github.com/hsandt/sonic-pico8/blob/6bf37c831569134d3fc14e3869c533d205f03eca/src/data/pcm_data.lua) which is just as big - it just happened to work because my title screen plays on a separate cart which only contains a menu + cinematic which is rather cheap in characters, compared to ingame. Also, I expect an audio signal to be quite complex, compared to a tilemap - but I understand that it's not so compact if not encoded as a sparse matrix.
one mode of multicart is compatible with bbs: not the one with data cards and reload used to copy bytes, but the one with a launcher cart that prepares data in high memory then uses load to run the main game cat which can access the data in high mem
@huulong,
After taking an initial look at your cart, it seems things are working correctly, which is a relief. Seems like you're very tight on characters, already, though, so the current setup isn't working that well for you.
The strings are taking up about 3.75KB of compressed data in total, with 3KB of that being compressed sprite data. The other smaller strings contain the "scratchpad" data, one that defines the properties of each metatile type, and one that tells the system where level data is located in the master table created at runtime. I lumped everything together this way to free up space in memory to hold the raw metatatile data, but in your case it looks like storing compressed sprites and map data via PX9 would work a lot better, that way you just have the small definition and data location strings, which will take only a fraction of the chars, and everything else can go into cart memory.
I can work on an update that incorporates PX9, I'm already using one, I'll just want to leave out any unstable features and do a few tests. I did notice the tile patterns in map memory look a lot like a full level, which could work if you're going to reuse chunks a lot, but so far I've found that packing it densely with specifically useful small pieces works well to minimize storage space. I updated the initial post with the editor cart I used to build the SMW demo, which also includes some test autotile tilesets to play around with, hopefully it illustrates what I mean.
Version 0.43 is up. Map area and sprite data are now stored in cart memory via PX9. This slightly increases token use, but can free up the compressed code size and character limit greatly. Also, an autotile list table will only be exported if you actually use them in your levels, and a flag data string will export automatically if data goes over 12KB. Details in the change log.
Was curious how big your total map was, so I downloaded the files from itch. Woah, never seen a map spread across several carts like that. o_O Anyway, I repurposed some code and dumped the contents of the carts to a massive string so it can be rendered from a single cart. It only takes up about 6KB with Pico-8's compression, but over 50,000 chars. Use the arrow keys to scroll.
After a little testing, I'm sure PicoMap could handle it alright, and could probably let you add a second (or maybe even a third) map using the same tileset. Here's a tiny little demo of what I mean. Just a couple screens with some simple metatile chunks, but it only takes 34 bytes for level data.
Pancelor's BigMap editor could also be a possibly simpler solution, but as that seems to have a max width of 256 tiles and compresses levels directly using PX9, you'd need to break the map up into two or more chunks and render them separately, or decompress to a table like PicoMap uses.
@merwok I see, so I can upload multi-cart but I need to make sure I have enough memory to prepare all the data in advance in the launch cart. Probably won't work on full pico sonic with menu and credits but maybe just on ingame using data strings or data as GFX (since the launcher cart will only have that, so no pb with chars limit / GFX limit). Probably data as GFX would be even simpler as I'd literally copy it to general memory (I already use almost 100% of it but part of it is for my custom level streaming system, which would be replaced by the new unified level loading system).
@JadeLombax Wow, thanks, I didn't think you'd go so far!
> Map area and sprite data are now stored in cart memory via PX9
Excellent, I'll try it. Are there some people here who have a lot of character space left and may still want to use the unicode strings instead, though? It seems that the unicode string copy to clipboard is still enabled, however since function px9s replaced decompress they'd have to use v0.42 to make it work. So if some people request it, maybe we should keep a copy of v0.42 around or commented code for the old compress method? I just want to avoid bothering other "less extreme users". But if everybody likes the new system, we can go with it.
> It only takes up about 6KB with Pico-8's compression, but over 50,000 chars. Use the arrow keys to scroll.
The demo full level is great! How did you make the native tilemaps work with your code though? When I asked first about porting existing tilemaps, you told me that because of metatiles I needed to redo everything from scratch. Is reading native tilemap data and converting it to a big string part of "repurposed some code"? In this case it's great, that would reduce my amount of work tremendously, in counterpart of losing the compactness and reusability of metatiles (but may matter less as I won't work as much on level design now).
> After a little testing, I'm sure PicoMap could handle it alright, and could probably let you add a second (or maybe even a third) map using the same tileset.
OK! According to my calculations I could fit everything in one PicoMap map though:
pico sonic originally splits levels with tilemaps without shared memory = 128x32 tiles A screen in 16x16 tiles So a level is 128/16 x 32/16 = 8x2 screens I have 4x2 levels (counting the goal area), which makes 4*8 x 2*2 = 32x4 screens So my new map parameters are (1 (type is WIP), 32, 4) |
and I verified that those parameters fit in PicoMap's limit. Did I count it wrong, or were you simply suggesting to split data in more carts for convenience, e.g. not having to scroll all the way to the third region?
So, to sum up and make I understood correctly:
- the runtime demo you posted with full map works well, but it uses the unicode string for 50k characters, so not usable directly in game
- the editor demo you posted just below demonstrates how I could recreate the full tilemap using metatiles. Since it uses the new version of PicoMap, it could export to PX9 and avoid the 50k chars usage. However I'd need more effort to define the metatiles and then place them (what I intended to do originally)
- there is a way to have the best of both worlds: avoid reconverting all tiles to meta-tiles manually and use PX9 export, however I'd need some glue code to convert my native tilemaps, not to a string this time, but to a format usable by the new PicoMap editor. And since currently, when it imports a PICO-8 map as import_cart, it only expects a set of tiles to define meta-tiles, not actual map data, it wouldn't work with the current version.
The 3rd point myself seems solvable by writing a converter that converts every single tile to a metatile made of 1 tile, but I'm not sure if that's the most efficient way to do it. I'll wait first to understand how your runtime demo works, because if it has been adapted to work with simple tiles and I want to work with that, then I may as well bypass the PicoMap editor entirely and instead focus on converting that giant unicode string to cart data using PX9.
Actually, how did you build the runtime demo map? With a custom cart that reloads each level one by one and packs everything into a string? I may just do the same except I'd apply PX9. So I'm not dependent on the unicode string from the runtime demo so if I change the level later, I can regenerate new PX9 data as many times as I need to.
> Pancelor's BigMap editor could also be a possibly simpler solution, but as that seems to have a max width of 256 tiles and compresses levels directly using PX9, you'd need to break the map up into two or more chunks and render them separately, or decompress to a table like PicoMap uses.
Yes, since I now see that just raw (non-meta) tiles can fit with efficient data compression/encoding, it seems easier for me to just work with a simple (but big) tile map system. I'd have to see how the chunks are: I already coded a streaming system which works in both horizontal and vertical directions, so with BigMap which has unlimited height, I would only need to care about streaming when character reaches a certain position on X, which should be even lighter. However that would combine with BigMap's own runtime code, so it may end up being more tokens/chars in the end.
Or, as you suggested at the beginning, I could try and compress the native tilemaps with PX9 to start with (but then I'd keep using my horizontal-vertical streaming system).
It seems that in all scenarios I have to start working on reducing chars usage before I attempt integrating new systems, so I'll spend some time on it first. Then I'll probably end up using raw tile data rather than metatile to spare level editing time, so I may end up not using PicoMap itself - in which case I hope your changes will still be beneficial to other users!
One last thing: how to concretely use the PX9 data with the new version? I didn't see clear changes in the instructions.
I had a look at the code and identified these memory positions:
map_adr,spr_adr,lvl_bytes=-32672,-32491,96 |
so I assume we should copy memory from these addresses (not sure how much length though) to current tilemap memory (resp. spritesheet).
I'm also not sure about negative addresses, is it a trick exploiting modulo?
@ huulong,
Well, your current setup looked so tricky to deal with I thought I'd see if I could make it a little easier for you, and it was an interesting challenge.
As far as encoding data in PX9, that's what I've been planning to do for my projects anyway. The one thing that's a little sticky is map data, as PX9 doesn't seem to compress it quite as well as Pico-8's code compression. I might make it an option to store that as a string.
The current full level map actually isn't using PicoMap's metatile sytsem, just loading the raw tile values that I compiled using some simple scripts into an array and mapping that using the map_array() function. Since this can handle huge maps of any size, the map doesn't need to be broken into chunks, so it's very simple to work with. If all you need is the one level, your best bet would be to just compress the existing raw data w/PX9, I just suggested rebuilding it w/ PicoMap if you were interested in having 2 or 3 levels, which should be possible but would take a good deal of work.
To build the demo map, I opened each data cart and dumped its contents to a string, then I combined all the strings into one large one, using a bit of extra logic to arrange the tiles properly for one large contiguous map.
Finally, about the memory addresses, those are the locations in upper memory where the compressed map data and spritesheet data are transferred to. In order for PX9 to decompress data, there has to be free space available in both map memory and the spritesheet, so I transfer the compressed data to upper memory and from there decompress it to low memory where it can be used. The addresses are negative, since they're >32768 and Pico-8 can't natively display numbers that high, so they read as negative but still work correctly.
I spent a while tonight trying to compress the map via PX9, though for some reason it wasn't compressing correctly until I compressed it in 2 chunks instead of 1. Anyway, it compresses to about 4.26KB in the spritesheet. I was going to compress the spritesheet as well, but I'm not sure if I can put the data in the spritesheet or it could overwrite itself and cause errors, and I can't use map memory because that's holding the decompressed map. Might be able to use sound memory, but that could be complex and/or waste a lot of space. I think the most efficient way could be using the setup in the demo cart where an array is holding the whole map, which also clears up 32KB for decompressing data (potentially including multiple spritesheets, which could be swapped in and out at runtime). The compressed map and sprite data should likely all fit in the spritesheet and the code will only take about 450 tokens and around 1600 characters total. I'll look into that.
Took a little debugging, but I was able to compress the whole map and spritesheet and store them in the spritesheet memory of a single cart, only takes 7KB! It decompresses the map directly to the level array, so upper memory is left free to store compressed data, or even decompressed data like extra spritesheets, but means the small included functions aget() and aset() need to be substituted for mget() and mset().
I used separate carts for compressing map data and sprites (used in that order) to avoid playing musical chairs with memory. To edit the map, you'll need to use a map editor that functions within upper memory, like Pancelor's BigMap. This allows 128 screens of workspace, but you'll need to cut the map in half and stack the halves vertically. Once you have that, this export method (minus the string-reading loop) will compress it.
Wow, thanks! I'm having a pass to simplify game code and was checking how much memory and how many carts I would need exactly to make the game work on BBS (loading everything in advance).
As you said, big map has just enough space for the full level but I'll have to cut it in two halves, I suppose what rearranges them at runtime in your code is that line in function map_array:
mset(111+col,row,array[r][(col+px)%w]) |
I haven't looked in depth at PX9 but apparently you need to apply a &255 mask to process compressed data. I thought it was a byte anyway looking at aset, is it just for safety?
Otherwise the final code looks very compact and will spare a lot of tokens/chars compared to my current level streaming code! (but this wasn't possible at the time without Big Maps)
I opened sonic_map_test-1.p8.png in PICO-8 and inspected map memory, it uses a bit more than 0x7000 (instead of 0x8000 is uncompressed), so I may even use a little extra general use memory on the space left (but risky since I can't predict the final compressed size if I change the map later - but feasible if I pay attention at offline time).
Not using level streaming, I already gain 0x300 (buffer memory I was using to rebuild the map live from stage cart data).
The extra ~0x1000 and 0x300 may not be enough to put all the game's sprites into general memory (title menu and stage clear have a lot of big sprites) but at least I should be able to fit the whole ingame into one cart now. The next bottleneck is code... what I'm working on right now.
Thanks for taking the time to do all of this and sorry for hijacking that thread into Big Map, but not PicoMap stuff. But maybe this can help other people who searched for a way to make bigger maps but realised they don't need metatiles, or need to convert old existing maps like me.
I'll tell you when I managed to integrate this in my game and whether I managed to make a BBS-compatible release! (probably multi-cart still)
I'll let other people discuss actual PicoMap features here now, but if we need to exchange further on Big Map usage or pico-sonic-specific stuff I will open a new thread.
That's alright, I've been thinking for a while that a map extending option that doesn't entail the complexity of metatiles could be very useful. I know Pancelor's BigMap editor exists, but my approach of storing levels in an array can allow for larger maps and leave all 32KB of upper memory free for things like storing multiple compressed spritesheets. Been meaning to work on it, but been busy with other things, this gave me a bit of a push to work on part of it.
Anyway, the code you pointed out streams a screen-size chunk of the map layout to a corner of map memory so it can be drawn to screen with a single map() command. Pico-8 has access to 2MB of RAM, and a 256-screen map only takes up 1/8 of that (4 bytes per array entry). The byte mask you see isn't actually for PX9, but rather a means to foolproof the array. Pico-8's default map uses byte entries only, simplifying any mset value to a byte, and only giving results for actual integer positions tested via mget, while an array can hold full 32-bit values, and will test for any entry specified, even a fractional one. I exploit these properties myself in my projects (for autotile data, or for info on what bricks contain in SMB, for example), but I figured I'd try and avoid the extra complexity for people using it generally. I guess a lot of the workings of PicoMap are counterintuitive because it's built on a lot of exploits to make Pico-8 do things it wasn't designed for.
[Please log in to post a comment]
