Hi everyone,

I am tinkering with procedural generation and I inevitably run out of memory. I expected to be able to delete some obsolete data to free some memory but I do not succeed. It looks like deleting variables does not free RAM. What did I miss?

I isolated that in the following script that just fill a global variable named STATE with random data, then set it to NIL, and track that it does not free any RAM. It outputs as such:

RAM: 0%
 FILL STATE WITH DATA...
RAM: 41%
 DELETE STATE...
RAM: 41%

The code:

state = {}

function _init()
 cls()
 print("ram: "..flr(stat(0)/20.48).."%")
 print(" fill state with data...")

 -- fill state with dumb data
 fill_state()

 -- test ram
 print("ram: "..flr(stat(0)/20.48).."%")
 print(" delete state...")
 state=nil
 print("ram: "..flr(stat(0)/20.48).."%")
end

function fill_state()
 state.data = {}
 local str=""
 for i=1,1000 do
  str=str.."abcdefghijklmnopqrstuvwxyz"..i
 end
 for i=1,100 do
  state.data[i] = {}
  for j=1,1000 do
   state.data[i][j] = str
  end  
 end
end

Isn't the garbage collector supposed to help me here? What did I miss? What is the way around the issue?
Thanks :)

OK it seems that the garbage collection just does not happen instantaneously. If I keep tracking the memory, it ultimately works.

I added an update loop to the previous code:

function _update()
 print("ram: "..flr(stat(0)/20.48).."%")
end

And it indeed outputs the right value at some point.

RAM: 0%
 FILL STATE WITH DATA...
RAM: 41%
 DELETE STATE...
RAM: 41%
RAM: 41%
RAM: 0%
RAM: 0%
RAM: 0%
...

(now I wonder why in my real-world case, ie. my game, RAM gets overloaded...)

did you do the math to get a sense of how much data you end up storing?
pico is limited to 2mb.

I adopted a design-to-cost approach. So my proof-of-concept, not yet optimized, showed me that the order of magnitude of what I need is coherent with what pico-8 offers. I plan to tune the content of the game to fit the limitation, and it seems to be working... except one thing.

I have a kind of "memory leak". I use a kind of landscape generator that constantly adds and removes cells to a grid. The grid has a constant size. It is not supposed to eat more RAM over time, but that is still what I observe.

I am still in the process of investigating that matter. Identifying which elements actually eat which amount of RAM is a complicated process. I hoped to be able to monitor that by deleting data and observing the memory gain, but (1) the garbage collector gives the answer after an undefined amount of time and (2) I observe some discrepancies. Namely, I sometimes need to delete the keys of a table, because only deleting the table itself does not free all the RAM it should, and I don't know why.

Deleting the table only deletes the hashmap that references all the keys/values, but the keys/values are all strings in memory which might have other references to them if they were ever stored in a variable, especially if there's intra-item links in the original table in a structure.

It's fairly common in garbage-collected languages to have to delete all the items from a table or array before deleting the meta-object unless you are absolutely flawless on tracking references, as any variable that was set with one of those values can keep the value/key/whatever around forever.

A huge amount of work goes into GC languages to try to detect such 'circular' references or 'dead but looks alive' ones, but with Lua being a fairly light-weight language simply brute-force nuking something component by component is likely best.

Depending on how you handle your grid for your landscape generator, it's possible that you are triggering the table growth (resize to the next power of 2) but the table will not shrink it's memory allocation by itself when your delete elements from it.