Hi! Some folks on the IRC were struggling with metatables, and Ivoah suggested I post an explanation here to help more people get to grips with them. Here goes nothing:
A table is a mapping of keys to values. They're explained quite well in the PICO-8 manual so I won't go into more detail. In particular you should know that t.foo is just a nicer way of writing t["foo"] and also that t:foo() is a nicer way of calling the function t.foo(t)
A metatable is a table with some specially named properties defined inside. You apply a metatable to any other table to change the way that table behaves. This can be used to:
- define custom operations for your table (+, -, etc.)
- define what should happen when somebody tries to look up a key that doesn't exist
- specify how your table should be converted to a string (e.g. for printing)
- change the way the garbage collector treats your table (e.g. tables with weak keys)
Point #2 is especially powerful because it allows you to set default values for missing properties, or specify a prototype object which contains methods shared by many tables.
You can attach a metatable to any other table using the setmetatable function.
All possible metatable events are explained on the lua-users wiki:
>>> list of metatable events <<<
And that's really all you need to know!
Edit: it appears that __tostring doesn't currently work in PICO-8 Lua. That means some of the code below is only relevant to vanilla Lua. See the end of the post for workarounds.
Vectors Example
I'll now demonstrate how metatables could be used to make 2D points/vectors with custom operators.
-- define a new metatable to be shared by all vectors
local mt = {}
-- function to create a new vector
function makevec2d(x, y)
local t = {
x = x,
y = y
}
setmetatable(t, mt)
return t
end
-- define some vector operations such as addition, subtraction:
function mt.__add(a, b)
return makevec2d(
a.x + b.x,
a.y + b.y
)
end
function mt.__sub(a, b)
return makevec2d(
a.x - b.x,
a.y - b.y
)
end
-- more fancy example, implement two different kinds of multiplication:
-- number*vector -> scalar product
-- vector*vector -> cross product
-- don't worry if you're not a maths person, this isn't important :)
function mt.__mul(a, b)
if type(a) == "number" then
return makevec2d(b.x * a, b.y * a)
elseif type(b) == "number" then
return makevec2d(a.x * b, a.y * b)
end
return a.x * b.x + a.y * b.y
end
-- check if two vectors with different addresses are equal to each other
function mt.__eq(a, b)
return a.x == b.x and a.y == b.y
end
-- custom format when converting to a string:
function mt.__tostring(a)
return "(" .. a.x .. ", " .. a.y .. ")"
end
|
Now we can use our newly defined 'vector' type like this:
local a = makevec2d(3, 4) local b = 2 * a print(a) -- calls __tostring internally, so this prints "(3, 4)" print(b) -- (6, 8) print(a + b) -- (9, 12) |
Pretty neat right?
Object Orientation
I mentioned that metatables can be used to define what should happen when a key lookup fails, and that this can be used to create custom methods shared by many tables. For example we might want to be able to do this:
a = makevec2d(3, 4) a:magnitude() -- calculate the length of the vector, returning 5 |
In Lua this is not always necessary, for example, we could define an ordinary function to do the job for us:
function magnitude(vec) return sqrt(vec.x^2 + vec.y^2) end magnitude(a) -- returns 5 |
In fact, for PICO-8 I would recommend that approach, because it's as efficient as you can get, and it uses the least number of tokens.
But I think it's educational to see how metatables can make it possible to use Lua in a more OOP style.
First off, we define all our methods in a table somewhere. Note, you can define them in the metatable itself, but I'll put them in a different table to prevent confusion.
local methods = {}
function methods.magnitude(self)
return sqrt(self.x^2 + self.y^2)
end
|
The __index property of a metatable is referred to when you try to look up a key 'k' which is not present in the original table 't'.
If index is a function, it is called like index(t, k)
If index is a table, a lookup is performed like index[k]
So we can add the magnitude function to all our existing vector objects like this:
mt.__index = methods |
And now we can do a:magnitude()
Which is a shortcut for a.magnitude(a)
Which is a shortcut for a["magnitude"](a)
Hopefully given all this information, it's clear what's happening:
We never defined a magnitude property in 'a', so when we try to lookup the string "magnitude", the lookup fails and Lua refers to the metatable's __index property instead.
Since __index is a table, it looks in there for any property called "magnitude" and finds the magnitude function we defined. This function is then called with the parameter 'a' which we implicitly passed when we used the : operator.
Well, that's it from me! I hope somebody finds this post useful, and please let me know if there is something you don't understand, or something that I left out or could have explained better.
Edit: at the time of writing, __tostring doesn't work in PICO-8 Lua.
This means you'll either have to convert your objects explicitly, like so:
function vec2str(vec)
return "(" .. vec.x .. ", " .. vec.y .. ")"
end
print(vec2str(a))
|
or, if you are using the object oriented approach outlined above, you could do this:
function methods.tostring(vec)
return "(" .. vec.x .. ", " .. vec.y .. ")"
end
local oldprint = print
-- override the print function to look for a method called 'tostring'
function print(val, ...)
if type(val) == "table" and val.tostring then
oldprint(val:tostring(), ...)
else
oldprint(val, ...)
end
end
print(a)
|
edit: minor bbs formatting fix
When WoW ui went secure was the first time I really dug into LUA. There are such interesting things that can be done when everything is a table. It's turtles all the way down.
Really nice introduction, btw. It's great how deep yet accessible Pico-8 is.
Thanks :)
Yeah, metatables are one of the things I love about Lua. Where other languages have several different concepts such as arrays, hash tables, classes, inheritance, operator overloading... Lua manages to pull it off with just tables and metatables. It's so elegant!
Nice, I never used metatables, so your post is full of great tips. I specialy like operators, as it may reduce token usage a lot.
This is a very interesting topic and a clear introduction, thank you! Coming from C++ it took me a little while to get used to the idea of tables and the fact that Lua is a dynamically typed language. I didn't even realise that metatables were a thing! Really cool :)
This is great, can't wait to use this, it will be very handy.
There is just 1 thing I can't quite get.. In the function mt._add it calls make vec2d, which I follow, but then make vec2d adds a new vec to the metatable, and then returns it. Do I have to add it to the table when adding them if I only need the result? Lua is very new to me and very different to what I'm used too.
Thank you again for this great post!
Yes i've been using vector operators to reduce token usage greatly!
Hi Vermeer, sorry for the late response!
Yes, technically if you only care about the result, you don't need to set the metatable for the new table.
You could write the __add function like this:
function mt.__add(a, b)
return {
x = a.x + b.x,
y = a.y + b.y
}
end
|
But this means that the result of the add operation won't have any of those custom operators available to it, because it doesn't have a metatable. In other words, you can't chain vector operations together without calling setmetatable on the intermediate results.
a = makevec2d(1, 2) b = makevec2d(3, 4) c = a + b -- ok because both a and b have metatables with __add defined d = c + b -- ok because b has a metatable with __add defined e = c + d -- not ok, because neither c or d have metatables |
I could really use a hand digesting this.
Currently, in my project, I have a 'worldbuilding' script that ups a "scene ID" (SID) by one every time it adds a new scene, and then it runs a loop set for a sequence of Metroidy items, before concocting the endgame. Not being aware of metatables, I've been trying to concoct a way so that each new scene ID also maps two "door IDs," that cross-reference one another so that the areas connect cohesively.
I have a little retooling to do, since I'm also changing the way scenes are generated... they used to copy Map data, now I'm making it so it copies sprite data, and pastes that as interpreted map data instead; keeps more map open and allows the game to "remember" more previous scenes before overwriting them. But that's kind of an aside.
Thank you for laying this out so well.
Has anyone done any profiling to compare, e.g., vector addition with operator overloading via metatables, and doing it manually on Pico-8? Obviously this wouldn't be worth thinking about much in a normal Lua environment but it's a different story when counting cycles on a fantasy console--especially if we're using this to simplify something like vector math that could be called repeatedly within an inner loop.
(I've found a number of tiny gotchas like this. x*x is significantly faster than x^2 in Pico-8, for example. Ditto a[#a+1]=b vs. add(a,b).)
musurca: I wonder how to do profiling in the first place? Checked the manual and I can't see any way to get the system time or count elapsed cycles. I'm aware that pico-8 has some sort of simulated cost for certain operations, my guess would be that it's about as expensive as a table lookup + function call?
TonyTheTGR: hey! This sounds more like a data-related problem, constructing a game world in which references between objects are set up correctly. Not much scope for using metatables here, it's probably better / less confusing to do without them :)
If you wanted to see how metatables could be applicable (I'm just taking a guess at how your system works) you could possibly use the __index metamethod here to return some kind of default door ID for a scene that has not been initialised correctly.
Here's an example:
local scene_mt = {}
-- __index can either be a table or a function. I'll present both possibilities:
-- A) a lookup table for undefined keys
scene_mt.__index = {
doorid1 = 0,
doorid2 = 0
}
-- B) a function to return default values for some undefined keys
function scene_mt.__index(t, k)
if k == "doorid1" or k == "doorid2" then
return 0
end
end
-- example usage (works the same for solution A or B)
local myscene = {
foo = "bar",
doorid2 = 5
}
setmetatable(myscene, scene_mt)
print(myscene.foo) -- prints "bar"
print(myscene.doorid1) -- prints 0
print(myscene.doorid2) -- prints 5
print(myscene.donut) -- prints nil
|
I'm not sure how useful it would actually be. A simpler solution, without using a metatable, would be to make an ordinary function that returns door IDs with default values.
-- function that returns a list containing both doorIDs in the scene, with default values
function getdoorids(scene)
return {
scene.doorid1 or 0,
scene.doorid2 or 0
}
end
print(getdoorids(myscene)[1]) -- prints 0
print(getdoorids(myscene)[2]) -- prints 5
|
Or to just remember throughout the code that door IDs could be nil, and treat them as such. Depending how often you accessed those doorid attributes, this might not be a waste of tokens.
I dunno, maybe this is an imaginary problem that doesn't actually exist in your code, I just thought it might be useful to show some more examples of doing things with / without metatables.
geckojsc: It's possible to profile in a rough way. While deprecated, time() is still present in 0.1.6. The resolution is poor (only accurate to within 1/30th second), so you have to test over a large sample set. I wrote the following code for comparing performance of functions:
function profile(label,func,tab)
local t=time()
local x
local r
for q=1,20 do
for i=1,#tab do
x=tab[i]
r=func(x)
end
end
t=time()-t
print(label..": "..t.." secs")
end
--example usage of profile()
--comparing x*x vs x^2
normtable={}
for i=1,32767 do
normtable[i]=rnd(1)
end
function hermite(a)
return (3-2*a)*a*a
end
function hermite2(a)
return (3-2*a)*a^2
end
profile("hermite",hermite,normtable)
profile("hermite2",hermite2,normtable)
|
That actually does help a lot; especially since the scenes have plenty of potential doors (8-16 isn't uncommon, some as many as 20; but only 2-5 get used). So having some kind of default value is good. The idea is that the door generation/placement is actually what determines the content generated in the scene too.
Then, I guess the other thing is how to make sceneID.doorID point to the actual "sceneID" and not just a string called "sceneID". I forget if it's percents or a dollar sign... ~.~'
Hi geckojsc, thanks for clearing that up for me, I follow it now and I find this so helpful, and I see what you mean about chaining operations. Thank you again. :)
Hey @geckojsc, I saw this post on the Lua.Space blog and it helped me a lot. I was stuck on trying to make my vectors too OOP, and I'm glad for your perspective on keeping things slim and simple for Pico.
I do have one question, I don't really understand the use of local in this situation. And why call the metatable just mt? I felt like I wanted to name it vec or vector so that i could recognize all of the functions related in that way. i know maybe its just a matter of style, but i'm curious. hope you're still around to read this post!
I believe the use of "local" at the top level is always extraneous. It results in a global variable, just as if "local" were omitted. It only does something inside a function. I would also agree that "mt" should be a more specific name in this example as a matter of style.
A more common pattern for defining a vector "class" using metatables is to create an object for the class, then set the metatable as "self" in the constructor. geckojsc may have (rightly) avoided this example in this tutorial to avoid introducing too many concepts at once. Example: http://pico-8.wikia.com/wiki/Setmetatable
I did read something in the Lua manual about how local is faster than global? I don't know enough about efficiency to understand it tho. And I don't think it really makes a global because I can't access the ones declared local in the script when I try calling them in the interpreter.
I have read about using metatabels for making objects, but I also agree that for Pico8 maybe it's better to stay light and flexible. I am not sure if I really need a vector class or not. Lua is nice in that it allows for that flexibility and it probably pays off well as you approach the token limit (which I've never come close to yet)
Good point about top-level locals not being accessible at the console, though I don't see a practical advantage. The console operator is the only possible user of globals outside of the top level of the program. I suppose if you wanted to redefine Lua built-ins globally to your program while preserving their original definitions for the console:
print('one')
orig_print = print
local print = function(msg)
orig_print('** '..msg..' **')
end
print('two')
|
:)
Its kind of a curious thing in this particular environment, because I often like to ESC out of a game I'm playing on SPLORE and poke around at the code, print the values of certain globals, etc. It can be handy to have access in the console to parts of the game that do stuff.
Using local would prevent certain parts of the code from being accessible to users, often good programming practice right? But in Pico8 it seems like exposing everything to the user can allow for experimentation and learning, who cares if you break something? just reload and try to figure out why. i guess the user can always just delete the local in front of the assignment, it probably shouldn't break everything.
however, seeing how this works makes me think that declaring things local must do some compiling of those local assignments, whereas if you assign globally, the lua is running uncompiled. this is just a guess, but i think maybe that is a reason to do local assignments this way. I also notice that LOCAL doesn't cost a token!
The game code is just running in its own function scope, with the Pico-8 driver Lua code appended and the whole thing fed to the interpreter. We've seen related side effects with exceptions and coroutines, and you can examine the driver code by running the Linux "strings" command on the Pico-8 binary.
It's easy to see that both the Pico-8 console and the Lua REPL run each command that you type in its own scope as well. "local x = 7" followed by "print(x)" prints "nil". "local x = 7; print(x)" on a single line prints "7".
I agree that hiding globals from the console doesn't seem to have real benefit and is detrimental to inspection by console.
I only just took a look at metatables (at last X-])
and tripped on the __tostring issue (still doesn't work as of 0.1.10c, not holding my breath)
though I found a quite versatile way around it by overriding the concat operator:
vec={}
function vec.new(x,y,z)
local v={x=x,y=y,z=z}
setmetatable(v,vec)
return v
end
function vec.__tostring(v)
return '('..v.x..','..v.y..','..v.z..')'
end
function vec.__concat(u,v)
-- if (getmetatable(u)==vec) u=vec.__tostring(u) --not available
-- if (getmetatable(v)==vec) v=vec.__tostring(v)
if(type(u)=="table") u=vec.__tostring(u)
if(type(v)=="table") v=vec.__tostring(v)
return u..v
end
u=vec.new(1,0,0)
v=vec.new(1,0,8)
print(u) -- !!! "table"
print("vec:"..u) -- ok
print(""..u) -- ok
print(u.."") -- ok
print(u..v) -- ok
|
so there's that, just don't print a table on its own.
I didn't know about metatables, this is pretty cool though I'd be cautious about overwriting default table behavior... If overdone that can lead to some pretty tricky bugs. The vector example is great though, for mathy things it's definitely appropriate.
FWIW, I use a slightly different approach to object construction and I tend to avoid the : syntax by using scope instead because it seems strange to me to require calling, for example, tostring() in a special way when you would never want to pass some other value in besides its own table. Here's an example of what I mean, converting the OP code:
-- function to create a new vector
function makevec2d(x, y)
local t
t = {
x = x,
y = y,
tostring = function()
return "(" .. t.x .. ", " .. t.y .. ")" -- note that we are always referring to t here, not a parameter to tostring()
end
}
setmetatable(t,{
-- these functions could refer to t instead of a, but their signatures can't be changed so there isn't a point in this case
__add = function(a, b)
return makevec2d(
a.x + b.x,
a.y + b.y
)
end,
__mul = function(a, b)
if type(a) == "number" then
return makevec2d(b.x * a, b.y * a)
elseif type(b) == "number" then
return makevec2d(a.x * b, a.y * b)
end
return a.x * b.x + a.y * b.y
end,
__eq = function(a, b)
return a.x == b.x and a.y == b.y
end
})
return t
end
|
There's more indentation involved, but IMO that's a good thing as it's easier to visually distinguish the code belonging to that "class" without having to use big block comments or something. Indentation spaces count as code unfortunately, but you can always write a script to strip out indentation and comments and such before publishing.
A nice side effect of using scope rather than :functionname() is that you can pass around the function and know it will always work as expected on the original table, even when no longer attached to the original table.
I do a similar thing in javascript to avoid having to use the "this" keyword, which bothers me for similar reasons... I don't like a function's behavior to change when it's attached to something else.
Do other people code in a similar way or am I just violating Lua best practices? Hopefully this doesn't come off as pedantic, I just really like thinking through ways to organize code as making code as understandable as possible is IMO super important.
@jcwilk: There's an important reason not to do it that way: memory usage. Your example creates all-new function objects in the environment of the scope for each instance. Similarly, each instance is getting its own metatable object when they could easily be sharing because none of the functions in the table refer to anything in the outer scope. (Your JavaScript code would have the same issue.)
Applications that construct and delete many objects would thrash the garbage collector more with much larger objects. Construction speed would also be an issue in some cases.
In the following demo, makevec2d_v1 is identical to yours. makevec2d_v2 is functionally equivalent except the tostring implementation and metatable are constructed only once. The driver creates one object per frame and displays the memory usage and object count. Try running this as is, then change the driver to use makevec2d_v2 and notice the difference.
function makevec2d_v1(x, y)
local t
t = {
x = x,
y = y,
tostring = function()
return "(" .. t.x .. ", " .. t.y .. ")"
end
}
setmetatable(t,{
__add = function(a, b)
return makevec2d_v1(
a.x + b.x,
a.y + b.y
)
end,
__mul = function(a, b)
if type(a) == "number" then
return makevec2d_v1(b.x * a, b.y * a)
elseif type(b) == "number" then
return makevec2d_v1(a.x * b, a.y * b)
end
return a.x * b.x + a.y * b.y
end,
__eq = function(a, b)
return a.x == b.x and a.y == b.y
end
})
return t
end
mt = {
__add = function(a, b)
return makevec2d_v2(
a.x + b.x,
a.y + b.y
)
end,
__mul = function(a, b)
if type(a) == "number" then
return makevec2d_v2(b.x * a, b.y * a)
elseif type(b) == "number" then
return makevec2d_v2(a.x * b, a.y * b)
end
return a.x * b.x + a.y * b.y
end,
__eq = function(a, b)
return a.x == b.x and a.y == b.y
end
}
function vec2d_tostring(t)
return "(" .. t.x .. ", " .. t.y .. ")"
end
function makevec2d_v2(x, y)
local t = {
x=x,
y=y,
tostring=vec2d_tostring
}
setmetatable(t, mt)
return t
end
function _init()
vecs = {}
ct = 0
end
function _update60()
-- try changing this line to use makevec2d_v2:
add(vecs, makevec2d_v1(3, 5))
ct += 1
end
function _draw()
cls()
print('ct = '..ct)
print('stat(0) = '..stat(0))
end
|
(Edit: Fixed v2 example to actually return the object, as noted below. Not doing so unfairly skewed the experiment. :) )
@dddaaannn - Wow, thanks, that's pretty eye opening... I sped it up a bit by making it create 100x each loop and output the memory/iteration ratio so I could see more fine grain changes depending on specific little tweaks.
For convenience in case anyone wants to follow along:
function _update60()
-- try changing this line to use makevec2d_v2:
for i=1,100 do
add(vecs, makevec2d_v1(3, 5))
end
ct += 100
end
function _draw()
cls()
print('ct = '..ct)
print('stat(0) = '..stat(0))
print('ct/st = '..stat(0)/ct)
end
|
Also the v2 constructor function was missing a return at the end, nbd.
It really surprised me how much memory was added to each constructed object from adding a reference to any of the constructor local variables to a function created inside the constructor. Particularly that there was a significant amount of memory added by referencing the t variable (the object that is eventually returned). That was strange to me since I know that that t variable refers to the exact same object as the one that gets stored in the array (same as in, change one and the other changes) so aside from the fact that the variable is marked as "in scope" to the function (seems like this shouldn't take much space) there isn't any actual additional data being persisted in memory. Also, if in the same function a reference is added to some other additional local variable, like x, it increases the memory somewhat significantly as well.
To give an idea of the range, here are some numbers: (higher ratio is worse)
0.125 the stripped down example you countered with
0.174 my version, modified by taking out and sharing the mt variable
0.284 my version I proposed above (as you can see, the mt object and its functions are pretty singificant)
0.41 added a few more references here and there which make total sense conceptually, but clearly add a lot of memory
Lame! Disappointing, lol, I was so pleased with the approach but it looks like it comes at a steep price. Using scope like that definitely has its place, but clearly also its cost. Most of the time in pico8 things aren't going to be instantiated thousands of times, of course, but memory does matter and a little bit of extra messiness in code structure for a 3x or even much higher (with many local functions it seems that scoped variables getting added for each keeps adding and adding up) reduction in resultant object size is pretty reasonable.
Anyways, long way of saying yep you're right, that's probably not a good go-to strategy I've been doing :P
Edit: I wonder if JavaScript handles things more efficiently, I'll have to run some tests next time I have a chance. Of course in JS the memory constraints are orders of magnitude more generous so a 5x increase in something you'll make a few times can be totally irrelevant, but still good to get a more specific idea of the kind of tradeoffs that are being made.
My version not returning t makes a big difference, so thanks for that fix. The point still stands (the fixed version uses about 1/3rd the memory), but the version I posted used almost no memory because it wasn't retaining the constructed objects, which is incorrect. :)
The primary memory consumption difference is the functions themselves, which are remade for each instance. It would also retain any (outer) local variables the functions reference when they are first created, basically none in either version above other than the t which as you noted is retained anyway.
From a bit of experimentation it appears Lua is smart enough to not retain the entire environment (local variables) from makevec2d_v1 for use by the inner functions. It only retains what is referenced by those functions. Fun thing to try: add this just inside makevec2d_v1:
local unused = {
a=1, b=2, c=3, d=4, e=5,
ba=1, bb=2, bc=3, bd=4, be=5,
ca=1, cb=2, cc=3, cd=4, ce=5,
da=1, db=2, dc=3, dd=4, de=5,
ea=1, eb=2, ec=3, ed=4, ee=5,
fa=1, fb=2, fc=3, fd=4, fe=5,
}
|
That alone results in no change to the memory usage. Now add this line to the "tostring = function() ... end" definition:
local zzz = unused['a'] |
That's enough to inspire Lua to keep an "unused" table around for each instance, because it is referenced from an inner environment.
oh wow, I never thought I'd get my head around metatables, this is the best explanation I've seen yet. thanks.
[Please log in to post a comment]
