Understanding ENVIRONMENTS
Inherited from native LUA, _𝘦𝘯𝘷 (NOTE: type it in PUNY FONT, won't work otherwise! For External editors, write it in CAPS _ENV so it transfers as PUNY FONT inside PICO-8) can be a tricky feature to understand and use (abuse?). There's been some talking around this subject and as I am actually using it heavily in my coding style (very OOP I'm afraid...) I thought I could share some knowledge around it.
If we take the information directly from LUA's manual what we are told is the following:
As will be discussed in §3.2 and §3.3.3, any reference to a free name (that is, a name not bound to any declaration) var is syntactically translated to _ENV.var. Moreover, every chunk is compiled in the scope of an external local variable named _ENV (see §3.3.2), so _ENV itself is never a free name in a chunk.
Despite the existence of this external _ENV variable and the translation of free names, _ENV is a completely regular name. In particular, you can define new variables and parameters with that name. Each reference to a free name uses the _ENV that is visible at that point in the program, following the usual visibility rules of Lua (see §3.5).
Any table used as the value of _ENV is called an environment.
Cryptic right? Let's try to distil the information...
Let's start with understanding what's a free name. Any declared identifier that does not correspond to a given scope, f.e. GLOBAL variable definitions, API defined symbols (identifiers, functions, etc) and so on, is considered a free name OK... so this starts to get us somewhere! Any time we are using a global scope variable name or an API function call, internally LUA's interpreter is actually converting that to _𝘦𝘯𝘷.identifier. These tables used as values for _𝘦𝘯𝘷 are usually called ENVIRONMENTS.
Let's move into the next part: _𝘦𝘯𝘷 itself is just a local identifier in your current scope, and like any other identifier you can assign anything to it. In particular, you can overwrite the automatically created _𝘦𝘯𝘷 in your current scope, which will always be the GLOBAL ENVIRONMENT, and point it to any table, and from that point on, any free name will be looked for inside that table and not in the _𝘦𝘯𝘷 scope originally received .
Everytime a scope is defined in the LUA execution context, the GLOBAL ENVIRONMENT (on isolated scopes, like function scopes) or the currently active environment (inside language scope constructs) is injected as a local external variable _𝘦𝘯𝘷. That means every function scope, any language scope construct (do ... end, if ... end, etc)
Great! So that's all there is to know about _𝘦𝘯𝘷... but how can we use this to our benefit? Let's find out!
Using ENVIRONMENTS
The core and simplest use-case for ENVIRONMENTS is mimicking WITH language constructs. It's quite typical that you have a table in your game holding all the information of the player... it's position, health level, sprite index, and many others probably. There's almost for sure some place in your code that handles the initialization of the player and that probably looks something similar to this:
player={}
player.x=10
player.y=20
player.lives=3
player.sp=1
-- many more here potentially... |
Depending on how many times you need to access the player table you can actually consume a big number of tokens (1 per each access to player). When you have more than 3 (the general cost of redeclaring _𝘦𝘯𝘷) or 4 (the cost if you require a scope definition for it) you can benefit from not needing to repeat PLAYER for every access like this:
player={}
do
local _𝘦𝘯𝘷=player
x=10
y=20
lives=3
sp=1
end |
The use of the DO ... END block in there prevents that we override _𝘦𝘯𝘷 for all our code, and that this applies only to the lines we want it to apply.
This technique is particularly useful if you use an OOP approach where you pass around SELF as a self-reference to table methods and will reduce drastically the need to repeatedly use SELF everywhere:
player={
x=10,
y=20,
lives=3,
sp=1,
dead=false,
hit=function(self)
local _𝘦𝘯𝘷=self
if not dead then
lives-=1
if (lives==0) dead=true
sp=2 -- dead sprite
end
end
}
?player.lives
player:hit()
?player.lives |
You can even reduce the need for the override this way (thank you @thisismypassword for the contribution):
player={
x=10,
y=20,
lives=3,
sp=1,
dead=false,
hit=function(_𝘦𝘯𝘷)
if not dead then
lives-=1
if (lives==0) dead=true
sp=2 -- dead sprite
end
end
} |
In a sense, this is very similar to WITH language constructs in other programming languages like Object Oriented BASIC or PASCAL. The benefit: reduce token count and char count. But don't just jump blindly into using this, get to the end of this post and understand the associated drawbacks!
Drawbacks of using ENVIRONMENTS in PICO-8
The main drawback for using an overriden ENVIRONMENT is loosing access to the GLOBAL ENVIRONMENT once we replace _𝘦𝘯𝘷 in a particular scope. When that happens, we stop seeing global variables and all API and base PICO-8 definitions, so we won't be able to call any function, use glyph symbols or access global vars. Luckily, METATABLES can help us here taking advantage of __INDEX metamethod.
METATABLES can be defined as a PROTOTYPE for other tables, and apart from defining elements to inherit in ohter tables, they can also define the BEHAVIOUR of these tables using METAMETHODS. Those are a quite advanced feature in LUA and I won't cover them in this post but for __INDEX that is going to be our solution for keeping our access to the global scope even if we have overriden our ENVIRONMENT (at a small performance cost...)
__INDEX METAMETHOD defines the behaviour of a table when we try to access an element that is not defined in it. Try this code to have a clear example of what it does:
d=4
mytable={
a=1,
b=2,
c=3,
}
setmetatable(mytable,
{
__index=function(tbl,key)
stop("accessing non existent key "..key.."\nin table "..tostr(tbl,true))
end
})
?mytable.a
?mytable.b
?mytable.c
?mytable.d |
__INDEX can be defined as a function or as a TABLE that will be where the missing identifier will be searched for... and there's our solution, just redefine __INDEX as _𝘦𝘯𝘷 and our problem is solved:
d=4
mytable={
a=1,
b=2,
c=3,
}
setmetatable(mytable,{__index=_𝘦𝘯𝘷})
?mytable.a
?mytable.b
?mytable.c
?mytable.d |
If we apply this approach to our previous example we can do things like this:
player=setmetatable({
init=function(_𝘦𝘯𝘷)
x=10
y=20
lives=3
sp=1
w=2
h=2
fliph=false
flipv=false
dead=false
end,
draw=function(_𝘦𝘯𝘷)
cls()
spr(sp,x,y,w,h,fliph,flipv)
end,
update=function(_𝘦𝘯𝘷)
if btnp(⬅️) then
if (x>0) x-=1
fliph=true
elseif btnp(➡️) then
if (x<128-8*w) x+=1
fliph=false
end
end,
hit=function(_𝘦𝘯𝘷)
if not dead then
lives-=1
if (lives==0) dead=true
sp=2 -- dead sprite
end
end
},{__index=_𝘦𝘯𝘷}) |
That's a very basic OOP-like approach to have entities inside your games, that expose entity:update() and entity:draw() methods you can call in your game loop (like having a list of entities in a table and iterate it calling update() and draw() for each entity inside your _update() and _draw() functions).
__INDEX will also come to the rescue when working inside functions to prevent losing global access... as functions themselves don't have metamethods you can "trick" things if you set _𝘦𝘯𝘷 to a table that has __INDEX set and leverage the lookup to it:
tbl=setmetatable({a=1},{__index=_𝘦𝘯𝘷})
b=2
function doit(_𝘦𝘯𝘷)
?"tbl.a=>"..a
?"global b=>"..b
end
doit(tbl) |
There is another effect that loosing access to the GLOBAL ENVIRONMENT generates, and this one is not fixed by __INDEX. Any new free name we create will be created inside the active ENVIRONMENT, so we won't be able to create any new GLOBAL variable from within the scopes affected by the override. If you need to be able to access the global space, one easy option is have a global variable pointing to the GLOBAL ENVIRONMENT in your code. Natively, LUA has _G environment available everywhere, but this one is not present in PICO-8, so you will need to create your own.
-- used by __index access
_g=_𝘦𝘯𝘷
mytable=setmetatable({
a=0,b=0,c=0,
init=function(_𝘦𝘯𝘷)
a,b,c=1,2,3
newvar=25 -- this is created in mytable
_g["initialized"]=true -- this is created in GLOBAL ENVIRONMENT
end
},{__index=_𝘦𝘯𝘷})
mytable:init()
function _draw()
-- outside table scopes (no __index access)
local _g,_𝘦𝘯𝘷=_𝘦𝘯𝘷,mytable
if _g.initialized and _g.btnp()>0 then
a+=1
b+=1
c+=1
end
cls()
?tostr(initialized)..":"..mytable.newvar..", "..mytable.a..", "..mytable.b..", "..mytable.c
end
|
Other uses
There's many more uses you can find for using ENVIRONMENTS... shadowing and extending the GLOBAL ENVIRONMENT, some Strategy Pattern approach (f.e. having several tables with alternate strategies extending the global scope and switching between them...). Don't be afraid to experiment! The basics are all here, the limits are yours to find!
Great writeup!
Instead of writing function(self) local _𝘦𝘯𝘷 = self, though - you can instead just write function(_𝘦𝘯𝘷).
This has the same effect since function parameters are really just locals, and it's shorter.
You can do the same trick with for _𝘦𝘯𝘷 in all(something), by the way.
@thisismypassword good trick... will add that to the post as alternatives... did not ocurr to me that would work :) Thnx for pointing them out!
Excellent writeup! This goes through everything you need to know, and ends with a very solid (and simple) example of usage. It's always great when someone puts in the time to demystify something into a BBS post like this, and I really respect the effort here to not only explain a new tool, but use it well.
THnx @shy, I've been through so many conversations in Discord about _env I thought it was time to dump all that into a post that could help out to propagate the knowledge.
Just added extra info on GLOBAL ENVIRONMENT access & var creation that was missing.
Wait, if I'm understanding this correctly, then it's the best token optimization I've ever heard? And it makes your code more readable and more amendable to OOP approaches like the update pattern?
God, I thought I knew Lua but this was eye-opening, thank you.
yes and no! it helps saving a ton of thing. tokens for code that’s already written in a object-oriented style, possibly also entity-component systems, but won’t do anything for code that uses closures (function that has local variables and return functions that work with these local vars) or global objects (messy, I know!).
all games use the update pattern, but not necessarily in an OO way :)
@shastabolicious glad it helped you out to understand environments. If your style is OOP friendly this and particularly combining it with __call can do wonders ;) I will write another post on my approach to an entity system (that is a bit... convoluted)
Good point, @merwok. Do you know any games making heavy use of closures or other functional programming tricks? I'd love to read their source.
Good write up on _ENV @slainte. Possibly worth pointing out for people using external editors that _ENV should be capitalized but will appear in puny font inside of the pico-8 editor. That's probably already common knowledge but I have seen a few people get tripped up by it from time to time.
@shastabolicious, you might be interested in the curried functions utility I posted a while back. It's probably not super useful/practical in a pico-8 context but it's implemented using closures and higher-order functions and it's quite short.
Maybe this is a stupid question, but why did you set _g.["initialized"] instead of just _g.initialized? They're equivalent.
Also the _g here in _draw() seems to be unnecessary, since the table you're assigning to _env contains a metatable with an __index that contains _g. It might gain you a small perf upgrade by having a local cache of _g in an inner loop, but there's otherwise no need for the token expenditure.
local _g,_𝘦𝘯𝘷=_𝘦𝘯𝘷,mytable |
Oh wait, maybe you meant to say it'd be useful for tables that don't have the metatable? I can see how you might have been iteratively writing up example code and then thought to yourself "Wait, I don't need two tables here, I can just use the first one," while forgetting the reason you'd set up the second. Is that what happened?
Hey @Felice, thank you for the comments...
About the [] access vs the . access you are right, they are equivalent... just used different access methods (as there's _g.initilized inside _draw) to show all the options
For the initialization of _G, you are also right, but it was written like an isolated example in mind for when you have a function (not a table) and need to access globals without having the option to use __index. Ended up a bit messed as it mixes table/non-table _env overrides and that renders some of the code useless... I will maybe split that into 2 separate examples with a table without __index involved.
Thnx for pointing those out!
I had a go at this as it looks like it's really useful and I think I'm understanding most of it, but I stumbled over a weird thing that I can't fully explain:
maker=function(x) return setmetatable(
{x=x,
printx=function(_ENV) printh(x) end,
changex=function(_ENV) x="Inside" end
}
,{__index=_ENV})
end
mytab=maker("Initial")
printh(mytab.x)
mytab.x="Outside"
printh(mytab.x)
mytab:printx()
mytab:changex()
mytab:printx()
printh(mytab.x)
|
The output is:
Initial Outside Initial Inside Outside |
It behaves like the x value is different for functions in the table than accessing it from outside.
I hit this when using a factory function where most of the time I define an entity's position at creation, but sometimes I didn't have a position then and needed to update it later. So I'd not pass a value for x or y to the constructor. I'd then get errors like: "attempt to perform arithmetic on upvalue 'x' (a nil value)" when I tried to draw the entity as the x value would still be nil.
It's easy enough to work around: don't have the argument names match the member names of the constructor function (e.g. new_x vs x), but it worries me that I'm not really clear what's going on despite reading more lua threads around the web than I ever want to see again :s
Can someone explain, please? Is there a better way round it that I've not found?
Originally I said you had closure like access and that local X was shadowing the internal X (but I edited the post)
@drakeblue forget what i said about closure access... I had not set my example as local :P (idiotic error on my side...)
Not really sure why this is happening, Native LUA has exactly the same behaviour... will try to investigate
@Cerb043_4 yes but you need to set an access to it... see the examples for _G
initialized=false
function _draw()
-- outside table scopes (no __index access)
local _g,_𝘦𝘯𝘷=_𝘦𝘯𝘷,mytable
if not _g.initialized and _g.btnp()>0 then
a+=1
b+=1
c+=1
_g.initialized=true
end
end |
NOTE: you can also need to access globals explicitly so this is not just a way to solve not having __index access, but to support writing on the global scope
Native LUA provides native _G and _ENV, but PICO does not provide access to _G (I guess the PICO8 custom interpreter uses an already tampered with scope and the native _G would make no sense there...)
@slainte This is practical for ingame right? I'm not understanding. You are defining the global native environment manually...to do this you are redefining all the variables of your game within a table? Restuffing the init of a game into a metatable...? Or defining the init of a game within a metatable to begin with...? Maybe the example is lost on me, humour me if you will with this example:
function _init() _G = _ENV; _G.__index = _G
outside_var=9
player={cx=44,cy=44,r=8,
draw=function(self)
--native env
do setmetatable(player,_G) local _ENV=player
circ(cx,cy,r,0) --local env
end
--native env
outside_var+=1
do --[[setmetatable(player,_G) alrdy is]] local _ENV=player
outside_var-=10 --is player.outside_var; local env again
circfill(self.cx,self.cy+30,self.r,14)
end
end}
end
function _draw()cls(1) player:draw()print(outside_var)print(player.outside_var) end |
So here a flipflip between the native/default and local environments seems to be achieved. What is the method you proposed applied to this example?
edit: Okay I understand now, thankyou for re-explaining.
@Cerb043_4 let's see if I get what is your question and analyze a bit on your example...
So in your example inside init you:
- create a global _G that points to the GLOBAL SCOPE (native _ENV) and self-index it
- create a global named outside_var, initilized to 9
- create a global table named player
- For this "player" table, inside it's draw function you metatable it to _G so all the GLOBAL SCOPE is accessible by __INDEX access. You have two scoped blocks (using DO ... END) that move the active environment to "player", so in the second one you access a variable "outside_var" that is not present in that environment and gets created inside it (player) so you effectively have two separate "outside_var", a global one and an internal player.outside_var
Correct? I think your example is quite clear and understandable... The only "odd" effect in your draw function is the initial value for outside_var in the player environment is coming from the global environment one (accessed the global through __index that is 9+1 from the first iteration) and set the new value as 10-10 but now the write happens in the new _ENV (player)
if you need to write into the GLOBAL SCOPE you need to explicitly write to _G.varname or _G["varname"] (equivalent result, just different access methods)
@slainte Thanks for taking a look. I've gone with my workaround of keeping the argument names different to the key names in the table and my project seems to be working fine now, but it would be nice to know what's happening.
@drakeblue had a quick look on LUA's source code... my guts tell me this is related with upvalue handling, but i'm not that expert on LUA's internals at this point (upvalues handle varname collisions and access btwn other things I think) and looks like the behaviour is not PICO's but LUA's... that's as far I've got without a wild goose hunt ;)
@drakeblue - if you write 'x' and there's a local variable named 'x' in either the current scope or any of the parent scopes - lua will use that variable instead of looking at the global scope (via _env).
That's why printh and changeh in your example mutate the 'x' function argument instead of '_env.x'. (function arguments and 'for' loop variables count as local variables)
OK, I am an idiot... I did the right assumption originally with the CLOSURE but implemented the example the wrong way so that derailed my hypothesis XD
I added a global c and a local c (so it is clear it's not accessing the global)
c="goodbye"
maker=function(x)
local c="hello"
return setmetatable(
{x=x,
printx=function(_env) print(c)print(x) end,
changex=function(_env) x="inside" end
},{__index=_𝘦𝘯𝘷})
end
mytab=maker("initial")
print(c)
mytab.x="outside"
print(mytab.x)
mytab:printx()
mytab:changex()
mytab:printx()
print(mytab.x) |
So what is happening is that _ENV access is the last resort to look for a symbol... first all closer scopes are checked, so the CLOSURE UPVALUE SCOPE for MAKER function takes precedence and that's why that "external local" x is used instead of the _ENV.X access
Well, this helped me learn something new about upvalues and closure scopes :)
Thank you for the help @thisismypassword
I've looked at this a couple of times and I think I've got my head round it now and it makes sense. Thanks very much for explaining @slainte and @thisismypassword.
Instead of writing function(self) local _𝘦𝘯𝘷 = self, though - you can instead just write function(_𝘦𝘯𝘷). |
@thisismypassword Wow, thanks so much for this trick, it freed up 83 tokens for me!
Thank you so much @slainte for this writeup! Finally had some time to sit down and understand it - should be a great resource for my current project.
One question though if you have the time.
When calling the add_entity function at the end, it seems like the parameters passed (x,y,dx) are automatically set in the ent object (hence the assignment for those are commented out). Are they actually set, or is there something else going on?
function add_entity(x,y,dx)
local ent=setmetatable({
-- x=64,y=64,
-- dx=rnd()+1,
on_update=function(_𝘦𝘯𝘷)
if (btn(⬅️)) x-=dx
if (btn(➡️)) x+=dx
end,
on_draw=function(_𝘦𝘯𝘷)
print("웃",x,y,7)
end
},{__index=_𝘦𝘯𝘷})
add(entities,ent)
end
-- elsewhere
add_entity(rnd(128),rnd(128),2) |
Took me 30 seconds after posting to maybe figure that one out myself. x, y, and dx are not inside each entity object?
However, that makes me even more puzzled. What is actually going on? Is x and y in the example above part of some private local scope?
what you observed is suspicious, I think there is something else going on.
when add_entity is called, x/y/dx are only local variables, there is no reason they would be auto-assigned to the entity.
on a related point, I wonder how calling a function such as this one behaves: on_draw=function(_ENV, a, b): would the local variables be set in self rather than the function scope?
hey @johanp glad you had the time to read it and it helped out you a bit... Let's see what goes on with your strange behaviour... and I'd say this matches the explanation of the CLOSURE UPVALUE SCOPE...
For that "entity" x,y and dx are closure-like scope variables and that scope is accessible when you reach out for "X", "Y" or "DX". So they are not "explicitly" set but they exist and can be accessed (precedence would be LOCAL -> EXTERNAL LOCAL UPVALUE -> _ENV). What I mean with they are not set but they exist is that you cannot access them with . operator in the table because they don't exist there Main issue with that is that any colliding var you want tu access through _ENV[<VAR>] will be shadowed if <VAR> exists already in the EXTERNAL LOCAL UPVALUE scope and will require explicit _ENV[<VAR>] accessors
Adding an example here illustrating it all:
entities={}
basex=1000
function add_entity(x,y,dx)
local basex=x
local basey=y
local ent=setmetatable({
name="entity",
basey=5000,
on_update=function(_𝘦𝘯𝘷)
if (btn(⬅️)) x-=dx
if (btn(➡️)) x+=dx
end,
on_draw=function(_𝘦𝘯𝘷)
? "name:"..e.name
? "basex:"..basex.. "|".._𝘦𝘯𝘷["basex"]
? "basey:"..basey.. "|".._𝘦𝘯𝘷["basey"]
? "x=" .. x
-- if uncommented, will crash
-- ? "e.x="..e.x
print("웃",x,y,7)
end
},{__index=_𝘦𝘯𝘷})
add(entities,ent)
return ent
end
-- elsewhere
e=add_entity(rnd(128),rnd(128),rnd(5))
function _draw()
cls()
for e in all(entities) do
e:on_draw()
end
end
function _update()
for e in all(entities) do
e:on_update()
end
end
|
- Global BASEX gets shadowed by EXTERNAL LOCAL UPVALUE BASEX in ADD_ENTITY function
- Local BASEY gets shadowed by EXTERNAL LOCAL UPVALUE BASEY in ADD_ENTITY function
- Global E holding our entity does not have a member X (E.X will make it crash, while f.e. E.NAME works just fine)
- LOCAL UPVALUE X,Y,DX work fine... exist but are not DEFINED in the actual table
I hope this clears things a bit for you and for @merwok... Scopes and Upvalues in LUA can be a bit tricky to master
No problem, took me a while to figure it out myself, happy to share the info and help out other ppl to figure it out ;) Considering your usual style, this can save you quite a few tokens @johanp, really looking forward what new miracle you can squeeze in with the extra space!
Added some extra info there... recently seen in conversations in the Discord help channel on how to use _ENV in functions.
I'm pretty new to Lua but have adopted an OOP/metatable/metamethod style in my explorations of Pico-8 and I'm kind of confused about using __index=_env in the setmetamethod(t,mt) function. Don't you lose the ability (or make it much more difficult) to supply inheritance by making that definition? Like then I couldn't have default values for an entity? Or am I missing something critical?
@Mushkrat you are right... this is a very basic approach to prevent loosing access to the global context, which is needed for just so many things, and it's normally "enough" for many use-cases. You can still provide full inheritance with a different approach:
function rootaccess(tbl,key) if getmetatable(tbl) then local v=getmetatable(tbl)[key] if (v~=nil) return v -- needed for bools end return _𝘦𝘯𝘷[key] end |
If you use this instead of _ENV you can have cascading checks on the metatable... I actually use this approach in my own code where everything falls down to a primitive OBJECT in cascade and using extensively __call metamethod.
@slainte That's pretty nifty. I'm studying your Aztec code right now and I can definitely see how doing the environment shifting thing with your rootaccess fallback gives you a lot of flexibility and would cut down a lot on token counts.
Aztec is in a "previous" iteration and given it was all done in a rush for a 2 weeks jam quite messy XD, I am currently working on a revision that handles almost everything with rootaccess, decorate and call... with some internal overrides of a custom new() constructor for children tables with full root fallback access to GLOBAL _ENV
function extend(parent,child)
return setmetatable(child or {},parent)
end
function noop() end
function empty() return {} end
function rootaccess(tbl,key)
if getmetatable(tbl) then
local v=getmetatable(tbl)[key]
if (v~=nil) return v -- needed for bools
end
return _ENV[key]
end
function decorate(tbl,base)
local mappers={
e=empty,
f=function(v) return _ENV[v] end,
c=function(v) return _ENV[v]() end,
b=function(v) return v=="true" and true or false end
}
for l in all(split(base,"#")) do
local list,val=unpack(split(l,"|"))
local tp,v=unpack(split(val,":",true))
for m in all(split(list)) do
if (tbl[m]==nil) tbl[m]=mappers[tp]==nil and v or mappers[tp](v)
end
end
return tbl
end
do
local id=0
function generator()
id+=1
return id
end
end
object=extend(
decorate({__call=function(...)
return extend(...)
end},"__name|s:root#__index|f:rootaccess"),
decorate({__call=function(_ENV,...)
__index=rootaccess
return decorate(extend(_ENV,decorate(_ENV:__new(...),"id|c:generator")),base)
end},"__name|s:object#__new|f:empty#__index|f:rootaccess")
)
entity=object(
decorate({__new=function(_ENV,proto)
local e=extend(_ENV,proto)
return e
end
},"__name|s:entity,__index|f:rootaccess#init,draw,update,frame|f:noop#x,y,z|i:0")
)
|
Quite obscure :P
I like it though. It's a really nice and flexible system for the sort of compositional approach you're taking with your games. I haven't looked at much of the older stuff but has this been the philosophy for your earlier projects as well?
It started far less complicated, but moved into this direction fast enough :P
[Please log in to post a comment]
