so i'm starting to get near the often mentioned but never explained token limit in my project. and while I think I might fit everything i want into this project, I think I need to keep tokens in mind from the get go of my next project.

so I have some questions:

what is a token? what counts towards the token limit? can i look at my existing code and try to optimize it with token limit in mind? and also do you have any tips & tricks while developing with this in mind?

each word is a token a pair of parens,brackets,braces count as one token, a string is one token, punctuation is generally one token. "local" does not count as a token. "end" does not count as a token

tip 1: don't repeat yourself (there are some exceptions)

find similar code you use frequently and make a function and use that instead, the more you use it the better the return.

tip 2: if you use lots of attributes, it might be worth making a local first to save tokens:

foo = a.x + b.x + c.x + d.x
bar = a.x + b.x + c.x + d.x
baz = as.x + b.x + c.x + d.x

= 39 tokens

local ax,bx,cx = a.x,b.x,c.x
foo = ax + bx + cx + dx
bar = ax + bx + cx + dx
baz = ax + bx + cx + dx

= 37 tokens

tip 3: (advanced) use operator overloading

if you use a lot of vectors you can save a heap of tokens by making a vector class with operator overloading

so instead of

pos.x += vel.x
pos.y += vel.y

you can do

vec = {}
vec.__index = vec
function vec:__add(b)
   return vec.new(self.x+b.x,self.y+b.y)
end
-- any other vector functions you need
function vec.new(x,y)
    local v = {x=x or 0,y=y or 0}
    setmetatable(v,vec)
    return v
end

pos += vel

There's a really good write up in one of the Pico 8 fanzines about tokens (the first one I think?).

Basically a token is one programming element in Pico 8 LUA. For example, keywords like "FOR" "FUNCTION" "IF" "END" all constitute one token each. Variable names you create are all one token, no matter how long they are. Operators like "+", "=", are all one token. Operators written with multiple characters like "*=" count as a single token too.

The easiest way to learn about them is to write code while paying attention to the token counter in the bottom right of the code editor.

thanks for the input!
@impbox your tips will be of great help, especially the first one!
@mrh i write all my code in atom. i should check to see if there is any way to display token count in it :)

also this reminds me i should read the fanzines...

Like the other replies said, a token is basically "something that the text parser considers functional". Think of a token as a delimiter or an instruction. One line of code such as "playerPos = { x=10, y=20 }" could consist of about 10 tokens or so.

Tokens are a feature of Lua, not Pico8. Remember that by design, you're limited to 8192 tokens, which must all be under 65536 characters in length total.

Speaking of operator overloading... does anyone know if Lua does return value optimization?

Where RVO would be something like taking this:

function x() return {x=0} end

v = x()

And, under the hood, producing code that actually works like this:

function x(rv) rv.x=0 end

x(v)

The goal being to avoid constructing temporaries and moving them around.