Possible bug / user error / 'feature' of rounding(?)...

cls()
print("0x7b18.3060")
print(tostr(0x7b18.3060))
print(tostr(31512.189,true))
print(tostr(31512.188965,true))

Why does PICO-8 return the s16.16 value for the hexadecimal with too few digits?
i.e. the conversion with tostr() hex>dec>hex is lossy.

the misconception here is that there is a conversion between hexa and decimal :-]

pico-8 only has one number type: 16.16 fixed-point numbers.
integers are numbers with 0 as decimal part*
when we write litterals, 0b1101 or 0xd or 13 are not different types or values, but equivalent ways of writing the exact same number. there is no conversion happening.

* or there’s a «high score» mode to interpret all 32 bits as one integer: https://www.lexaloffle.com/dl/docs/pico-8_manual.html#TOSTR

Thanks @merwork.

Let me share the actual issue. I'm creating a custom font, poking 4 bytes (x2) as hexadecimal:

poke4(
 22816,
 (0x7b18.3060),
 (0x487e.fcf9)
)
print"⁶@56000003⁸x⁸ᵉd"

That works as expected.

Now I want to change those hex values to decimal, so they are more compact. To save typing out the decimals, let's do it inline:

poke4(
 22816,
 tostr(0x7b18.3060,2),
 tostr(0x487e.fcf9,2)
)
print"⁶@56000003⁸x⁸ᵉd"

As you can see, this gives a different image... but it should not vary whether we use the hex or decimal ...?

I believe transforming the 4 byte representation hexadecimal --> decimal --> hexadecimal (where all values are stored with the same underlying s15.16 fixed point), should be lossless. ...? Right?..

e.g.

cls()
a=0x7b18.3060
print(a.." = "..tostr(a,1))
b=31512.189 -- Given by tostr(a,1)
print(b.." = "..tostr(b,1))
print(".3060 != .3062")

I appreciate at some stage precision is lost due to rounding the fractional part to 5 digits... is this the underlying limitation? If so, It will always be risky to use decimal numbers with poke4...

you can’t pass strings to poke functions! (and you shouldn’t interpret a number as 32-bit integer, but that’s irrelevant here as the tostr mode is meant for display, not processing)

replace 0x7b18.3060 with 31512.198 in your code and that will work.

@merwok - Try:

print(
 (tostr(31512.198,1))-- Gives '.32B0 not '.3060...
)

So the last 2 digits are still wrong...

That should be .189, not .198. Regardless that's still going to be slightly off since the tostr function rounds it—0x7b18.3060 in decimal exactly would be 31512.18896484375. If you do for some reason need a decimal that can be displayed by tostr as 0x7b18.3060 though you could round to 31512.18897.

I would not recommend actually doing that though. Even e.g. for tweetcarts, it doesn't help save characters at all in this case and just makes it harder to read.

Hi. After a deep dive (some lost hours!) and help from @freds72 & @merwok...

Warning: P8 may round values to 4 d.p. This can cause issues with precision loss when working with fixed point numbers.
It is worth experimenting with P8's native number format s15.16 using this online calculator.

Enhancement request @zep:
(1) This affects poke4 (not poke/poke2) as we are trying to cram data into all 32 bits. Please update Manual/Docs to advise (warn!) users, e.g. "It is best to use the hexadecimal representation of values with poke4, as loss of precision with decimal fractions may lead to unexpected data.", etc.

(2)

Many thanks,
Andrew

Update 1-Aug-2023: I'm investigating if a trivial fix with handling Q numbers would help too. If these values were displayed with 6 d.p. (truncated not rounded), those 2 extra digits should be enough to avoid these issues.

@aced

I agree that the default decimal format should print more than 4 places, but 6 is more than is needed.

With the raw storage of the fractional portion having 65,536 possible values, you only need 5 decimal digits to have enough resolution to represent every possible unique fraction, since 5 digits gets you 100,000 gradations between integers, which should be more than enough for a unique decimal representation of the 65,636 gradations that can actually be stored.

It just requires taking care with the conversion code to make sure that converting a number from binary to decimal results in a string that will convert back to binary correctly, which may require different rounding policies in each direction to prevent precision loss.

If it were me, I'd just be lazy and have two tables for converting fractions, where I'd made sure there was a 1:1 association between the two.

Thanks @Felice for looking at this.
With 16bits the smallest fraction that can be represented is 2^(-16) which equals 0.0000152587890625. I still believe 6 decimal places is the absolute minimum- and truncation rather than rounding should be tested.

I worry any look up tables may be prohibitively large, with 65,536 combinations of fractions -vs- a simpler numerical solution. Also, my issue came when calculating the values with external tools- that didn't match up with p8 internals. The solution should be compatible with how fixed point numbers are calculated generally, to accept data from other sources.

just express ‘other sources’ as hexadecimal values and you are good

Realistically, @freds72 is right. Like, if someone wants the most precise version of pi they can represent on PICO-8, they really ought to convert it to signed 16.16 themself and write it with explicit bits in a power of 2 base literal like hex, not bits implied by a literal in the base we humans like but computers don't actually speak.

Like, I always found it a little cringey that C/C++ math libraries specify everything in decimal, knowing full well the compiler and processor will not use decimal values under the hood. I think fractional hex literals are actually available these days, but for a long, lonnnng time they weren't and it bugged me that the alternative was to over-specify the number in decimal in hopes that the compiler would round/resolve the literal the way you hoped.

-- we *know* this is the exact value a PICO-8 program should use for pi
pi_from_hex = 0x3.243f   -- rounded down from 0x3.243f6a88...

-- we can only *hope* this will be the same value
pi_from_hex = 3.14159

(Not to mention that decimal literals slow down compilation, which seems kinda petty of me, but if you have a bazillion headers all full of decimal literals instead of hex, it'll actually matter a bit. But that's getting pretty far off-topic.)