Log In  


-32768 and 1 are both perfectly representable in the 16.16 fixed format, as 0x8000.0000 and 0x0001.0000. As far as I can see, this division should be perfect with no floating point shenanigans.

However, 0x8000.0000/0x0001.0000==0x8000.0001

which is throwing a wrench into some code I'm writing.

Other nearby operations work fine, such as

0x8000/2

and

0x7fff.ffff/1

and

0x8000.0001/1

Then there's another group of operations that seem to be technically misbehaving but in a possibly desirable way.

0x4000/0x0.8 0x2000/0x0.4 ... 0x1/0x0.0001

should technically all overflow to 0x8000 (-32768) but they actually evaluate to 0x7fff.ffff which renders as 32768 because it's higher than 32767.9999. Maybe this is intentional, and maybe it's helpful, but I mention it here because it might also be related.

PS: there are markdown rendering bugs affecting this post which is why it's weirdly formatted

1


Tried it myself:

That's definitely a low-level bug and it's the sort of thing that'll trip me up eventually too, because I tend to work with wonky edge cases a lot, like the outpost (0x8000) value. Hope @zep sees this.


1

Happy to see this one's fixed in 0.2.0c! :)



[Please log in to post a comment]