x,y,w,h=64-10.75,64-10.25,20.5,20.5
 rectfill(x,y,x+w-1,y+h-1,8)
 clip(x,y,w,h)
 rectfill(x,y,x+w-1,y+h-1,11)
 circ(64,64,12,12)

Note that on the top and left the circle extends to the edge of the rect, as is expected since the rect was drawn to match the impending clip region. Note the extra row on the bottom, inside the original rectfill, outside the clipping rectangle.

I think this is a bug. At the very least, it's a place where the documentation needs to be clarified.

(the circle being off center is irrelevant)

If I reduced your math properly, your rectfill() call becomes:

rectfill(53.25, 53.75, 72.75, 73.25)

However, every GPU-oriented API call only takes integers. Internally, those fractions get flr()'ed. So it's really just this:

rectfill(53, 53, 72, 73)

Similarly, if I reduced your clip() call correctly, it becomes:

clip(53.25, 53.75, 20.5, 20.5)

But again, the API only takes integers, so it's really just this:

clip(53, 53, 20, 20)

And thus you end up clipping pixels whose x or y values are <53 or >=73 (53+20=73). This doesn't match your rectangle call, where it wants to draw pixels at y=73.

I honestly think the lack of clarity here is that this is the one call in the API that takes x,y,w,h to describe a rectangle, rather than x1,y1,x2,y2 corner coordinates that describe the inside/inclusive limits of the rectangle.

If it makes life easier, you could use something like this to make it work more like other rectangle API calls, taking inclusive minimum and maximum valid values:

function cliprect(xmin,ymin,xmax,ymax)
  if x1 then
    clip(xmin,ymin,flr(xmax)-flr(xmin)+1,flr(ymax)-flr(ymin)+1)
  else
    clip()
  end
end

To really make it work like other rectangle calls, you might take arbitrary x1,y1,x2,y2 and sort them into xmin,ymin,xmax,ymax, but with cliprects it can often be useful to support having zero- or negative-area rectangles that allow nothing to be drawn, so I didn't do that here. My version puts the onus on the caller to provide parameters that are ordered properly.

And, hmm...

Here's a variation, with inclusive begin values but exclusive end values:

function cliprect(xb,yb,xe,ye)
  if xb then
    clip(xb,yb,flr(xe)-flr(xb),flr(ye)-flr(yb))
  else
    clip()
  end
end

This one is interesting because, based on the exposed pseudo-GPU registers, this call takes arguments in the same format the GPU actually uses, which would allow you to pass (modified) values directly in after peek()ing them from the current exposed GPU state.

...

MORE INFO THAN YOU EVER WANTED, I KNOW. IT'S HOW I DO, OKAY? 😅