My ball in pong has a different speed for y and x but this makes the ball super jittery and I'm not sure why. here is some of my code:
function game_init()
x=0
y=8*8
bx=0
by=0
dx=0.5
dy=1
end

function gane_draw()
cls(8)
spr(1,0,y)
spr(1,0,y-8)
spr(2,815,88)
spr(2,815,87)
spr(3,bx,by)
end

function gane_update()

--ball move
by+=dy
bx+=dx

--player input
if btn(⬇️) and y<120then
y+=1
end
if btn(⬆️) and y>8then
y-=1
end
end

(With the minor caution that my information on this is mainly 'absorbed theory', so someone else might have other information to impart, better sites to link to, or other solutions.)

The problem

As far as I know this is because Pico-8, to display the ball (or any sprite) at pixel coordinates floors (discards, rounds down) the fractional value, and because the pixel dimensions of the Pico-8 screen aren't very large, and because we're often running it on an enlarged screen which magnifies this, it's noticeable.

Note that which pixel(s) to set/round-to is a problem not limited to Pico-8, but one that all computers have to decide how to deal with.

Related keywords - if you want to do a web search - are jaggies, staircase effect, staircasing, or aliasing. But a lot of the sites I find look largely irrelevant to what you'll want to do in Pico-8.

Possible improvements

• Increasing the framerate by using _update60() rather than _update() so as to fool the eye with a faster framerate
• Using colours for the ball (or ball outline) and the background which don't make it so noticeable
• Drawing a ball of intermediary colour at slightly different coordinates (we'll put this under the heading 'anti-aliasing')

With any of these, judge the results for yourself after trying them out.

_update60()

Using this gameloop function in place of update() increases the framerate from 30 to 60, but the tradeoff is that "half the PICO-8 CPU is available per frame before dropping down to 30fps" and "not all host machines are capable of running at 60fps. Older machines, and / or web versions might also request PICO-8 to run at 30 fps (or 15 fps), even when the PICO-8 CPU is not over capacity."

Quotes from the Pico-8 manual.

My tablet web player handles _update60 badly - it seems to buffer my key presses, then when I change direction it continues in the direction I was pressing for a couple of seconds, making games with _update60 unplayable on it. But if it's for playing on your local machine, or online with a better-than-a-tablet device _update60 might help.

Using a ball colour that makes it less noticeable given the background colour

With the red background you're using, an orange outline for the ball might help blur things a little.

Anti-aliasing

We aren't talking proper sampling here (see link below) or calculating the intensity of the pixel. Pico-8 only has a limited range of colours to work with (about 32). As you know your background colour and your ball colour, you can just choose a colour in between the two (if there is one).

As a quick way of doing this you might try putting a ball sprite the colour of which is somewhere in between the background colour and your ball colour at slightly altered coordinates, before putting your normal coloured ball on top of it. I think you should only need this - presuming constant speeds of the ball from your code - when the ball's x coordinate is halfway between two pixels. Putting this second ball at a ceil(bx) value might be enough, but you can always experiment a little with tweaking the values of the coordinates it displays at - for example, perhaps the previous values of bx and by are better for it.

If necessary the 16 colour palette can be altered, see the user manual. There are an extra 16 colours which are numbered 128 to 143. If you want to use any of the extra 16 colours, you can send a third argument to the pal command, with the value 1, which will alter the screen palette. Experiment a little with swapping colours to see what the extra colours are like and if they would help. With a red background, probably 136 and 137 (the alternative red and orange colours) might be useful.

makeuseof.com What is Anti-Aliasing...