I've got this bunch of enemies that run after the player, but i can't seem to figure how to make them collide each other since they all are in a same table.
I've managed to do it with two for loops that calls them differently, but this makes a game overload after there's more then 40 enemies spawned.
Maybe somebody could give me a hint on this one.

Here's a snippet I worked out for an (as yet incomplete) project. Within your loop handling all enemies, add this inner loop so each enemy will push against each other enemy.

--push away from other enemies
for oen in all(ens) do
 if (oen != en) then
  local dx,dy,d=oen.x-en.x,oen.y-en.y,4
  if abs(dx)<d and abs(dy)<d and dx/d*dx/d+dy/d*dy/d<1 then
   local a=atan2(dx,dy)
   en.dx-=en.spd*cos(a)
   en.dy-=en.spd*sin(a)
  end
 end
end

ens is our table of enemies, en is the enemy we are currently working on, and oens are the other enemies. Our enemy objects must have at least properties x, y, spd, (speed) dx and dy. This code makes use of @Mot's nice little distance check.

This depends on your using deltas, a "dx" and "dy" by which the enemy will be moved at the conclusion of movement calcs. Rather than moving the enemies away from each other directly as it runs, this will influence the deltas that should be applied at the end of the outer loop. Let me know if you have questions about how that works.

I don't know how performant it will be for 40+ enemies but it seemed to run pretty well for dozens in my environment.

You saved me again!!!) thank you so much for sharing this, it worked great and looks pretty cool))
It starts overloading the game after 100 enemies which is much better then 40)

Didn't figure how it works yet, but i'm going to dig in. Amazing information! thank you @kozm0naut🙏🙏🙏

I'm glad it works for your game! :)

It works by taking the difference in x and y (our local dx,dy) between the current enemy en and the current other enemy oen and using a distance d of 4 (I forgot to mention as it's been a while since I used this, this is the push radius, or how close in px they can get before they start pushing. Change this as needed or you can make it dynamic by setting it to say, half enemy width).

Then on the next line we very quickly do a "Manhattan" dist check (if it's within dist d on both axes), if both are true it proceeds to Mot's distance check which is sort of a twist on the pythagorean theorem that doesn't use sqrt but only checks if a hypotenuse is shorter or longer than a given distance. If the Manhattan dist checks fail on either axis it won't even proceed to this part of the condition so we get to quickly skip anything simply too far away.

Once we know we are operating on something truly within distance, we can find the angle between the two enemies by using atan2(dx,dy) on the distances we took earlier. Using this angle, we impel the current enemy in the opposite direction (-=) on both axes, using cos(a) for x and sin(a) for y to get our ratio of how hard to push on each axis, and multiplying that by the enemy's speed to get the final forces to apply.

...

This can almost certainly be done more efficiently by simply checking for overlap in each enemy's bounding box and pushing them back in whichever axis has a smaller overlap. It could probably handle more enemies that way, they wouldn't quite push away in the "right" direction every time, but it would still look decent.

Thank you so much for this great explanation @kozm0naut!
I think i actually get it now.
Now i'm wondering how to do it like you say by the end. Actually if, by checking overlap you mean a usual collision function, i was trying it before your method and it overloaded on much less enemies for some reason. I guess it happens because of this second for loop.