Anyone know why the site was down for the past 7ish hours? maintenance? did we get hacked?

EDIT: DOWN AGAIN but it seems were back?

Ive also noticed its impossible to create posts/comments. strange. at least edits work?

@zep do you know whats going on? are we getting ddosed?

it also seems that times are incorrect. i am editing this at 10:09 GMT-7 , and the edit date is wrong.

edit 2: seems posts are back? someone posted abt their movement system, so i guess posts are back. comments still dont work. very odd.

edit 3: i guess posts are gone again. maybe its a me issue. someone try posting.

also ive noticed that if you try to post and cant, changing "pid=" in the search bar to "pid= any number" will, if there is a corresponding post, let you view that post as if you were editing posts. even drafts.

edit 4: ive still been getting update notifications from the automated lexaloffle thing, but the links are wrong. maybe an api is down or something broke somewhere.

edit 5: another short outage? hmm. im beginning to see a pattern.

edit 6: i dont know if this is new, but there is now a "#m" at the end of every lexaloffle url. huh.

edit 7: i guess now its just for some pages.

test

edit 8: seems like some people can still post/comment. i dont know why. maybe @zep does. i dont know if its daytime in japan (or wherever zep/the website developer is) right now, or if they are even awake, but my guess is it'll hopefully be fixed tomorrow. i have to go to sleep because its midnight where i am. gn everyone. lets hope its not serious and its simply server shenanigans and not a targeted attack.

Hey, thanks for the heads up @AntiBrain. Your notes were useful while I was debugging this.

The server was down intermittently between 19:00~23:00UTC (aug 17) and also 2:00~5:00UTC (aug 18). Nothing malicious, but I've temporarily blocked a few high traffic ips (overly aggressive scrapers?) while I figure it out. Unfortunately the unusual access patterns were causing part of the cache to flood the disk, in turn causing (among other things) one of the database files to become corrupted, breaking new posts & comments.

I was able to remedy this by turning off the cache (I'll improve that soon), and using built-in repair tools on the database to reindex it. That seems to have done the trick, but I'll be keeping an eye out for anomalies the next couple of days.

After comparing with backup snapshots, it looks like we didn't lose any previously posted data, but apologies to anyone who was trying to comment / post during that time and had their draft obliterated.

I'm glad I could help! I hope everything goes well over there! Thank you for the update!

Hey @zep! I've been contemplating @ing you here, but this is relativly important.

Some scrapers called "crawlers" we're found to be overstressing many a server near the time of the outages here.

This may sound crazy, but TickTok, more specifically their parent company ByteDance, have been sending out their own crawlers called "bytespider".

ByteDance did not limit crawl speeds, and accidently flooded and corrupted and sometimes crashed many small servers.

If you have any logs of the event, check the user agents of a few of those scrapers.

Regardless of if you find anything or not, I suggest redirecting all ByteSpider user agents to a 403 page.

If you need more info, Matt KC made a video about the situation here

Edit:

There are a ton of red flags with the bytespider bot. (Like ignoring robots.txt, bypassing blacklists and more)
Here is a list of odd things it does. I really do recommend dealing with this as soon as possible, because when it's too late, you might lose the server.

https://wordpress.org/support/topic/psa-bytedance-and-bytespider-bots-recommend-blocking/

Thanks for the info @AntiBrain -- it turns out they are not TikTok related (based on useragent, ip addresses), but are in a similar category in terms of annoying access patterns.

It seems some of them are spamming the search functionality to find for targeted threads to (attempt to) dump automated spam to.

A lot of such traffic got through recently (around 3x the usual bot activity), but fortunately it didn't have much impact on the server this time because of earlier mitigations (basically: stricter rate limiting & a safer caching scheme).

A side-effect is that visitors who are not logged in can't search more than ~20 times every 5 minutes, but I should be able to relax that once things calm down.

Well, I'm glad it's not dire and only annoying! Once again thank you for the update! Hope everything goes well over there.