mirrorbird (22:45:30)
well we don't have a robots.txt file at all, at the moment! (could be a very ancient bot that doesn't support 'nofollow' -- as that was introduced in later years)

nyingen (22:43:54)
Right. I would have thought that we'd already have that, but I will check. There is "nofollow" etc on the tag links

mirrorbird (22:27:48)
each tag page links to every other tag in a "narrowing down" way, so following all links would indeed hit all combinations.

mirrorbird (22:26:47)
very likely just a naively coded spider. i would put up a robots.txt excluding /demovibes/tags/ and /search/

nyingen (22:24:25)
*permutation

nyingen (22:24:05)
the crawler attack was hitting every combination of tags in the tag cloud

nyingen (22:23:42)
DB load, not HTTP server load, seemed to be the problem

zoi (21:52:22)
queue page is probably slow, too, but I guess better odds of hitting cache

zoi (21:51:29)
like new uploads, user favourites, songs by cyborg jeff or SIDs.

zoi (21:50:07)
seems those slow pages that list songs like to tie up the limited number of processes on the webserver

Czech, Slovakia and Back

Factor6

nyingen

Time Left: 4:37