Nectarine
Infamous OneLiner
mirrorbird
(22:45:30)
well we don't have a robots.txt file at all, at the moment! (could be a very ancient bot that doesn't support 'nofollow' -- as that was introduced in later years)
nyingen
(22:43:54)
Right. I would have thought that we'd already have that, but I will check. There is "nofollow" etc on the tag links
mirrorbird
(22:27:48)
each tag page links to every other tag in a "narrowing down" way, so following all links would indeed hit all combinations.
mirrorbird
(22:26:47)
very likely just a naively coded spider. i would put up a robots.txt excluding /demovibes/tags/ and /search/
nyingen
(22:24:25)
*permutation
nyingen
(22:24:05)
the crawler attack was hitting every combination of tags in the tag cloud
nyingen
(22:23:42)
DB load, not HTTP server load, seemed to be the problem
zoi
(21:52:22)
queue page is probably slow, too, but I guess better odds of hitting cache
zoi
(21:51:29)
like new uploads, user favourites, songs by cyborg jeff or SIDs.
zoi
(21:50:07)
seems those slow pages that list songs like to tie up the limited number of processes on the webserver
