mirrorbird (22:45:30)
well we don't have a robots.txt file at all, at the moment! (could be a very ancient bot that doesn't support 'nofollow' -- as that was introduced in later years)

nyingen (22:43:54)
Right. I would have thought that we'd already have that, but I will check. There is "nofollow" etc on the tag links

mirrorbird (22:27:48)
each tag page links to every other tag in a "narrowing down" way, so following all links would indeed hit all combinations.

mirrorbird (22:26:47)
very likely just a naively coded spider. i would put up a robots.txt excluding /demovibes/tags/ and /search/

nyingen (22:24:25)
*permutation

nyingen (22:24:05)
the crawler attack was hitting every combination of tags in the tag cloud

nyingen (22:23:42)
DB load, not HTTP server load, seemed to be the problem

zoi (21:52:22)
queue page is probably slow, too, but I guess better odds of hitting cache

zoi (21:51:29)
like new uploads, user favourites, songs by cyborg jeff or SIDs.

zoi (21:50:07)
seems those slow pages that list songs like to tie up the limited number of processes on the webserver

Comic Bakery (larger than no remix)

DHS (Carlo Demichelis)

Press Play on Tape

strim

Time Left: 4:44