mirrorbird (22:27:48)
each tag page links to every other tag in a "narrowing down" way, so following all links would indeed hit all combinations.

mirrorbird (22:26:47)
very likely just a naively coded spider. i would put up a robots.txt excluding /demovibes/tags/ and /search/

nyingen (22:24:25)
*permutation

nyingen (22:24:05)
the crawler attack was hitting every combination of tags in the tag cloud

nyingen (22:23:42)
DB load, not HTTP server load, seemed to be the problem

zoi (21:52:22)
queue page is probably slow, too, but I guess better odds of hitting cache

zoi (21:51:29)
like new uploads, user favourites, songs by cyborg jeff or SIDs.

zoi (21:50:07)
seems those slow pages that list songs like to tie up the limited number of processes on the webserver

mirrorbird (21:48:39)
less efficient tho, they have to pull the pages to see it

mirrorbird (21:48:11)
(ah just noticed a 'meta noindex' tag so maybe it's being done that way per page)

Aisyuu Breakthrough

Maak (Akira Shiozawa)

MedioLitro

Time Left: 2:38