Nectarine
Infamous OneLiner
mirrorbird
(22:26:47)
very likely just a naively coded spider. i would put up a robots.txt excluding /demovibes/tags/ and /search/
nyingen
(22:24:25)
*permutation
nyingen
(22:24:05)
the crawler attack was hitting every combination of tags in the tag cloud
nyingen
(22:23:42)
DB load, not HTTP server load, seemed to be the problem
zoi
(21:52:22)
queue page is probably slow, too, but I guess better odds of hitting cache
zoi
(21:51:29)
like new uploads, user favourites, songs by cyborg jeff or SIDs.
zoi
(21:50:07)
seems those slow pages that list songs like to tie up the limited number of processes on the webserver
mirrorbird
(21:48:39)
less efficient tho, they have to pull the pages to see it
mirrorbird
(21:48:11)
(ah just noticed a 'meta noindex' tag so maybe it's being done that way per page)
mirrorbird
(21:47:23)
no robots.txt file -- do we really want the (well-behaved) bots to grab every page? some like the last x hours of oneliner seem a bit pointless to put in search engines
