Method Web.Crawler.Crawler()->create()
- Method create
Web.Crawler.Crawler Web.Crawler.Crawler(Queue _queue, function(:void) _page_cb, function(:void) _error_cb, function(:void) _done_cb, function(:void) _prepare_cb, string|array(string)|Standards.URI|array(Standards.URI) start_uri, mixed ... _args)
- Parameter _page_cb
function called when a page is retreived. Arguments are: Standards.URI uri, mixed data, mapping headers, mixed ... args. should return an array containing additional links found within data that will be analyzed for insertion into the crawler queue (assuming they are allowed by the allow/deny rulesets.
- Parameter _error_cb
function called when an error is received from a server. Arguments are: Standards.URI real_uri, int status_code, mapping headers, mixed ... args. Returns void.
- Parameter _done_cb
function called when crawl is complete. Accepts mixed ... args and returns void.
- Parameter _prepare_cb
argument called before a uri is retrieved. may be used to alter the request. Argument is Standards.URI uri. Returns array with element 0 of Standards.URI uri, element 1 is a header mapping for the outgoing request.
- Parameter start_uri
location to start the crawl from.
- Parameter _args
optional arguments sent as the last argument to the callback functions.