crawler(1)

NAME

crawler - recursively explore the Internet and copy URLs documents locally

SYNOPSIS

crawler [options] -- [url] ...

DESCRIPTION

crawler is used to maintain a copy of a given set of URLs up to date on the local machine.

First it loads the specified URLs (args ...) locally. It stores the content of the URL in the url_content database table table. Then it analyses their contents to find other URLs starting at the same base. For instance, if crawling http://www.senga.org/mifluz/ it will only retain the URLs that start with http://www.senga.org/mifluz/. It loads the URLs found until there are no more or until it reaches the depth limit (see the -depth option).

Note that the -- must be specified to mark the end of the options. Otherwise the arguments will be mistaken as options.

The list of URLs to explore is stored in the start table and the list of URLs crawled is stored in the url table.

Unique key fields url_md5 in the start and url tables are MD5 hash of the value in the url field. To calculate the MD5 key use something similar to the following: perl -MMD5 -e 'print MD5->hexhash("http://www.chez.com/ancienstagiaires/")'.

While running Webbase in a mode that stores the content of crawler pages in a database it is possible to encounter problems with large SQL packets being sent to MySQL.

When the defaul maximal alloed packet size is reached MySQL will drop the connection to the crawler and log this:

  010109 17:38:47  Aborted connection 1 to db: 'DDDD'
user: 'XXXX' host: `localhost' (Got a packet bigger
than 'max_allowed_packet')
  010109 17:38:47  Aborted connection 2 to db: 'DDDD'
user: 'XXXX' host: `localhost' (Got a packet bigger
than 'max_allowed_packet')

The default maximal allowed packet size is 1 MB. To increase the maximal allowed packet size one has to start mysqld with the following option: -O max_allowed_packet=XXX where XXX is the size of the maximal allowed packet in bytes. Alternatively, one could enter the option in the default

[mysqld]
set-variable=max_allowed_packet=10M

More about this option can be found at: http://www.mysql.com/documentation/mysql/bychapter/manual_toc.html#Gone_away

The maximal allowed packet size should be slightly larger (e.g. 1024 bytes) than the number passed with Webbase crawler's -webtools_limit option. The default Webbase limit is 100 KB.

WHAT IS A SERVER

The crawler manage a list of servers in the servers table. By default the server of a given URL is simply the host name and port number of the URL.

This can be customized using the home_re table. The netloc field contains a host name and port number (for instance www.foo.com or www.bar.com:8080) and the re field contains a regular expression that is applied to the entire URL. The regular expression must capture a string and this string is taken as the server name. For instance, if the home_re contains the record

netloc        re
www.chez.com  http://(www.chez.com/[^/]*)

when the crawler finds the URL http://www.chez.com/mypage/, it will know that the server name is www.chez.com/mypage instead of the default www.chez.com. This feature is specially usefull for servers that host a large number of web sites.

DATABASE OPTIONS

-base <base name>

name of the MySQL database to use for the meta data information.

-user <name>

Name of the user to connect to database.

-password <password>

Password of the -user to connect to database.

-port <port>

TCP/IP port to connect to database, if not default MySQL port.

-socket <file>

Unix socket file full path name for local database connection, if not default MySQL socket file.

-host <hostname>

Hostname to connect to database, if not localhost.

-create

Create all the tables. Exclusive, no other option will be taken in account. The database must already exist. After creating the tables one should typically populate the start table (at least the url and url_md5 fields). The url_md5 field may contain an arbitrary unique value provided the consistentc -keys -base mybase command is run to fix the url_md5 field afterward.

-schema

Print the complete database schema on the standard output.

PARALLEL CRAWLING

-crawlers

Act as a crawler manager and run children crawlers. The output of the children crawlers is redirected to crawler.log number. If the -log option is provided, use it instead of crawler.log. By default all the entries of the start table are crawled. If the -where_start option is specified, it will reduce the scope of entries taken in account according to the restrictions. The number of parallel crawlers is driven by the -crawlers_nproc option and the number of entries from the start table handled by each crawler is driven by the -crawler_chunk option.

-crawlers_chunk <number> (default 50)

Each crawler run when the option -crawlers is given will handle number entries from the start table.

-crawlers_nproc <number> (default 5)

The -crawlers action will run at most number child crawlers.

FULL TEXT INDEXING

-hook <libname>

Use the hook library <libname> to perform full text indexing. crawler will dynamically load the shared library lib<libname>.la at initialization and call its functions whenever a document is added, deleted or updated. The library hooksmifluz is installed with webbase and can be used for test purposes.

-rehook

Check and fix the concordance between the meta information database and the full text index.

-rebuild

Remove all the records from the full text database and resubmit all the URLs for indexing.

-rebuild_resume

Resume from an interrupted -rebuild. Only the entries of the url table with the field hookid == 0 will be indexed.

-where_url <where clause> (default none)

associated with -rebuild option, only rebuild URLs that match the <where clause> restriction.

MIFLUZ OPTIONS

Those are only available if webbase was compiled with the mifluz indexing library. Those options will override the builtin values.

-hook_cache_size <bytes> (default 10Mb)

Indexing cache size hint, in bytes. Must roughly be 2% of the expected index size. The bigger the better.

-hook_page_size <bytes> (default 8192).

The page size of the underlying index file.

-hook_compress {1,0} (default 1)

The index file is compressed if 1, not compressed if 0.

-index_config <path>

Use this configuration file for indexing instead of the builtin defaults.

-index_file <path>

Use this index file instead of the default /var/lib/webbase/index.

CRAWL OPTIONS

-home_pages

Load all the URLs listed in the start table.

-touch

re-crawl the URLs given in argument. Each URL is not a starting point but a URL found in the url table. You may combine -touch with -noheuristics to force crawling regardless of the time based heuristics. The -touch option can only be used to crawl a URL that is already in the url table and the hypertext links found in the crawled URL are not explored. This is very different from saying crawler -base mybase -- http://www.foo.com/page.html since this command will add the URL http://www.foo.com/page.html in the start table. When you say crawler -base mybase -touch -- http://www.foo.com/page.html the URL http://www.foo.com/page.html is not added to the start table. When a URL is crawled with -touch and -noheuristics, the If-Modified-Since HTTP field is not used. This may be useful to recover the content of a given URL, regardless of its last modified date stored in the database.

-where_touch <where clause>

re-crawl URLs matching the argument <where clause> in the url table, using the same algorithm as in the -touch option. The URLs given in argument will be ignored, they are only useful when the -touch is specified alone, without the -where_touch option. See the documentation of the -touch option for more information.

-update

Rebuild the URL graph, therefore validating changes in the start table attributes. For instance, one should use -update after changing the depth of exploration (-depth option) to actually crawl or remove URLs accordingly.

-where_start <where clause>

Only consider entries of the start table that match the <where clause> restriction.

-loaded_delay <days> (default 7)

do not try to load URLs loaded successfully less than <days> days ago.

-modified_delay <days> (default 14)

do not try to load URLs stated not modified (304) less than <days> days ago.

-not_found_delay <days> (default 30)

do not try to load URLs stated not found (404) less than <days> days ago.

-timeout_delay <days> (default 1)

do not try to load URLs for which a timeout (may be a real timeout or a connection failed, in short any error that is likely to disapear within a short delay) less than <days> days ago.

-timeout <sec> (default 60)

Set the TCP/IP timeout delay to <sec> seconds.

-robot_delay <sec> (default 60)

number of seconds between two requests to the same server when robot exclusion is observed.

-accept <mime types> (default text/*,magnus-internal/*)

only accept the specified mime types. A coma separated list of mime types specifications is allowed. The * may be used to specify any type or subtype.

-size_hrefs <bytes> (default 100k)

Maximum size of all the hypertext references stored for a given URLs. If the total size of the absolute and relative URLs exceed this limit, the remaining URLs will be ignored.

-webtools_limit <bytes> (default 100k)

Maximum length of the document read for a given URL. The crawler will not read more than this amount when loading a given URL.

-noheuristics

Schedule all URLs to crawl now instead of using the stored crawl date in the crawl field of the url table. In normal mode each URL crawl is scheduled according to the values of the -loaded_delay, -timeout_delay flags and the crawler will ignore them if their crawl date is in the future. This flag is the simplest way to trigger the crawl of all URLs. The If-Modified-Since HTTP header field is not used.

-allow <list> (default none)

The URLs for which the file path match the <list> specification will be crawled. This list has precedence over the -disallow argument. This list does not have precedence over the robots.txt file specifications. The <list> is a white space separated list of strings. For instance /dir/foo /ba room /kat/cat.html is a valid value. The strings that start with a slash (/) will only match at the beginning of the file path. For instance http://www.my.org/bar/my.html will match because of /ba but http://www.my.org/foo/bar/my.html will not match. Likewise http://bar.org/my.html will not match because /bar.org is not part of the file path of the url: /my.html in this case. Another example is : http://www.my.org/clean/up/room.html will match because of room since the room string does not start with a slash (/) it will match anywhere in the file path.

-disallow <list>

The URLs for which the file path match the <list> specification will not be crawled. This list does not have precedence over the -allow argument. This list does not have precedence over the robots.txt file specifications. See the -allow option documentation for an explanation of the <list> format.

-agent <name> (default webbase/5.13)

Set the User-Agent parameter for HTTP communication to <name> instead of the default value.

-depth <depth> (default 2)

set the exploration threshold. At most <depth> URLs will be crawled from a given starting point. The robots.txt file is counted as one URL.

-sticky

Do not give up on a server even if it timeouts. The normal behaviour is to mark all the URLs of a given server as timeouted if at least 5 URLs from this server ended with a timeout.

DEBUG OPTIONS

-version

Show crawler version.

-show

show the records of the start table matching the url arguments and the records of the url, url_content and url_complete tables that are linked to each of them. The output is driven by the value of the -show_fields option.

-show_fields <fields> (default start.*,url.*,url_complete.*,url_content.*)

Must be used in conjunction with -show. Only list the specified fields. All field names must be qualified with the table name.

-show_indexable

display the text sent to the full text indexer for each URL given in argument.

-verbose

Main program debug messages.

-verbose_webbase

Database manipulation related messages.

-verbose_webtools

Network connection related messages.

-verbose_hooks

Show full text indexer related messages.

-verbose_robots

Robots.txt loading and parsing related messages.

-verbose_cookies

Cookies handling related messages.

-verbose_crawl

Many messages from the crawl logic.

-babil_crawl

Many messages from the crawl logic. Huge output.

-verbose_dirsel

Messages from the url exclusion implementation. This includes handling of the -allow and -disallow options and of the robots.txt lines as they are applied to each URL seen.

EXAMPLES

crawler -base crawl -depth 1000 -- http://www.droit.umontreal.ca/

Mirrors in depth the Web starting at URLs http://www.droit.umontreal.ca/ until it finds 1000 documents or there is no documents left, using the 'crawl' MySQL database.

crawler -base crawl -home_pages

Mirrors web pages specified in the start table. Because the start table contains all crawling params about depth, level, etc. there is no need to specify these params here.

crawler -base crawl -depth 0 -- http://www.law.indiana.edu:80/law/lawindex.html

Simple copy of the lawindex.html file on our local machine. No recursive exploration is done.

SEE ALSO

uri(3), consistentc(1), mifluz(3)