---
name: "crawler"
section: "1"
platform: "linux"
package: "webbase"
version: "5.17.0-14"
release: "Debian 3.0 (Woody)"
platforms: ["linux"]
url: "https://new.carta.tech/crawler"
---

# crawler(1)

recursively explore the Internet and copy URLs documents locally

## NAME

crawler -  recursively explore the Internet and copy URLs documents locally

## SYNOPSIS

crawler \[options\] -- \[url\] ...

## DESCRIPTION

**crawler** is used to maintain a copy of a given set of URLs up to date on the local machine.

First it loads the specified URLs (args ...) locally. It stores the content of the URL in the  **url\_content** database table table. Then it analyses their contents to find other URLs starting at the same base. For instance, if crawling http://www.senga.org/mifluz/ it will only retain the URLs that start with http://www.senga.org/mifluz/.  It loads the URLs found until there are no more or until it reaches the depth limit (see the *-depth* option).

Note that the  **--** must be specified to mark the end of the options. Otherwise the arguments will be mistaken as options.

The list of URLs to explore is stored in the  **start** table and the list of URLs crawled is stored in the  **url** table.

Unique key fields  **url\_md5** in the start and url tables are MD5 hash of the value in the url field. To calculate the MD5 key use something similar to the following: perl -MMD5 -e 'print MD5-\>hexhash("http://www.chez.com/ancienstagiaires/")'.

While running Webbase in a mode that stores the content of crawler pages in a database it is possible to encounter problems with large SQL packets being sent to MySQL.

When the defaul maximal alloed packet size is reached MySQL will drop the connection to the crawler and log this:

```
  010109 17:38:47  Aborted connection 1 to db: 'DDDD'
user: 'XXXX' host: `localhost' (Got a packet bigger
than 'max_allowed_packet')
  010109 17:38:47  Aborted connection 2 to db: 'DDDD'
user: 'XXXX' host: `localhost' (Got a packet bigger
than 'max_allowed_packet')
```

The default maximal allowed packet size is 1 MB. To increase the maximal allowed packet size one has to start mysqld with the following option: -O max\_allowed\_packet=XXX where XXX is the size of the maximal allowed packet in bytes. Alternatively, one could enter the option in the default

```
[mysqld]
set-variable=max_allowed_packet=10M
```

More about this option can be found at: http://www.mysql.com/documentation/mysql/bychapter/manual\_toc.html#Gone\_away

The maximal allowed packet size should be slightly larger (e.g. 1024 bytes) than the number passed with Webbase crawler's -webtools\_limit option. The default Webbase limit is 100 KB.

## WHAT IS A SERVER

The crawler manage a list of servers in the  **servers** table. By default the server of a given URL is simply the host name  and port number of the URL.

This can be customized using the  **home\_re** table. The  **netloc** field contains a host name and port number (for instance www.foo.com or www.bar.com:8080) and the **re** field contains a regular expression that is applied to the entire URL. The regular expression must capture a string and this string is taken as the server name. For instance, if the home\_re contains the record

```
netloc        re
www.chez.com  http://(www.chez.com/[^/]*)
```

when the crawler finds the URL http://www.chez.com/mypage/, it will know that the server name is www.chez.com/mypage instead of the default www.chez.com. This feature is specially usefull for servers that host a large number of web sites.

## DATABASE OPTIONS

- **-base \<base name\>**

  name of the  **MySQL** database to use for the meta data information.

- **-user \<name\>**

  Name of the user to connect to database.

- **-password \<password\>**

  Password of the -user to connect to database.

- **-port \<port\>**

  TCP/IP port to connect to database, if not default  **MySQL** port.

- **-socket \<file\>**

  Unix socket file full path name for local database connection, if not default **MySQL** socket file.

- **-host \<hostname\>**

  Hostname to connect to database, if not  **localhost.**

- **-create**

  Create all the tables. Exclusive, no other option will be taken in account. The database must already exist. After creating the tables one should typically populate the  **start** table (at least the  *url* and  *url\_md5* fields). The  *url\_md5* field may contain an arbitrary unique value provided the  **consistentc -keys -base mybase** command is run to fix the  **url\_md5** field afterward.

- **-schema**

  Print the complete database schema on the standard output.

## PARALLEL CRAWLING

- **-crawlers**

  Act as a crawler manager and run children crawlers. The output of the children crawlers is redirected to crawler.log *number.* If the **-log** option is provided, use it instead of  *crawler.log.* By default all the entries of the start table are crawled. If the **-where\_start** option is specified, it will reduce the scope of entries taken in account according to the restrictions. The number of parallel crawlers is driven by the  **-crawlers\_nproc** option and the number of entries from the  **start** table handled by each crawler is driven by the  **-crawler\_chunk** option.

- **-crawlers\_chunk \<number\> (default 50)**

  Each crawler run when the option **-crawlers** is given will handle  *number* entries from the  **start** table.

- **-crawlers\_nproc \<number\> (default 5)**

  The  **-crawlers** action will run at most  *number* child crawlers.

## FULL TEXT INDEXING

- **-hook \<libname\>**

  Use the hook library  *\<libname\>* to perform full text indexing.  **crawler** will dynamically load the shared library  *lib\<libname\>.la* at initialization and call its functions whenever a document is added, deleted or updated. The library  **hooksmifluz** is installed with  **webbase** and can be used for test purposes.

- **-rehook**

  Check and fix the concordance between the meta information database and the full text index.

- **-rebuild**

  Remove all the records from the full text database and resubmit all the URLs for indexing.

- **-rebuild\_resume**

  Resume from an interrupted -rebuild. Only the entries of the url table with the field **hookid** == 0 will be indexed.

- **-where\_url \<where clause\> (default none)**

  associated with **-rebuild** option, only rebuild URLs that match the **\<where clause\>** restriction.

## MIFLUZ OPTIONS

Those are only available if webbase was compiled with the mifluz indexing library. Those options will override the builtin values.

- **-hook\_cache\_size \<bytes\> (default 10Mb)**

  Indexing cache size hint, in bytes. Must roughly be 2% of the expected index size. The bigger the better.

- **-hook\_page\_size \<bytes\> (default 8192).**

  The page size of the underlying index file.

- **-hook\_compress {1,0} (default 1)**

  The index file is compressed if 1, not compressed if 0.

- **-index\_config \<path\>**

  Use this configuration file for indexing instead of the builtin defaults.

- **-index\_file \<path\>**

  Use this index file instead of the default /var/lib/webbase/index.

## CRAWL OPTIONS

- **-home\_pages**

  Load all the URLs listed in the  **start** table.

- **-touch**

  re-crawl the URLs given in argument. Each URL is not a starting point but a URL found in the  **url** table. You may combine  **-touch** with **-noheuristics** to force crawling regardless of the time based heuristics. The  **-touch** option can only be used to crawl a URL that is already  in the  **url** table and the hypertext links found in the crawled URL are not explored. This is very different from saying  *crawler -base mybase -- http://www.foo.com/page.html* since this command will add the URL *http://www.foo.com/page.html* in the  **start** table. When you say *crawler -base mybase -touch -- http://www.foo.com/page.html* the URL *http://www.foo.com/page.html* is not added to the start table. When a URL is crawled with **-touch** and  **-noheuristics,** the  *If-Modified-Since* HTTP field is not used. This may be useful to recover the content of  a given URL, regardless of its last modified date stored in the  database.

- **-where\_touch \<where clause\>**

  re-crawl URLs matching the argument  **\<where clause\>** in the url table, using the same algorithm as in the **-touch** option. The URLs given in argument will be ignored, they are only useful when the  **-touch** is specified alone, without the  **-where\_touch** option.  See the documentation of the  **-touch** option for more information.

- **-update**

  Rebuild the URL graph, therefore validating changes in the start table attributes. For instance, one should use -update after changing the depth of exploration (-depth option) to actually crawl or remove URLs accordingly.

- **-where\_start \<where clause\>**

  Only consider entries of the **start** table that match the \<where clause\> restriction.

- **-loaded\_delay \<days\> (default 7)**

  do not try to load *URLs* loaded successfully less than  **\<days\>** days ago.

- **-modified\_delay \<days\> (default 14)**

  do not try to load *URLs* stated not modified (304) less than  **\<days\>** days ago.

- **-not\_found\_delay \<days\> (default 30)**

  do not try to load *URLs* stated not found (404) less than  **\<days\>** days ago.

- **-timeout\_delay \<days\> (default 1)**

  do not try to load *URLs* for which a timeout (may be a real timeout or a connection failed, in short any error that is likely to disapear within a short delay) less than  **\<days\>** days ago.

- **-timeout \<sec\> (default 60)**

  Set the TCP/IP timeout delay to \<sec\> seconds.

- **-robot\_delay \<sec\> (default 60)**

  number of seconds between two requests to the same server when robot exclusion is observed.

- **-accept \<mime types\> (default text/\*,magnus-internal/\*)**

  only accept the specified mime types. A coma separated list  of mime types specifications is allowed. The  *\** may be used to specify any type or subtype.

- **-size\_hrefs \<bytes\> (default 100k)**

  Maximum size of all the hypertext references stored for a given URLs. If the total size of the absolute and relative URLs exceed this limit, the remaining URLs will be ignored.

- **-webtools\_limit \<bytes\> (default 100k)**

  Maximum length of the document read for a given URL. The crawler will not read more than this amount when loading a given URL.

- **-noheuristics**

  Schedule all URLs to crawl now instead of using the stored crawl date in the *crawl* field of the url table. In normal mode each URL crawl is scheduled according to the values of the -loaded\_delay, -timeout\_delay flags and the crawler will ignore them if their crawl date is in the future. This flag is the simplest way to trigger the crawl of all URLs. The **If-Modified-Since** HTTP header field is not used.

- **-allow \<list\> (default none)**

  The URLs for which the file path  match the \<list\> specification  will be crawled.  This list has precedence over the  **-disallow** argument. This list does not have precedence over the  **robots.txt** file specifications. The \<list\> is a white space separated list of strings. For  instance  */dir/foo /ba room /kat/cat.html* is a valid value. The strings that start with a slash (/) will only match at the beginning of the file path. For instance  *http://www.my.org/bar/my.html* will match because of */ba* but *http://www.my.org/foo/bar/my.html* will not match. Likewise *http://bar.org/my.html* will not match because  */bar.org* is not part of the file path of the url: /my.html in this case. Another example is : *http://www.my.org/clean/up/room.html* will match because of *room* since the  *room* string does not start with a slash (/) it will match anywhere in the file path.

- **-disallow \<list\>**

  The URLs for which the file path  match the \<list\> specification  will not be crawled.  This list does not have precedence over the  **-allow** argument. This list does not have precedence over the  **robots.txt** file specifications. See the  **-allow** option documentation for an explanation of the \<list\> format.

- **-agent \<name\> (default webbase/5.13)**

  Set the User-Agent parameter for HTTP communication to \<name\> instead of the default value.

- **-depth \<depth\> (default 2)**

  set the exploration threshold. At most \<depth\> URLs will be crawled from a given starting point. The  **robots.txt** file is counted as one URL.

- **-sticky**

  Do not give up on a server even if it timeouts. The normal behaviour is to mark all the URLs of a given server as timeouted if at least 5 URLs from this server ended with a timeout.

## DEBUG OPTIONS

- **-version**

  Show crawler version.

- **-show**

  show the records of the  **start** table matching the url arguments and the records of the **url, url\_content** and **url\_complete** tables that are linked to each of them. The output is driven by the value of the  *-show\_fields option.*

- **-show\_fields \<fields\> (default start.\*,url.\*,url\_complete.\*,url\_content.\*)**

  Must be used in conjunction with -show. Only list the specified fields. All field names must be qualified with the table name.

- **-show\_indexable**

  display the text sent to the full text indexer for each URL given in argument.

- **-verbose**

  Main program debug messages.

- **-verbose\_webbase**

  Database manipulation related messages.

- **-verbose\_webtools**

  Network connection related messages.

- **-verbose\_hooks**

  Show full text indexer related messages.

- **-verbose\_robots**

  Robots.txt loading and parsing related messages.

- **-verbose\_cookies**

  Cookies handling related messages.

- **-verbose\_crawl**

  Many messages from the crawl logic.

- **-babil\_crawl**

  Many messages from the crawl logic. Huge output.

- **-verbose\_dirsel**

  Messages from the url exclusion implementation. This includes handling of the -allow and -disallow options and of the robots.txt lines as they are applied to each URL seen.

## EXAMPLES

*crawler -base crawl -depth 1000 -- http://www.droit.umontreal.ca/*

Mirrors in depth the Web starting at  *URLs http://www.droit.umontreal.ca/* until it finds 1000 documents or there is no documents left, using the 'crawl' MySQL database.

*crawler -base crawl -home\_pages*

Mirrors web pages specified in the  **start** table. Because the **start** table contains all crawling params about depth, level, etc. there is no need to specify these params here.

*crawler -base crawl -depth 0 -- http://www.law.indiana.edu:80/law/lawindex.html*

Simple copy of the lawindex.html file on our local machine. No recursive exploration is done.

## SEE ALSO

[uri(3)](/uri/3.md), [consistentc(1)](/consistentc/1.md), [mifluz(3)](/mifluz/3.md)