NAME
linbot - WWW site link checker
SYNOPSIS
linbot [-x xurl] ... [-b][-a][-o reportdir] base_url [ihost] ...
DESCRIPTION
The program linbot will check the hyperdocument found at base_url and report on it in the directory reportdir. A running report of what the robot is doing is written to stdout.
When checking url's, linbot checks http:, ftp:, and file: schemas. All other schemas (like mailto:, gopher:, lynxcgi: etc.) are not checked and treated as "external".
The report can be viewed using most Web browsers. Browsers using frames technology should initially open the index.html file. Browsers not using frames or with frames disabled can initially open the navbar.html file. Note that these are the default filenames for linbot and may be changed via the config file.
The report consists of the files (with constant names, so use different directories for different invocations of linbot) broken.html, extern.html, images.html, index.html, map.html, navbar.html, new.html, and old.html. Here index.html and navbar.html only point at the other files. (The file index.html is a page with frames pointing at navbar.html and map.html, and the file navbar.html points at the remaining six pages.) The file map.html gives an overview of the hyperdocument (up to depth 5). The file broken.html reports the broken links; the file extern.html reports the external links; the file old.html reports the ancient links; the file new.html reports on the links that changed less than a week ago. Finally images.html collects the images found.
Executing linbot without any command-line arguments will cause it to give a simple synopsis of its usage and then quit.
You should decide beforehand which structures in your hyperdocument should be considered "internal" and which should be considered "external". An internal link is a part of your document that you have control of and should be checked, as well as the links that it points to. Basically an internal link is one that, if broken, you have the power to fix. An external link is one that your document points to, but you have no jurisdiction over. It can also be a link that you may have power to change, but need not be checked for broken links, such as CGI scripts or pages that were generated by an automated tool (such as linbot or any program that converts a document of one format to HTML).
Linbot honors the robots.txt file/protocol. This protocol is useful to indicate to robots that some parts or your site, such as CGI scripts, internal documents, or server stats, should not be explored. The robots.txt protocol is explained at http://info.webcrawler.com/mak/projects/robots/exclusion-admin.html. Currently linbot identifies itself as `User-Agent: Linbot'. You can allow linbot to search a directory but restrict other bots, for example, like this:
User-agent: * Disallow: / User-agent: Linbot Allow: /
OPTIONS
- -x url
-
Use this option to tell Linbot to consider any url starting with url to be external. This option can be used multiple times.
- -b
-
Base URLs only. This option tells linbot to consider any url that does not start with the base url to be external.
- -a
-
Avoid external links. Normally, if linbot is examining an HTML page and finds a link that points to an external document, it will check to see if that document exists. If the -a option is chosen, linbot will not check for the existence of external links.
- -o reportdir
-
Used to specify the directory where the report files must be written. The default is the current directory or as specified in config.py.
- ihost
-
This specifies that urls at ihost are to be considered internal. This can be useful, for example, it the base url is on one server but points to "internal" documents on another server. Ihost is the name of that server, for example www2.someplaceonthenet.com. This can also be used, for example, if you have an intranet where some urls may point to www2.someplaceonthenet.com whereas some urls may point to just `www'. This option may be used more than once, but must follow the base url.
BUGS
If you have any questions about Linbot or would like to report a bug, send electronic mail to [email protected]. Please mention Linbot in the subject of the message. In order to assist in tracking down bugs, please include either a URL where the problem can be found, an HTML file where the error occurs or a (small) tar file of a site where the error occurs. Suggestions for improvements are also welcomed.