webscan - "Ferret" search engine web crawler webscan [--index=<index-file>] [--machines=<allowed-machine-pattern>] [--include=<allowed-filename-pattern>] [--exclude=<disallowed-filename-pattern>] [--delay=<delay-time>] [--summary=<max-summary-bytes>] [--lines=<max-summary-lines>] [--name=<user's-name>] [--email=<contact-email-address>] [[--refresh=<url-pattern>] [--random] <new-url> [...] WebScan a full-featured web crawler for indexing documents. In its simplest form, WebScan can be run as:
webscan --index=sites.index www.hissite.com www.hersite.com
By default, webscan is very choosy about where it will search. Unless specified, it will not leave the machines on which it was run (hissite and hersite in the above example). This is done to prevent accidentally crawling over half of the Internet by mistake. WebScan will index HTML, Code (C/C++, Perl, shell-script), MIF (Maker Interchange Format), and plain text documents.
Options The following options alter the behavior of WebScan.
This option will cause WebScan to ignore all files matching the given regular expression. The regex is evaluated (case insensitive) by Perl and thus can use all of the Perl extensions.
If not specified, no documents will be excluded.
Example: --exclude='\.cgi$⎪/private/⎪/incoming/'
This option will cause WebScan to pause number seconds between each document fetch. This is a courtesy that should be extended when crawling over the Internet so as to not overload the sites being indexed. It's only a program, so why hurry?
If not specified, this option defaults to 2 seconds.
Example: --delay=5
This option provides information that is sent to the servers being scanned so the administrators there can contact you if the scan seems to be going awry for some reason. It is considered poor netiquette not to include this.
If not specified, no email address will be included.
Example: [email protected]
This option will cause WebScan to ignore all files not matching the given regular expression. The regex is evaluated (case insensitive) by Perl and thus can use all of the Perl extensions.
If not specified, all documents will be included.
Example: --include='/[^\.]*$⎪\.(htm⎪html)$'
This option tells WebScan what index it is to generate/update during its travels.
If not specified, an index name of "./ferret.index" will be used.
Example: --index=/var/web/data/mysite.index
This option tells WebScan the maximum number of lines allowed in text summaries. This option will not affect the summaries generated for HTML documents because there is no way to count lines until the document is actually displayed.
If not specified, a maximum of 5 lines will be generated.
Example: --lines=10
This option tells WebScan what machines it is allowed to scan. The regex is evaluated (case insensitive) by Perl and thus can use all of the Perl extensions. A value of '.*' would allow travelling anywhere, but this is highly discouraged! It is extremely easy for a Web spider to get sidetracked this way, and Ferret is simply not capable of indexing the entire Internet.
If not specified, WebScan will not leave the machines specified on the command line.
Example: --machines='.*\.mydomain\.com'
This option tells WebScan the name of the person or organization running the scan. This information is sent to the servers being scanned so the administrators there know who is scanning their information. It is considered poor netiquette not to include this.
If not specified, no information is sent.
Example: --name="My Company Name Here"
This option tells WebScan to periodically randomize the list of URLs it has yet to scan. This option is very useful when indexing multiple machines as it will tend to distribute the load evently instead of indexing one machine at a time.
If not specified, WebScan will scan in the order in which it finds the documents.
Example: --random
This option tells WebScan which of the URLs it has already indexed should be checked for updates. The regex is evaluated (case insensitive) by Perl and thus can use all of the Perl extensions. A value of '.*' would refresh all of the indexed documents. For a large index, it might be wise to only refresh certain parts every day and other parts every week.
If not specified, WebScan will not refresh any documents.
Example: --refresh='/dynamic-files/⎪/home-.*-dir/'
This option tells WebScan to shrink the index to its minimum size when the scan is completed. This will return any unsued space in the index back to the filesystem.
If not specified, WebScan will not shrink the index when finished.
Example: --shrink Ferret is copyright (c) 1996 by Verisim, Inc. For more information, join the Ferret mailing list by mailing the command "subscribe ferret-list" to the address: [email protected] Report bugs to: [email protected] Please include the version of Ferret being run, a detailed description of the problem, and how to reproduce it. Thank you!