---
name: "webscan"
section: "1"
platform: "linux"
package: "ferret"
version: "1.1.3-eval-1"
release: "Debian 1.2 (Rex)"
platforms: ["linux"]
url: "https://new.carta.tech/webscan"
---

# webscan(1)

webscan - "Ferret" search engine web crawler **webscan** \[**--index=**\<*index-file*\>\] \[**--machines=**\<*allowed-machine-pattern*\>\] \[**--include=**\<*allowed-filename-pattern*\>\] \[**--exclude=**\<*disallowed-filename-pattern*\>\] \[**--delay=**\<*delay-time*\>\] \[**--summary=**\<*max-summary-bytes*\>\] \[**--lines=**\<*max-summary-lines*\>\] \[**--name=**\<*user's-name*\>\] \[**--email=**\<*contact-email-address*\>\] \[\[**--refresh=**\<*url-pattern*\>\] \[**--random**\] \<*new-url*\> \[...\] WebScan a full-featured web crawler for indexing documents.  In its simplest form, WebScan can be run as:

```
  webscan --index=sites.index www.hissite.com www.hersite.com
```

By default, webscan is very choosy about where it will search.  Unless specified, it will not leave the machines on which it was run (*hissite* and *hersite* in the above example).  This is done to prevent accidentally crawling over half of the Internet by mistake. WebScan will index HTML, Code (C/C++, Perl, shell-script), MIF (Maker Interchange Format), and plain text documents.\
 **Options** The following options alter the behavior of **WebScan**.\
 This option will cause **WebScan** to ignore all files matching the given regular expression.  The regex is evaluated (case insensitive) by Perl and thus can use all of the Perl extensions.

If not specified, no documents will be excluded.

Example:  `--exclude='\.cgi$⎪/private/⎪/incoming/'`\
 This option will cause **WebScan** to pause *number* seconds between each document fetch.  This is a courtesy that should be extended when crawling over the Internet so as to not overload the sites being indexed.  It's only a program, so why hurry?

If not specified, this option defaults to 2 seconds.

Example:  `--delay=5`\
 This option provides information that is sent to the servers being scanned so the administrators there can contact you if the scan seems to be going awry for some reason.  It is considered poor netiquette not to include this.

If not specified, no email address will be included.

Example:  `--email=userid@mydomain.com`\
 This option will cause **WebScan** to ignore all files *not* matching the given regular expression.  The regex is evaluated (case insensitive) by Perl and thus can use all of the Perl extensions.

If not specified, all documents will be included.

Example:  `--include='/[^\.]*$⎪\.(htm⎪html)$'`\
 This option tells **WebScan** what index it is to generate/update during its travels.

If not specified, an index name of "*./ferret.index*" will be used.

Example:  `--index=/var/web/data/mysite.index`\
 This option tells **WebScan** the maximum number of lines allowed in text summaries.  This option will not affect the summaries generated for HTML documents because there is no way to count lines until the document is actually displayed.

If not specified, a maximum of 5 lines will be generated.

Example:  `--lines=10`\
 This option tells **WebScan** what machines it is allowed to scan.  The regex is evaluated (case insensitive) by Perl and thus can use all of the Perl extensions.  A value of '.\*' would allow travelling anywhere, but this is highly discouraged!  It is extremely easy for a Web spider to get sidetracked this way, and Ferret is simply not capable of indexing the entire Internet.

If not specified, **WebScan** will not leave the machines specified on the command line.

Example:  `--machines='.*\.mydomain\.com'`\
 This option tells **WebScan** the name of the person or organization running the scan.  This information is sent to the servers being scanned so the administrators there know who is scanning their information.  It is considered poor netiquette not to include this.

If not specified, no information is sent.

Example:  `--name="My Company Name Here"`\
 This option tells **WebScan** to periodically randomize the list of URLs it has yet to scan.  This option is very useful when indexing multiple machines as it will tend to distribute the load evently instead of indexing one machine at a time.

If not specified, **WebScan** will scan in the order in which it finds the documents.

Example:  `--random`\
 This option tells **WebScan** which of the URLs it has already indexed should be checked for updates.  The regex is evaluated (case insensitive) by Perl and thus can use all of the Perl extensions.  A value of '.\*' would refresh all of the indexed documents.  For a large index, it might be wise to only refresh certain parts every day and other parts every week.

If not specified, **WebScan** will not refresh any documents.

Example:  `--refresh='/dynamic-files/⎪/home-.*-dir/'`\
 This option tells **WebScan** to shrink the index to its minimum size when the scan is completed.  This will return any unsued space in the index back to the filesystem.

If not specified, **WebScan** will not shrink the index when finished.

Example:  `--shrink` **Ferret** is copyright (c) 1996 by Verisim, Inc. For more information, join the **Ferret** mailing list by mailing the command "subscribe ferret-list" to the address: `ferret-list-requst@verisim.com` Report bugs to: `ferret-bugs@verisim.com`   Please include the version of **Ferret** being run, a detailed description of the problem, and how to reproduce it.  Thank you!