--- name: "ferret" section: "3" platform: "linux" package: "ferret" version: "1.1.3-eval-1" release: "Debian 1.2 (Rex)" platforms: ["linux"] url: "https://new.carta.tech/ferret/3" --- # ferret(3) ferret - "Ferret" search engine Perl interface ``` use Ferret; ``` ``` [result =] Ferret::routine([...parameters...]); ``` ``` or ``` ``` $ferret = new Ferret; [result =] $ferret->method(...parameters...); ``` Routines:\ **ChangeHTMLForm** Convert form for local display\ **LimitLineCount** Limit text to a number of lines\ **LoadFile** Load entire file from disk\ **LoadHTTP** Fetch URL via HTTP protocol\ **LoadURL** Load URL from the 'net\ **MakeHTMLSummary** Convert summary to HTML\ **MarkHTMLWords** Highlight queried words in HTML\ **MarkTextWords** Highlight queried words in Text\ **QueryWords** Get words from query string\ **QuoteHTML** Convert reserved HTML characters\ **ResolveHTMLImages**Convert images for local view\ **ResolveURL** Determine fully-qualified URL\ **SetNetworkTimeout**Limit time accessing the network\ **StdExtFilters** Standard filters by file extension\ **StripCode** Remove non-index words from source\ **StripHTML** Remove non-index words from HTML\ **StripMIF** Remove non-index words from MIF files\ **StripMail** Remove non-index words from mail\ **StripText** Remove non-index words from text\ **TimetoRFC1123** Convert unix-time to web-time\ **Uniq** remove duplicate (adjacent) entries\ **UnquoteHTML** Convert reserved HTML characters Methods:\ **AddDocument** Add document words to index\ **AddStoppers** Add words to stopper list\ **Close** Close the index\ **CommonWords** Locate common words\ **DocumentList** Get list of indexed documents\ **DocumentTimestamp**Date/time document was added\ **GetSummary** Get summaries for documents\ **Open** Open an index for searching\ **Query** Search for document by words\ **RemoveDocument** Remove document from index\ **RemoveStoppers** Remove words from stopper list\ **SetOption** Set index options\ **Shrink** Reduce index size as much as possible\ **UnsetOption** Unset index options\ **Update** Open index for read/write The **ferret** library provides an easy way to build advanced indexing and query functions using Perl. This section provides some help on doing searches using Ferret. If you are new to Ferret, you may want to skip the summary (the table below), and read the section called "Query Structure Details" first, and then use the Summary section as a reference table. The following table summarizes all the operations that Ferret can perform. The "Quick Example" column shows how the operation is to be entered. More detailed explanation of the operation follow after the table. Please read the few notes that follow the table; they are important. ``` +=================================================+ ⎪ ⎪ ⎪ ⎪ Operation ⎪ Quick Example(s) ⎪ ⎪ ⎪ ⎪ +=================================================+ ⎪ ⎪ ⎪ ⎪ Single Word ⎪ word ⎪ ⎪ ⎪ ⎪ ⎪ Negation ⎪ NOT word ⎪ ⎪ ⎪ ! word ⎪ ⎪ ⎪ ⎪ ⎪ Required Words (AND) ⎪ word AND werd AND wird ⎪ ⎪ ⎪ word & werd & wird ⎪ ⎪ ⎪ word werd wird ⎪ ⎪ ⎪ ⎪ ⎪ Wild Characters ⎪ w?rd ⎪ ⎪ ⎪ w* ⎪ ⎪ ⎪ w[oei]rd ⎪ ⎪ ⎪ ⎪ ⎪ Optional Words (OR) ⎪ word OR werd OR wird ⎪ ⎪ ⎪ word ⎪ werd ⎪ wird ⎪ ⎪ ⎪ ⎪ ⎪ Priority (Parentheses) ⎪ word & (werd ⎪ wird) ⎪ ⎪ ⎪ ⎪ ⎪ Proximity ⎪ word NEAR werd ⎪ ⎪ ⎪ word 10 werd ⎪ ⎪ ⎪ ⎪ ⎪ Phrase ⎪ "werd is the word" ⎪ ⎪ ⎪ 1 werd is the word ⎪ ⎪ ⎪ ⎪ +------------------------+------------------------+ ``` Ferret operates only on alphabetical characters. You cannot perform searches for numbers. All characters other than alphabetical (numbers, symbols, punctuation) are ignored by the actual search engine; they are treated the same as spaces. However, when searching for phrases, including them can be useful because the search results will be displayed properly.\ If a word is typed in all lowercase letters, the search on it is case-insensetive. If there is at least one capital letter, the search is case-sensetive. The exception to this rule are the special keywords, described next.\ The words **AND**, **OR**, **NOT**, and **NEAR** (all capital letters) are special keywords. **AND** has the same meaning as the **&** operator; **OR** has the same meaining as the **⎪** operator; **NOT** has the same meaning as the **!** operator; and **NEAR** has the same meaning as a proximity-number of **10**.\ All operations take place left to right; certain operators do not take priority over others. Therefore, ``` first OR second AND third ``` will select documents containing "third," and one of the words "first" or "second." It does not select documents containing either "first" or both the words "second" and "third." The following gives a detailed description of **Ferret**'s query language and longer examples of how it can be used.\ **Single Word Search** To retrieve documents containing a single, specific word, just type the word by itself: ``` word ``` The word must not contain punctuation, numbers, or other special characters. Please see the note on non-alphabetic characters in the previous section. As stated earlier, if the word is all lowercase, then the search is not case-sensetive. If there is at least one capital in the word, then the search is case sensetive.\ **Multi-Word Search** Being able to join queries of multiple words is the primary goal of any search engine.\ To retrieve documents containing each and every word of a set, type: ``` first AND second AND third ``` (The **AND** keyword must be all uppercase.) Or, type the words separated by ampersands: ``` first & second & third ``` or, the above may be abbreviated to: ``` first second third ``` To retrieve documents containing at least one of a set of words: ``` first OR second OR third ``` (The **OR** keyword must be all uppercase.) Or, ``` first ⎪ second ⎪ third ``` As stated earlier, the **AND** ( **&** ) operation does not take priority over the **OR** ( **⎪** ) operation; priority is strictly left to right, unless you use parentheses.\ **Prioritization** All evaluation by ferret occurs left to right, regardless of the operation. To prioritize sections of a query, parentheses are required. For example, to retrieve documents containing some required words and at least one of several optional words: ``` firstreqd & secordreqd & (firstoptl ⎪ secondoptl) ``` **Negation** The negation operator reverses the meaning of the subsequent query. Therefore, to select documents that have none of a set of words, use: ``` !notthis & !northis & !northiseither ``` Or, using parentheses, ``` !(notthis ⎪ northis ⎪ northiseither) ``` The keyword **NOT** (note capitalization) also means negation. To retrieve documents containing one word but not another, you can type: ``` required AND NOT disallowed ``` **Phrase Search** To retrieve documents with a specific phrase: ``` "this is my phrase" ``` This will search for documents containing the specified words in sequence, ignoring punctuation and numbers.\ **Proximity** To retrieve documents with words within 10 words of each other: ``` first 10 second ``` or, ``` 10 first second ``` The **NEAR** keyword is an abbreviation for a proximity number of **10**. Therefore, the following means the same thing as the two queries above. Note that **NEAR** must be typed all uppercase. ``` first NEAR second ``` Each proximity is evaluated between two adjacent words. Therefore, ``` 12 first second third fourth ``` finds documents containing "second" within 12 words of "first," and "third" within 12 words of "second," and "fourth" within 12 words of "third." It does not find documents with all four words within 12 words of each other. A proximity number will remain in effect (left to right) until another number is specified. A number of "0" will cancel proximity checking. Note that proximity searches can be turned off when the index is generated and so may not be available on all sites.\ **Wild Characters** Ferret supports in its queries the same "globbing" characters as is common for matching filenames (i.e. the familiar "\*.\*" format). Such abilities are useful when searching for such things a the root of a word. Providing a single "\*" as the last character of the root will match any ending. ``` ? matches any single letter. * matches any zero or more letter. [abc] matches a single letter consisting of either an 'a', 'b', or a 'c.' ``` To retrieve documents containing a piece of a word: ``` onechar? ⎪ manychars* ⎪ rangechars[aeiou] ``` The following functions are all available from the Ferret module. Some can be accessed directly using a prefix of "Ferret::" while others need a Ferret object ($ferret) that with an open index (see the *Open* method for more information.) **AddDocument** will take a document and index all of the valid words within it. The document should have been previously stripped of all undesirable strings via **StripText**, **StripHTML**, etc. Parameters:\ *Document-Name* Name of document (returned by query)\ *Document-Data* Document data to be added to index\ \[ *Word-List* \] Words to omit for this document Output:\ *none* Example: `$ferret-`\>`AddDocument($url,$data);` **AddStoppers** will add words to the internal stopper list so they will not be indexed. All existing index data for these words will be deleted. Parameters:\ *Word-List* List of words not to index Output:\ *none* Example: `$ferret-`\>`AddStoppers("some","words",@wordlist);` **ChangeHTMLForm** will process HTML data (usually a generated summary) and convert the form tags into something else. This gives the ability to display a representation of the form within a summary displayed out of context. The *Conversion-Hash* is a table of form tags and what they should be converted into. The supported tags are: **form /form select radio** **radio-checked checkbox checkbox-checked submit reset text hidden image** **password textarea** For an example on how to use this function, see the "striphtml" example program that came with Ferret. Parameters:\ *HTML-Data* **Reference** to HTML data to be altered\ *Conversion-Hash* **Reference** to tags and replacements Output:\ *none* Example: `Ferret::ChangeHTMLForm(\$html,\$convhash);` **Close** should be called when all actions on the index are completed. Perl will automatically destruct the object when it goes out of scope, but it's good style to do these things manually. Parameters:\ *none* Output:\ *none* Example: `$ferret-`\>`Close()` **CommonWords** will search the entire index looking for words that appear within the specified range of number of documents. This is useful for determining additional words to add to the stopper list. If the specified counts are between 0 and 1, that value is considered to be a percentage of the total number of documents. If greater than 1, it is considered an absolute count. Parameters:\ *Min-Count* Minimum documents a word must be in\ \[ *Max-Count* \] Maximum documents a word can be in Output:\ *Word-String* CR separarted list of matching words Example: `$words = $ferret-`\>`CommonWords(0.50);` **DocumentList** will return a list of all documents currently in the index. Parameters:\ *none* Output:\ *Document-List* List of all indexed documents **DocumentTimestamp** will return the timestamp (unix-time) when the given document was last updated in the index. Parameters:\ *Document-Name* Name of document to get timestamp for Output:\ *Timestamp* Number of seconds since Jan 1st, 1970 Example: `$time = $ferret-`\>`DocumentTimestamp($url);` **GetSummary** will return stored summary information for documents. The summaries are returned either as a list or as a single string, depending upon what type the return variable is. Parameters:\ *Document-Names* One or more document names Output:\ *Summaries* One summary for each document Example: `$summary = $ferret-`\>`GetSummary($url);` **LimitLineCount** will reduce a string of text to the specified number of lines. Parameters:\ *Data* **Reference** to data to truncate\ *Line-Count* Maximum allowed number of lines Output:\ *none* Example: `Ferret::LimitLineCount(\$summary,10);` **LoadFile** will load an any file from disk, optionally passing it through a provided filter program, and return the contents as a single variable. See the "StdExtFilters" function for more information about filters. Parameters:\ *Pathname* Pathname of file to load\ \[ *Filter* \] Filter to pass data through. Output:\ *File-Data* Loaded (and filtered) data Example: `$data = Ferret::LoadFile("my.file");` **LoadHTTP** will use the HTTP protocol (the language of the World Wide Web) to fetch a given URL from over a network. Additional options for the "get" can be specified so the server knows who is requesting the information, and what kinds of return data are allowed. Parameters:\ *Web-File* URL of document to fetch\ \[ *Get-Options* \] Additional options for GET command Output:\ *Result-Data* Results of the GET, header *and* data Example: `Ferret::LoadHTTP("http://www.verisim.com/");` **LoadURL** will load an arbitrary URL and return both the data and information regarding the load operation. Currently supported methods are "file:" and "http:". The header information is different depending upon the access type, but this function will "die" if the fetch was not successful. In addition, if the "filter-flag" is present and non-zero, any loaded data will be passed through any filter deemed appropriate. Parameters:\ *URL* Address (url) of document to load\ *Results* **Reference** for storing result info\ \[ *Filter-Flag* \] Filter loaded data if non-zero Output:\ *Document* Loaded (and filtered) data Example: `Ferret::LoadURL("http://www.verisim.com/",1);` **MakeHTMLSummary** will take the summary created by the various strip functions and convert it into HTML suitable for display by a query engine. The original type (the same type used for Strip*Type*) is passed as a second parameter so the best possible conversion is done. Parameters:\ *Summary* **Reference** to summary data\ *Source-Type* Type of original document Output:\ *none* Example: `Ferret::MakeHTMLSummary(\$summary,"HTML");` **MarkHTMLWords** will take a list of words and phrases (such as that returned by the **QueryWords** function) in an HTML document and surrounded each instance of those with the specified tags. This is different from **MarkTextWords** in that it takes care not to match inside tags. Parameters:\ *HTML-Data* **Reference** to HTML text to highlight\ *Begin-String* String to put before each word/phrase\ *End-String* String to put after each word/phrase\ *Word-List* List of words to highlight Outptu:\ *none* Example: `Ferret::MarkHTMLWords(\$html,"`\`","`\`",@w);` **MarkTextWords** will take a list of words and phrases (such as that returned by the **QueryWords** function) and surrounded each instance of those with the specified tags. Parameters:\ *Text-Data* **Reference** to text to highlight\ *Begin-String* String to put before each word/phrase\ *End-String* String to put after each word/phrase\ *Word-List* List of words to highlight Output:\ *none* Example: `Ferret::MarkTextWords(\$text,"`\>\>\>`","`\<\<\<`",@w);` **Open** must be called before any other "$ferret" methods can function. It opens the specified index file, reads any set options, and generally initializes the index & search systems. Parameters:\ *Pathname* Path to generated index file Output:\ *none* Example: `$ferret-`\>`Open("ferret.index")` **Query** will search the open index and return an array of matches sorted by their score. Each string in the return list has the score prepended to the name of the matched document and can be separated with a Perl statement such as: ``` my($score,$document) = ($matches[0] =~ m/(\d+) (.*)/); ``` The query string is quite sophisticated and allows several different operators between words. ``` word look up given word (may have wild-characters) & boolean AND of the match on left and match on right ⎪ boolean OR of the match on left and match on right () group words and operators to be done first ! negate the match on right (i.e. docs "without") number left and right words must be within "number" words ``` Parameters:\ *Query-String* Set of words and operators Output:\ *Match-List* Array of matches, sorted by score Example: `@matches = $ferret-`\>`Query("search engine*");` **QueryWords** will return a list of all the words asked for in the query string. Quoted expressions are returned as one item in the list, and globbing characters are left unexpanded. Parameters:\ *Query-String* A query as would be passed to `Query` Output:\ *Word-List* List of words in the query string Example: `@words = Ferret::QueryWords("search engine*");` **QuoteHTML** will convert all of the reserved HTML characters (eg. "\<", "\>", "&", etc.) into their coded equivalents. Parameters:\ *HTML-Data* **Reference** to HTML data to be quoted Output:\ *none* Example: `Ferret::QuoteHTML(\$htmldata);` **RemoveDocument** will remove all information for the given document from the index. Space will not be returned to the file system, however, until the index is shrunk. Parameters:\ *Document-Name* Name of document to remove Output:\ *Removed* True, or *undef* in non-existant Example: `$ferret-`\>`RemoveDocument($url);` **RemoveStoppers** will remove words to the internal stopper list so they will be indexed by future documents. Older documents will continue without being indexed upon these words until they are added again. Parameters:\ *Word-List* List of words to index Output:\ *none* Example: `$ferret-`\>`RemoveStoppers(@wordlist);` **ResolveHTMLImages** will take HTML data, locate all of the embedded images, and replace them with fully-qualified names so they will appear when viewed out of context. In addition, a "shrink-factor" and a maximum hight for the images can be specified. Any image with a height and width specified will automatically have those values multiplied by the "shrink-factor". Any images that do not have these values specified will be fetched and their height and width determined. **Note:** Currently, size can only be determined automatically for GIF pictures. Parameters:\ *HTML-Data* **Reference** to HTML data\ *From-URL* Source document URL\ \[ *Shrink-Factor* \] Multiply sizes by this value\ \[ *Max-Height* \] Limit images to this height Output:\ *none* Example: `Ferret::ResolveHTMLImages(\$data,$url,0.75,100);` **ResolveURL** will take a URL and process it to determine the fully-qualified name required by most web software. Given a single name it will add the standard default information. This allows a URL to be specified simply as "www.verisim.com" and be used, after a call to this function, as "http://www.verisim.com/". If supplied with a second parameter, that will be used to modify the first as though it was relative to the first. If the second parameter is *not* relative, then it will be expanded on its own to a fully-qualified address. This mode allows for finding the fully-qualified URL of an "href" within another document. Parameters:\ *From-URL* current document URL\ \[ *To-URL* \] possibly relative destination URL Output:\ *Resolved-URL* fully-qualified URL Example: `$url = Ferret::ResolveURL($url);`\ `$url = Ferret::ResolveURL($url,$nexturl);` **SetNetworkTimeout** will enforce a limit on the amount of time Ferret will spend waiting for data to be sent or received. This is useful for automated programs that cannot be allow to hang. *WebScan* is a good example of this. Parameters:\ *Time* Max seconds to wait for network I/O. Output:\ *Time* Previous setting of the network timeout. Example: `Ferret::SetNetworkTime(60);` **SetOption** will affect how the index behaves. These options are stored and so stay in effect throughout all sessions until specifically unset. Supported options are currently:\ Make index as small as possible: A tiny index stores information is a much more compact manner, but does so at the expense of proximity matching (and thus phrase matching) during queries. If documents have already been indexed, setting this option will cause that data to be compacted to the smaller form and may take a while to complete. **Note:** Once set, this option cannot be unset.\ Index all words at all times: For most indexes, there is not point in indexing all words since people querying the system will be searching by content words. Stoppers are considered to have no content and are thus removed from the standard index. If indexing something with quotes, Shakespeare for example, it in absolutely necessary to include *all* words. Otherwise, searches for things like "to be or not to be" would fail since all the words in that query are considered stoppers. These options can be combined in one call by ORing the together with the '⎪' operator. Parameters:\ *Options* '⎪' separated list of options Output:\ *none* Example: `$ferret-`\>`SetOption(&Ferret::OPT_TINY);` **Shrink** will purge any deleted documents from the index and compact it to its minimum size. Parameters:\ *none* Output:\ *none* Example: `$ferret-`\>`Shrink();` **StdExtFilters** takes a filename and returns a filter through which the data should be passed in order to obtain more useful information. This is used mostly to deal with compressed data files. The filter returned will read data from standard-input and write to standard-output. In addition, if a reference to a scalar is passed as a second parameters, that variable will be given the value of the file without the final extension that defined the filter. For example, a pathname of "archive.tar.gz" would return a filter of "`gzip -d`" and a fixed pathname of "archive.tar". Parameters:\ *Pathname* Path of file to analyze\ \[ *Fixed-Pathname* \] **Reference** for storing fixed pathname Output:\ *Filter* Program (and options) to convert data **StripCode** will take source code data and strip out those words which are not suitable for indexing. If references for title and summary are provided, suitable data is returned there. Parameters:\ *Code-Data* Document data to be stripped\ \[ *Title* \] **Reference** for storing document title\ \[ *Summary* \] **Reference** for storing summary\ \[ *Summary-Size* \] Maximum size of summary to create Output:\ *none* Example: `Ferret::StripCode(\$text,\$title,\$summary,500);` **StripHTML** will take HTML data and strip out those words which are not suitable for indexing. This includes various tags and symbols. If references for title and summary are provided, suitable data is returned there. Parameters:\ *HTML-Data* Document data to be stripped\ \[ *Title* \] **Reference** for storing document title\ \[ *Summary* \] **Reference** for storing summary\ \[ *Summary-Size* \] Maximum size of summary to create Output:\ *none* Example: `Ferret::StripHTML(\$html,\$title,\$summary,500);` **StripMIF** will take MIF (Maker Interchange File) data and strip out those words which are not suitable for indexing. This includes various tags and symbols. If references for title and summary are provided, suitable data is returned there. Parameters:\ *MIF-Data* Document data to be stripped\ \[ *Title* \] **Reference** for storing document title\ \[ *Summary* \] **Reference** for storing summary\ \[ *Summary-Size* \] Maximum size of summary to create Output:\ *none* Example: `Ferret::StripMIF(\$mif,\$title,\$summary,500);` **StripMail** will take simple mail-message data and strip out those words which are not suitable for indexing. This includes embedded data such as UUencoded files. If references for title and summary are provided, suitable data is returned there. Parameters:\ *Mail-Data* Document data to be stripped\ \[ *Title* \] **Reference** for storing document title\ \[ *Summary* \] **Reference** for storing summary\ \[ *Summary-Size* \] Maximum size of summary to create Output:\ *none* Example: `Ferret::StripMail(\$mail,\$title,\$summary,500);` **StripText** will take simple text data and strip out those words which are not suitable for indexing. This includes embedded data such as UUencoded files. If references for title and summary are provided, suitable data is returned there. Parameters:\ *Text-Data* Document data to be stripped\ \[ *Title* \] **Reference** for storing document title\ \[ *Summary* \] **Reference** for storing summary\ \[ *Summary-Size* \] Maximum size of summary to create Output:\ *none* Example: `Ferret::StripText(\$text,\$title,\$summary,500);` **TimetoRFC1123** will take the standard unix time (seconds since January 1st, 1970) and return a time in the form of RFC1123 that is used for passing dates around with the HTTP protocol. Parameters:\ *Time* Unix-time (seconds since Unix epoch) Output:\ *Time* Time represented according to RFC1123 Example: `$webtime = Ferret::TimetoRFC1123(time());` **Uniq** will ensure that no adjacent elements in a list are identical. This is usually run on a sorted list so that each item in the list is unique. Parameters:\ *List* List to make unique Output:\ *List* List with no identical elements Example: `@list = Ferret::Uniq(sort @list);` **UnquoteHTML** will convert all of the reserved HTML characters (eg. "\<", "\>", "&", etc.) from their coded equivalents into their native ascii character. Parameters:\ *HTML-Data* **Reference** to HTML data to be unquoted Output:\ *none* Example: `Ferret:UnquoteHTML(\$htmldata);` **UnsetOption** will affect how the index behaves. These options are stored and so stay in effect throughout all sessions until specifically unset. For more information, see the "SetOption" function. **Update** will open the specified index so new documents and other information can be added to it. Parameters:\ *Pathname* Path to generated index file Output:\ *none* Example: `$ferret-`\>`Update("ferret.index")` **Ferret** is copyright (c) 1996 by Verisim, Inc. [**ferret**(1)](/ferret/1.md), [**webscan**(1)](/webscan/1.md) For more information, join the **Ferret** mailing list by mailing the command "subscribe ferret-list" to the address: `ferret-list-requst@verisim.com` Report bugs to: `ferret-bugs@verisim.com` Please include the version of **Ferret** being run, a detailed description of the problem, and how to reproduce it. Thank you!