ferret - "Ferret" search engine Perl interface
use Ferret;
[result =] Ferret::routine([...parameters...]);
or
$ferret = new Ferret;[result =] $ferret->method(...parameters...);
Routines:
ChangeHTMLForm Convert form for local display
LimitLineCount Limit text to a number of lines
LoadFile Load entire file from disk
LoadHTTP Fetch URL via HTTP protocol
LoadURL Load URL from the 'net
MakeHTMLSummary Convert summary to HTML
MarkHTMLWords Highlight queried words in HTML
MarkTextWords Highlight queried words in Text
QueryWords Get words from query string
QuoteHTML Convert reserved HTML characters
ResolveHTMLImagesConvert images for local view
ResolveURL Determine fully-qualified URL
SetNetworkTimeoutLimit time accessing the network
StdExtFilters Standard filters by file extension
StripCode Remove non-index words from source
StripHTML Remove non-index words from HTML
StripMIF Remove non-index words from MIF files
StripMail Remove non-index words from mail
StripText Remove non-index words from text
TimetoRFC1123 Convert unix-time to web-time
Uniq remove duplicate (adjacent) entries
UnquoteHTML Convert reserved HTML characters Methods:
AddDocument Add document words to index
AddStoppers Add words to stopper list
Close Close the index
CommonWords Locate common words
DocumentList Get list of indexed documents
DocumentTimestampDate/time document was added
GetSummary Get summaries for documents
Open Open an index for searching
Query Search for document by words
RemoveDocument Remove document from index
RemoveStoppers Remove words from stopper list
SetOption Set index options
Shrink Reduce index size as much as possible
UnsetOption Unset index options
Update Open index for read/write The ferret library provides an easy way to build advanced indexing and query functions using Perl. This section provides some help on doing searches using Ferret. If you are new to Ferret, you may want to skip the summary (the table below), and read the section called "Query Structure Details" first, and then use the Summary section as a reference table. The following table summarizes all the operations that Ferret can perform. The "Quick Example" column shows how the operation is to be entered. More detailed explanation of the operation follow after the table. Please read the few notes that follow the table; they are important.
+=================================================+⎪ ⎪ ⎪⎪ Operation ⎪ Quick Example(s) ⎪⎪ ⎪ ⎪+=================================================+⎪ ⎪ ⎪⎪ Single Word ⎪ word ⎪⎪ ⎪ ⎪⎪ Negation ⎪ NOT word ⎪⎪ ⎪ ! word ⎪⎪ ⎪ ⎪⎪ Required Words (AND) ⎪ word AND werd AND wird ⎪⎪ ⎪ word & werd & wird ⎪⎪ ⎪ word werd wird ⎪⎪ ⎪ ⎪⎪ Wild Characters ⎪ w?rd ⎪⎪ ⎪ w* ⎪⎪ ⎪ w[oei]rd ⎪⎪ ⎪ ⎪⎪ Optional Words (OR) ⎪ word OR werd OR wird ⎪⎪ ⎪ word ⎪ werd ⎪ wird ⎪⎪ ⎪ ⎪⎪ Priority (Parentheses) ⎪ word & (werd ⎪ wird) ⎪⎪ ⎪ ⎪⎪ Proximity ⎪ word NEAR werd ⎪⎪ ⎪ word 10 werd ⎪⎪ ⎪ ⎪⎪ Phrase ⎪ "werd is the word" ⎪⎪ ⎪ 1 werd is the word ⎪⎪ ⎪ ⎪+------------------------+------------------------+
Ferret operates only on alphabetical characters. You cannot perform searches for numbers. All characters other than alphabetical (numbers, symbols, punctuation) are ignored by the actual search engine; they are treated the same as spaces. However, when searching for phrases, including them can be useful because the search results will be displayed properly.
If a word is typed in all lowercase letters, the search on it is case-insensetive. If there is at least one capital letter, the search is case-sensetive. The exception to this rule are the special keywords, described next.
The words AND, OR, NOT, and NEAR (all capital letters) are special keywords. AND has the same meaning as the & operator; OR has the same meaining as the ⎪ operator; NOT has the same meaning as the ! operator; and NEAR has the same meaning as a proximity-number of 10.
All operations take place left to right; certain operators do not take priority over others. Therefore,
first OR second AND third
will select documents containing "third," and one of the words "first" or "second." It does not select documents containing either "first" or both the words "second" and "third." The following gives a detailed description of Ferret's query language and longer examples of how it can be used.
Single Word Search To retrieve documents containing a single, specific word, just type the word by itself:
word
The word must not contain punctuation, numbers, or other special characters. Please see the note on non-alphabetic characters in the previous section. As stated earlier, if the word is all lowercase, then the search is not case-sensetive. If there is at least one capital in the word, then the search is case sensetive.
Multi-Word Search Being able to join queries of multiple words is the primary goal of any search engine.
To retrieve documents containing each and every word of a set, type:
first AND second AND third
(The AND keyword must be all uppercase.) Or, type the words separated by ampersands:
first & second & third
or, the above may be abbreviated to:
first second third
To retrieve documents containing at least one of a set of words:
first OR second OR third
(The OR keyword must be all uppercase.) Or,
first ⎪ second ⎪ third
As stated earlier, the AND ( & ) operation does not take priority over the OR ( ⎪ ) operation; priority is strictly left to right, unless you use parentheses.
Prioritization All evaluation by ferret occurs left to right, regardless of the operation. To prioritize sections of a query, parentheses are required. For example, to retrieve documents containing some required words and at least one of several optional words:
firstreqd & secordreqd & (firstoptl ⎪ secondoptl)
Negation The negation operator reverses the meaning of the subsequent query. Therefore, to select documents that have none of a set of words, use:
!notthis & !northis & !northiseither
Or, using parentheses,
!(notthis ⎪ northis ⎪ northiseither)
The keyword NOT (note capitalization) also means negation. To retrieve documents containing one word but not another, you can type:
required AND NOT disallowed
Phrase Search To retrieve documents with a specific phrase:
"this is my phrase"
This will search for documents containing the specified words in sequence, ignoring punctuation and numbers.
Proximity To retrieve documents with words within 10 words of each other:
first 10 second
or,
10 first second
The NEAR keyword is an abbreviation for a proximity number of 10. Therefore, the following means the same thing as the two queries above. Note that NEAR must be typed all uppercase.
first NEAR second
Each proximity is evaluated between two adjacent words. Therefore,
12 first second third fourth
finds documents containing "second" within 12 words of "first," and "third" within 12 words of "second," and "fourth" within 12 words of "third." It does not find documents with all four words within 12 words of each other. A proximity number will remain in effect (left to right) until another number is specified. A number of "0" will cancel proximity checking. Note that proximity searches can be turned off when the index is generated and so may not be available on all sites.
Wild Characters Ferret supports in its queries the same "globbing" characters as is common for matching filenames (i.e. the familiar "*.*" format). Such abilities are useful when searching for such things a the root of a word. Providing a single "*" as the last character of the root will match any ending.
? matches any single letter.* matches any zero or more letter.[abc] matches a single letter consisting of either an 'a', 'b',or a 'c.'
To retrieve documents containing a piece of a word:
onechar? ⎪ manychars* ⎪ rangechars[aeiou]
The following functions are all available from the Ferret module. Some can be accessed directly using a prefix of "Ferret::" while others need a Ferret object ($ferret) that with an open index (see the Open method for more information.) AddDocument will take a document and index all of the valid words within it. The document should have been previously stripped of all undesirable strings via StripText, StripHTML, etc. Parameters:
Document-Name Name of document (returned by query)
Document-Data Document data to be added to index
[ Word-List ] Words to omit for this document Output:
none Example: $ferret->AddDocument($url,$data); AddStoppers will add words to the internal stopper list so they will not be indexed. All existing index data for these words will be deleted. Parameters:
Word-List List of words not to index Output:
none Example: $ferret->AddStoppers("some","words",@wordlist); ChangeHTMLForm will process HTML data (usually a generated summary) and convert the form tags into something else. This gives the ability to display a representation of the form within a summary displayed out of context. The Conversion-Hash is a table of form tags and what they should be converted into. The supported tags are: form /form select radio radio-checked checkbox checkbox-checked submit reset text hidden image password textarea For an example on how to use this function, see the "striphtml" example program that came with Ferret. Parameters:
HTML-Data Reference to HTML data to be altered
Conversion-Hash Reference to tags and replacements Output:
none Example: Ferret::ChangeHTMLForm(\$html,\$convhash); Close should be called when all actions on the index are completed. Perl will automatically destruct the object when it goes out of scope, but it's good style to do these things manually. Parameters:
none Output:
none Example: $ferret->Close() CommonWords will search the entire index looking for words that appear within the specified range of number of documents. This is useful for determining additional words to add to the stopper list. If the specified counts are between 0 and 1, that value is considered to be a percentage of the total number of documents. If greater than 1, it is considered an absolute count. Parameters:
Min-Count Minimum documents a word must be in
[ Max-Count ] Maximum documents a word can be in Output:
Word-String CR separarted list of matching words Example: $words = $ferret->CommonWords(0.50); DocumentList will return a list of all documents currently in the index. Parameters:
none Output:
Document-List List of all indexed documents DocumentTimestamp will return the timestamp (unix-time) when the given document was last updated in the index. Parameters:
Document-Name Name of document to get timestamp for Output:
Timestamp Number of seconds since Jan 1st, 1970 Example: $time = $ferret->DocumentTimestamp($url); GetSummary will return stored summary information for documents. The summaries are returned either as a list or as a single string, depending upon what type the return variable is. Parameters:
Document-Names One or more document names Output:
Summaries One summary for each document Example: $summary = $ferret->GetSummary($url); LimitLineCount will reduce a string of text to the specified number of lines. Parameters:
Data Reference to data to truncate
Line-Count Maximum allowed number of lines Output:
none Example: Ferret::LimitLineCount(\$summary,10); LoadFile will load an any file from disk, optionally passing it through a provided filter program, and return the contents as a single variable. See the "StdExtFilters" function for more information about filters. Parameters:
Pathname Pathname of file to load
[ Filter ] Filter to pass data through. Output:
File-Data Loaded (and filtered) data Example: $data = Ferret::LoadFile("my.file"); LoadHTTP will use the HTTP protocol (the language of the World Wide Web) to fetch a given URL from over a network. Additional options for the "get" can be specified so the server knows who is requesting the information, and what kinds of return data are allowed. Parameters:
Web-File URL of document to fetch
[ Get-Options ] Additional options for GET command Output:
Result-Data Results of the GET, header and data Example: Ferret::LoadHTTP("http://www.verisim.com/"); LoadURL will load an arbitrary URL and return both the data and information regarding the load operation. Currently supported methods are "file:" and "http:". The header information is different depending upon the access type, but this function will "die" if the fetch was not successful. In addition, if the "filter-flag" is present and non-zero, any loaded data will be passed through any filter deemed appropriate. Parameters:
URL Address (url) of document to load
Results Reference for storing result info
[ Filter-Flag ] Filter loaded data if non-zero Output:
Document Loaded (and filtered) data Example: Ferret::LoadURL("http://www.verisim.com/",1); MakeHTMLSummary will take the summary created by the various strip functions and convert it into HTML suitable for display by a query engine. The original type (the same type used for StripType) is passed as a second parameter so the best possible conversion is done. Parameters:
Summary Reference to summary data
Source-Type Type of original document Output:
none Example: Ferret::MakeHTMLSummary(\$summary,"HTML"); MarkHTMLWords will take a list of words and phrases (such as that returned by the QueryWords function) in an HTML document and surrounded each instance of those with the specified tags. This is different from MarkTextWords in that it takes care not to match inside tags. Parameters:
HTML-Data Reference to HTML text to highlight
Begin-String String to put before each word/phrase
End-String String to put after each word/phrase
Word-List List of words to highlight Outptu:
none Example: Ferret::MarkHTMLWords(\$html,"<b>","</b>",@w); MarkTextWords will take a list of words and phrases (such as that returned by the QueryWords function) and surrounded each instance of those with the specified tags. Parameters:
Text-Data Reference to text to highlight
Begin-String String to put before each word/phrase
End-String String to put after each word/phrase
Word-List List of words to highlight Output:
none Example: Ferret::MarkTextWords(\$text,">>>","<<<",@w); Open must be called before any other "$ferret" methods can function. It opens the specified index file, reads any set options, and generally initializes the index & search systems. Parameters:
Pathname Path to generated index file Output:
none Example: $ferret->Open("ferret.index") Query will search the open index and return an array of matches sorted by their score. Each string in the return list has the score prepended to the name of the matched document and can be separated with a Perl statement such as:
my($score,$document) = ($matches[0] =~ m/(\d+) (.*)/);
The query string is quite sophisticated and allows several different operators between words.
word look up given word (may have wild-characters)& boolean AND of the match on left and match on right⎪ boolean OR of the match on left and match on right() group words and operators to be done first! negate the match on right (i.e. docs "without")number left and right words must be within "number" words
Parameters:
Query-String Set of words and operators Output:
Match-List Array of matches, sorted by score Example: @matches = $ferret->Query("search engine*"); QueryWords will return a list of all the words asked for in the query string. Quoted expressions are returned as one item in the list, and globbing characters are left unexpanded. Parameters:
Query-String A query as would be passed to Query Output:
Word-List List of words in the query string Example: @words = Ferret::QueryWords("search engine*"); QuoteHTML will convert all of the reserved HTML characters (eg. "<", ">", "&", etc.) into their coded equivalents. Parameters:
HTML-Data Reference to HTML data to be quoted Output:
none Example: Ferret::QuoteHTML(\$htmldata); RemoveDocument will remove all information for the given document from the index. Space will not be returned to the file system, however, until the index is shrunk. Parameters:
Document-Name Name of document to remove Output:
Removed True, or undef in non-existant Example: $ferret->RemoveDocument($url); RemoveStoppers will remove words to the internal stopper list so they will be indexed by future documents. Older documents will continue without being indexed upon these words until they are added again. Parameters:
Word-List List of words to index Output:
none Example: $ferret->RemoveStoppers(@wordlist); ResolveHTMLImages will take HTML data, locate all of the embedded images, and replace them with fully-qualified names so they will appear when viewed out of context. In addition, a "shrink-factor" and a maximum hight for the images can be specified. Any image with a height and width specified will automatically have those values multiplied by the "shrink-factor". Any images that do not have these values specified will be fetched and their height and width determined. Note: Currently, size can only be determined automatically for GIF pictures. Parameters:
HTML-Data Reference to HTML data
From-URL Source document URL
[ Shrink-Factor ] Multiply sizes by this value
[ Max-Height ] Limit images to this height Output:
none Example: Ferret::ResolveHTMLImages(\$data,$url,0.75,100); ResolveURL will take a URL and process it to determine the fully-qualified name required by most web software. Given a single name it will add the standard default information. This allows a URL to be specified simply as "www.verisim.com" and be used, after a call to this function, as "http://www.verisim.com/". If supplied with a second parameter, that will be used to modify the first as though it was relative to the first. If the second parameter is not relative, then it will be expanded on its own to a fully-qualified address. This mode allows for finding the fully-qualified URL of an "href" within another document. Parameters:
From-URL current document URL
[ To-URL ] possibly relative destination URL Output:
Resolved-URL fully-qualified URL Example: $url = Ferret::ResolveURL($url);
$url = Ferret::ResolveURL($url,$nexturl); SetNetworkTimeout will enforce a limit on the amount of time Ferret will spend waiting for data to be sent or received. This is useful for automated programs that cannot be allow to hang. WebScan is a good example of this. Parameters:
Time Max seconds to wait for network I/O. Output:
Time Previous setting of the network timeout. Example: Ferret::SetNetworkTime(60); SetOption will affect how the index behaves. These options are stored and so stay in effect throughout all sessions until specifically unset. Supported options are currently:
Make index as small as possible: A tiny index stores information is a much more compact manner, but does so at the expense of proximity matching (and thus phrase matching) during queries. If documents have already been indexed, setting this option will cause that data to be compacted to the smaller form and may take a while to complete. Note: Once set, this option cannot be unset.
Index all words at all times: For most indexes, there is not point in indexing all words since people querying the system will be searching by content words. Stoppers are considered to have no content and are thus removed from the standard index. If indexing something with quotes, Shakespeare for example, it in absolutely necessary to include all words. Otherwise, searches for things like "to be or not to be" would fail since all the words in that query are considered stoppers. These options can be combined in one call by ORing the together with the '⎪' operator. Parameters:
Options '⎪' separated list of options Output:
none Example: $ferret->SetOption(&Ferret::OPT_TINY); Shrink will purge any deleted documents from the index and compact it to its minimum size. Parameters:
none Output:
none Example: $ferret->Shrink(); StdExtFilters takes a filename and returns a filter through which the data should be passed in order to obtain more useful information. This is used mostly to deal with compressed data files. The filter returned will read data from standard-input and write to standard-output. In addition, if a reference to a scalar is passed as a second parameters, that variable will be given the value of the file without the final extension that defined the filter. For example, a pathname of "archive.tar.gz" would return a filter of "gzip -d" and a fixed pathname of "archive.tar". Parameters:
Pathname Path of file to analyze
[ Fixed-Pathname ] Reference for storing fixed pathname Output:
Filter Program (and options) to convert data StripCode will take source code data and strip out those words which are not suitable for indexing. If references for title and summary are provided, suitable data is returned there. Parameters:
Code-Data Document data to be stripped
[ Title ] Reference for storing document title
[ Summary ] Reference for storing summary
[ Summary-Size ] Maximum size of summary to create Output:
none Example: Ferret::StripCode(\$text,\$title,\$summary,500); StripHTML will take HTML data and strip out those words which are not suitable for indexing. This includes various tags and symbols. If references for title and summary are provided, suitable data is returned there. Parameters:
HTML-Data Document data to be stripped
[ Title ] Reference for storing document title
[ Summary ] Reference for storing summary
[ Summary-Size ] Maximum size of summary to create Output:
none Example: Ferret::StripHTML(\$html,\$title,\$summary,500); StripMIF will take MIF (Maker Interchange File) data and strip out those words which are not suitable for indexing. This includes various tags and symbols. If references for title and summary are provided, suitable data is returned there. Parameters:
MIF-Data Document data to be stripped
[ Title ] Reference for storing document title
[ Summary ] Reference for storing summary
[ Summary-Size ] Maximum size of summary to create Output:
none Example: Ferret::StripMIF(\$mif,\$title,\$summary,500); StripMail will take simple mail-message data and strip out those words which are not suitable for indexing. This includes embedded data such as UUencoded files. If references for title and summary are provided, suitable data is returned there. Parameters:
Mail-Data Document data to be stripped
[ Title ] Reference for storing document title
[ Summary ] Reference for storing summary
[ Summary-Size ] Maximum size of summary to create Output:
none Example: Ferret::StripMail(\$mail,\$title,\$summary,500); StripText will take simple text data and strip out those words which are not suitable for indexing. This includes embedded data such as UUencoded files. If references for title and summary are provided, suitable data is returned there. Parameters:
Text-Data Document data to be stripped
[ Title ] Reference for storing document title
[ Summary ] Reference for storing summary
[ Summary-Size ] Maximum size of summary to create Output:
none Example: Ferret::StripText(\$text,\$title,\$summary,500); TimetoRFC1123 will take the standard unix time (seconds since January 1st, 1970) and return a time in the form of RFC1123 that is used for passing dates around with the HTTP protocol. Parameters:
Time Unix-time (seconds since Unix epoch) Output:
Time Time represented according to RFC1123 Example: $webtime = Ferret::TimetoRFC1123(time()); Uniq will ensure that no adjacent elements in a list are identical. This is usually run on a sorted list so that each item in the list is unique. Parameters:
List List to make unique Output:
List List with no identical elements Example: @list = Ferret::Uniq(sort @list); UnquoteHTML will convert all of the reserved HTML characters (eg. "<", ">", "&", etc.) from their coded equivalents into their native ascii character. Parameters:
HTML-Data Reference to HTML data to be unquoted Output:
none Example: Ferret:UnquoteHTML(\$htmldata); UnsetOption will affect how the index behaves. These options are stored and so stay in effect throughout all sessions until specifically unset. For more information, see the "SetOption" function. Update will open the specified index so new documents and other information can be added to it. Parameters:
Pathname Path to generated index file Output:
none Example: $ferret->Update("ferret.index") Ferret is copyright (c) 1996 by Verisim, Inc. ferret(1), webscan(1) For more information, join the Ferret mailing list by mailing the command "subscribe ferret-list" to the address: [email protected] Report bugs to: [email protected] Please include the version of Ferret being run, a detailed description of the problem, and how to reproduce it. Thank you!