--- name: "Search::QueryParser" section: "3pm" platform: "linux" package: "libsearch-queryparser-perl" version: "0.95-2" release: "Debian 13 (Trixie)" platforms: ["linux"] url: "https://new.carta.tech/Search::QueryParser" --- # Search::QueryParser(3pm) parses a query string into a data structure suitable for external search engines ## NAME Search::QueryParser - parses a query string into a data structure suitable for external search engines ## SYNOPSIS ``` my $qp = new Search::QueryParser; my $s = '+mandatoryWord -excludedWord +field:word "exact phrase"'; my $query = $qp->parse($s) or die "Error in query : " . $qp->err; $someIndexer->search($query); # query with comparison operators and implicit plus (second arg is true) $query = $qp->parse("txt~'^foo.*' date>='01.01.2001' date<='02.02.2002'", 1); # boolean operators (example below is equivalent to "+a +(b c) -d") $query = $qp->parse("a AND (b OR c) AND NOT d"); # subset of rows $query = $qp->parse("Id#123,444,555,666 AND (b OR c)"); ``` ## DESCRIPTION This module parses a query string into a data structure to be handled by external search engines. For examples of such engines, see File::Tabular and Search::Indexer. The query string can contain simple terms, "exact phrases", field names and comparison operators, '+/-' prefixes, parentheses, and boolean connectors. The parser can be parameterized by regular expressions for specific notions of "term", "field name" or "operator" ; see the new method. The parser has no support for lemmatization or other term transformations : these should be done externally, before passing the query data structure to the search engine. The data structure resulting from a parsed query is a tree of terms and operators, as described below in the parse method. The interpretation of the structure is up to the external search engine that will receive the parsed query ; the present module does not make any assumption about what it means to be "equal" or to "contain" a term. ## QUERY STRING The query string is decomposed into "items", where each item has an optional sign prefix, an optional field name and comparison operator, and a mandatory value. ### Sign prefix Prefix '+' means that the item is mandatory. Prefix '-' means that the item must be excluded. No prefix means that the item will be searched for, but is not mandatory. As far as the result set is concerned, `"+a +b c"` is strictly equivalent to `"+a +b"` : the search engine will return documents containing both terms 'a' and 'b', and possibly also term 'c'. However, if the search engine also returns relevance scores, query `"+a +b c"` might give a better score to documents containing also term 'c'. See also section "Boolean connectors" below, which is another way to combine items into a query. ### Field name and comparison operator Internally, each query item has a field name and comparison operator; if not written explicitly in the query, these take default values `''` (empty field name) and `':'` (colon operator). Operators have a left operand (the field name) and a right operand (the value to be compared with); for example, `"foo:bar"` means "search documents containing term 'bar' in field 'foo'", whereas `"foo=bar"` means "search documents where field 'foo' has exact value 'bar'". Here is the list of admitted operators with their intended meaning : - ":" treat value as a term to be searched within field. This is the default operator. - "~" or "=~" treat value as a regex; match field against the regex. - "!~" negation of above - "==" or "=", "\<=", "\>=", "!=", "\<", "\>" classical relational operators - "#" Inclusion in the set of comma-separated integers supplied on the right-hand side. Operators `":"`, `"~"`, `"=~"`, `"!~"` and `"#"` admit an empty left operand (so the field name will be `''`). Search engines will usually interpret this as "any field" or "the whole data record". ### Value A value (right operand to a comparison operator) can be - • just a term (as recognized by regex `"rxTerm"`, see new method below) - • A quoted phrase, i.e. a collection of terms within single or double quotes. Quotes can be used not only for "exact phrases", but also to prevent misinterpretation of some values : for example `"-2"` would mean "value '2' with prefix '-'", in other words "exclude term '2'", so if you want to search for value -2, you should write `"-2"` instead. In the last example of the synopsis, quotes were used to prevent splitting of dates into several search terms. - • a subquery within parentheses. Field names and operators distribute over parentheses, so for example `"foo:(bar bie)"` is equivalent to `"foo:bar foo:bie"`. Nested field names such as `"foo:(bar:bie)"` are not allowed. Sign prefixes do not distribute : `"+(foo bar) +bie"` is not equivalent to `"+foo +bar +bie"`. ### Boolean connectors Queries can contain boolean connectors 'AND', 'OR', 'NOT' (or their equivalent in some other languages). This is mere syntactic sugar for the '+' and '-' prefixes : `"a AND b"` is translated into `"+a +b"`; `"a OR b"` is translated into `"(a b)"`; `"NOT a"` is translated into `"-a"`. `"+a OR b"` does not make sense, but it is translated into `"(a b)"`, under the assumption that the user understands "OR" better than a '+' prefix. `"-a OR b"` does not make sense either, but has no meaningful approximation, so it is rejected. Combinations of AND/OR clauses must be surrounded by parentheses, i.e. `"(a AND b) OR c"` or `"a AND (b OR c)"` are allowed, but `"a AND b OR c"` is not. ## METHODS - new ``` new(rxTerm => qr/.../, rxOp => qr/.../, ...) ``` Creates a new query parser, initialized with (optional) regular expressions : - rxTerm Regular expression for matching a term. Of course it should not match the empty string. Default value is `"qr/[^\s()]+/"`. A term should not be allowed to include parenthesis, otherwise the parser might get into trouble. - rxField Regular expression for matching a field name. Default value is `"qr/\w+/"` (meaning of `"\w"` according to `"use locale"`). - rxOp Regular expression for matching an operator. Default value is `"qr/==|<=|>=|!=|=~|!~|:|=|<|>|~/"`. Note that the longest operators come first in the regex, because "alternatives are tried from left to right" (see "Version 8 Regular Expressions" in perlre) : this is to avoid `"a<=3"` being parsed as `"a < '=3'"`. - rxOpNoField Regular expression for a subset of the operators which admit an empty left operand (no field name). Default value is `"qr/=~|!~|~|:/"`. Such operators can be meaningful for comparisons with "any field" or with "the whole record" ; the precise interpretation depends on the search engine. - rxAnd Regular expression for boolean connector AND. Default value is `"qr/AND|ET|UND|E/"`. - rxOr Regular expression for boolean connector OR. Default value is `"qr/OR|OU|ODER|O/"`. - rxNot Regular expression for boolean connector NOT. Default value is `"qr/NOT|PAS|NICHT|NON/"`. - defField If no field is specified in the query, use *defField*. The default is the empty string "". - parse ``` $q = $queryParser->parse($queryString, $implicitPlus); ``` Returns a data structure corresponding to the parsed string. The second argument is optional; if true, it adds an implicit '+' in front of each term without prefix, so `"parse("+a b c -d", 1)"` is equivalent to `"parse("+a +b +c -d")"`. This is often seen in common WWW search engines as an option "match all words". The return value has following structure : ``` { '+' => [{field=>'f1', op=>':', value=>'v1', quote=>'q1'}, {field=>'f2', op=>':', value=>'v2', quote=>'q2'}, ...], '' => [...], '-' => [...] } ``` In other words, it is a hash ref with 3 keys `'+'`, `''` and `'-'`, corresponding to the 3 sign prefixes (mandatory, ordinary or excluded items). Each key holds either a ref to an array of items, or `"undef"` (no items with this prefix in the query). An *item* is a hash ref containing - "field" scalar, field name (may be the empty string) - "op" scalar, operator - "quote" scalar, character that was used for quoting the value ('"', "'" or undef) - "value" Either - • a scalar (simple term), or - • a recursive ref to another query structure. In that case, `"op"` is necessarily `'()'` ; this corresponds to a subquery in parentheses. In case of a parsing error, `"parse"` returns `"undef"`; method err can be called to get an explanatory message. - err ``` $msg = $queryParser->err; ``` Message describing the last parse error - unparse ``` $s = $queryParser->unparse($query); ``` Returns a string representation of the `$query` data structure. ## AUTHOR Laurent Dami, \ ## COPYRIGHT AND LICENSE Copyright (C) 2005, 2007 by Laurent Dami. This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.