---
name: "KinoSearch::Analysis::TokenBatch"
section: "3pm"
platform: "linux"
package: "libkinosearch-perl"
version: "0.165-2"
release: "Debian 6.0 (Squeeze)"
platforms: ["linux"]
url: "https://new.carta.tech/KinoSearch::Analysis::TokenBatch"
---

# KinoSearch::Analysis::TokenBatch(3pm)

a collection of tokens

Add many tokens to the batch, by supplying the string to be tokenized, and arrays of token starts and token ends (specified in bytes).

## NAME

KinoSearch::Analysis::TokenBatch - a collection of tokens

## SYNOPSIS

```
    while ( $batch->next ) {
        $batch->set_text( lc( $batch->get_text ) );
    }
```

## EXPERIMENTAL API

TokenBatch's API should be considered experimental and is likely to change.

## DESCRIPTION

A TokenBatch is a collection of Tokens which you can add to, then iterate over.

## METHODS

### new

```
    my $batch = KinoSearch::Analysis::TokenBatch->new;
```

Constructor.

### append

```
    $batch->append( $text, $start_offset, $end_offset, $pos_inc );
```

Add a Token to the end of the batch.  Accepts either three or four arguments: text, start\_offset, end\_offset, and an optional position increment which defaults to 1 if not supplied.  For a description of what these arguments mean, see the docs for Token.

### next

```
    while ( $batch->next ) {
        # ...
    }
```

Proceed to the next token in the TokenBatch.  Returns true if the TokenBatch ends up located at valid token.

## ACCESSOR METHODS

All of TokenBatch's accessor methods affect the current Token.  Calling any of these methods when the TokenBatch is not located at a valid Token will trigger an exception.

### set\_text get\_text

Set/get the text of the current Token.

### set\_start\_offset get\_start\_offset

Set/get the start\_offset of the current Token.

### set\_end\_offset get\_end\_offset

Set/get the end\_offset of the current Token.

### set\_pos\_inc get\_pos\_inc

Set/get the position increment of the current Token.

## COPYRIGHT

Copyright 2005-2009 Marvin Humphrey

## LICENSE, DISCLAIMER, BUGS, etc.

See KinoSearch version 0.165.