---
name: "Plucene::Analysis::Tokenizer"
section: "3pm"
platform: "linux"
package: "libplucene-perl"
version: "1.25-5"
release: "Debian 13 (Trixie)"
platforms: ["linux"]
url: "https://new.carta.tech/Plucene::Analysis::Tokenizer"
---

# Plucene::Analysis::Tokenizer(3pm)

base class for tokenizers

## NAME

Plucene::Analysis::Tokenizer - base class for tokenizers

## SYNOPSIS

```
        my $tokenizer = Plucene::Analysis::Tokenizer::Subclass
                ->new({ reader => $reader });
```

## DESCRIPTION

This is an abstract base class for tokenizers.

A Tokenizer is a TokenStream whose input is a Reader.

## METHODS

### new

```
        my $tokenizer = Plucene::Analysis::Tokenizer::Subclass
                ->new({ reader => $reader });
```

This will create a new tokenizer.

### next

This must be defined in a subclass

### close

Close the input reader.