---
name: "Plucene::Analysis::CharTokenizer"
section: "3pm"
platform: "linux"
package: "libplucene-perl"
version: "1.25-5"
release: "Debian 13 (Trixie)"
platforms: ["linux"]
url: "https://new.carta.tech/Plucene::Analysis::CharTokenizer"
---

# Plucene::Analysis::CharTokenizer(3pm)

base class for character tokenisers

## NAME

Plucene::Analysis::CharTokenizer - base class for character tokenisers

## SYNOPSIS

```
        # isa Plucene::Analysis::Tokenizer
        my $next = $chartokenizer->next;
```

## DESCRIPTION

This is an abstract base class for simple, character-oriented tokenizers.

## METHODS

### token\_re

This should be defined in subclasses.

### normalize

This will normalise the character before it is added to the token.

### next

```
        my $next = $chartokenizer->next;
```

This will return the next token in the string, or undef at the end  of the string.