---
name: "Plucene::Analysis::Standard::StandardTokenizer"
section: "3pm"
platform: "linux"
package: "libplucene-perl"
version: "1.25-5"
release: "Debian 13 (Trixie)"
platforms: ["linux"]
url: "https://new.carta.tech/Plucene::Analysis::Standard::StandardTokenizer"
---

# Plucene::Analysis::Standard::StandardTokenizer(3pm)

standard tokenizer

## NAME

Plucene::Analysis::Standard::StandardTokenizer - standard tokenizer

## SYNOPSIS

```
        # isa Plucene::Analysis::CharTokenizer
```

## DESCRIPTION

This is the standard tokenizer.

This should be a good tokenizer for most European-language documents.

## METHODS

### token\_re

The regular expression for tokenising.

### normalize

Remove 's and .