---
name: "ucto"
section: "1"
platform: "linux"
package: "ucto"
version: "0.35-2+b1"
release: "Debian 13 (Trixie)"
platforms: ["linux"]
url: "https://new.carta.tech/ucto"
---

# ucto(1)

Unicode Tokenizer

## NAME

ucto - Unicode Tokenizer

## SYNOPSIS

ucto \[\[options\]\] \[input‐file\] \[\[output‐file\]\]

## DESCRIPTION

**ucto** tokenizes text files: it separates words from punctuation, splits sentences (and optionally paragraphs), and finds paired quotes. Ucto is preconfigured with tokenisation rules for several languages.

Those rules are provided by **uctodata**

## OPTIONS

- **-c** configfile

  read settings from a 'configfile'

- **-B**

  run in batch mode. Process all inputfiles to an output directory specified with -O.

- **-d** value

  set debug mode to 'value'

- **-e** value

  set input encoding. (default UTF8)

- **-I** value

  set the input directory to 'value'. (batch mode only)

- **-O** value

  set the ouput directory to 'value'. (Required for batch mode)

- **-N** value

  set UTF8 output normalization. (default NFC)

- **--filter**=\[YES\|NO\]

  disable filtering of special characters, (default YES) These special characters can be specified in the \[FILTER\] block of the configuration file.

- **-L** language

  Automatically selects a configuration file by language code. The language code is generally a three-letter iso-639-3 code. For example, 'fra' will select the file tokconfig‐fra from the installation directory

- **--detectlanguages**=\<lang1,lang2,..langn\>

  try to detect all the specified languages. The default language will be 'lang1'. (only useful for FoLiA output).

  All values must be iso-639-3 codes.

  You can also use the special language code \`und\`. This ensures there is NO default language, and any language that is NOT in the list will remain unanalyzed.

  **Warning:** To be able to handle utterances of mixed language, Ucto uses a simple sentence splitter based on the markers '.' '?' and '!'. This may occasionally lead to surprising results.

- **-l**

  Convert output text to all lowercase

- **-u**

  Convert all input text to all uppercase

- **-n**

  Emit one sentence per line on output

- **-m**

  Assume one sentence per line on input

- **--normalize**=class1,class2,..,classn

  map all occurrences of tokens with class1,...class to their generic names. e.g --normalize=DATE will map all dates to the word {{DATE}}. Very useful to normalize tokens like URL's, DATE's, E-mail addresses and so on.

- **-T** value or **--textredundancy**=value

  set text redundancy level for text nodes in FoLiA output:\
    'full'    - add text to all levels: \<p\> \<s\> \<w\> etc.\
    'minimal' - don't introduce text on higher levels, but retain what is already\
    there.\
    'none'    - only introduce text on \<w\>, AND remove all text from higher levels

- **--allow-word-correction**

  Allow ucto to tokenize inside FoLiA Word elements, creating FoLiA Corrections

- **--ignore-tag-hints**

  Skip all **tag=token** hints from the FoLiA input. These hints can be used to signal text markup like subscript and superscript

- **--add-tokens**="file"

  Add additional tokens to the \[TOKENS\] block of the default language. The file should contain one TOKEN per line.

- **--passthru**

  Don't tokenize, but perform input decoding and simple token role detection

- **--filterpunct**

  remove most of the punctuation from the output. (not from abreviations and embedded punctuation like John's)

- **-P**

  Disable Paragraph Detection

- **-Q**

  Enable Quote Detection. (this is experimental and may lead to unexpected results)

- **-s** \<string\>

  Set End‐of‐sentence marker. (Default \<utt\>)

- **-V** or **-- version**

  Show version information

- **-v**

  set Verbose mode

- **-F**

  The input file(s) are assumed to be FoLiA XML. Text in the correct 'inputclass' will be tokenized. For files with an '.xml' extension, -F is the default.

  In batch mode, this forces to only select files with the '.xml' extension from the input directory.

- **--inputclass**="cls"

  When tokenizing a FoLiA XML document, search for text nodes of class 'cls'. The default is "current".

- **--outputclass**="cls"

  When tokenizing a FoLiA XML document, output the tokenized text in text nodes with 'cls'. The default is "current". It is recommended to have different classes for input and output.

- **--textclass**="cls"**(obsolete)**

  use 'cls' for input and output of text from FoLiA. Equivalent to both --inputclass='cls' and --outputclass='cls')

  This option is obsolete and NOT recommended. Please use the separate --inputclass= and --outputclass options.

- **--copyclass**

  when ucto is used on FoLiA with fully tokenized text in inputclass='inputclass', no text in textclass 'outputclass' is produced. (A warning will be given). To circumvent this. Add the **--copyclass** option. Which assures that text will be emitted in that class

- **-X**

  All output will be FoLiA XML. Document id's are autogenerated.

  Works in batch mode too.

- **--id** \<DocId\>

  Use the specified Document ID for the FoLiA XML. (not allowed in batch mode) When not provided, a document is is generated based on the nema of the input file.

## BUGS

likely

## AUTHORS

Maarten van Gompel

Ko van der Sloot

e-mail: lamasoftware@science.ru.nl