---
name: "tmx-tokenize"
section: "1p"
platform: "linux"
package: "libxml-tmx-perl"
version: "0.39-2"
release: "Debian 13 (Trixie)"
platforms: ["linux"]
url: "https://new.carta.tech/tmx-tokenize"
---

# tmx-tokenize(1p)

Tokenizes translation units on a tmx file.

## NAME

tmx-tokenize - Tokenizes translation units on a tmx file.

## VERSION

version 0.39

## SYNOPSIS

```
   tmx-tokenize file.tmx  # creates t_file.tmx
   tmx-tokenize -o=out.tmx file.tmx
```

## DESCRIPTION

Although this script is bundled in `"XML::TMX"`, it has a soft dependency on `"Lingua::FreeLing3"`. Soft means that the dependency is not ensured at install time, and other features of the module can still be used without `"Lingua::FreeLing3"`. Nevertheless, if you want to use this tool you should install that module.

At the moment the supported languages are the same as supported by FreeLing3: English, Spanish, Russian, Portuguese and Italian.

It your TMX file includes any other language, they will be maintained without a change.  This behavior can change in the future, as a basic regexp based tokenizer might be implemented.

## SEE ALSO

XML::TMX, Lingua::FreeLing3

## AUTHORS

- •

  Alberto Simões \<ambs@cpan.org\>

- •

  José João Almeida \<jj@di.uminho.pt\>

## COPYRIGHT AND LICENSE

This software is copyright (c) 2010-2017 by Projeto Natura \<natura@di.uminho.pt\>.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.