---
name: "LaTeXML::Core::Token"
section: "3pm"
platform: "linux"
package: "latexml"
version: "0.8.8-2"
release: "Debian 13 (Trixie)"
platforms: ["linux"]
url: "https://new.carta.tech/LaTeXML::Core::Token"
---

# LaTeXML::Core::Token(3pm)

representation of a Token: a pair of character and category code (catcode); It extends LaTeXML::Common::Object.

## NAME

"LaTeXML::Core::Token" - representation of a Token: a pair of character and category code (catcode); It extends LaTeXML::Common::Object.

### Exported functions

- "$catcode = CC\_ESCAPE;"

  Constants for the category codes:

  ```
    CC_BEGIN, CC_END, CC_MATH, CC_ALIGN, CC_EOL,
    CC_PARAM, CC_SUPER, CC_SUB, CC_IGNORE,
    CC_SPACE, CC_LETTER, CC_OTHER, CC_ACTIVE,
    CC_COMMENT, CC_INVALID, CC_CS.
  ```

  \[The last 2 are (apparent) extensions, with catcodes 16 and 17, respectively\].

- "$token = Token($string,$cc);"

  Creates a LaTeXML::Core::Token with the given content and catcode. The following shorthand versions are also exported for convenience:

  ```
    T_BEGIN, T_END, T_MATH, T_ALIGN, T_PARAM,
    T_SUB, T_SUPER, T_SPACE, T_LETTER($letter),
    T_OTHER($char), T_ACTIVE($char),
    T_COMMENT($comment), T_CS($cs)
  ```

- "@tokens = Explode($string);"

  Returns a list of the tokens corresponding to the characters in `$string`. All tokens have catcode CC\_OTHER, except for spaces which have catcode CC\_SPACE.

- "@tokens = ExplodeText($string);"

  Returns a list of the tokens corresponding to the characters in `$string`. All (roman) letters have catcode CC\_LETTER, all others have catcode CC\_OTHER, except for spaces which have catcode CC\_SPACE.

- "UnTeX($object, $suppress\_linebreaks);"

  Converts `$object` to a string containing TeX that created it (or could have). Note that this is not necessarily the original TeX code; expansions or other substitutions may have taken place.

  Line-breaking of the generated TeX can be explicitly requested or disabled by passing 0 or 1 as the second `$suppress_linebreaks` argument. The default behavior of line-breaking is controlled by the global State value `"SUPPRESS_UNTEX_LINEBREAKS"`.

### Methods

- "@tokens = $object-\>unlist;"

  Return a list of the tokens making up this `$object`.

- "$string = $object-\>toString;"

  Return a string representing `$object`.

- "$string = $token-\>getCSName;"

  Return the string or character part of the `$token`; for the special category codes, returns the standard string (eg. `"T_BEGIN->getCSName"` returns "{").

- "$string = $token-\>getString;"

  Return the string or character part of the `$token`.

- "$code = $token-\>getCharcode;"

  Return the character code of the character part of the `$token`, or 256 if it is a control sequence.

- "$code = $token-\>getCatcode;"

  Return the catcode of the `$token`.

## AUTHOR

pBruce Miller \<bruce.miller@nist.gov\>

## COPYRIGHT

Public domain software, produced as part of work done by the United States Government & not subject to copyright in the US.