--- name: "toke" section: "1" platform: "linux" package: "fcode-utils" version: "1.0.3-2" release: "Debian 13 (Trixie)" platforms: ["linux"] url: "https://new.carta.tech/toke" --- # toke(1) OpenBIOS tokenizer ## NAME toke - OpenBIOS tokenizer ## SYNOPSIS **toke** *\[options\] forth-file* ## DESCRIPTION **toke** is an open-source FCode Tokenizer from the OpenBIOS project. ## OPTIONS - **-h, -?** Print a brief help message and then exit. - **-i, --ignore-errors** Generate a Binary Output even if errors were reported. - **-l, --load-list** Collect the names of *fload*ed files into an FLoad-List file. The names collected are in the same form as they were presented in the *fload* statements. \ The name of the FLoad-List File is derived from the name of the binary output file, by replacing its extension with *.fl* , or, if the binary output file name had no extension, merely appending the extension *.fl*.\ The binary output file name used for this purpose is either the one specified on the command line, or the one created by default. - **-o, --output-name *OutputFileName*,** Direct the binary output (FCode result of Tokenization) to the named file instead of to the default-named file. This option is not valid when multiple input files are named. - **-P, --dependencies** Collect the fully-resolved pathnames of floaded and ENCODEd files into a dependency-list file. The names collected are in the form that is presented to the host operating system: shell environment Variables and related expressions will be fully expanded, and the directory within the include-list in which the file was found will be attached.\ The name of the dependency-list file will be the same as that of the FLoad-list file, except that its extension will be *.P* instead of *.fl*. - **-v, --verbose** Print additional messages (including advisories) during tokenization. - **-I, --Include *directory*** This tokenizer supports the notion of an include-list. The user creates the include-list by specifying a number of *-I directory* pairs on the command-line. All file-reads, whether for an fload command or an encode-file directive, will involve a search for the named file through the directories of the include-list, in the order they were supplied on the command-line. \ If no include-list is created, file-reads are relative to the current working Directory. If an include-list is created, file-reads are restricted to the directories within it. For the current working directory to be included in the file-search, it must be specified explicitly. **-I.** will accomplish that quite effectively. - **-T, --Trace *symbol*** This tokenizer supports the notion of a trace-list. The User creates the trace-list by specifying a number of *-T symbol* pairs on the command-line.\ When a name is defined, whether as an FCode, an alias, a Macro or anything else, either in normal tokenization mode or "Tokenizer Escape"‐mode, if it matches a symbol that has been added to the trace list, a trace note message will be issued indicating that a definition of that name has been created. Subsequent trace note Messages will be issued when the definition of that name is invoked.\ This trace-symbols feature can be helpful during maintenance of legacy code, for instance, when multiple symbols carry the same name. - **-d, --define *Symbol\[=Value\]*** Define a command-line symbol and optionally, assign a value to it. If you wish the *value* to contain spaces or quotes, you can accomplish that using the shell escape conventions. This sequence may be repeated. Once a Symbol is defined on the command-line, it stays in effect for the duration of the entire batch of tokenizations (i.e., if there are multiple input files named on the command line). Command-line Symbols can be tested for purposes of conditional tokenization, or their assigned values can be evaluated. - **-f, --flag *\[no\]\*** The tokenizer recognizes a specific set of special-feature flag-names; each is associated with a specific non-standard variant behavior. Pass the flag-name as an argument to the *-f* switch to enable the behavior; to disable it, precede the flag-name with the optional string *no*.\ The settings of the special-feature flags can also be changed or displayed from within the source input file.\ The special-feature flags are all initially set to be enabled, except where noted.\ The flag-names and their associated special-features are as follows: - **Local-Values** Support IBM-style Local Values ("LV"s). Initially disabled. - **LV-Legacy-Separator** Allow Semicolon for Local Values Separator ("Legacy"). - **LV-Legacy-Message** Display a Warning Message when Semicolon is used as the Local Values Separator. - **ABORT-Quote** Allow ABORT" macro. - **Sun-ABORT-Quote** ABORT" with implicit IF ... THEN - **Abort-Quote-Throw** Use -2 THROW, rather than ABORT, in an Abort" phrase - **String-remark-escape** Allow "\\ (Quote-Backslash) to interrupt string parsing. - **Hex-remark-escape** Allow \\ (Backslash) to interrupt hex-sequence parsing within a string. - **C-Style-string-escape** Allow the C-style String-Escape pairs \\n \\t and \\xx\\ to be treated as special characters in string parsing. - **Always-Headers** Override occurrences of the standard directive headerless in the source with -- effectively -- headers to make all definitions have a header. Occurrences of the directive external will continue to behave in the standard manner. Initially disabled. - **Always-External** All definitions will be made as though under the external directive; occurrences of either Standard directive headerless or headers in the source will be overridden. This special-feature flag will also override the *Always-Headers* special-feature flag in the event that both have been specified. Initially disabled. - **Warn-if-Duplicate** Display a WARNING message whenever a definition is made whose name duplicates that of an existing definition. Disabling this flag will suspend the duplicate-names test globally, until it is re-enabled. A Directive is supported that will suspend the test for the duration of only a single definition, without affecting global behavior. - **Obsolete-FCode-Warning** Display a WARNING message whenever an FCode function is invoked that the Standard identifies as obsolete. - **Trace-Conditionals** Issue advisory messages about the state of conditional tokenization. (Remember that advisory messages are displayed only if the *verbose* option *-v* is set.) Initially disabled. - **Upper-Case-Token-Names** **Lower-Case-Token-Names** When outputting the names of headered functions (**token-names**) to the binary output file, override the character-case in which the names appeared in the source, and convert them to *Upper-* or *Lower-* ‐Case, respectively. (These flags do not affect text string sequences, whose character-case is always preserved.) Initially disabled. - **Big-End-PCI-Rev-Level** Save the **Revision Level** of the Vendor's ROM field of the PCI Header in big-endian byte-order, rather than little-endian as per the general PCI Standard convention. (This flag does not affect any other field\ of the PCI Header). Initially disabled. - **Ret-Stk-Interp** Allow return-stack operations during interpretation. While the standard specifies that usage of the operators **\>r**, **r@**, and **r\>** while interpreting is allowed, actual practice in the industry is inconsistent. Developers who wish to take a more cautious approach to this question can disable this flag so that any attempt to use the operators **\>r**, **r@**, and **r\>** in the interpreting state will generate an ERROR Message. Also, the pseudo-flag-name *help* will cause a list of the flag-names and their associated special-features to be printed. ## SEE ALSO [**detok**(1)](/detok/1.md). ## AUTHORS Stefan Reinauer \\ David L. Paktor \ This manual page was written by Aurelien Jarno \ for the Debian GNU/Linux system (but may be used by others).