---
name: "ACE_Tokenizer"
section: "3"
platform: "linux"
package: "libace-doc"
version: "5.0.7-4"
release: "Debian 2.2 (Potato)"
platforms: ["linux"]
url: "https://new.carta.tech/ACE_Tokenizer"
---

# ACE\_Tokenizer(3)

Tokenizer

## NAME

ACE\_Tokenizer -  Tokenizer

## SYNOPSIS

```
#include <ace/SString.h>
class ACE_Tokenizer
{
  public:
    ACE_Tokenizer (LPTSTR buffer);
    int delimiter (TCHAR d);
    int delimiter_replace (TCHAR d, TCHAR replacement);
    int preserve_designators (TCHAR start, TCHAR stop, int strip=1);
    LPTSTR next (void);
    MAX_PRESERVES=16 }; protected: int is_delimiter (
        TCHAR d,
        int &replace,
        TCHAR &r
        );
    int is_preserve_designator (TCHAR start, TCHAR &stop, int &strip);
    class Delimiter_Entry
    {
      public:
        TCHAR delimiter_;
        TCHAR replacement_;
        int replace_;
    };
```

  private:     LPTSTR buffer\_;     int index\_;

    class Preserve\_Entry\
     {\
       public:         TCHAR start\_;         TCHAR stop\_;         int strip\_;\
     };     Preserve\_Entry preserves\_\[MAX\_PRESERVES\];     int preserves\_index\_;     Delimiter\_Entry delimiters\_\[MAX\_DELIMITERS\];     int delimiter\_index\_; };

## DESCRIPTION

Tokenizes a buffer.  Allows application to set delimiters and preserve designators.  Does not allow special characters, yet (e.g., printf ("\\"like a quoted string\\"")).

## PUBLIC MEMBERS

```
ACE_Tokenizer (LPTSTR buffer);
```

buffer will be parsed.

```
int delimiter (TCHAR d);
```

d is a delimiter.  Returns 0 on success, -1 if there is no memory left.

```
int delimiter_replace (TCHAR d, TCHAR replacement);
```

d is a delimiter and, when found, will be replaced by replacement.  Returns 0 on success, -1 if there is no memory left.

```
int preserve_designators (TCHAR start, TCHAR stop, int strip=1);
```

For instance, quotes, or '(' and ')'.  Returns 0 on success, -1 if there is no memory left.  If strip == 1, then the preserve designators will be stripped from the tokens returned by next.

```
LPTSTR next (void);
```

Returns the next token.

```
MAX_PRESERVES=16 }; protected: int is_delimiter (
    TCHAR d,
    int &replace,
    TCHAR &r
    );
```

Returns 1 if d is a delimiter, 0 otherwise.  If d should be replaced with r, replace is set to 1, otherwise 0.

```
int is_preserve_designator (TCHAR start, TCHAR &stop, int &strip);
```

If start is a start preserve designator, returns 1 and sets stop to the stop designator.  Returns 0 if start is not a preserve designator.

## PRIVATE MEMBERS

```
LPTSTR buffer_;
```

```
int index_;
```

```
Preserve_Entry preserves_[MAX_PRESERVES];
```

The application can specify MAX\_PRESERVES preserve designators.

```
int preserves_index_;
```

Pointer to the next free spot in preserves\_.

```
Delimiter_Entry delimiters_[MAX_DELIMITERS];
```

The tokenizer allows MAX\_DELIMITERS number of delimiters.

```
int delimiter_index_;
```

Pointer to the next free space in delimiters\_.

## AUTHOR

Douglas C. Schmidt (schmidt@cs.wustl.edu)

## LIBRARY

ace