---
name: "libwget-robots"
section: "3"
platform: "linux"
package: "wget2-dev"
version: "2.2.0+ds-1+deb13u1"
release: "Debian 13 (Trixie)"
platforms: ["linux"]
url: "https://new.carta.tech/libwget-robots"
---

# libwget-robots(3)

Robots Exclusion file parser

## NAME

libwget-robots - Robots Exclusion file parser

## SYNOPSIS

### Data Structures

struct **wget\_robots\_st**\

### Macros

\#define **parse\_record\_field**(d,  f)   parse\_record\_field(d, f, sizeof(f) - 1)\

### Functions

int **wget\_robots\_parse** (**wget\_robots** \*\*\_robots, const char \*data, const char \*client)\
 void **wget\_robots\_free** (**wget\_robots** \*\*robots)\
 int **wget\_robots\_get\_path\_count** (**wget\_robots** \*robots)\
 **wget\_string** \* **wget\_robots\_get\_path** (**wget\_robots** \*robots, int index)\
 int **wget\_robots\_get\_sitemap\_count** (**wget\_robots** \*robots)\
 const char \* **wget\_robots\_get\_sitemap** (**wget\_robots** \*robots, int index)\

## Detailed Description

The purpose of this set of functions is to parse a Robots Exclusion Standard file into a data structure for easy access.

## Function Documentation

### int wget\_robots\_parse (wget\_robots \*\* \_robots, const char \* data, const char \* client)

**Parameters**

*data* Memory with robots.txt content (with trailing 0-byte) \
 *client* Name of the client / user-agent

**Returns**

Return an allocated wget\_robots structure or NULL on error

The function parses the robots.txt `data` in accordance to https://www.robotstxt.org/orig.html#format and returns a ROBOTS structure including a list of the disallowed paths and including a list of the sitemap files.

The ROBOTS structure has to be freed by calling **wget\_robots\_free()**.

### void wget\_robots\_free (wget\_robots \*\* robots)

**Parameters**

*robots* Pointer to Pointer to wget\_robots structure

**wget\_robots\_free()** free's the formerly allocated wget\_robots structure.

### int wget\_robots\_get\_path\_count (wget\_robots \* robots)

**Parameters**

*robots* Pointer to instance of wget\_robots

**Returns**

Returns the number of paths listed in `robots`

### wget\_string \* wget\_robots\_get\_path (wget\_robots \* robots, int index)

**Parameters**

*robots* Pointer to instance of wget\_robots \
 *index* Index of the wanted path

**Returns**

Returns the path at `index` or NULL

### int wget\_robots\_get\_sitemap\_count (wget\_robots \* robots)

**Parameters**

*robots* Pointer to instance of wget\_robots

**Returns**

Returns the number of sitemaps listed in `robots`

### const char \* wget\_robots\_get\_sitemap (wget\_robots \* robots, int index)

**Parameters**

*robots* Pointer to instance of wget\_robots \
 *index* Index of the wanted sitemap URL

**Returns**

Returns the sitemap URL at `index` or NULL

## Author

Generated automatically by Doxygen for wget2 from the source code.