---
name: "urlgrabber"
section: "1"
platform: "linux"
package: "python-urlgrabber"
version: "3.10.2-1"
release: "Debian 10 (Buster)"
platforms: ["linux"]
url: "https://new.carta.tech/urlgrabber"
---

# urlgrabber(1)

a high-level cross-protocol url-grabber.

## NAME

urlgrabber - a high-level cross-protocol url-grabber.

## SYNOPSIS

*urlgrabber* \[OPTIONS\] URL \[FILE\]

## DESCRIPTION

urlgrabber is a binary program and python module for fetching files. It is designed to be used in programs that need common (but not necessarily simple) url-fetching features.

## OPTIONS

- \--help, -h

  help page specifying available options to the binary program.

- \--copy-local

  ignored except for file:// urls, in which case it specifies whether urlgrab should still make a copy of the file, or simply point to the existing copy.

- \--throttle=NUMBER

  if it's an int, it's the bytes/second throttle limit. If it's a float, it is first multiplied by bandwidth. If throttle == 0, throttling is disabled. If None, the module-level default (which can be set with set\_throttle) is used.

- \--bandwidth=NUMBER

  the nominal max bandwidth in bytes/second. If throttle is a float and bandwidth == 0, throttling is disabled. If None, the module-level default (which can be set with set\_bandwidth) is used.

- \--range=RANGE

  a tuple of the form first\_byte,last\_byte describing a byte range to retrieve. Either or both of the values may be specified. If first\_byte is None, byte offset 0 is assumed. If last\_byte is None, the last byte available is assumed. Note that both first and last\_byte values are inclusive so a range of (10,11) would return the 10th and 11th bytes of the resource.

- \--user-agent=STR

  the user-agent string provide if the url is HTTP.

- \--retry=NUMBER

  the number of times to retry the grab before bailing. If this is zero, it will retry forever. This was intentional... really, it was :). If this value is not supplied or is supplied but is None retrying does not occur.

- \--retrycodes

  a sequence of errorcodes (values of e.errno) for which it should retry. See the doc on URLGrabError for more details on this. retrycodes defaults to -1,2,4,5,6,7 if not specified explicitly.

## MODULE USE EXAMPLES

In its simplest form, urlgrabber can be a replacement for urllib2's open, or even python's file if you're just reading:

```
  from urlgrabber import urlopen
  fo = urlopen(url)
  data = fo.read()
  fo.close()
```

Here, the url can be http, https, ftp, or file. It's also pretty smart so if you just give it something like /tmp/foo, it will figure it out. For even more fun, you can also do:

```
  from urlgrabber import urlopen
  local_filename = urlgrab(url)  # grab a local copy of the file
  data = urlread(url)            # just read the data into a string
```

Now, like urllib2, what's really happening here is that you're using a module-level object (called a grabber) that kind of serves as a default. That's just fine, but you might want to get your own private version for a couple of reasons:

```
* it's a little ugly to modify the default grabber because you have to
  reach into the module to do it
* you could run into conflicts if different parts of the code
  modify the default grabber and therefore expect different
  behavior
```

Therefore, you're probably better off making your own. This also gives you lots of flexibility for later, as you'll see:

```
  from urlgrabber.grabber import URLGrabber
  g = URLGrabber()
  data = g.urlread(url)
```

This is nice because you can specify options when you create the grabber. For example, let's turn on simple reget mode so that if we have part of a file, we only need to fetch the rest:

```
  from urlgrabber.grabber import URLGrabber
  g = URLGrabber(reget='simple')
  local_filename = g.urlgrab(url)
```

The available options are listed in the module documentation, and can usually be specified as a default at the grabber-level or as options to the method:

```
from urlgrabber.grabber import URLGrabber
g = URLGrabber(reget='simple')
local_filename = g.urlgrab(url, filename=None, reget=None)
```

## AUTHORS

Written by: Michael D. Stenner \<mstenner@linux.duke.edu\> Ryan Tomayko \<rtomayko@naeblis.cx\>

This manual page was written by Kevin Coyner \<kevin@rustybear.com\> for the Debian system (but may be used by others). It borrows heavily on the documentation included in the urlgrabber module. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU General Public License, Version 2 any later version published by the Free Software Foundation.

## RESOURCES

Main web site: *http://linux.duke.edu/projects/urlgrabber/*