The HTTrack interface, step by step

Accept addresses matching the pattern, even ones the engine would otherwise leave alone. +*.example.com/*.jpg takes the JPEGs from a neighbouring host; +*/images/landscapes/* takes one directory wherever it appears.

Refuse addresses matching the pattern. -*.zip skips the archives, -*/forum/* skips a section you do not want.

Reads rules from a text file, one per line, instead of the box.

How many clicks from your starting addresses the engine may travel. A depth of 3 means the pages you listed, plus everything within two more clicks. Left empty it is effectively unlimited, which is usually right: the engine already refuses to leave the site.

How far to follow links off the site, past what the scan rules allow. It overrides the other limits, so raise it with care. The default is zero.

Per-file ceiling in bytes: a larger image or archive is skipped.

The same for pages. One command-line option carries both ceilings, non-HTML first.

Total bytes the mirror may download before it stops.

Pauses once that many bytes have arrived and waits for you to delete a lock file. The way to mirror a site larger than your free space: back up and clear the files during the pause.

Waits a random number of seconds between downloads. 2:8 picks a fresh delay in that range each time. The gentlest way to crawl a small server, and less blunt than capping the rate.

Seconds the whole mirror may take. 3600 is an hour.

Bytes per second, across the whole mirror. Keeps HTTrack from taking the whole line.

New connections per second, and the politest thing on this page. Fractions are allowed: 0.1 is one connection every ten seconds. Setting it to zero removes the limit and can flatten a small server; do not, unless the server is yours.

How many addresses may be held in memory, downloaded or not. The engine stops dead on reaching it, so do not set it low. The default of 100,000 suits most sites and costs memory to raise.

Transfers running at once. Four is the default and eight is comfortable on an ordinary site; drop to one or two when the files are large, since parallel transfers of big files help nobody.

Seconds to wait on a silent server before abandoning the link. It also bounds host name resolution. Around 120 suits most connections.

How many times to retry after a timeout or another non-fatal error. It will not rescue a 404: nothing retries a definite answer.

Bytes per second below which a transfer is treated as stalled and dropped.

Whether one bad transfer condemns the whole host: never (0), on timeout (1), on a slow transfer (2), or either (3). The checkboxes beside the timeout and rate fields set this. Anything but never can drop a lot of links over one slow moment, so leave it off unless a host is actively wasting your time.

Looks for addresses everywhere, including unknown tags and JavaScript. On by default. It earns its keep on script-heavy pages, at the price of the occasional request for something that was never a link.

Fetches the images and other files a page refers to even when they sit outside the mirror. The usual fix for a page whose pictures live on another host.

Requests every link found, including ones the rules forbid, and logs the failures. A link checker rather than a mirroring option.

Takes the pages before the images. The structure of the site is known sooner, which makes the rest of the crawl better informed.

The default reproduces the site's own folders and names. The alternatives sort files by kind instead: all the pages here, all the images there. A custom pattern such as -N "%h%p/%n%q.%t" builds names from the host, path, name, query and type.

Cuts every name to eight characters and a three-letter extension.

Names that survive being burned to a CD or DVD.

By default a page that returned 404 is saved as a note saying so. This drops those, leaving nothing at all where the page was.

Replaces links that leave the mirror with a page saying you need to be online. Keeps the offline copy from silently reaching for the network.

Strips credentials out of the links written into the saved pages, so the copy cannot leak them.

Local files rarely need the ?a=1&b=2 part, though it can carry a hint about what a page was. Some limited browsers choke on it.

After an update HTTrack deletes local files that are gone from the site or now excluded. This keeps them.

Rewrites each saved page with its stylesheets, scripts, images and fonts embedded, so a single file opens anywhere on its own. Links between mirrored pages stay relative; audio and video stay links. --single-file-max-size caps each embedded asset, 10 MB by default.

The User-Agent sent with every request. Some sites serve different pages, or refuse to serve at all, depending on what they read here.

A comment added to each saved page, recording where it came from. Named fields are substituted: {url}, {date}, {addr}, {path}, {lastmodified}, {version}, {mime}, {charset}, {status} and {size}. For example <!-- Mirrored from {url} on {date} -->. You can turn it off, but a page with no record of its origin is worth less later.

The Accept-Language header, as in "fr, en, *". It decides which translation a multilingual site hands over.

The Referer sent with the first request. Occasionally needed by sites that refuse traffic arriving from nowhere.

On by default. Sites that hand out a session before showing anything need it.

When the address does not reveal the file type, say a .cgi that returns an image, the engine asks the server so the file gets a sensible name locally. Turning this off entirely produces a mirror full of files no browser will open.

Reads scripts looking for addresses. On by default.

Whether to respect the site's crawling rules: never (0), sometimes (1), always (2, the default), or always including the strict rules (3). Ignoring them on a site you do not own is how mirroring gets people blocked.

Works around servers that misreport what changed, treating a file of identical size as unchanged even when the timestamp moved. Saves a great deal of traffic on dynamic sites, at a small risk of keeping a stale page.

Accepts responses that break the rules. Off by default, because a server that lies about a file's length usually leaves you with a truncated one.

For servers and proxies too old to cope with anything newer. It costs you most of the update machinery, so use it only when a site genuinely fails without it.

Treats addresses that differ only cosmetically as the same page (www.example.com and example.com, doubled slashes, reordered query keys), so the same page is not downloaded twice.

Three checkboxes opt out of one part each when a site really does treat the difference as meaningful: Keep the www. prefix (--keep-www-prefix), Keep double slashes (--keep-double-slashes) and Keep the original query-string order (--keep-query-order).

Reads the site's sitemap, the Sitemap: lines in robots.txt and then /sitemap.xml, and adds every address it lists as a starting point. It reaches pages no link on the site points to. Sitemap address (--sitemap-url) names one directly instead of probing.

Preloads cookies from a Netscape cookies.txt before the crawl starts. How you mirror something that needs you logged in, by exporting the session from your browser.

Comma-separated query keys to drop when naming saved files, such as sid,utm_source. Keeps one page from being saved several times under tracking parameters that change nothing.

The proxy protocol. HTTP is an ordinary proxy. HTTP (CONNECT tunnel) sends every request through a CONNECT tunnel, which CONNECT-only proxies such as Tor's HTTPTunnelPort require. SOCKS5 defaults to port 1080.

The proxy to send requests through. On the command line one string carries the lot, including a scheme and credentials where needed: --proxy socks5://user:pass@proxy.example.com:1080.

Filled in behind the Configure button, for a proxy that demands authentication.

not on Android The Android app takes a host and a port only.

Sends ftp:// transfers through the HTTP proxy as well. On by default, and worth keeping: it gets FTP links through a firewall, and proxied FTP tends to be more reliable than the engine's own client.

Normally only pages are cached, since that is all an update needs. This caches everything, which lets you rebuild the mirror in a different layout later without downloading it again, and makes the cache as large as the mirror.

A file you deleted stays deleted through the next update, instead of coming straight back. Useful while pruning something large by hand.

On, and worth leaving on: without a log you have no way to find out what failed. The debug level beside it controls how much detail lands there.

Writes an index.html at the top of the project, listing the mirrored sites. The front door to the copy.

Builds a searchable word index of every mirrored page.

Saves every fetched response into an ISO-28500 WARC/1.1 archive alongside the mirror, headers and all. The mirror is a browsable copy; the WARC is the archival record of what the server actually sent, which is what replay tools and web archives want.

WARC archive name (--warc-file) sets the base name, blank to auto-name it under the output directory. WARC segment size limit (--warc-max-size) starts a new segment past that many bytes; blank or 0 keeps one file.

Adds a sorted CDXJ index listing every record in the archive, which replay tools use to find a capture without reading the whole file.

Bundles the WARC, its CDXJ index and the mirrored pages into one WACZ file. It implies both of the two above.

Writes hts-changes.json listing what this crawl left new, changed, unchanged and gone against the previous mirror. The quick way to see what an update actually did.

Tells the engine what an extension always means, so it stops asking the server. On a site full of .asp links this is a large speed-up: without it every unknown extension costs a request before the engine knows what it has.

Enter a file type and the MIME type it maps to, asp to text/html, and list several at once with commas: asp,php,php3. On the command line the same thing reads --assume asp,php,php3=text/html, and --assume standard covers the usual page extensions in one go.

It also renames on the way in. If you know the site's .dat files are really ZIPs, map dat to application/x-zip and they land correctly named. Mapping an unknown type to application/octet-stream simply stops the engine checking it.

text/html	pages, which HTTrack parses for links
image/gif, image/jpeg, image/png	images
application/x-zip	ZIP archives
application/octet-stream	anything else, left alone

Android The table holds up to eight entries.

Keep this on. The cache is what makes Update existing download and Continue interrupted download possible at all; turning it off saves a little disk and costs you both.

What gets saved: nothing (0, for checking links), pages only (1), everything but pages (2), everything (3, the default), or pages first then the rest (7).

Which directories the crawl may enter relative to your starting address: this one only (-S), it and below (-D, the default), it and above (-U), or both directions (-B).

How far the crawl may stray from the address you gave it: same address only (-a, the default), same domain (-d), same top-level domain (-l), or anywhere at all (-e). The last one will follow the web until a limit stops it.

What the links in the saved pages look like: relative, so the copy browses offline (K0, the default), absolute, or the original online addresses (K4), which gives you the pages as archived documents rather than a browsable mirror.

Writes the HTTP headers and other internals to the log. For diagnosing a site that behaves strangely, not for normal use.