On the command-line#


Trafilatura offers a robust command-line interface and can be conveniently used without writing code. Learn how to perform various tasks and leverage the full power of the tool from the terminal.

For the very first steps please refer to this multilingual, step-by-step Introduction to the command-line interface and this section of the Introduction to Cultural Analytics & Python.

For instructions related to specific platforms see:


URLs can be used directly (-u/--URL):

# outputs main content and comments as plain text ...
$ trafilatura -u "https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/"

# outputs main text with basic XML structure ...
$ trafilatura --xml --URL "https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/"

# displays help message
$ trafilatura -h

You can also pipe a HTML document (and response body) to trafilatura:

# use the contents of an already existing file
$ cat myfile.html | trafilatura

# alternative syntax
$ < myfile.html trafilatura

# use a custom download utility and pipe it to trafilatura
$ wget -qO- "https://de.creativecommons.org/index.php/was-ist-cc/" | trafilatura

Extraction parameters#

Choice of HTML elements#

Several elements can be included or discarded (see list of options below):

  • Text elements: comments, tables

  • Structural elements: formatting, images, links

Only comments and text extracted from HTML <table> elements are extracted by default, --no-comments and --no-tables deactivate this setting.

Further options:


Keep structural elements related to formatting (<b>/<strong>, <i>/<emph> etc.)


Keep link targets (in href="..."), converting relative URLs to absolute where possible


Keep track of images along with their targets (<img> attributes: alt, src, title)


Certain elements are only visible in the output if the chosen format allows it (e.g. images and XML).

Including extra elements works best with conversion to XML/XML-TEI. If the output is buggy removing a constraint (e.g. formatting) can greatly improve the result.

Output format#

Output as TXT without metadata is the default, another format can be selected in two different ways:

  • --csv, --json, --xml or --xmltei

  • -out or --output-format {txt,csv,json,xml,xmltei}


Combining TXT, CSV and JSON formats with certain structural elements (e.g. formatting or links) triggers output in TXT+Markdown format.

Optimizing for precision and recall#

The arguments --precision & --recall can be passed to the extractor.

They slightly affect processing and volume of textual output, respectively concerning precision/accuracy (i.e. more selective extraction, yielding less and more central elements) and recall (i.e. more opportunistic extraction, taking more elements into account).

Language identification#

Passing the argument --target-language along with a 2-letter code (ISO 639-1) will trigger language filtering of the output if the identification component has been installed and if the target language is available.


Additional components are required: pip install trafilatura[all]

Changing default settings#

See documentation page on settings.

Process files locally#

In case web pages have already been downloaded and stored, it is possible to process single files or directories as a whole. It can be especially helpful to separate download and extraction to circumvent blocking mechanisms, either by scrambling IPs used to access the pages or by using web browser automation software to bypass issues related to cookies and paywalls.

Trafilatura will work as well provided web pages (HTML documents) are used as input. Two major command line arguments are necessary:

  • --input-dir to select a directory to read files from

  • -o or --output-dir to define a directory to eventually store the results


In case no directory is selected, results are printed to standard output (STDOUT, e.g. in the terminal window).

Further information#

For all usage instructions see trafilatura -h:

trafilatura [-h] [-i INPUTFILE | --input-dir INPUTDIR | -u URL]
               [--parallel PARALLEL] [-b BLACKLIST] [--list]
               [-o OUTPUTDIR] [--backup-dir BACKUP_DIR] [--keep-dirs]
               [--hash-as-name] [--feed [FEED] | --sitemap [SITEMAP] |
               --crawl [CRAWL] | --explore [EXPLORE]] [--archived]
               [--url-filter URL_FILTER [URL_FILTER ...]] [-f]
               [--formatting] [--links] [--images] [--no-comments]
               [--no-tables] [--only-with-metadata]
               [--target-language TARGET_LANGUAGE] [--deduplicate]
               [--config-file CONFIG_FILE]
               [-out {txt,csv,json,xml,xmltei} | --csv | --json | --xml | --xmltei]
               [--validate-tei] [-v] [--version]

Command-line interface for Trafilatura

optional arguments:
-h, --help

show this help message and exit

-v, --verbose

increase logging verbosity (-v or -vv)


show version information and exit


URLs, files or directories to process

-i INPUTFILE, --input-file INPUTFILE

name of input file for batch processing

--input-dir INPUTDIR

read files from a specified directory (relative path)


custom URL download

--parallel PARALLEL

specify a number of cores/threads for downloads and/or processing


file containing unwanted URLs to discard during processing


Determines if and how files will be written


display a list of URLs without downloading them

-o OUTPUTDIR, --output-dir OUTPUTDIR

write results in a specified directory (relative path)

--backup-dir BACKUP_DIR

preserve a copy of downloaded files in a backup directory


keep input directory structure and file names


use hash value as output file name instead of random default


Link discovery and web crawling

--feed URL

look for feeds and/or pass a feed URL as input

--sitemap URL

look for sitemaps for the given website and/or enter a sitemap URL

--crawl URL

crawl a fixed number of pages within a website starting from the given URL

--explore URL

explore the given websites (combination of sitemap and crawl)


try to fetch URLs from the Internet Archive if downloads fail

--url-filter URL_FILTER

only process/output URLs containing these patterns (space-separated strings)


Customization of text and metadata processing

-f, --fast

fast (without fallback detection)


include text formatting (bold, italic, etc.)


include links along with their targets (experimental)


include image sources in output (experimental)


don’t output any comments


don’t output any table elements


only output those documents with title, URL and date (for formats supporting metadata)

--target-language TARGET_LANGUAGE

select a target language (ISO 639-1 codes)


filter out duplicate documents and sections

--config-file CONFIG_FILE

override standard extraction parameters with a custom config file


favor extraction precision (less noise, possibly less text)


favor extraction recall (more text, possibly more noise)


Selection of the output format

-out, --output-format

determine output format, possible choices: txt, csv, json, xml, xmltei


CSV output


JSON output


XML output


XML TEI output


validate XML TEI output