Tutorial: Building a training corpus with Hugging Face Datasets

Why build a corpus this way?

Trafilatura is already used to build large-scale training corpora for language models, for instance the Allen Institute for AI’s Dolma toolkit, HuggingFace’s own DataTrove, and the RefinedWeb dataset behind the Falcon LLM. See the uses & citations page for more examples.

This tutorial shows a small-scale version of the same idea: crawl a set of pages, extract clean text and metadata with Trafilatura, and assemble the result into a Hugging Face Dataset that can be filtered, saved, and later used for training or fine-tuning.

Setup

$ pip install -U datasets

Crawl, extract, and collect metadata

For a training corpus you typically want more than the bare text: language, title, and date are useful for filtering and deduplication later on. bare_extraction() returns all of this in one call.

from trafilatura import fetch_url, bare_extraction

urls = [
    'https://www.tensorflow.org/',
    'https://pytorch.org/',
    'https://getbootstrap.com/',
]

records = []
for url in urls:
    downloaded = fetch_url(url)
    doc = bare_extraction(downloaded, url=url, with_metadata=True, target_language="en")
    if doc and doc.text:
        records.append({
            "url": url,
            "title": doc.title,
            "date": doc.date,
            "language": doc.language,
            "text": doc.text,
        })

print(f'{len(records)} pages extracted')

Hint

For a real corpus, gather your input URLs with sitemaps or feeds, consider courlan for link filtering, and use buffered downloads instead of looping over fetch_url() to crawl many pages efficiently.

Assemble and inspect the dataset

from datasets import Dataset

dataset = Dataset.from_list(records)
print(dataset)
print(dataset[0]["title"], "—", len(dataset[0]["text"]), "characters")

Dataset objects support the usual filtering, mapping, and shuffling operations, for example to drop very short pages before training:

dataset = dataset.filter(lambda row: len(row["text"]) > 500)

Save or share the dataset

Save it locally in a format that loads back instantly:

dataset.save_to_disk("trafilatura_corpus")

# later on
from datasets import load_from_disk
dataset = load_from_disk("trafilatura_corpus")

Or push it to the Hugging Face Hub to share it (this requires being logged in with huggingface-cli login):

dataset.push_to_hub("your-username/trafilatura-corpus")

See also

Tutorial: Building a web corpus, Deduplication, Uses & citations