Tutorials

Learn through practical examples. The following tutorials cover various scenarios, from text embedding for vector search to building custom web corpora and generating word frequency lists.

• Tutorial: Building a web corpus
  • Get your system up and running
  • Content discovery
  • Link filtering
  • Process a list of links
• Tutorial: Word frequency lists from web pages
  • Get your system up and running
  • Process a list of links
  • Build frequency lists
• Tutorial: Validation of TEI files
  • Producing TEI files
  • Validating existing files
• Tutorial: DWDS-Korpusdaten reproduzieren
  • Ziel
  • Von einer Abfrage zur Einsicht der Quellen
  • Interesse und Gestaltungsmöglichkeiten
  • Download und Verarbeitung der Daten
• Tutorial: Text embedding with ChromaDB
  • Why perform text embedding with crawled data?
  • Setup Chroma
  • Crawl project homepages and store their vector embeddings in Chroma
  • Perform vector search
• Tutorial: Retrieval-augmented generation (RAG)
  • Why build a RAG pipeline with crawled data?
  • Setup
  • Crawl and extract a small corpus
  • Build the index
  • Query with retrieval-augmented generation
• Tutorial: Building a training corpus with Hugging Face Datasets
  • Why build a corpus this way?
  • Setup
  • Crawl, extract, and collect metadata
  • Assemble and inspect the dataset
  • Save or share the dataset

Blog posts

• Extracting the main text content from web pages using Python

• Validating TEI-XML documents with Python

• Evaluating scraping and text extraction tools for Python

• Filtering links to gather texts on the web

• Using sitemaps to crawl websites on the command-line

• Using RSS and Atom feeds to collect web pages with Python

• Web scraping with R: Text and metadata extraction

• Web scraping with Trafilatura just got faster

Videos

Youtube playlist with video tutorials in several languages

Web scraping how-tos and tutorials.

External resources

• GLAM-Workbench

  • Harvesting collections of text from archived web pages

  • Compare two versions of an archived web page

• User Ethics & Legal Concerns

• Download von Web-Daten & Daten aufbereiten und verwalten (Tutorials in German by Noah Bubenhofer)