Tutorials
=========

.. meta::
    :description lang=en:
        Trafilatura tutorials: build web corpora, generate frequency lists, validate TEI files, and more.


Learn through practical examples. The following tutorials cover various scenarios, from text embedding for vector search to building custom web corpora and generating word frequency lists.


.. toctree::
   :maxdepth: 2

   tutorial-corpus
   tutorial-frequencies
   tutorial-tei
   tutorial-dwds
   tutorial-chromadb
   tutorial-rag
   tutorial-datasets



Blog posts
^^^^^^^^^^

- `Extracting the main text content from web pages using Python <https://adrien.barbaresi.eu/blog/trafilatura-main-text-content-python.html>`_
- `Validating TEI-XML documents with Python <https://adrien.barbaresi.eu/blog/validating-tei-xml-python.html>`_
- `Evaluating scraping and text extraction tools for Python <https://adrien.barbaresi.eu/blog/evaluating-text-extraction-python.html>`_
- `Filtering links to gather texts on the web <https://adrien.barbaresi.eu/blog/link-filtering-courlan-python.html>`_
- `Using sitemaps to crawl websites on the command-line <https://adrien.barbaresi.eu/blog/using-sitemaps-crawl-websites.html>`_
- `Using RSS and Atom feeds to collect web pages with Python <https://adrien.barbaresi.eu/blog/using-feeds-text-extraction-python.html>`_
- `Web scraping with R: Text and metadata extraction <https://adrien.barbaresi.eu/blog/web-scraping-text-metadata-r.html>`_
- `Web scraping with Trafilatura just got faster <https://adrien.barbaresi.eu/blog/web-scraping-trafilatura-faster.html>`_


Videos
^^^^^^

Youtube playlist with video tutorials in several languages
    `Web scraping how-tos and tutorials <https://www.youtube.com/watch?v=8GkiOM17t0Q&list=PL-pKWbySIRGMgxXQOtGIz1-nbfYLvqrci>`_.


.. raw:: html

    <iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/rEOoItpzlVw" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>



External resources
^^^^^^^^^^^^^^^^^^

- `GLAM-Workbench <https://github.com/GLAM-Workbench>`_
   - `Harvesting collections of text from archived web pages <https://github.com/GLAM-Workbench/web-archives/blob/master/getting_text_from_web_pages.ipynb>`_
   - `Compare two versions of an archived web page <https://github.com/GLAM-Workbench/web-archives/blob/master/show_diffs.ipynb>`_


- `User Ethics & Legal Concerns <https://melaniewalsh.github.io/Intro-Cultural-Analytics/Data-Collection/User-Ethics-Legal-Concerns.html>`_
- `Download von Web-Daten <https://www.bubenhofer.com/korpuslinguistik/kurs/index.php?id=eigenes_wwwdownload.html>`_ & `Daten aufbereiten und verwalten <https://www.bubenhofer.com/korpuslinguistik/kurs/index.php?id=eigenes_aufbereitenXML.html>`_ (Tutorials in German by Noah Bubenhofer)