4 - Webarchive

https://webarchiv.onb.ac.at

https://labs.onb.ac.at/dataset/webarchive/

In this block

• Overview Webarchive
• Overview Content
• Overview API

• Example: Interacting with the API
• Example: Wayback search via API
• Example: Full text search via API
• Example: Download preview SVG thumb of saved page

Overview Webarchive

https://webarchiv.onb.ac.at

What is the Webarchive Austria?

• Attempt to conserve online data for future generations
• Webarchive Austria crawls officially since March 2009
• All domains within .at, .ac.at, .gv.at, .wien, .tirol
• Selected other domains with 'Austrian content'
• About 2 million websites saved

Who is the Webarchive Austria?

• Andreas Predikaka
• webarchiv@onb.ac.at

What can I use?

• Websites: no public access
  • Access on premises at the ÖNB
  • Exception: onb.ac.at
• Metadata: public access
• Full text search: public access
  • Viewing the results of the full text search: no public access

What does that mean?

• Searching outside the ÖNB gives you URLs, doesn't give you page content

What if I really really need to see the content?

• You can come to the ÖNB in person and use one of two offline computers...

• ...to PRINT OUT THE INTERNET!

How is a search where I don't see detailed results useful to me?

• Sometimes the content is still online
• Sometimes the Internet Archive has a copy
• You can observe the emergence of certain terms ('Westbalkanroute', 'Soldatna')

• ???

Overview Content

https://webarchiv.onb.ac.at

What's inside?

• High crawl frequency (daily or weekly)
  • Media sites (ORF)
  • Political parties
• Low crawl frequency (a few times per year)
  • Topic: Gender
  • Austrian domains (via nic.at)
• Event crawls (daily or weekly within a certain timespan)
  • Elections
  • Olympia
  • Refugee crisis 2015
  • Song Contest 2015

Can I have a list?

• Sure, there you go:
  • Media, political, gender: https://webarchiv.onb.ac.at/data/selective.json
  • Events: https://webarchiv.onb.ac.at/data/events.json
  • All domains: https://webarchiv.onb.ac.at/data/domainnames.json

[
  {
    "id": 37,
    "name": "Frau/Gender",
    "begin": "29.11.2016",
    "groups": [
      {
        "seeds": [
          "http://abtreibung.at/"
        ],
        "group_id": 1,
        "name": "Abtreibung.at"
      },
      {
        "seeds": [
          "http://aep.at"
        ],
        "group_id": 2,
        "name": "Arbeitskreis Emanzipation Partnerschaft"
      },

How big is the Austrian Webarchive?

• About 500GiB indexed text
• About 100 million HTML documents
• Raw data: 115.28TiB uncompressed

Where's the catch?

• Social media is currently too hard to crawl
• Limited disk space necessitates a size limit per page
  • Ex: domain crawl 10MB -> 100MB -> 7GB
• Limitations of public access
  • Practically every webarchive except the Internet Archive

Overview API

https://webarchiv.onb.ac.at

How can I access the Austrian Webarchive?

• On site at the ONB
• Online: https://webarchiv.onb.ac.at
• REST API: https://webarchiv.onb.ac.at/api.html
  • Swagger definition: https://webarchiv.onb.ac.at/api/swagger.json
• Python module for easier access: https://labs.onb.ac.at/gitlab/labs-team/webarchive-api/blob/master/webarchiv.py (raw)

Why is access via API useful?

• Individual searches may take up to 1 minute
• Sift through loads of metadata
• API-only goodies
  • Easily nominate pages with Austrian content to be saved
  • Download SVG thumbnails of rendered websites
• It's way more fun

• Make Andreas happy :)

Tracking

We'd love to properly count the unique visitors in the Webarchive backend, so we kindly ask you to opt in to tracking by instantiating webarchiv.WebarchivSession with the parameter allow_tracking=True.

This sends your SHA256-hashed MAC address as a fingerprint to the server on authentication. It is only ever used to count unique users.

If you leave allow_tracking at the default value False, an empty string is sent as fingerprint.