# Esperanto Newspaper Excerpts

This repository is part of the project _Esperanto Newspaper Excerpts_
funded by the [Austrian Academy of
Sciences](https://www.oeaw.ac.at/en/) through
[CLARIAH-AT](https://clariah.at/about-clariah-at/). The project's aim
is to make searchable a collection of newspaper excerpts about the
planned language Esperanto. These excerpts have recently (2022) come
into possession of the [Austria National
Library's](https://www.onb.ac.at/en/) [Department of Planned
Languages](https://www.onb.ac.at/en/departments/department-of-planned-languages).

## Problem Statement

In order to make a collection searchable, it has to be transformed
from a purely visual data format such as images to text. The whole
collection is currently only available in a scanned image format and
images are by nature not searchable. Through the application of
Optical Character Recognition (OCR) these images will have searchable
text attached to them.

This dataset is challenging to apply OCR to because it is very
heterogeneous. There are a lot of individual strips/columns glued onto
a background paper. These strips are at times rotated or cut off
partially or have some other deformity. OCR-Programs usually at least
expect lines to be horizontal and have their own peculiar preferences
when it comes to the color of the background, contrast requirements,
resolution and file format. Furthermore, some scans contain
full-fledged newspaper front pages which also contain ads, multiple
articles in one column, artsy headlines and in general a quite complex
layout.

## Methods

Since the layout is so unique, this repository implements a two-stage
approach:

1. Detect all blocks of text in the scanned image.
2. Run OCR on the individual text blocks.

This approach has the benefit that the layout of the newspaper can be
as complex as it wants to be with additional ornamentations and other
typographic goodness without affecting the accuracy with which the
text is recognized (at least in theory).

### Block Detection

We chose to do block detection using a machine learning approach with
the object detection/segmentation model
[YOLOv8](https://github.com/ultralytics/ultralytics). YOLOv8 not only
draws bounding boxes around detected objects, but can also do instance
segmentation. Instance segmentation differs from pure object detection
in that the response from the network is not a bounding box with 90°
angles but a mask which either has a 0 if the pixel does not contain
the object of interest or a 1 if it does. This mask allows us to
recognize not only rectangular shapes but also skewed ones. Since a
lot of text blocks are rotated, instance segmentation gives us a
better prediction than bounding boxes.

The mask is used to cut out the individual text blocks from the
original image. The cutout is then processed by ocr/deskew-text> to
derotate the text block and is further passed to the text recognition
engine.

### Text Recognition

We have tried several engines to do text recognition, but most do not
work on smaller languages such as Esperanto because they have not
explicitly been trained on them. The only engine which has support for
Esperanto is
[Tesseract](https://github.com/tesseract-ocr/tesseract). Most OCR
engines work on lines of text and thus have to preprocess the data. In
our case, individual text blocks are sent to tesseract and the output
formats are specified as `TXT` and `ALTO-XML`.

Tesseract takes a parameter `--psm` which specifies the page
segmentation mode. In our tests, this parameter had a significant
impact on the recognition accuracy. As of now, all images are by
default processed with `--psm 12` (Sparse text with OSD) and—if the
response contains the string _Too few characters_—given a second pass
with `--psm 7` (Treat the image as a single text line). The second
pass is necessary for cutouts which only contain headlines and are
thus usually single lines. Single lines are almost never properly
recognized using `--psm 12`.

## Getting Started

It is first necessary to register the directory as an editable package
with `pip` so that the relative imports work:

```bash
# From the top-level directory
pip install -e .
```

The repository is organized in two main directories:

- _datasets_ which contains all datasets used to train the models or
  for inference. The whole newspaper collection would go in that
  directory.
- _src_ which contains the code necessary for training, inference,
  preprocessing and postprocessing.

Each directory should generally have its own README so that it is
clear how the repository is organized.

## Download training data and best model weights

The training data (200 annotated images with labels) and resulting best weights (including evaluation scores) for the YOLOv8 model, which are not part of the repository, can be downloaded here:

- [Training dataset](https://labs.onb.ac.at/static/media/esperanto-excerpts/training_dataset.zip)

- [Model weights](https://labs.onb.ac.at/static/media/esperanto-excerpts/best_weights.zip)
