README.md

PICCL: Philosophical Integrator of Computational and Corpus Libraries

PICCL offers a workflow for corpus building and builds on a variety of tools. The primary component of PICCL is TICCL; a Text-induced Corpus Clean-up system, which performs spelling correction and OCR post-correction (normalisation of spelling variants etc).

PICCL and TICCL constitute original research by Martin Reynaert (Tilburg University & Radboud University Nijmegen), and is currently developed in the scope of the CLARIAH project.

This repository hosts the relevant workflows that constitute PICCL, powered by Nextflow. These will be shipped as part of our LaMachine software distribution. The combination of these enable the PICCL workflow to be portable and scalable; it can be executed accross multiple computing nodes on a high performance cluster such as SGE, LSF, SLURM, PBS, HTCondor, Kubernetes and Amazon AWS. Parallellisation is handled automatically. Consult the Nextflow documentation for details regarding this.

All the modules that make up TICCL are part of the TicclTools collection, and are not part of the current repository. Certain other required components are in the FoLiA-Utils collection. There is no need to install either of these or other dependencies manually.

PICCL makes extensive use of the FoLiA format, a rich XML-based format for linguistic annotation.

Important Note: This is beta software still in development; for the old and deprecated version consult this repository.

Installation

PICCL is shipped as a part of LaMachine, although you need to explicitly select it for installation using lamachine-add piccl && lamachine-update (from inside a LaMachine installation). Once inside LaMachine, the command line interface can be invoked by directly specifying one of the workflows:

$ ocr.nf

Or

$ ticcl.nf

If you using a LaMachine installation, you can skip the rest of this section. If not, you can install Nextflow and Docker manually and then run the following to obtain the latest development release of PICCL:

$ nextflow pull LanguageMachines/PICCL

In this case you need to ensure to always run it with the -with-docker proycon/lamachine:piccl parameter, this lets nextflow manage your LaMachine docker container (this is not tested as much as running from inside the container directly):

$ nextflow run LanguageMachines/PICCL -with-docker proycon/lamachine:piccl

We have prepared PICCL for work in many languages, mainly on the basis of available open source lexicons due to Aspell, these data files serve as the input for TICCL and have to be downloaded once as follows;

$ nextflow run LanguageMachines/PICCL/download-data.nf -with-docker proycon/lamachine:piccl

This will generate a data/ directory in your current directory, and will be referenced in the usage examples in the next section. In a LaMachine environment, this directory is already available in $LM_PREFIX/opt/PICCL/data.

In addition, you can also download example corpora (>300MB), which will be placed in a corpora/ directory:

$ nextflow run LanguageMachines/PICCL/download-examples.nf -with-docker proycon/lamachine:piccl

Architecture

PICCL consists of two workflows, one for optical character recognition using tesseract, and a TICCL workflow for OCR-post-correction and normalisation. Third, PICCL provides a webservice that ties together both these workflows and also integrates two other workflows from aNtiLoPe: a workflow for tokenisation (using ucto) and Dutch Linguistic Enrichment (using frog).

The architecture of the PICCL webservice, and its two integral workflows, is visualised schematically as follows:

Usage

Command line interface

PICCL encompasses two workflows (and in webservice form it also integrates two more from aNtiLoPe)

• ocr.nf - A pipeline for Optical Character Recognition using Tesseract; takes PDF documents or images of scanned pages and produces FoLiA documents.
• ticcl.nf - The Text-induced Corpus Clean-up system: performs OCR-postcorrection, takes as input the result from ocr.nf, or standalone text or PDF (text; no OCR), and produces further enriched FoLiA documents.

If you are inside LaMachine, you can invoke these directly. If you let Nextflow manage LaMachine through docker, then you have to invoke them like nextflow run LanguageMachines/PICCL/ocr.nf -with-docker proycon/lamachine:piccl. This applies to all examples in this section.

Running with the --help parameter or absence of any parameters will output usage information.

$ ocr.nf --help
--------------------------
OCR Pipeline
--------------------------
Usage:
  ocr.nf [PARAMETERS]

Mandatory parameters:
  --inputdir DIRECTORY     Input directory
  --language LANGUAGE      Language (iso-639-3)

Optional parameters:
--inputtype STR          Specify input type, the following are supported:
        pdf (extension *.pdf)  - Scanned PDF documents (image content) [default]
        tif ($document-$sequencenumber.tif)  - Images per page (adhere to the naming convention!)
        jpg ($document-$sequencenumber.jpg)  - Images per page
        png ($document-$sequencenumber.png)  - Images per page
        gif ($document-$sequencenumber.gif)  - Images per page
        djvu (extension *.djvu)"
        (The hyphen delimiter may optionally be changed using --seqdelimiter)
--outputdir DIRECTORY    Output directory (FoLiA documents)
--virtualenv PATH        Path to Python Virtual Environment to load (usually path to LaMachine)
--pdfhandling reassemble Reassemble/merge all PDFs with the same base name and a number suffix; this can
                         for instance reassemble a book that has its chapters in different PDFs.
                         Input PDFs must adhere to a \$document-\$sequencenumber.pdf convention.
                         (The hyphen delimiter may optionally be changed using --seqdelimiter)
--seqdelimiter           Sequence delimiter in input files (defaults to: _)
--seqstart               What input field is the sequence number (may be a negative number to count from the end), default: -2


$ ticcl.nf --help
--------------------------
TICCL Pipeline
--------------------------
Usage:
  ticcl.nf [OPTIONS]

Mandatory parameters:
  --inputdir DIRECTORY     Input directory (FoLiA documents with an OCR text layer)
  --lexicon FILE           Path to lexicon file (*.dict)
  --alphabet FILE          Path to alphabet file (*.chars)
  --charconfus FILE        Path to character confusion list (*.confusion)

Optional parameters:
  --outputdir DIRECTORY    Output directory (FoLiA documents)
  --language LANGUAGE      Language
  --extension STR          Extension of FoLiA documents in input directory (default: folia.xml)
  --inputclass CLASS       FoLiA text class to use for input, defaults to 'current' for FoLiA input; must be set to 'OCR' for FoLiA documents produced by ocr.nf
  --inputtype STR          Input type can be either 'folia' (default), 'text', or 'pdf' (i.e. pdf with text; no OCR)
  --virtualenv PATH        Path to Virtual Environment to load (usually path to LaMachine)
  --artifrq INT            Default value for missing frequencies in the validated lexicon (default: 10000000)
  --distance INT           Levenshtein/edit distance (default: 2)
  --clip INT               Limit the number of variants per word (default: 10)
  --corpusfreqlist FILE    Corpus frequency list (skips the first step that would compute one for you)
  --low INT                skip entries from the anagram file shorter than 'low' characters. (default=5)
  --high INT               skip entries from the anagram file longer than 'high' characters. (default=35)
  --chainclean BOOLINT     enable chain clean or not (1 = on, 0 = off, default)

An example of invoking an OCR workflow for English is provided below, it assumes the sample data are installed in the corpora/ directory. It OCRs the OllevierGeets.pdf file, which contains scanned image data, therefore we choose the pdfimages input type.

$ ocr.nf --inputdir corpora/PDF/ENG/ --inputtype pdfimages --language eng

Alternative input types are images per page, in which case inputtype is set to either tif, jpg, gif or png. These input files should be placed in the designated input directory and follow the naming convention $documentname-$sequencenumber.$extension, for example harrypotter-032.png. An example invocation on dutch scanned pages in the example collection would be:

$ ocr.nf --inputdir corpora/TIFF/NLD/ --inputtype tif --language nld

In case of the first example the result will be a file OllevierGeets.folia.xml in the ocr_output/ directory. This in turn can serve as input for the TICCL workflow, which will attempt to correct OCR errors. Take care that that the --inputclass OCR parameter is mandatory if you want to use the FoLiA output of ocr.nf as input for TICCL:

$ ticcl.nf --inputdir ocr_output/ --inputclass OCR --lexicon $LM_PREFIX/opt/PICCL/data/int/eng/eng.aspell.dict --alphabet $LM_PREFIX/opt/PICCL/data/int/eng/eng.aspell.dict.lc.chars --charconfus $LM_PREFIX/opt/PICCL/data/int/eng/eng.aspell.dict.c0.d2.confusion

Note that here we pass a language-specific lexicon file, alphabet file, and character confusion file from the data files obtained by download-data.nf. Result will be a file OllevierGeets.folia.ticcl.xml in the ticcl_output/ directory, containing enriched corrections. The second example, on the dutch corpus data, can be run as follows:

$ ticcl.nf --inputdir ocr_output/ --inputclass OCR --lexicon $LM_PREFIX/opt/PICCL/data/int/nld/nld.aspell.dict --alphabet $LM_PREFIX/opt/PICCL/data/int/nld/nld.aspell.dict.lc.chars --charconfus $LM_PREFIX/opt/PICCL/data/int/nld/nld.aspell.dict.c20.d2.confusion

Webapplication / RESTful webservice

Installation

PICCL is also available as a webapplication and RESTful webservice, powered by CLAM. If you are in LaMachine with PICCL, the webservice is already installed, but you may need to run lamachine-start-webserver if it is not already running.

For production environments, you will want to adapt the CLAM configuration. To this end, copy $LM_PREFIX/etc/piccl.config.yml to $LM_PREFIX/etc/piccl.$HOST.yml, where $HOST corresponds with your hostname and edit the file with your host specific settings. Always enable authentication if your server is world-accessible (consult the CLAM documentation to read how).

Technical Details & Contributing

Please see CONTRIBUTE.md for technical details and information on how to contribute.