Skip to content

Latest commit

 

History

History
120 lines (95 loc) · 4.58 KB

README.rst

File metadata and controls

120 lines (95 loc) · 4.58 KB

libfnl™

Introduction

libfnl is an API and CLI facilitating data and text mining by providing a collection of easy-to-use tools. The library is designed to work with Python 3 (only). It is specifically tuned towards mining biomedical/scientific texts, but can be used in other contexts if need be, too. It is a complementary piece in the gnamed gene name repository daemon and the medic PubMed mirroring tool collection. In addtion, an (orphan) couchpy repository could provide a document storage facility.

The library contains the following packages:

fnl.nlp
tools to linguistically analyze text (tokenization, PoS tagging, phrase chunking, entity detection); modules to segment sentences (based on NLTK), and map text (strings) to entries in dictionaries this includes a Python wrapper for the GENIA Tagger, a Python wrapper for the NER Suite, and a handler for the GENIA corpus; furthermore, via NLTK 's wrapper for MegaM, a Maximum Entropy classifier is available, too;
fnl.stat
a module to evaluate inter-rater Kappa scores and a module to develop text classifiers based on Scikit-Learn
fnl.text
wrappers to work with text data (strings, tokens, segments, annotations, etc.)
fnl.utils
additional utilities and tools (currently, just for handling JSON)
scripts
the CLI scripts to manage data/text, representing the main value provided by this collection

The script directory provides the following command-line interfaces:

  • fnlclassi generate a classifier for [NER-tagged] text using Scikit-Learn.
  • fnlcorpus store corpora in JSON format in a CouchDB.
  • fnldgrep "grep" for tokens using a dictionary.
  • fnldictag tag semantic tokens from a dictionary in linguistically annotated text.
  • fnlgpcounter count gene/protein symbols in MEDLINE.
  • fnlkappa calculate inter-rater agreement scores.
  • fnlsegment segment text into sentences using NLTK (PunktSentenceTokenizer).
  • fnlsegtrain train a nltk.punkt.PunktSentenceTokenizer.
  • fnltok a fast, pure-Python, Unicode-aware string tokenizer.

Warning

This project is under "continuous development", better take your own snapshot.

Requirements

  • Python 3.2+
  • Numpy, SciPy, and Scikit-Learn 0.14+ (for fnlclassi)
  • NLTK 3.0+ (for the sentence segmenting tools fnlseg*)
  • DAWG (for fnlgpcounter; see Installation below)

Optional projects that work together with this project:

  • GENIA Tagger (optional, latest version)
  • NER Suite (optional, latest version, in turn requires CRF Suite)
  • MegaM - a MaxEnt classifier for NLTK with a (fast) L-BFGS optimizer
  • gnamed for creating gene/protein name repositories
  • medic for mirroring and handling PubMed citations
  • txtfnnl natural language processing tools based on Apache OpenNLP and UIMA

Installation

Into a Python 3 virtual environment:

pip install virtualenv # if virtualenv is not yet installed
git clone git://github.com/fnl/libfnl.git libfnl
virtualenv libfnl
cd libfnl
. bin/activate
pip install argparse # for python3 < 3.2
pip install numpy # because installing scipy fails if numpy isn't installed already
pip install -e . # installs all other dependencies

# if you prefer to install all other dependencies manually
# and/or prefer to use setup.py instead of pip:
# python setup.py install
pip install sqlalchemy
pip install sklearn
pip install matplotlib
pip install nltk --pre # to get 3.0

# if you want to install the test environment:
pip install pytest

# special steps to install DAWG
git clone [email protected]:fnl/DAWG.git
cd DAWG
python setup.py install
cd ..

License

All parts of this library are licensed under the GNU Affero GPL v3

See the attached LICENSE.txt file.

Copyright

© 2006-2014 Florian Leitner. All rights reserved.