IATI Dashboard

Summary

Product	IATI Dashboard
Description	A Django web application that provides key numbers, statistics and graphs about the data on the IATI registry. This repository is currently a development version where the IATI Dashboard/Publishing Statistics and Code for IATI Analytics are being merged.
Website	Development only; see IATI Dashboard, and Code for IATI Analytics for live versions.
Related	Repositories for the live version of the IATI Dashboard, live version of the IATI Publishing Stats, and Code for IATI Analytics. Data is generated from Code for IATI Stats.
Documentation	Rest of README.md
Technical Issues	See https://github.com/IATI/IATI-Dashboard/issues
Support	https://iatistandard.org/en/guidance/get-support/

High-level requirements

• Python 3.12
• Unix-based setup (e.g., Linux, MacOS X) with bash, wget and curl installed.
• Postgres database (see below for how to run this with docker)

Running the app locally

Overview

The IATI Dashboard is mostly written in Python but also has some helper Bash scripts to collect the data that the dashboard uses. Top-level steps required to run the Dashboard are:

1. Setup Python environment and install dependencies.
2. Fetch the data.
3. Build the static graphs and other data that will be served via the Dashboard.
4. Run the web server.

1. Setup environment

Assuming that this repository has been cloned and you are in the root directory of the repository.

# Setup and activate a virtual environment (recommended)
python3.12 -m venv .ve
source .ve/bin/activate

Now install the dependencies.

pip install -r requirements.txt

2. Fetching the data

Bash scripts are used to fetch the data that the Dashboard will present.

# Fetch the necessary calculated stats
# This will store data in the ./stats-calculated directory
# These are created by https://github.com/IATI/IATI-Stats (see below for more information)
# This downloads ~1.5GB of data, which uncompresses to ~50GB
./get_stats.sh

# Fetch some extra data from github and github gists and other sources on the internet
# This will store data in the ./data directory
# This downloads ~60MB of data
./fetch_data.sh

3. Start Postgres database, run migrations, import data

docker run --name iati-dashboard-postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=iati_dashboard -p 127.0.0.1:5432:5432 -d postgres:17
# Run this every time you open a new shell
export DATABASE_URL="postgres://postgres:postgres@localhost:5432/iati_dashboard"
python manage.py migrate
python manage.py dashboard_import

4. Build static data and graphs

python -m iati_dashboard.make_plots
python manage.py make_csv
python -m iati_dashboard.speakers_kit

5. Run the webserver.

DEBUG=True python manage.py runserver

The Dashboard will now be accessible from http://localhost:8000/.

Development

Automated tests

There are some unit tests written using pytest and site testing using Django's own testing framework.

Once the development dependencies have been installed the unit tests can be run with:

pytest

The Django site tests can be run from the dashboard/ directory with:

python manage.py test -v 2

Calculating your own statistics

The IATI Dashboard requires a stats-calculated directory, which can be downloaded using the get_stats.sh shell script as described above. This can also be calculated using Code for IATI Stats where stats-calculated corresponds to the gitout directory generated by git.sh in IATI-Stats.

Often you only want to regenerate the current stats, use get_stats.sh to download the pre-calculated historical stats and just replace the stats-calculated/current directory with the out directory produced by running the loop, aggregate and invert commands individually, then regenerate graphs and CSV files as per the above.

Adding new dependencies

If a change requires new dependencies then please add to requirements.in or requirements_dev.in as appropriate and recompile:

pip-compile requirements.in
pip-compile requirements_dev.in

Linting

Code linting is carried out using Black, isort and Flake8 and pyproject.toml has the configuration.

License

Copyright (C) 2013-2015 Ben Webb <[email protected]>
Copyright (C) 2013-2014 David Carpenter <[email protected]>
Copyright (C) 2021 Andy Lulham <[email protected]>

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.