Methods to import calculated XLSX.

README.md

@@ -4,13 +4,42 @@ Elcano - IEPG
 
 The Elcano [Global Presence Index](http://www.globalpresence.realinstitutoelcano.org) is an annual measurement of the projection in the world of 90 countries based on three dimensions: Economic, Military, Soft.
 
-This is a project from [Elcano Royal Institute](http://www.realinstitutoelcano.org) of strategic and international studies and [Geographica](https://geographica.gs). 
+This is a project from [Elcano Royal Institute](http://www.realinstitutoelcano.org) of strategic and international studies and [Geographica](https://geographica.gs).
 
-# Tech overview 
+# Git Submodules
+
+This project uses two submodules. To initialize them, at the git root:
+
+```Shell
+git submodule init
+git submodule update
+```
+
+# Tech overview
 This project is a web platform composed by 3 APIs developed in Python (Flask) at the server side. At the client side, it uses an HTML application who uses [Backbone JS](http://backbonejs.org).
 
 To deploy and develop it uses [Docker](https://www.docker.com).
 
+# How to import, load, and process the IEPG calculated XLSX
+
+In December 2016 the update process of IEPG data shifted from calculating the whole dataset from source data by the application by conducting the full IEPG calculations to providing in a XLSX the full calculation of the whole dataset. Please check file __calculus2015.xlsx__ to see the format of the data XLSX. Then:
+
+- apply the database patch in file _www-srv/calculated_xlsx-database_patch.sql_;
+
+- Import the XLSX:
+
+```Shell
+docker exec elcanoiepg_api_backend_1 python flux_updatefromcalculatedxlsx.py [XLSX file]
+  [# of years in IEPG] [# of countries in IEPG] [# of years in IEPE] [# of countries in IEPE]
+```
+
+- update REDIS from database:
+
+```Shell
+docker exec elcanoiepg_api_backend_1 python updatecache.py
+```
+
+
 # How to prepare the environment
 
 Install [Docker](https://docs.docker.com/engine/installation) 1.8+ and [docker-compose](https://docs.docker.com/compose/install/) 1.5.2+.
@@ -27,7 +56,7 @@ Create the config file using the sample and add your own data.
 cp config.sample.env config.env
 ```
 
-Set the environment variables 
+Set the environment variables
 ```
 source config.env
 ```
@@ -55,13 +84,13 @@ echo $(docker-machine ip default) 'dev.elcano-iepg.geographica.gs dev.explora.el
 
 Start
 ```
-docker-compose -f docker-compose.dev up
+docker-compose -f docker-compose.dev.yml up
 ```
 
 Propagate the data from Postgres to REDIS.
 
 ```
-docker exec elcanoiepg_api_backend_1 python updatecache.py 
+docker exec elcanoiepg_api_backend_1 python updatecache.py
 ```
 
 Prepare clients. TODO: we need to use the latest version of [Sting](https://github.com/GeographicaGS/Sting) to be able to read data of environment values (config.env) inside config.js files.
@@ -91,24 +120,24 @@ docker-compose -f docker-compose.dev up
 
 Update the whole platform from XLSX. From the input file www-srv/src/data_calculus/year2015.xlsx it updates de PostgreSQL and REDIS. It also makes a backup of the iepg_data schema of PostgreSQL.
 ```
-docker exec elcanoiepg_api_backend_1 python flux_updatewholeapp.py 
+docker exec elcanoiepg_api_backend_1 python flux_updatewholeapp.py
 ```
 
 RUN calculus engine XLSX to XLSX. From www-srv/src/data_calculus/year2015.xlsx, it generates the engine output at www-srv/src/data_calculus/calculus2015.xlsx.
 ```
-docker exec elcanoiepg_api_backend_1 python flux_xlsxtoxlsx.py 
+docker exec elcanoiepg_api_backend_1 python flux_xlsxtoxlsx.py
 ```
 
 Update REDIS from PostgreSQL
 ```
-docker exec elcanoiepg_api_backend_1 python updatecache.py 
+docker exec elcanoiepg_api_backend_1 python updatecache.py
 ```
 
 UPDATE population and GDP
 ```
 docker exec elcanoiepg_api_backend_1 bash -c "cd scripts && python pop_gdp.py" > gdp2015.sql
 docker exec -i elcanoiepg_pgsql_1 psql -U postgres -d elcano_iepg < gdp2015.sql
-docker exec elcanoiepg_api_backend_1 python updatecache.py 
+docker exec elcanoiepg_api_backend_1 python updatecache.py
 ```
 
 # Get latest database
@@ -154,4 +183,3 @@ It compiles the frontend apps using the geographica/elcano_iepg_webbuilder conta
 ```
 ./manager.sh buildapps <environment>
 ```
-

database/calculated_xlsx-database_patch.sql

@@ -0,0 +1,24 @@
+alter table iepg_data_redux.master_country add column xlsx_column_name varchar(150);
+
+alter table iepg_data_redux.master_country alter column iso_3166_1_2_code type varchar(4);
+
+update iepg_data_redux.master_country set xlsx_column_name=full_name_en_order;
+
+update iepg_data_redux.master_country set xlsx_column_name='Viet Nam'
+where trim(short_name_en_order)='Vietnam';
+
+update iepg_data_redux.master_country set xlsx_column_name='United Republic of Tanzania'
+where trim(short_name_en_order)='Tanzania';
+
+update iepg_data_redux.master_country set xlsx_column_name='United Kingdom'
+where trim(short_name_en_order)='United Kingdom';
+
+update iepg_data_redux.master_country set xlsx_column_name='Syria'
+where trim(short_name_en_order)='Syria';
+
+update iepg_data_redux.master_country set xlsx_column_name='Korea'
+where trim(short_name_en_order)='South Korea';
+
+insert into iepg_data_redux.master_country(
+  id_master_country, short_name_en_order, iso_3166_1_2_code, xlsx_column_name)
+values ('un999', 'European Union', 'XBEU', 'European Union');

docker-into_database_container.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+docker exec -ti elcanoiepg_pgsql_1 /bin/bash

docker-into_python_dev.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+docker exec -ti elcanoiepg_api_backend_1 /bin/bash