Engine WIP.

IvanoscaLV · Dec 26, 2014 · 126cb29 · 126cb29
1 parent 1ab0d4b
commit 126cb29
Show file tree

Hide file tree

Showing 771 changed files with 148 additions and 88 deletions.
diff --git a/.gitignore b/.gitignore
diff --git a/.gitmodules b/.gitmodules
diff --git a/LICENSE b/LICENSE
diff --git a/README.md b/README.md
@@ -1,6 +1,6 @@
-=======
-elcano-iepg
-===========
+=============
+Elcano - IEPG
+=============
 
 Web del Índice Elcano de presencia global del Real Instituo Elcano
 
@@ -11,68 +11,92 @@ Este proyecto esta compuesto por dos apliaciones webs, una para el backend y otr
 
 La aplicación JS se encuentra en el directorio www y los servicios en www-srv.
 
-## Install
+## Instalación del entorno de desarrollo para servicios
 
-### Instalamos virtualvenv.
+### Dependencias
 
-En Mac OSX : 
+Estas son las aplicaciones de las que depende Elcano - IEPG:
+
+  - Redis 2.8.19;
+  - PostgreSQL 9.1.2;
+  - PostGIS 1.5.8 (sobre la PostgreSQL anterior).
+
+Actualmente, dado que los nuevos proyectos en Geographica han comenzado a
+utilizar versiones más modernas de PostgreSQL y de PostGIS, estos dos elementos
+los utilizamos dockerizados (ver
+https://github.com/GeographicaGS/Docker-PostgreSQL-9.1.2-PostGIS-1.5.8).
+
+### Submodules
+
+Please note that this project uses GIT submodules. To check which ones are used:
+
+    git submodule
+
+To initialize them, enter the root of the repo and:
+
+    git submodule init
+	git submodule update
+
+### Configuración y restauración de la base de datos
+
+Primero, configurar los scripts de base de datos, renombrando el fichero
+__00-config.sql.config__ a __00-config.sql__ y rellenando los datos de acceso a
+la PostgreSQL. Para restaurar la copia funcional de la base de datos, entrar en
+el directorio __database__ y ejecutar el script __01-create_env-dev.sql__:
+
+    psql -h localhost -p 5454 -U postgres postgres -c "\i 01-create_env-dev.sql"
+    &> out
+
+para borrar toda la base de datos de desarrollo, utilizar el script
+__05-database-drop.sql__:
+
+    psql -h localhost -p 5454 -U postgres postgres -c "\i 05-database-drop.sql"
+    &> out
+
+### Entorno de desarrollo de servicios
+
+Primero hemos de instalar el Virtualenv de Python:
+
+- en Mac OSX : 
 
 ```$ sudo pip install virtualenv```
 
-En Ubuntu:
+- en Ubuntu:
 
 ```$ sudo apt-get install python-virtualenv```
 
-Sobre el directorio www-srv ejecutamos
+Creamos un entorno virtual en el directorio __www-src__:
 
 ```$ virtualenv venv```
 
-A continuación nos ponemos en el entorno virtual.
+A continuación nos ponemos en el entorno virtual:
 
 ```. venv/bin/activate ```
 
-### Librerías de python
+### Librerías de Python
 
-Una vez dentro del entorno virtual instalamos los paquetes necesarios con pip.
+Una vez dentro del entorno virtual instalamos los paquetes necesarios con PIP:
 
 ```
-$ pip install Flask
-$ pip install psycopg2
+pip install Flask
+pip install psycopg2
+pip install tweepy
+pip install python-memcached
+pip install redis
+pip install numpy
+pip install xlsxwriter
+pip install xlrd
 #only to debug purpose
-$ pip install tweepy
 $ pip install ipdb
-$ pip install python-memcached
 ```
 
-
 ### Arranque de las APIs
 
-Lo primero es crear el fichero de configuración de la base de datos a partir de la plantilla.
-
-```
-$ cp src/model/base/PostgreSQL/config_changes.py src/model/base/PostgreSQL/config.py
-```
-
-Ahora lo abrimos con cualquier editor de texto y lo modificamos con nuestros parámetros.
-
 Editamos el fichero de configuración global
 ```
 cp src/common/config_changes.py  src/common/config.py
 ```
 
-#### Backend
-
-Tenemos que crear los ficheros de configuración, estos no están en el trunk.
-
-```
-# copiamos el fichero de configuración de la base datos
-$ cp src/backend/config_changes.py src/backend/config.py
-```
-
-Abre el fichero config.py y modificalo con los parámetros adecuados.
-
-Una vez hecho esto ya podemos arrancar el backend ejecutando:
-
 ```
 $ python src/run_backend.py
 ```
@@ -93,6 +117,11 @@ chmod 755 /Users/alasarr/dev/elcano-iepg/www/cdn/backend/tmp
 chmod 755 /Users/alasarr/dev/elcano-iepg/www/cdn/media
 ```
 
+### Twitter y Explora
+
+Por último, para que funcionen ambos servicios, se deben rellenar los datos en
+__common/config.py__ de los datos de las claves de Twitter y de Explora.
+
 #### Frontend
 
 Tenemos que crear los ficheros de configuración, estos no están en el trunk.
@@ -277,6 +306,41 @@ Alias /img /Users/alasarr/dev/elcano-iepg/www/cdn/frontend/img
 Aquí las notas de la configuración de la web.
 
 
+# Checklist de entorno de desarrollo
+
+## Arranque
+
+Hay que seguir los siguientes pasos: 
+
+- arrancar PostgreSQL 9.1.2 / PostGIS 1.5.3 (posiblemente dockerizada):
+
+    pg_ctl-9.1.2-1.5.3 -D . start
+
+- arrancar Redis:
+
+    redis-server redis-2.8.14.conf &
+
+- activar el entorno virtual en __www-src__:
+
+    . venv/bin/activate
+
+- activar servicios Flask:
+
+    src/run_backend.py (port 5000)
+	src/run_explora.py (port 5002)
+	src/run_frontend.py (port 5001)
+
+## Parada
+
+Pasos: 
+
+- parar PostgreSQL (posiblemente dockerizada):
+
+    pg_ctl-9.1.2-1.5.3 -D . stop
+
+- parar Redis:
+
+    redis-cli shutdown save
 
 # Carga de datos en Redis
 

diff --git a/database/.gitignore b/database/.gitignore
diff --git a/database/00-config.sql.config b/database/00-config.sql.config
diff --git a/database/01-create_env-dev.sql b/database/01-create_env-dev.sql
diff --git a/database/01-create_env-production.sql b/database/01-create_env-production.sql
diff --git a/database/05-database-create.sql b/database/05-database-create.sql
diff --git a/database/05-database-drop.sql b/database/05-database-drop.sql
diff --git a/database/10-backend-ddl.sql b/database/10-backend-ddl.sql
diff --git a/database/10-backend-drop.sql b/database/10-backend-drop.sql
diff --git a/database/10-iepg_data-ddl.sql b/database/10-iepg_data-ddl.sql
@@ -510,58 +510,58 @@ order by id_master_country;
 copy iepg_data.master_country
 from :'copy_master_country'
 with delimiter ';'
-csv header quote '"';
+csv header quote '"' null '#@#@';
 
 copy iepg_data.country_relation
 from :'copy_country_relation'
 with delimiter ';'
-csv header quote '"';
+csv header quote '"' null '#@#@';
 
 copy iepg_data.iepg_final_data
 from :'copy_iepg_final_data'
 with delimiter ';'
-csv header quote '"';
+csv header quote '"' null '#@#@';
 
 copy iepg_data.pob_pib
 from :'copy_pob_pib'
 with delimiter ';'
-csv header quote '"';
+csv header quote '"' null '#@#@';
 
 copy iepg_data.iepg_comment
 from :'copy_iepg_comment'
 with delimiter '|'
-csv header quote '"';
+csv header quote '"' null '#@#@';
 
 copy iepg_data.country_geom
 from :'copy_country_geom'
 with delimiter '|'
-csv header quote '"';
+csv header quote '"' null '#@#@';
 
 copy iepg_data.iepe_final_data
 from :'copy_iepe_final_data'
 with delimiter '|'
-csv header quote '"';
+csv header quote '"' null '#@#@';
 
 copy iepg_data.iepe_quotas
 from :'copy_iepe_quotas'
 with delimiter '|'
-csv header quote '"';
+csv header quote '"' null '#@#@';
 
-\copy iepg_data.iepe_individual_contributions from 'iepe_individual_contributions.csv' with delimiter '|' csv header quote '"'
+\copy iepg_data.iepe_individual_contributions from 'iepe_individual_contributions.csv' with delimiter '|' csv header quote '"' null '#@#@'
 
-\copy iepg_data.iepe_relative_contributions from 'iepe_relative_contributions.csv' with delimiter '|' csv header quote '"'
+\copy iepg_data.iepe_relative_contributions from 'iepe_relative_contributions.csv' with delimiter '|' csv header quote '"' null '#@#@'
 
-\copy iepg_data.iepg_quotas from 'iepg_quotas.csv' with delimiter '|' csv header quote '"'
+\copy iepg_data.iepg_quotas from 'iepg_quotas.csv' with delimiter '|' csv header quote '"' null '#@#@'
 
-\copy iepg_data.iepg_individual_contributions from 'iepg_individual_contributions.csv' with delimiter '|' csv header quote '"'
+\copy iepg_data.iepg_individual_contributions from 'iepg_individual_contributions.csv' with delimiter '|' csv header quote '"' null '#@#@'
 
-\copy iepg_data.iepg_relative_contributions from 'iepg_relative_contributions.csv' with delimiter '|' csv header quote '"'
+\copy iepg_data.iepg_relative_contributions from 'iepg_relative_contributions.csv' with delimiter '|' csv header quote '"' null '#@#@'
 
-\copy iepg_data.iepg_final_data_eu from 'iepg_final_data_eu.csv' with delimiter '|' csv header quote '"'
+\copy iepg_data.iepg_final_data_eu from 'iepg_final_data_eu.csv' with delimiter '|' csv header quote '"' null '#@#@'
 
-\copy iepg_data.iepg_individual_contributions_eu from 'iepg_individual_contributions_eu.csv' with delimiter '|' csv header quote '"'
+\copy iepg_data.iepg_individual_contributions_eu from 'iepg_individual_contributions_eu.csv' with delimiter '|' csv header quote '"' null '#@#@'
 
-\copy iepg_data.iepg_relative_contributions_eu from 'iepg_relative_contributions_eu.csv' with delimiter '|' csv header quote '"'
+\copy iepg_data.iepg_relative_contributions_eu from 'iepg_relative_contributions_eu.csv' with delimiter '|' csv header quote '"' null '#@#@'
 
 
 analyze;

diff --git a/database/10-iepg_data-drop.sql b/database/10-iepg_data-drop.sql
diff --git a/database/10-iepg_data_redux-ddl.sql b/database/10-iepg_data_redux-ddl.sql
diff --git a/database/10-iepg_data_redux-drop.sql b/database/10-iepg_data_redux-drop.sql
diff --git a/database/10-maplex-ddl.sql b/database/10-maplex-ddl.sql
@@ -321,23 +321,23 @@ order by a.geoentity_desc;
 
 -- Restore data
 
-\copy maplex.geoentity from 'maplex_geoentity.csv' with delimiter '|' csv header quote '"'
+\copy maplex.geoentity from 'maplex_geoentity.csv' with delimiter '|' csv header quote '"' null '#@#@'
 
-\copy maplex.name_family from 'maplex_name_family.csv' with delimiter '|' csv header quote '"'
+\copy maplex.name_family from 'maplex_name_family.csv' with delimiter '|' csv header quote '"' null '#@#@'
 
-\copy maplex.name from 'maplex_name.csv' with delimiter '|' csv header quote '"'
+\copy maplex.name from 'maplex_name.csv' with delimiter '|' csv header quote '"' null '#@#@'
 
-\copy maplex.geometry_family from 'maplex_geometry_family.csv' with delimiter '|' csv header quote '"'
+\copy maplex.geometry_family from 'maplex_geometry_family.csv' with delimiter '|' csv header quote '"' null '#@#@'
 
-\copy maplex.block from 'maplex_block.csv' with delimiter '|' csv header quote '"'
+\copy maplex.block from 'maplex_block.csv' with delimiter '|' csv header quote '"' null '#@#@'
 
-\copy maplex.geometry from 'maplex_geometry.csv' with delimiter '|' csv header quote '"'
+\copy maplex.geometry from 'maplex_geometry.csv' with delimiter '|' csv header quote '"' null '#@#@'
 
-\copy maplex.geoentity_geometry from 'maplex_geoentity_geometry.csv' with delimiter '|' csv header quote '"'
+\copy maplex.geoentity_geometry from 'maplex_geoentity_geometry.csv' with delimiter '|' csv header quote '"' null '#@#@'
 
-\copy maplex.geoentity_name from 'maplex_geoentity_name.csv' with delimiter '|' csv header quote '"'
+\copy maplex.geoentity_name from 'maplex_geoentity_name.csv' with delimiter '|' csv header quote '"' null '#@#@'
 
-\copy maplex.iepg_short_names from 'csv/maplex-country_short_iepg.csv' with delimiter ',' csv header
+\copy maplex.iepg_short_names from 'csv/maplex-country_short_iepg.csv' with delimiter ',' csv header null '#@#@'
 
 commit;
 

diff --git a/database/10-maplex-drop.sql b/database/10-maplex-drop.sql
diff --git a/database/10-varengine-ddl.sql b/database/10-varengine-ddl.sql
diff --git a/database/10-varengine-drop.sql b/database/10-varengine-drop.sql
diff --git a/database/2014-05-14_delta_translation.sql b/database/2014-05-14_delta_translation.sql
diff --git a/database/2014-07-22_delta_publishing_dates.sql b/database/2014-07-22_delta_publishing_dates.sql
diff --git a/database/2014-08-26_translation.sql b/database/2014-08-26_translation.sql
diff --git a/database/90-backend-delete_data.sql b/database/90-backend-delete_data.sql
diff --git a/database/90-export_data.sql b/database/90-export_data.sql
diff --git a/database/98-load_data.sql b/database/98-load_data.sql
diff --git a/database/99-basic_data.sql b/database/99-basic_data.sql
diff --git a/database/99-iepe_excel_exports_views.sql b/database/99-iepe_excel_exports_views.sql
diff --git a/database/99-iepg_excel_exports_views.sql b/database/99-iepg_excel_exports_views.sql