Initialise DVC

We now use DVC to track our /data directory. All files are versioned and cached using DVC, with remote storage on S3. closes #161
nestauk · Sep 23, 2022 · 760f75d · 760f75d
1 parent f13117b
commit 760f75d
Show file tree

Hide file tree

Showing 128 changed files with 17 additions and 219,927 deletions.
diff --git a/.dvc/.gitignore b/.dvc/.gitignore
@@ -0,0 +1,3 @@
+/config.local
+/tmp
+/cache
diff --git a/.dvc/config b/.dvc/config
@@ -0,0 +1,4 @@
+[core]
+    remote = remote
+['remote "remote"']
+    url = s3://dap-dv-backends
diff --git a/.dvcignore b/.dvcignore
@@ -0,0 +1,3 @@
+# Add patterns of files dvc should ignore, which could improve
+# the performance. Learn more at
+# https://dvc.org/doc/user-guide/dvcignore
diff --git a/.gitignore b/.gitignore
@@ -6,4 +6,5 @@ node_modules
 **/secrets/**
 src/bin/spotlight/nginx.conf
 src/servers/**/terraform
-src/servers/**/nginx.conf
+src/servers/**/nginx.conf
+/data
diff --git a/data.dvc b/data.dvc
@@ -0,0 +1,5 @@
+outs:
+- md5: 2961519e5dd56934effbdacf079132b3.dir
+  size: 28522654
+  nfiles: 125
+  path: data
diff --git a/data/ai_in_genomics/inputs/ai_genomics_cb_samples.csv b/data/ai_in_genomics/inputs/ai_genomics_cb_samples.csv
diff --git a/data/ai_in_genomics/inputs/ai_genomics_gtr_samples.csv b/data/ai_in_genomics/inputs/ai_genomics_gtr_samples.csv
diff --git a/data/ai_in_genomics/inputs/ai_genomics_openalex_samples.csv b/data/ai_in_genomics/inputs/ai_genomics_openalex_samples.csv
diff --git a/data/ai_in_genomics/inputs/ai_genomics_patent_samples.csv b/data/ai_in_genomics/inputs/ai_genomics_patent_samples.csv
diff --git a/data/ai_in_genomics/outputs/ai_genomics_cb_samples.csv b/data/ai_in_genomics/outputs/ai_genomics_cb_samples.csv
diff --git a/data/ai_in_genomics/outputs/ai_genomics_gtr_samples.csv b/data/ai_in_genomics/outputs/ai_genomics_gtr_samples.csv
diff --git a/data/ai_in_genomics/outputs/ai_genomics_openalex_samples.csv b/data/ai_in_genomics/outputs/ai_genomics_openalex_samples.csv
diff --git a/data/ai_in_genomics/outputs/ai_genomics_patent_samples.csv b/data/ai_in_genomics/outputs/ai_genomics_patent_samples.csv
diff --git a/data/ai_map/inputs/ai_map_orgs_places.json b/data/ai_map/inputs/ai_map_orgs_places.json
diff --git a/data/ai_map/outputs/ai_map_orgs_places_populated.json b/data/ai_map/outputs/ai_map_orgs_places_populated.json
diff --git a/data/ai_map/outputs/ai_map_orgs_places_populated_subset.json b/data/ai_map/outputs/ai_map_orgs_places_populated_subset.json
diff --git a/data/ai_map/outputs/entity_details.json b/data/ai_map/outputs/entity_details.json