Importing data
There are two ways to export/import Django data: (1) using Django's (dumpdata) and (loaddata) and (2) manually exporting/importing the PostgreSQL.
For our purposes, the latter seems to work way better and more reliably. So, that's what we recommend!
Note: for whatever reason, I have not had as much luck using this approach as manually exporting/importing the PostgreSQL database. So, we recommend not using this approach.
Django makes it relatively easy to dump a database from one computer (dumpdata) and reload it on another (loaddata). This is useful and important for testing and bootstrapping your local development environment.
Because so much of the Makeability Lab website relies on artifacts (like PDF and PowerPoint files as well as thumbnails), it's not sufficient to just dump/load the database. We must also copy these files (stored in /code/media) from the exporting computer to the importing computer. I'll walk through everything below.
To dump the database data, you need to create an interactive bash shell into the Docker container and run the dumpdata command.
-
Run the interactive shell in terminal:
> docker exec -it makeabilitylabwebsite_website_1 bash -
From the Docker shell, run the
dumpdatacommand. Make sure you have admin access.apache@43cd75e1d80c:/code$ python manage.py dumpdata > dumped_data.json bash: dumped_data.json: Permission denied apache@43cd75e1d80c:/code$ su Password: root@43cd75e1d80c:/code# python manage.py dumpdata > dumped_data.json -
You might also have to copy the
dumped_data.jsonfile out of the Docker container and into your local filesystem. So, go back into terminal (not the Docker shell) and type> docker cp makeabilitylabwebsite_website_1:/code/dumped_data.json . -
Now you should have a
dumped_data.jsonfile that you can copy over to your 2nd computer.
The second primary step is to copy the media directory from the Docker container into your local filesystem.
- From terminal (not the Docker shell), type
> docker cp makeabilitylabwebsite_website_1:/code/media ., which should copy thismediadirectory to your local filesystem - I also zipped up this dir (because it can be huge) to transport it over to my 2nd computer for importing.
Now, on a second computer, we can import that dumped_data.json file into Django and copy over the media dir.
We need to first copy over the media dir into the Docker container.
- From your second computer's terminal, type
> docker cp ./media makeabilitylabwebsite_website_1:/code/., which will copy themediadir into the Docker container at/code/. - Make sure the
mediadir and subdirs have appropriate permissions. I launched an interactive shelldocker exec -it makeabilitylabwebsite_website_1 bashand then set all media tochmod -R 777(but probably could usechmod -R 755). If you get anOperation not permittederror, you'll need to first switch to sudo user (e.g.,> sucommand) - In the interactive shell, you can also check that the contents have been copied over correctly
Finally, we can import the dumped_data.json file:
- First, launch an interactive shell. From your second computer's terminal, type:
> docker exec -it makeabilitylabwebsite_website_1 bash - From within this shell, type
$ python manage.py loaddata dumped_data.jsonand it should work. Update: I just tried this again and the command failed with a permissions issue (see below), so I had to rerun the command with sudo permissions:
No need to save, the thumbnail '/code/media/talks/images/Social_Fabrics_Designing_Wearable_Electronics_Textiles_for_Interaction_Introspection_and_Inquiry.jpg' already exists!
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 316, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 353, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
self.loaddata(fixture_labels)
File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 113, in loaddata
self.load_label(fixture_label)
File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 177, in load_label
obj.save(using=self.using)
File "/usr/local/lib/python3.8/site-packages/django/core/serializers/base.py", line 222, in save
getattr(self.object, accessor_name).set(object_list)
File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 997, in set
self.add(*new_objs)
File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 926, in add
self._add_items(self.source_field_name, self.target_field_name, *objs)
File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 1099, in _add_items
signals.m2m_changed.send(
File "/usr/local/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 173, in send
return [
File "/usr/local/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 174, in <listcomp>
(receiver, receiver(signal=self, sender=sender, **named))
File "/code/website/models/talk.py", line 108, in update_file_name_talks
os.rename(initial_path, new_path)
PermissionError: [Errno 13] Permission denied: '/code/media/talks/Social_Fabrics_Designing_Wearable_Electronics_Textiles_for_Interaction_Introspection_and_Inquiry.pdf' -> '/code/media/talks/Froehlich_SocialFabricsDesigningWearableElectronicsTextilesForInteractionIntrospectionAndInquiry_2016.pdf'
Here's an example:
apache@e86363656952:/code$ python manage.py loaddata dumped_data.json
Publication 'This is a test' has just been saved with PDF=/code/media/publications/Froehlich_ThisIsATest_ASSETS2021.pdf, checking to see if we should auto-generate a thumbnail
Checking for thumbnail at '/code/media/publications/images/Froehlich_ThisIsATest_ASSETS2021.jpg', otherwise will auto-generate
The thumbnail at `/code/media/publications/images/Froehlich_ThisIsATest_ASSETS2021.jpg` exists
No need to save, the thumbnail '/code/media/publications/images/Froehlich_ThisIsATest_ASSETS2021.jpg' already exists!
Publication 'This is a test' has just been saved with PDF=/code/media/publications/Froehlich_ThisIsATest_ASSETS2021.pdf, checking to see if we should auto-generate a thumbnail
Checking for thumbnail at '/code/media/publications/images/Froehlich_ThisIsATest_ASSETS2021.jpg', otherwise will auto-generate
The thumbnail at `/code/media/publications/images/Froehlich_ThisIsATest_ASSETS2021.jpg` exists
No need to save, the thumbnail '/code/media/publications/images/Froehlich_ThisIsATest_ASSETS2021.jpg' already exists!
Publication 'The Future of Global-Scale Spatial Data Collection and Analyses on Urban (in)Accessibility for People with Disabilities' has just been saved with PDF=/code/media/publications/Froehlich_TheFutureOfGlobalScaleSpatialDataCollectionAndAnalysesOnUrbanInAccessibilityForPeopleWithDisabilities_SPATIALDATASCIENCESYMPOSIUM2021.pdf, checking to see if we should auto-generate a thumbnail
Checking for thumbnail at '/code/media/publications/images/Froehlich_TheFutureOfGlobalScaleSpatialDataCollectionAndAnalysesOnUrbanInAccessibilityForPeopleWithDisabilities_SPATIALDATASCIENCESYMPOSIUM2021.jpg', otherwise will auto-generate
The thumbnail at `/code/media/publications/images/Froehlich_TheFutureOfGlobalScaleSpatialDataCollectionAndAnalysesOnUrbanInAccessibilityForPeopleWithDisabilities_SPATIALDATASCIENCESYMPOSIUM2021.jpg` exists
No need to save, the thumbnail '/code/media/publications/images/Froehlich_TheFutureOfGlobalScaleSpatialDataCollectionAndAnalysesOnUrbanInAccessibilityForPeopleWithDisabilities_SPATIALDATASCIENCESYMPOSIUM2021.jpg' already exists!
Publication 'The Future of Global-Scale Spatial Data Collection and Analyses on Urban (in)Accessibility for People with Disabilities' has just been saved with PDF=/code/media/publications/Froehlich_TheFutureOfGlobalScaleSpatialDataCollectionAndAnalysesOnUrbanInAccessibilityForPeopleWithDisabilities_SPATIALDATASCIENCESYMPOSIUM2021.pdf, checking to see if we should auto-generate a thumbnail
Checking for thumbnail at '/code/media/publications/images/Froehlich_TheFutureOfGlobalScaleSpatialDataCollectionAndAnalysesOnUrbanInAccessibilityForPeopleWithDisabilities_SPATIALDATASCIENCESYMPOSIUM2021.jpg', otherwise will auto-generate
The thumbnail at `/code/media/publications/images/Froehlich_TheFutureOfGlobalScaleSpatialDataCollectionAndAnalysesOnUrbanInAccessibilityForPeopleWithDisabilities_SPATIALDATASCIENCESYMPOSIUM2021.jpg` exists
No need to save, the thumbnail '/code/media/publications/images/Froehlich_TheFutureOfGlobalScaleSpatialDataCollectionAndAnalysesOnUrbanInAccessibilityForPeopleWithDisabilities_SPATIALDATASCIENCESYMPOSIUM2021.jpg' already exists!
Installed 189 object(s) from 1 fixture(s)
To export the data, you must have your Docker containers running (at least the database one).
First, connect to the Docker database container. To do so, you need to know the CONTAINER ID. So, from your command line, run docker ps:
jonf@jonfhome:~/makeabilitylabwebsite$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
00496813ffb5 makeabilitylabwebsite-website "./docker-entrypoint…" 2 weeks ago Up 3 days 127.0.0.1:8571->8000/tcp makeabilitylabwebsite-website-1
fd509ce466df postgres:16 "docker-entrypoint.s…" 2 weeks ago Up 3 days 0.0.0.0:6543->5432/tcp makeabilitylabwebsite-db-1
Then run an interactive shell into that db container:
jonf@jonfhome:~/makeabilitylabwebsite$ docker exec -it -u root fd509ce466df /bin/bash
Then verify that you can connect to the database:
root@fd509ce466df:/# psql makeability admin
psql (16.0 (Debian 16.0-1.pgdg120+1))
Type "help" for help.
makeability=# \dt
List of relations
Schema | Name | Type | Owner
--------+---------------------------------------+-------+-------
public | auth_group | table | admin
public | auth_group_permissions | table | admin
public | auth_permission | table | admin
public | auth_user | table | admin
public | auth_user_groups | table | admin
public | auth_user_user_permissions | table | admin
Then actually make the dump:
pg_dump -U admin -d makeability -f makeabilitylab-test-nov10.sql
You can verify that the dump succeeded (or at least created the file):
root@fd509ce466df:/# ls -al
total 1492
drwxr-xr-x 1 root root 4096 Nov 10 14:24 .
drwxr-xr-x 1 root root 4096 Nov 10 14:24 ..
lrwxrwxrwx 1 root root 7 Sep 19 00:00 bin -> usr/bin
drwxr-xr-x 2 root root 4096 Sep 29 17:08 docker-entrypoint-initdb.d
drwxr-xr-x 1 root root 4096 Oct 27 12:01 etc
drwxr-xr-x 2 root root 4096 Jul 14 16:00 home
-rw-r--r-- 1 root root 1460968 Nov 10 14:24 makeabilitylab-test-nov10.sql
Now, finally, we have to get this file out into your OS filesystem. You can copy using the following command:
docker cp <containerId>:/file/path/within/container /host/path/target
Let's try it:
jonf@jonfhome:~/makeabilitylabwebsite$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
00496813ffb5 makeabilitylabwebsite-website "./docker-entrypoint…" 2 weeks ago Up 3 days 127.0.0.1:8571->8000/tcp makeabilitylabwebsite-website-1
fd509ce466df postgres:16 "docker-entrypoint.s…" 2 weeks ago Up 3 days 0.0.0.0:6543->5432/tcp makeabilitylabwebsite-db-1
jonf@jonfhome:~/makeabilitylabwebsite$ docker cp fd509ce466df:/makeabilitylab-test-nov10.sql .
Successfully copied 1.46MB to /home/jonf/makeabilitylabwebsite/.
I already have a SQL dump that Matt made, so these notes are assuming you already have a dump.
Note: you need to perform this step before following the dev install directions.
So, we want to create a fresh DB instance that is then populated by our .sql dump.
I first started on a machine that already had the makeabilitylabwebsite container and the website-1 and db-1 images. I wanted to see if I could create a docker-compose simply aimed at connecting to the db.
First, I made my own docker file called docker-compose-db-only.yml.
I ran this file from Ubuntu by calling `docker-compose -f docker-compose-db-only.yml up"
From another Ubuntu terminal, I created an interactive shell into that instance: docker exec -it makeabilitylabwebsite-db-1 bash (but the name could be different)
From the Docker shell, I ran psql makeability admin:
root@327457f1fad7:/# psql makeability admin
psql (16.0 (Debian 16.0-1.pgdg120+1))
Type "help" for help.
Then, I executed \dt to see the tables:
makeability=# \dt
List of relations
Schema | Name | Type | Owner
--------+---------------------------------------+-------+-------
public | django_admin_log | table | admin
public | django_content_type | table | admin
public | django_migrations | table | admin
public | django_session | table | admin
public | easy_thumbnails_source | table | admin
public | easy_thumbnails_thumbnail | table | admin
public | easy_thumbnails_thumbnaildimensions | table | admin
public | website_banner | table | admin
public | website_keyword | table | admin
public | website_news | table | admin
public | website_news_project | table | admin
Now, I am going to completely delete/remove the makeabilitylabwebsite container and the website-1 and db-1 images. And try this step over!
So, now I'm rerunning docker-compose -f docker-compose-db-only.yml up from Ubuntu:
jonf@jonfhome:~/makeabilitylabwebsite$ docker-compose -f docker-compose-db-only.yml up
[+] Running 14/14
✔ db 13 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 11.1s
✔ a803e7c4b030 Pull complete 1.7s
✔ 009c876521a0 Pull complete 0.4s
✔ 9c412905cca2 Pull complete 0.8s
✔ 6463d4bf467a Pull complete 1.0s
✔ bd8b983728ed Pull complete 1.8s
✔ febc167f3560 Pull complete 1.6s
✔ d73c81c4ade3 Pull complete 2.0s
✔ 34b3b0ac6e9e Pull complete 2.1s
✔ 9bd86d074f4e Pull complete 5.3s
✔ 406f63329750 Pull complete 2.4s
✔ ec40772694b7 Pull complete 9.1s
✔ 7d3dfa1637e9 Pull complete 2.9s
✔ e217ca41159f Pull complete 3.0s
[+] Running 2/2
✔ Network makeabilitylabwebsite_default Created 0.0s
✔ Container makeabilitylabwebsite-db-1 Created 0.1s
Attaching to makeabilitylabwebsite-db-1
makeabilitylabwebsite-db-1 | The files belonging to this database system will be owned by user "postgres".
makeabilitylabwebsite-db-1 | This user must also own the server process.
makeabilitylabwebsite-db-1 |
makeabilitylabwebsite-db-1 | The database cluster will be initialized with locale "en_US.utf8".
makeabilitylabwebsite-db-1 | The default database encoding has accordingly been set to "UTF8".
makeabilitylabwebsite-db-1 | The default text search configuration will be set to "english".
...
makeabilitylabwebsite-db-1 | PostgreSQL init process complete; ready for start up.
makeabilitylabwebsite-db-1 |
makeabilitylabwebsite-db-1 | 2023-09-30 13:05:53.812 UTC [1] LOG: starting PostgreSQL 16.0 (Debian 16.0-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
makeabilitylabwebsite-db-1 | 2023-09-30 13:05:53.813 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
makeabilitylabwebsite-db-1 | 2023-09-30 13:05:53.813 UTC [1] LOG: listening on IPv6 address "::", port 5432
makeabilitylabwebsite-db-1 | 2023-09-30 13:05:53.816 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
makeabilitylabwebsite-db-1 | 2023-09-30 13:05:53.819 UTC [64] LOG: database system was shut down at 2023-09-30 13:05:53 UTC
makeabilitylabwebsite-db-1 | 2023-09-30 13:05:53.823 UTC [1] LOG: database system is ready to accept connections
From another Ubuntu terminal, I created an interactive shell into that instance: docker exec -it makeabilitylabwebsite-db-1 bash (but the name could be different)
root@327457f1fad7:/var/lib/postgresql/data# jonf@jonfhome:~$ docker exec -it makeabilitylabwebsite-db-1 bash
root@ccd87abc9e1c:/#
From the Docker shell, I ran psql makeability admin:
root@327457f1fad7:/# psql makeability admin
psql (16.0 (Debian 16.0-1.pgdg120+1))
Type "help" for help.
But now when we run \dt, we are expecting no schema or data:
makeability=# \dt
Did not find any relations.
makeability=#
OK, let's quit out of psql with \q and get back to the interactive shell.
Move to the app dir by cd usr/app, which we exposed in our volumes:
root@e11a3117ad94:/# pwd
/
root@e11a3117ad94:/# cd usr/app
root@e11a3117ad94:/usr/app# ls -al
total 1592
drwxr-xr-x 32 root root 1024 Sep 30 17:52 .
drwxr-xr-x 1 root root 4096 Sep 30 17:56 ..
drwxr-xr-x 6 root root 192 Nov 4 2022 auto-deploy
...
-rw-r--r-- 1 root root 1541363 Sep 27 00:20 makeabilitylab-test.sql
drwxr-xr-x 5 root root 160 Sep 28 23:34 makeabilitylabwebsite
-rw-r--r-- 1 root root 257 Oct 31 2022 manage.py
drwxr-xr-x 12 root root 384 Sep 29 20:14 media
...
drwxr-xr-x 3 root root 96 Nov 4 2022 .vscode
drwxr-xr-x 20 root root 640 Sep 29 19:44 website
Double check you're in the right dir:
root@ccd87abc9e1c:/usr/app# pwd
/usr/app
root@ccd87abc9e1c:/usr/app# ls -al
total 1648
drwxr-xr-x 16 1000 1000 4096 Sep 30 13:20 .
drwxr-xr-x 1 root root 4096 Sep 30 13:16 ..
drwxr-xr-x 2 1000 1000 4096 Sep 15 23:24 auto-deploy
drwxr-xr-x 2 1000 1000 4096 Sep 15 23:24 db
drwxr-xr-x 2 1000 1000 4096 Sep 15 23:24 design
-rw-r--r-- 1 1000 1000 561 Sep 30 13:16 docker-compose-db-only.yml
-rw-r--r-- 1 1000 1000 1218 Sep 30 12:22 docker-compose-local-dev.yml
-rw-r--r-- 1 1000 1000 988 Sep 29 00:48 docker-compose.yml
-rwxr-xr-x 1 1000 1000 2755 Sep 30 12:22 docker-entrypoint.sh
-rw-r--r-- 1 1000 1000 1541363 Sep 27 00:20 makeabilitylab-test.sql
Note that one file in there is makeabilitylab-test.sql. You can download this file from Dropbox and move it to this dir.
Let's try to import this sql file into PostgreSQL.
So, then I ran: psql -U admin -d makeability -f makeabilitylab-test.sql and it appeared to work:
root@ccd87abc9e1c:/usr/app# psql -U admin -d makeability -f makeabilitylab-test.sql
SET
SET
SET
psql:makeabilitylab-test.sql:14: ERROR: role "admin" already exists
ALTER ROLE
You are now connected to database "template1" as user "admin".
SET
SET
SET
SET
SET
set_config
------------
(1 row)
SET
SET
SET
SET
ALTER SCHEMA
...
You are now connected to database "postgres" as user "admin".
SET
SET
SET
SET
SET
set_config
------------
(1 row)
SET
SET
SET
SET
ALTER SCHEMA
REVOKE
GRANT
To verify, I reconnected to the database and tried to list the tables:
root@ccd87abc9e1c:/usr/app# psql makeability admin
psql (16.0 (Debian 16.0-1.pgdg120+1))
Type "help" for help.
makeability=# \dt
List of relations
Schema | Name | Type | Owner
--------+---------------------------------------+-------+-------
public | auth_group | table | admin
...
public | django_admin_log | table | admin
public | django_content_type | table | admin
public | django_migrations | table | admin
public | django_session | table | admin
public | easy_thumbnails_source | table | admin
public | easy_thumbnails_thumbnail | table | admin
public | easy_thumbnails_thumbnaildimensions | table | admin
public | website_banner | table | admin
public | website_poster | table | admin
public | website_poster_authors | table | admin
public | website_poster_projects | table | admin
...
public | website_talk_speakers | table | admin
public | website_video | table | admin
(43 rows)
Previously, I had zipped up the /media directory on makeabilitylab-test, which has all of the publications, talk files, images, etc. This zipped file was 25 GB due to the large .pptx files. I removed these and the smaller zipped file is 1GB: www-test-no-talk-files.zip. I unzipped this file and copied the data into the media directory (I did this using Windows because I have the Ubuntu filesystem mapped):
jonf@jonfhome:~/makeabilitylabwebsite/media$ pwd
/home/jonf/makeabilitylabwebsite/media
jonf@jonfhome:~/makeabilitylabwebsite/media$ ls -al
total 272
drwxrwxrwx 10 jonf jonf 4096 Sep 30 06:42 .
drwxr-xr-x 17 jonf jonf 4096 Sep 30 06:29 ..
drwxr-xr-x 2 jonf jonf 16384 Sep 30 06:41 banner
drwxr-xr-x 2 jonf jonf 4096 Sep 30 06:41 bibtex
drwxrwxrwx 3 jonf jonf 4096 Sep 15 16:24 images
drwxr-xr-x 2 jonf jonf 32768 Sep 30 06:41 news
drwxr-xr-x 4 jonf jonf 163840 Sep 30 06:41 person
-rw-r--r-- 1 jonf jonf 3612 Dec 30 2021 policy.xml
drwxr-xr-x 3 jonf jonf 4096 Sep 30 06:41 projects
drwxr-xr-x 3 jonf jonf 20480 Sep 30 06:41 publications
-rw-r--r-- 1 jonf jonf 27 Mar 9 2022 robots.txt
drwxr-xr-x 3 jonf jonf 4096 Sep 30 06:42 talks
-rw-r--r-- 1 jonf jonf 20 Dec 15 2017 test.txt
Now, I'm gonna try to run the regular docker-compose and see if we can get this to work!
jonf@jonfhome:~/makeabilitylabwebsite$ docker-compose -f docker-compose-local-dev.yml up
And amazingly, it worked and has our content!
And I just successfully replicated this on Mac.