Import without database creation?

[mfizia]

Hey Threre,

maybe someone can help me out. I don't want to open an issue for that, so: Why is it necessary to create a datebase by default during the import task of nominatim?

I'm asking because i'm working in a company that has strict rules for database-user and the pgsql is running in a huge cluster of customized pgsql-instances. So I've got an empty database with a db-owner with full rights inside the database and a second read-only user. Our goal is to deploy a own OSM/Nominatim-Instance for internal usage, so that we don't need to call from LAN to WAN.

First thing we have done, was to build and ship the nominatim.so library and it's requirements inside of our pgsql-cluster images. As second step i tried with the latest-Version of Nominatim (3.7.1) to import a small region and got - as i expected - an error from "createdb". I took a look around the issues and discussions here and found this: #423 (comment)

This comment seems to be for the older versions of Nominatim, so is there a way to do an import without creating a new database?

Kind regards

[mtmail]

If I understand right you're saying a suitable database with database user and access rights already exists, but the user executing the import scripts cannot create a new database, either because of permissions, policy, missing file or convience.

You're right, with setup.php in earlier Nominatim versions skipping was possible. Or rather the setup with -all executed 10 steps and specificying 9 step instead made it possible to skip the first step.

Starting Nominatim 3.7 the setup.php got replaced by a nominatim command-line interface (https://nominatim.org/2021/02/25/nominatim-tool.html) and almost all code rewritten in python. By reading the code it looks like the createdb step can't currently be skipped

Nominatim/nominatim/tools/database_import.py

Line 23 in 9257262

def setup_database_skeleton(dsn, data_dir, no_partitions, rouser=None):

Essentially a feature request to be able to skip it again.

One work-around is editing the logic locally (nominatim/tools/database_import.py). Another might be to create a script createdb that does nothing and make sure it get found (shell $PATH environment variable) before the "real" postgresql createdb binary.

[mfizia]

Good Evening @mtmail,

thank you for your fast reply.
I tried to modfiy the import line and there we are. The creating is skipped.
In my case, i changed the command to ls instead of createdb. Sweet little hack so far, but if i get this working in our environment and i got the time to do it - i will implement this feature. But i think there is a lot more to do, so i'm stucked now in the strict rights management of our company environment.

What i get now is:
Installing database module
Error accessing database module: permission denied for language c

But this could be a problem in our environment. I will speak to our database adminstrators tomorrow.
Just in case i'm wrong, i would like to double check the module path:
Is it correct that the NOMINATIM_DATABASE_MODULE_PATH uses a trailing slash?
NOMINATIM_DATABASE_MODULE_PATH=/usr/lib/postgresql/13/lib/

Kind Regards.

[lonvia]

Non-superuser imports were never really officially supported because the database creation may not be the only step that requires special database rights. 3.7.x still has the old util functions, so the hack described in #423 should still work. No guarantees. You basically have to follow the installation instructions from 3.6. and run Nominatim from the build directory.

The 4.0 version of Nominatim will get rid of the database module which is probably the biggest obstacle with non-superuser imports. I think its time to revisit the work flow then and see if we can have an official non-privileged import mode.

[mfizia]

Hey @lonvia,

thank you for your reply.

Today i made it.

For those who wants to know how i got this working:
First of all: I'm lazy i used the mediagis/nominatim-docker container for the import.

I ran the container with this command and used the nominatim-cli for importing:
docker run -it -v "/home/mfizia/nominatim-import-workaround/bremen-latest.osm.pbf:/tmp/bremen-latest.osm.pbf" -v "/home/mfizia/nominatim-import-workaround/nominatim:/nominatim" -v "/home/mfizia/nominatim-import-workaround/database_import.py:/usr/local/lib/nominatim/lib-python/nominatim/tools/database_import.py" -v "/home/mfizia/nominatim-import-workaround/status.py:/usr/local/lib/nominatim/lib-python/nominatim/db/status.py" -v "/home/mfizia/nominatim-import-workaround/data:/data" -p 7070:8080 <internal docker-hub-proxy>/mediagis/nominatim:3.7 /bin/bash

As @lonvia mentioned, the first obstacle to take was the nominatim.so.
We integrated the build of the library - as described here - into our container builds for our pgsql-cluster image. So we ship it on every new image version directly. This was relative easy for us. The more annoying obstacle for the library was the privilege to add a FUNCTION for the Language C. I need to ask a database administrator to activate this right. By default this is disabled in our environment due to securtity reasons.

The second step was to work around the database creation. As @mtmail mentioned above there are multiple ways for doing this.
I took the possibility to remap the database_import.py for my nominatim-docker container:
-v "/home/mfizia/nominatim-import-workaround/database_import.py:/usr/local/lib/nominatim/lib-python/nominatim/tools/database_import.py"
I changed the command 'createdb' to an simple 'ls' in this line,
So you are skipping the database creation and get an simple ls output during this step. Maybe not the best solution, but it worked.

The last obstacle was in the status.py. In our environment there is no internet at any machine in the DC-LAN. So the very last import step failed in this line. Because we can not get the timestamp for the maximum osmid to update the status table.
I managed this, by a simple manual lookup for the printed osmid. After this i hardcoded the timestamp attribute into the status.py and remapped the file again against the nominatim-docker container:
-v "/home/mfizia/nominatim-import-workaround/status.py:/usr/local/lib/nominatim/lib-python/nominatim/db/status.py"

Back to topic:

The 4.0 version of Nominatim will get rid of the database module which is probably the biggest obstacle with non-superuser imports. I think its time to revisit the work flow then and see if we can have an official non-privileged import mode.

That's good news. I also love the new environment and project-dir handling. it makes the administration a lot easier.
But in my point of view, you need a possibility for non-internet environments, too. In my company a way out the DC-LAN is long and bureaucratic. You discuss every outgoing or incoming connection with the it security. That's why we want to use it "locally".
Maybe there is another way to get this information.

Thank you for your suggestions and replies at this topic. The next thing i will do is a test for updating the data. Hopefully it will work the same way.

[lonvia]

Not being able to get the data should not be a fatal error. It will print a warning but not fail. There was a bug in status.php for databases without a date but that has been fixed in #2314.