Difference in standalone and cloud nominatim databases

[danielabelski]

Describe the bug
Standalone nominatim installation accumulated differences from the nominatim.openstreetmap.org and those are not fixed in updates. We had this in nominatim 3 and after the update still have it here.

Here is an example
our results

and
nominatim.openstreetmap.org

you see, that the local name section is entirely different. And those diffs pile and add all the time.

To Reproduce
Run daily updates and observe the drift. Its difficult to predict where it will occurs. Fresh nominatim install seems to be ok, bit after running the updates we get this. What are we doing wrong?

We run daily - nominatim replication --catch-up --threads 8

NOMINATIM_REPLICATION_URL="https://planet.openstreetmap.org/replication/day"
NOMINATIM_REPLICATION_MAX_DIFF=5000
NOMINATIM_FLATNODE_FILE="/srv/nominatim/flatnodePlace/flatnode.file"

Software Environment (please complete the following information):

• Nominatim version: 4.0.1
• Postgresql version: 14.2-1.pgdg20.04+1+b1
• Postgis version: 3.2.1
• OS: ubuntu 20.04

Hardware Configuration (please complete the following information):

• RAM: 64
• number of CPUs: 8
• type and size of disks: ZFS on SSD
• bare metal/AWS/other cloud service: AWS

Postgresql Configuration:

max_connections = 2000
shared_buffers = 16GB
maintenance_work_mem = 4GB
work_mem = 100MB
effective_cache_size = 24GB
synchronous_commit = off
checkpoint_timeout = 60min
checkpoint_completion_target = 0.9
fsync = on
full_page_writes = on
max_parallel_workers_per_gather = 0
jit = off

Additional info
For nominatim 3 there were partial updates that allowed to reload an object and it helped somehow, but I dont see this in v4. How people fix this in their installations?

[lonvia]

Have you kept the flatnode file from the import and use the same for the updates?

[danielabelski]

Have you kept the flatnode file from the import and use the same for the updates?

yes

[lonvia]

Well, it looks like you are loosing your areas on updates and that's usually a sign that the node coordinates are not properly stored. And that in turn is pretty much always an error in your setup.

A couple of things that might help you debug the issue:

• Do the relations that are in the osm.org response still exist on your server. Check first the details page for 'R3488095' and then in your database 'SELECT count(*) FROM planet_osm_rels WHERE id = 3488095'.
• Is there a node backstore table planet_osm_nodes? IF yes, your initial import was not done with a flatnode file.

[lonvia]

Moving this to discussion as it is likely an installation issue and not a bug.

[danielabelski]

@lonvia I am a bit puzzled here.

1. We do the install as in https://nominatim.org/release-docs/latest/appendix/Install-on-Ubuntu-20/
2. We inport planet pbf as in https://nominatim.org/release-docs/latest/admin/Import/
3. Update we do catchup - https://nominatim.org/release-docs/latest/admin/Update/

there are no errors no nothing that can hint why this is happening.

As for your questions

1. SELECT count(*) FROM planet_osm_rels WHERE id = 3488095 gives nothing
2. nominatim=# SELECT table_name FROM information_schema.tables where table_name like '%planet%';
   table_name

---

planet_osm_ways
planet_osm_rels
(2 rows)

so there is no planet_osm_nodes

Still need help.

[lonvia]

You will have to provide exact details about all commands used for the import and the updates including adaptions you had to do for making it work on AWS. "We followed the installation instructions." is not enough. The log of an update run might have useful hints as well.

[mtmail]

For nominatim 3 there were partial updates that allowed to reload an object

You probably mean reindex an object. You can do that with SELECT place_force_update(place_id) FROM placex WHERE osm_type = ... and osm_id = ... That will mark (set indexed_status = 2) of that place and all children recursively. Then start the indexing with nominatim index (single threaded, can run in background or add --threads ... and use all CPU cores but that will affect search performance). Thus you can reindex a whole

https://github.com/osm-search/Nominatim/pull/2666/files added an admin command to make these steps easier. I will be part of the next stable release.

SELECT count(*) FROM planet_osm_rels WHERE id = 3488095 gives nothing

Reindexing won't help in this case though. Reindexing take data from the planet_osm_* (and flatnode file) and updates data in the placex table. Here something already went wrong applying the OSM update to the planet_osm_* data.

[lonvia]

I think he means the command that since 3.7 is called nominatim add-data --node/--way/--relation. It can function as a band-aid for the symptoms but it won't fix the root cause.

[danielabelski]

Yes, this band-aid helped

[danielabelski]

I think he means the command that since 3.7 is called nominatim add-data --node/--way/--relation. It can function as a band-aid for the symptoms but it won't fix the root cause.

The reason updated ruin our database unknown to me. I will make a step-by-step and post it here... it just needs some time