wrong parent place for no apparent reason

[jtopenpetition]

We have a local installation of nominatim and we built a query that outputs the hierarchy for administrative boundaries that are stored in placex.parent_place_id / place_addressline. When we set it up, it produced correct results for our sample data (which was all administrative (sub) boundaries for Berlin, Germany), specifically the place with the osm id R55764 had the parent place with osm id R62422 (which is correct, thus, expected).

However, since we installed nominatim a few updates were run and now R55764 apparently belongs to R1318337 (Bernau, which is distinctly outside of Berlin and has no overlap at all). We are very confused about how that could happen and are wondering, if there's a way to "fix" that without a complete reinstall, those are very inconvenient ... and we would have to do those regularly. (in case it's relevant: obviously, we don't modify the data. we only import the changes from osm)

nominatim response when we lookup R55764

the parent place should be Berlin, but it's Bernau instead ...

Obivously, openstreetmap produces the correct results, which is the results we would like to get ...

is there any known bug that could cause these inconsistencies? any advice on how to handle this?

[lonvia]

What data did you use for your original import? What data did you use when you applied the updates? What do the relations (and in particular the geometries) of 'Berlin' and 'Bernau' look like when you look at them in the details view?

[jtopenpetition]

The data we imported were from geofabrik.de, the Europe extract, updates are also from geofabrik.de. Last update was 2022-05-13, after which the update service died due to network problems, it seems... but that shouldn't matter, I hope.

The geometries so far appear to be correct, in that containment and intersections are fine. I also checked their placex.geometry, and they look exactly as they're supposed to look.

WITH places AS (
   SELECT * 
   FROM placex 
   WHERE osm_id IN ('1318337', '55764', '62422', '62504')
   AND osm_type = 'R'
) 
SELECT 
   l.name->'name' as left, 
   ST_DISTANCE(l.geometry, r.geometry), 
   ST_CONTAINS(l.geometry, r.geometry), 
   ST_INTERSECTS(l.geometry, r.geometry), 
   r.name->'name' as right 
FROM places l 
CROSS JOIN places r;

left	st_distance	st_contains	st_intersects	right
Friedrichshain-Kreuzberg	0	t	t	Friedrichshain-Kreuzberg
Friedrichshain-Kreuzberg	0	f	t	Berlin
Friedrichshain-Kreuzberg	0.0676549075334527	f	f	Brandenburg
Friedrichshain-Kreuzberg	0.135453322558765	f	f	Bernau
Berlin	0	t	t	Friedrichshain-Kreuzberg
Berlin	0	t	t	Berlin
Berlin	0	f	t	Brandenburg
Berlin	0.0126248228502393	f	f	Bernau
Brandenburg	0.0676549075334527	f	f	Friedrichshain-Kreuzberg
Brandenburg	0	f	t	Berlin
Brandenburg	0	t	t	Brandenburg
Brandenburg	0	t	t	Bernau
Bernau	0.135453322558765	f	f	Friedrichshain-Kreuzberg
Bernau	0.0126248228502393	f	f	Berlin
Bernau	0	f	t	Brandenburg
Bernau	0	t	t	Bernau
(16 rows)

[jtopenpetition]

follow-up question/comment: is there a way to let nominatim recalculate parent places? I believe it's not really feasible to modify the parent_place_id manually. I would be really grateful for any advice on what to (preferably) do. (also updated my original comment, we update with extracts from geofabrik.de)

[mtmail]

You can run SELECT place_force_update(place_id) FROM placex WHERE osm_type = ... and osm_id = ... on for example Berlin. That will mark (set indexed_status = 2) of that place and all children recursively. Then run nominatim index which recalculates those relationships. It will affected 1000s of places but can run in the background.

https://github.com/osm-search/Nominatim/pull/2666/files added an admin command to make these steps easier. I will be part of the next stable release.

[jtopenpetition]

thank you @mtmail for your quick response. I tried the command, forcing updates for both Berlin and Brandenburg, but after running for quite some time, the results are still the same. Is there anything else we can try? I'm wondering if some different process is involved in calculating administrative borders (which all of the above are). In generell I'm confused about how the parent_place_id could ever be determined if not by geometry...

[lonvia]

I suspect that this is another manifestation of #2635. Friedrichshain-Kreuzberg has a couple of is_in:* tags that range from useless to outright wrong. Ironically, it is one of the more correct ones that trips Nominatim. is_in:city=Berlin gets a partial match on 'Bernau bei Berlin' and that's how Bernau ends up in the address. (Clearly a bug as there is a city 'Berlin' that is closer.)

[jtopenpetition]

that's fair. we actually have been using place_addressline for getting the hierarchies. (however, a significantly wrong parent_place_id appeared to be a strong indicator of something being quite wrong in place_addressline - if place_addressline has correct data, parent_place_id shouldn't be completely wrong.)

There is a very similar case with another suburb of Berlin, Marzahn-Hellersdorf (i joined the right side for more helpful information than place_ids):

SELECT DISTINCT ON (p.place_id) p.osm_id, p.osm_type, p.name->'name' as childname, a.*, r.name->'name' as parent_name, r.osm_id, r.osm_type 
FROM placex p, place_addressline a, placex r
WHERE p.class = 'boundary' and p.type = 'administrative' and p.rank_search < 25
      and a.place_id = p.place_id
      and a.fromarea and a.isaddress and p.osm_id in (164712) and p.osm_type = 'R' and r.place_id = address_place_id 
ORDER BY p.place_id, a.cached_rank_address DESC;

which produces as it's "parent" Schöneiche bei Berlin, which is not in Berlin, and also isn't the "parent" region of Marzahn-Hellersdorf. Removing the distinct from the query also produces Berlin (the correct parent):

 osm_id | osm_type |      childname      | place_id  | address_place_id |     distance      | cached_rank_address | fromarea | isaddress |      parent_name      | osm_id  | osm_type 
--------+----------+---------------------+-----------+------------------+-------------------+---------------------+----------+-----------+-----------------------+---------+----------
 164712 | R        | Marzahn-Hellersdorf | 139572752 |        139667808 | 0.116935159740301 |                  16 | t        | t         | Schöneiche bei Berlin | 1352215 | R
 164712 | R        | Marzahn-Hellersdorf | 139572752 |        139547488 |  0.19081845319546 |                   8 | t        | t         | Berlin                |   62422 | R
(2 rows)

The distance is greater (although the suburb is very clearly contained in Berlin).

Is it correct to assume that since Berlin fulfills multiple roles (county/city, state), but only is marked as state, it has too great of an admin_level to be considered the closest ancestor, hierarchy-wise?

I hope it's not relevant, but we use the icu tokenizer.

[lonvia]

Yes, the 'place=state' on the Berlin admin boundary is creating a serious mess for Nominatim because it breaks the assumption that 'place' contains the settlement form. Eventually this goes back to my comment in #2635: the only way to make it better is to disallow partial matches between names and addr:* tags. Problem is as always that it breaks other cases.

I'm tempted to just remove all that 'is_in:*' mess in Berlin and be done with it.

If you use the database for the hierarchies only, then you might actually be better off with the legacy tokenizer because this one does the matching by full names only, if I remember correctly.

[jtopenpetition]

sadly, we don't use it for the hierarchies only. ;o/

I'm already stealing too much of your time (sorry). I was wondering why nominatim uses the minimum distance from the (potential parent) place's centroid, ordering by having an intersection and then by that potentially dangerous distance. Seemed odd to me and I presumed performance being the main reason. Upwards a certain level/rank - where there are mostly somewhat well-defined boundaries - I would've expected not just ordering by intersection but filtering by intersection, which would most likely remove some of the weirder candidates. But as you already mentioned, there are probably other cases that would break... of which I know nothing (yet). And I'm wondering, if this problem will have a less "tags-vulnerable" (or maybe tags-quality dependant) solution or if no real solution should be expected...

Disclaimer: I'm most likely far less proficient with nominatim's inner workings and I have the feeling I'm overstepping quite a lot here, so I apologize. I want to emphasize, that I'm really grateful for the insights you have provided so far, and I'm somewhat invested in finding a solution, yet, not equipped with the resources or knowledge to do so. Also Thank you for your work and support. You've gone far and beyond.

[lonvia]

The query with the fix has a bit of a special function. It is only applied to find address components that have been explicitly referenced in the OSM object with a addr:* or is_in:* tag. It explicitly allows for more faraway objects because when in comes to addresses, the administrative boundaries are not always the same as the postal boundaries. See #1120 (comment) for an example in the US. That said, I have already wondered if Nominatim should go back to being strict for anything above street level. The OSM data is just too much of a mess.

[lonvia]

Oh well, I've cleaned up the tags in Berlin. That was long overdue anyways. Lets see if there are any complaints.