Different search query results between own and public nominatim installation

[onock]

Hello,

We run our own nominatim servers with https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf (Nov 27 03:29) and nominatim version 4.0.1 and database updates running via nominatim replication. We add wikimedia, postcodes_gb, postcodes_us and tiger data. In addition we install cronjobs for nominatim refresh --postcodes and removing of large deleted objects.

OS: Ubuntu 20.04.4 LTS
Database version: 12.10 (Ubuntu 12.10-0ubuntu0.20.04.1)
PostGIS version: 3.0

Now we get some different output for queries like

curl "https://own.server.tld/search?city=Lienz&format=jsonv2"
[{"place_id":119181,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"node","osm_id":25181326,"boundingbox":["48.1459078","48.4659078","14.126198","14.446198"],"lat":"48.3059078","lon":"14.286198","display_name":"Linz, Oberösterreich, 4010, Österreich","place_rank":16,"category":"place","type":"city","importance":0.62171012248016055},{"place_id":284790196,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"relation","osm_id":156566,"boundingbox":["46.8139875","46.8451726","12.6953694","12.8086157"],"lat":"46.82958615","lon":"12.750316209629112","display_name":"Stadt Lienz, Bezirk Lienz, Tirol, 9900, Österreich","place_rank":16,"category":"boundary","type":"administrative","importance":0.60069263038760268},{"place_id":525279,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"node","osm_id":206378027,"boundingbox":["46.7898662","46.8698662","12.7281269","12.8081269"],"lat":"46.8298662","lon":"12.7681269","display_name":"Lienz, Stadt Lienz, Bezirk Lienz, Tirol, 9900, Österreich","place_rank":18,"category":"place","type":"town","importance":0.41000000000000003},{"place_id":578592,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"node","osm_id":240115278,"boundingbox":["45.9","47.9","11.5","13.5"],"lat":"46.9","lon":"12.5","display_name":"Lienz, Gemeinde Hopfgarten in Defereggen, Bezirk Lienz, Tirol, 9961, Österreich","place_rank":18,"category":"place","type":"region","importance":0.41000000000000003},{"place_id":551909,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"node","osm_id":240028641,"boundingbox":["47.2574546","47.2974546","9.4960452","9.5360452"],"lat":"47.2774546","lon":"9.5160452","display_name":"Lienz, Altstätten, Wahlkreis Rheintal, Sankt Gallen, 9464, Schweiz/Suisse/Svizzera/Svizra","place_rank":19,"category":"place","type":"village","importance":0.29147283909189603},{"place_id":561455,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"node","osm_id":240041742,"boundingbox":["46.7093787","46.7493787","14.0310624","14.0710624"],"lat":"46.7293787","lon":"14.0510624","display_name":"Linz, Tiffnerwinkl, Himmelberg, Bezirk Feldkirchen, Kärnten, 9562, Österreich","place_rank":20,"category":"place","type":"hamlet","importance":0.25},{"place_id":493441,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"node","osm_id":175277149,"boundingbox":["51.3214241","51.3614241","13.7145255","13.7545255"],"lat":"51.3414241","lon":"13.7345255","display_name":"Linz, Schönfeld, Meißen, Sachsen, 01561, Deutschland","place_rank":19,"category":"place","type":"village","importance":0.22254953489334617},{"place_id":16876117,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"node","osm_id":1598419331,"boundingbox":["50.1404087","50.1804087","13.2972963","13.3372963"],"lat":"50.1604087","lon":"13.3172963","display_name":"Mlýnce, Vroutek, okres Louny, Ústecký kraj, Severozápad, 43983, Česko","place_rank":19,"category":"place","type":"village","importance":0.18147283909189599},{"place_id":16962722,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"node","osm_id":1599437564,"boundingbox":["49.4511518","49.4711518","12.7931147","12.8131147"],"lat":"49.4611518","lon":"12.8031147","display_name":"Mlýnec, Postřekov, okres Domažlice, Plzeňský kraj, Jihozápad, 34535, Česko","place_rank":20,"category":"place","type":"neighbourhood","importance":0.18147283909189599}]

vs

curl "https://nominatim.openstreetmap.org/search.php?city=Lienz&format=jsonv2"
[{"place_id":281809102,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"relation","osm_id":156566,"boundingbox":["46.8139875","46.8451726","12.6953694","12.8086157"],"lat":"46.82958615","lon":"12.750316209629112","display_name":"Stadt Lienz, Bezirk Lienz, Tirol, 9900, Österreich","place_rank":16,"category":"boundary","type":"administrative","importance":0.6006926303876027,"icon":"https://nominatim.openstreetmap.org/ui/mapicons//poi_boundary_administrative.p.20.png"},{"place_id":521946,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"node","osm_id":206378027,"boundingbox":["46.7898662","46.8698662","12.7281269","12.8081269"],"lat":"46.8298662","lon":"12.7681269","display_name":"Lienz, Stadt Lienz, Bezirk Lienz, Tirol, 9900, Österreich","place_rank":18,"category":"place","type":"town","importance":0.41000000000000003,"icon":"https://nominatim.openstreetmap.org/ui/mapicons//poi_place_town.p.20.png"},{"place_id":558402,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"node","osm_id":240115278,"boundingbox":["45.9","47.9","11.5","13.5"],"lat":"46.9","lon":"12.5","display_name":"Lienz, Gemeinde Hopfgarten in Defereggen, Bezirk Lienz, Tirol, 9961, Österreich","place_rank":18,"category":"place","type":"region","importance":0.41000000000000003},{"place_id":546001,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"node","osm_id":240028641,"boundingbox":["47.2574546","47.2974546","9.4960452","9.5360452"],"lat":"47.2774546","lon":"9.5160452","display_name":"Lienz, Altstätten, Wahlkreis Rheintal, Sankt Gallen, 9464, Schweiz/Suisse/Svizzera/Svizra","place_rank":19,"category":"place","type":"village","importance":0.29147283909189603,"icon":"https://nominatim.openstreetmap.org/ui/mapicons//poi_place_village.p.20.png"}]

or

curl "https://own.server.tld/search?street=Bahnhofstrasse+4a&city=Lichtenberg&postalcode=09638&country=Deutschland&countrycodes=DE&format=json&accept-language=de&addressdetails=1&limit=1"
[{"place_id":5210601,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"node","osm_id":559168411,"boundingbox":["52.5451239","52.5452239","13.4978276","13.4979276"],"lat":"52.5451739","lon":"13.4978776","display_name":"4a, Bahnhofstraße, Alt-Hohenschönhausen, Lichtenberg, Berlin, 13055, Deutschland","class":"place","type":"house","importance":0.33100000000000007,"address":{"house_number":"4a","road":"Bahnhofstraße","suburb":"Alt-Hohenschönhausen","borough":"Lichtenberg","city":"Berlin","state":"Berlin","postcode":"13055","country":"Deutschland","country_code":"de"}}]

vs

curl "https://nominatim.openstreetmap.org/search.php?street=Bahnhofstrasse+4a&city=Lichtenberg&country=Deutschland&postalcode=09638&accept-language=de&countrycodes=DE&limit=1&format=jsonv2"[{"place_id":158622118,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"way","osm_id":231008647,"boundingbox":["50.8436437","50.8438467","13.4044304","13.4046916"],"lat":"50.8437452","lon":"13.404561000000001","display_name":"4a, Bahnhofstraße, Lichtenberg/Erzgebirge, Lichtenberg-Weißenborn, Mittelsachsen, Sachsen, 09638, Deutschland","place_rank":30,"category":"building","type":"yes","importance":0.431}]

Every time the public installation gives the correct result in contrast to the own installation.
I tried to update nominatim to current master but the differences still exist.

Status report after nominatim update from 4.0.1 to current master:
{"status":0,"message":"OK","data_updated":"2022-05-20T10:40:02+00:00","software_version":"4.0.99-6","database_version":"4.0.99-6"}

My next solution attempt would now be a fresh planet import. But since this takes a very long time, I wanted to ask here before once:
What am I missing here? How can I solve this so that the same output as public appears in our own installation?

Kind regards

[mtmail]

Reformatting to highlight the differences:

1. query city=Lienz

• nominatim.osm.org: Stadt Lienz, Bezirk Lienz, Tirol, 9900, Österreich
• own server: Linz, Oberösterreich, 4010, Österreich

2. query street=Bahnhofstrasse+4a&city=Lichtenberg&country=Deutschland&postalcode=09638

• nominatim.osm.org: 4a, Bahnhofstraße, Lichtenberg/Erzgebirge, Lichtenberg-Weißenborn, Mittelsachsen, Sachsen, 09638
• own server: 4a, Bahnhofstraße, Alt-Hohenschönhausen, Lichtenberg, Berlin, 13055

nominatim.osm.org runs the latest master (unstable) version of Nominatim. It also uses the new ICU tokenizer, which you can enable in your .env file https://nominatim.org/2021/11/03/release-40.html

I checked on my local fresh installation and it matches nominatim.osm.org.

$ nominatim search --city Lienz | jq -c '.[] | [.osm_type, .osm_id, .display_name]'
["relation",156566,"Stadt Lienz, Bezirk Lienz, Tirol, 9900, Österreich"]
["node",206378027,"Lienz, Stadt Lienz, Bezirk Lienz, Tirol, 9900, Österreich"]

$ nominatim search --street 'Bahnhofstrasse 4a' --city Lichtenberg --country Deutschland --postalcode 09638 | jq -c '.[] | [.osm_type, .osm_id, .display_name]'
["way",231008647,"4a, Bahnhofstraße, Lichtenberg/Erzgebirge, Lichtenberg-Weißenborn, Mittelsachsen, Sachsen, 09638, Deutschland"]

So highly likely that installting the master version (https://nominatim.org/release-docs/develop/admin/Installation/), which now has ICU tokenizer enbled by default, will help. Or wait for Nominatim 4.1 stable release.

[onock]

Looks good, thanks for your help!

[knee11]

Hi All,

We are also encountering this problem on our newly installed Nominatim 4.0.1 and the Tokenizer is present.

Seeking your help for this one, thank you!

[mtmail]

@knee11 Open a new discussion and give specific examples.

[knee11]

@mtmail noted on this sir, I just have one question. If we have an existing Nominatim, is it possible to change this legacy_tokenizer.php to icu_tokenizer.php?

[mtmail]

Please create a new discussion. I’m locking this discussion for new comments.