Setup Error: Error loading transliteration module

[TheLegend29]

Trying to install on the following :
Linux HP-Pavilion-15-Notebook-PC 4.8.0-41-generic #44~16.04.1-Ubuntu SMP Fri Mar 3 17:11:16 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

When I try to check ./src/address_parser I get the following error:
Loading models...
ERR Error loading transliteration module, dir=(null)
at libpostal_setup_datadir (libpostal.c:1069) errno: No such file or directory

The following are the steps I've used:

cd ~
rm -rf libpostal
git clone https://github.com/openvenues/libpostal
cd libpostal
./bootstrap.sh
./configure LDFLAGS=-L/usr/lib64 --datadir=$(pwd)/data --prefix=$(realpath $(pwd)) --bindir=$(realpath $(pwd)/bin)
make install
sudo ldconfig
./src/address_parser

Any guidance would be great. I'm a newbie ubuntu user but would love to check this module out for python.

[albarrentine]

That sounds like the data files didn't download. Does the dir you specified for --datadir when running configure have enough space? It needs approximately 2.2G free to build the current version of libpostal (that number may change in subsequent updates to the models).

Also, looking at the "dir=(null)" piece, which prints the value of LIBPOSTAL_DATA_DIR on error, it's possible that something went wrong in configure. Did all of those commands run successfully?

Another random thing I noticed: it looks like $(realpath $(pwd)) is used for the other paths but only $(pwd) is used --datadir, if that has anything to do with it (it shouldn't, pwd returns an absolute path already AFAIK but if it doesn't on your system that might be it).

[TheLegend29]

According to the properties there is 682gb of free space. Is there a better way you'd suggest to check this?
If I try to put realpath in the datadir like in the following :
./configure LDFLAGS=-L/usr/lib64 --datadir=$(realpath $(pwd)) /data --prefix=$(realpath $(pwd)) --bindir=$(realpath $(pwd)/bin)
I get a config fail with the output looking like this:
configure: WARNING: you should use --build, --host, --target
configure: WARNING: invalid host type: /data
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking build system type... Invalid configuration /data': machine /data' not recognized
configure: error: /bin/bash ./config.sub /data failed

When running configure using the original command above I get the following output, which seems successful to me?:
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking how to print strings... printf
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... gcc3
checking for a sed that does not truncate output... /bin/sed
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for fgrep... /bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @file support... @
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for a working dd... /bin/dd
checking how to truncate binary pipes... /bin/dd bs=4096 count=1
checking for mt... mt
checking if mt is a manifest tool... no
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for gcc option to accept ISO C99... none needed
checking for library containing snappy_compress... -lsnappy
checking for library containing log... -lm
checking for ANSI C header files... (cached) yes
checking whether time.h and sys/time.h may both be included... yes
checking for dirent.h that defines DIR... yes
checking for library containing opendir... none required
checking for stdbool.h that conforms to C99... yes
checking for _Bool... yes
checking fcntl.h usability... yes
checking fcntl.h presence... yes
checking for fcntl.h... yes
checking float.h usability... yes
checking float.h presence... yes
checking for float.h... yes
checking for inttypes.h... (cached) yes
checking limits.h usability... yes
checking limits.h presence... yes
checking for limits.h... yes
checking locale.h usability... yes
checking locale.h presence... yes
checking for locale.h... yes
checking malloc.h usability... yes
checking malloc.h presence... yes
checking for malloc.h... yes
checking for memory.h... (cached) yes
checking stddef.h usability... yes
checking stddef.h presence... yes
checking for stddef.h... yes
checking for stdint.h... (cached) yes
checking for stdlib.h... (cached) yes
checking for string.h... (cached) yes
checking for unistd.h... (cached) yes
checking for inline... inline
checking for int16_t... yes
checking for int32_t... yes
checking for int64_t... yes
checking for int8_t... yes
checking for off_t... yes
checking for size_t... yes
checking for ssize_t... yes
checking for uint16_t... yes
checking for uint32_t... yes
checking for uint64_t... yes
checking for uint8_t... yes
checking for ptrdiff_t... yes
checking for stdlib.h... (cached) yes
checking for unistd.h... (cached) yes
checking for sys/param.h... yes
checking for getpagesize... yes
checking for working mmap... yes
checking for malloc... yes
checking for realloc... yes
checking for getcwd... yes
checking for gettimeofday... yes
checking for memmove... yes
checking for memset... yes
checking for munmap... yes
checking for regcomp... yes
checking for setlocale... yes
checking for sqrt... yes
checking for strdup... yes
checking for strndup... yes
checking for shuf... yes
configure: extra cflags for scanner.c:
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating libpostal.pc
config.status: creating src/Makefile
config.status: creating src/sparkey/Makefile
config.status: creating test/Makefile
config.status: creating config.h
config.status: executing depfiles commands
config.status: executing libtool commands

Lastly, in the make install I noticed a few lines. Are these normal to get?
In file included from collections.h:8:0,
from averaged_perceptron.h:26,
from address_parser.h:49,
from address_parser.c:1:
address_parser.c: In function ‘address_parser_context_fill’:
log/log.h:26:55: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 3 has type ‘int64_t {aka long int}’ [-Wformat=]
#define log_debug(M, ...) do { if (0) fprintf(stderr, "\33[34mDEBUG\33[39m " M " \33[90m at %s (%s:%d) \33[39m\n", ##VA_ARGS, func, _
^
address_parser.c:329:17: note: in expansion of macro ‘log_debug’
log_debug("token i=%lld, null phrase membership\n", i);
^
log/log.h:26:55: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 3 has type ‘int64_t {aka long int}’ [-Wformat=]
#define log_debug(M, ...) do { if (0) fprintf(stderr, "\33[34mDEBUG\33[39m " M " \33[90m at %s (%s:%d) \33[39m\n", ##VA_ARGS, func, _
^
address_parser.c:333:17: note: in expansion of macro ‘log_debug’
log_debug("token i=%lld, phrase membership=%lld\n", i, j);
^
log/log.h:26:55: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 4 has type ‘int64_t {aka long int}’ [-Wformat=]
#define log_debug(M, ...) do { if (0) fprintf(stderr, "\33[34mDEBUG\33[39m " M " \33[90m at %s (%s:%d) \33[39m\n", ##VA_ARGS, func, _
^
address_parser.c:333:17: note: in expansion of macro ‘log_debug’
log_debug("token i=%lld, phrase membership=%lld\n", i, j);
^
log/log.h:26:55: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 3 has type ‘int64_t {aka long int}’ [-Wformat=]
#define log_debug(M, ...) do { if (0) fprintf(stderr, "\33[34mDEBUG\33[39m " M " \33[90m at %s (%s:%d) \33[39m\n", ##VA_ARGS, func, _
^
address_parser.c:340:9: note: in expansion of macro ‘log_debug’
log_debug("token i=%lld, null phrase membership\n", i);
^
log/log.h:26:55: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 3 has type ‘int64_t {aka long int}’ [-Wformat=]
#define log_debug(M, ...) do { if (0) fprintf(stderr, "\33[34mDEBUG\33[39m " M " \33[90m at %s (%s:%d) \33[39m\n", ##VA_ARGS, func, _
^
address_parser.c:356:17: note: in expansion of macro ‘log_debug’
log_debug("token i=%lld, null geo phrase membership\n", i);
^
log/log.h:26:55: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 3 has type ‘int64_t {aka long int}’ [-Wformat=]
#define log_debug(M, ...) do { if (0) fprintf(stderr, "\33[34mDEBUG\33[39m " M " \33[90m at %s (%s:%d) \33[39m\n", ##VA_ARGS, func, _
^
address_parser.c:361:17: note: in expansion of macro ‘log_debug’
log_debug("token i=%lld, geo phrase membership=%lld\n", i, j);
^
log/log.h:26:55: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 4 has type ‘int64_t {aka long int}’ [-Wformat=]
#define log_debug(M, ...) do { if (0) fprintf(stderr, "\33[34mDEBUG\33[39m " M " \33[90m at %s (%s:%d) \33[39m\n", ##VA_ARGS, func, _
^
address_parser.c:361:17: note: in expansion of macro ‘log_debug’
log_debug("token i=%lld, geo phrase membership=%lld\n", i, j);
^
log/log.h:26:55: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 3 has type ‘int64_t {aka long int}’ [-Wformat=]
#define log_debug(M, ...) do { if (0) fprintf(stderr, "\33[34mDEBUG\33[39m " M " \33[90m at %s (%s:%d) \33[39m\n", ##VA_ARGS, func, _
^
address_parser.c:367:9: note: in expansion of macro ‘log_debug’
log_debug("token i=%lld, null geo phrase membership\n", i);
^
log/log.h:26:55: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 3 has type ‘int64_t {aka long int}’ [-Wformat=]
#define log_debug(M, ...) do { if (0) fprintf(stderr, "\33[34mDEBUG\33[39m " M " \33[90m at %s (%s:%d) \33[39m\n", ##VA_ARGS, func, _
^
address_parser.c:383:17: note: in expansion of macro ‘log_debug’
log_debug("token i=%lld, null component phrase membership\n", i);
^
log/log.h:26:55: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 3 has type ‘int64_t {aka long int}’ [-Wformat=]
#define log_debug(M, ...) do { if (0) fprintf(stderr, "\33[34mDEBUG\33[39m " M " \33[90m at %s (%s:%d) \33[39m\n", ##VA_ARGS, func, _
^
address_parser.c:388:17: note: in expansion of macro ‘log_debug’
log_debug("token i=%lld, component phrase membership=%lld\n", i, j);
^
log/log.h:26:55: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 4 has type ‘int64_t {aka long int}’ [-Wformat=]
#define log_debug(M, ...) do { if (0) fprintf(stderr, "\33[34mDEBUG\33[39m " M " \33[90m at %s (%s:%d) \33[39m\n", ##VA_ARGS, func, _
^
address_parser.c:388:17: note: in expansion of macro ‘log_debug’
log_debug("token i=%lld, component phrase membership=%lld\n", i, j);
^
log/log.h:26:55: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 3 has type ‘int64_t {aka long int}’ [-Wformat=]
#define log_debug(M, ...) do { if (0) fprintf(stderr, "\33[34mDEBUG\33[39m " M " \33[90m at %s (%s:%d) \33[39m\n", ##VA_ARGS, func, _
^
address_parser.c:394:9: note: in expansion of macro ‘log_debug’
log_debug("token i=%lld, null component phrase membership\n", i);
gcc -DHAVE_CONFIG_H -I.. -I/usr/local/include -Wfloat-equal -Wpointer-arith -DLIBPOSTAL_DATA_DIR='"/home/omer/libpostal/data/libpostal"' -g -g -O2 -O3 -MT build_numex_table-numex.o -MD -MP -MF .deps/build_numex_table-numex.Tpo -c -o build_numex_table-numex.o test -f 'numex.c' || echo './'numex.c
In file included from collections.h:8:0,
from numex.h:14,
from numex.c:3:
numex.c: In function ‘numex_table_read’:
log/log.h:26:55: warning: format ‘%llu’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘uint64_t {aka long unsigned int}’ [-Wformat=]
#define log_debug(M, ...) do { if (0) fprintf(stderr, "\33[34mDEBUG\33[39m " M " \33[90m at %s (%s:%d) \33[39m\n", ##VA_ARGS, func, _
^
numex.c:424:5: note: in expansion of macro ‘log_debug’
log_debug("read num_languages = %llu\n", num_languages);
^
log/log.h:26:55: warning: format ‘%llu’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘uint64_t {aka long unsigned int}’ [-Wformat=]
#define log_debug(M, ...) do { if (0) fprintf(stderr, "\33[34mDEBUG\33[39m " M " \33[90m at %s (%s:%d) \33[39m\n", ##VA_ARGS, func, _
^
numex.c:446:5: note: in expansion of macro ‘log_debug’
log_debug("read num_rules = %llu\n", num_rules);

[albarrentine]

There's an extra space in --datadir=$(realpath $(pwd)) /data, I think you want: --datadir=$(realpath $(pwd))/data instead.

It looks like everything compiled properly. The main thing to check is: du -h $(realpath $(pwd))/data and make sure that it's 2.2G. If not, try rm-ing the data dir and run make again. Toward the end of the make command there should be a line like ./libpostal_data download all $YOUR_DATA_DIR. If something went wrong when downloading the data files, that's where to look. If all else fails, nuke the entire checkout and start from a fresh clone.

[TheLegend29]

I think you're onto something with the 2.2G it's stuck on producing the below output before and after. I've also tried the whole process from scratch a few times with the same outcomes (the added realpath doesn't appear to make a difference). Not sure why the 2.2G isn't downloading there's no folder limit or read/write issues associated with the datadir. Sorry for the lengthy back and forth trying to get this to work.
8.0K /home/user/libpostal/data/address_parser
20K /home/user/libpostal/data/libpostal
8.0K /home/user/libpostal/data/geonames
8.0K /home/user/libpostal/data/transliteration
8.0K /home/user/libpostal/data/numex
8.0K /home/user/libpostal/data/geodb
8.0K /home/user/libpostal/data/address_expansions
72K /home/user/libpostal/data

[albarrentine]

Try running ./src/libpostal_data download all $(pwd)/data and see what happens.

[TheLegend29]

I get the following output -- aren't these the files that should be in data not in source?

Checking for new libpostal data file...
./src/libpostal_data: 108: ./src/libpostal_data: curl: not found
libpostal data file up to date
Checking for new libpostal geodb data file...
./src/libpostal_data: 108: ./src/libpostal_data: curl: not found
libpostal geodb data file up to date
Checking for new libpostal parser data file...
./src/libpostal_data: 108: ./src/libpostal_data: curl: not found
libpostal parser data file up to date
Checking for new libpostal language classifier data file...
./src/libpostal_data: 108: ./src/libpostal_data: curl: not found
libpostal language classifier data file up to date

looks like it's saving in src, looking at the size is this the expected behavior?

12K	./src/murmur
44K	./src/utf8proc/.deps
516K	./src/utf8proc/.libs
7.2M	./src/utf8proc
96K	./src/sparkey/.deps
752K	./src/sparkey/.libs
1.5M	./src/sparkey
20K	./src/cmp/.deps
188K	./src/cmp/.libs
1.5M	./src/cmp
8.0K	./src/log
20K	./src/geohash/.deps
52K	./src/geohash/.libs
320K	./src/geohash
64K	./src/klib
12K	./src/linenoise/.deps
156K	./src/linenoise
1.6M	./src/.deps
42M	./src/.libs
230M	./src/

[albarrentine]

curl is not installed. apt-get install curl, delete the datadir, and run make again.

[TheLegend29]

oh wow, that was a silly oversight, now it works. That likely means this problem is probably also silly but what would be the reason now when I try to bring it into python I get a similar error. I've done the following:

cd libpostal/scripts
python3 setup.py build_ext --inplace
cd /home/user/.local/lib/python3.5/site-packages/
nosetests postal/tests

I get the same kind of error (dir=null) (both in the nose test and if i try to import in python)

ERR   Error loading transliteration module, dir=(null)
   at libpostal_setup_datadir (libpostal.c:1069) errno: No such file or directory
ERR   Error loading transliteration module, dir=(null)
   at libpostal_setup_datadir (libpostal.c:1069) errno: No such file or directory
EE
======================================================================
ERROR: Failure: SystemError (initialization of _expand raised unreported exception)

[albarrentine]

Those are not the Python bindings, these are. Try: pip3 install postal

[TheLegend29]

yup so that's what I did before the above. However, when I ran in python I end up with the following error. i thought it was an error like the one we just solved where the libpostal didn't get put on properly. But now I'm confused since it works outside of python.

from postal.expand import expand_address
ERR Error loading transliteration module, dir=(null)
at libpostal_setup_datadir (libpostal.c:1069) errno: No such file or directory
Traceback (most recent call last):
File "", line 1, in
File "/home/omer/.local/lib/python3.5/site-packages/postal/expand.py", line 5, in
from postal import _expand
SystemError: initialization of _expand raised unreported exception

[albarrentine]

It seems like there were some non-standard steps taken during the install, so it may be in a weird half-installed state. Try a clean build now that curl is installed and the data files can be downloaded using the exact steps from the README for Ubuntu.