apertium crashes with bad_alloc

[jelmervdl]

I've attached an excerpt of about 25 lines that causes apertium to be busy for a long time and eventually crash with a bad_alloc exception.

Troublesome excerpt: apertium-bug-ca-es-bad-alloc.txt

I'm feeding apertium (with the cat-spa model) internet archive text to translate it line by line. The data itself is pretty noisy, which might give me garbage output, but that I'm fine with.

My "pipeline":

cat apertium-bug-ca-es-bad-alloc.txt \
    | apertium-destxt -i \
    | apertium -f none -u cat-spa \
    | apertium-retxt \
    > out.txt

Apertium is compiled from the source in this repository, at 267d7555af270261916d89980250cf3cd7df8f0c.

[unhammer]

Can you see what part of the pipeline causes it? Take the contents of the cat-spa.mode file (if you installed the package, it'll be in /usr/share/apertium/modes), remove the $2 in there, and chop off bits of the pipeline from the back until it no longer crashes, e.g.

cat apertium-bug-ca-es-bad-alloc.txt \
    | apertium-destxt -i \
    |  lt-proc … | apertium-transfer …  >/dev/null

[TinoDidriksen]

It's the very first lt-proc that crashes.

[TinoDidriksen]

The input line is, and I kid you not, 7401 bytes without spaces:

artisanmade-na.comurtisanmade-ne.comawrtisanmade-ne.comartisanmade-nel.comartisanjmade-ne.comartisanmade-nde.comartisanmide-ne.comartisanmade-nse.comartisanmadye-ne.comartiasnmade-ne.comartisanmade-re.comqartisanmade-ne.comarticanmade-ne.comartisajnmade-ne.comartiwsanmade-ne.comartisanmaded-ne.comartisanmadae-ne.comartisanmader-ne.comattisanmade-ne.comarttisanmade-ne.comartisanmadet-ne.comartisarmade-ne.comartisanmade-ne1.comartisanmade-nur.comartisammade-ne.comartisasnmade-ne.comaltisanmade-ne.comartisanmade-nec.comtartisanmade-ne.comartyisanmade-ne.comartiwanmade-ne.comartistanmade-ne.comartisanmaode-ne.comatrisanmade-ne.comartisanmqde-ne.comartyesanmade-ne.comartisanmoade-ne.comartisaunmade-ne.comadrtisanmade-ne.comartisanmkade-ne.comartisanjade-ne.comartisaynmade-ne.comartisanmade-mne.comartiasanmade-ne.comartisanmado-ne.comartisanmwade-ne.comarutisanmade-ne.comartisanmaide-ne.comartisahmade-ne.comaretisanmade-ne.comartisanmzde-ne.comartisanmade-n4e.comartfisanmade-ne.comartisanmazde-ne.comartisanmade-nre.coma4tisanmade-ne.comartisanmade-ne3.comartisanmadea-ne.comartisanmade-ne4.comasrtisanmade-ne.comartisaonmade-ne.comartisanmade-no.coma5rtisanmade-ne.comartisanmadfe-ne.comartisanmjade-ne.comartisanmede-ne.comartsanmade-ne.comaortisanmade-ne.comartisianmade-ne.comyartisanmade-ne.comartisanmadey-ne.comartijsanmade-ne.comartcisanmade-ne.comartisanmsde-ne.comartisurnmade-ne.comartisanmade-nea.comwwwartisanmade-ne.comartisanmaden-e.comartisanmzade-ne.comzrtisanmade-ne.comrartisanmade-ne.comartisynmade-ne.comjartisanmade-ne.comartisamnade-ne.comnartisanmade-ne.comartisanmad4-ne.comartisinmade-ne.comar4tisanmade-ne.comartisanmate-ne.comartiysanmade-ne.comlartisanmade-ne.comartisanade-ne.comalrtisanmade-ne.comartisanmade-nee.comartisanamde-ne.comartixanmade-ne.comartysanmade-ne.com1artisanmade-ne.comartisanmace-ne.comarytisanmade-ne.comarltisanmade-ne.comarteesanmade-ne.comarteisanmade-ne.comartisanmade-neo.comartisanmade-en.comartisanmad4e-ne.comadtisanmade-ne.comartisanmade-nek.comartisoanmade-ne.comartisannmade-ne.com6artisanmade-ne.comartsisanmade-ne.comartisajmade-ne.comartisanmade-nen.comarftisanmade-ne.comartisanmade-neg.comartisanmafe-ne.comartisanmade-ny.comartisanmadwe-ne.comartrisanmade-ne.comartjisanmade-ne.com9artisanmade-ne.comartisanmades-ne.comcartisanmade-ne.comartisanmadew-ne.comartisanmade-ne2.comartisanmade-nex.comartisanmady-ne.comarti8sanmade-ne.comartisanmaude-ne.comartiksanmade-ne.comarcisanmade-ne.comar5isanmade-ne.comortisanmade-ne.comartisuanmade-ne.comartisanmode-ne.comartisanhmade-ne.comartisarnmade-ne.comart9isanmade-ne.combartisanmade-ne.comartisaznmade-ne.comartisanmadce-ne.comartisanmad3e-ne.comartusanmade-ne.comartisanmade-nu.comartisanmade-ns.comairtisanmade-ne.comartisanbmade-ne.comartisanmade-net.comarti9sanmade-ne.comartisanmnade-ne.comartisanmayde-ne.comartisanmuade-ne.comartisaenmade-ne.comartisqanmade-ne.comartiosanmade-ne.comartisanmade-nr.comartasanmade-ne.comaartisanmade-ne.comartisanmade-hne.comartiusanmade-ne.comartiswanmade-ne.comartisanmade-nhe.comyrtisanmade-ne.comartisaqnmade-ne.comartisanmade-neb.comartisanmade-ne8.comartisenmade-ne.comartisanmade-nev.comarrtisanmade-ne.comartisanmiade-ne.compartisanmade-ne.comartidsanmade-ne.comdartisanmade-ne.comar5tisanmade-ne.comartisanmade-nef.comartisannade-ne.comartisanmahde-ne.comartisonmade-ne.comartisdanmade-ne.comartisahnmade-ne.comartjsanmade-ne.comartisanmade-neu.comartisanmade-ne9.comxartisanmade-ne.com7artisanmade-ne.comaertisanmade-ne.comartisznmade-ne.comafrtisanmade-ne.comartisanmacde-ne.comartisanmade4-ne.comazrtisanmade-ne.comartisanmade-nez.comartisanmare-ne.comaurtisanmade-ne.comartieanmade-ne.comartisanmadei-ne.comartisanmadre-ne.comartisanmade-ned.comarticsanmade-ne.com8artisanmade-ne.comartisabnmade-ne.comartdisanmade-ne.comvartisanmade-ne.comgartisanmade-ne.comartisanmadeh-ne.comarfisanmade-ne.comaritsanmade-ne.comartiscanmade-ne.comart6isanmade-ne.comartisanmade-noe.comartisamade-ne.comartisanmad-ne.comartisanmadu-ne.comartisanmase-ne.comiartisanmade-ne.comartisyanmade-ne.comahrtisanmade-ne.comartisqnmade-ne.comartisanmade-neh.comarhisanmade-ne.comartisanrmade-ne.comartissnmade-ne.comartisanmawde-ne.comartisanmadde-ne.comartisanmads-ne.comartiseanmade-ne.comartisanmade-rne.coma5tisanmade-ne.comart8isanmade-ne.comertisanmade-ne.comartissanmade-ne.comrtisanmade-ne.com0artisanmade-ne.comartsianmade-ne.comaryisanmade-ne.comartiesanmade-ne.comartisanmade3-ne.comartisanmade-nw.comartisanmdae-ne.comartisanmade-n.comartisanmadai-ne.commartisanmade-ne.comatrtisanmade-ne.comartisanmade-ni.comartisanmade-nep.comoartisanmade-ne.comartisanmadur-ne.comartisanmadd-ne.comaftisanmade-ne.comartisanmade-nem.comartisanmad-ene.comartisanmade-e.comartisanmade-ne7.comartisanmade-be.comwartisanmade-ne.comartisanmada-ne.comartisanmade-nwe.comartisanmad3-ne.comartisanmurde-ne.comartisanmadoe-ne.coma4rtisanmade-ne.comartisanmeade-ne.comartisanmae-ne.comartisanmade-ney.comartisanmade-n3.comartisanmade-je.comartuisanmade-ne.comartisanmade-ne0.comartizsanmade-ne.comartisanmaade-ne.comartisanmaed-ne.com4artisanmade-ne.comartisanmade-nbe.comartisanmade-neq.comartisanmadene.comarisanmade-ne.comartitanmade-ne.comartisanmade-nej.comardisanmade-ne.comartisabmade-ne.comartisanmatde-ne.comartisanmade-jne.comartisanmade-n3e.comartisanmade-nue.comartisanmade-nd.comartisanmasde-ne.comartisunmade-ne.comartisatnmade-ne.comartisanmqade-ne.comart9sanmade-ne.comartisanmade-nne.comartisanmadie-ne.comart8sanmade-ne.comartisainmade-ne.comartixsanmade-ne.comartisanmadw-ne.comartisanmaxde-ne.comartiaanmade-ne.comaetisanmade-ne.comartisnmade-ne.comartisanmadee-ne.comartgisanmade-ne.comartaisanmade-ne.comartoisanmade-ne.comarctisanmade-ne.comurrtisanmade-ne.comartisanmude-ne.comartisanmaqde-ne.com3artisanmade-ne.comartisanmde-ne.comartisanmadue-ne.comartosanmade-ne.comirtisanmade-ne.comaztisanmade-ne.comartisankmade-ne.comartizanmade-ne.comartisanmade-ne5.comaqrtisanmade-ne.comartisanmade-new.comartisankade-ne.com2artisanmade-ne.comartisanmyade-ne.comarthisanmade-ne.comartisanmafde-ne.comar6isanmade-ne.com5artisanmade-ne.comartisanmade-n4.comartisnamade-ne.comartisawnmade-ne.comartisanmaee-ne.comhartisanmade-ne.comartiisanmade-ne.comartisanmade-nai.comartisanmade-nye.comartisanmadeu-ne.comartisxanmade-ne.comartisanmade-me.comfartisanmade-ne.comartisanmade--ne.comartkisanmade-ne.comartitsanmade-ne.comartisanmade-ner.comartisamnmade-ne.comartiswnmade-ne.comarztisanmade-ne.comartiszanmade-ne.comarhtisanmade-ne.comsrtisanmade-ne.comwrtisanmade-ne.comardtisanmade-ne.comart5isanmade-ne.comeartisanmade-ne.comartesanmade-ne.comartisanmade-nie.comartisanmmade-ne.comartisanmsade-ne.comargtisanmade-ne.comaratisanmade-ne.comartisanmade-nme.comartisanmade-ne.comartisanmadr-ne.comartisanmadse-ne.comartisanmaxe-ne.comarsisanmade-ne.comartisanmade-nje.comartisaanmade-ne.comatisanmade-ne.comartisanmarde-ne.comartianmade-ne.comuartisanmade-ne.comartisanmyde-ne.comartisanmade-bne.comartisanmadxe-ne.comsartisanmade-ne.comqrtisanmade-ne.comkartisanmade-ne.comar6tisanmade-ne.comartisanmade-nei.comartisanmadeo-ne.comwwartisanmade-ne.comartisanmadte-ne.comartisanmade-he.comarstisanmade-ne.comartisanmaede-ne.comartisanmade-ne6.comartisanmade-nae.comayrtisanmade-ne.comzartisanmade-ne.comartksanmade-ne.comartisanmwde-ne.comargisanmade-ne.comartidanmade-ne.comartisanmadi-ne.comarrisanmade-ne.comartisanmade-nes.comratisanmade-ne.com

[TinoDidriksen]

If I clip the input to 6001 bytes, it works. 6002 bytes segfaults. Also, moving this to lttoolbox.

[jelmervdl]

Indeed. This is the backtrace I get when I hook gdb to that first lt-proc:

Program received signal SIGSEGV, Segmentation fault.
0x00002aaaaad10234 in FSTProcessor::readAnalysis(_IO_FILE*) () at fst_processor.cc:246
246	    return input_buffer.next();
Missing separate debuginfos, use: debuginfo-install glibc-2.17-307.el7.1.x86_64 libxml2-2.9.1-6.el7.4.x86_64 xz-libs-5.2.2-1.el7.x86_64
(gdb) bt
#0  0x00002aaaaad10234 in FSTProcessor::readAnalysis(_IO_FILE*) () at fst_processor.cc:246
#1  0x00002aaaaad106b7 in FSTProcessor::analysis(_IO_FILE*, _IO_FILE*) ()
    at fst_processor.cc:1357
#2  0x0000000000402a0d in main () at lt_proc.cc:400
#3  0x00002aaaabb71555 in __libc_start_main () from /lib64/libc.so.6
#4  0x0000000000402dbf in _start () at lt_proc.cc:417
(gdb)

[jelmervdl]

The 6000 doesn't seem to be a magic threshold. I found another one of these "sentences", except that the difference between okay and error is cut -b1-4095 (4094 works).

Attached: trouble.txt

[TinoDidriksen]

It'll vary depending on runs. The Buffer class which causes it can by default hold 2048 characters, so any more than 2 KiB garbage input is out-of-bounds. But the OS won't notice this until it goes out of the allocated pages' bounds.

[ftyers]

Here is the relevant code: https://github.com/apertium/lttoolbox/blob/master/lttoolbox/buffer.h#L74