scripts not working with unicode input on mac

[roeeaharoni]

Hi!

I Tried to use the tool according to the readme file, on macosx, with hebrew as the source language and arabic as the target. When I executed the following command (after installing the dependencies):

./build-corpus.sh ar hewiki-20141102 > titles_he_ar.txt

I got the following output:

Target language code: ar
Using hewiki-20141102-langlinks.sql.gz
Using hewiki-20141102-page.sql.gz
Reading page data from hewiki-20141102-page.sql.gz...
iconv: conversion from utf8 unsupported
iconv: try 'iconv -l' to get the list of supported encodings
read 0 documents
Reading langlinks data from hewiki-20141102-langlinks.sql.gz...
iconv: conversion from utf8 unsupported
iconv: try 'iconv -l' to get the list of supported encodings
read 0 documents

I tried to fix this by changing the perl scripts that called iconv with parameter 'utf8' to call it with 'utf-8', and it seems to work fine now.

Best regards,
Roee

[randinterval]

Hi there, Can you upload the updated script?I seem to be having some issues.
saad@Arc:/Desktop/wiki$ ./build-corpus.sh en enwiki-20151102 > titles.txt
Target language code: en
Using enwiki-20151102-langlinks.sql.gz
Using enwiki-20151102-page.sql.gz
Reading page data from enwiki-20151102-page.sql.gz...
read 37804388 documents
Reading langlinks data from enwiki-20151102-langlinks.sql.gz...
read 0 documents
saad@Arc:/Desktop/wiki$ ./build-corpus.sh ur urwiki-20151123 > titles.txt
Target language code: ur
Using urwiki-20151123-langlinks.sql.gz
Using urwiki-20151123-page.sql.gz
Reading page data from urwiki-20151123-page.sql.gz...
read 401280 documents
Reading langlinks data from urwiki-20151123-langlinks.sql.gz...
read 0 documents

[randinterval]

@roeeaharoni @redpony @wammar

[wammar]

Hi Saad,

Try the following command:
./build-corpus.sh en urwiki-20151123

Waleed
On Dec 1, 2015 9:46 AM, "Saad Ahmed" [email protected] wrote:

@roeeaharoni https://github.com/roeeaharoni @redpony
https://github.com/redpony @wammar https://github.com/wammar

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

[randinterval]

Hi Waleed,
Thank you so much! 👍 @wammar

[randinterval]

Hey @wammar , Hope you're well - I also want to extract entire articles in English and Target Language (Urdu) versions. Could you please point me in the right direction?I haven't programming in Perl before, so just little bit confused.

[wammar]

Sorry Saad but I can hardly write perl myself.
On Dec 14, 2015 4:14 AM, "Saad Ahmed" [email protected] wrote:

Hey @wammar https://github.com/wammar , Hope you're well - I also want
to extract entire articles in English and Target Language (Urdu) versions.
Could you please point me in the right direction?I haven't programming Perl
before, so just little bit confused.

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

[randinterval]

Hey @roeeaharoni I also want to extract entire articles in English and Target Language (Urdu) versions. Could you please point me in the right direction?I haven't programming in Perl before, so just little bit confused.

[ghost]

if you're getting the following error:

iconv: conversion from utf8 unsupported
iconv: try 'iconv -l' to get the list of supported encodings

Go to scripts > extract.pl
change two instances of "utf8" to "utf-8" in Line 29 & Line 53.
It will work.

PS: This change worked on Ubuntu & Mac

[imrrahul]

@randinterval @wammar please I am getting below error ---

[twielfaert]

@imrrahul You're using both the wrong path and the wrong files (enwikivoyage).