Unable to download experimental features using icu_datagen

[riajain0412]

I installed icu_datagen using command cargo install icu_datagen. After this I was trying to download experimental features using cargo install --features experimental,bin icu_datagen. But this command says failed to compile icu_datagen v1.1.2

Caused by:
none of the selected packages contains these features: experimental

[Manishearth]

I believe datagen no longer has the feature split and can just be installed with cargo install icu_datagen

[Manishearth]

(We're thinking of reintroducing the feature split but it won't change the defaults)

[riajain0412]

So, how will segmenter features get downloaded? If I run icu4x-datagen with all keys then segmenter data is not getting downloaded.

[Manishearth]

Okay, your main problem is that you're using icu_segmenter 0.7, which is from ICU4X 1.0 (and you're using newer datagen). You want to use icu_segmenter 0.8 with ICU4X 1.1, since your data is for ICU4X 1.1. This typically won't matter that much but segmenter is an experimental component and it can change. The good news is that we're planning for segmenter to stabilize in ICU4X 1.2.

For now make sure you are using ICU4X 1.1 (so all icu_ crates should be version 1.1 or newer, and if you're directly depending on icu_segmenter, depend on version 0.8).

Note that you may still see the ICU4X data error: Missing data for locale (key: segmenter/lstm@1). error due to vertical fallback. As mentioned before this does not mean the data is not being found, it's just a bug that the error is seen. The upcoming stable segmenter plans to fix this.

[riajain0412]

Okay, I upgraded to icu_segmenter 0.8.0 but still unable to run it for SEA languages. Build is successful but when I do cargo run then it says missing data key 'segmenter/word@1'

[Manishearth]

What locales does it say it is missing that key for?
What invocation did you use for datagen to generate the data?

Please provide as much information as is possible.

[riajain0412]

Okay, so below is the code which I'm trying to implement:

main.rs

use icu_segmenter::WordSegmenter; 

use icu_provider_blob::BlobDataProvider; 

fn main() { 

       let blob = std::fs::read("my_data_blob").expect("Failed to read file"); 

    let buffer_provider = BlobDataProvider::try_new_from_blob(blob.into_boxed_slice()).expect("Failed to initialize Data Provider."); 

      let segmenter = WordSegmenter::try_new_with_buffer_provider(&buffer_provider).expect("Data exists"); 

      let breakpoints: Vec<usize> = segmenter.segment_str("မင်္ဂလာပါကမ္ဘာလောက").collect(); 

       println!("hello"); 

    for i in &breakpoints { 

        println!("{}",i); 

    } 

 } 

Cargo.toml

[package]
name = "check"
version = "0.1.0"
edition = "2021"

[dependencies]
icu = {version= "1.1.0", features = ["serde", "experimental"]} 

icu_provider_blob = "1.1.0" 

icu_provider = "1.1.0" 

icu_segmenter = {version= "0.8.0", features = ["serde"]} 

serde = { version = "1.0.101", optional = true, features = ["derive"] } 

serde_json = "*" 

serde_macros = { version = "*", optional = true } 

icu_testdata = {version= "1.1.0", features = ["icu_segmenter", "buffer"]} 

unicode-segmentation = "1.0.0" 

postcard = "1.0.2" 

I firstly ran the icu-datagen command to create the data blob: icu4x-datagen --keys all --locales full --format blob --out my_data_blob

Then I'm doing cargo build and cargo run. Cargo run is giving the below error:
thread 'main' panicked at 'Data exists: Data(DataError { kind: MissingDataKey, key: Some(DataKey{segmenter/word@1}), str_context: None })', check/src/main.rs:11:85 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

[sffc]

icu4x-datagen version 1.1 changed the invocation to --keys experimental-all if you want to include experimental keys. You can also list them out explicitly: --keys "segmenter/word@1"

[riajain0412]

One more thing, what is the location of ICU4X C++ API's source files?

[Manishearth]

Please read through https://github.com/unicode-org/icu4x/blob/main/docs/tutorials/cpp.md