How to use segmenter

[andrewbaxter]

I have a simple use case, I'm wrapping some text for a text file at 80 columns. I take an unwrapped line and need to find the last line break before the max line width. I'm 99% sure what I want is segment_str, but I can't figure out how to use it.

The documentation example shows

let segmenter =
     LineBreakSegmenter::try_new_unstable(&icu_testdata::unstable())
         .expect("Data exists");

let breakpoints: Vec<usize> =
     segmenter.segment_str("Hello World").collect();

1. icu_testdata doesn't look like end users should be using it, but I'm not sure what I need as a replacement

2. icu_testdata doesn't seem to work anyway, I get this error: the trait bound `UnstableDataProvider: icu_provider::data_provider::DataProvider<LineBreakDataV1Marker>` is not satisfied

3. I'm not sure why the segmenter would need a data provider when the string to split is passed in to segment_str. I thought maybe testdata takes takes the key and returns it as data, and Hello World here is a key, but the documentation for segment_str clearly states that the function argument is the string to split, not a key.

4. Additionally, it states

   It can be enabled with the "experimental" feature of the icu meta-crate.

   Using icu_segmenter it seems like the lstm feature needs to be enabled. Is there another feature I'm missing somewhere?

Sorry for the dump of topics, I've read the data provider, tutorials, etc. but I'm not getting the full picture here so I'm not sure what's relevant and what's not.

[sffc]

Hi @andrewbaxter!

These are all super valid questions! ICU4X Segmenter is still experimental and currently planned for a full stable release in 1.2 (spring 2023), at which point we'll make sure that the developer experience is where it should be. In the mean time:

1. The Tutorial (Part 1, Part 2) discusses where to get and how to generate the data for ICU4X. It's totally fine to use icu_testdata to get bootstrapped. In general, the recommendation is that you use icu4x-datagen to build your data file. For certain components like Segmenter, I'd like to see us start shipping data directly on crates.io; Ship data for certain components directly on crates.io #2888 is a follow-up.
2. You need to enable the icu_segmenter feature on icu_testdata. You won't need to do this after it becomes a stable component.
3. The data provider is passed to the segmenter constructor, not segment_str. Can you point to where in the docs you found something that says otherwise?
4. The lstm feature is optional; if you enable it, you get LSTM segmentation for South-East Asian languages. If you leave it disabled, you get dictionary segmentation for those languages. This may change in 1.2: Document and potentially fix LSTM feature in icu_segmenter #2829

Hope this helps!

[andrewbaxter]

Thanks for the reply! I can probably get by with just splitting by spaces for now, in case I'm just getting in the way.

Reading your answer and especially this I think I had confusion over what "Data" was. I thought that was "user provided data" i.e. strings, and none of the discussions about keys, rust data structures, etc. made sense (maybe they were string tables, or templates, or something). It's clear now that it's "locale data". With your answer to 2 I'm probably good (for now), thanks again!

FWIW I missed that data_management.md doc, but AFAICT no documentation pages explicitly define what Data or a DataProvider is.

I got to https://icu4x.unicode.org/doc/icu_segmenter/index.html via google, and from there followed links to DataProvider, how to choose a constructor, etc. and couldn't figure out what those are. None of them seem to say what a DataProvider is or what Data refers to.

After that, I did come to the main readme, and read
https://github.com/unicode-org/icu4x/blob/main/docs/tutorials/intro.md - the "Locale" section doesn't mention Data/DataProviders, and the "Basic Data Management" section doesn't say what a DataProvider is either.

https://github.com/unicode-org/icu4x/blob/main/docs/tutorials/index.md links back to the intro, and I think at that point I assumed there was no more information and bounced back to the top readme.

From the top readme I went to https://github.com/unicode-org/icu4x/blob/main/docs/README.md and clicked on https://github.com/unicode-org/icu4x/blob/main/docs/tutorials/data_provider.md but that also somehow avoids mentioning the concrete definition too:

DataProvider is a general mechanism for loading data required for ICU4X components to operate from a source.

i.e. it avoids mentioning locales again.

Eventually https://github.com/unicode-org/icu4x/blob/main/docs/tutorials/data_management.md is what helped me:

Unless you're happy shipping your app with the ~10 locales supported by icu_testdata (and with the size impact of all keys), you will need generate your own data that is customized to suit your needs.

However it only implies that data is locale data, but it requires reading between the lines.

I assume this is one of those "everyone is so familiar with it it's a blind spot" type of things. An explicit definition (in the api docs?) that says something like:

DataProvider is the interface for providing locale data to this library.

would be amazing.

Anyway, I'm a documentation nerd so I just thought I'd leave some details. Thanks again!

[zbraniecki]

@andrewbaxter I know this is asking for a lot, but since you self-described yourself as a documentation nerd- would you be willing to help with the rewrite?

One of the inherent challenges of learning new concepts is that once you learned them, it's challenging to understand the hurdles. So the best time to shape the tutorials/onboarding is when you just learned it, which is where you are right now.

It makes you the best expert we have right now in helping the next person onboarding from experiencing the same obstacles you faced.

[andrewbaxter]

Thanks, I'd be glad to add an explicit definition somewhere. Is the main documentation markdown files in the repo, or the api docs? The DataProvider api docs seem like they'd be the most central location for discussion of the concept at the moment, so I could put it there.

I know next to nothing about DataProvider and how it relates to other concepts though, so I can't do much more than the above.

I think it would help greatly to gather all the information about DataProvider and Data in the various tutorials, markdown pages, etc and put them all in one document about the concept, and just link them from other places rather than repeat the information.

[sffc]

The primary landing page for the DataProvider docs is the icu_provider crate. I think it's a sensible place to improve clarity about what it does and what it is for and why you need it.

https://icu4x.unicode.org/doc/icu_provider/index.html

[sffc]

The actual file to edit is: https://github.com/unicode-org/icu4x/blob/main/provider/core/src/lib.rs