RFC: 1.0 release?

[BurntSushi]

For those coming here that don't know what bstr is: it is a string library for &[u8]. The essential difference between the strings in bstr and the &str type in std is that bstr treats &[u8] as conventionally UTF-8 instead of requiring that it be UTF-8. Its main utility is in contexts where you believe your data is UTF-8 (but it might not be completely UTF-8) and you either don't have any information about what its actual encoding is or do not want to pay for the UTF-8 validity check. A common example of this is reading data from files. The bstr documentation says a lot more.

This issue is about releasing 1.0. Since I do not currently have any plans for a 2.0, I would like to get as many eyes on this as possible. If you have any feedback with respect to API breaking changes, I would love to hear about it.

OK, so I promise that the 1.0 release is imminent. Here is an exhaustive list of planned breaking API changes, all of which are currently present on master (some brought in via #104, others via #123):

• Bytes::as_slice is renamed to Bytes::as_bytes.
• ByteVec::into_os_string now returns Result<OsString, FromUtf8Error> instead of Result<OsString, Vec<u8>>.
• ByteVec::into_path_buf now returns Result<PathBuf, FromUtf8Error> instead of Result<PathBuf, Vec<u8>>.
• Find<'a> has been changed to Find<'h, 'n>, which represents the lifetimes of both the haystack and the needle, instead of the shorter of the two.
• FindReverse<'a> has been changed to FindReverse<'h, 'n>, which represents the lifetimes of both the haystack and the needle, instead of the shorter of the two.
• Split<'a> has been changed to Split<h, 's>, which represents the lifetimes of both the haystack and the splitter, instead of the shorter of the two.
• SplitReverse<'a> has been changed to SplitReverse<'h, 's>, which represents the lifetimes of both the haystack and the splitter, instead of the shorter of the two.
• SplitN<'a> has been changed to SplitN<h, 's>, which represents the lifetimes of both the haystack and the splitter, instead of the shorter of the two.
• SplitNReverse<'a> has been changed to SplitNReverse<h, 's>, which represents the lifetimes of both the haystack and the splitter, instead of the shorter of the two.
• ByteSlice::fields is now gated behind the unicode feature. Previously, it was available unconditionally.
• serde1 has been renamed to serde1-std, and serde1-nostd has been split into serde1-alloc and serde1-core.
• BufReadExt::for_byte_line now accepts &mut self instead of self.
• BufReadExt::for_byte_record now accepts &mut self instead of self.
• BufReadExt::for_byte_line_with_terminator now accepts &mut self instead of self.
• BufReadExt::for_byte_record_with_terminator now accepts &mut self instead of self.
• The OsStr and Path conversion routines had their API docs tweaked slightly so that they could defer to a possible OsStr::as_bytes (and OsStr::from_bytes) routine in the future, if it's added. But their behavior otherwise currently remains the same.
• The serde1-* features have been dropped. bstr now just has a serde feature and uses the new dep: and pkg? syntax so that it will combine as one would expect with other features.
• ByteSlice::copy_within_str has been removed, since slice::copy_within has been stable since Rust 1.37. slice::copy_within does the exact same thing.

Assuming this is an exhaustive list, and given that these are all very minor changes, I'm hopefully that migration from 0.2 to 1.0 will be very easy. Hopefully requiring no changes in most cases.

My plan is to release 1.0 on July 11, 2022 July 18, 2022 September 6, 2022. If you have feedback to give, please do so now. :-)

Note that I've published 1.0.0-pre.3 to crates.io. Docs: https://docs.rs/bstr/1.0.0-pre.3

(Below is the message I initial wrote a couple years ago. I'm way late to the party.)

---

It has been almost a year since I released bstr 0.2 with the major breaking change of moving most of the routines to extension traits. It seems like this has been a success. Namely, bstr's reverse dependency list is growing. More generally, I personally like working with the new API better than the old one. While I still bemoan the loss of a distinct type and its corresponding Debug impl as the One True Byte String, I think the benefits of the extension trait API have ended up outweighing that cost.

I've been giving some thought to bstr's API and its future evolution, and nothing immediately comes to mind in terms of breaking changes. That is, most everything I can think of are API additions, or at worst, deprecations. The only breaking change I can think of is to more carefully audit which API routines are available when unicode mode is disabled. I just want to make sure I'm not boxing myself into any corners there. (e.g., Some extant implementations might currently rely on std for its Unicode support, but it may wind up being the case that we want to re-implement some of those, which will require bringing in our own Unicode data. If that occurs, then those APIs should be gated behind the unicode feature.)

Otherwise, my feeling is that, unless I hear otherwise, I will make a 1.0 release in a few months. June 2020 is the 1 year anniversay of the 0.2 release, so that sounds like a good a time as any.

Thoughts?

cc @thomcc I know you've done some work on bstr and are actually using it, so would definitely appreciate if you have any thoughts here! Mostly what I'm looking for are things that we might want to do that will break the current API. While 1.0 doesn't necessarily mean "breaking changes must stop," I generally try to commit to a long termish period of stability for each major version in core libraries.

[thomcc]

My main question would be: do the BStr and BString types pull their own weight? They feel a bit redundant to me -- I essentially only ever use the former and even then, only as an argument in a fmt call, and that could just be a one-off wrapper returned by a method on the extension trait.

That said, part of the reason I don't use them much has been due to the fact that the library is pre-1.0 and these seem the most likely item to be removed to me.

In principal I like the idea of being able to specify something is a BString rather than a Vec<u8> -- in compound types like Vec<Vec<u8>> it can be a bit non-obvious what this represents, whereas Vec<BString> is much more clear (to me, anyway).

I think the main other changes I can think of would be API additions, and thus compatible.

[BurntSushi]

do the BStr and BString types pull their own weight? They feel a bit redundant to me -- I essentially only ever use the former and even then, only as an argument in a fmt call, and that could just be a one-off wrapper returned by a method on the extension trait.

I think so. There are two main use cases I envision for them:

1. They provide a target for trait impls that is distinct from &[u8] and Vec<u8>. This is significant because the latter may be covered by blanket impls. (This is why serde_bytes exists. One can use bstr instead of serde_bytes since bstr has optional serde support.
2. You can use BString and &BStr types internally instead of Vec<u8> and &[u8] to automatically benefit from their std::fmt::Debug impls. That way, you don't have to write custom debug impls.

[thomcc]

Yep, that all makes sense to me, and I hadn't considered point 1 at all.

[lopopolo]

Another place that BString and &BStr are useful is to wrap bytes before using them in assert_eq! in tests to get more readable debug output.

[Byron]

Now that gitoxide is thinking about stability as well it became clear that without a 1.0 bstr release this couldn't ever happen. I am glad to see this is actively explored here and will be looking forward to the eventual 1.0 release.

For reference, not using &BStr in gitoxide would be a great degradation of usability of parsed git objects which can't be forced into UTF-8 encoding. Turning these back into &[u8] would be something so regrettable that it simply has to be avoided.

[BurntSushi]

For reference, not using &BStr in gitoxide would be a great degradation of usability of parsed git objects which can't be forced into UTF-8 encoding.

Could you say more about this? In particular, I would like to hear more about its importance in public APIs.

[Byron]

A good example for its use is a freshly parsed commit whose memory is backed by a buffer somewhere else. The goal is to make clear that these fields represent user-readable strings, but in Rust there is no way to do this without enforcing an encoding or a custom type. That custom type is BStr and once I found it there was no way I would roll my own.

It also works nicely for the mutable version of such a commit.

When testing, working with BString/BStr felt natural and worked exactly as if one would expect.

Note that I put the bstr crate into public APIs knowing it goes against the prominent advice provided in the readme file, that's how indispensable it was 😅.

From there it only proliferated and it's now used in no less than 13 gitoxide crates.

Could you say more about this? In particular, I would like to hear more about its importance in public APIs.

To find a more concise answer: bstr is just too practical and there is a huge void for strings without known encoding in Rust that it seems to fill well for my use.

[BurntSushi]

@Byron Ah wow, that is great feedback. I agree that using BStr/BString in those APIs is probably the right thing, because otherwise the derived Debug representation of those types is going to be totally useless I imagine.

I don't think there is much work to be done to get bstr to 1.0. I'll do my best to prioritize it in the next month or so.

When do you plan to release 1.0 versions of crates that depend on bstr?

[Byron]

Thanks so much! Part of me was afraid I'd be told it's all wrong as it was so strongly discouraged 😅. Also thanks for the prioritization, even though I think no matter how long it takes you, I will be slower. If it happens at the end of the year I would be served well already.

[BurntSushi]

Ah perfect. My time is super limited and devoting pretty much all of what little free time I have to regex. But I'll do my best to take a detour and whip bstr into a 1.0 release. I think it has had more than enough time to bake and there don't appear to be any obvious design flaws when compared to alternatives.

[chrysn]

Having this stable would also enable its use with the riot-wrappers crate, where for example process names are exposed (they're by all probability ASCII even, but an unsafe assume-it-is opens up for UB if a C user on the same system does something weird, and converting them to &str means pulling in UTF-8 checks, and either fallible operations or panics).

No hurry from me (I'll need to wait with this for my next API bump anyway), just a data point, and thanks for working on this!

[Byron]

Since there hasn't been a lot of movement in the past 9 months and since bstr is a very, very important dependency for gitoxide and probably countless other crates, I wonder what I can do to help. The reason for me getting a bit more pushy is the ongoing work to get gitoxide into cargo.

I am hereby happily offering to step in as collaborator or maintainer and serve as helping hand for whatever @BurntSushi deems right.

Thank a lot :)!

[BurntSushi]

Folks, I've updated the top comment of this issue to include a list of breaking changes (all of which are minor). My plan is to push out the 1.0 release on Monday, July 11, 2022. So please, if you have comments or feedback or other ideas for breaking changes, now is the time to do it.

I am hereby happily offering to step in as collaborator or maintainer and serve as helping hand for whatever @BurntSushi deems right.

For what it's worth, it would have probably been an order of magnitude more work to onboard someone else to do the 1.0 release than to just do it myself. (Which ended up taking almost a full day of work.) It's important to remember this when offering to maintain a project. It's not as simple as me just clicking a button and giving someone permissions. :-)

[Byron]

For what it's worth, it would have probably been an order of magnitude more work to onboard someone else to do the 1.0 release than to just do it myself.

That's certainly true, yet I find it worth clarifying that my offer wasn't limited to the 1.0 release. In any case, thanks for this wonderful crate and the continued maintenance efforts!

[joshtriplett]

I sincerely hope that the version of bstr after 1.0 is simply std (or, more precisely, core and alloc as appropriate).

Thank you for such a fundamental building block of the ecosystem.

[BurntSushi]

Note that I've published 1.0.0-pre.1 to crates.io. It's not on docs.rs yet though.

[BurntSushi]

dddd0 on reddit asked:

Might as well ask here: Why does for_byte_record_with_terminator in BufReadExt consume self / the underlying BufRead? It doesn't seem to be necessary to me.

Indeed, this is I believe a good point. In fact, all of the methods on BufReadExt consume self. I believe the only ones we actually want to consume self are byte_lines and byte_records, since they each return an iterator generic over Self. But the other methods all execute closures, and so I suspect &mut self would be more appropriate there. (It's likely this isn't a show-stopping issue in practice since you can probably always convert your T: BufReadExt to a &mut T: BufReadExt and get away with calling the for_ methods without actually consuming your reader, but it does look like a wart.)

[BurntSushi]

@omac777 Let's please try to keep this thread focused on 1.0 concerns. This isn't an RFC for any arbitrary feature requests, but issues specific to a 1.0 release. I've moved your comment into #109 and responded there.

[BurntSushi]

OK, the docs for what's currently on master have been published: https://docs.rs/bstr/1.0.0-pre.1/bstr/

(I haven't made the changes to stop consuming self for some of the methods on BufReadExt yet.)

[BurntSushi]

I found myself creating little helper methods for that throughout the codebase.

Can you create a new issue with these? That is, just copy and paste your helper routines. (Or if they fit better in an existing issue, that's fine too.)

[BurntSushi]

I've added another breaking change: restructure serde feature flags

Now that bstr has an 'alloc' feature, we need to rethink how we setup
the serde feature flags. Previously, all we had was 'std' and 'no std'.
But now we have 'std', 'alloc only' and 'core only'. In particular, 'no
std' is split into 'alloc only' and 'core only', since neither one bring
in std. To reflect this trichotomy, we rename 'serde1' to 'serde1-std',
and split 'serde1-nostd' into 'serde1-alloc' and 'serde1-core'.

[BurntSushi]

I've added another breaking change: several methods on BufReadExt now take &mut self instead of self.

[epage]

Now that bstr has an 'alloc' feature, we need to rethink how we setup
the serde feature flags. Previously, all we had was 'std' and 'no std'.
But now we have 'std', 'alloc only' and 'core only'. In particular, 'no
std' is split into 'alloc only' and 'core only', since neither one bring
in std. To reflect this trichotomy, we rename 'serde1' to 'serde1-std',
and split 'serde1-nostd' into 'serde1-alloc' and 'serde1-core'.

Would it be worth bumping the MSRV to 1.60 and use weak / namespaced feature flags?

This way you have

• serde1 could possibly be named serde
• std depends on serde?/std
• alloc depends on serde?/alloc
• Optional dependencies that are not meant to be features can be properly hidden with dep:

[BurntSushi]

@epage Yeah I mused about this here: #111 (comment)

Basically, I kind of feel like 1.60 is really way too new. My first approximation is to track Debian stable (which is at Rust 1.48), but in principle, I'm okay with something newer.

Unfortunately, there does seem to be a fundamental conflict here. Namely, if we move forward with the existing trichotomy of serde features, then it's going to have to stay that way since I think moving to the scheme you propose would be a breaking change. (Not the MSRV bump, that's fine, but changing the features around.) Since I don't see myself putting out a bstr 2.0 any time soon, it would effectively freeze bstr into a slightly sub-optimal feature configuration. But if I do adopt the nicer feature setup, then I'm being somewhat aggressive with the MSRV.

I personally don't mind being a bit aggressive, but I suppose it depends on my users.

@lopopolo @thomcc @Byron @TethysSvensson How do you feel about Rust 1.60 as the MSRV?

[BurntSushi]

With respect to MSRV, if Rust 1.60 is too new for some folks, then I suppose they could stick with bstr 0.2 until Rust 1.60 is no longer too new.

[BurntSushi]

Given there's a question open here, I'm going to delay the 1.0 release another week.

[epage]

Speaking of

ByteVec::into_os_string now returns Result<OsString, FromUtf8Error> instead of Result<OsString, Vec>.

In the docs, it says

1. One could re-implement WTF-8 and re-encode file paths on Windows to WTF-8 by accessing their underlying 16-bit integer representation. Unfortunately, this isn’t zero cost (it introduces a second WTF-8 decoding step) and it’s not clear this is a good thing to do, since WTF-8 should ideally remain an internal implementation detail.
   ...

While this library may provide facilities for (1) in the future, currently, this library only provides facilities for (2) and (3).

(1) could be simplified, depending on what is done with OsStr, like rust-lang/rust#95290

Should the existing functions have more specific names to open the door for the more general functions to be added later, if possible?

Non-blocker for 1.0: would it also be worth pointing people to os_str_bytes to help people who want (1)?

EDIT: Now split out as #115, #116

[epage]

As a usability note, I wrote an application with BStr in the internal APis and kept getting tripped up that I needed as_bstr() or .map(ByteSlice::as_bstr) at the end of any transformation I did on a BStr to get it back into its own type. In the ideal world, the output type would match the input type.

I'm assuming this is fully intentional and won't be changing but figured it was still worth noting.

EDIT: Split out into #117

[lopopolo]

@BurntSushi 1.60.0 MSRV works for me. Applications I build with bstr target the latest stable and library crates I've built with bstr have a "bump MSRV in minor versions" policy which I'd be happy to exercise to accommodate a higher bstr MSRV.

On a related note, I've been using the weak features and namespaced features in recent cargo and have found them to be quite pleasant to work with. Using serde1 = ["dep:serde", ...] means that a --features serde flag is no longer valid like it is now.

[BurntSushi]

As a usability note, I wrote an application with BStr in the internal APis and kept getting tripped up that I needed as_bstr() or .map(ByteSlice::as_bstr) at the end of any transformation I did on a BStr to get it back into its own type. In the ideal world, the output type would match the input type.

I'm assuming this is fully intentional and won't be changing but figured it was still worth noting.

Right yeah. That's what bstr 0.1 was: https://docs.rs/bstr/0.1.4/bstr/

[TethysSvensson]

How do you feel about Rust 1.60 as the MSRV?

I think it's probably the right choice in this situation, given that you are trying to stabilize.

In planus, our base policy is to only bump the MSRV to versions that are at least six months old, which would mean that we could not start using bstr until October. That being said, it's a policy we try to follow, but break when there is sufficient reason to do so.

(While writing this, I am also realizing that our MSRV simultaneously says 4 versions of rust and 6 months, which are two very different things)

[BurntSushi]

@lopopolo Yeah the new Cargo feature stuff is awesome. I very much like that it gives more control over what's a public feature.

@epage See #5 and #8 for more details on why bstr 0.1 -> 0.2 went from concrete BString/BStr types to extension traits on Vec<u8>/[u8]. It is slightly sad that if you have a BStr and call an extension trait method that returns a subslice of it that you get a &[u8] back instead. I'm not sure there is any simple machinery that would let me fix that while maintaining the existing extension trait API on [u8].

Happy to answer more questions on that but would prefer opening a new issue for it if you have more questions.

[Byron]

How do you feel about Rust 1.60 as the MSRV?

I'd be OK with it, thanks for asking.

[lopopolo]

Hi @BurntSushi I know this is late in the game for 1.0, but I wanted to float an idea of an optional dependency on simdutf8 for a SIMD-accelerated fast path to crate::utf8::validate:

bstr/src/utf8.rs

Lines 463 to 471 in 0d9d222

	/// Returns OK if and only if the given slice is completely valid UTF-8.
	///
	/// If the slice isn't valid UTF-8, then an error is returned that explains
	/// the first location at which invalid UTF-8 was detected.
	pub fn validate(slice: &[u8]) -> Result<(), Utf8Error> {
	// The fast path for validating UTF-8. It steps through a UTF-8 automaton
	// and uses a SIMD accelerated ASCII fast path on x86_64. If an error is
	// detected, it backs up and runs the slower version of the UTF-8 automaton
	// to determine correct error information.

I'm not sure how you'd want to structure this, but maybe an optional performance feature a la regex (that maybe does nothing for 1.0 but leaves the possibility for additional deps open in subsequent releases?).

Edit: coming back to this, I'm actually not sure whether I should expect the DFA in bstr or SIMD routines in simdutf8 to be faster. I just associate SIMD with 🏎️

[BurntSushi]

Yeah that's not a 1.0 concern. I'm unlikely to take a dependency for that. Instead, I would rather see the implementation ported to bstr or maybe even memchr. That said, I've never looked at it in depth, so I'm unsure of its complexity.

Feel free to open a new issue about this to avoid cluttering up the 1.0 release thread.

For this thread, I would really like to keep it scoped to 1.0 concerns.

[Lokathor]

September update on Rust versions in the Linux world:

• https://packages.debian.org/bullseye/rustc
  • Stable is 1.48
  • Testing 1.59
• https://packages.fedoraproject.org/pkgs/rust/rust/
  • Fedora 35 is at 1.55
  • Fedora 37 is at 1.63

1.60 seems a completely reasonable MSRV to start with for 1.0. Linux distros will catch up to 1.60 "very soon", and the 0.2 version is still usable until then.

You could also document that the MSRV will stay up to date with Debian Stable once they eventually get past 1.60, or whatever other reference point.

[BurntSushi]

OK, bstr 1.0.0-pre.3 is up: https://docs.rs/bstr/1.0.0-pre.3/bstr/

I've rejiggered/simplified the features (just a serde feature and no more serde1-{std,alloc,core} goop) and bumped the MSRV to Rust 1.60.

My new 1.0 release date is September 6, 2022. If there are any last comments on backward incompatible changes, now is the time to make them. :)

[BurntSushi]

Done!

1.0 tag: https://github.com/BurntSushi/bstr/releases/tag/1.0.0

Blog post: https://blog.burntsushi.net/bstr/

[lopopolo]

Congrats @BurntSushi! This is huge. Thank you so much for all of the effort put into this high quality, foundational crate. bstr makes so much possible for me and my projects. 🙇