Should the hexadecimal representation for control chars be escaped ?

[bnchi]

In the current Rust implementation when I try to call to_mdast on a markdown like this :

![](<\x0C>)

It generates the correct tree but the url value is being escaped like that "\\x0C"

while in the JS implementation when I provide it with the same markdown the url value isn't being escaped and instead it generates "\f" which is the valid control char.

For my use case it's quite tricky when trying to match for a control char in the string with Rust, while in JS it's relatively simple just a matter of checking whether the character is in a specific range.

Maybe there's a decision here that I don't quite get ?

[wooorm]

Hey hey!

It might be that you are writing tests for both projects?
In a JavaScript file, strings are different from strings in a Rust file.
Perhaps you are running into that?

Can you show the code for these things?

[bnchi]

For one of the test cases in rust it's like this :

assert_eq!(
    to(&Node::Image(Image {
        position: None,
        alt: String::new(),
        url: String::from("\\f"),
        title: None
    }))
    .unwrap(),
    "![](<\\f>)\n",
);

This is the same JavaScript test here: https://github.com/syntax-tree/mdast-util-to-markdown/blob/main/test/index.js#L1865

In Rust \f for form feed is not valid char however this hexadecimal representation is valid \x0C

For the test case above it's quite trivial to change the escaped \\f into \x0C and simply check for the presence of the control char.

So in rust a string like this is completely valid : "test\x0C"

Though I tried again to read a file as a valid utf-8 string with JS and the resulting string is being correctly escaped and generating "![](\\f)\n" (previously I was hard coding things in a string literal in JS which is not right I assume)

If I put this correctly the test above is actually unit testing a "hand written" AST object where a user could provide a control char and not an AST generated using the to_mdast, right ?

[wooorm]

The \f in the JavaScript string you show is an escape. Just like \n. Or \t. It is a form feed character: '\f'.charCodeAt(0).toString(16) is c.

If you type f or \\f in Rust, it’s something else. It could definitely be that \f isn‘t allowed to be typed literally in Rust, I don’t know from the top of my head.

So it’s not the same as the JavaScript test case.

I believe your test case in Rust should be:

assert_eq!(
    to(&Node::Image(Image {
        position: None,
        alt: String::new(),
        url: String::from("\x0C"),
        title: None
    }))
    .unwrap(),
    "![](<\x0C>)\n",
);

If I put this correctly the test above is actually unit testing a "hand written" AST object where a user could provide a control char and not an AST generated using the to_mdast, right ?

The test case there in JavaScript is checking if control characters (C0), which are not simple whitespace with meaning in markdown (spaces, tabs, line endings), whether they work. Somewhat common of these characters on the web platform \f.

Indeed, for manual ASTs.
Though, this can be generated in human authored markdown with:

![](<&#xc;>)

![](<&#12;>)

Which generates:

<p><img src="%0C" alt="" /></p>
<p><img src="%0C" alt="" /></p>

https://spec.commonmark.org/dingus/?text=!%5B%5D(%3C%26%23xc%3B%3E)%0A%0A!%5B%5D(%3C%26%2312%3B%3E)

[bnchi]

Yes thank you so much for your quick response, this makes a lot of sense!

It's a trivial thing, but been scratching my head for a while on it :)

[wooorm]

No worries! Feel free to ping me when you get stuck. Really appreciate you digging in!