Parser fixes

[jaytaph]

Currently trying to get tests10.dat to pass. This tests foreign elements (svg, mathml), which the current parser cannot cope with without an extra layer. I've got most of it working, but the majority of the large state-machine is moved to a separate function. Codewise not much is changed, but might result in a large diff nevertheless..

[CharlesChen0823]

I currently has finished, all tests2.dat, tests8.dat, tests22.data, tests24.dat, tests25.dat, tests15.dat. tests18.dat remains 3 cases not passed. And some tests data is associate fragment case and the order of attributes display. I think these tests is easy to pass.

[jaytaph]

I have done some other fixes too, so lets try and merge things together to see how far we are.

[jaytaph]

@CharlesChen0823 I've added a html5_parser_test binary that will give an overview of the current parser tests and their status:

Test: ( 18) adoption01.dat [................XX]
Test: (  2) adoption02.dat [..]
Test: ( 48) blocks.dat [................................................]
Test: ( 16) comments01.dat [................]
Test: ( 37) doctype01.dat [.............XXX......XXXXX.XX.XXXXXX]
Test: ( 49) domjs-unsafe.dat [XXX...............XXX............................]
Test: ( 75) entities01.dat [...........................................................................]
Test: ( 26) entities02.dat [..........................]
Test: ( 66) foreign-fragment.dat [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]
Test: ( 24) html5test-com.dat [........................]
Test: (  4) inbody01.dat [....]
Test: (  4) isindex.dat [....]    
Test: (  3) main-element.dat [...]
Test: (  8) math.dat [XXXXXXXX]
Test: ( 20) menuitem-element.dat [....................]
Test: (  1) namespace-sensitivity.dat [X]
Test: ( 18) noscript01.dat [..................]
Test: (  1) pending-spec-changes-plain-text-unsafe.dat [.]
Test: (  3) pending-spec-changes.dat [...]
Test: ( 33) plain-text-unsafe.dat [..........X......................]
Test: (  4) quirks01.dat [XXXX]
Test: ( 21) ruby.dat [.....................]
Test: ( 26) scriptdata01.dat [...............XXX.XXXX.X.]
Test: (  1) scripted\adoption01.dat [X]                  
Test: (  1) scripted\ark.dat [X]                         
Test: (  2) scripted\webkit01.dat [XX]                   
Test: (  3) search-element.dat [...]
Test: (  8) svg.dat [XXXXXXXX]
Test: ( 19) tables01.dat [...................]
Test: (112) template.dat [XXXXXXX...X........XXX...XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX]
Test: (112) tests1.dat [................................................................................................................]
Test: ( 54) tests10.dat [..............................XX..XX...X..............]
Test: ( 13) tests11.dat [.............]
Test: (  2) tests12.dat [X.]
Test: (  7) tests14.dat [.......]
Test: ( 14) tests15.dat [..............]
Test: (197) tests16.dat [.....................................................................................................................................................................................................]
Test: ( 13) tests17.dat [.............]
Test: ( 36) tests18.dat [...............X....................]
Test: (103) tests19.dat [..............................XXXXX.............................................X.X....................]
Test: ( 63) tests2.dat [...............................................................]
Test: ( 64) tests20.dat [................................................................]
Test: ( 23) tests21.dat [XX.XXXXXXXXXX.XXXXXXXXX]
Test: (  5) tests22.dat [.....]
Test: (  5) tests23.dat [XXX..]
Test: (  8) tests24.dat [........]
Test: ( 26) tests25.dat [..........................]
Test: ( 20) tests26.dat [....................]
Test: ( 24) tests3.dat [........................]
Test: (  9) tests4.dat [XXXXXXXXX]
Test: ( 17) tests5.dat [.................]
Test: ( 52) tests6.dat [......X..........X..X...X.X..X.X.XX.X.X....XX.X.....]
Test: ( 34) tests7.dat [...........................X......]
Test: ( 10) tests8.dat [..........]
Test: ( 27) tests9.dat [...........................]
Test: ( 81) tests_innerHTML_1.dat [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]
Test: (  9) tricky01.dat [.X......X]
Test: ( 52) webkit01.dat [....................................................]
Test: ( 35) webkit02.dat [...........X....XXX................]
All tests completed. 1399/1768 (79.13%) succeeded.

[CharlesChen0823]

@jaytaph I had pass tests23.dat, and the problem of tests19.dat is found. when your PR is merge, I will send an new PR.

[jaytaph]

@CharlesChen0823 awesome! I will wait for @emwalker and/or @Kiyoshika approval, and then I will merge the PR into main.

[jaytaph]

@CharlesChen0823 I have been looking at svg.dat tests, but it is very strange. The result should be a "svg" root node, but instead the parser will output "". I cannot find any way how the parser would create the svg node without also creating the html and body tags. Maybe you have an idea?

[CharlesChen0823]

I think that might be fragment case, currently not support. the parser should be split two parser, one for current normal case, and the other for fragment_case.

[jaytaph]

@CharlesChen0823 I've merged my work into the main branch.

[CharlesChen0823]

IMO, tokenzier and parser has circular dependency, which tokenizer hardly get information from parser. We should rethink this problem after finish the fragment case. @jaytaph

[jaytaph]

@CharlesChen0823 yes, the tokenizer and parser really are tightly close together. I've setup the document fragment case PR and it is merged. We now have a 90% pass rate on the tests. The way the parser and the tokenizer are initialized have been changed and it should be simple enough to get a parser up and running:

   // Something like this. I think we should have examples on how to use it.

   let stream = InputStream::new();
   let document = Document::share();
   let errors = Html5Parser::parse_document(&mut stream, Document::clone(&document), None);

There are a few .dat files that are failing mostly, but I think these are quick fixes. THanks for your awesome work on this.

[CharlesChen0823]

some test case failed, which the problem is possible in function pop_until, becuase pop_until sometimes also need justify node namespace.

[jaytaph]

@CharlesChen0823 Because of the template.dat tests, I needed to make adjustments to the test code, which resulting in a (big) refactor. Because of this, I've found that there we some tests that are actually failing, but were passing in our test code.

Now, there are a few tests still open that used to pass but we have to check these manually.

The PR is in #223 but I hope to get it approved tonight by @emwalker @Kiyoshika so we can merge it to main.

The current testst that are failing can be simply be viewed with cargo run --bin html5-parser-test.

Bwt, we are current at 96.91% 🎉

[jaytaph]

there is still an issue where the tokenizer needs to call the parser (i needs to know the namespace of the adjusted current node).. connecting the parser back to the tokenizer will create a cyclic dependency and makes this very hard to get right with the borrow checker. This I think is one of the reasons many implementations of a html5parser will actually have 3 actors: the tokenizer, the parser and a tree-builder (or treesink). Here, the parser will tell the tree-builder to add / retrieve nodes, and the tokenizer can ask the tree-builder for some additional information.

Now, in order to not completely rewrite the parser (yet).. i opt for just adding the information (the adjusted-current-node's namespace) to the tokenizer call in the parser. This is a bit of an ugly hack, but it works for now, since we know that no other data from the parser is needed, and it saves a lot of time rewriting the parser.

[CharlesChen0823]

when I testing the cdata section for passing parser state to tokenizer, I also using this workaround way. But this is still question, finally, IMO, the parser could rewrite.......

[jaytaph]

i'll add the latest code I have to a new PR (this is including the unmerged PR 223)... I'm currently at 99.37% pass rate

[jaytaph]

9bf6eb3

[CharlesChen0823]

I look at this branch, and add some modify comments. @jaytaph