Meeting of the minds: architectural considerations and goals

[jaytaph]

I would like to propose to see if we can discuss the architectural layout of what we are trying to achieve. I see the current system more as a proof-of-concept on different aspects of a browser, but it have a solid foundation on which we can build a stable engine yet.

In order to do this, we should keep going the way we are going: figuring out little bits, see how they function and try to implement it, and secondly: try to come up with a good architectural foundation to incorporate these components (or rewrite them with the gained knowledge).

It might be too early to do this yet, but at least we can think of some idea's for it.

[blaumeise20]

I found this great image:

(https://viethung.space/blog/2020/05/29/Browser-from-Scratch-Introduction/)

[emwalker]

I like this diagram. It makes some sense of the browser code bases I've been looking at. I was a little surprised to discover that the rendering engine is what is responsible for parsing HTML. This is the case in Chromium at least.

[blaumeise20]

I honestly expected nothing else, because the "user agent" and the "browser/rendering engine" are basically two separate things, and you should be able to just use the engine in a headless way and still be able to completely control it (e.g. take "screenshots" of the page).

[emwalker]

I learn best by doing and by osmosis, as I work with a code base, and by studying what has already been done. I have a general sense of what some of the moving parts of a browser engine are now. But at this stage I'm hardly able to see the forest beyond the trees. I find myself latching on to specific concepts like a "parser" and a "tree builder" and a document that holds the DOM tree that is mutated in place. Downstream from there there's the code that will traverse the DOM (possibly multiple times) and turn it into something a display can understand. And then surrounding all of this there's all of the UI that will eventually need to be built out with all of its components, like bookmarks, a search bar, tabs, and so on.

A browser engine, and around it a browser, are like an old-fashioned mechanical watch, with thousands of moving pieces. Gradually you can see the different gears and other components become reality. But I do not know how much of that it will be possible for me personally to discern without more experience working with the code.

[jaytaph]

i've been thinking about the network layer as well.. here again there are multiple ways to do it, but this layer could be pretty isolated to the rest of the system (i think)... should be support os name resolving, our own dns client, doh etc.. and how do we handle things like TLS, http2/quic/speedy/etc.. even socks proxies etc should have some consideration I guess...

[emwalker]

Where does something else go?

Just guessing, but I believe that fragment would be parsed in three stages. First the parsing code would process everything up until the script, and then return. Then the script would then be passed over to the JS code with a document it can mutate. When the script completes, a second parsing pass would pick up something else. Something like that. I'd recommend reviewing the WebKit and Chromium implementations.

[jaytaph]

Where does something else go?

Once the </script> tag is tokenized and parsed, the script's content is executed in the javascript engine. This will remove the <div> from the DOM. At that point, the parser is resumed and the next tokens are read.. that would be the text "something else". This text would be added to the current node (the last node on the open stack), which is still the <div> node, since javascript cannot touch parser data. At that point this node does not exist anymore in the dom, but it DOES exist in the arena. So the parser add's the "something else" to the "div" node (which is a dangling node at this point).

in the end, the whole div is not shown, but it IS parsed

[emwalker]

Probably premature, but I want to suggest this as a starting point for the high-level approach (this is a strawman proposal intended for consultation / discussion):

• One process per browser tab (like Chrome). This makes the browser more resilient against page crashes. A page crash in a browser tab might bring down the process for that tab but usually not the whole browser.
• Within the process for a tab, several threads, up to the available number of cores, with a main thread and background threads.
• Background threads receive tasks from a multi-producer, single-consumer queue (std::sync::mpsc). An example of a background task is fetching a stylesheet from a remote server.
• We do not attempt to use Rust async for this as a first pass (e.g., tokio). Rust async is good for servers and other processes that must handle thousands of actor-like tasks. But it also introduces function coloring and spreads async fn to all callers that call into the async code. We should use Rust async if it's needed, but I suspect it might not be needed for our use case.
• HTML rendering happens on the main thread for a browser tab, and the main thread blocks on the rendering of short fragments of HTML. But it doesn't block for long, because it isn't parsing the whole document. It's parsing small fragments of the document and then returning control to the UI and then resuming parsing again, quickly back and forth.
• The handoff from the main thread to a background thread happens through implementors of traits, so that a single-threaded test implementation can be made that runs in the unit tests without needing to spawn separate threads.
• You can also make unit tests that specifically test the spawning of threads, but in many cases we could keep the tests simple and not test with multiple threads.

[blaumeise20]

in the end, the whole div is not shown, but it IS parsed

Oh, I understand. Yeah, that makes sense.

[jaytaph]

I havent checked this in outher engines, but it sounds to me that this makes the most sense. Maybe they can optimize out things once the parser becomes aware of a dangling node, but foster parenting and the adoption agency makes this quite difficult to implement