Add first draft of a spin-test SIP
#2462
Conversation
Signed-off-by: Ryan Levick <[email protected]>
|
|
||
| ## Open Questions | ||
|
|
||
| - Should `spin-test` be a stand alone CLI experience, a plugin for `spin` CLI, or both? |
There was a problem hiding this comment.
I would say plugin is probably best - at least as a starting point - as it would benefit from the plugin update framework.
There was a problem hiding this comment.
+1 on the plugin route for delivering the functionality to users.
| ## Open Questions | ||
|
|
||
| - Should `spin-test` be a stand alone CLI experience, a plugin for `spin` CLI, or both? | ||
| - How do we handle Spin applications with more than component? |
There was a problem hiding this comment.
Ignoring service chaining (?), presumably you could create a test composition per Spin component? That might even be ideal if you could compose multiple components as you wouldn't have to build anything you aren't currently testing.
There was a problem hiding this comment.
But which test composition do we end up actually calling? The spin-test host calls the run function to kick off the test. This run function is exported by the user-written test runner component. This test runner component is ultimately what calls the router (one or more times) as part of the test. If we create N test compositions, N - 1 of those compositions are not meant to receive that request (and calling their run function would just fail). At the core of the issue is that we don't really known which component needs to called before hand - the router is what makes that decision, but the router cannot compose with an arbitrary number of components to route to.
The only way I can see see to work around this are the two proposed solutions: moving routing into the host or synthesizing a router component.
There was a problem hiding this comment.
Ohhh I was thinking that the developer would explicitly map tests to components, but you are looking at doing that via regular URL path routing. Yeah I would think in the short term that would require the host to do the routing.
There was a problem hiding this comment.
I think this is a scenario that would become (at least for me) much clearer with an example of the (proposed/desired/currently possible/default) developer experience for writing tests. I think there are a few decisions that might impact the decisions here:
- does each Spin component come with its own sets of tests, or is there a single test "project" for the entire Spin application?
- depending on ^^, there could be a scenario where the test is linked to a single component, which means the question is no longer difficult to answer.
| - Should `spin-test` be a stand alone CLI experience, a plugin for `spin` CLI, or both? | ||
| - How do we handle Spin applications with more than component? | ||
| - There’s no way to compose a statically defined component with N number of components. This means the router component must know how many components it will be composed with ahead of time, and so the reasonable number of 1 is picked. | ||
| - We could work around this by moving the router out of a component and into the host, but this makes the host smarter which would like to avoid. We could also potentially synthesize a router component based on the `spin.toml` manifest instead of relying on a statically defined one, but this is certainly not trivial. |
There was a problem hiding this comment.
You could also (statically/macro-ly) generate a family of routers (router1, router2, ...). Annoying, but probably fine as a medium-term hack.
There was a problem hiding this comment.
This wouldn't work either (without changing the host to be smarter) as something needs to split one request into a call to one of N possible components. Composing N components with 1 component (where N is not known ahead of time) is not possible.
There was a problem hiding this comment.
What about 1 router with many identical component instance imports? It would be a little fiddly but you should be able to derive one consistent ordering of spin components from a manifest which would let the composition step fill the imports in the same order that the router then expects them. You would also need some way to fill all the unused import slots, either a static dummy component that always traps or some Linker jiggery-pokery.
There was a problem hiding this comment.
So you're suggesting a single router component with some arbitrary amount of imports that components can plug into and we either plug those with actual Spin app components or with a dummy trapping component once we run out of Spin app components?
| - How exactly should the various worlds look like? | ||
| - The proposal above is enough to get things working, but there is plenty of room for bike-shedding the exact shape of the worlds. | ||
| - Which triggers should we support? | ||
| - Can we support an arbitrary number of triggers through some sort of plugin system? How would that work? |
There was a problem hiding this comment.
This feels like something that may become dramatically easier with CM async (to the extent that magically fixes all our nested composition woes).
There was a problem hiding this comment.
Hmm... I'm not sure I follow - none of the triggers (with the exception of wasi:http/incoming-handler which we already support) are necessarily built to try to accommodate async. I don't see how CM async improves the situation here. Could you explain more about what you mean?
There was a problem hiding this comment.
Ah yeah sorry the train of consciousness left the station early there...
I think each trigger type could have a test driver adapter to wrap the component-under-test and test driver to implement the runner world; this is impossible today because the host has to provide resource factories to glue everything together.
Signed-off-by: Ryan Levick <[email protected]>
Signed-off-by: Ryan Levick <[email protected]>
| - How should test discovery work? | ||
| - The most straightforward way would be for `spin-test` to require the user to specify the path to a `spin-test` compliant test component binary. | ||
| - Another way would be for there to be a convention around a `tests` directory. This `tests` directory could include both built `spin-test` compliant test component binaries as well as directories with source code and a `spin-test.toml` configuration file that specifies how the source code is built into a `spin-test` compliant test component binary. `spin-test` would then build all the tests that need to be built and run each test binary. | ||
| - We could also carve away space in the `spin.toml` manifest (e.g., by using the `[component.<id>.tool]` section). Such a solution might be more appropriate if we determine that users will normally want very few test component binaries. If, however, it proves to be useful to potentially have many test binaries, splitting the configuration out to each test binary in the form of a bespoke `spin-test.toml` per test binary might be the better solution. |
There was a problem hiding this comment.
I suppose we could also do both (eventually), with a compatible schema reused between spin.toml [component.<id>.tool] for one-test-per-component and tests/spin-test.toml for more complex scenarios.
There was a problem hiding this comment.
Another option to consider might be the application.tool section in spin.toml, e.g. [application.tool.spin-test.test-a]. I'm not sure I would actually like that but it would avoid having another config file.
|
|
||
| At the core of `spin-test` is component composition. The user’s Spin app is composed together with other components to ultimately produce a component (referred to as “the composition” below) that has none of the original Spin or WASI imports. The composition has the following components internal to it: | ||
|
|
||
| - A single component from the Spin app (see the *Open Questions* section for why only one component from a Spin app can be supported) |
There was a problem hiding this comment.
This refers to "a single component" from the perspective of the Spin application manifest I assume — if my Spin component build process generates a statically linked component, I would expect testing to continue to work as expected.
Is that accurate?
|
|
||
| - A single component from the Spin app (see the *Open Questions* section for why only one component from a Spin app can be supported) | ||
| - A Spin virtualization component that virtualizes all `fermyon:spin/platform` interfaces. | ||
| - A WASI virtualization component that virtualizes all the `wasi:cli/imports` interfaces. |
There was a problem hiding this comment.
As I was going through the test examples from the implementation repository, I was thinking about the wasi:cli/imports imports — will this contain a programmatic API to set the behavior of all WASI functionality?
Will there be a subset available to begin with?
I'm curious what the API through which we expose that to users in tests will look like.
| *Note*: | ||
|
|
||
| > **A few notes:** | ||
| * We may still wish to have the composition import `wasi:cli/stdout` and `wasi:cli/stderr` so that it can easily log things. |
There was a problem hiding this comment.
The example test I ran from the implementation repo did log to stdout -- is this already supported?
| * It might be possible to statically configure the virtualized Spin runtime component given a manifest file by synthesizing the component on the file (in much the way that WASI-virt works). This would make it possible to no longer require that the manifest is passed to the composition component at runtime since it has already been statically configured to run correctly with that Spin manifest. This has its downsides though as composition components for the same app but different manifests cannot be shared between tests. | ||
| * The `wasi:http` work around imports may hopefully eventually overcome with changes to the `wasi:http` package. | ||
| * If this and the manifest import are eliminated, tests could potentially be run by an component runtime with no imports whatsoever. | ||
| > |
| /// The ability to call the Spin app | ||
| import wasi:http/[email protected]; | ||
|
|
||
| /// Actually call the test |
There was a problem hiding this comment.
nit: formatting for WIT source and comments seems off in parts of this file.
|
|
||
| ## Open Questions | ||
|
|
||
| - Should `spin-test` be a stand alone CLI experience, a plugin for `spin` CLI, or both? |
There was a problem hiding this comment.
+1 on the plugin route for delivering the functionality to users.
| ## Open Questions | ||
|
|
||
| - Should `spin-test` be a stand alone CLI experience, a plugin for `spin` CLI, or both? | ||
| - How do we handle Spin applications with more than component? |
There was a problem hiding this comment.
I think this is a scenario that would become (at least for me) much clearer with an example of the (proposed/desired/currently possible/default) developer experience for writing tests. I think there are a few decisions that might impact the decisions here:
- does each Spin component come with its own sets of tests, or is there a single test "project" for the entire Spin application?
- depending on ^^, there could be a scenario where the test is linked to a single component, which means the question is no longer difficult to answer.
|
What is the status of the SIP wrt the actual implementation from https://github.com/fermyon/spin-test? |
|
@radu-matei There haven't been any fundamental objections to the approach and we're far enough along in the implementation that I think this could be merged. I would just need to update the document to reflect the true state of the proposal. |
This SIP describes the
spin-testtool that allows running unmodified Spin applications in WebAssembly component based testing environment where all potential imports to the Spin app are virtualized and can be configured by the user in a language independent way.Implementation of this SIP has already begun here.