Testing with Injectables

[thegridman]

After the discussion on yesterday's zoom call about testing I wanted to throw out there the ideas I had based on some of the comments from people on the call.

The Problem

The issue is how to run a test for a class that requires something to be injected when that test is running locally and not running in the platform, for example, injecting a database.

The example used in the call was how to test the welcome example shown below

@WebApp
module welcome.examples.org {
    package web import web.xtclang.org;
    package db  import welcomeDB.examples.org;

    import web.*;

    @WebService("/welcome")
    service SimpleApi {
        @Inject db.WelcomeSchema schema;

        @Get
        Int count() {
            return schema.count.next();
        }
    }

    @StaticContent("/", /webapp)
    service Content {}
}

As tests will be running locally via the XUnit engine, this line is the issue, as XUnit has no idea how to inject a WelcomeSchema:

@Inject db.WelcomeSchema schema;

The Possible Solution

We talked through various things but something Mark mentioned seemed like a sensible idea. Rather than having to add something to the command line, really the tests themselves should be able to say what gets injected here so different tests can inject different things to test different scenarios. We already know how to create a DB on the fly, we have other examples that do it. We obviously do not want the developer to have to write that code (even though there is not too much) but it can be hidden away in a utility.

Everything in XUnit is pretty much controlled using the concept of an Extension, so it is simple to add functionality using custom extensions. My idea is that we have an Extension that is an "Injectable Provider".

The welcome example with a simple test might look like this:

@WebApp
module welcome.examples.org {
    package web import web.xtclang.org;
    package db  import welcomeDB.examples.org;

    import web.*;
    import xunit.annotations.*;
    import xunit.mocks.*;

    @WebService("/welcome")
    service SimpleApi {
        @Inject db.WelcomeSchema schema;

        @Get
        Int count() {
            return schema.count.next();
        }

       @TestInjectable("schema")
       static db.WelcomeSchema testSchema = Mocks.createDB(db.WelcomeSchema);

       @Test
       void testCountIsNextValue() {
              testSchema.count.set(19);
              assert count() == 20;       
       }
    }

    @StaticContent("/", /webapp)
    service Content {}
}

The important bits are:

       @TestInjectable("schema")
       static db.WelcomeSchema testSchema = Mocks.createDB(db.WelcomeSchema);

       @Test
       void testCountIsNextValue() {
              testSchema.count.set(19);
              assert count() == 20;       
       }

The first part is the @TestInjectable which the XUnit engine processes when it builds up the model for the tests. This annotation has a name, and basically means for the given name and property type inject this value instead.
The code below says create an injectable with name "schema" and type db.WelcomeSchema. (Exactly how Mocks.createDB() works is TBD...)

       @TestInjectable("schema")
       static db.WelcomeSchema testSchema = Mocks.createDB(db.WelcomeSchema);

This would allow test code to specify something that will be injected into the code being tested, and the tests are able to configure that with whatever state is required before the tests execute. So in this case a DB could be initialised with whatever data the tests required. This could be done in @BeforeAll or @BeforeEach methods in the test or some other way when the mock DM is created. And obviously like all the other XUnit mixins TestInjectable would is only be present during testing.

As that property is static the XUnit engine can access it and do whatever is required "before" creating the instance of SimpleApi required to run the test. So the mock injectable is added to the XUnit container's injector and when the instance of SimpleApi is created, the mock DB will be injected.

The test method simply sets a value for the count in the mock DB and asserts the call to count() returns the correct value.

       @Test
       void testCountIsNextValue() {
              testSchema.count.set(19);
              assert count() == 20;       
       }

I could obviously write a whole suite of test classes to test SimpleApi that use different injectable DBs with different states. The mock DBs are created and destroyed as part of the test lifecycle. The same pattern should work for other injectable things. What if I want to inject an instance of a Clock that I can control from my tests to test specific time related functionality.

Potential Gotchas

That all seems workable based on my possibly lacking understanding of the container and injection functionality, but I can see some possible issues.

In the example above the mock DB is accessible by XUnit before it needed to create an instance of SimpleApi to execute the tests in it. But what if the test methods were in a singleton, like a module or package, as these are not created by XUnit calling a constructor. If there is a static @TestInjectable property in say a Module, how can XUnit process that and add it to its injector before it needs to be injected into the module? Maybe this is just my lack of understanding of the lifecycle of singletons and how the container works.

[markandrewfalco]

I’m curious as to how things would look if we separate the tests and test injectables from the class/code being tested. The tests would run in container N and the code under test would run in container N+1, and presumably XUnit in N-1? This would seem to solve your singleton problem and also in my opinion result in cleaner test and impl code. Also it would allow @TestInjectables to share the same name as the @Injectable .I.e “schema” in your example.

[markandrewfalco]

This also raises the question of how does one container provide injections to a child container in non-test scenarios. I suspect whatever the answer is there we can directly use that for testing and not even need @TestInjectable

[cpurdy]

Mark, there are reasons for both. One reason that we will want to run tests in a container is so that we can run tests in parallel, without any concern for interaction among tests. So in theory, we'll be spinning up as much as one container per test -- although, more likely is one container per "parallel test".

The tests would run in container N and the code under test would run in container N+1, and presumably XUnit in N-1?

Possible, but unlikely in the general case (I think?)

Also it would allow @TestInjectables to share the same name as the @Injectable .I.e “schema” in your example.

Right. See my response to JK.

how does one container provide injections to a child container in non-test scenarios

We have both the ability to provide the injections into a child container (this is quite normal stuff in Ecstasy), and even the ability to provide injections within a container (on a module by module basis). @ggleyzer -- do we have an example of this somewhere in the repo?

[thegridman]

Right, this is where I need to learn more about how all the container and injection works. If there is a way to simplify it, that is obviously better.

Regarding separating the test injectables, maybe they can go into a separate module. Then we have a module containing a set of our injectable things configured however we need for the tests. Then we can add that injectables module on the XUnit command line so XUnit can do whatever it needs with those injectables before it fires up the test suite.

[ggleyzer]

I like that idea very much. It definitely covers what I tried to do manually. Imagine that you would write a test like this:

@TestModule
module welcomeTest {

    package welcome import welcome.examples.org;
    package welcomeDB import welcomeDB.examples.org;

    @BeforeAll
    void init() {
        configureWeb(welcome);
        configureDb(welcomeDB, createNew=True, allowWrite=True);
    }

    @Test
    void testCount(welcome.SimpleApi webService) {
        assert webService.count() % 2 == 1;
    }
}

where "configure*" are methods of TestModule mixin providing the necessary context to the "xtc test" facility. Then you'd be able to run it either directly from a command line or in the hosting environment as a "hosted test"

[cpurdy]

Reading this, I'd suggest @TestPrepare or at least @BeforeAllTests so it's clear (without any context) that the annotations are explicitly test related.

I also think we need to be more declarative than explicitly prescriptive, when possible. At the same time, it's a balance: Making things "too declarative" tends to become its own confusing DSL that you have to become an expert on (see: Gradle).

[cpurdy]

Regarding:

testSchema.count.set(19);

I'd suggest that we want the schema injection itself to be that mock (or test instance of the database or whatever it is), instead of having to introduce an additional schema property.

I could obviously write a whole suite of test classes to test SimpleApi that use different injectable DBs with different states.

Right. And it would be handy if we could somehow describe that at the use site, i.e. "this test needs a database derived from that generic test database image that I gave you, but I need you to corrupt the date to the year 1492 on the order number 123 before you give me the database". Or alternatively, "here's the exact content that I need in the database, spelled out in some literal form".

The same pattern should work for other injectable things. What if I want to inject an instance of a Clock that I can control from my tests to test specific time related functionality.

This is one of the reasons why we selected injection as the metaphor.

What we need to figure out is how to best represent this, so that it's easy to make a predictable environment for the tests, yet also easy on a test-by-test basis to tweak that environment (i.e. start from the predictable environment, then modify) in very specific ways.

-- EDIT --

Gene asked me about my use of the term "mock". Just to be clear, I use it as a catch-all term for "something that we provide in testing that doesn't screw up production", so it could be an empty implementation of an interface that just throws an exception on every method, or it could be a fake database, or it could be a real database that the test can't mess up, etc.

[ggleyzer]

Cam came up with a very elegant idea of having an injectable Injector, which could be used for a pass-through resource provider. I just pushed all the relevant pieces into the master.

Please take a look at the PassThroughResourceProvider and its usage by the Runner.

I think it addresses all the needs, though the orchestration could be a bit tricky.