Parallelism control

[MelGrubb]

I'm pretty excited about the parallel execution features here, but I have a couple questions. In the ReadMe, there's a bullet point for "Test ordering (if running not in parallel)". Does this mean it's an all-or-nothing thing? I'd love the ability to write CRUD style integration tests that can "fan out" in the middle.

1 - Create a new User
2 (Depends on 1) - Retrieve the newly-created User by Id
3 (Depends on 2) - Check that certain fields were filled in by the back-end
4 (Depends on 2) - Update user's email
5 (Depends on 4) - Retrieve the user based on the updated email
6 (Depends on 3, 4, & 5) - Delete the user
7 (Depends on 6) - Check that the user no longer exists

Tests 3 & 4 can be run in parallel since the initial user was retrieved in step 2.
4 needs to wait on 2 & 3 to make sure they're finished before it goes changing things.
6 needs to wait on 3, 4, & 5 so it doesn't delete the user before they're done with it, but importantly, 3, 4, & 5 don't have to PASS. They just have to complete.

This is a simple example, but I think you see what I mean. Step 3 could actually be multiple individual tests, checking multiple different properties in parallel. #4 would need to wait for 3a, 3b, and 3c to all finish before continuing. I think this is where I'd get the biggest payoff. As long as a set of tests don't mutate the object, they should all be able to run at the same time. I think this would speed a lot of things up in integration tests where each individual step can incur a delay. Those really start to add up when they run in series.

[thomhurst]

Hey @MelGrubb - That ordering section in the readme refers to a more primitive "Order = 1" "Order = 2" functionality, similar to what other frameworks have.

But there's also the "DependsOn" functionality which effectively orders your tests too, but without having to turn off parallelism.

This should entirely be possible for you.

Pseudocode:

[Test]
public void CreateUser { ... }

[Test]
[DependsOn(nameof(CreateUser))
public void RetrieveUser { ... }

[Test]
[DependsOn(nameof(RetrieveUser))
public void CheckBackend { ... }

[Test]
[DependsOn(nameof(RetrieveUser))
public void UpdateEmail { ... }

[Test]
[DependsOn(nameof(UpdateEmail))
public void RetrieveUserAgain { ... }

[Test]
[DependsOn(nameof(RetrieveUser))
[DependsOn(nameof(UpdateEmail))
[DependsOn(nameof(RetrieveUserAgain))
public void DeleteUser { ... }

[Test]
[DependsOn(nameof(DeleteUser))
public void CheckDeleted { ... }

Does that help? Bits that can run in parallel should automatically be worked out by the framework so you don't have to work it out yourself. Otherwise it'll wait as it needs to.

[MelGrubb]

I think that's exactly what I'm looking for. A few experiments will let me know for sure. Thanks.

[thomhurst]

@MelGrubb did this all work for you?

[MelGrubb]

I was out of town yesterday and will be away part of today as well, so I haven't been at my desk all weekend. I'll try this week though. Thank you.

[thomhurst]

but importantly, 3, 4, & 5 don't have to PASS. They just have to complete.

DependsOn doesn't currently supporting starting tests if its dependencies have failed.
Instead I'd recommend just having an [After(Class)] method to make sure your user is deleted if you need to clean up your environment

[MelGrubb]

That's probably what I'd use now, but in the case of CRUD tests, the "D" part truly is one of the tests. It's not just cleanup. I want to assert that the item was successfully deleted. I put asserts in my teardown right now because there are no frameworks that support this use case, but the experience isn't the best. the failure ends up showing at the class level. I'm intentionally misusing the existing frameworks, but I'd love to find one that recognizes this scenario as an actual, legitimate case.

[thomhurst]

If depending on a test, and still starting even if it failed does give benefit, I can implement this. Probably a property on the attribute like "ProceedOnFailure=true".

Do you want to raise an issue and I can add it as a feature?

[MelGrubb]

Sure. I'll create an issue for it.

[thomhurst]

@MelGrubb Available in 0.1.783

Docs here: https://thomhurst.github.io/TUnit/docs/tutorial-extras/depends-on#failures

[MelGrubb]

Man, you're fast. Thanks.

[thomhurst]

Step 3 could actually be multiple individual tests, checking multiple different properties in parallel. 4 would need to wait for 3a, 3b, and 3c to all finish before continuing. I think this is where I'd get the biggest payoff.

This could be optimised by storing your data in the TestContext and using that in split out tests, to avoid having to retrieve over and over.

[Test]
public async Task RetrieveUser()
{
    var user = await GetUser();
    TestContext.Current!.ObjectBag["User"] = user;
}

[Test]
public async Task CheckName()
{
    var user = TestContext.Current!.GetTests(nameof(RetrieveUser)).First().ObjectBag["User"] as MyUserModel;
    await Assert.That(user.Name).IsEqualTo(whatever);
}

[MelGrubb]

I was thinking I'd just throw it in a class-level field after retrieval, but the property/object bag works fine too, just as long as you can ensure that CheckName won't try to run before RetrieveUser has finished. I'm just looking for a way to say that the first couple steps happen serially, then everything can fan out before coming back together at the end. The comment saying that test ordering only works when not running in parallel is why I'm asking. I'd really like to say that some of the tests can run in parallel, but there are definite checkpoints along the way where I need to tell it to wait for all the tests up to that point to complete before continuing.

[thomhurst]

I was thinking I'd just throw it in a class-level field after retrieval, but the property/object bag works fine too, just as long as you can ensure that CheckName won't try to run before RetrieveUser has finished. I'm just looking for a way to say that the first couple steps happen serially, then everything can fan out before coming back together at the end. The comment saying that test ordering only works when not running in parallel is why I'm asking. I'd really like to say that some of the tests can run in parallel, but there are definite checkpoints along the way where I need to tell it to wait for all the tests up to that point to complete before continuing.

Yep see my comment 2 comments up. This is entirely possible :)

[thomhurst]

Class level field can work too but each test will have its own class instance so you'll have to make it static to share it between different tests just fyi

[RedX2501]

The example given by him where testcase 3 could be many smaller tests 3a, 3b and so on makes me wonder whether it makes sense for the DependsOn attribute to support wildcards (or regex patterns).

Fow example DependsOn("CheckAttribute.*") would match all tests starting with CheckAttribute for example CheckAttributeName, CheckAttributeEmaul and so on.

[thomhurst]

Since tests are defined at compile time, you'll know the method names at compile time. Due to that, I think keeping it explicit makes things clearer. It's a couple more attributes on your test but then your test is clearly showing what it's doing, and if you use nameof then your ide will easily let you navigate to that method with ctrl + click

[skoging]

Is there a chance the experience of this kind of test setup could be improved when running a specific test?

Currently what happens is that all dependant tests are run in addition to the specific test. There is no possible way of specifying that a test is not required to be run for this this test to be run, but if it is included in the filter, it must be run before this test.

Example:

[Test]
public void CreateUser { ... }

[Test]
[DependsOn(nameof(CreateUser))
public void RetrieveUser { ... }

[Test]
[DependsOn(nameof(RetrieveUser))
public void CheckBackend { ... }

[Test]
[DependsOn(nameof(RetrieveUser))
public void UpdateEmail { ... }

[Test]
[DependsOn(nameof(UpdateEmail))
public void RetrieveUserAgain { ... }

[Test]
[DependsOn(nameof(CreateUser))
[DependsOn(nameof(RetrieveUser), ProceedOnFailure = true, Optional = true)
[DependsOn(nameof(UpdateEmail), ProceedOnFailure = true, Optional = true)
[DependsOn(nameof(RetrieveUserAgain), ProceedOnFailure = true, Optional = true)
public void DeleteUser { ... }

[Test]
[DependsOn(nameof(DeleteUser))
public void CheckDeleted { ... }

Running just the DeleteUser test would also run the CreateUser test first, but not RetrieveUser, UpdateEmail, or RetrieveUserAgain as these are not required in order for the DeleteUser test to run.
Running all of these tests would still keep the ordering to ensure the user is not deleted while other tests depends on it.

Another "nice to have" feature would be to specify a reverse dependency, to ensure DeleteUser is always run after CreateUser, even if not explicitly included. This would be a way to ensure clean up of resources that are not automatically reset after the tests (e.g. a common or "live" database). [After(Class)] could be a way to do this, but would require some way to use a shared fixture like from [ClassDataSource<Fixture>(Shared = SharedType.PerTestSession)], this is currently not possible as the method must be static, and takes no arguments.

[thomhurst]

There is no possible way of specifying that a test is not required to be run for this this test to be run, but if it is included in the filter, it must be run before this test.

Sounds sensible.

Another "nice to have" feature would be to specify a reverse dependency

Not sure I like this. Makes things more convoluted and is essentially mimicking After hooks.

this is currently not possible as the method must be static, and takes no arguments

You can extract objects out of your class hook context, it just involves a bit of casting and filtering from within an After(Class):

    [After(Class)]
    public static void EnsureDeleted(ClassHookContext context)
    {
        if (context.Tests.FirstOrDefault(x => x.TestDetails.TestName == nameof(DeleteUser)) 
            is { Result.Status: Status.Passed })
        {
            return;
        }
        
        ((MyClass)context.Tests.Last().TestDetails.ClassInstance).DeleteUser();
    }