Wanted Framework Features?

[thomhurst]

Is there a feature that other frameworks provide that TUnit doesn't?

Is there a feature that they don't support but you wish they did?

I'm open to suggestions on anything that'd help make TUnit meet your testing needs.

[Ergamon]

Maybe I overlooked something (I made my first TUnit steps today), but I miss a feature that you can generate the data on the specified types.

Currently I am using xUnit in conjunction with AutoFixture.
If you want to provide your own data you can use the MemberData, ClassData or InlineData attributes. But these are just out of the box implementions. You can always make your own DataAttribute.

Basically you have to implement a Create method with all defined types as parameters.

The attribute subclassing is not the perfect solution, but it does what is needed.

In the end I want to write a test like this:

[Test]
[AutoData]
public Task Test(int param1, string param2)
{

From your documentation I can only find the MethodDataSource, but there is no way to get the information about the arguments of the test.

[dev-in-disguise]

@thomhurst I had a look and I still think it should be feasible.
AutoDataAttribute plus all customisation attributes should work with the addition of the ParameterInfo array.
InlineAutoDataAttribute, which lets a user define parts of the arguments and generate the rest, should be also possible.

And out of interest: do you plan to create an analyzer for the attribute? Because right now if there is a mismatch between the parameters of the test method and the type arguments provided to the attribute this will result in a runtime error.

[thomhurst]

I'll pass in a parameter info array for you then! Hopefully I can do that tonight.

And yeah I do intend to write some analyzers for it. Just haven't gotten round to it yet

[thomhurst]

@dev-in-disguise this is done now! You can be passed either a property info (for property injection) or a parameter info array. Let me know if it lets you achieve what you need!

[dev-in-disguise]

@thomhurst I started today with the implementation. So far it looks great (and also awesome btw. that you could already add the analyzer as well!).

Just one thing: when I add more than one attribute on a test method I encounter an (at least for me) unexpected behaviour.
When I add the attribute twice I will receive 2 tests and generate the data 4 times.
When I add the attribute thrice I will receive 3 tests and generate the data 6 times.
And so on and so forth.

I would have expected one test and one data generation per attribute. I checked that against XUnits DataAttribute which also performs as I expected.

What is your reasoning behind doing it this way? Or maybe is this behaviour unintended?

[thomhurst]

Ah yes. I think I know why! It'll be a bug. If you want to raise it I'll try and take a look later

[MiroKov]

Hi, Do I have the ability to repeat the test in growing intervals? For instance, if the test fails because of a loss of connection, can I repeat it after 10 sec min, and if still fails, then after 25 seconds, then after 1 min?

…

On Thu, Sep 26, 2024 at 7:11 PM Tom Longhurst ***@***.***> wrote: Is there a feature that other frameworks provide that TUnit doesn't? Is there a feature that they don't support but you wish they did? I'm open to suggestions on anything that'd help make TUnit meet your testing needs. — Reply to this email directly, view it on GitHub <#646>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AENYVK4NKS4QI3QKYPCOLZ3ZYQ55RAVCNFSM6AAAAABO5PCE6WVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZXGI2DAMZSGI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[thomhurst]

Hey @MiroKov ! Yeah this can easily be done.

You can create your own retry attributes that simply inherit from RetryAttribute. Then you can plug your own logic into the ShouldRetry method.

It returns a task so you can wait asynchronously in there.

E.g.

public class ExponentialRetryAttribute(int times) : RetryAttribute(times)
{
    public override async Task<bool> ShouldRetry(TestContext context, Exception exception, int currentRetryCount)
    {
        await Task.Delay(TimeSpan.FromSeconds(currentRetryCount));
        
        return true;
    }
}

[MiroKov]

Vow! Do you have any notion of when will this come out of prerelease?Can’t wait to start using this.On 4. Oct 2024, at 00:49, Tom Longhurst ***@***.***> wrote: Hey @MiroKov ! Yeah this can easily be done. You can create your own retry attributes that simply inherit from RetryAttribute. Then you can plug your own logic into the ShouldRetry method. It returns a task so you can wait asynchronously in there. E.g. public class ExponentialRetryAttribute(int times) : RetryAttribute(times) { public override async Task<bool> ShouldRetry(TestContext context, Exception exception, int currentRetryCount) { await Task.Delay(TimeSpan.FromSeconds(currentRetryCount)); return true; } } —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[thomhurst]

Hopefully in the next couple of months.

I'm just collecting user feedback before I mark the API as stable

[MiroKov]

👍On 4. Oct 2024, at 07:29, Tom Longhurst ***@***.***> wrote: Hopefully in the next couple of months. I'm just collecting user feedback before I mark the API as stable —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[VMelnalksnis]

Is there some way to combine multiple DataSourceGeneratorAttribute on a class? My current tests are in NUnit, and I'm not sure how to migrate my current setup.

In my case I want to run all my tests with different with different fixtures, so I have a base class with a TestFixtureSource. One thing I noticed that this does not work in TUnit, as I have to add my DataSourceGeneratorAttribute on each class that inherits from the base class. For some of the test classes, I also have another TestFixtureSource with another argument, and all the combinations of the base class source and this one are run.

Here's a quick example:

[TestFixtureSource(typeof(DatabaseFixtureSource))]
public abstract class WebserverTests
{
    protected WebserverTests(WebserverFixture fixture)
    {
        Fixture = fixture;
    }

    protected WebserverFixture Fixture { get; }
}

public sealed class ExampleTests(WebserverFixture fixture) : WebserverTests(fixture)
{
    [Test]
    public void Test() => Assert.Pass();
}

[TestFixtureSource(typeof(OtherFixtureSource))]
public sealed class CombinedTests(WebserverFixture fixture, OtherFixture other) : WebserverTests(fixture)
{
    [Test]
    public void Test() => Assert.Pass();
}

I'm not really sure what would be the best way to port this to TUnit, since the fixtures also require async setup and should be create once per assembly. Probably have to write a for each combination, and store the fixtures outside of the data sources? I'm not sure how to handle setup/teardown then though.

One thing I couldn't figure out in NUnit, and wish was possible, is ordering all tests based on global fixtures. My issue is that Dapper can only be configured globally, and different databases require different configuration, so I'd like to run all PostgreSQL tests for different versions, then all Sqlite tests, etc.

[thomhurst]

Does WebserverFixture have a parameterless constructor? If so you can use

[ClassDataSource<WebserverFixture>(Shared = Shared.Type.Globally)

and in WebserverFixture:

public class WebserverFixture : IAsyncInitializer, IAsyncDisposable
{
        public async Task InitializeAsync()
        {
            ...
        }

        ...

        public async ValueTask DisposeAsync()
        {
            ...
        }

}

[thomhurst]

One thing I couldn't figure out in NUnit, and wish was possible, is ordering all tests based on global fixtures. My issue is that Dapper can only be configured globally, and different databases require different configuration, so I'd like to run all PostgreSQL tests for different versions, then all Sqlite tests, etc.

I don't quite understand. Can you give a code example of what you're trying to achieve?

[VMelnalksnis]

One thing I couldn't figure out in NUnit, and wish was possible, is ordering all tests based on global fixtures. My issue is that Dapper can only be configured globally, and different databases require different configuration, so I'd like to run all PostgreSQL tests for different versions, then all Sqlite tests, etc.

I don't quite understand. Can you give a code example of what you're trying to achieve?

I want to run all API integration tests against different databases, in my case different versions of PostgreSQL and Sqlite. The issue is that Dapper requires static configuration, see sqlite and PostgreSQL. Right now I have a hacky workaround for it - including the same source files in separate projects for each database.

[VMelnalksnis]

Does WebserverFixture have a parameterless constructor? If so you can use

[ClassDataSource<WebserverFixture>(Shared = Shared.Type.Globally)

and in WebserverFixture:

public class WebserverFixture : IAsyncInitializer, IAsyncDisposable
{
        public async Task InitializeAsync()
        {
            ...
        }

        ...

        public async ValueTask DisposeAsync()
        {
            ...
        }

}

Right now it does have parameters, see example, but in this case I guess I can make subclases if needed.

[VMelnalksnis]

One thing I couldn't figure out in NUnit, and wish was possible, is ordering all tests based on global fixtures. My issue is that Dapper can only be configured globally, and different databases require different configuration, so I'd like to run all PostgreSQL tests for different versions, then all Sqlite tests, etc.

I don't quite understand. Can you give a code example of what you're trying to achieve?

I got it working using ITestExecutor, not sure if there's a general solution for this. Here's an example - https://github.com/VMelnalksnis/Gnomeshade/blob/c232d3a5fc9e5fe92d724c52a3e968c0174a8046/tests/Gnomeshade.WebApi.Tests.Integration.Oidc/Fixtures/GroupedTestExecutor.cs and https://github.com/VMelnalksnis/Gnomeshade/blob/c232d3a5fc9e5fe92d724c52a3e968c0174a8046/tests/Gnomeshade.WebApi.Tests.Integration.Oidc/Fixtures/KeycloakGroupedTestExecutor.cs

[Vespion]

A feature that would be really useful would be what xUnit is calling two-phase parametrised tests. (Although it has yet to be implemented)

The idea is that a data source (such as from a DataSourceGeneratorAttribute) can interrupt the enumeration of generated test cases. This is very useful for property-based testing libraries that generate multiple random test cases for a single parameterised test method.

The original xUnit issue is here xunit/xunit#1719 but to paraphrase, the process is:

1. Write a parametrised test.
2. Generate lots of random data for test cases.
3. Start running the test method/function with the random data.
4. If all tests succeed, run to completion. (Yay!)
5. If one test fails, stop further enumeration of test cases and start a shrinking phase.

In the shrinking phase, the property-based testing library uses various heuristics to simplify the (randomly generated) test case to a single human-readable repro.

The problem with xUnit and other existing libraries is that they see both these generated test cases and the test cases generated by the shrinking algorithm as a single test, not a parameterised one with multiple inputs, making the test runner results unhelpful.

I don't think TUnit supports this yet (although if I'm wrong, I'd happily be corrected), but I'd have thought it'd be much easier to implement while the framework is still young.

[thomhurst]

@Vespion been looking into this.

What do you think of this?

https://github.com/thomhurst/TUnit/pull/1005/files#diff-561a5ccc1508320a5d6c3ef8a4ee3b6eca2e61b32c7c4be7b4fe495a1e93f9fc

In the example it's all abstracted away behind a custom attribute of DynamicDataGenerator. Behind the scenes that's generating data, setting up linked cancellation tokens so it can cancel the original enumeration, and re-registering tests if failures occur.

[Vespion]

From what I can tell, it solves the problem perfectly. I love how extensible TUnit is without getting in the way.

Without playing around with it, I can't be entirely sure it'll work, but I don't see any obvious issues.
I guess I'll grab a copy of the pull and have a play around, see if I can't get FsCheck to play nicely (you never know, it might work so well that I can package it as a nugget nuget extension?)

Unless that'd get in the way?

[thomhurst]

If you have time, go for it! I'm off to bed now 😴

[Vespion]

Oof, yeah looking at it I really should be too XD

[thomhurst]

@Vespion that FsCheck library is way beyond my understanding. I'm struggling to understand it even with the documentation haha.

Seems you need to construct a Property object, but that doesn't give you an IEnumerable of data, so I can't figure out how you'd inject in multiple test cases.

So yeah I'll leave it up to you to experiment!

[Ergamon]

It took a while for me to check back to this thread.

As for the AutoFixture part, I dont think that the current implementation would fit my developers. There is no point in type checking. It just makes writing tests painful. The whole idea of writing tests with AutoFixture is that you just write parameters in your tests and they will be created behind the scenes.

When I change the type of the parameter to another, then I need another parameter. This would force to change every parameter twice with no benefit.

Also if you write a migration from xUnit, where you currently just subclass the AutoDataAttribute to e.g. AutoNSubstiuteDataAttribute. You would be forced to subclass every generic attribute. Writing a lot of useless code. This is done just to override the Fixture factory, so there are other ways to solve this. A matter of how much different TUnit should feel.

It makes combinations also very hard to implement. Something like InlineAutoDataAttribute would need number of generic arguments minus one constructors.

[thomhurst]

@Ergamon can you pull the latest version and try inheriting from NonTypedDataSourceGeneratorAttribute instead? This will let you return a dumb object array instead, so no type checking!

[siewers]

I'm trying out TUnit by replacing xUnit in a small project. I'm using XUnit.Combinatorial, where the use of enums as test method parameters automatically generates a test case for each value in the enum.
Is this something that's planned to support?

This is what I'm able to do with XUnit.Combinatorial:

public enum MyEnum { Foo, Bar, Baz }

[Theory]
[Combinatorial]
public void MyTest(MyEnum enumValue)
{
    // This generates a test for Foo, Bar, and Baz
}

[thomhurst]

You can easily implement this yourself 😄

public class EnumGeneratorAttribute : NonTypedDataSourceGeneratorAttribute
{
    public override IEnumerable<Func<object?[]?>> GenerateDataSources(DataGeneratorMetadata dataGeneratorMetadata)
    {
        if (dataGeneratorMetadata.ParameterInfos is not { Length: 1 })
        {
            throw new Exception("Expecting one parameter");
        }

        var parameterType = dataGeneratorMetadata.ParameterInfos[0].ParameterType;
        if (parameterType is not { IsEnum: true })
        {
            throw new Exception("Expecting Enum parameter");
        }
        
        foreach (var enumValue in parameterType.GetEnumValues())
        {
            yield return () => [enumValue];
        }
    }
}

public enum MyEnum { Foo, Bar, Baz }

[Test]
[EnumGenerator]
public void MyTest(MyEnum enumValue)
{
    // This generates a test for Foo, Bar, and Baz
}

[siewers]

Okay, that doesn't look that bad. I could fork XUnit.Combinatorial into a version that works with TUnit :)
It has other combinatorial support like booleans, but the majority of the library is a port from NUnit to get the same combinatorial support in xUnit.

[AlexandreBossard]

With the demise of FluentAssertions, I find myself missing the CompleteWithinAsync() and NotThrowAfterAsync assertions.

Is this available and/or planned ?

[dev-in-disguise]

Hey @thomhurst!
I'm loving Matrix tests but can't use them in one scenario though they still would be very beneficial. Essentially I have a method that accepts multiple arguments and the arguments can also be null. But the method does not allow all values of the arguments to be null. And since I have anyway a dedicated test for these edge cases I kind of would love if Matrix tests would offer some kind of exclusion.

Something like this maybe where you could specify a method in the MatrixDataSource that can return the exclusions that should not be generated?

[Test]
[MatrixDataSource(ExclusionSource)]
public async Task T1_OnWithOneSelector_Should_ExecuteTActionOnSuccessResult([Matrix("success", null)]string? out0Result, [Matrix("success", null)]string? out1Result)

Would that be feasible and something you can imagine as a good addition?

[thomhurst]

Should be feasible! Maybe something similar to the Arguments attribute.

[MatrixExclusion(null, null)]

[dev-in-disguise]

If that fits better then sure!

[thomhurst]

Give v0.10.33 a try 😄

[dev-in-disguise]

Amazing! Again out there adapting stuff faster than anyone would be able to try out over the weekend 🤣
Works like a charm so thank you sooo much!