Idea to improve performance of unit tests

[johnwest80]

Brandon, right now, you mock everything the module depends on before running a test, correct? Do you think it would be possible to only mock dependencies as angular attempts to inject them? Is there some hook that would enable this (or hack :))? I bet we're not alone in having modules that have lots of imports for all the different components, while each component might only use a fraction of them? Or, in the case where a ui module is imported, it might be that only a few of all the possible ui components are used?

Having this would significantly increase the speed of unit tests. Thoughts/

[getsaf]

This is a great idea. We do some caching now but it only serves to prevent duplicating a module during a test so the caching doesn't last beyond an individual test.

I don't know of any first-class hooks from Angular that would allow for this but it might be possible to do the kind of caching you're describing using Proxy wrappers around modules and/or providers so once we traverse a module one time, we can re-use the mock for the next test to save ourselves another traversal.

I may give this a look next week or so. If you have any ideas about where to add a hook so we can do on-demand mocking let me know.

[johnwest80]

unfortunately, i'm not really familiar with the angular internals. but if you make this happen, it'll be a huge improvement to test run speeds!

[johnwest80]

@getsaf not to be a pain :), but did you ever get a chance to look into this? really hopeful this can be done!

[getsaf]

Hey, sorry for the delay, I just had some time to do some inspection and as my first task, I set out to get an idea of the current performance of module mocking is.

In short, I'm not seeing any heavy performance issues when mocking modules.

I setup some tests which mocked some rather large Angular internal modules in a loop:

  it.each(new Array<number>(2000).fill(0).map((_, i) => i))('mocks angular modules', () => {
    class Foo {}
    class Bar {}
    // Avoid using internal module-mock cache by creating a new TestSetup for each mock (shallow-mock cache lives on the TestSetup instance)
    expect(isMocked(mockModule(FormsModule, new TestSetup(Foo, Bar)))).toBe(true);
    expect(isMocked(mockModule(CommonModule, new TestSetup(Foo, Bar)))).toBe(true);
    expect(isMocked(mockModule(ReactiveFormsModule, new TestSetup(Foo, Bar)))).toBe(true);
    expect(isMocked(mockModule(RouterModule, new TestSetup(Foo, Bar)))).toBe(true);
    expect(isMocked(mockModule(HttpClientModule, new TestSetup(Foo, Bar)))).toBe(true);
  });

In total the test suite mocked:
modules: 18,000
providers: 22,000,
declarations: 126,000

Test Suites: 1 passed, 1 total
Tests:       2000 passed, 2000 total
Snapshots:   0 total
Time:        5.616 s, estimated 6 s

The whole thing ran in <6 seconds on a single thread which seems reasonable to me for producing >160k mocks.

It's possible my test setup isn't mirroring your situation though. Is there any way you could help me pinpoint where the slowness lives in your test runs?

[johnwest80]

I’m not sure what makes ours slower (how we reference files, some config?) but we have modules that reference other modules like three iterations of ux libraries, that, when I specifically reduce imports for one test, I see the tests within it execute much faster. And considering we have modules importing modules importing modules, well, you get the point 😀.

Right now we can see tests take half a second each or more, and it does seem to correlate with how many imports the module has.

Wish I could give you more, but does that help make sense why I think caching will help considerably?

[getsaf]

Gotcha. I could see that if you are running one test suite at a time, you may be taking a penalty for simply importing the module tree which means they would need to be transpiled ahead of time (this would happen well before the actual tests run, before this library has been loaded).

This could be verified if you see minimal difference in performance when artificially inflating the tests in a single test file. (i.e., copy/paste the same tests like 10x a single test-file). You could also try running manual timings on the shallow.render calls just to see if any one of those calls is taking a long time (if the render calls are fast, it's more likely you're seeing the transpilation issue).

If this is the issue you're seeing, it would probably have an outsized effect on single test file runs but should even out with transpilation cache when running many tests (i.e., each module only builds once and is cached for subsequent runs). It's worth a check to see that you have the basic transpilation cache in Karma/Jest hasn't been disabled.

[johnwest80]

so did a little more testing. i have a spec file whose module imports 30 other modules. i have 20 tests in the spec file. when i run it, i see that each of the 20 tests takes the same amount of time, 500ms. if i remove 15 of the imports (they're not needed for this component), each of the 20 tests takes the same time, 300ms. this suggests to me that the mocking of all the classes in the module is what making the tests slower. i'm not surprised by that - it's what i would expect, almost. the question i have around caching is, could the mocks be cached before the first test such that time to first test is slower, but then, if the mocks could be reused for the other 19 tests, those other 19 tests would be faster?

or do you have to create new mocks every time for each of the 20 tests because they can be changed during the execution of a given test, and therefore you have no choice?

[johnwest80]

and also thinking, do you create a tree of what a component is using before mocking? if so, could you only mock what the component is using? lots of ifs and i'm sure there are complexities, just thinking aloud really.

[johnwest80]

so here is a test run with extra modules that aren't needed by the particular component... note that all runs take approximately the same time.

[johnwest80]

and here it is after taking out extra modules...

[johnwest80]

and with my suggestions above, in the scenario above with extra modules, i would hope to see the first test take longer, but then each test after take more around the 8-11ms time that we see when those extra modules aren't mocked. just some way to cache mocked modules or eliminate the time needed to mock if the module isn't needed.... that all make sense?