Server-side coverage not reported when using next.js

[romandecker]

Proof of concept repository: https://github.com/romandecker/next-ts-with-cypress-coverage

Steps to reproduce:

1. npm install
2. npm run dev
3. In another terminal: DEBUG=code-coverage npm run cypress run
4. npm run see-cy-coverage and verify that getServerSideProps() in pages/index.tsx was called and coverage was tracked successfully, but the function it calls, in no-coverage-here/some-file.ts still shows up as having 0% coverage, even though it was clearly executed.

Logs and screenshots

> [email protected] cypress
> cypress "run"

Missing baseUrl in compilerOptions. tsconfig-paths will be skipped
  code-coverage combined NYC options { 'report-dir': 'cypress-coverage', reporter: [ 'json', 'lcov' ], extension: [ '.js', '.jsx', '.ts', '.tsx' ], excludeAfterRemap: false, extends: './.nycrc.json', include: [ 'pages/**/*', 'no-coverage-here/**/*' ], exclude: [ 'pages/api/__coverage__.js' ], all: true, sourceMap: true, instrument: true } +0ms

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        5.3.0                                                                          │
  │ Browser:        Electron 83 (headless)                                                         │
  │ Node Version:   v16.14.0 (/Users/romande/.nvm/versions/node/v16.14.0/bin/node)                 │
  │ Specs:          2 found (about.spec.ts, index.spec.ts)                                         │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────

  Running:  about.spec.ts                                                                   (1 of 2)


  app
  code-coverage parsed sent coverage +0ms
  code-coverage wrote coverage file /Users/romande/projects/next-ts-with-cypress-coverage/.nyc_output/out.json +7ms
  code-coverage parsed sent coverage +174ms
  code-coverage wrote coverage file /Users/romande/projects/next-ts-with-cypress-coverage/.nyc_output/out.json +3ms
  code-coverage NYC file /Users/romande/projects/next-ts-with-cypress-coverage/.nyc_output/out.json has 3 key(s) +6s
  code-coverage 1 key /Users/romande/projects/next-ts-with-cypress-coverage/pages/_app.tsx file path /Users/romande/projects/next-ts-with-cypress-coverage/pages/_app.tsx +0ms
  code-coverage 2 key /Users/romande/projects/next-ts-with-cypress-coverage/pages/about.tsx file path /Users/romande/projects/next-ts-with-cypress-coverage/pages/about.tsx +0ms
  code-coverage 3 key /Users/romande/projects/next-ts-with-cypress-coverage/pages/index.tsx file path /Users/romande/projects/next-ts-with-cypress-coverage/pages/index.tsx +0ms
  code-coverage in file /Users/romande/projects/next-ts-with-cypress-coverage/.nyc_output/out.json all files are not found? false +1ms
  code-coverage NYC file /Users/romande/projects/next-ts-with-cypress-coverage/.nyc_output/out.json has 3 key(s) +0ms
  code-coverage nyc needs to report on all included files +18ms
  code-coverage include all files options: { all: true, include: [ 'pages/**/*', 'no-coverage-here/**/*' ], exclude: [ 'pages/api/__coverage__.js' ], extension: [ '.js', '.jsx', '.ts', '.tsx' ] } +1ms
  code-coverage searching files to include using patterns [ 'pages/**/*', 'no-coverage-here/**/*', '!pages/api/__coverage__.js', '!**/node_modules/**' ] +0ms
  code-coverage found 4 file(s) +10ms
/Users/romande/projects/next-ts-with-cypress-coverage/pages/_app.tsx
/Users/romande/projects/next-ts-with-cypress-coverage/pages/about.tsx
/Users/romande/projects/next-ts-with-cypress-coverage/pages/index.tsx
/Users/romande/projects/next-ts-with-cypress-coverage/no-coverage-here/some-file.ts
  code-coverage coverage has 3 record(s) +9ms
coverage has the following first paths
/Users/romande/projects/next-ts-with-cypress-coverage/pages/_app.tsx
/Users/romande/projects/next-ts-with-cypress-coverage/pages/about.tsx
/Users/romande/projects/next-ts-with-cypress-coverage/pages/index.tsx
  code-coverage adding empty coverage for file /Users/romande/projects/next-ts-with-cypress-coverage/no-coverage-here/some-file.ts +0ms
  code-coverage includeAllFiles saving updated file /Users/romande/projects/next-ts-with-cypress-coverage/.nyc_output/out.json +0ms
  code-coverage there are 4 keys in the file +0ms
  code-coverage calling NYC reporter with options { 'report-dir': '/Users/romande/projects/next-ts-with-cypress-coverage/cypress-coverage', reporter: [ 'json', 'lcov' ], extension: [ '.js', '.jsx', '.ts', '.tsx' ], excludeAfterRemap: false, extends: './.nycrc.json', include: [ 'pages/**/*', 'no-coverage-here/**/*' ], exclude: [ 'pages/api/__coverage__.js' ], all: true, sourceMap: true, instrument: true, 'temp-dir': '/Users/romande/projects/next-ts-with-cypress-coverage/.nyc_output', tempDir: '/Users/romande/projects/next-ts-with-cypress-coverage/.nyc_output', reportDir: '/Users/romande/projects/next-ts-with-cypress-coverage/cypress-coverage' } +22ms
  code-coverage current working directory is /Users/romande/projects/next-ts-with-cypress-coverage +1ms
  code-coverage after reporting, returning the report folder name /Users/romande/projects/next-ts-with-cypress-coverage/cypress-coverage +988ms
  code-coverage Final coverage in /Users/romande/projects/next-ts-with-cypress-coverage/cypress-coverage/coverage-final.json +0ms
  code-coverage There are 4 key(s) in /Users/romande/projects/next-ts-with-cypress-coverage/cypress-coverage/coverage-final.json +1ms
  code-coverage ❓ /Users/romande/projects/next-ts-with-cypress-coverage/no-coverage-here/some-file.ts statements covered 0/0 +0ms
  code-coverage ✅ /Users/romande/projects/next-ts-with-cypress-coverage/pages/_app.tsx statements covered 1/1 +0ms
  code-coverage ✅ /Users/romande/projects/next-ts-with-cypress-coverage/pages/about.tsx statements covered 8/8 +0ms
  code-coverage ✅ /Users/romande/projects/next-ts-with-cypress-coverage/pag    ✓ redirects to / (2070ms)


  1 passing (3s)

es/index.tsx statements covered 3/3 +0ms

  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        1                                                                                │
  │ Passing:      1                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     3 seconds                                                                        │
  │ Spec Ran:     about.spec.ts                                                                    │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


  (Video)

  -  Started processing:  Compressing to 32 CRF
  -  Finished processing: /Users/romande/projects/next-ts-with-cypress-coverage/cypre    (0 seconds)
                          ss/videos/about.spec.ts.mp4


────────────────────────────────────────────────────────────────────────────────────────────────────

  Running:  index.spec.ts                                                                   (2 of 2)


  landing
  code-coverage parsed sent coverage +4s
  code-coverage wrote coverage file /Users/romande/projects/next-ts-with-cypress-coverage/.nyc_output/out.json +2ms
  code-coverage parsed sent coverage +31ms
  code-coverage wrote coverage file /Users/romande/projects/next-ts-with-cypress-coverage/.nyc_output/out.json +1ms
  code-coverage NYC file /Users/romande/projects/next-ts-with-cypress-coverage/.nyc_output/out.json has 3 key(s) +5s
  code-coverage 1 key /Users/romande/projects/next-ts-with-cypress-coverage/pages/_app.tsx file path /Users/romande/projects/next-ts-with-cypress-coverage/pages/_app.tsx +0ms
  code-coverage 2 key /Users/romande/projects/next-ts-with-cypress-coverage/pages/about.tsx file path /Users/romande/projects/next-ts-with-cypress-coverage/pages/about.tsx +0ms
  code-coverage 3 key /Users/romande/projects/next-ts-with-cypress-coverage/pages/index.tsx file path /Users/romande/projects/next-ts-with-cypress-coverage/pages/index.tsx +0ms
  code-coverage in file /Users/romande/projects/next-ts-with-cypress-coverage/.nyc_output/out.json all files are not found? false +1ms
  code-coverage NYC file /Users/romande/projects/next-ts-with-cypress-coverage/.nyc_output/out.json has 3 key(s) +0ms
  code-coverage nyc needs to report on all included files +17ms
  code-coverage include all files options: { all: true, include: [ 'pages/**/*', 'no-coverage-here/**/*' ], exclude: [ 'pages/api/__coverage__.js' ], extension: [ '.js', '.jsx', '.ts', '.tsx' ] } +0ms
  code-coverage searching files to include using patterns [ 'pages/**/*', 'no-coverage-here/**/*', '!pages/api/__coverage__.js', '!**/node_modules/**' ] +1ms
  code-coverage found 4 file(s) +18ms
/Users/romande/projects/next-ts-with-cypress-coverage/pages/_app.tsx
/Users/romande/projects/next-ts-with-cypress-coverage/pages/about.tsx
/Users/romande/projects/next-ts-with-cypress-coverage/pages/index.tsx
/Users/romande/projects/next-ts-with-cypress-coverage/no-coverage-here/some-file.ts
  code-coverage coverage has 3 record(s) +1ms
coverage has the following first paths
/Users/romande/projects/next-ts-with-cypress-coverage/pages/_app.tsx
/Users/romande/projects/next-ts-with-cypress-coverage/pages/about.tsx
/Users/romande/projects/next-ts-with-cypress-coverage/pages/index.tsx
  code-coverage adding empty coverage for file /Users/romande/projects/next-ts-with-cypress-coverage/no-coverage-here/some-file.ts +0ms
  code-coverage includeAllFiles saving updated file /Users/romande/projects/next-t    ✓ should render (753ms)


  1 passing (888ms)

s-with-cypress-coverage/.nyc_output/out.json +0ms
  code-coverage there are 4 keys in the file +0ms
  code-coverage calling NYC reporter with options { 'report-dir': '/Users/romande/projects/next-ts-with-cypress-coverage/cypress-coverage', reporter: [ 'json', 'lcov' ], extension: [ '.js', '.jsx', '.ts', '.tsx' ], excludeAfterRemap: false, extends: './.nycrc.json', include: [ 'pages/**/*', 'no-coverage-here/**/*' ], exclude: [ 'pages/api/__coverage__.js' ], all: true, sourceMap: true, instrument: true, 'temp-dir': '/Users/romande/projects/next-ts-with-cypress-coverage/.nyc_output', tempDir: '/Users/romande/projects/next-ts-with-cypress-coverage/.nyc_output', reportDir: '/Users/romande/projects/next-ts-with-cypress-coverage/cypress-coverage' } +20ms
  code-coverage current working directory is /Users/romande/projects/next-ts-with-cypress-coverage +0ms
  code-coverage after reporting, returning the report folder name /Users/romande/projects/next-ts-with-cypress-coverage/cypress-coverage +19ms
  code-coverage Final coverage in /Users/romande/projects/next-ts-with-cypress-coverage/cypress-coverage/coverage-final.json +0ms
  code-coverage There are 4 key(s) in /Users/romande/projects/next-ts-with-cypress-coverage/cypress-coverage/coverage-final.json +0ms
  code-coverage ❓ /Users/romande/projects/next-ts-with-cypress-coverage/no-coverage-here/some-file.ts statements covered 0/0 +0ms
  code-coverage ✅ /Users/romande/projects/next-ts-with-cypress-coverage/pages/_app.tsx statements covered 1/1 +0ms
  code-coverage ✅ /Users/romande/projects/next-ts-with-cypress-coverage/pages/about.tsx statements covered 8/8 +0ms
  code-coverage ✅ /Users/romande/projects/next-ts-with-cypress-coverage/pages/index.tsx statements covered 3/3 +0ms

  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        1                                                                                │
  │ Passing:      1                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     0 seconds                                                                        │
  │ Spec Ran:     index.spec.ts                                                                    │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


  (Video)

  -  Started processing:  Compressing to 32 CRF
  -  Finished processing: /Users/romande/projects/next-ts-with-cypress-coverage/cypre    (0 seconds)
                          ss/videos/index.spec.ts.mp4


====================================================================================================

  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  about.spec.ts                            00:03        1        1        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  index.spec.ts                            885ms        1        1        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:04        2        2        -        -        -

Versions

• What is this plugin's version? 3.9.12
• What is Cypress version? 9.6.0
• What is your operating system? Mac OS
• What is the shell? zsh
• What is the Node version? v16.14.0
• What is the NPM version? v8.5.0
• How do you instrument your application? istanbul via the babel-plugin, see the example repo
• When running tests, if you open the web application in regular browser, and open DevTools, do you see window.__coverage__ object? yes
• Is there .nyc_output folder? Is there .nyc_output/out.json file. Is it empty? Can you paste at least part of it so we can see the keys and file paths? yes, it just doesn't cover certainly included files
• Do you run Cypress tests in a Docker container? no

Describe the bug

Coverage is being measured correctly, but ONLY for the pages folder. Anything outside the pages folder (e.g. no-coverage-here) always remains uncovered, despite being called.

Link to the repo
https://github.com/romandecker/next-ts-with-cypress-coverage

Originally found this issue here: kylemh/next-ts-with-cypress-coverage#3

[jpierson-at-riis]

Were running into this issue as well with a project which uses SSG and CSR. Any code that is run as part of the build to generate the pages is not producing coverage information. This is also true when using the dev server as illustrated in the example by @romandecker which directly impacts our ability to gather coverage for our project through our current build pipeline.

Thinking this through, server side code runs at build time with SSG or at runtime with SSR would need to generate coverage data and either make it available in a separate file output or it would need to be tacked into the client side coverage data as soon as the page loads. Has there been any thoughts on which of these approaches would be appropriate or even feasible? Getting this to work for the dev server would be useful too and happens to be the way that we are collecting coverage data currently in our build pipeline through CypressJS.

My only workaround that I can think of for now is to disable coverage for all code that runs through SSG or SSR but identifying all of this code is not as simple as looking at a simple file name pattern or location of the code because of how code organized in a Next.js project with server side and client side code comingled and somewhat dynamically determined depending on the specific runtime scenario (ex. dev server vs. production build using export).

We're currently generated coverage using the swc-plugin-coverage-instrument plugin instead of babel in order to keep our test builds closer to our production builds.

[jwickens]

Is this perhaps a nextjs limitation? I opened a feature request for a integration between nyc and nextjs here: vercel/next.js#70713