Improve first-time testing (& coverage) experience

[joaomoreno]

I asked an LLM to give me an example Python unit test suite. I got this:

# calculator.py

def add(x, y):
    return x + y

def subtract(x, y):
    return x - y

def multiply(x, y):
    return x * y

def divide(x, y):
    if y == 0:
        raise ValueError("Cannot divide by zero")
    return x / y

# test_calculator.py

import unittest
from calculator import add, subtract, multiply, divide

class TestCalculator(unittest.TestCase):

    def test_add(self):
        self.assertEqual(add(1, 2), 3)
        self.assertEqual(add(-1, 1), 0)
        self.assertEqual(add(-1, -1), -2)

    def test_subtract(self):
        self.assertEqual(subtract(10, 5), 5)
        self.assertEqual(subtract(-1, 1), -2)
        self.assertEqual(subtract(-1, -1), 0)

    def test_multiply(self):
        self.assertEqual(multiply(4, 5), 20)
        self.assertEqual(multiply(-1, 1), -1)
        self.assertEqual(multiply(-1, -1), 1)

    def test_divide(self):
        self.assertEqual(divide(8, 4), 2)
        self.assertEqual(divide(-1, 1), -1)
        self.assertEqual(divide(-1, -1), 1)
        with self.assertRaises(ValueError):
            divide(1, 0)

if __name__ == '__main__':
    unittest.main()

I created these two files. I installed the Python extension. The Testing view appeared. But now the troubles start:

---

🐛 The editor is unable to detect that I use unittest, so it asks me to configure stuff. The configuration is weird since it asks me for the root of the tests and for the right test glob to use (why aren't we just picking good defaults here, instead of forcing the user to make a decision?):

---

🐛 After configuring the tests I am immediately greeted with an error. The error is also weirdly shown as a list row (cc @connor4312).

---

🐛 After reloading I can now run the tests. I then run Run Tests with Coverage and immediately the next error comes up. Should the extension automatically take care of installing this for me?

[joaomoreno]

Actually pip install coverage didn't solve the last error for me. I don't even know how to proceed.

[connor4312]

The error is also weirdly shown as a list row

This is intentional so that test providers can show 'per-item' information, e.g. if an AST-parsing test extension is unable to parse a single file out of the workspace, it can show that.

[karthiknadig]

We are working on improving this flow. But, before we do the test part, we actually need to "fix" install flow. See in python when you run pip install <package> you might be installing the wrong thing in the wrong place. Wrong thing, because install names in python can be different from package names. Wrong place because (here is Brett's detailed explanation https://snarky.ca/why-you-should-use-python-m-pip/). We need installation in many areas, like notebook, terminal quick fix, make it easy for copilot to install things if needed, also need to handle install permissions. @eleanorjboyd will be looking into this part of it.

You are right about making it easier to do test config. We are going towards a no-config solution for everything python. We just shipped no-config debugging for python (since the area covered by that is the largest) and works with/without AI. we plan on doing no-config for testing as well. What we have now is the legacy support for this, and I agree that it is cumbersome. Eleanors work will feed into the no-config solution, also into many other places, eliminating the wrong thing in the wrong place problem.