Snapshot manager

[leowrites]

Proposed Changes

• Add a context manager python_ta.debug.SnapshotManager

Type of Change

(Write an X or a brief description next to the type or types that best describe your changes.)

Type	Applies?
🚨 Breaking change (fix or feature that would cause existing functionality to change)
✨ New feature (non-breaking change that adds functionality)	x
🐛 Bug fix (non-breaking change that fixes an issue)
♻️ Refactoring (internal change to codebase, without changing functionality)
🚦 Test update (change that only adds or modifies tests)	x
📚 Documentation update (change that only updates documentation)	x
📦 Dependency update (change that updates a dependency)
🔧 Internal (change that only affects developers or continuous integration)

Checklist

(Complete each of the following items for your pull request. Indicate that you have completed an item by changing the [ ] into a [x] in the raw text, or by clicking on the checkbox in the rendered description on GitHub.)

Before opening your pull request:

• I have performed a self-review of my changes.
  • Check that all changed files included in this pull request are intentional changes.
  • Check that all changes are relevant to the purpose of this pull request, as described above.
• I have added tests for my changes, if applicable.
  • This is required for all bug fixes and new features.
• I have updated the project documentation, if applicable.
  • This is required for new features.
• I have updated the project Changelog (this is required for all changes).
• If this is my first contribution, I have added myself to the list of contributors.

After opening your pull request:

• I have verified that the pre-commit.ci checks have passed.
• I have verified that the CI tests have passed.
• I have reviewed the test coverage changes reported by Coveralls.
• I have requested a review from a project maintainer.

Questions and Comments

(Include any questions or comments you have regarding your changes.)

[coveralls]

Pull Request Test Coverage Report for Build 11110823253

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 92 unchanged lines in 9 files lost coverage.
• Overall coverage increased (+0.001%) to 92.051%

---

Files with Coverage Reduction	New Missed Lines	%
python_ta/checkers/possibly_undefined_checker.py	1	98.89%
python_ta/checkers/unnecessary_indexing_checker.py	1	98.36%
python_ta/reporters/core.py	2	98.52%
python_ta/cfg/cfg_generator.py	4	92.86%
python_ta/z3/z3_parser.py	8	95.06%
python_ta/init.py	9	92.09%
python_ta/checkers/invalid_name_checker.py	12	93.19%
python_ta/contracts/main.py	24	0.0%
python_ta/contracts/init.py	31	90.09%

Totals
Change from base Build 11018465322:	0.001%
Covered Lines:	3057
Relevant Lines:	3321

---

💛 - Coveralls

[david-yz-liu]

@leowrites good work. In this first review, I focused on the API of SnapshotManager as well as some high-level comments to improve the test file. Make sure to also update the documentation accordingly, which I'll review more closely after you've addressed these comments.

[david-yz-liu]

pytest provides a fixture for creating a temporary directory, use that instead of defining your own here

[leowrites]

I defined a directory here because the output directory is captured in the snapshots, so we want the directories to be consistent across each test run.

I looked into using tmp_path and tmp_path_factory but I can't seem to get the same base output directory for each run (an example directory I got is /private/var/folders/rt/x1pynv8j1fb1hkbhs6y9v62m0000gn/T/pytest-of-leoliu/pytest-0). The only way I found to ensure the base directory is the same is to set the --basetemp. Should I rely on this flag to ensure the directories are consistent?

[david-yz-liu]

@leowrites that's a great question. This by itself suggests a limitation with the tests, because (of course) you mean to only test the part inside the with block, and not this output_directory parameter.

I think you should be able to change your function structure by nesting the "function input" definition within the test case itself, making the output_directory variable non-local to the function. Something like:

def test_func_one_line(tmp_dir)
    output_directory = ... # tmp_dir's path
    
    def func_one_line():
        with SnapshotTracer(...):
            num = 123

    func_one_line()

    assert_output_files_match(..., ...)

This loses the nice use of pytest.parameterize that you have, but I think overall the code complexity stays almost the same because you move the function input definitions into the test case code.

[leowrites]

@david-yz-liu By nesting the functions, the output_directory is still captured in f_locals inside func_one_line. I think this is because f_locals also includes free variables in the parent scope when they are referenced. One solution I found is using a global variable so that the variable won't show up in f_locals, but it feels unconventional and could break if we were to run the tests concurrently

[david-yz-liu]

@leowrites wow, yeah okay. I'm not sure how I feel about free variables appearing the stack frame, but I get why it's happening.

All that said, it's not good to use an absolute path (can't assume specific permissions) and it's almost certainly going to look different on Windows.

Why don't we take a pause on this and introduce a different approach. One snapshot feature that's on our roadmap is to allow the user to exclude certain variables (again, as an iterable string/regex). It's a different kind of filtering, I suppose, now at the individual frame level. If you implement that, you'll be able to use it here to exclude output_directory from the memory model diagram