Vendor the antlr4 runtime library #487

pelson · 2024-09-27T07:09:20Z

Vendors the antlr4 runtime. The copyright and license information (BSD) is included in the header of each of the vendored files (because that is how the code is shipped by antlr4-python3-runtime)
Drops the latex extra - it is no longer necessary/used to have the antlr4 runtime library installed (since we vendored it)
Re-formats all of the generated code - this makes life easier to dig into the code, and also means that there is less likely to be diffs when upstream code becomes more standardised wrt. formatting
Makes generated code not shown by default in a PR.

pyproject.toml

pelson · 2024-09-27T07:34:41Z

cf_units/tests/integration/parse/test_graph.py

@@ -3,9 +3,6 @@
 # This file is part of cf-units and is released under the BSD license.
 # See LICENSE in the root of the repository for full licensing details.
 # ruff: noqa: E402
-import pytest
-
-antlr4 = pytest.importorskip("antlr4")


Reverts the change from #423. cc @ocefpaf - the idea here is to vendor antlr4 runtime so that we don't have to worry about it as a dependency.

If that was on the table I would not do #423. This is better. Thanks!

pelson · 2024-09-27T07:36:58Z

Sadly, the simplest vendoring approach of using absolute import names fails...

/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py:126: in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
  ../venv-test-x86_64/lib/python3.10/site-packages/cf_units/tests/integration/parse/test_parse.py:11: in <module>
      from cf_units._udunits2_parser import normalize
  ../venv-test-x86_64/lib/python3.10/site-packages/cf_units/_udunits2_parser/__init__.py:8: in <module>
      from cf_units._udunits2_parser.parser._antlr4_runtime import (
  ../venv-test-x86_64/lib/python3.10/site-packages/cf_units/_udunits2_parser/parser/_antlr4_runtime/__init__.py:8: in <module>
      from cf_units._udunits2_parser.parser._antlr4_runtime.atn.ParserATNSimulator import (
  ../venv-test-x86_64/lib/python3.10/site-packages/cf_units/_udunits2_parser/parser/_antlr4_runtime/atn/ParserATNSimulator.py:236: in <module>
      from cf_units._udunits2_parser.parser._antlr4_runtime import DFA
  E   ImportError: cannot import name 'DFA' from partially initialized module 'cf_units._udunits2_parser.parser._antlr4_runtime' (most likely due to a circular import)

Looks like I will have to do a bit more work to do relative imports.

pelson · 2024-10-01T04:39:37Z

The codacy test is failing. I don't know how to configure the tool, and can't run it locally. I don't see any docs on this topic other than https://docs.codacy.com/repositories-configure/codacy-configuration-file/. I also don't have an option to configure the project in the codacy website (screenshot:

).

Would you have any suggestions on how best to proceed @trexfeathers?

trexfeathers · 2024-10-02T10:20:10Z

Would you have any suggestions on how best to proceed @trexfeathers?

In short: no. I'm afraid I don't know anything about Codacy, other than the fact that it occasionally produces confusing objections. I hope I'm not picking names out of a hat: @bjlittle / @lbdreyer can either of you speak to why cf-units uses Codacy where most of our other repos use Codecov?

bjlittle · 2024-10-02T12:21:01Z

We dabbled with Codacy adoption, but for me it's out of favour now.

Personally, I think we should drop it's usage as codecov and ruff easily replace it along with repo-review 👍

lbdreyer · 2024-10-02T12:22:24Z

In short: no. I'm afraid I don't know anything about Codacy, other than the fact that it occasionally produces confusing objections. I hope I'm not picking names out of a hat: @bjlittle / @lbdreyer can either of you speak to why cf-units uses Codacy where most of our other repos use Codecov?

I'm afraid I don't know. I remember briefly talking about codacy a couple of years ago. We thought it may be nice as it had some extra functionality over other coding standards/coverage/checking tools. I believe I gave it access to the SciTools org, but I don't remember enabling it for cf-units. Perhaps that was someone else?
I remember deciding codacy wasn't suitable, but I can't remember why now!

trexfeathers · 2024-10-02T12:38:49Z

We dabbled with Codacy adoption, but for me it's out of favour now.

Personally, I think we should drop it's usage as codecov and ruff easily replace it along with repo-review 👍

OK sounds good. @pp-mo as of Monday you are the 'developer in charge' here; are you happy with us switching to Codecov? I'm fairly confident I can see how to make it happen.

bjlittle · 2024-10-02T13:06:48Z

codacy has now been unconfigured from cf-units (and scitools, as cf-units was the only repo using it)

trexfeathers · 2024-10-02T14:17:18Z

codacy has now been unconfigured from cf-units (and scitools, as cf-units was the only repo using it)

#497

pelson · 2024-10-03T03:35:21Z

Requesting a review of this before a release is considered. cc @pp-mo as I think #423 took us in the wrong direction.

pelson · 2024-10-03T03:36:56Z

Also: big thanks for updating/removing the codacy configuration - it was a bit uncomfortable that I had no idea what was going on, how to configure it, nor how to run it locally. Much appreciated that it got straightened out! 👍

rcomer · 2024-10-03T09:55:42Z

I don't think I'm qualified to review this, but I think it would be good to update the README so future devs know how to look after this. In particular, the section I added last year about updating antlr would no longer be correct I think.

…ular dependencies

…amed

pelson · 2024-10-04T03:52:59Z

Thanks for pointing it out. I've rebased and added the comment to the README.

pp-mo · 2024-10-16T11:28:30Z

Despite the apparent silence, I'm keen to include this in a v3.3 release.

We now think will be within 1-2 weeks to provide full working with numpy2 for Iris 3.11
-- that is by 2024-10-28

I did have some suggested addition to the readme, but otherwise about ready to merge this.
Sorry for the delay, I'll post that soon.

I'm assuming that the work in https://github.com/pelson/pyudunits2/ has not affected your enthusiasm for this approach @pelson ?

pelson

I'm assuming that the work in https://github.com/pelson/pyudunits2/ has not affected your enthusiasm for this approach @pelson ?

No, I still think this is the right thing to do here (I lifted it verbatim in pyudunits2).

I wrote a few comments for which I forgot to hit "submit review", so include them when commenting now (they may or may not be useful...)

pelson · 2024-10-03T02:55:17Z

cf_units/_udunits2_parser/compile.py

@@ -14,10 +14,13 @@
 You're welcome ;).



The changes in this file are relevant to the review (pointing this out because the diff is large due to the changes to generated files).

pelson · 2024-10-03T02:56:13Z

cf_units/_udunits2_parser/compile.py

+
+    # Straighten out some wonky imports.
+    if antlr_file_path.name == "XPathLexer.py":
+        contents = contents.replace(


I'm working on upstreaming these, but think they are fine here for now given we have no need to update the antlr version any time soon (antlr/antlr4#4705 etc.)

pelson commented Sep 27, 2024

View reviewed changes

pyproject.toml Show resolved Hide resolved

pelson commented Sep 27, 2024

View reviewed changes

pelson force-pushed the feature/compile-update branch 3 times, most recently from c1d360b to 350bca5 Compare September 29, 2024 06:04

pelson force-pushed the feature/compile-update branch 2 times, most recently from 95de3af to b0b38bc Compare October 1, 2024 07:28

pelson force-pushed the feature/compile-update branch from b0b38bc to f1de9d2 Compare October 3, 2024 02:52

pp-mo mentioned this pull request Oct 3, 2024

Make antlr optional #423

Merged

pelson added 7 commits October 4, 2024 05:52

Format the generated code at generation time

5084fe1

Code to vendor the antlr runtime

ebb3565

Vendor the antlr4 runtime library

3a89502

Remove the path from the generated code

373142e

Move the antlr runtime outside of the parser directory, to avoid circ…

7a2d243

…ular dependencies

Use relative imports within the antlr4 runtime. This should be upstre…

b9413c9

…amed

Update the readme to explain more about ANTLR4 and the vendoring process

0a0fc02

pelson force-pushed the feature/compile-update branch from 2148654 to 0a0fc02 Compare October 4, 2024 03:52

pelson commented Oct 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Vendor the antlr4 runtime library #487

Vendor the antlr4 runtime library #487

pelson commented Sep 27, 2024

pelson Sep 27, 2024

ocefpaf Sep 27, 2024

pelson commented Sep 27, 2024

pelson commented Oct 1, 2024

trexfeathers commented Oct 2, 2024

bjlittle commented Oct 2, 2024

lbdreyer commented Oct 2, 2024

trexfeathers commented Oct 2, 2024

bjlittle commented Oct 2, 2024

trexfeathers commented Oct 2, 2024

pelson commented Oct 3, 2024

pelson commented Oct 3, 2024

rcomer commented Oct 3, 2024

pelson commented Oct 4, 2024

pp-mo commented Oct 16, 2024

pelson left a comment

pelson Oct 3, 2024

pelson Oct 3, 2024

Vendor the antlr4 runtime library #487

Are you sure you want to change the base?

Vendor the antlr4 runtime library #487

Conversation

pelson commented Sep 27, 2024

pelson Sep 27, 2024

Choose a reason for hiding this comment

ocefpaf Sep 27, 2024

Choose a reason for hiding this comment

pelson commented Sep 27, 2024

pelson commented Oct 1, 2024

trexfeathers commented Oct 2, 2024

bjlittle commented Oct 2, 2024

lbdreyer commented Oct 2, 2024

trexfeathers commented Oct 2, 2024

bjlittle commented Oct 2, 2024

trexfeathers commented Oct 2, 2024

pelson commented Oct 3, 2024

pelson commented Oct 3, 2024

rcomer commented Oct 3, 2024

pelson commented Oct 4, 2024

pp-mo commented Oct 16, 2024

pelson left a comment

Choose a reason for hiding this comment

pelson Oct 3, 2024

Choose a reason for hiding this comment

pelson Oct 3, 2024

Choose a reason for hiding this comment