Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AVRO-3832: [Python] Make Python test work with Docker #2444

Merged
merged 2 commits into from
Aug 18, 2023

Conversation

sarutak
Copy link
Member

@sarutak sarutak commented Aug 15, 2023

AVRO-3832

What is the purpose of the change

This PR fixes an issue that Python test doesn't work successfully with ./build.sh docker-test due to some reasons.

Reason1:

/bin/sh: 1: BUILDARCH: parameter not set or null

Reason2:

py37: install_package> python -I -m pip install --force-reinstall --no-deps /avro/lang/py/.tox/.tmp/package/258/avro-1.12.0+snapshot.tar.gz
py37: exit 1 (0.27 seconds) /avro/lang/py> python -I -m pip install --force-reinstall --no-deps /avro/lang/py/.tox/.tmp/package/258/avro-1.12.0+snapshot.tar.gz pid=243
py37: FAIL ✖ in 0.34 seconds

Reason3:

py311: install_deps> python -I -m pip install coverage python-snappy zstandard
Collecting coverage
  Obtaining dependency information for coverage from https://files.pythonhosted.org/packages/55/63/f2dcc8f7f1587ae54bf8cc1c3b08e07e442633a953537dfaf658a0cbac2c/coverage-7.3.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Downloading coverage-7.3.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (8.1 kB)
Collecting python-snappy
  Downloading python-snappy-0.6.1.tar.gz (24 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting zstandard
  Downloading zstandard-0.21.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.7/2.7 MB 9.4 MB/s eta 0:00:00
Downloading coverage-7.3.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (232 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 232.9/232.9 kB 12.0 MB/s eta 0:00:00
Building wheels for collected packages: python-snappy
  Building wheel for python-snappy (setup.py): started
  Building wheel for python-snappy (setup.py): finished with status 'error'
  Running setup.py clean for python-snappy
Failed to build python-snappy
py311: exit 1 (2.64 seconds) /avro/lang/py> python -I -m pip install coverage python-snappy zstandard pid=3179
py311: FAIL ✖ in 2.94 seconds

Reason4:

pypy3.9: commands_pre[2]> coverage run -pm avro.test.gen_interop_data avro/interop.avsc avro/test/interop/data/py.avro
pypy3.9: commands_pre[3]> cp -r avro/test/interop/data /avro/lang/py/../../build/interop
pypy3.9: commands[0]> coverage run -pm unittest discover --buffer --failfast
/avro/lang/py/avro/schema.py:1233: IgnoredLogicalType: Unknown unknown-logical-type, using string.
  warnings.warn(avro.errors.IgnoredLogicalType(f"Unknown {logical_type}, using {type_}."))
/avro/lang/py/avro/schema.py:1229: IgnoredLogicalType: Logical type timestamp-millis requires literal type long, not string.
  warnings.warn(
/avro/lang/py/avro/schema.py:1233: IgnoredLogicalType: Unknown unknown-logical-type, using string.
  warnings.warn(avro.errors.IgnoredLogicalType(f"Unknown {logical_type}, using {type_}."))
/avro/lang/py/avro/schema.py:1229: IgnoredLogicalType: Logical type timestamp-millis requires literal type long, not string.
  warnings.warn(
..............F
======================================================================
FAIL: test_interop (avro.test.test_datafile_interop.TestDataFileInterop)
Test Interop
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/avro/lang/py/avro/test/test_datafile_interop.py", line 38, in test_interop
    self.assertGreater(os.stat(filename).st_size, 0)
AssertionError: 0 not greater than 0

Verifying this change

Confirmed Python test passed with ./build.sh docker-test

  build: OK (3.17 seconds)
  docs: OK (6.36=setup[2.93]+cmd[3.43] seconds)
  typechecks: OK (4.04=setup[2.89]+cmd[0.00,0.00,0.13,0.00,0.49,0.10,0.43] seconds)
  py37: OK (19.67=setup[5.74]+cmd[0.00,0.00,0.13,0.00,13.24,0.09,0.45] seconds)
  py38: OK (15.87=setup[2.76]+cmd[0.00,0.00,0.13,0.00,12.42,0.08,0.46] seconds)
  py39: OK (15.58=setup[2.82]+cmd[0.00,0.00,0.13,0.00,12.05,0.10,0.47] seconds)
  py310: OK (15.41=setup[2.91]+cmd[0.00,0.00,0.14,0.00,11.89,0.09,0.38] seconds)
  py311: OK (18.49=setup[5.62]+cmd[0.00,0.00,0.14,0.00,12.20,0.09,0.43] seconds)
  pypy3.7: SKIP (3.12 seconds)
  pypy3.8: OK (44.44=setup[7.62]+cmd[0.01,0.01,0.91,0.00,34.72,0.26,0.92] seconds)
  pypy3.9: OK (40.33=setup[7.42]+cmd[0.00,0.00,0.71,0.00,31.13,0.22,0.84] seconds)
  pypy3.10: OK (39.95=setup[6.97]+cmd[0.00,0.00,0.68,0.00,31.23,0.24,0.82] seconds)
  congratulations :) (226.57 seconds)

Documentation

No new features added.

@sarutak sarutak changed the title AVRO-3832: [Python] Make Python test works with Docker AVRO-3832: [Python] Make Python test work with Docker Aug 15, 2023
@@ -73,6 +73,7 @@ def generate(schema_file: TextIO, output_path: IO) -> None:
continue
base, ext = os.path.splitext(output_path.name)
Path(f"{base}_{codec}{ext}").write_bytes(data)
output_path.close()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be in finally or with closing()

https://docs.python.org/3/library/contextlib.html#contextlib.closing

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry. I've fixed it.

@kojiromike
Copy link
Contributor

Thanks for doing this work. If you are happy making the slight change I mentioned then I think this looks pretty good.

@kojiromike kojiromike merged commit 1711661 into apache:master Aug 18, 2023
22 checks passed
RanbirK pushed a commit to RanbirK/avro that referenced this pull request May 13, 2024
* AVRO-3832: [Python] Make Python test work with Docker.

* Use contextlib.closing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants