Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#49 Added export_slc_async fixture #50

Merged
merged 26 commits into from
Aug 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
fd44ea1
#49 Added export_slc_async fixture
ahsimb Aug 23, 2024
81bef2e
#49 Added export_slc_async fixture
ahsimb Aug 23, 2024
ca1c52d
#49 Added export_slc_async fixture
ahsimb Aug 23, 2024
19d9646
#49 Made a proper integration test
ahsimb Aug 23, 2024
eb9132e
#49 Trying to fix the integration test
ahsimb Aug 23, 2024
f34544a
#49 Added debugging messages to the integration test
ahsimb Aug 23, 2024
6f66074
#49 Reduced the test to minimum
ahsimb Aug 23, 2024
5222645
#49 Added the stuff back but disabled the export
ahsimb Aug 23, 2024
9f08687
#49 Added the stuff back but disabled the export
ahsimb Aug 27, 2024
94d963b
#49 Disabled export_slc_async
ahsimb Aug 27, 2024
d78b4d8
#49 Disabled upload_slc
ahsimb Aug 27, 2024
90ee5a7
#49 Disabled prepare_flavor
ahsimb Aug 27, 2024
119cad5
#49 Checking if find_path_backwards works
ahsimb Aug 27, 2024
b0dab6b
#49 Tried a workaround for the find_path_backwards bug
ahsimb Aug 27, 2024
b5a08d6
#49 Re-enabled the test, but only for ITDE
ahsimb Aug 27, 2024
b6235ba
#49 Re-enabled the test, but only for ITDE
ahsimb Aug 27, 2024
5d24cd5
#49 Re-enabled the test, but only for ITDE
ahsimb Aug 27, 2024
6292e49
#49 Re-enabled the test, but only for ITDE
ahsimb Aug 27, 2024
c26dc80
#49 Prepared for the release
ahsimb Aug 27, 2024
2a39190
#49 Fixed the issue with multiple wait.
ahsimb Aug 27, 2024
cb10b2d
#49 Fixed the issue with multiple wait.
ahsimb Aug 27, 2024
d0c274c
#49 Prepared for the release again.
ahsimb Aug 27, 2024
fc4b301
#49 Experiment with overriding a fixture
ahsimb Aug 28, 2024
54356e2
#49 Experiment with overriding a fixture
ahsimb Aug 28, 2024
54853b3
#49 Experiment with overriding a fixture
ahsimb Aug 28, 2024
1993d06
#49 Re-enabled everything again
ahsimb Aug 28, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 15 additions & 10 deletions pytest-slc/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# pytest-exasol-slc Plugin

The `pytest-exasol-slc` plugin provides a pytest fixture for uploading a script language container
into the database. The fixture is backend agnostic. It runs for the selected backends
The `pytest-exasol-slc` plugin provides a pytest fixtures for building and uploading a script language container
into the database. The fixtures are backend agnostic. They run for the selected backends
(see the documentation for the `pytest-exasol-backend` plugin).

## Installation
Expand All @@ -14,15 +14,20 @@ pip install pytest-exasol-slc

## Usage in Tests

Below is an example of a test that requires a script language container to be uploaded into the database.
Below is an example of a test that requires a script language container to be built and uploaded into the database.
Note, that by default this test will run twice - once for each backend.

```python
import pyexasol

def test_something_with_slc(upload_slc, container_file_path, language_alias):

upload_slc(container_file_path=container_file_path, language_alias=language_alias)

# Now run the actual test
import pytest

@pytest.fixture(scope='session')
def slc_builder(use_onprem, use_saas):
if use_onprem or use_saas:
with language_container_factory() as container_builder:
yield container_builder
else:
yield None

def test_something_with_slc(upload_slc):
...
```
2 changes: 2 additions & 0 deletions pytest-slc/doc/changes/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

* [unreleased](unreleased.md)
* [0.1.0](changes_0.1.0.md)
* [0.2.0](changes_0.2.0.md)

<!--- This MyST Parser Sphinx directive is necessary to keep Sphinx happy. We need list here all release letters again, because release droid and other scripts assume Markdown --->
```{toctree}
Expand All @@ -10,5 +11,6 @@ hidden:
---
unreleased
changes_0.1.0
changes_0.2.0

```
9 changes: 9 additions & 0 deletions pytest-slc/doc/changes/changes_0.2.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# 0.2.0 - 2024-08-27

## Summary

🚀 The plugin now covers both building and uploading of the language container.

## Feature

* #49: Create export and upload fixture in the SLC plugin
3 changes: 3 additions & 0 deletions pytest-slc/error_code_config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
error-tags:
PYTSLC:
highest-index: 0
79 changes: 67 additions & 12 deletions pytest-slc/exasol/pytest_slc/__init__.py
Original file line number Diff line number Diff line change
@@ -1,25 +1,80 @@
from __future__ import annotations
from typing import Callable
from pathlib import Path
from exasol.pytest_backend import paralleltask
import pytest

import pyexasol
import exasol.bucketfs as bfs
from exasol.python_extension_common.deployment.language_container_deployer import LanguageContainerDeployer
from exasol.python_extension_common.deployment.language_container_builder import LanguageContainerBuilder

BFS_CONTAINER_DIRECTORY = 'container'

@pytest.fixture(scope="session")
def upload_slc(backend_aware_database_params,
backend_aware_bucketfs_params) -> Callable[[str | Path, str, str], None]:

def func(container_file_path: str | Path,
language_alias: str,
bucketfs_path: str = ''):
@pytest.fixture(scope='session')
def slc_builder() -> LanguageContainerBuilder | None:
"""
This fixture must be provided by the user. Below is a simple implementation.

@pytest.fixture(scope='session')
def slc_builder():
with language_container_factory() as container_builder:
yield container_builder

Here the language_container_factory is a context-manager function that creates a
LanguageContainerBuilder object. This function performs all necessary preparations for
building a language container, but does not call its export function. The idea is to share
the code that builds a language container for the release and for integration tests.
"""
return None


@pytest.fixture(scope='session', autouse=True)
def export_slc_async(slc_builder, use_onprem: bool, use_saas: bool):
"""
The fixture starts the export() function of the provided LanguageContainerBuilder object as
an asynchronous task.

The operation will be skipped if none of the backends is in use.
ahsimb marked this conversation as resolved.
Show resolved Hide resolved
"""
if (not (use_onprem or use_saas)) or (slc_builder is None):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it makes also sense to add a cli option to enable building specifically also when no backend is enabled for tests that only inspect the tar archive. But, I think we can add this also later, when we see the need for it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the last statement :). We do need to test the container building and uploading, but the code for that is in the python-extension-common. We cannot use this plugin there as it would create a circular reference.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PEC needs to test some general things. I meant more, if a extension needs add special things to the language container and wants to check if they are where they should be. But as we agree we can figure that out later.

return None

@paralleltask
def export_runner():
yield slc_builder.export()

with export_runner() as export_task:
return export_task


@pytest.fixture(scope='session')
def export_slc(slc_builder, export_slc_async) -> Path | None:
"""
The fixture waits for the LanguageContainerBuilder.export() function to finish.
It returns the path of the exported container.
"""
if (slc_builder is None) or (export_slc_async is None):
# Perhaps none of the backends is enabled.
return None

export_result = export_slc_async.get_output()
export_info = export_result.export_infos[str(slc_builder.flavor_path)]["release"]
return Path(export_info.cache_file)


@pytest.fixture(scope="session")
def upload_slc(slc_builder, export_slc,
backend_aware_database_params, backend_aware_bucketfs_params):
"""
The fixture uploads language container to a database, according to the selected
ahsimb marked this conversation as resolved.
Show resolved Hide resolved
backends.
"""
if (slc_builder is not None) and (export_slc is not None):
pyexasol_connection = pyexasol.connect(**backend_aware_database_params)
bucketfs_path = bfs.path.build_path(**backend_aware_bucketfs_params, path=bucketfs_path)
bucketfs_path = bfs.path.build_path(**backend_aware_bucketfs_params,
path=BFS_CONTAINER_DIRECTORY)
deployer = LanguageContainerDeployer(pyexasol_connection=pyexasol_connection,
bucketfs_path=bucketfs_path,
language_alias=language_alias)
deployer.run(container_file=Path(container_file_path), alter_system=True, allow_override=True)

return func
language_alias=slc_builder.language_alias)
deployer.run(container_file=export_slc, alter_system=True, allow_override=True)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question is do we need to clean up. Meaning removing the the slc and resetting the language definitions for the system. And maybe we actually want to use the file name we get from export_slc for the filename in the bucketfs. I am actually not sure if the deployer does support this at moment. However, if we do it we could cache the uploaded file.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And, another question, do we wait here for the upload?

Copy link
Contributor Author

@ahsimb ahsimb Aug 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the moment the deployer doesn't support the cleanup. There is a ticket for that.

Yes, uploading the container is a blocking operation.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, so the cleanup is something we can improve later

2 changes: 1 addition & 1 deletion pytest-slc/exasol/pytest_slc/version.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,6 @@
# Do not edit this file manually!
# If you need to change the version, do so in the project.toml, e.g. by using `poetry version X.Y.Z`.
MAJOR = 0
MINOR = 1
MINOR = 2
PATCH = 0
VERSION = f"{MAJOR}.{MINOR}.{PATCH}"
Loading
Loading