[17.0][MIG] queue_job + test_queue_job: Migration to 17.0 #593

Currently translated at 3.7% (5 of 135 strings) Translation: queue-15.0/queue-15.0-queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-15-0/queue-15-0-queue_job/es/

Currently translated at 100.0% (135 of 135 strings) Translation: queue-15.0/queue-15.0-queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-15-0/queue-15-0-queue_job/es/

Currently translated at 83.2% (114 of 137 strings) Translation: queue-15.0/queue-15.0-queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-15-0/queue-15-0-queue_job/zh_CN/

Updated by "Update PO files to match POT (msgmerge)" hook in Weblate. Translation: queue-16.0/queue-16.0-queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-16-0/queue-16-0-queue_job/

It can still be activated by overriding a method.

…n test if addon not loaded. TT35938

…se_sparse_field) from server_wide_modules. OCA#443 (comment)

Co-authored-by: Stéphane Bidoul <[email protected]>

Currently translated at 86.3% (114 of 132 strings) Translation: queue-16.0/queue-16.0-queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-16-0/queue-16-0-queue_job/de/

…fined in abstract model

These fields should not be changed by users.

Several fields on queue.job are initialized using computed fields, then never changed again. On creation of a queue.job record, we'll have an initial INSERT + at least one following UPDATE for the computed fields. Replace all the stored computed fields by a raw initialization of the values in `Job.store()` when the job is created, so we have only a single INSERT. Highlights: * as channel is no longer a compute/inverse field, override_channel is useless, I dropped it (actually the same value was stored in both channel and override_channel as the channel field was stored) * one functional diff is that now, when changing a channel on a job.function, the channel is no longer synchronized on existing jobs, it will be applied only on new jobs: actually this is an improvement, because it was impossible to change the channel of a job function in a large queue_job table as it meant writing on all the done/started jobs * searching the queue.job.function is now cached, as each job using the same will run a query on queue_job_function

Everytime a job is created, a mail.message "Queue Job created" is created. This is useless, as we already have the creation date and user, and nobody will ever want to receive a notification for a created job anyway. Removing the on creation auto-subscription of the user that created the job makes sense as well since we automatically subscribe the queue job managers for failures. Add the owner of the jobs to the followers on failures only as well. It allows to remove a lot of insertions of records (and of deletions when autovacuuming jobs).

When RetryableJobError was raised, any change done by the job was not rollbacked. Using `raise` would throw the exception up to the core and rollback, but we would have a stack trace in the logs for each try. Calling rollback manually (rollback also clears the env) hide the tracebacks, however, when the last try fails, the full traceback is still shown in the logs. Fixes OCA#261

When Job.date_done has been set, for instance because the job has been performed, if the job is set back to pending (e.g. a RetryableJobError is raised), the date_done is kept.

The float_time widget shows hours seconds, we only want seconds. The widget had been removed on the form view, but not on the tree view.

Use case: * you have a root channel per scope/app (eg: root.edi) * you have several sub channels (eg: root.edi.ubl.sales, root.edi.gs1.delivery) * you want to configure capacity only for the main channel "root.edi" Before this change, the channel manager falls back on root channel. However, if you have a specific parent channel configured it sounds a good idea to use it.

Contributing since a while... :)

Updated by "Update PO files to match POT (msgmerge)" hook in Weblate. Translation: queue-15.0/queue-15.0-queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-15-0/queue-15-0-queue_job/

test_queue_job will create channels root.sub and root.sub.sub, so the tests doing the same thing will fail with a unique constraint error.

When displayed in a tab, the widget show the nodes offset, the d3 network is probably confused by the size or visibility of its canvas. Install a listener on tabs to trigger a fit() on the network.

A graph of jobs always share the same graph_uuid, which can be used to group jobs, but is also a faster way to find all the jobs of the graph. Then we can use the dependencies field to build the whole graph from the pre-selection of jobs.

Jobs waiting that their dependencies are executed cannot be requeued. They have to keep waiting their turn.

* mock_jobs context manager with assert methods to check enqueued jobs * jobs are not stored in database, yet they can be inspected and performed during the test * the context manager works with graphs enqueued by delayable and with with_delay() * direct execution of jobs works with graphs by executing them synchronously following their topological sort

The previous implementation was incomplete, as it was setting the graph_uuid when `Job.add_depends()` was called. It had the advantage of being correct when (for e.g. a test) manually created a Job and called `add_depends()`, but it could not work when in some situations involving a group where one of the job has no job depending on it. Also, remove ``Job.add_reverse_depends()`` which is not used.

* Fix a js bug when there is no graph. * Use a single read group for multiple jobs

From https://visjs.github.io/vis-network/examples/network/other/html-in-titles.html

When showing large graphs, like 300 jobs (not really a real life case maybe), the graph takes a dozen of seconds to render. Disabling the stabilization helps a lot.

* Renaming the TestXxx models in test_queue_job is because of pytest that complains, confusing them for test classes * logger.warn is deprecated in favor of logger.warning * Add invalidation of cache in SQL update that enqueue dependent jobs * Flush in a test, the same is done currently in the controller * Weird one: we can no longer compare the bound methods of the models, compare their __func__ which is equal.

The assert fails to show the method's details if we store the __func__ in JobCall. Anyway, it was hackish to change the content of "method".

The fact that mocks are used is an implementation detail, and is actually hidden to the end user of the test API.

on_done() is maybe a better choice, it is less ambiguous than then() (clearer on the fact it happens on done()) and done() (less confusion with set_done()). I'll do the renaming soon, unless you have other suggestions. Ref: OCA#154 (comment)

Updated by "Update PO files to match POT (msgmerge)" hook in Weblate. Translation: queue-15.0/queue-15.0-queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-15-0/queue-15-0-queue_job/

This forward port takes all the changes from 15.0 not yet into 16.0 up to OCA@173b6ca This FP contains the new queue dependencies functionality. The dedicated widget used to display the dependecies graph on the job form is also migrated to owl

Updated by "Update PO files to match POT (msgmerge)" hook in Weblate. Translation: queue-16.0/queue-16.0-queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-16-0/queue-16-0-queue_job/

In servers with multiple databases db is not captured from URL when processing jobs in /queue_job/runjob With this patch db is set from the db param in URL when calling the runjob endpoint Fixes OCA#503

…ou generated too much records

Currently translated at 82.0% (128 of 156 strings) Translation: queue-16.0/queue-16.0-queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-16-0/queue-16-0-queue_job/es/

When a job is postponed because of a retryable error, as the error is not re-raised, we reach the new method self._enqueue_dependent_jobs(env, job) With a closed cursor. Return early in case of postponing, as there is no chance dependent jobs can become pending anyway.

jobrunner_db_user, jobrunner_db_password

Current situation: * multiple keys for no good reason * half baked: not all of them used everywhere * no centralization * poor naming With this change we'll have: * 1 and only one key to disable via ctx: ``queue_job__no_delay`` * 1 and only one key to disable via os env: ``QUEUE_JOB__NO_DELAY`` * backward compatibility with deprecation for old keys

The trap_jobs() test helper accepts any model/recordset as long as the job function is the same, which is not enough to verify that the job has been delayed on the expected recordset or model. Tested job functions are "bound methods" so we can compare their `__self__`. Example of assertion that would be valid: partner = self.env["res.partner"].create({"name": "foo"}) with trap_jobs() as trap: partner.with_delay().do_something(42) trap.assert_enqueued_job( self.env["res.partner"].do_something, args=(42,) ) trap.assert_jobs_count(1, only=self.env["res.partner"].do_something) Now, it would fail with an error message like: E AssertionError: Job <res.partner()>.do_something(42) with properties (channel=root.test, description=Do something, eta=15, identity_key=<function identity_exact at 0x7f01bfb54cb0>, max_retries=1, priority=15) was not enqueued. E Actual enqueued jobs: E * <res.partner(64,)>.do_something(42) with properties (priority=15, max_retries=1, eta=15, description=Do something, channel=root.test, identity_key=<function identity_exact at 0x7f01bfb54cb0>)

Currently translated at 98.0% (153 of 156 strings) Translation: queue-16.0/queue-16.0-queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-16-0/queue-16-0-queue_job/es/

Currently translated at 100.0% (156 of 156 strings) Translation: queue-16.0/queue-16.0-queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-16-0/queue-16-0-queue_job/es/

Use the most efficient Selector implementation available on the current platform Odoo supports only SelectSelector but it is a little obsolete python >= 3.4 supports a new high-level library Selectors: - https://docs.python.org/es/3/library/selectors.html It could to auto-choose the following ones: - SelectSelector - PollSelector - EpollSelector - DevpollSelector - KqueueSelector Using the DefaultSelector class the most efficient implementation available on the current platform will be use: - https://docs.python.org/3/library/selectors.html#selectors.DefaultSelector It helps to support better the resources of the system Using SelectSelector you are not able to run workers >=255 If you set `ulimit -n 10240` and run `odoo-bin --workers=255` the following error is raised: Traceback (most recent call last): File "odoo/service/server.py", line 926, in run self.sleep() File "odoo/service/server.py", line 852, in sleep sel.select(self.beat) File "python3.8/lib/python3.8/selectors.py", line 323, in select r, w, _ = self._select(self._readers, self._writers, [], timeout) ValueError: filedescriptor out of range in select() But using PollSelector it is not reproduced even using more workers Most of platform supports PollSelector but using DefaultSelector we can be sure that even too old system are supported too And using this High-level library will allow to use the future new improvements e.g. Epoll has better performance improvements More info about: - https://devarea.com/linux-io-multiplexing-select-vs-poll-vs-epoll - redis/redis-py#486 - odoo/odoo#84684

Updated by "Update PO files to match POT (msgmerge)" hook in Weblate. Translation: queue-16.0/queue-16.0-queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-16-0/queue-16-0-queue_job/

Currently translated at 100.0% (156 of 156 strings) Translation: queue-16.0/queue-16.0-queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-16-0/queue-16-0-queue_job/es/

Updated by "Update PO files to match POT (msgmerge)" hook in Weblate. Translation: queue-16.0/queue-16.0-queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-16-0/queue-16-0-queue_job/

1. In production, a job which is waiting dependencies or which has started, but not completed, should not be repeated if the identity_key matches. 2. In tests, the mock queue handler is now enhanced to allow better mimicking of the identity_key blocks from production. 3. In tests, the mock queue handler now clears the enqueued jobs after performing them, to better reproduce what a production environment would do.

Good explanation from Guewen: Depending on the use case, we should or should not include. As part of my current work, I use Sidekiq daily. They have a similar feature, but you can set a per-job parameter unique_until with options: start (that would mean up to ENQUEUED here) or success (that would be up to STARTED here). Think about this use case: a job refreshes a cache. Data have changed, we create a pending job. Data change again, no new job because the job is still pending. Job starts. Data change while the job is running. In this very case, we'd like to enqueue a new job otherwise the cache will be outdated. I reckon that both cases are valid, but I fear adding this state in the domain may, silently and in subtle ways, existing behaviors.

…tance with list_db = False The scenario is a multi database environment where you can't set db_name as the databases change frequently and you use dbfilter to allow users to access their DB according to domain

…e with list_db=false

Separate tests that need special methods in a dedicated test_queue_job module. Rework the registry of the jobs. The 'job' decorator is applied on functions before they are bound to their class, so we don't know the class yet. Now, the decorator marks job methods with 'delayable' and when a model is registered (Base._register_hook), it looks for the marked methods and register them.

To avoid to pollute 'queue.job.function' with a test job.

The model name, method name, recordset ids, arguments and keyword arguments must never ever be modified after the creation of a job. This change fix a bug when a job with a mutable argument see the content of the argument being modified during the execution of a job.

The automatic registration of job methods and their defaut channel has been a bit chaotic. The initial version for the new Odoo API could crashing as soon as a method was decorated by @Property. There is such a property in the code code, the method '_cache'. The problem of the crash was that, the introspection basically uses a 'getattr' on every attribute of the instance. The '_cache' method could then be called on an empty recordset, which is not supported by '_cache'. A first correction (49d8f37) was to naively skip the '_cache' method from the introspection. In any case, it is wrong to access the property of an instance only to instrospect its members. That's why the correction 4ebb245 changed the inspection from the instance to the class. Properties are no longer accessed, however this correction was not correct for Python 3. When members of the class are introspected, they are neither bound neither unbound methods. they are mere functions. The change here is to lookup for functions. _register_job must now takes the model as input argument, because there is no way to get the name of the model from the function. Closes OCA#64 Follows OCA#50

The related action button now opens by default the record on which the job is working. It uses the default views of the job. The exact behavior depends of the number of records in the job: * No record (job called on a model): display a message indicating that there is no available action * One record: open the default view in form * Several records: open the default view in list The related action can still be overridden by the 'related_action' decorator.

Currently translated at 100.0% (12 of 12 strings) Translation: queue-12.0/queue-12.0-test_queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-12-0/queue-12-0-test_queue_job/zh_CN/

The former syntax is deprecated

This change has been done and accepted on the component and connector addons: OCA/connector#356 This is a low-level technical module, we may need to use jobs on enterprise features.

Add a field in the "queue_job" model to store the worker pid. No usage for new besides "logging" for debugging (helps to find which process to investigate in case a job is stuck). Later, it may be useful to improve the recovery of job that stay in the wrong state when the job runner restarts. Closes OCA#177

@job

* @job and @related_action are still supported but emit warnings and will be removed at some point (code to remove has a TODO with :job-no-decorator:) * the options previously set by the decorators are now stored on "queue.job.function" records, which can be explicitly created as XML records (and then editable!) * any method can be used with "with_delay()", if no "queue.job.function" exists, default values are used (root channel, default related action, ...)

A many2one to select the model + a char for the method name is more convenient and error-proof than having to type the expected format correctly.

These fields contain the current model and record ids and user id the job works on. Storing them as separate fields does not allow to take benefits on the "JobSerialized" field, which keeps the original user, and later the 'su' flag. We could store the context there as well (or a cleaned one). Use computed fields to keep the model_name, record_ids, user_id fields for backward compatibility. The model_name should stay anyway for the searches on the UI, and the user_id for the same reason + possibility to edit the user.

Which means that the record the job is working on, or any record argument of the job keeps the 'su' flag.

- set module uninstallable - add new .dot file config for v14

Now that the `@job` decorater is deleted, `identity_exact` is the only thing we may need to import from "job.py". Being able to import it from "odoo.addons.queue_job" is nicer than from "odoo.addons.queue_job.job".

Co-authored-by: Simone Orsi <[email protected]>

Updated by "Update PO files to match POT (msgmerge)" hook in Weblate. Translation: queue-14.0/queue-14.0-test_queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-14-0/queue-14-0-test_queue_job/

…inate a job + test"

Updated by "Update PO files to match POT (msgmerge)" hook in Weblate. Translation: queue-14.0/queue-14.0-test_queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-14-0/queue-14-0-test_queue_job/

This patch method has to be called in ``_register_hook``. When a method is patched, any call to the method will not directly execute the method's body, but will instead enqueue a job. When a ``context_key`` is set when calling ``_patch_job_auto_delay``, the patched method is automatically delayed only when this key is ``True`` in the caller's context. It is advised to patch the method with a ``context_key``, because making the automatic delay *in any case* can produce nasty and unexpected side effects (e.g. another module calls the method and expects it to be computed before doing something else, expecting a result, ...). A typical use case is when a method in a module we don't control is called synchronously in the middle of another method, and we'd like all the calls to this method become asynchronous. It relies on OCA#274 that deprecates the `@job` decorator.

Updated by "Update PO files to match POT (msgmerge)" hook in Weblate. Translation: queue-16.0/queue-16.0-test_queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-16-0/queue-16-0-test_queue_job/

It can still be activated by overriding a method.

Currently translated at 75.0% (9 of 12 strings) Translation: queue-16.0/queue-16.0-test_queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-16-0/queue-16-0-test_queue_job/de/

Everytime a job is created, a mail.message "Queue Job created" is created. This is useless, as we already have the creation date and user, and nobody will ever want to receive a notification for a created job anyway. Removing the on creation auto-subscription of the user that created the job makes sense as well since we automatically subscribe the queue job managers for failures. Add the owner of the jobs to the followers on failures only as well. It allows to remove a lot of insertions of records (and of deletions when autovacuuming jobs).

Updated by "Update PO files to match POT (msgmerge)" hook in Weblate. Translation: queue-15.0/queue-15.0-test_queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-15-0/queue-15-0-test_queue_job/

A graph of jobs always share the same graph_uuid, which can be used to group jobs, but is also a faster way to find all the jobs of the graph. Then we can use the dependencies field to build the whole graph from the pre-selection of jobs.

Jobs waiting that their dependencies are executed cannot be requeued. They have to keep waiting their turn.

As jobs are created in the setup, but not always delayed, the __del__ warning that the job was not delayed is triggered. Create only the jobs necessary for each test.

* mock_jobs context manager with assert methods to check enqueued jobs * jobs are not stored in database, yet they can be inspected and performed during the test * the context manager works with graphs enqueued by delayable and with with_delay() * direct execution of jobs works with graphs by executing them synchronously following their topological sort

The previous implementation was incomplete, as it was setting the graph_uuid when `Job.add_depends()` was called. It had the advantage of being correct when (for e.g. a test) manually created a Job and called `add_depends()`, but it could not work when in some situations involving a group where one of the job has no job depending on it. Also, remove ``Job.add_reverse_depends()`` which is not used.

* Renaming the TestXxx models in test_queue_job is because of pytest that complains, confusing them for test classes * logger.warn is deprecated in favor of logger.warning * Add invalidation of cache in SQL update that enqueue dependent jobs * Flush in a test, the same is done currently in the controller * Weird one: we can no longer compare the bound methods of the models, compare their __func__ which is equal.

The fact that mocks are used is an implementation detail, and is actually hidden to the end user of the test API.

on_done() is maybe a better choice, it is less ambiguous than then() (clearer on the fact it happens on done()) and done() (less confusion with set_done()). I'll do the renaming soon, unless you have other suggestions. Ref: OCA#154 (comment)

This forward port takes all the changes from 15.0 not yet into 16.0 up to OCA@173b6ca This FP contains the new queue dependencies functionality. The dedicated widget used to display the dependecies graph on the job form is also migrated to owl

Updated by "Update PO files to match POT (msgmerge)" hook in Weblate. Translation: queue-16.0/queue-16.0-test_queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-16-0/queue-16-0-test_queue_job/

Current situation: * multiple keys for no good reason * half baked: not all of them used everywhere * no centralization * poor naming With this change we'll have: * 1 and only one key to disable via ctx: ``queue_job__no_delay`` * 1 and only one key to disable via os env: ``QUEUE_JOB__NO_DELAY`` * backward compatibility with deprecation for old keys

The trap_jobs() test helper accepts any model/recordset as long as the job function is the same, which is not enough to verify that the job has been delayed on the expected recordset or model. Tested job functions are "bound methods" so we can compare their `__self__`. Example of assertion that would be valid: partner = self.env["res.partner"].create({"name": "foo"}) with trap_jobs() as trap: partner.with_delay().do_something(42) trap.assert_enqueued_job( self.env["res.partner"].do_something, args=(42,) ) trap.assert_jobs_count(1, only=self.env["res.partner"].do_something) Now, it would fail with an error message like: E AssertionError: Job <res.partner()>.do_something(42) with properties (channel=root.test, description=Do something, eta=15, identity_key=<function identity_exact at 0x7f01bfb54cb0>, max_retries=1, priority=15) was not enqueued. E Actual enqueued jobs: E * <res.partner(64,)>.do_something(42) with properties (priority=15, max_retries=1, eta=15, description=Do something, channel=root.test, identity_key=<function identity_exact at 0x7f01bfb54cb0>)

Currently translated at 100.0% (13 of 13 strings) Translation: queue-16.0/queue-16.0-test_queue_job Translate-URL: https://translation.odoo-community.org/projects/queue-16-0/queue-16-0-test_queue_job/es/

1. In production, a job which is waiting dependencies or which has started, but not completed, should not be repeated if the identity_key matches. 2. In tests, the mock queue handler is now enhanced to allow better mimicking of the identity_key blocks from production. 3. In tests, the mock queue handler now clears the enqueued jobs after performing them, to better reproduce what a production environment would do.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[17.0][MIG] queue_job + test_queue_job: Migration to 17.0 #593

[17.0][MIG] queue_job + test_queue_job: Migration to 17.0 #593

Commits on Nov 24, 2023

Commits on Nov 29, 2023

Commits on Dec 4, 2023