Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: avoid race conditions when removing multiple instance #15495

Merged
merged 4 commits into from
Sep 13, 2024

Conversation

pb82
Copy link
Contributor

@pb82 pb82 commented Sep 9, 2024

SUMMARY

Removing multiple instance groups at the same time currently fails. The failure seems to be caused by a race condition, when the UI sends multiple disassociate requests.

Locking the m2m relationship table during each disassociation resolves the problem (likely because this forces all operations to be sequential in the database).

ISSUE TYPE
  • Bug, Docs Fix or other nominal change
COMPONENT NAME
  • API
AWX VERSION
devel
ADDITIONAL INFORMATION

@AlanCoding
Copy link
Member

I would like to avoid cursor.execute calls at all possible costs. There's another problem here, that another solution mechanism already exists. You can see that there's a very different code pattern to this, but it does the same thing of obtaining a row-level lock.

with transaction.atomic():
instance_groups_queryset = InstanceGroup.objects.select_for_update()
if self.parent_model is Instance:
ig_obj = get_object_or_400(instance_groups_queryset, pk=sub_id)
else:
# similar to get_parent_object, but selected for update
parent_filter = {self.lookup_field: self.kwargs.get(self.lookup_field, None)}
ig_obj = get_object_or_404(instance_groups_queryset, **parent_filter)
if inst_name not in ig_obj.policy_instance_list:
ig_obj.policy_instance_list.append(inst_name)
ig_obj.save(update_fields=['policy_instance_list'])

That implementation lives in InstanceGroupMembershipMixin and is used in:

  • InstanceInstanceGroupsList
  • InstanceGroupInstanceList

Those 2 views do appear very similar on a surface-level to the view your patch is concerned with, which I believe is the organization instance group list. So I'd like to see more research on whether that mixin can or can't be reused here, and if not, if the ORM code pattern can be used, and whether a fully general implementation is needed for all attach-detach views as you have, or if a more targeted solution can work.

@fosterseth
Copy link
Member

had some CI breakage yesterday, rebasing should help

@pb82
Copy link
Contributor Author

pb82 commented Sep 11, 2024

@AlanCoding @chrismeyersfsu I've changed it to a Mixin & select_for_update now, no more full table lock. Could you have another look please?

to ensure instance group updates are persisted
"""

def unattach(self, request, *args, **kwargs):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe this needs to be applied to attach as well

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried it, but it doesn't solve the problem we have with attaching (which is that the order ends up random)

Copy link

@pb82 pb82 merged commit 1c170c3 into ansible:devel Sep 13, 2024
19 checks passed
@pb82 pb82 deleted the AAP-30335 branch September 13, 2024 07:35
djyasin pushed a commit to djyasin/awx that referenced this pull request Nov 11, 2024
…5495)

* fix: avoid race conditions when removing multiple instance groups at once

* remove unused imports
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants