Psych DisalowedClass MiqAeEngine::StateVarHash error while retiring an Openstack Instance

[agrare]

[----] I, [2024-07-17T12:30:20.546139#109140:9ad8]  INFO -- evm: MIQ(MiqQueue#deliver) Message id: [10931], Delivering...
[----] I, [2024-07-17T12:30:20.569415#109140:9ad8]  INFO -- evm: Delivering {"request"=>"vm_retire", "VmOrTemplate::vm"=>185} for object [VmRetireTask.2] with state [CheckRemovedFromProvider] to Automate
[----] I, [2024-07-17T12:30:20.582462#109140:9ad8]  INFO -- evm: MIQ(VmRetireTask#after_ae_delivery) ae_result="error"
[----] E, [2024-07-17T12:30:20.625368#109140:9ad8] ERROR -- evm: MIQ(MiqQueue#deliver) Message id: [10931], Error: [Tried to load unspecified class: MiqAeEngine::StateVarHash]
[----] E, [2024-07-17T12:30:20.625477#109140:9ad8] ERROR -- evm: [Psych::DisallowedClass]: Tried to load unspecified class: MiqAeEngine::StateVarHash  Method:[block (2 levels) in <class:LogProxy>]
[----] E, [2024-07-17T12:30:20.625518#109140:9ad8] ERROR -- evm: /home/agrare/.gem/ruby/3.1.4/gems/psych-4.0.6/lib/psych/class_loader.rb:99:in `find'
/home/agrare/.gem/ruby/3.1.4/gems/psych-4.0.6/lib/psych/class_loader.rb:28:in `load'
/home/agrare/.gem/ruby/3.1.4/gems/psych-4.0.6/lib/psych/visitors/to_ruby.rb:426:in `resolve_class'
/home/agrare/src/manageiq/manageiq/config/initializers/yaml_autoloader.rb:15:in `resolve_class'
/home/agrare/.gem/ruby/3.1.4/gems/psych-4.0.6/lib/psych/visitors/to_ruby.rb:215:in `visit_Psych_Nodes_Mapping'
/home/agrare/.gem/ruby/3.1.4/gems/psych-4.0.6/lib/psych/visitors/visitor.rb:30:in `visit'
/home/agrare/.gem/ruby/3.1.4/gems/psych-4.0.6/lib/psych/visitors/visitor.rb:6:in `accept'
/home/agrare/.gem/ruby/3.1.4/gems/psych-4.0.6/lib/psych/visitors/to_ruby.rb:35:in `accept'
/home/agrare/.gem/ruby/3.1.4/gems/psych-4.0.6/lib/psych/visitors/to_ruby.rb:320:in `visit_Psych_Nodes_Document'
/home/agrare/.gem/ruby/3.1.4/gems/psych-4.0.6/lib/psych/visitors/visitor.rb:30:in `visit'
/home/agrare/.gem/ruby/3.1.4/gems/psych-4.0.6/lib/psych/visitors/visitor.rb:6:in `accept'
/home/agrare/.gem/ruby/3.1.4/gems/psych-4.0.6/lib/psych/visitors/to_ruby.rb:35:in `accept'
/home/agrare/.gem/ruby/3.1.4/gems/psych-4.0.6/lib/psych.rb:334:in `safe_load'
/home/agrare/src/manageiq/manageiq/lib/extensions/yaml_load_aliases.rb:7:in `safe_load'
/home/agrare/.gem/ruby/3.1.4/gems/psych-4.0.6/lib/psych.rb:369:in `load'
/home/agrare/.gem/ruby/3.1.4/bundler/gems/manageiq-automation_engine-770768456550/lib/miq_automation_engine/engine/miq_ae_engine/miq_ae_workspace_runtime.rb:166:in `instantiate'
/home/agrare/.gem/ruby/3.1.4/bundler/gems/manageiq-automation_engine-770768456550/lib/miq_automation_engine/engine/miq_ae_engine/miq_ae_workspace_runtime.rb:54:in `instantiate_with_user'
/home/agrare/.gem/ruby/3.1.4/bundler/gems/manageiq-automation_engine-770768456550/lib/miq_automation_engine/engine/miq_ae_engine/miq_ae_workspace_runtime.rb:48:in `block in instantiate'
/home/agrare/src/manageiq/manageiq/app/models/user.rb:383:in `with_user'
/home/agrare/.gem/ruby/3.1.4/bundler/gems/manageiq-automation_engine-770768456550/lib/miq_automation_engine/engine/miq_ae_engine/miq_ae_workspace_runtime.rb:48:in `instantiate'
/home/agrare/.gem/ruby/3.1.4/bundler/gems/manageiq-automation_engine-770768456550/lib/miq_automation_engine/engine/miq_ae_engine.rb:343:in `resolve_automation_object'
/home/agrare/.gem/ruby/3.1.4/bundler/gems/manageiq-automation_engine-770768456550/lib/miq_automation_engine/engine/miq_ae_engine.rb:106:in `deliver'
/home/agrare/src/manageiq/manageiq/app/models/miq_queue.rb:517:in `block in dispatch_method'

[jrafanie]

First thought is automation engine should have a list of allowed classes and pass it to safe_load here:

manageiq-automation_engine/lib/miq_automation_engine/engine/miq_ae_engine/miq_ae_workspace_runtime.rb

Line 166 in 7707684

@persist_state_hash.merge!(YAML.load(ae_state_data))

Or we need to add this class to the default list:
https://github.com/ManageIQ/manageiq/blob/31ce59bae092487710e925a3a8f78fa622d782dc/lib/yaml_permitted_classes.rb#L2-L14

I don't know if it makes sense to pass it at runtime.rb at that point.

[agrare]

Yeah we either need to allow loading that class or better yet find where it is being saved as an object and serialize/deserialize it properly which I think would be better

Here is what the payload looks like, :ae_state_data=>"--- !ruby/object:MiqAeEngine::StateVarHash\nbinary_blob_id: 53\n" seems to just be a binary_blob_id

Full payload:

[----] I, [2024-07-17T12:29:19.696379#109140:9ad8]  INFO -- evm: Requeuing {:object_type=>"VmRetireTask", :object_id=>2, :attrs=>{"request"=>"vm_retire", "VmOrTemplate::vm"=>185}, :instance_name=>"AUTOMATION", :user_id=>1, :miq_group_id=>2, :tenant_id=>1, :miq_task_id=>nil, :state=>"CheckRemovedFromProvider", :ae_fsm_started=>nil, :ae_state_started=>"2024-07-17 16:29:18 UTC", :ae_state_retries=>1, :ae_state_data=>"--- !ruby/object:MiqAeEngine::StateVarHash\nbinary_blob_id: 53\n", :ae_state_previous=>"---\n\"/ManageIQ/Cloud/VM/Retirement/StateMachines/VMRetirement/Default\":\n  ae_state: CheckRemovedFromProvider\n  ae_state_retries: 1\n  ae_state_started: 2024-07-17 16:29:18 UTC\n  ae_state_max_retries: 100\n"} for object [VmRetireTask.2] with state [CheckRemovedFromProvider] to Automate for delivery in [60] seconds

[jrafanie]

related... #538. fixing the code to not serialize it would be good. when can we delete automate?

[jrafanie]

FYI, #554 is a generic workaround for the loading of aliases. We probably need to treat allowed classes similarly and add it to allowed list directly on the call or centrally as described in #552 (comment). If we knew where these were coming from, it would be better to not serialize these objects in the first place.