Inputs with incomplete variable substitutions will be silently ignored

[cmacknz]

discarded-input.yml.txt

The attached agent policy (with .txt extension so GitHub would let me upload it) should start two instances of the httpjson input, but only ends up starting the input with ID httpjson-ti_abusech-5c2d6965-8136-4237-8c08-20a54da01a13.

Looking the diagnostics the second input with ID httpjson-ti_misp-911740ef-8690-424f-8e0c-e7a540a178d6 is present in the pre-config.yaml but removed in the computed-config.yaml suggesting that the agent has decided it does not need to run it without explaining why.

Looking closer at the agent policy reveals the use of ${MISP_API_KEY} as a variable that does not resolve to anything. The agent determines that the input configuration is incomplete and decides not to run it without any error messages in the logs or the agent status output.

sudo elastic-agent status --output=yaml
info:
  id: f8447627-952a-4be0-a3e5-3b4db4da126b
  version: 8.7.0
  commit: fdd1465eb0551eed52130810e52859658b5367f9
  build_time: 2023-02-07 03:10:25 +0000 UTC
  snapshot: true
state: 2
message: Running
components:
- id: beat/metrics-monitoring
  name: beat/metrics
  state: 2
  message: 'Healthy: communicating with pid ''22094'''
  units:
  - unit_id: beat/metrics-monitoring-metrics-monitoring-beats
    unit_type: 0
    state: 2
    message: Healthy
  - unit_id: beat/metrics-monitoring
    unit_type: 1
    state: 2
    message: Healthy
  version_info:
    name: beat-v2-client
    version: 8.7.0
    meta:
      build_time: 2023-02-06 17:06:15 +0000 UTC
      commit: c8578a0f674f5cde47e3c9b021a5800d46abc170
- id: httpjson-default
  name: httpjson
  state: 2
  message: 'Healthy: communicating with pid ''22091'''
  units:
  - unit_id: httpjson-default-httpjson-ti_abusech-5c2d6965-8136-4237-8c08-20a54da01a13
    unit_type: 0
    state: 2
    message: Healthy
  - unit_id: httpjson-default
    unit_type: 1
    state: 2
    message: Healthy
  version_info:
    name: beat-v2-client
    version: 8.7.0
    meta:
      build_time: 2023-02-06 17:06:07 +0000 UTC
      commit: c8578a0f674f5cde47e3c9b021a5800d46abc170
- id: http/metrics-monitoring
  name: http/metrics
  state: 2
  message: 'Healthy: communicating with pid ''22092'''
  units:
  - unit_id: http/metrics-monitoring-metrics-monitoring-agent
    unit_type: 0
    state: 2
    message: Healthy
  - unit_id: http/metrics-monitoring
    unit_type: 1
    state: 2
    message: Healthy
  version_info:
    name: beat-v2-client
    version: 8.7.0
    meta:
      build_time: 2023-02-06 17:06:15 +0000 UTC
      commit: c8578a0f674f5cde47e3c9b021a5800d46abc170
- id: filestream-monitoring
  name: filestream
  state: 2
  message: 'Healthy: communicating with pid ''22093'''
  units:
  - unit_id: filestream-monitoring-filestream-monitoring-agent
    unit_type: 0
    state: 2
    message: Healthy
  - unit_id: filestream-monitoring
    unit_type: 1
    state: 2
    message: Healthy
  version_info:
    name: beat-v2-client
    version: 8.7.0
    meta:
      build_time: 2023-02-06 17:06:07 +0000 UTC
      commit: c8578a0f674f5cde47e3c9b021a5800d46abc170

We can continue not running the input at all, but a much better user experience would be for the removed input to appear in the elastic-agent status input with an error indicating that the input could not be started because of an error during variable substitution.

We could consider a simple log statement, but including the incomplete input in the agent runtime model allows its status to be reported back to Fleet in a much more obvious way.

[cmacknz]

@blakerouse any ideas what would be causing this?

[lucabelluccini]

It can be quite easy to overlook an input is not running especially in very complex policies.
It would be great to report a degraded or unhealthy status of the components which couldn't be rendered.

It might become more complex to handle when using autodiscovery, but the example in this issue should be manageable to report.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)