Skip to content

Commit

Permalink
[Obs AI Assistant] Fix chat on the Alerts page (elastic#197126)
Browse files Browse the repository at this point in the history
Closes elastic#184214

## Summary

### Problem
The Observability AI Assistant doesn't work on the Alerts page - errors
out with a 400 status code from OpenAI.
The reason for this is that the description for the function
`get_data_on_screen` is too long, and there is a token limit for
function descriptions by OpenAI.

**Note:** This error does not occur with Gemini or Bedrock because
simulated function calling is enabled by default for them. With
simulated function calling, all functions and their descriptions are
appended to the system message, therefore doesn't run into a token limit
error as opposed to OpenAI.

### Solution
Append the function description to the system message instead of sending
it with the function.

The implementation includes:
- Registering an AdHoc instruction
- Retrieving AdHoc instructions
- Combine the retrieved AdHoc instructions with the other adHoc
instructions passed to the chat and pass all AdHoc instructions to the
`getSystemMessageFromInstructions` function.

This correctly orders the description for the `get_data_on_screen`
function at the end of the system message

_OpenAI request object **before** the above update:_
<details>
  <summary>Click to expand JSON</summary>

```json
{
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant for Elastic Observability. Your goal is to help the Elastic Observability users to quickly assess what is happening in their observed systems. You can help them visualise and analyze data, investigate their systems, perform root cause analysis or identify optimisation opportunities.\n\n  It's very important to not assume what the user is meaning. Ask them for clarification if needed.\n\n  If you are unsure about which function should be used and with what arguments, ask the user for clarification or confirmation.\n\n  In KQL (\"kqlFilter\")) escaping happens with double quotes, not single quotes. Some characters that need escaping are: ':()\\  /\". Always put a field value in double quotes. Best: service.name:\"opbeans-go\". Wrong: service.name:opbeans-go. This is very important!\n\n  You can use Github-flavored Markdown in your responses. If a function returns an array, consider using a Markdown table to format the response.\n\n  Note that ES|QL (the Elasticsearch Query Language which is a new piped language) is the preferred query language.\n\n  If you want to call a function or tool, only call it a single time per message. Wait until the function has been executed and its results\n  returned to you, before executing the same tool or another tool again if needed.\n\n  DO NOT UNDER ANY CIRCUMSTANCES USE ES|QL syntax (`service.name == \"foo\"`) with \"kqlFilter\" (`service.name:\"foo\"`).\n\n  The user is able to change the language which they want you to reply in on the settings page of the AI Assistant for Observability, which can be found in the Stack Management app under the option AI Assistants.\n  If the user asks how to change the language, reply in the same language the user asked in.\n\nYou MUST use the \"query\" function when the user wants to:\n  - visualize data\n  - run any arbitrary query\n  - breakdown or filter ES|QL queries that are displayed on the current page\n  - convert queries from another language to ES|QL\n  - asks general questions about ES|QL\n\n  DO NOT UNDER ANY CIRCUMSTANCES generate ES|QL queries or explain anything about the ES|QL query language yourself.\n  DO NOT UNDER ANY CIRCUMSTANCES try to correct an ES|QL query yourself - always use the \"query\" function for this.\n\n  If the user asks for a query, and one of the dataset info functions was called and returned no results, you should still call the query function to generate an example query.\n\n  Even if the \"query\" function was used before that, follow it up with the \"query\" function. If a query fails, do not attempt to correct it yourself. Again you should call the \"query\" function,\n  even if it has been called before.\n\n  When the \"visualize_query\" function has been called, a visualization has been displayed to the user. DO NOT UNDER ANY CIRCUMSTANCES follow up a \"visualize_query\" function call with your own visualization attempt.\n  If the \"execute_query\" function has been called, summarize these results for the user. The user does not see a visualization in this case.\n\nYou MUST use the \"get_dataset_info\"  function before calling the \"query\" or the \"changes\" functions.\n\nIf a function requires an index, you MUST use the results from the dataset info functions.\n\nYou have access to data on the screen by calling the \"get_data_on_screen\" function.\nUse it to help the user understand what they are looking at. A short summary of what they are looking at is available in the return of the \"context\" function.\nData that is compact enough automatically gets included in the response for the \"context\" function.\n\nYou can use the \"summarize\" function to store new information you have learned in a knowledge database.\nOnly use this function when the user asks for it.\nAll summaries MUST be created in English, even if the conversation was carried out in a different language."
    },
    { "role": "user", "content": "Can you explain this page?" },
    { "role": "assistant", "content": "", "function_call": { "name": "context", "arguments": "{}" } },
    {
      "role": "user",
      "content": "{\"screen_description\":\"The user is looking at http://localhost:5601/phq/app/observability/alerts?_a=(filters:!(),kuery:%27%27,rangeFrom:now-24h,rangeTo:now,status:all). The current time range is 2024-10-21T18:23:54.539Z - 2024-10-21T18:38:54.539Z.\",\"learnings\":[],\"data_on_screen\":[{\"name\":\".es-query\",\"value\":\"Elasticsearch query Alert when matches are found during the latest query run.\",\"description\":\"An available rule is Elasticsearch query.\"},{\"name\":\"observability.rules.custom_threshold\",\"value\":\"Custom threshold Alert when any Observability data type reaches or exceeds a given value.\",\"description\":\"An available rule is Custom threshold.\"},{\"name\":\"xpack.ml.anomaly_detection_alert\",\"value\":\"Anomaly detection Alert when anomaly detection jobs results match the condition.\",\"description\":\"An available rule is Anomaly detection.\"},{\"name\":\"slo.rules.burnRate\",\"value\":\"SLO burn rate Alert when your SLO burn rate is too high over a defined period of time.\",\"description\":\"An available rule is SLO burn rate.\"},{\"name\":\"metrics.alert.threshold\",\"value\":\"Metric threshold Alert when the metrics aggregation exceeds the threshold.\",\"description\":\"An available rule is Metric threshold.\"},{\"name\":\"metrics.alert.inventory.threshold\",\"value\":\"Inventory Alert when the inventory exceeds a defined threshold.\",\"description\":\"An available rule is Inventory.\"},{\"name\":\"logs.alert.document.count\",\"value\":\"Log threshold Alert when the log aggregation exceeds the threshold.\",\"description\":\"An available rule is Log threshold.\"},{\"name\":\"xpack.uptime.alerts.tlsCertificate\",\"value\":\"Uptime TLS Alert when the TLS certificate of an Uptime monitor is about to expire.\",\"description\":\"An available rule is Uptime TLS.\"},{\"name\":\"xpack.uptime.alerts.monitorStatus\",\"value\":\"Uptime monitor status Alert when a monitor is down or an availability threshold is breached.\",\"description\":\"An available rule is Uptime monitor status.\"},{\"name\":\"xpack.uptime.alerts.durationAnomaly\",\"value\":\"Uptime Duration Anomaly Alert when the Uptime monitor duration is anomalous.\",\"description\":\"An available rule is Uptime Duration Anomaly.\"},{\"name\":\"xpack.synthetics.alerts.monitorStatus\",\"value\":\"Synthetics monitor status Alert when a monitor is down.\",\"description\":\"An available rule is Synthetics monitor status.\"},{\"name\":\"xpack.synthetics.alerts.tls\",\"value\":\"Synthetics TLS certificate Alert when the TLS certificate of a Synthetics monitor is about to expire.\",\"description\":\"An available rule is Synthetics TLS certificate.\"},{\"name\":\"apm.error_rate\",\"value\":\"Error count threshold Alert when the number of errors in a service exceeds a defined threshold.\",\"description\":\"An available rule is Error count threshold.\"},{\"name\":\"apm.transaction_error_rate\",\"value\":\"Failed transaction rate threshold Alert when the rate of transaction errors in a service exceeds a defined threshold.\",\"description\":\"An available rule is Failed transaction rate threshold.\"},{\"name\":\"apm.transaction_duration\",\"value\":\"Latency threshold Alert when the latency of a specific transaction type in a service exceeds a defined threshold.\",\"description\":\"An available rule is Latency threshold.\"},{\"name\":\"apm.anomaly\",\"value\":\"APM Anomaly Alert when either the latency, throughput, or failed transaction rate of a service is anomalous.\",\"description\":\"An available rule is APM Anomaly.\"}]}",
      "name": "context"
    }
  ],
  "stream": true,
  "tools": [
    {
      "function": {
        "name": "get_data_on_screen",
        "description": "Get data that is on the screen:\n.es-query: An available rule is Elasticsearch query.\nobservability.rules.custom_threshold: An available rule is Custom threshold.\nxpack.ml.anomaly_detection_alert: An available rule is Anomaly detection.\nslo.rules.burnRate: An available rule is SLO burn rate.\nmetrics.alert.threshold: An available rule is Metric threshold.\nmetrics.alert.inventory.threshold: An available rule is Inventory.\nlogs.alert.document.count: An available rule is Log threshold.\nxpack.uptime.alerts.tlsCertificate: An available rule is Uptime TLS.\nxpack.uptime.alerts.monitorStatus: An available rule is Uptime monitor status.\nxpack.uptime.alerts.durationAnomaly: An available rule is Uptime Duration Anomaly.\nxpack.synthetics.alerts.monitorStatus: An available rule is Synthetics monitor status.\nxpack.synthetics.alerts.tls: An available rule is Synthetics TLS certificate.\napm.error_rate: An available rule is Error count threshold.\napm.transaction_error_rate: An available rule is Failed transaction rate threshold.\napm.transaction_duration: An available rule is Latency threshold.\napm.anomaly: An available rule is APM Anomaly.",
        "parameters": {
          "type": "object",
          "properties": {
            "data": {
              "type": "array",
              "description": "The pieces of data you want to look at it. You can request one, or multiple",
              "items": {
                "type": "string",
                "enum": [
                  ".es-query",
                  "observability.rules.custom_threshold",
                  "xpack.ml.anomaly_detection_alert",
                  "slo.rules.burnRate",
                  "metrics.alert.threshold",
                  "metrics.alert.inventory.threshold",
                  "logs.alert.document.count",
                  "xpack.uptime.alerts.tlsCertificate",
                  "xpack.uptime.alerts.monitorStatus",
                  "xpack.uptime.alerts.durationAnomaly",
                  "xpack.synthetics.alerts.monitorStatus",
                  "xpack.synthetics.alerts.tls",
                  "apm.error_rate",
                  "apm.transaction_error_rate",
                  "apm.transaction_duration",
                  "apm.anomaly"
                ]
              }
            }
          },
          "required": ["data"]
        }
      },
      "type": "function"
    },
    {
      "function": {
        "name": "query",
        "description": "This function generates, executes and/or visualizes a query\n      based on the user's request. It also explains how ES|QL works and how to\n      convert queries from one language to another. Make sure you call one of\n      the get_dataset functions first if you need index or field names. This\n      function takes no input.",
        "parameters": { "type": "object", "properties": {} }
      },
      "type": "function"
    },
    {
      "function": {
        "name": "get_alerts_dataset_info",
        "description": "Use this function to get information about alerts data.",
        "parameters": {
          "type": "object",
          "properties": {
            "start": {
              "type": "string",
              "description": "The start of the current time range, in datemath, like now-24h or an ISO timestamp"
            },
            "end": {
              "type": "string",
              "description": "The end of the current time range, in datemath, like now-24h or an ISO timestamp"
            }
          }
        }
      },
      "type": "function"
    },
    {
      "function": {
        "name": "alerts",
        "description": "Get alerts for Observability.  Make sure get_alerts_dataset_info was called before.\n        Use this to get open (and optionally recovered) alerts for Observability assets, like services,\n        hosts or containers.\n        Display the response in tabular format if appropriate.\n      ",
        "parameters": {
          "type": "object",
          "properties": {
            "start": {
              "type": "string",
              "description": "The start of the time range, in Elasticsearch date math, like `now`."
            },
            "end": {
              "type": "string",
              "description": "The end of the time range, in Elasticsearch date math, like `now-24h`."
            },
            "kqlFilter": { "type": "string", "description": "Filter alerts by field:value pairs" },
            "includeRecovered": {
              "type": "boolean",
              "description": "Whether to include recovered/closed alerts. Defaults to false, which means only active alerts will be returned"
            }
          },
          "required": ["start", "end"]
        }
      },
      "type": "function"
    },
    {
      "function": {
        "name": "changes",
        "description": "Returns change points like spikes and dips for logs and metrics.",
        "parameters": {
          "type": "object",
          "properties": {
            "start": {
              "type": "string",
              "description": "The beginning of the time range, in datemath, like now-24h, or an ISO timestamp"
            },
            "end": {
              "type": "string",
              "description": "The end of the time range, in datemath, like now, or an ISO timestamp"
            },
            "logs": {
              "description": "Analyze changes in log patterns. If no index is given, the default logs index pattern will be used",
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "name": { "type": "string", "description": "The name of this set of logs" },
                  "index": { "type": "string", "description": "The index or index pattern where to find the logs" },
                  "kqlFilter": {
                    "type": "string",
                    "description": "A KQL filter to filter the log documents by, e.g. my_field:foo"
                  },
                  "field": {
                    "type": "string",
                    "description": "The text field that contains the message to be analyzed, usually `message`. ONLY use field names from the conversation."
                  }
                },
                "required": ["name"]
              }
            },
            "metrics": {
              "description": "Analyze changes in metrics. DO NOT UNDER ANY CIRCUMSTANCES use date or metric fields for groupBy, leave empty unless needed.",
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "name": { "type": "string", "description": "The name of this set of metrics" },
                  "index": { "type": "string", "description": "The index or index pattern where to find the metrics" },
                  "kqlFilter": {
                    "type": "string",
                    "description": "A KQL filter to filter the log documents by, e.g. my_field:foo"
                  },
                  "field": {
                    "type": "string",
                    "description": "Metric field that contains the metric. Only use if the metric aggregation type is not count."
                  },
                  "type": {
                    "type": "string",
                    "description": "The type of metric aggregation to perform. Defaults to count",
                    "enum": ["count", "avg", "sum", "min", "max", "p95", "p99"]
                  },
                  "groupBy": {
                    "type": "array",
                    "description": "Optional keyword fields to group metrics by.",
                    "items": { "type": "string" }
                  }
                },
                "required": ["index", "name"]
              }
            }
          },
          "required": ["start", "end"]
        }
      },
      "type": "function"
    },
    {
      "function": {
        "name": "summarize",
        "description": "Use this function to store facts in the knowledge database if the user requests it.\n        You can score the learnings with a confidence metric, whether it is a correction on a previous learning.\n        An embedding will be created that you can recall later with a semantic search.\n        When you create this summarisation, make sure you craft it in a way that can be recalled with a semantic\n        search later, and that it would have answered the user's original request.",
        "parameters": {
          "type": "object",
          "properties": {
            "id": {
              "type": "string",
              "description": "An id for the document. This should be a short human-readable keyword field with only alphabetic characters and underscores, that allow you to update it later."
            },
            "text": {
              "type": "string",
              "description": "A human-readable summary of what you have learned, described in such a way that you can recall it later with semantic search, and that it would have answered the user's original request."
            },
            "is_correction": {
              "type": "boolean",
              "description": "Whether this is a correction for a previous learning."
            },
            "confidence": {
              "type": "string",
              "description": "How confident you are about this being a correct and useful learning",
              "enum": ["low", "medium", "high"]
            },
            "public": {
              "type": "boolean",
              "description": "Whether this information is specific to the user, or generally applicable to any user of the product"
            }
          },
          "required": ["id", "text", "is_correction", "confidence", "public"]
        }
      },
      "type": "function"
    },
    {
      "function": {
        "name": "elasticsearch",
        "description": "Call Elasticsearch APIs on behalf of the user. Make sure the request body is valid for the API that you are using. Only call this function when the user has explicitly requested it.",
        "parameters": {
          "type": "object",
          "properties": {
            "method": {
              "type": "string",
              "description": "The HTTP method of the Elasticsearch endpoint",
              "enum": ["GET", "PUT", "POST", "DELETE", "PATCH"]
            },
            "path": {
              "type": "string",
              "description": "The path of the Elasticsearch endpoint, including query parameters"
            },
            "body": { "type": "object", "description": "The body of the request" }
          },
          "required": ["method", "path"]
        }
      },
      "type": "function"
    },
    {
      "function": {
        "name": "kibana",
        "description": "Call Kibana APIs on behalf of the user. Only call this function when the user has explicitly requested it, and you know how to call it, for example by querying the knowledge base or having the user explain it to you. Assume that pathnames, bodies and query parameters may have changed since your knowledge cut off date.",
        "parameters": {
          "type": "object",
          "properties": {
            "method": {
              "type": "string",
              "description": "The HTTP method of the Kibana endpoint",
              "enum": ["GET", "PUT", "POST", "DELETE", "PATCH"]
            },
            "pathname": {
              "type": "string",
              "description": "The pathname of the Kibana endpoint, excluding query parameters"
            },
            "query": { "type": "object", "description": "The query parameters, as an object" },
            "body": { "type": "object", "description": "The body of the request" }
          },
          "required": ["method", "pathname"]
        }
      },
      "type": "function"
    },
    {
      "function": {
        "name": "get_dataset_info",
        "description": "Use this function to get information about indices/datasets available and the fields available on them.\n\n      providing empty string as index name will retrieve all indices\n      else list of all fields for the given index will be given. if no fields are returned this means no indices were matched by provided index pattern.\n      wildcards can be part of index name.",
        "parameters": {
          "type": "object",
          "properties": {
            "index": {
              "type": "string",
              "description": "index pattern the user is interested in or empty string to get information about all available indices"
            }
          },
          "required": ["index"]
        }
      },
      "type": "function"
    },
    {
      "function": {
        "name": "execute_connector",
        "description": "Use this function when user explicitly asks to call a kibana connector.",
        "parameters": {
          "type": "object",
          "properties": {
            "id": { "type": "string", "description": "The id of the connector" },
            "params": { "type": "object", "description": "The connector parameters" }
          },
          "required": ["id", "params"]
        }
      },
      "type": "function"
    }
  ],
  "temperature": 0
}
```
</details>

_OpenAI request object **after** the above update:_
<details>
  <summary>Click to expand JSON</summary>

```json
{
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant for Elastic Observability. Your goal is to help the Elastic Observability users to quickly assess what is happening in their observed systems. You can help them visualise and analyze data, investigate their systems, perform root cause analysis or identify optimisation opportunities.\n\n  It's very important to not assume what the user is meaning. Ask them for clarification if needed.\n\n  If you are unsure about which function should be used and with what arguments, ask the user for clarification or confirmation.\n\n  In KQL (\"kqlFilter\")) escaping happens with double quotes, not single quotes. Some characters that need escaping are: ':()\\  /\". Always put a field value in double quotes. Best: service.name:\"opbeans-go\". Wrong: service.name:opbeans-go. This is very important!\n\n  You can use Github-flavored Markdown in your responses. If a function returns an array, consider using a Markdown table to format the response.\n\n  Note that ES|QL (the Elasticsearch Query Language which is a new piped language) is the preferred query language.\n\n  If you want to call a function or tool, only call it a single time per message. Wait until the function has been executed and its results\n  returned to you, before executing the same tool or another tool again if needed.\n\n  DO NOT UNDER ANY CIRCUMSTANCES USE ES|QL syntax (`service.name == \"foo\"`) with \"kqlFilter\" (`service.name:\"foo\"`).\n\n  The user is able to change the language which they want you to reply in on the settings page of the AI Assistant for Observability, which can be found in the Stack Management app under the option AI Assistants.\n  If the user asks how to change the language, reply in the same language the user asked in.\n\nYou MUST use the \"query\" function when the user wants to:\n  - visualize data\n  - run any arbitrary query\n  - breakdown or filter ES|QL queries that are displayed on the current page\n  - convert queries from another language to ES|QL\n  - asks general questions about ES|QL\n\n  DO NOT UNDER ANY CIRCUMSTANCES generate ES|QL queries or explain anything about the ES|QL query language yourself.\n  DO NOT UNDER ANY CIRCUMSTANCES try to correct an ES|QL query yourself - always use the \"query\" function for this.\n\n  If the user asks for a query, and one of the dataset info functions was called and returned no results, you should still call the query function to generate an example query.\n\n  Even if the \"query\" function was used before that, follow it up with the \"query\" function. If a query fails, do not attempt to correct it yourself. Again you should call the \"query\" function,\n  even if it has been called before.\n\n  When the \"visualize_query\" function has been called, a visualization has been displayed to the user. DO NOT UNDER ANY CIRCUMSTANCES follow up a \"visualize_query\" function call with your own visualization attempt.\n  If the \"execute_query\" function has been called, summarize these results for the user. The user does not see a visualization in this case.\n\nYou MUST use the \"get_dataset_info\"  function before calling the \"query\" or the \"changes\" functions.\n\nIf a function requires an index, you MUST use the results from the dataset info functions.\n\nYou have access to data on the screen by calling the \"get_data_on_screen\" function.\nUse it to help the user understand what they are looking at. A short summary of what they are looking at is available in the return of the \"context\" function.\nData that is compact enough automatically gets included in the response for the \"context\" function.\n\nYou can use the \"summarize\" function to store new information you have learned in a knowledge database.\nOnly use this function when the user asks for it.\nAll summaries MUST be created in English, even if the conversation was carried out in a different language.\nThe \"get_data_on_screen\" function will Get data that is on the screen:\n.es-query: An available rule is Elasticsearch query.\nobservability.rules.custom_threshold: An available rule is Custom threshold.\nxpack.ml.anomaly_detection_alert: An available rule is Anomaly detection.\nslo.rules.burnRate: An available rule is SLO burn rate.\nmetrics.alert.threshold: An available rule is Metric threshold.\nmetrics.alert.inventory.threshold: An available rule is Inventory.\nlogs.alert.document.count: An available rule is Log threshold.\nxpack.uptime.alerts.tlsCertificate: An available rule is Uptime TLS.\nxpack.uptime.alerts.monitorStatus: An available rule is Uptime monitor status.\nxpack.uptime.alerts.durationAnomaly: An available rule is Uptime Duration Anomaly.\nxpack.synthetics.alerts.monitorStatus: An available rule is Synthetics monitor status.\nxpack.synthetics.alerts.tls: An available rule is Synthetics TLS certificate.\napm.error_rate: An available rule is Error count threshold.\napm.transaction_error_rate: An available rule is Failed transaction rate threshold.\napm.transaction_duration: An available rule is Latency threshold.\napm.anomaly: An available rule is APM Anomaly."
    },
    { "role": "user", "content": "Can you explain this page?" },
    { "role": "assistant", "content": "", "function_call": { "name": "context", "arguments": "{}" } },
    {
      "role": "user",
      "content": "{\"screen_description\":\"The user is looking at http://localhost:5601/phq/app/observability/alerts?_a=(filters:!(),kuery:%27%27,rangeFrom:now-24h,rangeTo:now,status:all). The current time range is 2024-10-21T18:23:54.539Z - 2024-10-21T18:38:54.539Z.\",\"learnings\":[],\"data_on_screen\":[{\"name\":\".es-query\",\"value\":\"Elasticsearch query Alert when matches are found during the latest query run.\",\"description\":\"An available rule is Elasticsearch query.\"},{\"name\":\"observability.rules.custom_threshold\",\"value\":\"Custom threshold Alert when any Observability data type reaches or exceeds a given value.\",\"description\":\"An available rule is Custom threshold.\"},{\"name\":\"xpack.ml.anomaly_detection_alert\",\"value\":\"Anomaly detection Alert when anomaly detection jobs results match the condition.\",\"description\":\"An available rule is Anomaly detection.\"},{\"name\":\"slo.rules.burnRate\",\"value\":\"SLO burn rate Alert when your SLO burn rate is too high over a defined period of time.\",\"description\":\"An available rule is SLO burn rate.\"},{\"name\":\"metrics.alert.threshold\",\"value\":\"Metric threshold Alert when the metrics aggregation exceeds the threshold.\",\"description\":\"An available rule is Metric threshold.\"},{\"name\":\"metrics.alert.inventory.threshold\",\"value\":\"Inventory Alert when the inventory exceeds a defined threshold.\",\"description\":\"An available rule is Inventory.\"},{\"name\":\"logs.alert.document.count\",\"value\":\"Log threshold Alert when the log aggregation exceeds the threshold.\",\"description\":\"An available rule is Log threshold.\"},{\"name\":\"xpack.uptime.alerts.tlsCertificate\",\"value\":\"Uptime TLS Alert when the TLS certificate of an Uptime monitor is about to expire.\",\"description\":\"An available rule is Uptime TLS.\"},{\"name\":\"xpack.uptime.alerts.monitorStatus\",\"value\":\"Uptime monitor status Alert when a monitor is down or an availability threshold is breached.\",\"description\":\"An available rule is Uptime monitor status.\"},{\"name\":\"xpack.uptime.alerts.durationAnomaly\",\"value\":\"Uptime Duration Anomaly Alert when the Uptime monitor duration is anomalous.\",\"description\":\"An available rule is Uptime Duration Anomaly.\"},{\"name\":\"xpack.synthetics.alerts.monitorStatus\",\"value\":\"Synthetics monitor status Alert when a monitor is down.\",\"description\":\"An available rule is Synthetics monitor status.\"},{\"name\":\"xpack.synthetics.alerts.tls\",\"value\":\"Synthetics TLS certificate Alert when the TLS certificate of a Synthetics monitor is about to expire.\",\"description\":\"An available rule is Synthetics TLS certificate.\"},{\"name\":\"apm.error_rate\",\"value\":\"Error count threshold Alert when the number of errors in a service exceeds a defined threshold.\",\"description\":\"An available rule is Error count threshold.\"},{\"name\":\"apm.transaction_error_rate\",\"value\":\"Failed transaction rate threshold Alert when the rate of transaction errors in a service exceeds a defined threshold.\",\"description\":\"An available rule is Failed transaction rate threshold.\"},{\"name\":\"apm.transaction_duration\",\"value\":\"Latency threshold Alert when the latency of a specific transaction type in a service exceeds a defined threshold.\",\"description\":\"An available rule is Latency threshold.\"},{\"name\":\"apm.anomaly\",\"value\":\"APM Anomaly Alert when either the latency, throughput, or failed transaction rate of a service is anomalous.\",\"description\":\"An available rule is APM Anomaly.\"}]}",
      "name": "context"
    }
  ],
  "stream": true,
  "tools": [
    {
      "function": {
        "name": "get_data_on_screen",
        "parameters": {
          "type": "object",
          "properties": {
            "data": {
              "type": "array",
              "description": "The pieces of data you want to look at it. You can request one, or multiple",
              "items": {
                "type": "string",
                "enum": [
                  ".es-query",
                  "observability.rules.custom_threshold",
                  "xpack.ml.anomaly_detection_alert",
                  "slo.rules.burnRate",
                  "metrics.alert.threshold",
                  "metrics.alert.inventory.threshold",
                  "logs.alert.document.count",
                  "xpack.uptime.alerts.tlsCertificate",
                  "xpack.uptime.alerts.monitorStatus",
                  "xpack.uptime.alerts.durationAnomaly",
                  "xpack.synthetics.alerts.monitorStatus",
                  "xpack.synthetics.alerts.tls",
                  "apm.error_rate",
                  "apm.transaction_error_rate",
                  "apm.transaction_duration",
                  "apm.anomaly"
                ]
              }
            }
          },
          "required": ["data"]
        }
      },
      "type": "function"
    },
    {
      "function": {
        "name": "query",
        "description": "This function generates, executes and/or visualizes a query\n      based on the user's request. It also explains how ES|QL works and how to\n      convert queries from one language to another. Make sure you call one of\n      the get_dataset functions first if you need index or field names. This\n      function takes no input.",
        "parameters": { "type": "object", "properties": {} }
      },
      "type": "function"
    },
    {
      "function": {
        "name": "get_alerts_dataset_info",
        "description": "Use this function to get information about alerts data.",
        "parameters": {
          "type": "object",
          "properties": {
            "start": {
              "type": "string",
              "description": "The start of the current time range, in datemath, like now-24h or an ISO timestamp"
            },
            "end": {
              "type": "string",
              "description": "The end of the current time range, in datemath, like now-24h or an ISO timestamp"
            }
          }
        }
      },
      "type": "function"
    },
    {
      "function": {
        "name": "alerts",
        "description": "Get alerts for Observability.  Make sure get_alerts_dataset_info was called before.\n        Use this to get open (and optionally recovered) alerts for Observability assets, like services,\n        hosts or containers.\n        Display the response in tabular format if appropriate.\n      ",
        "parameters": {
          "type": "object",
          "properties": {
            "start": {
              "type": "string",
              "description": "The start of the time range, in Elasticsearch date math, like `now`."
            },
            "end": {
              "type": "string",
              "description": "The end of the time range, in Elasticsearch date math, like `now-24h`."
            },
            "kqlFilter": { "type": "string", "description": "Filter alerts by field:value pairs" },
            "includeRecovered": {
              "type": "boolean",
              "description": "Whether to include recovered/closed alerts. Defaults to false, which means only active alerts will be returned"
            }
          },
          "required": ["start", "end"]
        }
      },
      "type": "function"
    },
    {
      "function": {
        "name": "changes",
        "description": "Returns change points like spikes and dips for logs and metrics.",
        "parameters": {
          "type": "object",
          "properties": {
            "start": {
              "type": "string",
              "description": "The beginning of the time range, in datemath, like now-24h, or an ISO timestamp"
            },
            "end": {
              "type": "string",
              "description": "The end of the time range, in datemath, like now, or an ISO timestamp"
            },
            "logs": {
              "description": "Analyze changes in log patterns. If no index is given, the default logs index pattern will be used",
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "name": { "type": "string", "description": "The name of this set of logs" },
                  "index": { "type": "string", "description": "The index or index pattern where to find the logs" },
                  "kqlFilter": {
                    "type": "string",
                    "description": "A KQL filter to filter the log documents by, e.g. my_field:foo"
                  },
                  "field": {
                    "type": "string",
                    "description": "The text field that contains the message to be analyzed, usually `message`. ONLY use field names from the conversation."
                  }
                },
                "required": ["name"]
              }
            },
            "metrics": {
              "description": "Analyze changes in metrics. DO NOT UNDER ANY CIRCUMSTANCES use date or metric fields for groupBy, leave empty unless needed.",
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "name": { "type": "string", "description": "The name of this set of metrics" },
                  "index": { "type": "string", "description": "The index or index pattern where to find the metrics" },
                  "kqlFilter": {
                    "type": "string",
                    "description": "A KQL filter to filter the log documents by, e.g. my_field:foo"
                  },
                  "field": {
                    "type": "string",
                    "description": "Metric field that contains the metric. Only use if the metric aggregation type is not count."
                  },
                  "type": {
                    "type": "string",
                    "description": "The type of metric aggregation to perform. Defaults to count",
                    "enum": ["count", "avg", "sum", "min", "max", "p95", "p99"]
                  },
                  "groupBy": {
                    "type": "array",
                    "description": "Optional keyword fields to group metrics by.",
                    "items": { "type": "string" }
                  }
                },
                "required": ["index", "name"]
              }
            }
          },
          "required": ["start", "end"]
        }
      },
      "type": "function"
    },
    {
      "function": {
        "name": "summarize",
        "description": "Use this function to store facts in the knowledge database if the user requests it.\n        You can score the learnings with a confidence metric, whether it is a correction on a previous learning.\n        An embedding will be created that you can recall later with a semantic search.\n        When you create this summarisation, make sure you craft it in a way that can be recalled with a semantic\n        search later, and that it would have answered the user's original request.",
        "parameters": {
          "type": "object",
          "properties": {
            "id": {
              "type": "string",
              "description": "An id for the document. This should be a short human-readable keyword field with only alphabetic characters and underscores, that allow you to update it later."
            },
            "text": {
              "type": "string",
              "description": "A human-readable summary of what you have learned, described in such a way that you can recall it later with semantic search, and that it would have answered the user's original request."
            },
            "is_correction": {
              "type": "boolean",
              "description": "Whether this is a correction for a previous learning."
            },
            "confidence": {
              "type": "string",
              "description": "How confident you are about this being a correct and useful learning",
              "enum": ["low", "medium", "high"]
            },
            "public": {
              "type": "boolean",
              "description": "Whether this information is specific to the user, or generally applicable to any user of the product"
            }
          },
          "required": ["id", "text", "is_correction", "confidence", "public"]
        }
      },
      "type": "function"
    },
    {
      "function": {
        "name": "elasticsearch",
        "description": "Call Elasticsearch APIs on behalf of the user. Make sure the request body is valid for the API that you are using. Only call this function when the user has explicitly requested it.",
        "parameters": {
          "type": "object",
          "properties": {
            "method": {
              "type": "string",
              "description": "The HTTP method of the Elasticsearch endpoint",
              "enum": ["GET", "PUT", "POST", "DELETE", "PATCH"]
            },
            "path": {
              "type": "string",
              "description": "The path of the Elasticsearch endpoint, including query parameters"
            },
            "body": { "type": "object", "description": "The body of the request" }
          },
          "required": ["method", "path"]
        }
      },
      "type": "function"
    },
    {
      "function": {
        "name": "kibana",
        "description": "Call Kibana APIs on behalf of the user. Only call this function when the user has explicitly requested it, and you know how to call it, for example by querying the knowledge base or having the user explain it to you. Assume that pathnames, bodies and query parameters may have changed since your knowledge cut off date.",
        "parameters": {
          "type": "object",
          "properties": {
            "method": {
              "type": "string",
              "description": "The HTTP method of the Kibana endpoint",
              "enum": ["GET", "PUT", "POST", "DELETE", "PATCH"]
            },
            "pathname": {
              "type": "string",
              "description": "The pathname of the Kibana endpoint, excluding query parameters"
            },
            "query": { "type": "object", "description": "The query parameters, as an object" },
            "body": { "type": "object", "description": "The body of the request" }
          },
          "required": ["method", "pathname"]
        }
      },
      "type": "function"
    },
    {
      "function": {
        "name": "get_dataset_info",
        "description": "Use this function to get information about indices/datasets available and the fields available on them.\n\n      providing empty string as index name will retrieve all indices\n      else list of all fields for the given index will be given. if no fields are returned this means no indices were matched by provided index pattern.\n      wildcards can be part of index name.",
        "parameters": {
          "type": "object",
          "properties": {
            "index": {
              "type": "string",
              "description": "index pattern the user is interested in or empty string to get information about all available indices"
            }
          },
          "required": ["index"]
        }
      },
      "type": "function"
    },
    {
      "function": {
        "name": "execute_connector",
        "description": "Use this function when user explicitly asks to call a kibana connector.",
        "parameters": {
          "type": "object",
          "properties": {
            "id": { "type": "string", "description": "The id of the connector" },
            "params": { "type": "object", "description": "The connector parameters" }
          },
          "required": ["id", "params"]
        }
      },
      "type": "function"
    }
  ],
  "temperature": 0
}
```
</details>

---------

Co-authored-by: Søren Louv-Jansen <[email protected]>
(cherry picked from commit 354bec4)
  • Loading branch information
viduni94 committed Oct 31, 2024
1 parent 5f2506f commit 6cc7534
Show file tree
Hide file tree
Showing 10 changed files with 94 additions and 12 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ const getFunctionsRoute = createObservabilityAIAssistantServerRoute({
systemMessage: getSystemMessageFromInstructions({
applicationInstructions: functionClient.getInstructions(),
userInstructions,
adHocInstructions: [],
adHocInstructions: functionClient.getAdhocInstructions(),
availableFunctionNames,
}),
};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
import dedent from 'dedent';
import { ChatFunctionClient, GET_DATA_ON_SCREEN_FUNCTION_NAME } from '.';
import { FunctionVisibility } from '../../../common/functions/types';
import { AdHocInstruction } from '../../../common/types';

describe('chatFunctionClient', () => {
describe('when executing a function with invalid arguments', () => {
Expand Down Expand Up @@ -86,6 +87,7 @@ describe('chatFunctionClient', () => {
]);

const functions = client.getFunctions();
const adHocInstructions = client.getAdhocInstructions();

expect(functions[0]).toEqual({
definition: {
Expand All @@ -97,7 +99,7 @@ describe('chatFunctionClient', () => {
respond: expect.any(Function),
});

expect(functions[0].definition.description).toContain(
expect(adHocInstructions[0].text).toContain(
dedent(`my_dummy_data: My dummy data
my_other_dummy_data: My other dummy data
`)
Expand Down Expand Up @@ -128,4 +130,52 @@ describe('chatFunctionClient', () => {
});
});
});

describe('when adhoc instructions are provided', () => {
let client: ChatFunctionClient;

beforeEach(() => {
client = new ChatFunctionClient([]);
});

describe('register an adhoc Instruction', () => {
it('should register a new adhoc instruction', () => {
const adhocInstruction: AdHocInstruction = {
text: 'Test adhoc instruction',
instruction_type: 'application_instruction',
};

client.registerAdhocInstruction(adhocInstruction);

expect(client.getAdhocInstructions()).toContainEqual(adhocInstruction);
});
});

describe('retrieve adHoc instructions', () => {
it('should return all registered adhoc instructions', () => {
const firstAdhocInstruction: AdHocInstruction = {
text: 'First adhoc instruction',
instruction_type: 'application_instruction',
};

const secondAdhocInstruction: AdHocInstruction = {
text: 'Second adhoc instruction',
instruction_type: 'application_instruction',
};

client.registerAdhocInstruction(firstAdhocInstruction);
client.registerAdhocInstruction(secondAdhocInstruction);

const adhocInstructions = client.getAdhocInstructions();

expect(adhocInstructions).toEqual([firstAdhocInstruction, secondAdhocInstruction]);
});

it('should return an empty array if no adhoc instructions are registered', () => {
const adhocInstructions = client.getAdhocInstructions();

expect(adhocInstructions).toEqual([]);
});
});
});
});
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,18 @@ import Ajv, { type ErrorObject, type ValidateFunction } from 'ajv';
import dedent from 'dedent';
import { compact, keyBy } from 'lodash';
import { FunctionVisibility, type FunctionResponse } from '../../../common/functions/types';
import type { Message, ObservabilityAIAssistantScreenContextRequest } from '../../../common/types';
import type {
AdHocInstruction,
Message,
ObservabilityAIAssistantScreenContextRequest,
} from '../../../common/types';
import { filterFunctionDefinitions } from '../../../common/utils/filter_function_definitions';
import type {
FunctionCallChatFunction,
FunctionHandler,
FunctionHandlerRegistry,
InstructionOrCallback,
RegisterAdHocInstruction,
RegisterFunction,
RegisterInstruction,
} from '../types';
Expand All @@ -35,6 +40,8 @@ export const GET_DATA_ON_SCREEN_FUNCTION_NAME = 'get_data_on_screen';

export class ChatFunctionClient {
private readonly instructions: InstructionOrCallback[] = [];
private readonly adhocInstructions: AdHocInstruction[] = [];

private readonly functionRegistry: FunctionHandlerRegistry = new Map();
private readonly validators: Map<string, ValidateFunction> = new Map();

Expand All @@ -49,9 +56,7 @@ export class ChatFunctionClient {
this.registerFunction(
{
name: GET_DATA_ON_SCREEN_FUNCTION_NAME,
description: dedent(`Get data that is on the screen:
${allData.map((data) => `${data.name}: ${data.description}`).join('\n')}
`),
description: `Retrieve the structured data of content currently visible on the user's screen. Use this tool to understand what the user is viewing at this moment to provide more accurate and context-aware responses to their questions.`,
visibility: FunctionVisibility.AssistantOnly,
parameters: {
type: 'object',
Expand All @@ -75,6 +80,13 @@ export class ChatFunctionClient {
};
}
);

this.registerAdhocInstruction({
text: `The ${GET_DATA_ON_SCREEN_FUNCTION_NAME} function will retrieve specific content from the user's screen by specifying a data key. Use this tool to provide context-aware responses. Available data: ${dedent(
allData.map((data) => `${data.name}: ${data.description}`).join('\n')
)}`,
instruction_type: 'application_instruction',
});
}

this.actions.forEach((action) => {
Expand All @@ -95,6 +107,10 @@ export class ChatFunctionClient {
this.instructions.push(instruction);
};

registerAdhocInstruction: RegisterAdHocInstruction = (instruction: AdHocInstruction) => {
this.adhocInstructions.push(instruction);
};

validate(name: string, parameters: unknown) {
const validator = this.validators.get(name)!;
if (!validator) {
Expand All @@ -111,6 +127,10 @@ export class ChatFunctionClient {
return this.instructions;
}

getAdhocInstructions(): AdHocInstruction[] {
return this.adhocInstructions;
}

hasAction(name: string) {
return !!this.actions.find((action) => action.name === name)!;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ describe('Observability AI Assistant client', () => {
getActions: jest.fn(),
validate: jest.fn(),
getInstructions: jest.fn(),
getAdhocInstructions: jest.fn(),
} as any;

let llmSimulator: LlmSimulator;
Expand Down Expand Up @@ -173,6 +174,7 @@ describe('Observability AI Assistant client', () => {
knowledgeBaseServiceMock.getUserInstructions.mockResolvedValue([]);

functionClientMock.getInstructions.mockReturnValue(['system']);
functionClientMock.getAdhocInstructions.mockReturnValue([]);

return new ObservabilityAIAssistantClient({
actionsClient: actionsClientMock,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,12 +47,12 @@ import {
} from '../../../common/conversation_complete';
import { CompatibleJSONSchema } from '../../../common/functions/types';
import {
AdHocInstruction,
type Conversation,
type ConversationCreateRequest,
type ConversationUpdateRequest,
type KnowledgeBaseEntry,
type Message,
type AdHocInstruction,
} from '../../../common/types';
import { withoutTokenCountEvents } from '../../../common/utils/without_token_count_events';
import { CONTEXT_FUNCTION_NAME } from '../../functions/context';
Expand Down Expand Up @@ -210,6 +210,9 @@ export class ObservabilityAIAssistantClient {

const userInstructions$ = from(this.getKnowledgeBaseUserInstructions()).pipe(shareReplay());

const registeredAdhocInstructions = functionClient.getAdhocInstructions();
const allAdHocInstructions = adHocInstructions.concat(registeredAdhocInstructions);

// from the initial messages, override any system message with
// the one that is based on the instructions (registered, request, kb)
const messagesWithUpdatedSystemMessage$ = userInstructions$.pipe(
Expand All @@ -219,7 +222,7 @@ export class ObservabilityAIAssistantClient {
getSystemMessageFromInstructions({
applicationInstructions: functionClient.getInstructions(),
userInstructions,
adHocInstructions,
adHocInstructions: allAdHocInstructions,
availableFunctionNames: functionClient
.getFunctions()
.map((fn) => fn.definition.name),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ export function continueConversation({
chat,
signal,
functionCallsLeft,
adHocInstructions,
adHocInstructions = [],
userInstructions,
logger,
disableFunctions,
Expand Down Expand Up @@ -213,11 +213,14 @@ export function continueConversation({
disableFunctions,
});

const registeredAdhocInstructions = functionClient.getAdhocInstructions();
const allAdHocInstructions = adHocInstructions.concat(registeredAdhocInstructions);

const messagesWithUpdatedSystemMessage = replaceSystemMessage(
getSystemMessageFromInstructions({
applicationInstructions: functionClient.getInstructions(),
userInstructions,
adHocInstructions,
adHocInstructions: allAdHocInstructions,
availableFunctionNames: definitions.map((def) => def.name),
}),
initialMessages
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import type {
Message,
ObservabilityAIAssistantScreenContextRequest,
InstructionOrPlainText,
AdHocInstruction,
} from '../../common/types';
import type { ObservabilityAIAssistantRouteHandlerResources } from '../routes/types';
import { ChatFunctionClient } from './chat_function_client';
Expand Down Expand Up @@ -76,6 +77,8 @@ export type RegisterInstructionCallback = ({

export type RegisterInstruction = (...instruction: InstructionOrCallback[]) => void;

export type RegisterAdHocInstruction = (...instruction: AdHocInstruction[]) => void;

export type RegisterFunction = <
TParameters extends CompatibleJSONSchema = any,
TResponse extends FunctionResponse = any,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ export function getSystemMessageFromInstructions({

const adHocInstructionsWithId = adHocInstructions.map((adHocInstruction) => ({
...adHocInstruction,
doc_id: adHocInstruction.doc_id ?? v4(),
doc_id: adHocInstruction?.doc_id ?? v4(),
}));

// split ad hoc instructions into user instructions and application instructions
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ describe('observabilityAIAssistant rule_connector', () => {
getFunctionClient: async () => ({
getFunctions: () => [],
getInstructions: () => [],
getAdhocInstructions: () => [],
}),
},
context: {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,7 @@ If available, include the link of the conversation at the end of your answer.`
availableFunctionNames: functionClient.getFunctions().map((fn) => fn.definition.name),
applicationInstructions: functionClient.getInstructions(),
userInstructions: [],
adHocInstructions: [],
adHocInstructions: functionClient.getAdhocInstructions(),
}),
},
},
Expand Down

0 comments on commit 6cc7534

Please sign in to comment.