+
+
+**SDK:**
+
+- Allow users to configure the project name in the LangChain integration
diff --git a/apps/opik-documentation/documentation/fern/docs/changelog/10-18-24.mdx b/apps/opik-documentation/documentation/fern/docs/changelog/10-18-24.mdx
new file mode 100644
index 0000000000..1b97eab2e8
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/changelog/10-18-24.mdx
@@ -0,0 +1,11 @@
+**Opik Dashboard**:
+
+- Added a new `Feedback modal` in the UI so you can easily provide feedback on any parts of the platform.
+
+**SDK**:
+
+- Released new evaluation metric: [GEval](/evaluation/metrics/g_eval) - This LLM as a Judge metric is task agnostic and can be used to evaluate any LLM call based on your own custom evaluation criteria.
+- Allow users to specify the path to the Opik configuration file using the `OPIK_CONFIG_PATH` environment variable, read more about it in the [Python SDK Configuration guide](/tracing/sdk_configuration#using-a-configuration-file).
+- You can now configure the `project_name` as part of the `evaluate` method so that traces are logged to a specific project instead of the default one.
+- Added a new `Opik.search_traces` method to search for traces, this includes support for a search string to return only specific traces.
+- Enforce structured outputs for LLM as a Judge metrics so that they are more reliable (they will no longer fail when decoding the LLM response).
diff --git a/apps/opik-documentation/documentation/fern/docs/changelog/10-21-24.mdx b/apps/opik-documentation/documentation/fern/docs/changelog/10-21-24.mdx
new file mode 100644
index 0000000000..ef8d77f950
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/changelog/10-21-24.mdx
@@ -0,0 +1,33 @@
+**Opik Dashboard**:
+
+- Added the option to download traces and LLM calls as CSV files from the UI:
+
+ 
+
+- Introduce a new quickstart guide to help you get started:
+
+ 
+
+- Updated datasets to support more flexible data schema, you can now insert items with any key value pairs and not just `input` and `expected_output`. See more in the SDK section below.
+- Multiple small UX improvements (more informative empty state for projects, updated icons, feedback tab in the experiment page, etc).
+- Fix issue with `\t` characters breaking the YAML code block in the traces page.
+
+**SDK**:
+
+- Datasets now support more flexible data schema, we now support inserting items with any key value pairs:
+
+ ```python
+ import opik
+
+ client = opik.Opik()
+ dataset = client.get_or_create_dataset(name="Demo Dataset")
+ dataset.insert([
+ {"user_question": "Hello, what can you do ?", "expected_output": {"assistant_answer": "I am a chatbot assistant that can answer questions and help you with your queries!"}},
+ {"user_question": "What is the capital of France?", "expected_output": {"assistant_answer": "Paris"}},
+ ])
+ ```
+
+- Released WatsonX, Gemini and Groq integration based on the LiteLLM integration.
+- The `context` field is now optional in the [Hallucination](/tracing/integrations/overview) metric.
+- LLM as a Judge metrics now support customizing the LLM provider by specifying the `model` parameter. See more in the [Customizing LLM as a Judge metrics](/evaluation/metrics/overview#customizing-llm-as-a-judge-metrics) section.
+- Fixed an issue when updating feedback scores using the `update_current_span` and `update_current_trace` methods. See this Github issue for more details.
diff --git a/apps/opik-documentation/documentation/fern/docs/changelog/11-04-24.mdx b/apps/opik-documentation/documentation/fern/docs/changelog/11-04-24.mdx
new file mode 100644
index 0000000000..0dc4189032
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/changelog/11-04-24.mdx
@@ -0,0 +1,12 @@
+**Opik Dashboard**:
+
+- Added a new `Prompt library` page to manage your prompts in the UI.
+
+ 
+
+
+**SDK**:
+
+- Introduced the `Prompt` object in the SDK to manage prompts stored in the library. See the [Prompt Management](/prompt_engineering/managing_prompts_in_code) guide for more details.
+- Introduced a `Opik.search_spans` method to search for spans in a project. See the [Search spans](/tracing/export_data#exporting-spans) guide for more details.
+- Released a new integration with [AWS Bedrock](/tracing/integrations/bedrock) for using Opik with Bedrock models.
diff --git a/apps/opik-documentation/documentation/fern/docs/changelog/11-11-24.mdx b/apps/opik-documentation/documentation/fern/docs/changelog/11-11-24.mdx
new file mode 100644
index 0000000000..a1ee3791ab
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/changelog/11-11-24.mdx
@@ -0,0 +1,36 @@
+**Opik Dashboard**:
+
+- Added the option to sort the projects table by `Last updated`, `Created at` and `Name` columns.
+- Updated the logic for displaying images, instead of relying on the format of the response, we now use regex rules to detect if the trace or span input includes a base64 encoded image or url.
+- Improved performance of the Traces table by truncating trace inputs and outputs if they contain base64 encoded images.
+- Fixed some issues with rendering trace input and outputs in YAML format.
+- Added grouping and charts to the experiments page:
+
+ 
+
+
+**SDK**:
+
+- **New integration**: Anthropic integration
+
+ ```python
+ from anthropic import Anthropic, AsyncAnthropic
+ from opik.integrations.anthropic import track_anthropic
+
+ client = Anthropic()
+ client = track_anthropic(client, project_name="anthropic-example")
+
+ message = client.messages.create(
+ max_tokens=1024,
+ messages=[
+ {
+ "role": "user",
+ "content": "Tell a fact",
+ }
+ ],
+ model="claude-3-opus-20240229",
+ )
+ print(message)
+ ```
+
+- Added a new `evaluate_experiment` method in the SDK that can be used to re-score an existing experiment, learn more in the [Update experiments](/evaluation/update_existing_experiment) guide.
diff --git a/apps/opik-documentation/documentation/fern/docs/changelog/11-18-24.mdx b/apps/opik-documentation/documentation/fern/docs/changelog/11-18-24.mdx
new file mode 100644
index 0000000000..bc507beda0
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/changelog/11-18-24.mdx
@@ -0,0 +1,12 @@
+**Opik Dashboard**:
+
+- Updated the majority of tables to increase the information density, it is now easier to review many traces at once.
+- Images logged to datasets and experiments are now displayed in the UI. Both images urls and base64 encoded images are supported.
+
+**SDK**:
+
+- The `scoring_metrics` argument is now optional in the `evaluate` method. This is useful if you are looking at evaluating your LLM calls manually in the Opik UI.
+- When uploading a dataset, the SDK now prints a link to the dataset in the UI.
+- Usage is now correctly logged when using the LangChain OpenAI integration.
+- Implement a batching mechanism for uploading spans and dataset items to avoid `413 Request Entity Too Large` errors.
+- Removed pandas and numpy as mandatory dependencies.
diff --git a/apps/opik-documentation/documentation/fern/docs/changelog/11-25-24.mdx b/apps/opik-documentation/documentation/fern/docs/changelog/11-25-24.mdx
new file mode 100644
index 0000000000..0aec7a9cfe
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/changelog/11-25-24.mdx
@@ -0,0 +1,26 @@
+**Opik Dashboard**:
+
+- Feedback scores are now displayed as separate columns in the traces and spans table
+- Introduce a new project dashboard to see trace count, feedback scores and token count over time.
+
+ 
+
+- Project statistics are now displayed in the traces and spans table header, this is especially useful for tracking the average feedback scores
+
+ 
+
+- Redesigned the experiment item sidebar to make it easier to review experiment results
+
+ 
+
+- Annotating feedback scores in the UI now feels much faster
+- Support exporting traces as JSON file in addition to CSV
+- Sidebars now close when clicking outside of them
+- Dataset groups in the experiment page are now sorted by last updated date
+- Updated scrollbar styles for Windows users
+
+**SDK**:
+
+- Improved the robustness to connection issues by adding retry logic.
+- Updated the OpenAI integration to track structured output calls using `beta.chat.completions.parse`.
+- Fixed issue with `update_current_span` and `update_current_trace` that did not support updating the `output` field.
diff --git a/apps/opik-documentation/documentation/fern/docs/changelog/12-02-24.mdx b/apps/opik-documentation/documentation/fern/docs/changelog/12-02-24.mdx
new file mode 100644
index 0000000000..24816af4f2
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/changelog/12-02-24.mdx
@@ -0,0 +1,13 @@
+**Opik Dashboard**:
+
+- Added a new `created_by` column for each table to indicate who created the record
+- Mask the API key in the user menu
+
+**SDK**:
+
+- Implement background batch sending of traces to speed up processing of trace creation requests
+- Updated OpenAI integration to track cost of LLM calls
+- Updated `prompt.format` method to raise an error when it is called with the wrong arguments
+- Updated the `Opik` method so it accepts the `api_key` parameter as a positional argument
+- Improved the prompt template for the `hallucination` metric
+- Introduced a new `opik_check_tls_certificate` configuration option to disable the TLS certificate check.
diff --git a/apps/opik-documentation/documentation/fern/docs/changelog/12-09-24.mdx b/apps/opik-documentation/documentation/fern/docs/changelog/12-09-24.mdx
new file mode 100644
index 0000000000..959afa7aee
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/changelog/12-09-24.mdx
@@ -0,0 +1,13 @@
+**Opik Dashboard**:
+
+- Updated the experiments pages to make it easier to analyze the results of each experiment. Columns are now organized based on where they came from (dataset, evaluation task, etc) and output keys are now displayed in multiple columns to make it easier to review
+
+ 
+
+- Improved the performance of the experiments so experiment items load faster
+- Added descriptions for projects
+
+**SDK**:
+
+- Add cost tracking for OpenAI calls made using LangChain
+- Fixed a timeout issue when calling `get_or_create_dataset`
diff --git a/apps/opik-documentation/documentation/fern/docs/changelog/12-16-24.mdx b/apps/opik-documentation/documentation/fern/docs/changelog/12-16-24.mdx
new file mode 100644
index 0000000000..9bb15b32c0
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/changelog/12-16-24.mdx
@@ -0,0 +1,18 @@
+**Opik Dashboard**:
+
+- The Opik playground is now in public preview
+
+ 
+
+- You can now view the prompt diff when updating a prompt from the UI
+- Errors in traces and spans are now displayed in the UI
+- Display agent graphs in the traces sidebar
+- Released a new plugin for the [Kong AI Gateway](/production/gateway)
+
+**SDK**:
+
+- Added support for serializing Pydantic models passed to decorated functions
+- Implemented `get_experiment_by_id` and `get_experiment_by_name` methods
+- Scoring metrics are now logged to the traces when using the `evaluate` method
+- New integration with [aisuite](/tracing/integrations/aisuite)
+- New integration with [Haystack](/tracing/integrations/haystack)
diff --git a/apps/opik-documentation/documentation/fern/docs/changelog/12-23-24.mdx b/apps/opik-documentation/documentation/fern/docs/changelog/12-23-24.mdx
new file mode 100644
index 0000000000..6e902869ef
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/changelog/12-23-24.mdx
@@ -0,0 +1,7 @@
+**SDK**:
+
+- Improved error messages when getting a rate limit when using the `evaluate` method
+- Added support for a new metadata field in the `Prompt` object, this field is used to store any additional information about the prompt.
+- Updated the library used to create uuidv7 IDs
+- New Guardrails integration
+- New DSPY integration
diff --git a/apps/opik-documentation/documentation/fern/docs/changelog/12-30-24.mdx b/apps/opik-documentation/documentation/fern/docs/changelog/12-30-24.mdx
new file mode 100644
index 0000000000..6a38b219be
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/changelog/12-30-24.mdx
@@ -0,0 +1,10 @@
+**Opik Dashboard**:
+
+- Added duration chart to the project dashboard
+- Prompt metadata can now be set and viewed in the UI, this can be used to store any additional information about the prompt
+- Playground prompts and settings are now cached when you navigate away from the page
+
+**SDK**:
+
+- Introduced a new `OPIK_TRACK_DISABLE` environment variable to disable the tracking of traces and spans
+- We now log usage information for traces logged using the LlamaIndex integration
diff --git a/apps/opik-documentation/documentation/fern/docs/cookbook/aisuite.mdx b/apps/opik-documentation/documentation/fern/docs/cookbook/aisuite.mdx
new file mode 100644
index 0000000000..d89ecd1472
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/cookbook/aisuite.mdx
@@ -0,0 +1,108 @@
+---
+description: Cookbook that showcases Opik's integration with the aisuite Python SDK
+---
+
+# Using Opik with aisuite
+
+Opik integrates with aisuite to provide a simple way to log traces for all aisuite LLM calls.
+
+
+## Creating an account on Comet.com
+
+[Comet](https://www.comet.com/site?from=llm&utm_source=opik&utm_medium=colab&utm_content=openai&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm&utm_source=opik&utm_medium=colab&utm_content=aisuite&utm_campaign=opik) and grab you API Key.
+
+> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm&utm_source=opik&utm_medium=colab&utm_content=aisuite&utm_campaign=opik) for more information.
+
+
+```python
+%pip install --upgrade opik "aisuite[openai]"
+```
+
+
+```python
+import opik
+
+opik.configure(use_local=False)
+```
+
+## Preparing our environment
+
+First, we will set up our OpenAI API keys.
+
+
+```python
+import os
+import getpass
+
+if "OPENAI_API_KEY" not in os.environ:
+ os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
+```
+
+## Logging traces
+
+In order to log traces to Opik, we need to wrap our OpenAI calls with the `track_openai` function:
+
+
+```python
+from opik.integrations.aisuite import track_aisuite
+import aisuite as ai
+
+client = track_aisuite(ai.Client(), project_name="aisuite-integration-demo")
+
+messages = [
+ {"role": "user", "content": "Write a short two sentence story about Opik."},
+]
+
+response = client.chat.completions.create(
+ model="openai:gpt-4o", messages=messages, temperature=0.75
+)
+print(response.choices[0].message.content)
+```
+
+The prompt and response messages are automatically logged to Opik and can be viewed in the UI.
+
+
+
+## Using it with the `track` decorator
+
+If you have multiple steps in your LLM pipeline, you can use the `track` decorator to log the traces for each step. If OpenAI is called within one of these steps, the LLM call with be associated with that corresponding step:
+
+
+```python
+from opik import track
+from opik.integrations.aisuite import track_aisuite
+import aisuite as ai
+
+client = track_aisuite(ai.Client(), project_name="aisuite-integration-demo")
+
+
+@track
+def generate_story(prompt):
+ res = client.chat.completions.create(
+ model="openai:gpt-3.5-turbo", messages=[{"role": "user", "content": prompt}]
+ )
+ return res.choices[0].message.content
+
+
+@track
+def generate_topic():
+ prompt = "Generate a topic for a story about Opik."
+ res = client.chat.completions.create(
+ model="openai:gpt-3.5-turbo", messages=[{"role": "user", "content": prompt}]
+ )
+ return res.choices[0].message.content
+
+
+@track(project_name="aisuite-integration-demo")
+def generate_opik_story():
+ topic = generate_topic()
+ story = generate_story(topic)
+ return story
+
+
+generate_opik_story()
+```
+
+The trace can now be viewed in the UI:
+
+
diff --git a/apps/opik-documentation/documentation/fern/docs/cookbook/anthropic.mdx b/apps/opik-documentation/documentation/fern/docs/cookbook/anthropic.mdx
new file mode 100644
index 0000000000..a5dc6b1e81
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/cookbook/anthropic.mdx
@@ -0,0 +1,122 @@
+---
+description: Cookbook that showcases Opik's integration with the Anthropic Python SDK
+---
+
+# Using Opik with Anthropic
+
+Opik integrates with Anthropic to provide a simple way to log traces for all Anthropic LLM calls. This works for all supported models, including if you are using the streaming API.
+
+
+## Creating an account on Comet.com
+
+[Comet](https://www.comet.com/site?from=llm&utm_source=opik&utm_medium=colab&utm_content=anthropic&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm&utm_source=opik&utm_medium=colab&utm_content=anthropic&utm_campaign=opik) and grab you API Key.
+
+> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm&utm_source=opik&utm_medium=colab&utm_content=anthropic&utm_campaign=opik) for more information.
+
+
+```python
+%pip install --upgrade opik anthropic
+```
+
+
+```python
+import opik
+
+opik.configure(use_local=False)
+```
+
+## Preparing our environment
+
+First, we will set up our anthropic client. You can [find or create your Anthropic API Key in this page page](https://console.anthropic.com/settings/keys) and paste it below:
+
+
+```python
+import os
+import getpass
+import anthropic
+
+if "ANTHROPIC_API_KEY" not in os.environ:
+ os.environ["ANTHROPIC_API_KEY"] = getpass.getpass("Enter your Anthropic API key: ")
+```
+
+## Logging traces
+
+In order to log traces to Opik, we need to wrap our Anthropic calls with the `track_anthropic` function:
+
+
+```python
+import os
+
+from opik.integrations.anthropic import track_anthropic
+
+anthropic_client = anthropic.Anthropic()
+anthropic_client = track_anthropic(anthropic, project_name="anthropic-integration-demo")
+```
+
+
+```python
+PROMPT = "Why is it important to use a LLM Monitoring like CometML Opik tool that allows you to log traces and spans when working with Anthropic LLM Models?"
+
+response = anthropic_client.messages.create(
+ model="claude-3-5-sonnet-20241022",
+ max_tokens=1024,
+ messages=[{"role": "user", "content": PROMPT}],
+)
+print("Response", response.content[0].text)
+```
+
+The prompt and response messages are automatically logged to Opik and can be viewed in the UI.
+
+
+
+## Using it with the `track` decorator
+
+If you have multiple steps in your LLM pipeline, you can use the `track` decorator to log the traces for each step. If Anthropic is called within one of these steps, the LLM call with be associated with that corresponding step:
+
+
+```python
+import anthropic
+
+from opik import track
+from opik.integrations.anthropic import track_anthropic
+
+os.environ["OPIK_PROJECT_NAME"] = "anthropic-integration-demo"
+
+anthropic_client = anthropic.Anthropic()
+anthropic_client = track_anthropic(anthropic)
+
+
+@track
+def generate_story(prompt):
+ res = anthropic_client.messages.create(
+ model="claude-3-5-sonnet-20241022",
+ max_tokens=1024,
+ messages=[{"role": "user", "content": prompt}],
+ )
+ return res.content[0].text
+
+
+@track
+def generate_topic():
+ prompt = "Generate a topic for a story about Opik."
+ res = anthropic_client.messages.create(
+ model="claude-3-5-sonnet-20241022",
+ max_tokens=1024,
+ messages=[{"role": "user", "content": prompt}],
+ )
+ return res.content[0].text
+
+
+@track
+def generate_opik_story():
+ topic = generate_topic()
+ story = generate_story(topic)
+ return story
+
+
+generate_opik_story()
+```
+
+The trace can now be viewed in the UI:
+
+
diff --git a/apps/opik-documentation/documentation/fern/docs/cookbook/bedrock.mdx b/apps/opik-documentation/documentation/fern/docs/cookbook/bedrock.mdx
new file mode 100644
index 0000000000..d0550ed7dc
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/cookbook/bedrock.mdx
@@ -0,0 +1,215 @@
+---
+description: Cookbook that showcases Opik's integration with AWS Bedrock
+---
+
+# Using Opik with AWS Bedrock
+
+Opik integrates with AWS Bedrock to provide a simple way to log traces for all Bedrock LLM calls. This works for all supported models, including if you are using the streaming API.
+
+
+## Creating an account on Comet.com
+
+[Comet](https://www.comet.com/site?from=llm&utm_source=opik&utm_medium=colab&utm_content=bedrock&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm&utm_source=opik&utm_medium=colab&utm_content=bedrock&utm_campaign=opik) and grab you API Key.
+
+> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm&utm_source=opik&utm_medium=colab&utm_content=bedrock&utm_campaign=opik) for more information.
+
+
+```python
+%pip install --upgrade opik boto3
+```
+
+
+```python
+import opik
+
+opik.configure(use_local=False)
+```
+
+## Preparing our environment
+
+First, we will set up our bedrock client. Uncomment the following lines to pass AWS Credentials manually or [checkout other ways of passing credentials to Boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html). You will also need to request access to the model in the UI before being able to generate text, here we are gonna use the Llama 3.2 model, you can request access to it in [this page for the us-east1](https://us-east-1.console.aws.amazon.com/bedrock/home?region=us-east-1#/providers?model=meta.llama3-2-3b-instruct-v1:0) region.
+
+
+```python
+import boto3
+
+REGION = "us-east-1"
+
+MODEL_ID = "us.meta.llama3-2-3b-instruct-v1:0"
+
+bedrock = boto3.client(
+ service_name="bedrock-runtime",
+ region_name=REGION,
+ # aws_access_key_id=ACCESS_KEY,
+ # aws_secret_access_key=SECRET_KEY,
+ # aws_session_token=SESSION_TOKEN,
+)
+```
+
+## Logging traces
+
+In order to log traces to Opik, we need to wrap our Bedrock calls with the `track_bedrock` function:
+
+
+```python
+import os
+
+from opik.integrations.bedrock import track_bedrock
+
+bedrock_client = track_bedrock(bedrock, project_name="bedrock-integration-demo")
+```
+
+
+```python
+PROMPT = "Why is it important to use a LLM Monitoring like CometML Opik tool that allows you to log traces and spans when working with LLM Models hosted on AWS Bedrock?"
+
+response = bedrock_client.converse(
+ modelId=MODEL_ID,
+ messages=[{"role": "user", "content": [{"text": PROMPT}]}],
+ inferenceConfig={"temperature": 0.5, "maxTokens": 512, "topP": 0.9},
+)
+print("Response", response["output"]["message"]["content"][0]["text"])
+```
+
+The prompt and response messages are automatically logged to Opik and can be viewed in the UI.
+
+
+
+# Logging traces with streaming
+
+
+```python
+def stream_conversation(
+ bedrock_client,
+ model_id,
+ messages,
+ system_prompts,
+ inference_config,
+):
+ """
+ Sends messages to a model and streams the response.
+ Args:
+ bedrock_client: The Boto3 Bedrock runtime client.
+ model_id (str): The model ID to use.
+ messages (JSON) : The messages to send.
+ system_prompts (JSON) : The system prompts to send.
+ inference_config (JSON) : The inference configuration to use.
+ additional_model_fields (JSON) : Additional model fields to use.
+
+ Returns:
+ Nothing.
+
+ """
+
+ response = bedrock_client.converse_stream(
+ modelId=model_id,
+ messages=messages,
+ system=system_prompts,
+ inferenceConfig=inference_config,
+ )
+
+ stream = response.get("stream")
+ if stream:
+ for event in stream:
+ if "messageStart" in event:
+ print(f"\nRole: {event['messageStart']['role']}")
+
+ if "contentBlockDelta" in event:
+ print(event["contentBlockDelta"]["delta"]["text"], end="")
+
+ if "messageStop" in event:
+ print(f"\nStop reason: {event['messageStop']['stopReason']}")
+
+ if "metadata" in event:
+ metadata = event["metadata"]
+ if "usage" in metadata:
+ print("\nToken usage")
+ print(f"Input tokens: {metadata['usage']['inputTokens']}")
+ print(f":Output tokens: {metadata['usage']['outputTokens']}")
+ print(f":Total tokens: {metadata['usage']['totalTokens']}")
+ if "metrics" in event["metadata"]:
+ print(f"Latency: {metadata['metrics']['latencyMs']} milliseconds")
+
+
+system_prompt = """You are an app that creates playlists for a radio station
+ that plays rock and pop music. Only return song names and the artist."""
+
+# Message to send to the model.
+input_text = "Create a list of 3 pop songs."
+
+
+message = {"role": "user", "content": [{"text": input_text}]}
+messages = [message]
+
+# System prompts.
+system_prompts = [{"text": system_prompt}]
+
+# inference parameters to use.
+temperature = 0.5
+top_p = 0.9
+# Base inference parameters.
+inference_config = {"temperature": temperature, "topP": 0.9}
+
+
+stream_conversation(
+ bedrock_client,
+ MODEL_ID,
+ messages,
+ system_prompts,
+ inference_config,
+)
+```
+
+
+
+## Using it with the `track` decorator
+
+If you have multiple steps in your LLM pipeline, you can use the `track` decorator to log the traces for each step. If Bedrock is called within one of these steps, the LLM call with be associated with that corresponding step:
+
+
+```python
+from opik import track
+from opik.integrations.bedrock import track_bedrock
+
+bedrock = boto3.client(
+ service_name="bedrock-runtime",
+ region_name=REGION,
+ # aws_access_key_id=ACCESS_KEY,
+ # aws_secret_access_key=SECRET_KEY,
+ # aws_session_token=SESSION_TOKEN,
+)
+
+os.environ["OPIK_PROJECT_NAME"] = "bedrock-integration-demo"
+bedrock_client = track_bedrock(bedrock)
+
+
+@track
+def generate_story(prompt):
+ res = bedrock_client.converse(
+ modelId=MODEL_ID, messages=[{"role": "user", "content": [{"text": prompt}]}]
+ )
+ return res["output"]["message"]["content"][0]["text"]
+
+
+@track
+def generate_topic():
+ prompt = "Generate a topic for a story about Opik."
+ res = bedrock_client.converse(
+ modelId=MODEL_ID, messages=[{"role": "user", "content": [{"text": prompt}]}]
+ )
+ return res["output"]["message"]["content"][0]["text"]
+
+
+@track
+def generate_opik_story():
+ topic = generate_topic()
+ story = generate_story(topic)
+ return story
+
+
+generate_opik_story()
+```
+
+The trace can now be viewed in the UI:
+
+
diff --git a/apps/opik-documentation/documentation/fern/docs/cookbook/crewai.mdx b/apps/opik-documentation/documentation/fern/docs/cookbook/crewai.mdx
new file mode 100644
index 0000000000..e49ee827f1
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/cookbook/crewai.mdx
@@ -0,0 +1,111 @@
+---
+description: Cookbook that showcases Opik's integration with the CrewAI
+---
+
+# Using Opik with CrewAI
+
+This notebook showcases how to use Opik with CrewAI. [CrewAI](https://github.com/crewAIInc/crewAI) is a cutting-edge framework for orchestrating autonomous AI agents.
+> CrewAI enables you to create AI teams where each agent has specific roles, tools, and goals, working together to accomplish complex tasks.
+
+> Think of it as assembling your dream team - each member (agent) brings unique skills and expertise, collaborating seamlessly to achieve your objectives.
+
+For this guide we will use CrewAI's quickstart example.
+
+## Creating an account on Comet.com
+
+[Comet](https://www.comet.com/site?from=llm&utm_source=opik&utm_medium=colab&utm_content=llamaindex&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm&=opik&utm_medium=colab&utm_content=llamaindex&utm_campaign=opik) and grab you API Key.
+
+> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm&utm_source=opik&utm_medium=colab&utm_content=llamaindex&utm_campaign=opik) for more information.
+
+
+```python
+%pip install crewai crewai-tools opik --upgrade
+```
+
+
+```python
+import opik
+
+opik.configure(use_local=False)
+```
+
+## Preparing our environment
+
+First, we set up our API keys for our LLM-provider as environment variables:
+
+
+```python
+import os
+import getpass
+
+if "OPENAI_API_KEY" not in os.environ:
+ os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
+```
+
+## Using CrewAI
+The first step is to create our project. We will use an example from CrewAI's documentation:
+
+
+```python
+from crewai import Agent, Crew, Task, Process
+
+
+class YourCrewName:
+ def agent_one(self) -> Agent:
+ return Agent(
+ role="Data Analyst",
+ goal="Analyze data trends in the market",
+ backstory="An experienced data analyst with a background in economics",
+ verbose=True,
+ )
+
+ def agent_two(self) -> Agent:
+ return Agent(
+ role="Market Researcher",
+ goal="Gather information on market dynamics",
+ backstory="A diligent researcher with a keen eye for detail",
+ verbose=True,
+ )
+
+ def task_one(self) -> Task:
+ return Task(
+ name="Collect Data Task",
+ description="Collect recent market data and identify trends.",
+ expected_output="A report summarizing key trends in the market.",
+ agent=self.agent_one(),
+ )
+
+ def task_two(self) -> Task:
+ return Task(
+ name="Market Research Task",
+ description="Research factors affecting market dynamics.",
+ expected_output="An analysis of factors influencing the market.",
+ agent=self.agent_two(),
+ )
+
+ def crew(self) -> Crew:
+ return Crew(
+ agents=[self.agent_one(), self.agent_two()],
+ tasks=[self.task_one(), self.task_two()],
+ process=Process.sequential,
+ verbose=True,
+ )
+```
+
+Now we can import Opik's tracker and run our `crew`:
+
+
+```python
+from opik.integrations.crewai import track_crewai
+
+track_crewai(project_name="crewai-integration-demo")
+
+my_crew = YourCrewName().crew()
+result = my_crew.kickoff()
+
+print(result)
+```
+
+You can now go to the Opik app to see the trace:
+
+
diff --git a/apps/opik-documentation/documentation/fern/docs/cookbook/dspy.mdx b/apps/opik-documentation/documentation/fern/docs/cookbook/dspy.mdx
new file mode 100644
index 0000000000..2179fc51b5
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/cookbook/dspy.mdx
@@ -0,0 +1,63 @@
+---
+description: Cookbook that showcases Opik's integration with DSPy
+---
+
+# Using Opik with DSPy
+
+[DSPy](https://dspy.ai/) is the framework for programming—rather than prompting—language models.
+
+In this guide, we will showcase how to integrate Opik with DSPy so that all the DSPy calls are logged as traces in Opik.
+
+## Creating an account on Comet.com
+
+[Comet](https://www.comet.com/site?from=llm&utm_source=opik&utm_medium=colab&utm_content=dspy&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm&utm_source=opik&utm_medium=colab&utm_content=dspy&utm_campaign=opik) and grab you API Key.
+
+> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm&utm_source=opik&utm_medium=colab&utm_content=dspy&utm_campaign=opik) for more information.
+
+
+```python
+%pip install --upgrade opik dspy
+```
+
+
+```python
+import opik
+
+opik.configure(use_local=False)
+```
+
+
+```python
+import os
+import getpass
+
+if "OPENAI_API_KEY" not in os.environ:
+ os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
+```
+
+## Logging traces
+
+In order to log traces to Opik, you will need to set the `opik` callback:
+
+
+```python
+import dspy
+from opik.integrations.dspy.callback import OpikCallback
+
+lm = dspy.LM("openai/gpt-4o-mini")
+
+project_name = "DSPY"
+opik_callback = OpikCallback(project_name=project_name)
+
+dspy.configure(lm=lm, callbacks=[opik_callback])
+```
+
+
+```python
+cot = dspy.ChainOfThought("question -> answer")
+cot(question="What is the meaning of life?")
+```
+
+The trace is now logged to the Opik platform:
+
+
diff --git a/apps/opik-documentation/documentation/fern/docs/cookbook/evaluate_hallucination_metric.mdx b/apps/opik-documentation/documentation/fern/docs/cookbook/evaluate_hallucination_metric.mdx
new file mode 100644
index 0000000000..8e902bb73a
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/cookbook/evaluate_hallucination_metric.mdx
@@ -0,0 +1,138 @@
+---
+sidebar_label: Evaluating Opik's Hallucination Metric
+description: Cookbook that evalutes Opik's Hallucination Metric, showcasing both how to use the `evaluation` functionality in the platform as well as the quality of the Hallucination metric included in the SDK. It is a complex example that doesn't always align with how the `evaluate` function works.
+---
+
+# Evaluating Opik's Hallucination Metric
+
+For this guide we will be evaluating the Hallucination metric included in the LLM Evaluation SDK which will showcase both how to use the `evaluation` functionality in the platform as well as the quality of the Hallucination metric included in the SDK.
+
+## Creating an account on Comet.com
+
+[Comet](https://www.comet.com/site/?from=llm&utm_source=opik&utm_medium=colab&utm_content=eval_hall&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm&utm_source=opik&utm_medium=colab&utm_content=eval_hall&utm_campaign=opik) and grab you API Key.
+
+> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm&utm_source=opik&utm_medium=colab&utm_content=eval_hall&utm_campaign=opik) for more information.
+
+
+```python
+%pip install opik pyarrow pandas fsspec huggingface_hub --upgrade --quiet
+```
+
+
+```python
+import opik
+
+opik.configure(use_local=False)
+```
+
+## Preparing our environment
+
+First, we will install configure the OpenAI API key and create a new Opik dataset
+
+
+```python
+import os
+import getpass
+
+if "OPENAI_API_KEY" not in os.environ:
+ os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
+```
+
+We will be using the [HaluEval dataset](https://huggingface.co/datasets/pminervini/HaluEval?library=pandas) which according to this [paper](https://arxiv.org/pdf/2305.11747) ChatGPT detects 86.2% of hallucinations. The first step will be to create a dataset in the platform so we can keep track of the results of the evaluation.
+
+Since the insert methods in the SDK deduplicates items, we can insert 50 items and if the items already exist, Opik will automatically remove them.
+
+
+```python
+# Create dataset
+import opik
+import pandas as pd
+
+client = opik.Opik()
+
+# Create dataset
+dataset = client.get_or_create_dataset(name="HaluEval", description="HaluEval dataset")
+
+# Insert items into dataset
+df = pd.read_parquet(
+ "hf://datasets/pminervini/HaluEval/general/data-00000-of-00001.parquet"
+)
+df = df.sample(n=50, random_state=42)
+
+dataset_records = [
+ {
+ "input": x["user_query"],
+ "llm_output": x["chatgpt_response"],
+ "expected_hallucination_label": x["hallucination"],
+ }
+ for x in df.to_dict(orient="records")
+]
+
+dataset.insert(dataset_records)
+```
+
+## Evaluating the hallucination metric
+
+In order to evaluate the performance of the Opik hallucination metric, we will define:
+
+- Evaluation task: Our evaluation task will use the data in the Dataset to return a hallucination score computed using the Opik hallucination metric.
+- Scoring metric: We will use the `Equals` metric to check if the hallucination score computed matches the expected output.
+
+By defining the evaluation task in this way, we will be able to understand how well Opik's hallucination metric is able to detect hallucinations in the dataset.
+
+
+```python
+from opik.evaluation.metrics import Hallucination, Equals
+from opik.evaluation import evaluate
+from opik import Opik
+from opik.evaluation.metrics.llm_judges.hallucination.template import generate_query
+from typing import Dict
+
+
+# Define the evaluation task
+def evaluation_task(x: Dict):
+ metric = Hallucination()
+ try:
+ metric_score = metric.score(input=x["input"], output=x["llm_output"])
+ hallucination_score = metric_score.value
+ hallucination_reason = metric_score.reason
+ except Exception as e:
+ print(e)
+ hallucination_score = None
+ hallucination_reason = str(e)
+
+ return {
+ "hallucination_score": "yes" if hallucination_score == 1 else "no",
+ "hallucination_reason": hallucination_reason,
+ }
+
+
+# Get the dataset
+client = Opik()
+dataset = client.get_dataset(name="HaluEval")
+
+# Define the scoring metric
+check_hallucinated_metric = Equals(name="Correct hallucination score")
+
+# Add the prompt template as an experiment configuration
+experiment_config = {
+ "prompt_template": generate_query(
+ input="{input}", context="{context}", output="{output}", few_shot_examples=[]
+ )
+}
+
+res = evaluate(
+ dataset=dataset,
+ task=evaluation_task,
+ scoring_metrics=[check_hallucinated_metric],
+ experiment_config=experiment_config,
+ scoring_key_mapping={
+ "reference": "expected_hallucination_label",
+ "output": "hallucination_score",
+ },
+)
+```
+
+We can see that the hallucination metric is able to detect ~80% of the hallucinations contained in the dataset and we can see the specific items where hallucinations were not detected.
+
+
diff --git a/apps/opik-documentation/documentation/fern/docs/cookbook/evaluate_moderation_metric.mdx b/apps/opik-documentation/documentation/fern/docs/cookbook/evaluate_moderation_metric.mdx
new file mode 100644
index 0000000000..9ee3d07815
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/cookbook/evaluate_moderation_metric.mdx
@@ -0,0 +1,147 @@
+---
+sidebar_label: Evaluate Opik's Moderation Metric
+description: Cookbook that evalutes Opik's Moderation Metric, showcasing both how to use the `evaluation` functionality in the platform as well as the quality of the Modelration metric included in the SDK. It is a complex example that doesn't always align with how the `evaluate` function works.
+---
+
+# Evaluating Opik's Moderation Metric
+
+For this guide we will be evaluating the Moderation metric included in the LLM Evaluation SDK which will showcase both how to use the `evaluation` functionality in the platform as well as the quality of the Moderation metric included in the SDK.
+
+## Creating an account on Comet.com
+
+[Comet](https://www.comet.com/site?from=llm&utm_source=opik&utm_medium=colab&utm_content=eval_mod&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup/?from=llm&utm_source=opik&utm_medium=colab&utm_content=eval_mod&utm_campaign=opik) and grab you API Key.
+
+> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm&utm_source=opik&utm_medium=colab&utm_content=eval_mod&utm_campaign=opik) for more information.
+
+
+```python
+%pip install --upgrade --quiet opik pandas
+```
+
+
+```python
+import opik
+
+opik.configure(use_local=False)
+```
+
+## Preparing our environment
+
+First, we will configure the OpenAI API key and download a reference moderation dataset.
+
+
+```python
+import os
+import getpass
+
+if "OPENAI_API_KEY" not in os.environ:
+ os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
+```
+
+We will be using the [OpenAI Moderation API Release dataset](https://github.com/openai/moderation-api-release/tree/main/data) which according to this [blog post](https://openai.com/index/using-gpt-4-for-content-moderation/) GPT-4o detects ~60~% of hallucinations. The first step will be to create a dataset in the platform so we can keep track of the results of the evaluation.
+
+Since the insert methods in the SDK deduplicates items, we can insert 50 items and if the items already exist, Opik will automatically remove them.
+
+
+```python
+# Create dataset
+import opik
+import pandas as pd
+import requests
+from io import BytesIO
+
+client = opik.Opik()
+
+# Create dataset
+dataset = client.get_or_create_dataset(
+ name="OpenAIModerationDataset", description="OpenAI Moderation Dataset"
+)
+
+# Insert items into dataset
+url = "https://github.com/openai/moderation-api-release/raw/main/data/samples-1680.jsonl.gz"
+response = requests.get(url)
+df = pd.read_json(BytesIO(response.content), lines=True, compression="gzip")
+
+df = df.sample(n=50, random_state=42)
+
+dataset_records = []
+for x in df.to_dict(orient="records"):
+ moderation_fields = ["S", "H", "V", "HR", "SH", "S3", "H2", "V2"]
+ moderated_fields = [field for field in moderation_fields if x[field] == 1.0]
+ expected_output = "moderated" if moderated_fields else "not_moderated"
+
+ dataset_records.append(
+ {
+ "output": x["prompt"],
+ "expected_output": expected_output,
+ "moderated_fields": moderated_fields,
+ }
+ )
+
+dataset.insert(dataset_records)
+```
+
+## Evaluating the moderation metric
+
+In order to evaluate the performance of the Opik moderation metric, we will define:
+
+- Evaluation task: Our evaluation task will use the data in the Dataset to return a moderation score computed using the Opik moderation metric.
+- Scoring metric: We will use the `Equals` metric to check if the moderation score computed matches the expected output.
+
+By defining the evaluation task in this way, we will be able to understand how well Opik's moderation metric is able to detect moderation violations in the dataset.
+
+We can use the Opik SDK to compute a moderation score for each item in the dataset:
+
+
+```python
+from opik.evaluation.metrics import Moderation, Equals
+from opik.evaluation import evaluate
+from opik import Opik
+from opik.evaluation.metrics.llm_judges.moderation.template import generate_query
+from typing import Dict
+
+
+# Define the evaluation task
+def evaluation_task(x: Dict):
+ metric = Moderation()
+ try:
+ metric_score = metric.score(output=x["output"])
+ moderation_score = "moderated" if metric_score.value > 0.5 else "not_moderated"
+ moderation_reason = metric_score.reason
+ except Exception as e:
+ print(e)
+ moderation_score = None
+ moderation_reason = str(e)
+
+ return {
+ "moderation_score": moderation_score,
+ "moderation_reason": moderation_reason,
+ }
+
+
+# Get the dataset
+client = Opik()
+dataset = client.get_dataset(name="OpenAIModerationDataset")
+
+# Define the scoring metric
+moderation_metric = Equals(name="Correct moderation score")
+
+# Add the prompt template as an experiment configuration
+experiment_config = {
+ "prompt_template": generate_query(output="{output}", few_shot_examples=[])
+}
+
+res = evaluate(
+ dataset=dataset,
+ task=evaluation_task,
+ scoring_metrics=[moderation_metric],
+ experiment_config=experiment_config,
+ scoring_key_mapping={"reference": "expected_output", "output": "moderation_score"},
+)
+```
+
+We are able to detect ~85% of moderation violations, this can be improved further by providing some additional examples to the model. We can view a breakdown of the results in the Opik UI:
+
+
+
+
diff --git a/apps/opik-documentation/documentation/fern/docs/cookbook/gemini.mdx b/apps/opik-documentation/documentation/fern/docs/cookbook/gemini.mdx
new file mode 100644
index 0000000000..876ea72dfd
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/cookbook/gemini.mdx
@@ -0,0 +1,127 @@
+---
+description: Cookbook that showcases Opik's integration with the Gemini Python SDK
+---
+
+# Using Opik with Gemini
+
+Opik integrates with Gemini to provide a simple way to log traces for all Gemini LLM calls. This works for all Gemini models.
+
+## Creating an account on Comet.com
+
+[Comet](https://www.comet.com/site?from=llm&utm_source=opik&utm_medium=colab&utm_content=openai&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm&utm_source=opik&utm_medium=colab&utm_content=openai&utm_campaign=opik) and grab you API Key.
+
+> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm&utm_source=opik&utm_medium=colab&utm_content=openai&utm_campaign=opik) for more information.
+
+
+```python
+%pip install --upgrade opik google-generativeai litellm
+```
+
+
+```python
+import opik
+
+opik.configure(use_local=False)
+```
+
+## Preparing our environment
+
+First, we will set up our OpenAI API keys.
+
+
+```python
+import os
+import getpass
+import google.generativeai as genai
+
+if "GEMINI_API_KEY" not in os.environ:
+ genai.configure(api_key=getpass.getpass("Enter your Gemini API key: "))
+```
+
+## Configure LiteLLM
+
+Add the LiteLLM OpikTracker to log traces and steps to Opik:
+
+
+```python
+import litellm
+import os
+from litellm.integrations.opik.opik import OpikLogger
+from opik import track
+from opik.opik_context import get_current_span_data
+
+os.environ["OPIK_PROJECT_NAME"] = "gemini-integration-demo"
+opik_logger = OpikLogger()
+litellm.callbacks = [opik_logger]
+```
+
+## Logging traces
+
+Now each completion will logs a separate trace to LiteLLM:
+
+
+```python
+prompt = """
+Write a short two sentence story about Opik.
+"""
+
+response = litellm.completion(
+ model="gemini/gemini-pro",
+ messages=[{"role": "user", "content": prompt}],
+)
+
+print(response.choices[0].message.content)
+```
+
+The prompt and response messages are automatically logged to Opik and can be viewed in the UI.
+
+
+
+## Using it with the `track` decorator
+
+If you have multiple steps in your LLM pipeline, you can use the `track` decorator to log the traces for each step. If Gemini is called within one of these steps, the LLM call with be associated with that corresponding step:
+
+
+```python
+@track
+def generate_story(prompt):
+ response = litellm.completion(
+ model="gemini/gemini-pro",
+ messages=[{"role": "user", "content": prompt}],
+ metadata={
+ "opik": {
+ "current_span_data": get_current_span_data(),
+ },
+ },
+ )
+ return response.choices[0].message.content
+
+
+@track
+def generate_topic():
+ prompt = "Generate a topic for a story about Opik."
+ response = litellm.completion(
+ model="gemini/gemini-pro",
+ messages=[{"role": "user", "content": prompt}],
+ metadata={
+ "opik": {
+ "current_span_data": get_current_span_data(),
+ },
+ },
+ )
+ return response.choices[0].message.content
+
+
+@track
+def generate_opik_story():
+ topic = generate_topic()
+ story = generate_story(topic)
+ return story
+
+
+generate_opik_story()
+```
+
+The trace can now be viewed in the UI:
+
+
diff --git a/apps/opik-documentation/documentation/fern/docs/cookbook/groq.mdx b/apps/opik-documentation/documentation/fern/docs/cookbook/groq.mdx
new file mode 100644
index 0000000000..0900fda70f
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/cookbook/groq.mdx
@@ -0,0 +1,126 @@
+---
+description: Cookbook that showcases Opik's integration with Groq
+---
+
+# Using Opik with Groq
+
+Opik integrates with Groq to provide a simple way to log traces for all Groq LLM calls. This works for all Groq models.
+
+## Creating an account on Comet.com
+
+[Comet](https://www.comet.com/site?from=llm&utm_source=opik&utm_medium=colab&utm_content=openai&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm&utm_source=opik&utm_medium=colab&utm_content=openai&utm_campaign=opik) and grab you API Key.
+
+> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm&utm_source=opik&utm_medium=colab&utm_content=openai&utm_campaign=opik) for more information.
+
+
+```python
+%pip install --upgrade opik litellm
+```
+
+
+```python
+import opik
+
+opik.configure(use_local=False)
+```
+
+## Preparing our environment
+
+First, we will set up our OpenAI API keys.
+
+
+```python
+import os
+import getpass
+
+if "GROQ_API_KEY" not in os.environ:
+ os.environ["GROQ_API_KEY"] = getpass.getpass("Enter your Groq API key: ")
+```
+
+## Configure LiteLLM
+
+Add the LiteLLM OpikTracker to log traces and steps to Opik:
+
+
+```python
+import litellm
+import os
+from litellm.integrations.opik.opik import OpikLogger
+from opik import track
+from opik.opik_context import get_current_span_data
+
+os.environ["OPIK_PROJECT_NAME"] = "grok-integration-demo"
+opik_logger = OpikLogger()
+litellm.callbacks = [opik_logger]
+```
+
+## Logging traces
+
+Now each completion will logs a separate trace to LiteLLM:
+
+
+```python
+prompt = """
+Write a short two sentence story about Opik.
+"""
+
+response = litellm.completion(
+ model="groq/llama3-8b-8192",
+ messages=[{"role": "user", "content": prompt}],
+)
+
+print(response.choices[0].message.content)
+```
+
+The prompt and response messages are automatically logged to Opik and can be viewed in the UI.
+
+
+
+## Using it with the `track` decorator
+
+If you have multiple steps in your LLM pipeline, you can use the `track` decorator to log the traces for each step. If Groq is called within one of these steps, the LLM call with be associated with that corresponding step:
+
+
+```python
+@track
+def generate_story(prompt):
+ response = litellm.completion(
+ model="groq/llama3-8b-8192",
+ messages=[{"role": "user", "content": prompt}],
+ metadata={
+ "opik": {
+ "current_span_data": get_current_span_data(),
+ },
+ },
+ )
+ return response.choices[0].message.content
+
+
+@track
+def generate_topic():
+ prompt = "Generate a topic for a story about Opik."
+ response = litellm.completion(
+ model="groq/llama3-8b-8192",
+ messages=[{"role": "user", "content": prompt}],
+ metadata={
+ "opik": {
+ "current_span_data": get_current_span_data(),
+ },
+ },
+ )
+ return response.choices[0].message.content
+
+
+@track
+def generate_opik_story():
+ topic = generate_topic()
+ story = generate_story(topic)
+ return story
+
+
+generate_opik_story()
+```
+
+The trace can now be viewed in the UI:
+
+
diff --git a/apps/opik-documentation/documentation/fern/docs/cookbook/guardrails-ai.mdx b/apps/opik-documentation/documentation/fern/docs/cookbook/guardrails-ai.mdx
new file mode 100644
index 0000000000..eef561404f
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/cookbook/guardrails-ai.mdx
@@ -0,0 +1,79 @@
+---
+description: Cookbook that showcases Opik's integration with the Guardrails AI Python SDK
+---
+
+# Using Opik with Guardrails AI
+
+[Guardrails AI](https://github.com/guardrails-ai/guardrails) is a framework for validating the inputs and outputs
+
+For this guide we will use a simple example that logs guardrails validation steps as traces to Opik, providing them with the validation result tags.
+
+## Creating an account on Comet.com
+
+[Comet](https://www.comet.com/site?from=llm&utm_source=opik&utm_medium=colab&utm_content=openai&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm&utm_source=opik&utm_medium=colab&utm_content=openai&utm_campaign=opik) and grab you API Key.
+
+> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm&utm_source=opik&utm_medium=colab&utm_content=openai&utm_campaign=opik) for more information.
+
+
+```python
+%pip install --upgrade opik guardrails-ai
+```
+
+
+```python
+import opik
+
+opik.configure(use_local=False)
+```
+
+## Preparing our environment
+
+In order to use Guardrails AI, we will configure the OpenAI API Key, if you are using any other providers you can replace this with the required API key:
+
+
+```python
+import os
+import getpass
+
+if "OPENAI_API_KEY" not in os.environ:
+ os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
+```
+
+We will also need to install the guardrails check for politeness from the Guardrails Hub
+
+
+```python
+!guardrails hub install hub://guardrails/politeness_check
+```
+
+## Logging validation traces
+
+In order to log traces to Opik, you will need to call the track the Guard object with `track_guardrails` function.
+
+
+```python
+from guardrails import Guard, OnFailAction
+from guardrails.hub import PolitenessCheck
+
+from opik.integrations.guardrails import track_guardrails
+
+politeness_check = PolitenessCheck(
+ llm_callable="gpt-3.5-turbo", on_fail=OnFailAction.NOOP
+)
+
+guard: Guard = Guard().use_many(politeness_check)
+guard = track_guardrails(guard, project_name="guardrails-integration-example")
+
+guard.validate(
+ "Would you be so kind to pass me a cup of tea?",
+)
+guard.validate(
+ "Shut your mouth up and give me the tea.",
+);
+```
+
+Every validation will now be logged to Opik as a trace
+
+The trace will now be viewable in the Opik platform:
+
+
diff --git a/apps/opik-documentation/documentation/fern/docs/cookbook/haystack.mdx b/apps/opik-documentation/documentation/fern/docs/cookbook/haystack.mdx
new file mode 100644
index 0000000000..303c9255b1
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/cookbook/haystack.mdx
@@ -0,0 +1,130 @@
+---
+description: Cookbook that showcases Opik's integration with Haystack
+---
+
+# Using Opik with Haystack
+
+[Haystack](https://docs.haystack.deepset.ai/docs/intro) is an open-source framework for building production-ready LLM applications, retrieval-augmented generative pipelines and state-of-the-art search systems that work intelligently over large document collections.
+
+In this guide, we will showcase how to integrate Opik with Haystack so that all the Haystack calls are logged as traces in Opik.
+
+## Creating an account on Comet.com
+
+[Comet](https://www.comet.com/site?from=llm&utm_source=opik&utm_medium=colab&utm_content=haystack&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm&utm_source=opik&utm_medium=colab&utm_content=haystack&utm_campaign=opik) and grab you API Key.
+
+> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm&utm_source=opik&utm_medium=colab&utm_content=haystack&utm_campaign=opik) for more information.
+
+
+```python
+%pip install --upgrade --quiet opik haystack-ai
+```
+
+
+```python
+import opik
+
+opik.configure(use_local=False)
+```
+
+
+```python
+import os
+import getpass
+
+if "OPENAI_API_KEY" not in os.environ:
+ os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
+```
+
+## Creating the Haystack pipeline
+
+In this example, we will create a simple pipeline that uses a prompt template to translate text to German.
+
+To enable Opik tracing, we will:
+1. Enable content tracing in Haystack by setting the environment variable `HAYSTACK_CONTENT_TRACING_ENABLED=true`
+2. Add the `OpikConnector` component to the pipeline
+
+Note: The `OpikConnector` component is a special component that will automatically log the traces of the pipeline as Opik traces, it should not be connected to any other component.
+
+
+```python
+import os
+
+os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"
+
+from haystack import Pipeline
+from haystack.components.builders import ChatPromptBuilder
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.dataclasses import ChatMessage
+
+from opik.integrations.haystack import OpikConnector
+
+
+pipe = Pipeline()
+
+# Add the OpikConnector component to the pipeline
+pipe.add_component("tracer", OpikConnector("Chat example"))
+
+# Continue building the pipeline
+pipe.add_component("prompt_builder", ChatPromptBuilder())
+pipe.add_component("llm", OpenAIChatGenerator(model="gpt-3.5-turbo"))
+
+pipe.connect("prompt_builder.prompt", "llm.messages")
+
+messages = [
+ ChatMessage.from_system(
+ "Always respond in German even if some input data is in other languages."
+ ),
+ ChatMessage.from_user("Tell me about {{location}}"),
+]
+
+response = pipe.run(
+ data={
+ "prompt_builder": {
+ "template_variables": {"location": "Berlin"},
+ "template": messages,
+ }
+ }
+)
+
+trace_id = response["tracer"]["trace_id"]
+print(f"Trace ID: {trace_id}")
+print(response["llm"]["replies"][0])
+```
+
+The trace is now logged to the Opik platform:
+
+
+
+## Advanced usage
+
+### Ensuring the trace is logged
+
+By default the `OpikConnector` will flush the trace to the Opik platform after each component in a thread blocking way. As a result, you may disable flushing the data after each component by setting the `HAYSTACK_OPIK_ENFORCE_FLUSH` environent variable to `false`.
+
+**Caution**: Disabling this feature may result in data loss if the program crashes before the data is sent to Opik. Make sure you will call the `flush()` method explicitly before the program exits:
+
+
+```python
+from haystack.tracing import tracer
+
+tracer.actual_tracer.flush()
+```
+
+### Getting the trace ID
+
+If you would like to log additional information to the trace you will need to get the trace ID. You can do this by the `tracer` key in the response of the pipeline:
+
+
+```python
+response = pipe.run(
+ data={
+ "prompt_builder": {
+ "template_variables": {"location": "Berlin"},
+ "template": messages,
+ }
+ }
+)
+
+trace_id = response["tracer"]["trace_id"]
+print(f"Trace ID: {trace_id}")
+```
diff --git a/apps/opik-documentation/documentation/fern/docs/cookbook/langchain.mdx b/apps/opik-documentation/documentation/fern/docs/cookbook/langchain.mdx
new file mode 100644
index 0000000000..2e1773b8ee
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/cookbook/langchain.mdx
@@ -0,0 +1,207 @@
+---
+description: Cookbook that showcases Opik's integration with the LangChain Python SDK
+---
+
+# Using Opik with Langchain
+
+For this guide, we will be performing a text to sql query generation task using LangChain. We will be using the Chinook database which contains the SQLite database of a music store with both employee, customer and invoice data.
+
+We will highlight three different parts of the workflow:
+
+1. Creating a synthetic dataset of questions
+2. Creating a LangChain chain to generate SQL queries
+3. Automating the evaluation of the SQL queries on the synthetic dataset
+
+## Creating an account on Comet.com
+
+[Comet](https://www.comet.com/site?from=llm&utm_source=opik&utm_medium=colab&utm_content=langchain&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm&utm_source=opik&utm_medium=colab&utm_content=langchain&utm_campaign=opik) and grab you API Key.
+
+> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm&utm_source=opik&utm_medium=colab&utm_content=langchain&utm_campaign=opik) for more information.
+
+
+```python
+%pip install --upgrade --quiet opik langchain langchain-community langchain-openai
+```
+
+
+```python
+import opik
+
+opik.configure(use_local=False)
+```
+
+## Preparing our environment
+
+First, we will download the Chinook database and set up our different API keys.
+
+
+```python
+import os
+import getpass
+
+if "OPENAI_API_KEY" not in os.environ:
+ os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
+```
+
+
+```python
+# Download the relevant data
+import os
+from langchain_community.utilities import SQLDatabase
+
+import requests
+import os
+
+url = "https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite.sqlite"
+filename = "./data/chinook/Chinook_Sqlite.sqlite"
+
+folder = os.path.dirname(filename)
+
+if not os.path.exists(folder):
+ os.makedirs(folder)
+
+if not os.path.exists(filename):
+ response = requests.get(url)
+ with open(filename, "wb") as file:
+ file.write(response.content)
+ print("Chinook database downloaded")
+
+db = SQLDatabase.from_uri(f"sqlite:///{filename}")
+```
+
+## Creating a synthetic dataset
+
+In order to create our synthetic dataset, we will be using the OpenAI API to generate 20 different questions that a user might ask based on the Chinook database.
+
+In order to ensure that the OpenAI API calls are being tracked, we will be using the `track_openai` function from the `opik` library.
+
+
+```python
+from opik.integrations.openai import track_openai
+from openai import OpenAI
+import json
+
+os.environ["OPIK_PROJECT_NAME"] = "langchain-integration-demo"
+client = OpenAI()
+
+openai_client = track_openai(client)
+
+prompt = """
+Create 20 different example questions a user might ask based on the Chinook Database.
+
+These questions should be complex and require the model to think. They should include complex joins and window functions to answer.
+
+Return the response as a json object with a "result" key and an array of strings with the question.
+"""
+
+completion = openai_client.chat.completions.create(
+ model="gpt-3.5-turbo", messages=[{"role": "user", "content": prompt}]
+)
+
+print(completion.choices[0].message.content)
+```
+
+Now that we have our synthetic dataset, we can create a dataset in Comet and insert the questions into it.
+
+Since the insert methods in the SDK deduplicates items, we can insert 20 items and if the items already exist, Opik will automatically remove them.
+
+
+```python
+# Create the synthetic dataset
+import opik
+
+synthetic_questions = json.loads(completion.choices[0].message.content)["result"]
+
+client = opik.Opik()
+
+dataset = client.get_or_create_dataset(name="synthetic_questions")
+dataset.insert([{"question": question} for question in synthetic_questions])
+```
+
+## Creating a LangChain chain
+
+We will be using the `create_sql_query_chain` function from the `langchain` library to create a SQL query to answer the question.
+
+We will be using the `OpikTracer` class from the `opik` library to ensure that the LangChan trace are being tracked in Comet.
+
+
+```python
+# Use langchain to create a SQL query to answer the question
+from langchain.chains import create_sql_query_chain
+from langchain_openai import ChatOpenAI
+from opik.integrations.langchain import OpikTracer
+
+opik_tracer = OpikTracer(tags=["simple_chain"])
+
+llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)
+chain = create_sql_query_chain(llm, db).with_config({"callbacks": [opik_tracer]})
+response = chain.invoke({"question": "How many employees are there ?"})
+response
+
+print(response)
+```
+
+## Automating the evaluation
+
+In order to ensure our LLM application is working correctly, we will test it on our synthetic dataset.
+
+For this we will be using the `evaluate` function from the `opik` library. We will evaluate the application using a custom metric that checks if the SQL query is valid.
+
+
+```python
+from opik import Opik, track
+from opik.evaluation import evaluate
+from opik.evaluation.metrics import base_metric, score_result
+from typing import Any
+
+
+class ValidSQLQuery(base_metric.BaseMetric):
+ def __init__(self, name: str, db: Any):
+ self.name = name
+ self.db = db
+
+ def score(self, output: str, **ignored_kwargs: Any):
+ # Add you logic here
+
+ try:
+ db.run(output)
+ return score_result.ScoreResult(
+ name=self.name, value=1, reason="Query ran successfully"
+ )
+ except Exception as e:
+ return score_result.ScoreResult(name=self.name, value=0, reason=str(e))
+
+
+valid_sql_query = ValidSQLQuery(name="valid_sql_query", db=db)
+
+client = Opik()
+dataset = client.get_dataset("synthetic_questions")
+
+
+@track()
+def llm_chain(input: str) -> str:
+ response = chain.invoke({"question": input})
+
+ return response
+
+
+def evaluation_task(item):
+ response = llm_chain(item["question"])
+
+ return {"output": response}
+
+
+res = evaluate(
+ experiment_name="SQL question answering",
+ dataset=dataset,
+ task=evaluation_task,
+ scoring_metrics=[valid_sql_query],
+ nb_samples=20,
+)
+```
+
+The evaluation results are now uploaded to the Opik platform and can be viewed in the UI.
+
+
+
+
diff --git a/apps/opik-documentation/documentation/fern/docs/cookbook/langgraph.mdx b/apps/opik-documentation/documentation/fern/docs/cookbook/langgraph.mdx
new file mode 100644
index 0000000000..fc79356f5c
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/cookbook/langgraph.mdx
@@ -0,0 +1,127 @@
+---
+description: Cookbook that showcases Opik's integration with the LangGraph Python SDK
+---
+
+# Using Opik with LangGraph
+
+This notebook showcases how to use Opik with LangGraph. [LangGraph](https://langchain-ai.github.io/langgraph/) is a library for building stateful, multi-actor applications with LLMs, used to create agent and multi-agent workflows
+
+In this notebook, we will create a simple LangGraph workflow and focus on how to track it's execution with Opik. To learn more about LangGraph, check out the [official documentation](https://langchain-ai.github.io/langgraph/).
+
+## Creating an account on Opik Cloud
+
+[Comet](https://www.comet.com/site?from=llm&utm_source=opik&utm_medium=colab&utm_content=langgraph&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm&=opik&utm_medium=colab&utm_content=langgraph&utm_campaign=opik) and grab you API Key.
+
+> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm&utm_source=opik&utm_medium=colab&utm_content=langgraph&utm_campaign=opik) for more information.
+
+
+```python
+%pip install --quiet -U langchain langgraph opik
+```
+
+
+```python
+import opik
+
+opik.configure(use_local=False)
+```
+
+## Create the LangGraph graph
+
+The LangGraph graph we will be created in made up of 3 nodes:
+
+1. `classify_input`: Classify the input question
+2. `handle_greeting`: Handle the greeting question
+3. `handle_search`: Handle the search question
+
+*Note*: We will not be using any LLM calls or tools in this example to keep things simple. However in most cases, you will want to use tools to interact with external systems.
+
+
+```python
+# We will start by creating simple functions to classify the input question and handle the greeting and search questions.
+def classify(question: str) -> str:
+ return "greeting" if question.startswith("Hello") else "search"
+
+
+def classify_input_node(state):
+ question = state.get("question", "").strip()
+ classification = classify(question) # Assume a function that classifies the input
+ return {"classification": classification}
+
+
+def handle_greeting_node(state):
+ return {"response": "Hello! How can I help you today?"}
+
+
+def handle_search_node(state):
+ question = state.get("question", "").strip()
+ search_result = f"Search result for '{question}'"
+ return {"response": search_result}
+```
+
+
+```python
+from langgraph.graph import StateGraph, END
+
+from typing import TypedDict, Optional
+
+
+class GraphState(TypedDict):
+ question: Optional[str] = None
+ classification: Optional[str] = None
+ response: Optional[str] = None
+
+
+workflow = StateGraph(GraphState)
+workflow.add_node("classify_input", classify_input_node)
+workflow.add_node("handle_greeting", handle_greeting_node)
+workflow.add_node("handle_search", handle_search_node)
+
+
+def decide_next_node(state):
+ return (
+ "handle_greeting"
+ if state.get("classification") == "greeting"
+ else "handle_search"
+ )
+
+
+workflow.add_conditional_edges(
+ "classify_input",
+ decide_next_node,
+ {"handle_greeting": "handle_greeting", "handle_search": "handle_search"},
+)
+
+workflow.set_entry_point("classify_input")
+workflow.add_edge("handle_greeting", END)
+workflow.add_edge("handle_search", END)
+
+app = workflow.compile()
+
+# Display the graph
+try:
+ from IPython.display import Image, display
+
+ display(Image(app.get_graph().draw_mermaid_png()))
+except Exception:
+ # This requires some extra dependencies and is optional
+ pass
+```
+
+## Calling the graph with Opik tracing enabled
+
+In order to log the execution of the graph, we need to define the OpikTracer callback:
+
+
+```python
+from opik.integrations.langchain import OpikTracer
+
+tracer = OpikTracer(graph=app.get_graph(xray=True))
+inputs = {"question": "Hello, how are you?"}
+result = app.invoke(inputs, config={"callbacks": [tracer]})
+print(result)
+```
+
+The graph execution is now logged on the Opik platform and can be viewed in the UI:
+
+
diff --git a/apps/opik-documentation/documentation/fern/docs/cookbook/litellm.mdx b/apps/opik-documentation/documentation/fern/docs/cookbook/litellm.mdx
new file mode 100644
index 0000000000..de12a0a8e7
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/cookbook/litellm.mdx
@@ -0,0 +1,102 @@
+---
+description: Cookbook that showcases Opik's integration with the LiteLLM Python SDK
+---
+
+# Using Opik with LiteLLM
+
+Lite allows you to call all LLM APIs using the OpenAI format [Bedrock, Huggingface, VertexAI, TogetherAI, Azure, OpenAI, Groq etc.]. You can learn more about LiteLLM [here](https://github.com/BerriAI/litellm).
+
+There are two main approaches to using LiteLLM, either using the `litellm` [python library](https://docs.litellm.ai/docs/#litellm-python-sdk) that will query the LLM API for you or by using the [LiteLLM proxy server](https://docs.litellm.ai/docs/#litellm-proxy-server-llm-gateway). In this cookbook we will focus on the first approach but you can learn more about using Opik with the LiteLLM proxy server in our [documentation](https://www.comet.com/docs/opik/tracing/integrations/litellm).
+
+## Creating an account on Comet.com
+
+[Comet](https://www.comet.com/site?from=llm&utm_source=opik&utm_medium=colab&utm_content=openai&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm&utm_source=opik&utm_medium=colab&utm_content=openai&utm_campaign=opik) and grab you API Key.
+
+> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm&utm_source=opik&utm_medium=colab&utm_content=openai&utm_campaign=opik) for more information.
+
+
+```python
+%pip install --upgrade opik litellm
+```
+
+
+```python
+import opik
+
+opik.configure(use_local=False)
+```
+
+## Preparing our environment
+
+In order to use LiteLLM, we will configure the OpenAI API Key, if you are using any other providers you can replace this with the required API key:
+
+
+```python
+import os
+import getpass
+
+if "OPENAI_API_KEY" not in os.environ:
+ os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
+```
+
+## Logging traces
+
+In order to log traces to Opik, you will need to set the `opik` callback:
+
+
+```python
+from litellm.integrations.opik.opik import OpikLogger
+from opik.opik_context import get_current_span_data
+from opik import track
+import litellm
+
+os.environ["OPIK_PROJECT_NAME"] = "litellm-integration-demo"
+opik_logger = OpikLogger()
+litellm.callbacks = [opik_logger]
+```
+
+Every LiteLLM call will now be logged to Opik:
+
+
+```python
+response = litellm.completion(
+ model="gpt-3.5-turbo",
+ messages=[
+ {"role": "user", "content": "Why is tracking and evaluation of LLMs important?"}
+ ],
+)
+
+print(response.choices[0].message.content)
+```
+
+The trace will now be viewable in the Opik platform:
+
+
+
+## Logging LLM calls within a tracked function
+
+
+If you are using LiteLLM within a function tracked with the `@track` decorator, you will need to pass the `current_span_data` as metadata to the `litellm.completion` call:
+
+
+
+```python
+@track
+def streaming_function(input):
+ messages = [{"role": "user", "content": input}]
+ response = litellm.completion(
+ model="gpt-3.5-turbo",
+ messages=messages,
+ metadata={
+ "opik": {
+ "current_span_data": get_current_span_data(),
+ "tags": ["streaming-test"],
+ },
+ },
+ )
+ return response
+
+
+response = streaming_function("Why is tracking and evaluation of LLMs important?")
+chunks = list(response)
+```
diff --git a/apps/opik-documentation/documentation/fern/docs/cookbook/llama-index.mdx b/apps/opik-documentation/documentation/fern/docs/cookbook/llama-index.mdx
new file mode 100644
index 0000000000..92afe9feb8
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/cookbook/llama-index.mdx
@@ -0,0 +1,110 @@
+---
+description: Cookbook that showcases Opik's integration with the LlamaIndex Python SDK
+---
+
+# Using Opik with LlamaIndex
+
+This notebook showcases how to use Opik with LlamaIndex. [LlamaIndex](https://github.com/run-llama/llama_index) is a flexible data framework for building LLM applications:
+> LlamaIndex is a "data framework" to help you build LLM apps. It provides the following tools:
+>
+> - Offers data connectors to ingest your existing data sources and data formats (APIs, PDFs, docs, SQL, etc.).
+> - Provides ways to structure your data (indices, graphs) so that this data can be easily used with LLMs.
+> - Provides an advanced retrieval/query interface over your data: Feed in any LLM input prompt, get back retrieved context and knowledge-augmented output.
+> - Allows easy integrations with your outer application framework (e.g. with LangChain, Flask, Docker, ChatGPT, anything else).
+
+For this guide we will be downloading the essays from Paul Graham and use them as our data source. We will then start querying these essays with LlamaIndex.
+
+## Creating an account on Comet.com
+
+[Comet](https://www.comet.com/site?from=llm&utm_source=opik&utm_medium=colab&utm_content=llamaindex&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm&=opik&utm_medium=colab&utm_content=llamaindex&utm_campaign=opik) and grab you API Key.
+
+> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm&utm_source=opik&utm_medium=colab&utm_content=llamaindex&utm_campaign=opik) for more information.
+
+
+```python
+%pip install opik llama-index llama-index-agent-openai llama-index-llms-openai --upgrade --quiet
+```
+
+
+```python
+import opik
+
+opik.configure(use_local=False)
+```
+
+## Preparing our environment
+
+First, we will download the Chinook database and set up our different API keys.
+
+And configure the required environment variables:
+
+
+```python
+import os
+import getpass
+
+if "OPENAI_API_KEY" not in os.environ:
+ os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
+```
+
+In addition, we will download the Paul Graham essays:
+
+
+```python
+import os
+import requests
+
+# Create directory if it doesn't exist
+os.makedirs("./data/paul_graham/", exist_ok=True)
+
+# Download the file using requests
+url = "https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt"
+response = requests.get(url)
+with open("./data/paul_graham/paul_graham_essay.txt", "wb") as f:
+ f.write(response.content)
+```
+
+## Using LlamaIndex
+
+### Configuring the Opik integration
+
+You can use the Opik callback directly by calling:
+
+
+```python
+from llama_index.core import Settings
+from llama_index.core.callbacks import CallbackManager
+from opik.integrations.llama_index import LlamaIndexCallbackHandler
+
+opik_callback_handler = LlamaIndexCallbackHandler()
+Settings.callback_manager = CallbackManager([opik_callback_handler])
+```
+
+Now that the callback handler is configured, all traces will automatically be logged to Opik.
+
+### Using LLamaIndex
+
+The first step is to load the data into LlamaIndex. We will use the `SimpleDirectoryReader` to load the data from the `data/paul_graham` directory. We will also create the vector store to index all the loaded documents.
+
+
+```python
+from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
+
+documents = SimpleDirectoryReader("./data/paul_graham").load_data()
+index = VectorStoreIndex.from_documents(documents)
+query_engine = index.as_query_engine()
+```
+
+We can now query the index using the `query_engine` object:
+
+
+```python
+response = query_engine.query("What did the author do growing up?")
+print(response)
+```
+
+You can now go to the Opik app to see the trace:
+
+
+
+
diff --git a/apps/opik-documentation/documentation/fern/docs/cookbook/ollama.mdx b/apps/opik-documentation/documentation/fern/docs/cookbook/ollama.mdx
new file mode 100644
index 0000000000..e793b0530d
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/cookbook/ollama.mdx
@@ -0,0 +1,116 @@
+---
+description: Cookbook that showcases Opik's integration with the Ollama Python SDK
+---
+
+# Using Opik with Ollama
+
+[Ollama](https://ollama.com/) allows users to run, interact with, and deploy AI models locally on their machines without the need for complex infrastructure or cloud dependencies.
+
+In this notebook, we will showcase how to log Ollama LLM calls using Opik by utilizing either the OpenAI or LangChain libraries.
+
+## Getting started
+
+### Configure Ollama
+
+In order to interact with Ollama from Python, we will to have Ollama running on our machine. You can learn more about how to install and run Ollama in the [quickstart guide](https://github.com/ollama/ollama/blob/main/README.md#quickstart).
+
+### Configuring Opik
+
+Opik is available as a fully open source local installation or using Comet.com as a hosted solution. The easiest way to get started with Opik is by creating a free Comet account at comet.com.
+
+If you'd like to self-host Opik, you can learn more about the self-hosting options [here](https://www.comet.com/docs/opik/self-host/overview).
+
+In addition, you will need to install and configure the Opik Python package:
+
+
+```python
+%pip install --upgrade --quiet opik
+
+import opik
+
+opik.configure()
+```
+
+## Tracking Ollama calls made with OpenAI
+
+Ollama is compatible with the OpenAI format and can be used with the OpenAI Python library. You can therefore leverage the Opik integration for OpenAI to trace your Ollama calls:
+
+
+
+```python
+from openai import OpenAI
+from opik.integrations.openai import track_openai
+
+import os
+
+os.environ["OPIK_PROJECT_NAME"] = "ollama-integration"
+
+# Create an OpenAI client
+client = OpenAI(
+ base_url="http://localhost:11434/v1/",
+ # required but ignored
+ api_key="ollama",
+)
+
+# Log all traces made to with the OpenAI client to Opik
+client = track_openai(client)
+
+# call the local ollama model using the OpenAI client
+chat_completion = client.chat.completions.create(
+ messages=[
+ {
+ "role": "user",
+ "content": "Say this is a test",
+ }
+ ],
+ model="llama3.1",
+)
+
+print(chat_completion.choices[0].message.content)
+```
+
+Your LLM call is now traced and logged to the Opik platform.
+
+## Tracking Ollama calls made with LangChain
+
+In order to trace Ollama calls made with LangChain, you will need to first install the `langchain-ollama` package:
+
+
+```python
+%pip install --quiet --upgrade langchain-ollama
+```
+
+You will now be able to use the `OpikTracer` class to log all your Ollama calls made with LangChain to Opik:
+
+
+```python
+from langchain_ollama import ChatOllama
+from opik.integrations.langchain import OpikTracer
+
+# Create the Opik tracer
+opik_tracer = OpikTracer(tags=["langchain", "ollama"])
+
+# Create the Ollama model and configure it to use the Opik tracer
+llm = ChatOllama(
+ model="llama3.1",
+ temperature=0,
+).with_config({"callbacks": [opik_tracer]})
+
+# Call the Ollama model
+messages = [
+ (
+ "system",
+ "You are a helpful assistant that translates English to French. Translate the user sentence.",
+ ),
+ (
+ "human",
+ "I love programming.",
+ ),
+]
+ai_msg = llm.invoke(messages)
+ai_msg
+```
+
+You can now go to the Opik app to see the trace:
+
+
diff --git a/apps/opik-documentation/documentation/fern/docs/cookbook/openai.mdx b/apps/opik-documentation/documentation/fern/docs/cookbook/openai.mdx
new file mode 100644
index 0000000000..87a26fa27d
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/cookbook/openai.mdx
@@ -0,0 +1,120 @@
+---
+description: Cookbook that showcases Opik's integration with the OpenAI Python SDK
+---
+
+# Using Opik with OpenAI
+
+Opik integrates with OpenAI to provide a simple way to log traces for all OpenAI LLM calls. This works for all OpenAI models, including if you are using the streaming API.
+
+
+## Creating an account on Comet.com
+
+[Comet](https://www.comet.com/site?from=llm&utm_source=opik&utm_medium=colab&utm_content=openai&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm&utm_source=opik&utm_medium=colab&utm_content=openai&utm_campaign=opik) and grab you API Key.
+
+> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm&utm_source=opik&utm_medium=colab&utm_content=openai&utm_campaign=opik) for more information.
+
+
+```python
+%pip install --upgrade opik openai
+```
+
+
+```python
+import opik
+
+opik.configure(use_local=False)
+```
+
+## Preparing our environment
+
+First, we will set up our OpenAI API keys.
+
+
+```python
+import os
+import getpass
+
+if "OPENAI_API_KEY" not in os.environ:
+ os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
+```
+
+## Logging traces
+
+In order to log traces to Opik, we need to wrap our OpenAI calls with the `track_openai` function:
+
+
+```python
+from opik.integrations.openai import track_openai
+from openai import OpenAI
+
+os.environ["OPIK_PROJECT_NAME"] = "openai-integration-demo"
+
+client = OpenAI()
+openai_client = track_openai(client)
+```
+
+
+```python
+prompt = """
+Write a short two sentence story about Opik.
+"""
+
+completion = openai_client.chat.completions.create(
+ model="gpt-3.5-turbo", messages=[{"role": "user", "content": prompt}]
+)
+
+print(completion.choices[0].message.content)
+```
+
+The prompt and response messages are automatically logged to Opik and can be viewed in the UI.
+
+
+
+## Using it with the `track` decorator
+
+If you have multiple steps in your LLM pipeline, you can use the `track` decorator to log the traces for each step. If OpenAI is called within one of these steps, the LLM call with be associated with that corresponding step:
+
+
+```python
+from opik import track
+from opik.integrations.openai import track_openai
+from openai import OpenAI
+
+os.environ["OPIK_PROJECT_NAME"] = "openai-integration-demo"
+
+client = OpenAI()
+openai_client = track_openai(client)
+
+
+@track
+def generate_story(prompt):
+ res = openai_client.chat.completions.create(
+ model="gpt-3.5-turbo", messages=[{"role": "user", "content": prompt}]
+ )
+ return res.choices[0].message.content
+
+
+@track
+def generate_topic():
+ prompt = "Generate a topic for a story about Opik."
+ res = openai_client.chat.completions.create(
+ model="gpt-3.5-turbo", messages=[{"role": "user", "content": prompt}]
+ )
+ return res.choices[0].message.content
+
+
+@track
+def generate_opik_story():
+ topic = generate_topic()
+ story = generate_story(topic)
+ return story
+
+
+generate_opik_story()
+```
+
+The trace can now be viewed in the UI:
+
+
+
+
diff --git a/apps/opik-documentation/documentation/fern/docs/cookbook/overview.mdx b/apps/opik-documentation/documentation/fern/docs/cookbook/overview.mdx
new file mode 100644
index 0000000000..056e24341a
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/cookbook/overview.mdx
@@ -0,0 +1,94 @@
+---
+title: Opik Cookbooks
+subtitle: An open-source collection of notebooks and guides for using the Opik platform.
+---
+
+## Guides
+
+
+
In this section, we will walk through all the concepts associated with Opik's evaluation framework.
@@ -28,15 +28,16 @@ Given the importance of datasets in the evaluation process, teams often spend a
1. **Manually curating examples**: As a first step, you can manually curate a set of examples based on your knowledge of the application you are building. You can also leverage subject matter experts to help in the creation of the dataset.
-2. **Using synthetic data**: If you don't have enough data to create a diverse set of examples, you can turn to synthetic data generation tools to help you create a dataset. The [LangChain cookbook](/docs/cookbook/langchain.md) has a great example of how to use synthetic data generation tools to create a dataset.
+2. **Using synthetic data**: If you don't have enough data to create a diverse set of examples, you can turn to synthetic data generation tools to help you create a dataset. The [LangChain cookbook](/cookbook/langchain) has a great example of how to use synthetic data generation tools to create a dataset.
3. **Leveraging production data**: If you application is in production, you can leverage the data that is being generated to augment your dataset. While this is often not the first step in creating a dataset, it can be a great way to to enrich your dataset with real world data.
If you are using Opik for production monitoring, you can easily add traces to your dataset by selecting them in the UI and selecting `Add to dataset` in the `Actions` dropdown.
-:::tip
-You can learn more about how to manage your datasets in Opik in the [Manage Datasets](/docs/evaluation/manage_datasets.md) section.
-:::
+
+
### Experiment Items
Experiment items store the input, expected output, actual output and feedback scores for each dataset sample that was processed during an experiment. In addition, a trace is associated with each item to allow you to easily understand why a given item scored the way it did.
-
+
+ 
+
## Learn more
We have provided some guides to help you get started with Opik's evaluation framework:
-1. [Overview of Opik's evaluation features](/docs/evaluation/overview.mdx)
-2. [Evaluate prompts](/docs/evaluation/evaluate_prompt.md)
-3. [Evaluate your LLM application](/docs/evaluation/evaluate_your_llm.md)
+1. [Overview of Opik's evaluation features](/evaluation/overview)
+2. [Evaluate prompts](/evaluation/evaluate_prompt)
+3. [Evaluate your LLM application](/evaluation/evaluate_your_llm)
diff --git a/apps/opik-documentation/documentation/docs/evaluation/evaluate_prompt.md b/apps/opik-documentation/documentation/fern/docs/evaluation/evaluate_prompt.mdx
similarity index 92%
rename from apps/opik-documentation/documentation/docs/evaluation/evaluate_prompt.md
rename to apps/opik-documentation/documentation/fern/docs/evaluation/evaluate_prompt.mdx
index 11415a6745..3315770096 100644
--- a/apps/opik-documentation/documentation/docs/evaluation/evaluate_prompt.md
+++ b/apps/opik-documentation/documentation/fern/docs/evaluation/evaluate_prompt.mdx
@@ -1,15 +1,14 @@
---
-sidebar_label: Evaluate Prompts
-description: Step by step guide on how to evaluate LLM prompts
+subtitle: Step by step guide on how to evaluate LLM prompts
---
-# Evaluate Prompts
-
When developing prompts and performing prompt engineering, it can be challenging to know if a new prompt is better than the previous version.
Opik Experiments allow you to evaluate the prompt on multiple samples, score each LLM output and compare the performance of different prompts.
-
+
+
+
There are two way to evaluate a prompt in Opik:
@@ -56,13 +55,15 @@ evaluate_prompt(
Once the evaluation is complete, you can view the responses in the Opik UI and score each LLM output.
-
+
+
+
### Automate the scoring process
Manually reviewing each LLM output can be time-consuming and error-prone. The `evaluate_prompt` function allows you to specify a list of scoring metrics which allows you to score each LLM output. Opik has a set of built-in metrics that allow you to detect hallucinations, answer relevance, etc and if we don't have the metric you need, you can easily create your own.
-You can find a full list of all the Opik supported metrics in the [Metrics Overview](/evaluation/metrics/overview.md) section or you can define your own metric using [Custom Metrics](/evaluation/metrics/custom_metric.md).
+You can find a full list of all the Opik supported metrics in the [Metrics Overview](/evaluation/metrics/overview) section or you can define your own metric using [Custom Metrics](/evaluation/metrics/custom_metric) section.
By adding the `scoring_metrics` parameter to the `evaluate_prompt` function, you can specify a list of metrics to use for scoring. We will update the example above to use the `Hallucination` metric for scoring:
@@ -121,4 +122,4 @@ evaluate_prompt(
## Next steps
-To evaluate complex LLM applications like RAG applications or agents, you can use the [`evaluate`](/evaluation/evaluate_your_llm.md) function.
+To evaluate complex LLM applications like RAG applications or agents, you can use the [`evaluate`](/evaluation/evaluate_your_llm) function.
diff --git a/apps/opik-documentation/documentation/docs/evaluation/evaluate_your_llm.md b/apps/opik-documentation/documentation/fern/docs/evaluation/evaluate_your_llm.mdx
similarity index 86%
rename from apps/opik-documentation/documentation/docs/evaluation/evaluate_your_llm.md
rename to apps/opik-documentation/documentation/fern/docs/evaluation/evaluate_your_llm.mdx
index 00c5e7480d..48a7f9fb10 100644
--- a/apps/opik-documentation/documentation/docs/evaluation/evaluate_your_llm.md
+++ b/apps/opik-documentation/documentation/fern/docs/evaluation/evaluate_your_llm.mdx
@@ -1,16 +1,13 @@
---
-sidebar_label: Evaluate Complex LLM Applications
-description: Step by step guide on how to evaluate your LLM application
-pytest_codeblocks_execute_previous: true
+subtitle: Step by step guide on how to evaluate your LLM application
---
-# Evaluate Complex LLM Applications
-
Evaluating your LLM application allows you to have confidence in the performance of your LLM application. In this guide, we will walk through the process of evaluating complex applications like LLM chains or agents.
-:::tip
-In this guide, we will focus on evaluating complex LLM applications. If you are looking at evaluating single prompts you can refer to the [Evaluate A Prompt](/evaluation/evaluate_prompt.md) guide.
-:::
+
+
### Logging traces to a specific project
@@ -242,7 +246,7 @@ In order to evaluate datasets more efficiently, Opik uses multiple background th
You can access all the experiments logged to the platform from the SDK with the [`Opik.get_experiments_by_name`](https://www.comet.com/docs/opik/python-sdk-reference/Opik.html#opik.Opik.get_experiment_by_name) and [`Opik.get_experiment_by_id`](https://www.comet.com/docs/opik/python-sdk-reference/Opik.html#opik.Opik.get_experiment_by_id) methods:
-```python pytest_codeblocks_skip=true
+```python {pytest_codeblocks_skip=true}
import opik
# Get the experiment
diff --git a/apps/opik-documentation/documentation/docs/evaluation/manage_datasets.md b/apps/opik-documentation/documentation/fern/docs/evaluation/manage_datasets.mdx
similarity index 81%
rename from apps/opik-documentation/documentation/docs/evaluation/manage_datasets.md
rename to apps/opik-documentation/documentation/fern/docs/evaluation/manage_datasets.mdx
index 192a9bf253..56c3750341 100644
--- a/apps/opik-documentation/documentation/docs/evaluation/manage_datasets.md
+++ b/apps/opik-documentation/documentation/fern/docs/evaluation/manage_datasets.mdx
@@ -1,10 +1,7 @@
---
-sidebar_label: Manage Datasets
-description: Guides you through the process of creating and managing datasets
+subtitle: Guides you through the process of creating and managing datasets
---
-# Manage Datasets
-
Datasets can be used to track test cases you would like to evaluate your LLM on. Each dataset is made up of dictionary
with any key value pairs. When getting started, we recommend having an `input` and optional `expected_output` fields for
example. These datasets can be created from:
@@ -50,21 +47,23 @@ dataset.insert([
])
```
-:::tip
-Opik automatically deduplicates items that are inserted into a dataset when using the Python SDK. This means that you
-can insert the same item multiple times without duplicating it in the dataset. This combined with
-the `get_or_create_dataset` method means that you can use the SDK to manage your datasets in a "fire and forget" manner.
-:::
+
+
#### Inserting items from a JSONL file
You can also insert items from a JSONL file:
-```python pytest_codeblocks_skip=true
+```python {pytest_codeblocks_skip=true}
dataset.read_jsonl_from_file("path/to/file.jsonl")
```
@@ -79,13 +78,13 @@ The format of the JSONL file should be a JSON object per line. For example:
You can also insert items from a Pandas DataFrame:
-```python pytest_codeblocks_skip=true
+```python {pytest_codeblocks_skip=true}
dataset.insert_from_pandas(dataframe=df)
```
The `keys_mapping` parameter maps the column names in the DataFrame to the keys in the dataset items, this can be useful if you want to rename columns before inserting them into the dataset:
-```python pytest_codeblocks_skip=true
+```python {pytest_codeblocks_skip=true}
dataset.insert_from_pandas(dataframe=df, keys_mapping={"Expected output": "expected_output"})
```
@@ -93,7 +92,7 @@ dataset.insert_from_pandas(dataframe=df, keys_mapping={"Expected output": "expec
You can delete items in a dataset by using the `delete` method:
-```python pytest_codeblocks_skip=true
+```python {pytest_codeblocks_skip=true}
from opik import Opik
# Get or create a dataset
@@ -103,10 +102,10 @@ dataset = client.get_dataset(name="My dataset")
dataset.delete(items_ids=["123", "456"])
```
-:::tip
+
+
In the experiment pages, you will be able to:
@@ -145,8 +144,8 @@ In the experiment pages, you will be able to:
You can learn more about Opik's evaluation features in:
-1. [Evaluation concepts](/evaluation/concepts.md)
-1. [Evaluate prompts](/evaluation/evaluate_prompt.md)
-1. [Evaluate complex LLM applications](/evaluation/evaluate_your_llm.md)
-1. [Evaluation metrics](/evaluation/metrics/overview.md)
-1. [Manage datasets](/evaluation/manage_datasets.md)
+1. [Evaluation concepts](/evaluation/concepts)
+1. [Evaluate prompts](/evaluation/evaluate_prompt)
+1. [Evaluate complex LLM applications](/evaluation/evaluate_your_llm)
+1. [Evaluation metrics](/evaluation/metrics/overview)
+1. [Manage datasets](/evaluation/manage_datasets)
diff --git a/apps/opik-documentation/documentation/docs/evaluation/update_existing_experiment.md b/apps/opik-documentation/documentation/fern/docs/evaluation/update_existing_experiment.mdx
similarity index 82%
rename from apps/opik-documentation/documentation/docs/evaluation/update_existing_experiment.md
rename to apps/opik-documentation/documentation/fern/docs/evaluation/update_existing_experiment.mdx
index 9e3ee22204..b2939bf812 100644
--- a/apps/opik-documentation/documentation/docs/evaluation/update_existing_experiment.md
+++ b/apps/opik-documentation/documentation/fern/docs/evaluation/update_existing_experiment.mdx
@@ -1,15 +1,12 @@
---
-sidebar_label: Update an Existing Experiment
-description: Guides you through the process of updating an existing experiment
+subtitle: Guides you through the process of updating an existing experiment
---
-# Update an Existing Experiment
-
Sometimes you may want to update an existing experiment with new scores, or update existing scores for an experiment. You can do this using the [`evaluate_experiment` function](https://www.comet.com/docs/opik/python-sdk-reference/evaluation/evaluate_existing.html).
This function will re-run the scoring metrics on the existing experiment items and update the scores:
-```python pytest_codeblocks_skip=true
+```python {pytest_codeblocks_skip=true}
from opik.evaluation import evaluate_experiment
from opik.evaluation.metrics import Hallucination
@@ -19,9 +16,10 @@ hallucination_metric = Hallucination()
evaluate_experiment(experiment_name="my-experiment", scoring_metrics=[hallucination_metric])
```
-:::tip
-The `evaluate_experiment` function can be used to update existing scores for an experiment. If you use a scoring metric with the same name as an existing score, the scores will be updated with the new values.
-:::
+
+
-:::tip
-Opik is Open Source! You can find the full source code on [GitHub](https://github.com/comet-ml/opik) and the complete self-hosting guide can be found [here](/self-host/local_deployment.md).
-:::
+
+
diff --git a/apps/opik-documentation/documentation/docs/production/production_monitoring.md b/apps/opik-documentation/documentation/fern/docs/production/production_monitoring.mdx
similarity index 83%
rename from apps/opik-documentation/documentation/docs/production/production_monitoring.md
rename to apps/opik-documentation/documentation/fern/docs/production/production_monitoring.mdx
index 0276d32609..d24d6c145c 100644
--- a/apps/opik-documentation/documentation/docs/production/production_monitoring.md
+++ b/apps/opik-documentation/documentation/fern/docs/production/production_monitoring.mdx
@@ -1,33 +1,31 @@
---
-sidebar_label: Production Monitoring
-description: Describes how to monitor your LLM applications in production using Opik
+subtitle: Describes how to monitor your LLM applications in production using Opik
---
-# Production Monitoring
-
Opik has been designed from the ground up to support high volumes of traces making it the ideal tool for monitoring your production LLM applications.
You can use the Opik dashboard to review your feedback scores, trace count and tokens over time at both a daily and hourly granularity.
-
+
+ 
+
In addition to viewing scores over time, you can also view the average feedback scores for all the traces in your project from the traces table.
## Logging feedback scores
-To monitor the performance of your LLM application, you can log feedback scores using the [Python SDK and through the UI](/tracing/annotate_traces.md).
+To monitor the performance of your LLM application, you can log feedback scores using the [Python SDK and through the UI](/tracing/annotate_traces).
### Defining online evaluation metrics
-You can define LLM as a Judge metrics in the Opik platform that will automatically score all, or a subset, of your production traces. You can find more information about how to define LLM as a Judge metrics in the [Online evaluation](/production/rules.md) section.
-
-
+You can define LLM as a Judge metrics in the Opik platform that will automatically score all, or a subset, of your production traces. You can find more information about how to define LLM as a Judge metrics in the [Online evaluation](/production/rules) section.
Once a rule is defined, Opik will score all the traces in the project and allow you to track these feedback scores over time.
-:::tip
-In addition to allowing you to define LLM as a Judge metrics, Opik will soon allow you to define Python metrics to give you even more control over the feedback scores.
-:::
+
+
When creating a new rule, you will be presented with the following options:
@@ -42,18 +42,23 @@ Opik comes pre-configured with 3 different LLM as a Judge metrics:
2. Moderation: This metric checks if the LLM output contains any offensive content.
3. Answer Relevance: This metric checks if the LLM output is relevant to the given context.
-:::tip
-If you would like us to add more LLM as a Judge metrics to the platform, do raise an issue on [GitHub](https://github.com/comet-ml/opik/issues) and we will do our best to add them !
-:::
+
+
+When writing your own LLM as a Judge metric you will need to specify the prompt variables using the mustache syntax, ie.
+`{{ variable_name }}`. You can then map these variables to your trace data using the `variable_mapping` parameter. When the
+rule is executed, Opik will replace the variables with the values from the trace data.
You can control the format of the output using the `Scoring definition` parameter. This is were you can define the scores you want the LLM as a Judge metric to return. Under the hood, we will use this definition in conjunction with the [structured outputs](https://platform.openai.com/docs/guides/structured-outputs) functionality to ensure that the the LLM as a Judge metric always returns trace scores.
diff --git a/apps/opik-documentation/documentation/docs/prompt_engineering/managing_prompts_in_code.mdx b/apps/opik-documentation/documentation/fern/docs/prompt_engineering/managing_prompts_in_code.mdx
similarity index 73%
rename from apps/opik-documentation/documentation/docs/prompt_engineering/managing_prompts_in_code.mdx
rename to apps/opik-documentation/documentation/fern/docs/prompt_engineering/managing_prompts_in_code.mdx
index 72ac7fc4ee..d9cb01dba7 100644
--- a/apps/opik-documentation/documentation/docs/prompt_engineering/managing_prompts_in_code.mdx
+++ b/apps/opik-documentation/documentation/fern/docs/prompt_engineering/managing_prompts_in_code.mdx
@@ -1,18 +1,8 @@
----
-sidebar_label: Versioning prompts stored in code
-description: Describes how to version prompts stored in code
----
-
-import Tabs from "@theme/Tabs";
-import TabItem from "@theme/TabItem";
-
-# Managing prompts stored in code
-
If you already have prompts stored in code, you can use the the [`Prompt`](https://www.comet.com/docs/opik/python-sdk-reference/library/Prompt.html) object in the SDK to sync these prompts with the library.
This allows you to store the prompt text in your code while also having it versioned and stored in the library:
+
-:::tip
+
+
+
+_All of the conversations from the playground are logged to the `playground` project so that you can easily refer back to them later._
+
+## Configuring the prompt playground
+
+The playground supports the following LLM providers:
+
+- OpenAI
+- Anthropic
+- Ollama
+- LM Studio (coming soon)
+
+
+
+
+
+
+
+_Don't forget to update the model list with the models supported by your proxy server!_
+
+
+
You can also edit a prompt by clicking on the prompt name in the library and clicking `Edit prompt`.
-
+
diff --git a/apps/opik-documentation/documentation/docs/quickstart.mdx b/apps/opik-documentation/documentation/fern/docs/quickstart.mdx
similarity index 69%
rename from apps/opik-documentation/documentation/docs/quickstart.mdx
rename to apps/opik-documentation/documentation/fern/docs/quickstart.mdx
index ac2e393222..28b1f7f1f9 100644
--- a/apps/opik-documentation/documentation/docs/quickstart.mdx
+++ b/apps/opik-documentation/documentation/fern/docs/quickstart.mdx
@@ -1,45 +1,37 @@
----
-sidebar_label: Quickstart
-description: This guide helps you integrate the Opik platform with your existing LLM application
----
-
-import Tabs from "@theme/Tabs";
-import TabItem from "@theme/TabItem";
-
-# Quickstart
-
This guide helps you integrate the Opik platform with your existing LLM application. The goal of this guide is to help you log your first LLM calls and chains to the Opik platform.
-
+
+ 
+
## Set up
-Getting started is as simple as creating an [account on Comet](https://www.comet.com/signup?from=llm) or [self-hosting the platform](/self-host/overview.md).
+Getting started is as simple as creating an [account on Comet](https://www.comet.com/signup?from=llm) or [self-hosting the platform](/self-host/overview).
Once your account is created, you can start logging traces by installing the Opik Python SDK:
+
+
+## Backend Service
+
+Opik's backend uses Java 21 LTS and Dropwizard 4, structured as a RESTful web service offering public API
+endpoints for core functionality. Full API documentation is available [here](/reference/rest-api/overview).
+
+For observability Opik uses OpenTelemetry due its vendor-neutral approach and wide support across languages
+and frameworks. It provides a single, consistent way to collect telemetry data from all services and applications.
+
+_You can find the full backend codebase in Github under the [`apps/opik-backend`](https://github.com/comet-ml/opik/tree/main/apps/opik-backend) folder._
+
+## Frontend Application
+
+Opik's frontend is a TypeScript + React application served by Nginx. It provides a user-friendly interface for
+interacting with the backend services. The frontend is built using a modular approach, with each module
+encapsulating a specific feature or functionality.
+
+_You can find the full frontend codebase in Github under the [`apps/opik-frontend`](https://github.com/comet-ml/opik/tree/main/apps/opik-frontend) folder._
+
+## SDK's
+
+Opik provides SDKs for Python, and JavaScript. These SDKs allow developers to interact with Opik's backend
+services programmatically. The SDKs are designed to be easy to use and provide a high-level abstraction over
+the REST API and many additional features.
+
+_You can find the full SDK codebase in Github under the [`sdks/python`](https://github.com/comet-ml/opik/tree/main/sdks/python) for the Python SDK
+and [`sdks/typescript`](https://github.com/comet-ml/opik/tree/main/sdks/typescript) for the TypeScript SDK._
+
+## ClickHouse
+
+ClickHouse is a column-oriented database management system developed by Yandex. It is optimized for fast analytics on large datasets and is capable of processing hundreds of millions to more than a billion rows and tens of gigabytes of data per single server per second.
+
+Opik uses ClickHouse for datasets that require near real-time ingestion and analytical queries, such as:
+
+- LLM calls and traces
+- Feedback scores
+- Datasets and experiments
+- Experiments
+
+The image below details the schema used by Opik in ClickHouse:
+
+
+ 
+
+
+ Liquibase automates schema management
+
+
+## MySQL
+
+Opik uses MySQL for transactional data, it provides ACID-compliant transactional storage for Opik's lower-volume but critical data, such as:
+
+- Feedback definitions
+- Metadata containers e.g., projects that group related traces
+- Configuration data
+
+The image below details the schema used by Opik in MySQL:
+
+
+ 
+
+
+ Liquibase automates schema management
+
+
+## Redis
+
+Redis is an in-memory data structure store, used as a database, cache, and message broker. It supports a vast range of data structures. Opik uses Redis for:
+
+- A distributed cache: for high-speed lookups.
+- A distributed lock: for coordinating safe access to certain shared resources.
+- A rate limiter: to enforce throughput limits and protect scalability.
+- A streaming mechanism: Redis streams power Opik's Online evaluation functionality; future iterations may integrate Kafka or similar platforms for even higher scalability.
+
+## Observability
+
+Opik is built and runs on top of open-source infrastructure (MySQL, Redis, Kubernetes, and more), making it straightforward to integrate with popular observability stacks such as Grafana and Prometheus. Specifically:
+
+- The backend uses OpenTelemetry for vendor-neutral instrumentation.
+- ClickHouse deployments include an operator for real-time performance monitoring and metric exports to Grafana/Prometheus.
+- Other components (MySQL, Redis, Kubernetes) also have well-documented strategies for monitoring.
diff --git a/apps/opik-documentation/documentation/docs/self-host/configure/anonymous_usage_statistics.md b/apps/opik-documentation/documentation/fern/docs/self-host/configure/anonymous_usage_statistics.mdx
similarity index 92%
rename from apps/opik-documentation/documentation/docs/self-host/configure/anonymous_usage_statistics.md
rename to apps/opik-documentation/documentation/fern/docs/self-host/configure/anonymous_usage_statistics.mdx
index 9ddab38eb5..e06cb4d979 100644
--- a/apps/opik-documentation/documentation/docs/self-host/configure/anonymous_usage_statistics.md
+++ b/apps/opik-documentation/documentation/fern/docs/self-host/configure/anonymous_usage_statistics.mdx
@@ -1,10 +1,7 @@
---
-sidebar_label: Anonymous Usage Statistics
-description: Describes the usage statistics that are collected by Opik
+subtitle: Describes the usage statistics that are collected by Opik
---
-# Anonymous Usage Statistics
-
Opik includes a system that optionally sends anonymous reports non-sensitive, non-personally identifiable information about the usage of the Opik platform. This information is used to help us understand how the Opik platform is being used and to identify areas for improvement.
The anonymous usage statistics reporting is enabled by default. You can opt-out by setting the `OPIK_USAGE_REPORT_ENABLED` environment variable to `false`.
diff --git a/apps/opik-documentation/documentation/docs/self-host/kubernetes.md b/apps/opik-documentation/documentation/fern/docs/self-host/kubernetes.mdx
similarity index 88%
rename from apps/opik-documentation/documentation/docs/self-host/kubernetes.md
rename to apps/opik-documentation/documentation/fern/docs/self-host/kubernetes.mdx
index 070528c8b9..db4fe31ac8 100644
--- a/apps/opik-documentation/documentation/docs/self-host/kubernetes.md
+++ b/apps/opik-documentation/documentation/fern/docs/self-host/kubernetes.mdx
@@ -1,11 +1,7 @@
---
-sidebar_label: Production (Kubernetes)
-description: Describes how to run Opik on a Kubernetes cluster
-pytest_codeblocks_skip: true
+subtitle: Describes how to run Opik on a Kubernetes cluster
---
-# Production ready Kubernetes deployment
-
For production deployments, we recommend using our Kubernetes Helm chart. This chart is designed to be highly configurable and has been battle-tested in Comet's managed cloud offering.
## Prerequisites
diff --git a/apps/opik-documentation/documentation/docs/self-host/local_deployment.md b/apps/opik-documentation/documentation/fern/docs/self-host/local_deployment.mdx
similarity index 72%
rename from apps/opik-documentation/documentation/docs/self-host/local_deployment.md
rename to apps/opik-documentation/documentation/fern/docs/self-host/local_deployment.mdx
index 81b1ffbca2..3eb7247255 100644
--- a/apps/opik-documentation/documentation/docs/self-host/local_deployment.md
+++ b/apps/opik-documentation/documentation/fern/docs/self-host/local_deployment.mdx
@@ -1,21 +1,18 @@
---
-sidebar_label: Local (Docker Compose)
-description: Describes how to run Opik locally using Docker Compose
-pytest_codeblocks_skip: true
+subtitle: Describes how to run Opik locally using Docker Compose
---
-# Local Deployments using Docker Compose
-
-To run Opik locally we recommend using [Docker Compose](https://docs.docker.com/compose/). It's easy to setup and allows you to get started in a couple of minutes **but** is not meant for production deployments. If you would like to run Opik in a production environment, we recommend using our [Kubernetes Helm chart](./kubernetes.md).
+To run Opik locally we recommend using [Docker Compose](https://docs.docker.com/compose/). It's easy to setup and allows you to get started in a couple of minutes **but** is not meant for production deployments. If you would like to run Opik in a production environment, we recommend using our [Kubernetes Helm chart](/self-host/kubernetes).
Before running the installation, make sure you have Docker and Docker Compose installed:
- [Docker](https://docs.docker.com/get-docker/)
- [Docker Compose](https://docs.docker.com/compose/install/)
-:::note
-If you are using Mac or Windows, both `docker` and `docker compose` are included in the [Docker Desktop](https://docs.docker.com/desktop/) installation.
-:::
+
+
-:::tip
-If you are evaluating your LLM application during development, we recommend using the `evaluate` function as it will provide you with a more detailed report. You can learn more about the `evaluate` function in the [evaluation documentation](/evaluation/evaluate_your_llm.md).
-:::
+
+
-:::tip
-In order to ensure a consistent set of feedback, you will need to define feedback definitions in the `Feedback Definitions` page which supports both numerical and categorical annotations.
-:::
+
+
### Trace-Level Costs
Opik automatically aggregates costs from all spans within a trace to compute total trace costs:
-
+
+ 
+
### Project-Level Analytics
Track your overall project costs in:
1. The main project view, through the Estimated Cost column:
- 
+
+
+ 
+
2. The project Metrics tab, which shows cost trends over time:
- 
+
+ 
+
## Retrieving Costs Programmatically
-You can retrieve the estimated cost programmatically for both spans and traces. Note that the cost will be `None` if the span or trace used an unsupported model. See [Exporting Traces and Spans](./export_data.md) for more ways of exporting traces and spans.
+You can retrieve the estimated cost programmatically for both spans and traces. Note that the cost will be `None` if the span or trace used an unsupported model. See [Exporting Traces and Spans](/tracing/export_data) for more ways of exporting traces and spans.
### Retrieving Span Costs
@@ -74,6 +79,7 @@ update_current_span(total_cost=0.05) # Cost in USD will override any automatic
```
This is particularly useful when:
+
- Using models or providers not yet supported by automatic cost tracking
- You have a custom pricing agreement with your provider
- You want to track additional costs beyond model usage
@@ -81,9 +87,11 @@ This is particularly useful when:
## Supported Models and Integrations
Opik currently calculates costs automatically for:
-- [OpenAI Integration](./integrations/openai.md) with Text Models hosted on openai.com
-- [Langchain Integration](./integrations/langchain.md) with Vertex AI Gemini text generation models
-:::tip
-We are actively expanding our cost tracking support. Need support for additional models or providers? Please [open a feature request](https://github.com/comet-ml/opik/issues) to help us prioritize development.
-:::
+- [OpenAI Integration](/tracing/integrations/openai) with Text Models hosted on openai.com
+- [Langchain Integration](/tracing/integrations/langchain) with Vertex AI Gemini text generation models
+
+
+
-:::tip
-The UI only allows you to export up to 100 traces or spans at a time as it is linked to the page size of the traces table. If you need to export more traces or spans, we recommend using the Opik SDK.
-:::
+
- You can check out the Colab Notebook if you'd like to jump straight to the code:
-
- 
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/anthropic.md b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/anthropic.mdx
similarity index 76%
rename from apps/opik-documentation/documentation/docs/tracing/integrations/anthropic.md
rename to apps/opik-documentation/documentation/fern/docs/tracing/integrations/anthropic.mdx
index 529414495c..f6528f6aeb 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/anthropic.md
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/anthropic.mdx
@@ -1,18 +1,25 @@
---
-sidebar_label: Anthropic
-description: Describes how to track Anthropic LLM calls using Opik
+subtitle: Describes how to track Anthropic LLM calls using Opik
---
-# Anthropic
-
[Anthropic](https://www.anthropic.com/) is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.
This guide explains how to integrate Opik with the Anthropic Python SDK. By using the `track_anthropic` method provided by opik, you can easily track and evaluate your Anthropic API calls within your Opik projects as Opik will automatically log the input prompt, model used, token usage, and response generated.
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
- You can check out the Colab Notebook if you'd like to jump straight to the code:
-
-
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
@@ -38,7 +45,7 @@ In order to configure Anthropic, you will need to have your Anthropic API Key se
Once you have it, you can set it as an environment variable:
-```bash pytest_codeblocks_skip=true
+```bash {pytest_codeblocks_skip=true}
export ANTHROPIC_API_KEY="YOUR_API_KEY"
```
@@ -65,4 +72,6 @@ response = anthropic_client.messages.create(
print("Response", response.content[0].text)
```
-
+
+
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
+
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/bedrock.md b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/bedrock.mdx
similarity index 81%
rename from apps/opik-documentation/documentation/docs/tracing/integrations/bedrock.md
rename to apps/opik-documentation/documentation/fern/docs/tracing/integrations/bedrock.mdx
index 583e5deea1..530786e0e0 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/bedrock.md
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/bedrock.mdx
@@ -1,19 +1,25 @@
---
-sidebar_label: Bedrock
-description: Describes how to track Bedrock LLM calls using Opik
-pytest_codeblocks_skip: true
+subtitle: Describes how to track Bedrock LLM calls using Opik
---
-# AWS Bedrock
-
[AWS Bedrock](https://aws.amazon.com/bedrock/) is a fully managed service that provides access to high-performing foundation models (FMs) from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon through a single API.
This guide explains how to integrate Opik with the Bedrock Python SDK. By using the `track_bedrock` method provided by opik, you can easily track and evaluate your Bedrock API calls within your Opik projects as Opik will automatically log the input prompt, model used, token usage, and response generated.
- You can check out the Colab Notebook if you'd like to jump straight to the code:
-
-
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
@@ -76,4 +82,6 @@ response = bedrock_client.converse(
print("Response", response["output"]["message"]["content"][0]["text"])
```
-
+
+
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
+
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/crewai.md b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/crewai.mdx
similarity index 80%
rename from apps/opik-documentation/documentation/docs/tracing/integrations/crewai.md
rename to apps/opik-documentation/documentation/fern/docs/tracing/integrations/crewai.mdx
index 2dc9acf6ba..3d73c3d60b 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/crewai.md
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/crewai.mdx
@@ -1,18 +1,25 @@
---
-sidebar_label: CrewAI
-description: Describes how to track CrewAI calls using Opik
+subtitle: Describes how to track CrewAI calls using Opik
---
-# CrewAI
-
[CrewAI](https://www.crewai.com/) is a cutting-edge framework for orchestrating autonomous AI agents.
Opik integrates with CrewAI to log traces for all CrewAI activity.
- You can check out the Colab Notebook if you'd like to jump straight to the code:
-
-
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
@@ -92,4 +99,6 @@ print(result)
Each run will now be logged to the Opik platform:
-
+
+
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
+
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/dify.mdx b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/dify.mdx
similarity index 72%
rename from apps/opik-documentation/documentation/docs/tracing/integrations/dify.mdx
rename to apps/opik-documentation/documentation/fern/docs/tracing/integrations/dify.mdx
index ec3f5bf454..2e669ef1e7 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/dify.mdx
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/dify.mdx
@@ -1,14 +1,7 @@
---
-sidebar_label: Dify
-description: Describes how to use Opik with Dify
-pytest_codeblocks_skip: true
+subtitle: Describes how to use Opik with Dify
---
-import Tabs from "@theme/Tabs";
-import TabItem from "@theme/TabItem";
-
-# Dify Integration
-
Learn how to connect Opik with Dify to monitor your applications' performance.
## Setup Instructions
@@ -21,10 +14,12 @@ Follow these simple steps to connect Dify with Opik:
4. Click on **Configure** for Opik
5. Enter your connection details based on your Opik version:
-
+
+ 
+
+
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/dspy.md b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/dspy.mdx
similarity index 67%
rename from apps/opik-documentation/documentation/docs/tracing/integrations/dspy.md
rename to apps/opik-documentation/documentation/fern/docs/tracing/integrations/dspy.mdx
index 3e3a0d94de..300da04f46 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/dspy.md
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/dspy.mdx
@@ -1,18 +1,25 @@
---
-sidebar_label: DSPy
-description: Describes how to track DSPy calls using Opik
+subtitle: Describes how to track DSPy calls using Opik
---
-# DSPy
-
[DSPy](https://dspy.ai/) is the framework for programming—rather than prompting—language models.
Opik integrates with DSPy to log traces for all DSPy calls.
- You can check out the Colab Notebook if you'd like to jump straight to the code:
-
-
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
@@ -57,4 +64,6 @@ cot(question="What is the meaning of life?")
Each run will now be logged to the Opik platform:
-
+
+
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
+
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/gemini.md b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/gemini.mdx
similarity index 71%
rename from apps/opik-documentation/documentation/docs/tracing/integrations/gemini.md
rename to apps/opik-documentation/documentation/fern/docs/tracing/integrations/gemini.mdx
index c2f63b4a2c..25d9ddac15 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/gemini.md
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/gemini.mdx
@@ -1,17 +1,23 @@
---
-sidebar_label: Gemini - Google AI Studio
-description: Describes how to track Gemini LLM calls using Opik
-pytest_codeblocks_skip: true
+subtitle: Describes how to track Gemini LLM calls using Opik
---
-# Gemini - Google AI Studio
-
[Gemini](https://aistudio.google.com/welcome) is a family of multimodal large language models developed by Google DeepMind.
- You can check out the Colab Notebook if you'd like to jump straight to the code:
-
-
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
@@ -19,7 +25,7 @@ pytest_codeblocks_skip: true
### Configuring Opik
-To start tracking your Gemini LLM calls, you can use our [LiteLLM integration](/tracing/integrations/litellm.md). You'll need to have both the `opik`, `litellm` and `google-generativeai` packages installed. You can install them using pip:
+To start tracking your Gemini LLM calls, you can use our [LiteLLM integration](/tracing/integrations/litellm). You'll need to have both the `opik`, `litellm` and `google-generativeai` packages installed. You can install them using pip:
```bash
pip install opik litellm google-generativeai
@@ -31,11 +37,11 @@ In addition, you can configure Opik using the `opik configure` command which wil
opik configure
```
-:::info
+
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
+
## Logging LLM calls within a tracked function
-If you are using LiteLLM within a function tracked with the [`@track`](/tracing/log_traces.mdx#using-function-decorators) decorator, you will need to pass the `current_span_data` as metadata to the `litellm.completion` call:
+If you are using LiteLLM within a function tracked with the [`@track`](/tracing/log_traces#using-function-decorators) decorator, you will need to pass the `current_span_data` as metadata to the `litellm.completion` call:
```python
from opik import track, opik_context
@@ -119,4 +127,6 @@ def generate_opik_story():
generate_opik_story()
```
-
+
+ 
+
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/groq.md b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/groq.mdx
similarity index 72%
rename from apps/opik-documentation/documentation/docs/tracing/integrations/groq.md
rename to apps/opik-documentation/documentation/fern/docs/tracing/integrations/groq.mdx
index 7e0d72735a..41e63f7266 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/groq.md
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/groq.mdx
@@ -1,17 +1,23 @@
---
-sidebar_label: Groq
-description: Describes how to track Groq LLM calls using Opik
-pytest_codeblocks_skip: true
+subtitle: Describes how to track Groq LLM calls using Opik
---
-# Groq
-
[Groq](https://groq.com/) is Fast AI Inference.
- You can check out the Colab Notebook if you'd like to jump straight to the code:
-
-
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
@@ -19,7 +25,7 @@ pytest_codeblocks_skip: true
### Configuring Opik
-To start tracking your Groq LLM calls, you can use our [LiteLLM integration](/tracing/integrations/litellm.md). You'll need to have both the `opik` and `litellm` packages installed. You can install them using pip:
+To start tracking your Groq LLM calls, you can use our [LiteLLM integration](/tracing/integrations/litellm). You'll need to have both the `opik` and `litellm` packages installed. You can install them using pip:
```bash
pip install opik litellm
@@ -31,11 +37,11 @@ In addition, you can configure Opik using the `opik configure` command which wil
opik configure
```
-:::info
+
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
+
## Logging LLM calls within a tracked function
-If you are using LiteLLM within a function tracked with the [`@track`](/tracing/log_traces.mdx#using-function-decorators) decorator, you will need to pass the `current_span_data` as metadata to the `litellm.completion` call:
+If you are using LiteLLM within a function tracked with the [`@track`](/tracing/log_traces#using-function-decorators) decorator, you will need to pass the `current_span_data` as metadata to the `litellm.completion` call:
```python
from opik import track, opik_context
@@ -119,4 +127,6 @@ def generate_opik_story():
generate_opik_story()
```
-
+
+ 
+
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/guardrails-ai.md b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/guardrails-ai.mdx
similarity index 79%
rename from apps/opik-documentation/documentation/docs/tracing/integrations/guardrails-ai.md
rename to apps/opik-documentation/documentation/fern/docs/tracing/integrations/guardrails-ai.mdx
index 6b4e7f1ec6..bbb5277c55 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/guardrails-ai.md
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/guardrails-ai.mdx
@@ -1,10 +1,7 @@
---
-sidebar_label: Guardrails AI
-description: Cookbook that showcases Opik's integration with the Guardrails AI Python SDK
+subtitle: Cookbook that showcases Opik's integration with the Guardrails AI Python SDK
---
-# Guardrails AI
-
[Guardrails AI](https://github.com/guardrails-ai/guardrails) is a framework for validating the inputs and outputs
For this guide we will use a simple example that logs guardrails validation steps as traces to Opik, providing them with the validation result tags.
@@ -25,7 +22,7 @@ guardrails hub install hub://guardrails/politeness_check
In order to log traces to Opik, you will need to call the track the Guard object with `track_guardrails` function.
-```python pytest_codeblocks_skip=true
+```python {pytest_codeblocks_skip=true}
from guardrails import Guard, OnFailAction
from guardrails.hub import PolitenessCheck
@@ -50,4 +47,6 @@ Every validation will now be logged to Opik as a trace
The trace will now be viewable in the Opik platform:
-
+
+ 
+
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/haystack.md b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/haystack.mdx
similarity index 93%
rename from apps/opik-documentation/documentation/docs/tracing/integrations/haystack.md
rename to apps/opik-documentation/documentation/fern/docs/tracing/integrations/haystack.mdx
index e493233519..fc5432d428 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/haystack.md
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/haystack.mdx
@@ -1,10 +1,7 @@
---
-sidebar_label: Haystack
-description: Describes how to track Haystack pipeline runs using Opik
+subtitle: Describes how to track Haystack pipeline runs using Opik
---
-# Haystack
-
[Haystack](https://docs.haystack.deepset.ai/docs/intro) is an open-source framework for building production-ready LLM applications, retrieval-augmented generative pipelines and state-of-the-art search systems that work intelligently over large document collections.
Opik integrates with Haystack to log traces for all Haystack pipelines.
@@ -19,7 +16,7 @@ pip install opik haystack-ai
In addition, you can configure Opik using the `opik configure` command which will prompt you for the correct local server address or if you are using the Cloud platform your API key:
-```bash pytest_codeblocks_skip=true
+```bash {pytest_codeblocks_skip=true}
opik configure
```
@@ -73,13 +70,15 @@ print(response["llm"]["replies"][0])
Each pipeline run will now be logged to the Opik platform:
-
+
+ 
+
-:::tip
+
- You can check out the Colab Notebook if you'd like to jump straight to the code:
-
-
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
@@ -138,3 +145,16 @@ from opik.integrations.langchain import OpikTracer
opik_tracer = OpikTracer()
opik_tracer.flush()
```
+
+## Important notes
+1. If you are using asynchronous streaming mode (calling `.astream()` method), the `input` field in the trace UI will be empty due to a Langchain limitation for this mode.
+However, you can find the input data inside the nested spans of this chain.
+
+2. if you are planning to use streaming with LLM-calls, and you want to calculate LLM-call tokens/cost, you need explicitly set argument `stream_usage` to `True`,
+```python
+from langchain_openai import ChatOpenAI
+llm = ChatOpenAI(
+ temperature=0,
+ stream_usage=True,
+)
+```
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/langgraph.md b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/langgraph.mdx
similarity index 84%
rename from apps/opik-documentation/documentation/docs/tracing/integrations/langgraph.md
rename to apps/opik-documentation/documentation/fern/docs/tracing/integrations/langgraph.mdx
index 797a2c6540..46eb21af20 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/langgraph.md
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/langgraph.mdx
@@ -1,16 +1,23 @@
---
-sidebar_label: LangGraph
-description: Describes how to track LangGraph Agent executions using Opik
+subtitle: Describes how to track LangGraph Agent executions using Opik
---
-# LangGraph
-
Opik provides a seamless integration with LangGraph, allowing you to easily log and trace your LangGraph-based applications. By using the `OpikTracer` callback, you can automatically capture detailed information about your LangGraph graph executions during both development and production.
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
- You can check out the Colab Notebook if you'd like to jump straight to the code:
-
-
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
@@ -69,7 +76,9 @@ result = app.invoke({"messages": [HumanMessage(content = "How to use LangGraph ?
Once the OpikTracer is configured, you will start to see the traces in the Opik UI:
-
+
+
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
+
## Updating logged traces
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/litellm.md b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/litellm.mdx
similarity index 90%
rename from apps/opik-documentation/documentation/docs/tracing/integrations/litellm.md
rename to apps/opik-documentation/documentation/fern/docs/tracing/integrations/litellm.mdx
index 283b04a109..01e8575e59 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/litellm.md
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/litellm.mdx
@@ -1,10 +1,7 @@
---
-sidebar_label: LiteLLM
-description: Describes how to track LiteLLM LLM calls using Opik
+subtitle: Describes how to track LiteLLM LLM calls using Opik
---
-# LiteLLM
-
[LiteLLM](https://github.com/BerriAI/litellm) allows you to call all LLM APIs using the OpenAI format [Bedrock, Huggingface, VertexAI, TogetherAI, Azure, OpenAI, Groq etc.]. There are two main ways to use LiteLLM:
1. Using the [LiteLLM Python SDK](https://docs.litellm.ai/docs/#litellm-python-sdk)
@@ -45,11 +42,13 @@ response = litellm.completion(
)
```
-
+
+ 
+
### Logging LLM calls within a tracked function
-If you are using LiteLLM within a function tracked with the [`@track`](/tracing/log_traces.mdx#using-function-decorators) decorator, you will need to pass the `current_span_data` as metadata to the `litellm.completion` call:
+If you are using LiteLLM within a function tracked with the [`@track`](/tracing/log_traces#using-function-decorators) decorator, you will need to pass the `current_span_data` as metadata to the `litellm.completion` call:
```python
from opik import track
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/llama_index.md b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/llama_index.mdx
similarity index 83%
rename from apps/opik-documentation/documentation/docs/tracing/integrations/llama_index.md
rename to apps/opik-documentation/documentation/fern/docs/tracing/integrations/llama_index.mdx
index 509b40d0e4..a2bdc8a8e3 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/llama_index.md
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/llama_index.mdx
@@ -1,10 +1,7 @@
---
-sidebar_label: LlamaIndex
-description: Describes how to track LlamaIndex pipelines using Opik
+subtitle: Describes how to track LlamaIndex pipelines using Opik
---
-# LlamaIndex
-
[LlamaIndex](https://github.com/run-llama/llama_index) is a flexible data framework for building LLM applications:
LlamaIndex is a "data framework" to help you build LLM apps. It provides the following tools:
@@ -15,9 +12,19 @@ description: Describes how to track LlamaIndex pipelines using Opik
- Allows easy integrations with your outer application framework (e.g. with LangChain, Flask, Docker, ChatGPT, anything else).
- You can check out the Colab Notebook if you'd like to jump straight to the code:
-
-
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
@@ -108,4 +115,6 @@ print(response)
Given that the integration with Opik has been set up, all the traces are logged to the Opik platform:
-
+
+
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
+
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/ollama.md b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/ollama.mdx
similarity index 85%
rename from apps/opik-documentation/documentation/docs/tracing/integrations/ollama.md
rename to apps/opik-documentation/documentation/fern/docs/tracing/integrations/ollama.mdx
index 3c4b6c1418..02d2d1ae4a 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/ollama.md
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/ollama.mdx
@@ -1,19 +1,25 @@
---
-sidebar_label: Ollama
-description: Describes how to track Ollama LLM calls using Opik
-pytest_codeblocks_skip: true
+subtitle: Describes how to track Ollama LLM calls using Opik
---
-# Ollama
-
[Ollama](https://ollama.com/) allows users to run, interact with, and deploy AI models locally on their machines without the need for complex infrastructure or cloud dependencies.
There are multiple ways to interact with Ollama from Python including but not limited to the [ollama python package](https://pypi.org/project/ollama/), [LangChain](https://python.langchain.com/docs/integrations/providers/ollama/) or by using the [OpenAI library](https://github.com/ollama/ollama/blob/main/docs/openai.md). We will cover how to trace your LLM calls for each of these methods.
- You can check out the Colab Notebook if you'd like to jump straight to the code:
-
-
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
@@ -35,9 +41,10 @@ pip install --upgrade --quiet opik
opik configure
```
-:::tip
-Opik is fully open-source and can be run locally or through the Opik Cloud platform. You can learn more about hosting Opik on your own infrastructure in the [self-hosting guide](/docs/self-host/overview.md).
-:::
+
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
+
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/openai.md b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/openai.mdx
similarity index 53%
rename from apps/opik-documentation/documentation/docs/tracing/integrations/openai.md
rename to apps/opik-documentation/documentation/fern/docs/tracing/integrations/openai.mdx
index b8690f2b18..9720c86436 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/openai.md
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/openai.mdx
@@ -1,16 +1,23 @@
---
-sidebar_label: OpenAI
-description: Describes how to track OpenAI LLM calls using Opik
+subtitle: Describes how to track OpenAI LLM calls using Opik
---
-# OpenAI
-
This guide explains how to integrate Opik with the OpenAI Python SDK. By using the `track_openai` method provided by opik, you can easily track and evaluate your OpenAI API calls within your Opik projects as Opik will automatically log the input prompt, model used, token usage, and response generated.
- You can check out the Colab Notebook if you'd like to jump straight to the code:
-
-
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
@@ -56,7 +63,33 @@ print(response.choices[0].message.content)
The `track_openai` will automatically track and log the API call, including the input prompt, model used, and response generated. You can view these logs in your Opik project dashboard.
-By following these steps, you can seamlessly integrate Opik with the OpenAI Python SDK and gain valuable insights into your model's performance and usage.
+## Using Azure OpenAI
+
+The OpenAI integration also supports Azure OpenAI Services. To use Azure OpenAI, initialize your client with Azure configuration and use it with `track_openai` just like the standard OpenAI client:
+
+```python
+from opik.integrations.openai import track_openai
+from openai import AzureOpenAI
+
+# gets the API Key from environment variable AZURE_OPENAI_API_KEY
+azure_client = AzureOpenAI(
+ # https://learn.microsoft.com/azure/ai-services/openai/reference#rest-api-versioning
+ api_version="2023-07-01-preview",
+ # https://learn.microsoft.com/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal#create-a-resource
+ azure_endpoint="https://example-endpoint.openai.azure.com",
+)
+azure_client = track_openai(azure_client)
+
+completion = azure_client.chat.completions.create(
+ model="deployment-name", # e.g. gpt-35-instant
+ messages=[
+ {
+ "role": "user",
+ "content": "How do I output all files in a directory using Python?",
+ },
+ ],
+)
+```
## Supported OpenAI methods
diff --git a/apps/opik-documentation/documentation/fern/docs/tracing/integrations/openrouter.mdx b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/openrouter.mdx
new file mode 100644
index 0000000000..6e3c3258e0
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/openrouter.mdx
@@ -0,0 +1,73 @@
+---
+subtitle: Describes how to track OpenRouter LLM calls using Opik
+---
+
+This guide explains how to integrate Opik with OpenRouter using the OpenAI SDK. OpenRouter provides a unified API for accessing hundreds of AI models through a single OpenAI-compatible interface.
+
+## Getting started
+
+First, ensure you have both `opik` and `openai` packages installed:
+
+```bash
+pip install opik openai
+```
+
+You'll also need an OpenRouter API key which you can get from [OpenRouter](https://openrouter.ai/).
+
+## Tracking OpenRouter API calls
+
+```python
+from opik.integrations.openai import track_openai
+from openai import OpenAI
+
+# Initialize the OpenAI client with OpenRouter base URL
+client = OpenAI(
+ base_url="https://openrouter.ai/api/v1",
+ api_key="YOUR_OPENROUTER_API_KEY"
+)
+client = track_openai(client)
+
+# Optional headers for OpenRouter leaderboard
+headers = {
+ "HTTP-Referer": "YOUR_SITE_URL", # Optional. Site URL for rankings
+ "X-Title": "YOUR_SITE_NAME" # Optional. Site title for rankings
+}
+
+response = client.chat.completions.create(
+ model="openai/gpt-4", # You can use any model available on OpenRouter
+ extra_headers=headers,
+ messages=[
+ {"role": "user", "content": "Hello, world!"}
+ ],
+ temperature=0.7,
+ max_tokens=100
+)
+
+print(response.choices[0].message.content)
+```
+
+## Available Models
+
+OpenRouter provides access to a wide variety of models, including many open source models from different providers.
+
+- [OpenAI models](https://openrouter.ai/openai) (GPT-4o, o1, o3-mini)
+- [Anthropic models](https://openrouter.ai/anthropic) (Opus, Sonnet, Haiku)
+- [Google models](https://openrouter.ai/google) (Gemini Pro, Flash, Flash Thinking)
+- And many open source models
+
+You can find the complete list of available models in the [OpenRouter documentation](https://openrouter.ai/docs/overview/models).
+
+## Supported Methods
+
+OpenRouter supports the following methods:
+
+### Chat Completions
+- `client.chat.completions.create()`: Works with all models
+- Provides standard chat completion functionality
+- Compatible with the OpenAI SDK interface
+
+### Structured Outputs
+- `client.beta.chat.completions.parse()`: Only compatible with OpenAI models
+- For non-OpenAI models, see OpenRouter's [Structured Outputs documentation](https://openrouter.ai/docs/features/structured-outputs)
+
+For detailed information about available methods, parameters, and best practices, refer to the [OpenRouter API documentation](https://openrouter.ai/docs).
\ No newline at end of file
diff --git a/apps/opik-documentation/documentation/fern/docs/tracing/integrations/overview.mdx b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/overview.mdx
new file mode 100644
index 0000000000..e1013e12bf
--- /dev/null
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/overview.mdx
@@ -0,0 +1,31 @@
+---
+subtitle: >-
+ Describes all the integrations provided by Opik and what each framework can be
+ used for
+---
+
+Opik aims to make it as easy as possible to log, view and evaluate your LLM traces. We do this by providing a set of integrations:
+
+| Integration | Description | Documentation | Try in Colab |
+| ----------- | ---------------------------------------------------------------------------- | ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| OpenAI | Log traces for all OpenAI LLM calls | [Documentation](/tracing/integrations/openai) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/openai.ipynb) |
+| OpenRouter | Log traces for all OpenRouter LLM calls using OpenAI SDK | [Documentation](/tracing/integrations/openrouter) | |
+| LiteLLM | Call any LLM model using the OpenAI format | [Documentation](/tracing/integrations/litellm) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/litellm.ipynb) |
+| LangChain | Log traces for all LangChain LLM calls | [Documentation](/tracing/integrations/langchain) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/langchain.ipynb) |
+| Haystack | Log traces for all Haystack pipelines | [Documentation](/tracing/integrations/haystack) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/haystack.ipynb) |
+| aisuite | Log traces for all aisuite LLM calls | [Documentation](/tracing/integrations/aisuite) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/aisuite.ipynb) |
+| Anthropic | Log traces for all Anthropic LLM calls | [Documentation](/tracing/integrations/anthropic) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/anthropic.ipynb) |
+| Bedrock | Log traces for all AWS Bedrock LLM calls | [Documentation](/tracing/integrations/bedrock) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/bedrock.ipynb) |
+| CrewAI | Log traces for all CrewAI LLM calls | [Documentation](/tracing/integrations/crewai) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/crewai.ipynb) |
+| DeepSeek | Log traces for all LLM calls made with DeepSeek | [Documentation](/tracing/integrations/deepseek) | |
+| Dify | Log traces and LLM calls for your Dify Apps | [Documentation](/tracing/integrations/dify) | |
+| DSPy | Log traces for all DSPy runs | [Documentation](/tracing/integrations/dspy) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/dspy.ipynb) |
+| Guardrails | Log traces for all Guardrails validations | [Documentation](/tracing/integrations/guardrails-ai) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/guardrails-ai.ipynb) |
+| LangGraph | Log traces for all LangGraph executions | [Documentation](/tracing/integrations/langgraph) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/langgraph.ipynb) |
+| LlamaIndex | Log traces for all LlamaIndex LLM calls | [Documentation](/tracing/integrations/llama_index) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/llama-index.ipynb) |
+| Ollama | Log traces for all Ollama LLM calls | [Documentation](/tracing/integrations/ollama) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/ollama.ipynb) |
+| Predibase | Fine-tune and serve open-source LLMs | [Documentation](/tracing/integrations/predibase) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/predibase.ipynb) |
+| Ragas | Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines | [Documentation](/tracing/integrations/ragas) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/ragas.ipynb) |
+| watsonx | Log traces for all watsonx LLM calls | [Documentation](/tracing/integrations/watsonx) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/watsonx.ipynb) |
+
+If you would like to see more integrations, please open an issue on our [GitHub repository](https://github.com/comet-ml/opik/issues/new/choose).
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/predibase.md b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/predibase.mdx
similarity index 73%
rename from apps/opik-documentation/documentation/docs/tracing/integrations/predibase.md
rename to apps/opik-documentation/documentation/fern/docs/tracing/integrations/predibase.mdx
index 83e877d131..1893e5a6fd 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/predibase.md
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/predibase.mdx
@@ -1,17 +1,23 @@
---
-sidebar_label: Predibase
-description: Describes how to track Predibase LLM calls using Opik
-pytest_codeblocks_skip: true
+subtitle: Describes how to track Predibase LLM calls using Opik
---
-# Using Opik with Predibase
-
Predibase is a platform for fine-tuning and serving open-source Large Language Models (LLMs). It's built on top of open-source [LoRAX](https://loraexchange.ai/).
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
- You can check out the Colab Notebook if you'd like to jump straight to the code:
-
-
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
@@ -65,13 +71,16 @@ response = model.invoke(
print(response)
```
-:::tip
-You can learn more about the Opik integration with LangChain in our [LangChain integration guide](/docs/tracing/integrations/langchain.md) or in the [Predibase cookbook](/docs/cookbook/predibase.md).
-:::
+
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
+
## Tracking your fine-tuning training runs
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/ragas.md b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/ragas.mdx
similarity index 91%
rename from apps/opik-documentation/documentation/docs/tracing/integrations/ragas.md
rename to apps/opik-documentation/documentation/fern/docs/tracing/integrations/ragas.mdx
index 36702901cd..bfc45dccc5 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/ragas.md
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/integrations/ragas.mdx
@@ -1,11 +1,7 @@
---
-sidebar_label: Ragas
-description: Describes how to log Ragas scores to the Opik platform
-pytest_codeblocks_skip: true
+subtitle: Describes how to log Ragas scores to the Opik platform
---
-# Ragas
-
The Opik SDK provides a simple way to integrate with Ragas, a framework for evaluating RAG systems.
There are two main ways to use Ragas with Opik:
@@ -14,9 +10,19 @@ There are two main ways to use Ragas with Opik:
2. Using Ragas to evaluate a RAG pipeline.
- You can check out the Colab Notebook if you'd like to jump straight to the code:
-
-
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
@@ -127,7 +133,9 @@ print(rag_pipeline("What is the capital of France?"))
In the Opik UI, you will be able to see the full trace including the score calculation:
-
+
+
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
+
## Using Ragas metrics to evaluate a RAG pipeline
@@ -191,7 +199,7 @@ class AnswerRelevancyWrapper(base_metric.BaseMetric):
answer_relevancy = AnswerRelevancyWrapper(ragas_answer_relevancy)
```
-:::tip
+
- You can check out the Colab Notebook if you'd like to jump straight to the code:
-
-
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
@@ -19,7 +25,7 @@ pytest_codeblocks_skip: true
### Configuring Opik
-To start tracking your watsonx LLM calls, you can use our [LiteLLM integration](/tracing/integrations/litellm.md). You'll need to have both the `opik` and `litellm` packages installed. You can install them using pip:
+To start tracking your watsonx LLM calls, you can use our [LiteLLM integration](/tracing/integrations/litellm). You'll need to have both the `opik` and `litellm` packages installed. You can install them using pip:
```bash
pip install opik litellm
@@ -31,11 +37,11 @@ In addition, you can configure Opik using the `opik configure` command which wil
opik configure
```
-:::info
+
+
+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+
+
+
## Logging LLM calls within a tracked function
-If you are using LiteLLM within a function tracked with the [`@track`](/tracing/log_traces.mdx#using-function-decorators) decorator, you will need to pass the `current_span_data` as metadata to the `litellm.completion` call:
+If you are using LiteLLM within a function tracked with the [`@track`](/tracing/log_traces#using-function-decorators) decorator, you will need to pass the `current_span_data` as metadata to the `litellm.completion` call:
```python
@track
@@ -124,4 +132,6 @@ def generate_opik_story():
generate_opik_story()
```
-
+
+ 
+
diff --git a/apps/opik-documentation/documentation/docs/tracing/log_agents.mdx b/apps/opik-documentation/documentation/fern/docs/tracing/log_agents.mdx
similarity index 65%
rename from apps/opik-documentation/documentation/docs/tracing/log_agents.mdx
rename to apps/opik-documentation/documentation/fern/docs/tracing/log_agents.mdx
index 37aad33cb0..59df09f48a 100644
--- a/apps/opik-documentation/documentation/docs/tracing/log_agents.mdx
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/log_agents.mdx
@@ -1,34 +1,27 @@
---
-sidebar_label: Track Agents
-description: Describes how to track agents using Opik
-toc_min_heading_level: 2
-toc_max_heading_level: 4
-pytest_codeblocks_skip: true
+subtitle: Describes how to track agents using Opik
---
-import Tabs from "@theme/Tabs";
-import TabItem from "@theme/TabItem";
-
-# Track Agents
-
When working with agents, it can become challenging to track the flow of the agent and its interactions with the environment.
Opik provides a way to track both the agent definition and it's flow.
-Opik includes an integration with many popular Agent frameworks ([LangGrah](/tracing/integrations/langgraph.md), [LLamaIndex](/tracing/integrations/llama_index.md))
+Opik includes an integration with many popular Agent frameworks ([LangGrah](/tracing/integrations/langgraph), [LLamaIndex](/tracing/integrations/llama_index))
and can also be used to log agents manually using the `@track` decorator.
-:::tip
-We are working on improving Opik's support for agent workflows, if you have any ideas or suggestions for the roadmap, you can create a [new Feature Request issue](https://github.com/comet-ml/opik/issues/new/choose)
-in the Opik Github repo or book a call with the Opik team: [Talk to the Opik team](https://calendly.com/jacques-comet/opik-agent-support).
-:::
+
+
## Track the agent definition
-If you are using out [LangGraph](/tracing/integrations/langgraph.md) integration, you can also track the agent definition by passing in the `graph` argument to the `OpikTracer` callback:
+If you are using out [LangGraph](/tracing/integrations/langgraph) integration, you can also track the agent definition by passing in the `graph` argument to the `OpikTracer` callback:
```python
from opik.integrations.langchain import OpikTracer
@@ -139,4 +134,6 @@ opik_tracer = OpikTracer(graph=app.get_graph(xray=True))
This allows you to view the agent definition in the Opik dashboard:
-
+
+ 
+
diff --git a/apps/opik-documentation/documentation/docs/tracing/log_distributed_traces.md b/apps/opik-documentation/documentation/fern/docs/tracing/log_distributed_traces.mdx
similarity index 82%
rename from apps/opik-documentation/documentation/docs/tracing/log_distributed_traces.md
rename to apps/opik-documentation/documentation/fern/docs/tracing/log_distributed_traces.mdx
index 93128a3067..5c28de0d6a 100644
--- a/apps/opik-documentation/documentation/docs/tracing/log_distributed_traces.md
+++ b/apps/opik-documentation/documentation/fern/docs/tracing/log_distributed_traces.mdx
@@ -1,11 +1,7 @@
---
-sidebar_label: Log Distributed Traces
-description: Describes how to log distributed traces to the Opik platform
-pytest_codeblocks_skip: true
+subtitle: Describes how to log distributed traces to the Opik platform
---
-# Log Distributed Traces
-
When working with complex LLM applications, it is common to need to track a traces across multiple services. Opik supports distributed tracing out of the box when integrating using function decorators using a mechanism that is similar to how OpenTelemetry implements distributed tracing.
For the purposes of this guide, we will assume that you have a simple LLM application that is made up of two services: a client and a server. We will assume that the client will create the trace and span, while the server will add a nested span. In order to do this, the `trace_id` and `span_id` will be passed in the headers of the request from the client to the server.
@@ -47,6 +43,7 @@ def generate_llm_response(request: Request) -> str:
return my_llm_application(opik_distributed_trace_headers=request.headers)
```
-:::note
-The `opik_distributed_trace_headers` parameter is added by the `track` decorato to each function that is decorated and is a dictionary with the keys `opik_trace_id` and `opik_parent_span_id`.
-:::
+
+
## Log a trace with an image using OpenAI SDK
@@ -56,6 +53,7 @@ If you are not using the OpenAI SDK, you can still log images to the platform. T
}
```
-:::tip
-Let's us know on [Github](https://github.com/comet-ml/opik/issues/new/choose) if you would like to us to support additional image formats or models.
-:::
+
+
Opik provides different ways to log your LLM calls and traces to the platform:
-1. **Using one of our [integrations](/tracing/integrations/overview.md):** This is the easiest way to get started.
+1. **Using one of our [integrations](/tracing/integrations/overview):** This is the easiest way to get started.
2. **Using the `@track` decorator:** This allows you to track not just LLM calls but any function call in your application, it is often used in conjunction with the integrations.
3. **Using the Python SDK:** This allows for the most flexibility and customizability and is recommended if you want to have full control over the logging process.
4. **Using the Opik REST API:** If you are not using Python, you can use the REST API to log traces to the platform. The REST API is currently in beta and subject to change.
@@ -33,7 +30,7 @@ Opik provides different ways to log your LLM calls and traces to the platform:
In order to use the Opik Python SDK, you will need to install it and configure it:
docs directory.
-
- ),
- },
- {
- title: "Powered by React",
- Svg: require("@site/static/img/undraw_docusaurus_react.svg").default,
- description: (
- <>
- Extend or customize your website layout by reusing React. Docusaurus can be extended while reusing the same
- header and footer.
-
- ),
- },
-];
-
-function Feature({ title, Svg, description }: FeatureItem) {
- return (
-
-
- );
-}
-
-export default function HomepageFeatures(): JSX.Element {
- return (
-
-
-
-
- {title}
-
- {description}
-
-
-
- {FeatureList.map((props, idx) => (
-
-
-
- );
- }
-}
-
-export default function ContentWrapper(props: Props): JSX.Element {
- return (
- <>
-
-
-
-
-
-
- This is a jupyter notebook
-
-
-
-
-
-
- );
-}
-
-export default function NavbarContent() {
- const mobileSidebar = useNavbarMobileSidebar();
- const items = useNavbarItems();
- let [leftItems, rightItems] = splitNavbarItems(items);
-
- return (
- {left}
- {right}
-
{groupOptions.map((group) => (
-
+
= ({
chartData,
className,
}) => {
- const [hiddenLines, setHiddenLines] = useState([]);
+ const [activeLine, setActiveLine] = useState(null);
const config = useMemo(() => {
return getDefaultHashedColorsChartConfig(chartData.lines);
@@ -105,12 +111,17 @@ const ExperimentChartContainer: React.FC = ({
[],
);
+ const isSinglePoint = chartData.data.length === 1;
+
return (
-
-
- {name}
+
+
+ {name}
+
+ Average scores
+
-
+
{noData ? (
= ({
domain={domain}
/>
= ({
align="right"
content={
= ({
height={128}
/>
{chartData.lines.map((line) => {
- const hide = hiddenLines.includes(line);
+ const isActive = line === activeLine;
+
+ let strokeOpacity = 1;
+
+ if (activeLine) {
+ strokeOpacity = isActive ? 1 : 0.4;
+ }
return (
record.scores[line]}
name={config[line].label as string}
stroke={config[line].color as string}
- dot={{ strokeWidth: 1, r: 1 }}
- activeDot={{ strokeWidth: 1, r: 3 }}
- hide={hide}
+ dot={
+ isSinglePoint
+ ? { fill: config[line].color, strokeWidth: 0 }
+ : false
+ }
+ activeDot={{ strokeWidth: 1.5, r: 4, stroke: "white" }}
+ strokeWidth={1.5}
+ strokeOpacity={strokeOpacity}
/>
);
})}
diff --git a/apps/opik-frontend/src/components/pages/ExperimentsPage/charts/ExperimentChartLegendContent.tsx b/apps/opik-frontend/src/components/pages/ExperimentsPage/charts/ExperimentChartLegendContent.tsx
index dce117546b..132e920474 100644
--- a/apps/opik-frontend/src/components/pages/ExperimentsPage/charts/ExperimentChartLegendContent.tsx
+++ b/apps/opik-frontend/src/components/pages/ExperimentsPage/charts/ExperimentChartLegendContent.tsx
@@ -8,10 +8,18 @@ const ExperimentChartLegendContent = React.forwardRef<
HTMLDivElement,
React.ComponentProps &
React.ComponentProps<"div"> & {
- setHideState: OnChangeFn;
+ setActiveLine: OnChangeFn;
chartId: string;
}
->(({ payload, color, setHideState }, ref) => {
+>(({ payload, color, setActiveLine }, ref) => {
+ const handleMouseEnter = (id: string) => {
+ setActiveLine(id);
+ };
+
+ const handleMouseLeave = () => {
+ setActiveLine(null);
+ };
+
if (!payload?.length) {
return null;
}
@@ -19,7 +27,8 @@ const ExperimentChartLegendContent = React.forwardRef<
return (
setOpenProviderMenu(group.label)}
+ onMouseEnter={() => setOpenProviderMenu(group.provider)}
onMouseLeave={() => setOpenProviderMenu(null)}
className={cn(
"comet-body-s flex h-10 w-full items-center rounded-sm p-0 pl-2 hover:bg-primary-foreground justify-center",
{
- "bg-primary-foreground": group.label === openProviderMenu,
+ "bg-primary-foreground":
+ group.provider === openProviderMenu,
},
)}
>
@@ -215,7 +218,7 @@ const PromptModelSelect = ({
align="start"
className="max-h-[400px] overflow-y-auto p-1"
sideOffset={-5}
- onMouseEnter={() => setOpenProviderMenu(group.label)}
+ onMouseEnter={() => setOpenProviderMenu(group.provider)}
hideWhenDetached
>
{group.options.map((option) => {
@@ -308,6 +311,7 @@ const PromptModelSelect = ({
-
diff --git a/apps/opik-frontend/src/components/pages/ConfigurationPage/AIProvidersTab/AIProvidersRowActionsCell.tsx b/apps/opik-frontend/src/components/pages/ConfigurationPage/AIProvidersTab/AIProvidersRowActionsCell.tsx
index c05b1fefa8..c16d80cbe5 100644
--- a/apps/opik-frontend/src/components/pages/ConfigurationPage/AIProvidersTab/AIProvidersRowActionsCell.tsx
+++ b/apps/opik-frontend/src/components/pages/ConfigurationPage/AIProvidersTab/AIProvidersRowActionsCell.tsx
@@ -12,9 +12,10 @@ import { Button } from "@/components/ui/button";
import ConfirmDialog from "@/components/shared/ConfirmDialog/ConfirmDialog";
import CellWrapper from "@/components/shared/DataTableCells/CellWrapper";
import AddEditAIProviderDialog from "@/components/shared/AddEditAIProviderDialog/AddEditAIProviderDialog";
-import { ProviderKey } from "@/types/providers";
+import { PROVIDER_LOCATION_TYPE, ProviderKey } from "@/types/providers";
import useProviderKeysDeleteMutation from "@/api/provider-keys/useProviderKeysDeleteMutation";
import { PROVIDERS } from "@/constants/providers";
+import useLocalAIProviderData from "@/hooks/useLocalAIProviderData";
const AIProvidersRowActionsCell: React.FunctionComponent<
CellContext
@@ -23,13 +24,25 @@ const AIProvidersRowActionsCell: React.FunctionComponent<
const providerKey = context.row.original;
const [open, setOpen] = useState(false);
+ const { deleteLocalAIProviderData } = useLocalAIProviderData();
+
const { mutate: deleteProviderKey } = useProviderKeysDeleteMutation();
const deleteProviderKeyHandler = useCallback(() => {
- deleteProviderKey({
- providerId: providerKey.id,
- });
- }, [providerKey.id, deleteProviderKey]);
+ const config = PROVIDERS[providerKey.provider];
+ if (config.locationType === PROVIDER_LOCATION_TYPE.local) {
+ deleteLocalAIProviderData(providerKey.provider);
+ } else {
+ deleteProviderKey({
+ providerId: providerKey.id,
+ });
+ }
+ }, [
+ providerKey.provider,
+ providerKey.id,
+ deleteLocalAIProviderData,
+ deleteProviderKey,
+ ]);
return (
{
const providerKeys = useMemo(() => data?.content ?? [], [data?.content]);
+ const configuredProviderKeys = useMemo(
+ () => providerKeys.map((p) => p.provider),
+ [providerKeys],
+ );
+
const filteredProviderKeys = useMemo(() => {
if (providerKeys?.length === 0 || search === "") {
return providerKeys;
@@ -140,6 +145,7 @@ const AIProvidersTab = () => {
/>
{payload.map((item) => {
const key = `${item.value || "value"}`;
@@ -29,24 +38,17 @@ const ExperimentChartLegendContent = React.forwardRef<
{
- setHideState((state) => {
- return item.inactive
- ? state.filter((value) => value !== item.value)
- : [...state, item.value];
- });
- }}
+ onMouseEnter={() => handleMouseEnter(item.value)}
>
-
+
{
/>
;
customYTickFormatter?: (value: number, maxDecimalLength?: number) => string;
+ chartId: string;
}
type TransformedDataValueType = null | number | string;
type TransformedData = { [key: string]: TransformedDataValueType };
+const predefinedColorMap = {
+ traces: TAG_VARIANTS_COLOR_MAP.purple,
+ cost: TAG_VARIANTS_COLOR_MAP.purple,
+ "duration.p50": TAG_VARIANTS_COLOR_MAP.turquoise,
+ "duration.p90": TAG_VARIANTS_COLOR_MAP.burgundy,
+ "duration.p99": TAG_VARIANTS_COLOR_MAP.purple,
+ completion_tokens: TAG_VARIANTS_COLOR_MAP.turquoise,
+ prompt_tokens: TAG_VARIANTS_COLOR_MAP.burgundy,
+ total_tokens: TAG_VARIANTS_COLOR_MAP.purple,
+};
+
const MetricChart = ({
name,
+ description,
metricName,
projectId,
interval,
@@ -56,7 +83,9 @@ const MetricChart = ({
renderValue = renderTooltipValue,
labelsMap,
customYTickFormatter,
+ chartId,
}: MetricChartProps) => {
+ const [activeLine, setActiveLine] = useState(null);
const { data: traces, isPending } = useProjectMetric(
{
projectId,
@@ -72,7 +101,7 @@ const MetricChart = ({
);
const [data, lines, values] = useMemo(() => {
- if (!traces?.length) {
+ if (!traces?.filter((trace) => !!trace.name).length) {
return [[], [], []];
}
@@ -96,7 +125,11 @@ const MetricChart = ({
}, [traces]);
const config = useMemo(() => {
- return getDefaultHashedColorsChartConfig(lines, labelsMap);
+ return getDefaultHashedColorsChartConfig(
+ lines,
+ labelsMap,
+ predefinedColorMap,
+ );
}, [lines, labelsMap]);
const {
@@ -132,6 +165,15 @@ const MetricChart = ({
);
const renderContent = () => {
+ const isSingleLine = lines.length === 1;
+ const isSinglePoint =
+ data.filter((point) => lines.every((line) => point[line] !== null))
+ .length === 1;
+
+ const [firstLine] = lines;
+
+ const activeDot = { strokeWidth: 1.5, r: 4, stroke: "white" };
+
if (isPending) {
return (
{item.value}
= ({
? "w-full"
: chartsData.length === 2
? "basis-1/2"
- : "basis-2/5";
+ : "basis-[520px]";
return (
(false);
-
- const [search, setSearch] = useState("");
- const [page, setPage] = useState(1);
- const [size, setSize] = useState(10);
-
- const [rowSelection, setRowSelection] = useState({});
+ const [search = "", setSearch] = useQueryParam("search", StringParam, {
+ updateType: "replaceIn",
+ });
+ const [page = 1, setPage] = useQueryParam("page", NumberParam, {
+ updateType: "replaceIn",
+ });
+ const [size = 10, setSize] = useQueryParam("size", NumberParam, {
+ updateType: "replaceIn",
+ });
+ const [rowSelection = {}, setRowSelection] = useQueryParam(
+ "selection",
+ JsonParam,
+ {
+ updateType: "replaceIn",
+ },
+ );
const [sortedColumns, setSortedColumns] = useLocalStorageState(
COLUMNS_SORT_KEY,
@@ -192,7 +204,7 @@ const ProjectsPage: React.FunctionComponent = () => {
const { data, isPending } = useProjectWithStatisticsList(
{
workspaceName,
- search,
+ search: search!,
sorting: sortedColumns.map((column) => {
if (column.id === "last_updated_at") {
return {
@@ -202,8 +214,8 @@ const ProjectsPage: React.FunctionComponent = () => {
}
return column;
}),
- page,
- size,
+ page: page!,
+ size: size!,
},
{
placeholderData: keepPreviousData,
@@ -292,7 +304,7 @@ const ProjectsPage: React.FunctionComponent = () => {
0 && "mb-6",
+ "flex items-center gap-4 overflow-y-auto",
+ chartsData.length > 0 && "mb-4",
)}
>
{chartsData.map((data) => (
diff --git a/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundOutputs/PlaygroundOutput.tsx b/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundOutputs/PlaygroundOutput.tsx
index 36a67e8301..e1ed354522 100644
--- a/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundOutputs/PlaygroundOutput.tsx
+++ b/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundOutputs/PlaygroundOutput.tsx
@@ -1,7 +1,7 @@
import React from "react";
import ReactMarkdown from "react-markdown";
-import { getAlphabetLetter } from "@/lib/utils";
+import { cn, getAlphabetLetter } from "@/lib/utils";
import PlaygroundOutputLoader from "@/components/pages/PlaygroundPage/PlaygroundOutputs/PlaygroundOutputLoader/PlaygroundOutputLoader";
import {
useOutputLoadingByPromptDatasetItemId,
@@ -25,7 +25,11 @@ const PlaygroundOutput = ({ promptId, index }: PlaygroundOutputProps) => {
}
return (
-
+
{value}
);
diff --git a/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundOutputs/PlaygroundOutputActions/usePromptDatasetItemCombination.ts b/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundOutputs/PlaygroundOutputActions/usePromptDatasetItemCombination.ts
index 75197e8bca..00681d2ca7 100644
--- a/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundOutputs/PlaygroundOutputActions/usePromptDatasetItemCombination.ts
+++ b/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundOutputs/PlaygroundOutputActions/usePromptDatasetItemCombination.ts
@@ -16,6 +16,7 @@ import mustache from "mustache";
import cloneDeep from "lodash/cloneDeep";
import set from "lodash/set";
import isObject from "lodash/isObject";
+import useLocalAIProviderData from "@/hooks/useLocalAIProviderData";
export interface DatasetItemPromptCombination {
datasetItem?: DatasetItem;
@@ -85,6 +86,8 @@ const usePromptDatasetItemCombination = ({
// the prop is just taken as the value on the moment of creation
const isToStopRef = useRef(isToStop);
+ const { getLocalIAProviderURL } = useLocalAIProviderData();
+
const runStreaming = useCompletionProxyStreaming({
workspaceName,
});
@@ -138,6 +141,7 @@ const usePromptDatasetItemCombination = ({
);
const run = await runStreaming({
+ url: getLocalIAProviderURL(prompt.provider),
model: prompt.model,
messages: providerMessages,
configs: prompt.configs,
@@ -149,7 +153,8 @@ const usePromptDatasetItemCombination = ({
},
});
- const error = run.opikError || run.providerError;
+ const error =
+ run.opikError || run.providerError || run.pythonProxyError;
updateOutput(prompt.id, datasetItemId, {
isLoading: false,
@@ -160,7 +165,7 @@ const usePromptDatasetItemCombination = ({
providerMessages,
configs: prompt.configs,
model: prompt.model,
-
+ provider: prompt.provider,
promptId: prompt.id,
datasetName,
datasetItemId: datasetItemId,
@@ -182,10 +187,11 @@ const usePromptDatasetItemCombination = ({
},
[
- datasetName,
- runStreaming,
- updateOutput,
addAbortController,
+ updateOutput,
+ runStreaming,
+ getLocalIAProviderURL,
+ datasetName,
deleteAbortController,
],
);
diff --git a/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundOutputs/PlaygroundOutputTable/PlaygroundOutputCell.tsx b/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundOutputs/PlaygroundOutputTable/PlaygroundOutputCell.tsx
index 626b16a7d3..17a48cbe09 100644
--- a/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundOutputs/PlaygroundOutputTable/PlaygroundOutputCell.tsx
+++ b/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundOutputs/PlaygroundOutputTable/PlaygroundOutputCell.tsx
@@ -9,6 +9,7 @@ import {
} from "@/store/PlaygroundStore";
import ReactMarkdown from "react-markdown";
import PlaygroundOutputLoader from "@/components/pages/PlaygroundPage/PlaygroundOutputs/PlaygroundOutputLoader/PlaygroundOutputLoader";
+import { cn } from "@/lib/utils";
interface PlaygroundOutputCellData {
dataItemId: string;
@@ -46,7 +47,11 @@ const PlaygroundOutputCell: React.FunctionComponent<
}
return (
-
+
{value}
);
diff --git a/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundPrompts/PlaygroundPrompt.tsx b/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundPrompts/PlaygroundPrompt.tsx
index 8b6bbfc8d6..523add9792 100644
--- a/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundPrompts/PlaygroundPrompt.tsx
+++ b/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundPrompts/PlaygroundPrompt.tsx
@@ -15,11 +15,7 @@ import {
generateDefaultPrompt,
getDefaultConfigByProvider,
} from "@/lib/playground";
-import {
- generateDefaultLLMPromptMessage,
- getModelProvider,
- getNextMessageType,
-} from "@/lib/llm";
+import { generateDefaultLLMPromptMessage, getNextMessageType } from "@/lib/llm";
import LLMPromptMessages from "@/components/pages-shared/llm/LLMPromptMessages/LLMPromptMessages";
import PromptModelSelect from "@/components/pages-shared/llm/PromptModelSelect/PromptModelSelect";
import { getAlphabetLetter } from "@/lib/utils";
@@ -33,9 +29,11 @@ import {
useUpdateOutput,
useUpdatePrompt,
} from "@/store/PlaygroundStore";
-import { getDefaultProviderKey } from "@/lib/provider";
-import { PROVIDERS } from "@/constants/providers";
import useLastPickedModel from "@/components/pages/PlaygroundPage/PlaygroundPrompts/useLastPickedModel";
+import {
+ ModelResolver,
+ ProviderResolver,
+} from "@/hooks/useLLMProviderModelsData";
interface PlaygroundPromptProps {
workspaceName: string;
@@ -43,6 +41,8 @@ interface PlaygroundPromptProps {
promptId: string;
providerKeys: PROVIDER_TYPE[];
isPendingProviderKeys: boolean;
+ providerResolver: ProviderResolver;
+ modelResolver: ModelResolver;
}
const PlaygroundPrompt = ({
@@ -51,6 +51,8 @@ const PlaygroundPrompt = ({
index,
providerKeys,
isPendingProviderKeys,
+ providerResolver,
+ modelResolver,
}: PlaygroundPromptProps) => {
const checkedIfModelIsValidRef = useRef(false);
@@ -66,7 +68,7 @@ const PlaygroundPrompt = ({
const deletePrompt = useDeletePrompt();
const updateOutput = useUpdateOutput();
- const provider = model ? getModelProvider(model) : "";
+ const provider = providerResolver(model);
const hintMessage = datasetVariables?.length
? `Reference dataset variables using mustache syntax: ${datasetVariables
@@ -91,6 +93,8 @@ const PlaygroundPrompt = ({
const newPrompt = generateDefaultPrompt({
initPrompt: prompt,
setupProviders: providerKeys,
+ providerResolver: providerResolver,
+ modelResolver: modelResolver,
});
addPrompt(newPrompt, index + 1);
@@ -116,32 +120,40 @@ const PlaygroundPrompt = ({
);
const handleUpdateModel = useCallback(
- (model: PROVIDER_MODEL_TYPE) => {
- updatePrompt(promptId, { model });
- setLastPickedModel(model);
+ (newModel: PROVIDER_MODEL_TYPE, newProvider: PROVIDER_TYPE) => {
+ updatePrompt(promptId, {
+ model: newModel,
+ provider: newProvider,
+ ...(newProvider !== provider && {
+ configs: getDefaultConfigByProvider(newProvider),
+ }),
+ });
+ setLastPickedModel(newModel);
},
- [updatePrompt, promptId, setLastPickedModel],
+ [updatePrompt, promptId, provider, setLastPickedModel],
);
const handleAddProvider = useCallback(
(provider: PROVIDER_TYPE) => {
- const modelProvider = model ? getModelProvider(model) : "";
- const noCurrentModel = !modelProvider;
-
- if (noCurrentModel) {
- const newModel = PROVIDERS[provider]?.defaultModel || "";
-
- const newDefaultConfigs = provider
- ? getDefaultConfigByProvider(provider)
- : {};
+ const newModel = modelResolver(model, providerKeys, provider);
+ if (newModel !== model) {
+ const newProvider = providerResolver(newModel);
updatePrompt(promptId, {
model: newModel,
- configs: newDefaultConfigs,
+ provider: newProvider,
+ configs: getDefaultConfigByProvider(newProvider),
});
}
},
- [model, promptId, updatePrompt],
+ [
+ modelResolver,
+ model,
+ providerKeys,
+ providerResolver,
+ updatePrompt,
+ promptId,
+ ],
);
useEffect(() => {
@@ -149,40 +161,24 @@ const PlaygroundPrompt = ({
if (!checkedIfModelIsValidRef.current && !isPendingProviderKeys) {
checkedIfModelIsValidRef.current = true;
- const modelProvider = model ? getModelProvider(model) : "";
+ const newModel = modelResolver(model, providerKeys);
- const noModelProviderWhenProviderKeysSet =
- !modelProvider && providerKeys.length > 0;
- const modelProviderIsNotFromProviderKeys =
- modelProvider && !providerKeys.includes(modelProvider);
-
- const needToChangeProvider =
- noModelProviderWhenProviderKeysSet ||
- modelProviderIsNotFromProviderKeys;
+ if (newModel !== model) {
+ const newProvider = providerResolver(newModel);
+ updatePrompt(promptId, {
+ model: newModel,
+ provider: newProvider,
+ configs: getDefaultConfigByProvider(newProvider),
+ });
- if (!needToChangeProvider) {
- return;
+ updateOutput(promptId, "", { value: "" });
}
-
- const newProvider = getDefaultProviderKey(providerKeys);
- const newModel = newProvider
- ? PROVIDERS[newProvider]?.defaultModel || ""
- : "";
-
- const newDefaultConfigs = newProvider
- ? getDefaultConfigByProvider(newProvider)
- : {};
-
- updatePrompt(promptId, {
- model: newModel,
- configs: newDefaultConfigs,
- });
-
- updateOutput(promptId, "", { value: "" });
}
}, [
providerKeys,
isPendingProviderKeys,
+ providerResolver,
+ modelResolver,
updateOutput,
updatePrompt,
promptId,
diff --git a/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundPrompts/PlaygroundPrompts.tsx b/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundPrompts/PlaygroundPrompts.tsx
index c14df24854..8d7eae76d2 100644
--- a/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundPrompts/PlaygroundPrompts.tsx
+++ b/apps/opik-frontend/src/components/pages/PlaygroundPage/PlaygroundPrompts/PlaygroundPrompts.tsx
@@ -11,6 +11,7 @@ import {
useSetPromptMap,
} from "@/store/PlaygroundStore";
import useLastPickedModel from "@/components/pages/PlaygroundPage/PlaygroundPrompts/useLastPickedModel";
+import useLLMProviderModelsData from "@/hooks/useLLMProviderModelsData";
interface PlaygroundPromptsState {
workspaceName: string;
@@ -32,11 +33,15 @@ const PlaygroundPrompts = ({
const promptIds = usePromptIds();
const [lastPickedModel] = useLastPickedModel();
+ const { calculateModelProvider, calculateDefaultModel } =
+ useLLMProviderModelsData();
const handleAddPrompt = () => {
const newPrompt = generateDefaultPrompt({
setupProviders: providerKeys,
lastPickedModel,
+ providerResolver: calculateModelProvider,
+ modelResolver: calculateDefaultModel,
});
addPrompt(newPrompt);
};
@@ -45,10 +50,19 @@ const PlaygroundPrompts = ({
const newPrompt = generateDefaultPrompt({
setupProviders: providerKeys,
lastPickedModel,
+ providerResolver: calculateModelProvider,
+ modelResolver: calculateDefaultModel,
});
setPromptMap([newPrompt.id], { [newPrompt.id]: newPrompt });
onResetHeight();
- }, [setPromptMap, providerKeys, lastPickedModel, onResetHeight]);
+ }, [
+ providerKeys,
+ lastPickedModel,
+ calculateModelProvider,
+ calculateDefaultModel,
+ setPromptMap,
+ onResetHeight,
+ ]);
useEffect(() => {
// hasn't been initialized yet or the last prompt is removed
@@ -84,6 +98,8 @@ const PlaygroundPrompts = ({
key={promptId}
providerKeys={providerKeys}
isPendingProviderKeys={isPendingProviderKeys}
+ providerResolver={calculateModelProvider}
+ modelResolver={calculateDefaultModel}
/>
))}
diff --git a/apps/opik-frontend/src/components/pages/ProjectsPage/ProjectsPage.tsx b/apps/opik-frontend/src/components/pages/ProjectsPage/ProjectsPage.tsx
index 137f698962..80ae01ffe5 100644
--- a/apps/opik-frontend/src/components/pages/ProjectsPage/ProjectsPage.tsx
+++ b/apps/opik-frontend/src/components/pages/ProjectsPage/ProjectsPage.tsx
@@ -1,6 +1,13 @@
import React, { useCallback, useMemo, useRef, useState } from "react";
import { keepPreviousData } from "@tanstack/react-query";
+import {
+ JsonParam,
+ NumberParam,
+ StringParam,
+ useQueryParam,
+} from "use-query-params";
import isNumber from "lodash/isNumber";
+import get from "lodash/get";
import { formatNumericData } from "@/lib/utils";
import DataTable from "@/components/shared/DataTable/DataTable";
@@ -30,18 +37,13 @@ import {
} from "@/types/shared";
import { convertColumnDataToColumn, mapColumnDataFields } from "@/lib/table";
import useLocalStorageState from "use-local-storage-state";
-import {
- ColumnPinningState,
- ColumnSort,
- RowSelectionState,
-} from "@tanstack/react-table";
+import { ColumnPinningState, ColumnSort } from "@tanstack/react-table";
import {
generateActionsColumDef,
generateSelectColumDef,
} from "@/components/shared/DataTable/utils";
import { RESOURCE_TYPE } from "@/components/shared/ResourceLink/ResourceLink";
import FeedbackScoreListCell from "@/components/shared/DataTableCells/FeedbackScoreListCell";
-import { get } from "lodash";
export const getRowId = (p: ProjectWithStatistic) => p.id;
@@ -175,12 +177,22 @@ const ProjectsPage: React.FunctionComponent = () => {
const resetDialogKeyRef = useRef(0);
const [openDialog, setOpenDialog] = useState
@@ -145,7 +187,7 @@ const MetricChart = ({
config={config}
className="h-[var(--chart-height)] w-full"
>
-
+
+
+
+
+
+ ) : (
+ lines.map((line) => {
+ const isActive = line === activeLine;
+
+ let strokeOpacity = 1;
+
+ if (activeLine) {
+ strokeOpacity = isActive ? 1 : 0.4;
+ }
+
+ return (
+
-
- {name}
+
+ {name}
+
+ {description}
+
- {renderContent()}
+ {renderContent()}
);
};
diff --git a/apps/opik-frontend/src/components/pages/TracesPage/MetricsTab/MetricChart/MetricChartLegendContent.tsx b/apps/opik-frontend/src/components/pages/TracesPage/MetricsTab/MetricChart/MetricChartLegendContent.tsx
new file mode 100644
index 0000000000..adb12c2e7b
--- /dev/null
+++ b/apps/opik-frontend/src/components/pages/TracesPage/MetricsTab/MetricChart/MetricChartLegendContent.tsx
@@ -0,0 +1,66 @@
+import React from "react";
+import * as RechartsPrimitive from "recharts";
+import { OnChangeFn } from "@/types/shared";
+import TooltipWrapper from "@/components/shared/TooltipWrapper/TooltipWrapper";
+
+const MetricChartLegendContent = React.forwardRef<
+ HTMLDivElement,
+ React.ComponentProps &
+ React.ComponentProps<"div"> & {
+ setActiveLine: OnChangeFn;
+ chartId: string;
+ }
+>(({ payload, color, setActiveLine }, ref) => {
+ const handleMouseEnter = (id: string) => {
+ setActiveLine(id);
+ };
+
+ const handleMouseLeave = () => {
+ setActiveLine(null);
+ };
+
+ if (!payload?.length) {
+ return null;
+ }
+
+ return (
+
{
{
{
{
= ({
value={field.value}
onChange={field.onChange}
id="sampling_rate"
- label="Samping rate"
+ label="Sampling rate"
tooltip="Percentage of traces to evaluate"
/>
)}
diff --git a/apps/opik-frontend/src/components/pages/TracesPage/RulesTab/AddEditRuleDialog/LLMJudgeRuleDetails.tsx b/apps/opik-frontend/src/components/pages/TracesPage/RulesTab/AddEditRuleDialog/LLMJudgeRuleDetails.tsx
index 0d8a7d5536..7c6be7cc0c 100644
--- a/apps/opik-frontend/src/components/pages/TracesPage/RulesTab/AddEditRuleDialog/LLMJudgeRuleDetails.tsx
+++ b/apps/opik-frontend/src/components/pages/TracesPage/RulesTab/AddEditRuleDialog/LLMJudgeRuleDetails.tsx
@@ -25,10 +25,11 @@ import {
LLMMessage,
LLMPromptTemplate,
} from "@/types/llm";
-import { generateDefaultLLMPromptMessage, getModelProvider } from "@/lib/llm";
+import { generateDefaultLLMPromptMessage } from "@/lib/llm";
import { PROVIDER_MODEL_TYPE } from "@/types/providers";
import { safelyGetPromptMustacheTags } from "@/lib/prompt";
import { EvaluationRuleFormType } from "@/components/pages/TracesPage/RulesTab/AddEditRuleDialog/schema";
+import useLLMProviderModelsData from "@/hooks/useLLMProviderModelsData";
const MESSAGE_TYPE_OPTIONS = [
{
@@ -61,6 +62,7 @@ const LLMJudgeRuleDetails: React.FC = ({
form,
}) => {
const cache = useRef>({});
+ const { calculateModelProvider } = useLLMProviderModelsData();
return (
<>
@@ -69,7 +71,7 @@ const LLMJudgeRuleDetails: React.FC = ({
name="llmJudgeDetails.model"
render={({ field, formState }) => {
const model = field.value as PROVIDER_MODEL_TYPE | "";
- const provider = model ? getModelProvider(model) : "";
+ const provider = calculateModelProvider(model);
const validationErrors = get(formState.errors, [
"llmJudgeDetails",
"model",
diff --git a/apps/opik-frontend/src/components/shared/AddEditAIProviderDialog/AddEditAIProviderDialog.tsx b/apps/opik-frontend/src/components/shared/AddEditAIProviderDialog/AddEditAIProviderDialog.tsx
index 47ac3aef21..d4e042f282 100644
--- a/apps/opik-frontend/src/components/shared/AddEditAIProviderDialog/AddEditAIProviderDialog.tsx
+++ b/apps/opik-frontend/src/components/shared/AddEditAIProviderDialog/AddEditAIProviderDialog.tsx
@@ -1,4 +1,12 @@
-import React, { useCallback, useState } from "react";
+import React, { useCallback, useMemo } from "react";
+import { useForm, UseFormReturn } from "react-hook-form";
+import { zodResolver } from "@hookform/resolvers/zod";
+import { z } from "zod";
+import get from "lodash/get";
+import isFunction from "lodash/isFunction";
+import isArray from "lodash/isArray";
+
+import { cn } from "@/lib/utils";
import { Button } from "@/components/ui/button";
import {
Dialog,
@@ -8,22 +16,40 @@ import {
DialogHeader,
DialogTitle,
} from "@/components/ui/dialog";
+import {
+ Form,
+ FormControl,
+ FormField,
+ FormItem,
+ FormMessage,
+} from "@/components/ui/form";
import { Label } from "@/components/ui/label";
-import { ProviderKey, PROVIDER_TYPE } from "@/types/providers";
+import {
+ LocalAIProviderData,
+ PROVIDER_LOCATION_TYPE,
+ PROVIDER_TYPE,
+ ProviderKey,
+} from "@/types/providers";
+import SelectBox from "@/components/shared/SelectBox/SelectBox";
import useProviderKeysUpdateMutation from "@/api/provider-keys/useProviderKeysUpdateMutation";
import useProviderKeysCreateMutation from "@/api/provider-keys/useProviderKeysCreateMutation";
-import SelectBox from "@/components/shared/SelectBox/SelectBox";
+import useLocalAIProviderData from "@/hooks/useLocalAIProviderData";
import { PROVIDERS, PROVIDERS_OPTIONS } from "@/constants/providers";
import { SelectItem } from "@/components/ui/select";
import { DropdownOption } from "@/types/shared";
-import EyeInput from "@/components/shared/EyeInput/EyeInput";
-import isFunction from "lodash/isFunction";
+import {
+ AIProviderFormSchema,
+ AIProviderFormType,
+} from "@/components/shared/AddEditAIProviderDialog/schema";
+import CloudAIProviderDetails from "@/components/shared/AddEditAIProviderDialog/CloudAIProviderDetails";
+import LocalAIProviderDetails from "@/components/shared/AddEditAIProviderDialog/LocalAIProviderDetails";
type AddEditAIProviderDialogProps = {
providerKey?: ProviderKey;
open: boolean;
setOpen: (open: boolean) => void;
onAddProvider?: (provider: PROVIDER_TYPE) => void;
+ excludedProviders?: PROVIDER_TYPE[];
};
const AddEditAIProviderDialog: React.FC = ({
@@ -31,18 +57,40 @@ const AddEditAIProviderDialog: React.FC = ({
open,
setOpen,
onAddProvider,
+ excludedProviders,
}) => {
+ const { getLocalAIProviderData, setLocalAIProviderData } =
+ useLocalAIProviderData();
const { mutate: createMutate } = useProviderKeysCreateMutation();
const { mutate: updateMutate } = useProviderKeysUpdateMutation();
- const [provider, setProvider] = useState(
- providerKey?.provider || "",
- );
- const [apiKey, setApiKey] = useState("");
- const isEdit = Boolean(providerKey);
- const isValid = Boolean(apiKey.length);
+ const localData = useMemo(() => {
+ return providerKey?.provider
+ ? getLocalAIProviderData(providerKey.provider)
+ : undefined;
+ }, [getLocalAIProviderData, providerKey]);
+
+ const form: UseFormReturn = useForm<
+ z.infer
+ >({
+ resolver: zodResolver(AIProviderFormSchema),
+ defaultValues: {
+ provider: providerKey?.provider || "",
+ locationType: providerKey?.provider
+ ? PROVIDERS[providerKey.provider]?.locationType
+ : PROVIDER_LOCATION_TYPE.cloud,
+ apiKey: "",
+ url: localData?.url ?? "",
+ models: localData?.models ?? "",
+ },
+ });
- const providerName = (provider && PROVIDERS[provider]?.label) || "";
+ const provider = form.watch("provider") as PROVIDER_TYPE | "";
+
+ const isEdit = Boolean(providerKey);
+ const isCloudProvider =
+ provider === "" ||
+ PROVIDERS[provider].locationType === PROVIDER_LOCATION_TYPE.cloud;
const title = isEdit
? "Edit AI provider configuration"
@@ -50,9 +98,33 @@ const AddEditAIProviderDialog: React.FC = ({
const buttonText = isEdit ? "Update configuration" : "Save configuration";
- const apiKeyLabel = provider ? `${providerName} API Key` : "API Key";
+ const localConfigHandler = useCallback(() => {
+ const data: LocalAIProviderData = {
+ created_at: new Date().toISOString(),
+ ...localData,
+ url: form.getValues("url"),
+ models: form.getValues("models"),
+ };
+
+ if (provider) {
+ setLocalAIProviderData(provider, data);
+ if (isFunction(onAddProvider)) {
+ onAddProvider(provider);
+ }
+ }
- const submitHandler = useCallback(() => {
+ setOpen(false);
+ }, [
+ localData,
+ form,
+ provider,
+ setOpen,
+ setLocalAIProviderData,
+ onAddProvider,
+ ]);
+
+ const cloudConfigHandler = useCallback(() => {
+ const apiKey = form.getValues("apiKey");
if (isEdit) {
updateMutate({
providerKey: {
@@ -72,21 +144,41 @@ const AddEditAIProviderDialog: React.FC = ({
},
});
}
+ setOpen(false);
}, [
- createMutate,
+ form,
isEdit,
- apiKey,
- updateMutate,
provider,
+ setOpen,
+ updateMutate,
providerKey,
onAddProvider,
+ createMutate,
]);
+ const onSubmit = useCallback(
+ () => (isCloudProvider ? cloudConfigHandler() : localConfigHandler()),
+ [isCloudProvider, cloudConfigHandler, localConfigHandler],
+ );
+
+ const options = useMemo(() => {
+ return isArray(excludedProviders)
+ ? PROVIDERS_OPTIONS.filter(
+ ({ value }) => !excludedProviders.includes(value),
+ )
+ : PROVIDERS_OPTIONS;
+ }, [excludedProviders]);
+
const renderOption = (option: DropdownOption) => {
const Icon = PROVIDERS[option.value as PROVIDER_TYPE]?.icon;
return (
-
+ {option.description} }
+ withoutCheck
+ >
= ({
{title}
-
-
-
-
+
diff --git a/apps/opik-frontend/src/components/shared/AddEditAIProviderDialog/CloudAIProviderDetails.tsx b/apps/opik-frontend/src/components/shared/AddEditAIProviderDialog/CloudAIProviderDetails.tsx
new file mode 100644
index 0000000000..016aeb573c
--- /dev/null
+++ b/apps/opik-frontend/src/components/shared/AddEditAIProviderDialog/CloudAIProviderDetails.tsx
@@ -0,0 +1,78 @@
+import React from "react";
+import { UseFormReturn } from "react-hook-form";
+import { Button } from "@/components/ui/button";
+import { Label } from "@/components/ui/label";
+import { PROVIDER_TYPE } from "@/types/providers";
+import { CLOUD_PROVIDER_OPTION_TYPE, PROVIDERS } from "@/constants/providers";
+import EyeInput from "@/components/shared/EyeInput/EyeInput";
+import { AIProviderFormType } from "@/components/shared/AddEditAIProviderDialog/schema";
+import get from "lodash/get";
+import {
+ FormControl,
+ FormField,
+ FormItem,
+ FormMessage,
+} from "@/components/ui/form";
+import { cn } from "@/lib/utils";
+
+type CloudAIProviderDetailsProps = {
+ provider: PROVIDER_TYPE | "";
+ form: UseFormReturn;
+};
+
+const CloudAIProviderDetails: React.FC = ({
+ provider,
+ form,
+}) => {
+ const providerName = (provider && PROVIDERS[provider]?.label + " ") || "";
+ const apiKeyLabel = `${providerName}API Key`;
+
+ return (
+ ;
+};
+
+const LocalAIProviderDetails: React.FC = ({
+ provider,
+ form,
+}) => {
+ const providerName = (provider && PROVIDERS[provider]?.label + " ") || "";
+ const urlLabel = `${providerName}URL`;
+
+ return (
+ <>
+ ;
diff --git a/apps/opik-frontend/src/components/shared/ChartTooltipContent/ChartTooltipContent.tsx b/apps/opik-frontend/src/components/shared/ChartTooltipContent/ChartTooltipContent.tsx
index ec0b98f893..77a0cf8191 100644
--- a/apps/opik-frontend/src/components/shared/ChartTooltipContent/ChartTooltipContent.tsx
+++ b/apps/opik-frontend/src/components/shared/ChartTooltipContent/ChartTooltipContent.tsx
@@ -84,7 +84,7 @@ const ChartTooltipContent = React.forwardRef<
className="flex h-6 w-full flex-wrap items-center gap-1.5 px-2"
>
{!isUndefined(item.value) && (
-
+
{renderValue(item.value)}
)}
diff --git a/apps/opik-frontend/src/components/ui/select.tsx b/apps/opik-frontend/src/components/ui/select.tsx
index 6d8d40637a..4844b3c89c 100644
--- a/apps/opik-frontend/src/components/ui/select.tsx
+++ b/apps/opik-frontend/src/components/ui/select.tsx
@@ -112,7 +112,7 @@ SelectLabel.displayName = SelectPrimitive.Label.displayName;
interface SelectItemProps
extends React.ComponentPropsWithoutRef {
withoutCheck?: boolean;
- description?: string;
+ description?: React.ReactNode;
}
const SelectItem = React.forwardRef<
@@ -143,7 +143,7 @@ const SelectItem = React.forwardRef<
{children}
{description && (
-
+
+ );
+});
+MetricChartLegendContent.displayName = "MetricChartLegendContent";
+
+export default MetricChartLegendContent;
diff --git a/apps/opik-frontend/src/components/pages/TracesPage/MetricsTab/MetricsTab.tsx b/apps/opik-frontend/src/components/pages/TracesPage/MetricsTab/MetricsTab.tsx
index 1bc0213d2e..1c797a2c48 100644
--- a/apps/opik-frontend/src/components/pages/TracesPage/MetricsTab/MetricsTab.tsx
+++ b/apps/opik-frontend/src/components/pages/TracesPage/MetricsTab/MetricsTab.tsx
@@ -146,7 +146,9 @@ const MetricsTab = ({ projectId }: MetricsTabProps) => {
+ {payload.map((item) => {
+ const key = `${item.value || "value"}`;
+ const indicatorColor = color || item.color;
+
+ return (
+
+ handleMouseEnter(item.value)}
+ >
+
+
+
+
+ );
+ })}
+
+ {item.value}
+
+
-
- setProvider(v as PROVIDER_TYPE)}
- options={PROVIDERS_OPTIONS}
- placeholder="Select a provider"
- />
-
-
-
- setApiKey(e.target.value)}
- />
- {provider && (
-
- Get your {providerName} API key{" "}
-
- .
-
- )}
-
+
+
+ {
+ const validationErrors = get(formState.errors, ["apiKey"]);
+
+ return (
+
+
+
+ field.onChange(e.target.value)}
+ className={cn({
+ "border-destructive": Boolean(validationErrors?.message),
+ })}
+ />
+
+
+ );
+ }}
+ />
+ {provider && (
+
+ Get your {providerName} API key{" "}
+
+ .
+
+ )}
+
+ );
+};
+
+export default CloudAIProviderDetails;
diff --git a/apps/opik-frontend/src/components/shared/AddEditAIProviderDialog/LocalAIProviderDetails.tsx b/apps/opik-frontend/src/components/shared/AddEditAIProviderDialog/LocalAIProviderDetails.tsx
new file mode 100644
index 0000000000..2053afb3de
--- /dev/null
+++ b/apps/opik-frontend/src/components/shared/AddEditAIProviderDialog/LocalAIProviderDetails.tsx
@@ -0,0 +1,113 @@
+import React from "react";
+import { UseFormReturn } from "react-hook-form";
+import { Label } from "@/components/ui/label";
+import { PROVIDER_TYPE } from "@/types/providers";
+import { PROVIDERS } from "@/constants/providers";
+import { AIProviderFormType } from "@/components/shared/AddEditAIProviderDialog/schema";
+import get from "lodash/get";
+import {
+ FormControl,
+ FormField,
+ FormItem,
+ FormMessage,
+} from "@/components/ui/form";
+import { buildDocsUrl, cn } from "@/lib/utils";
+import { Button } from "@/components/ui/button";
+import { Input } from "@/components/ui/input";
+import { Description } from "@/components/ui/description";
+
+type LocalAIProviderDetailsProps = {
+ provider: PROVIDER_TYPE | "";
+ form: UseFormReturn
+ {
+ const validationErrors = get(formState.errors, ["url"]);
+
+ return (
+
+
+
+ To use {providerName}you will need to configure the Opik proxy
+ to avoid network issues, learn more in the{" "}
+
+ .
+
+
+ field.onChange(e.target.value)}
+ className={cn({
+ "border-destructive": Boolean(validationErrors?.message),
+ })}
+ />
+
+
+ );
+ }}
+ />
+
+
+ {
+ const validationErrors = get(formState.errors, ["models"]);
+
+ return (
+
+
+
+ Comma separated list of available models
+
+
+ field.onChange(e.target.value)}
+ className={cn({
+ "border-destructive": Boolean(validationErrors?.message),
+ })}
+ />
+
+
+ );
+ }}
+ />
+
+
+ );
+};
+
+export default LocalAIProviderDetails;
diff --git a/apps/opik-frontend/src/components/shared/AddEditAIProviderDialog/schema.ts b/apps/opik-frontend/src/components/shared/AddEditAIProviderDialog/schema.ts
new file mode 100644
index 0000000000..9b42d0db24
--- /dev/null
+++ b/apps/opik-frontend/src/components/shared/AddEditAIProviderDialog/schema.ts
@@ -0,0 +1,43 @@
+import { z } from "zod";
+import { PROVIDER_LOCATION_TYPE, PROVIDER_TYPE } from "@/types/providers";
+import uniq from "lodash/uniq";
+
+const ProviderSchema = z
+ .union([z.nativeEnum(PROVIDER_TYPE), z.string().length(0)], {
+ required_error: "Provider is required",
+ })
+ .refine((model) => model.length >= 1, { message: "Provider is required" });
+
+export const CloudAIProviderDetailsFormSchema = z.object({
+ provider: ProviderSchema,
+ locationType: z.literal(PROVIDER_LOCATION_TYPE.cloud),
+ apiKey: z
+ .string({
+ required_error: "API key is required",
+ })
+ .min(1, { message: "API key is required" }),
+});
+
+export const LocalAIProviderDetailsFormSchema = z.object({
+ provider: ProviderSchema,
+ locationType: z.literal(PROVIDER_LOCATION_TYPE.local),
+ url: z.string().url(),
+ models: z
+ .string()
+ .min(1, { message: "Models is required" })
+ .refine(
+ (models) => {
+ const modelsArray = models.split(",");
+
+ return modelsArray.length === uniq(modelsArray).length;
+ },
+ { message: "All model names should be unique" },
+ ),
+});
+
+export const AIProviderFormSchema = z.discriminatedUnion("locationType", [
+ CloudAIProviderDetailsFormSchema,
+ LocalAIProviderDetailsFormSchema,
+]);
+
+export type AIProviderFormType = z.infer
+ = {};
+
+ PROVIDERS_OPTIONS.forEach((option) => {
+ if (option.locationType === PROVIDER_LOCATION_TYPE.local) {
+ const providerData: LocalAIProviderData | undefined =
+ getLocalAIProviderData(option.value);
+
+ if (providerData) {
+ retVal[option.value] = providerData.models.split(",").map((m) => ({
+ value: m.trim() as PROVIDER_MODEL_TYPE,
+ label: m.trim(),
+ }));
+ }
+ }
+ });
+
+ return retVal;
+ },
+ });
+
+ const setLocalAIProviderData = useCallback(
+ (provider: PROVIDER_TYPE, data: LocalAIProviderData) => {
+ const config = PROVIDERS[provider] as LOCAL_PROVIDER_OPTION_TYPE;
+ localStorage.setItem(config.lsKey, JSON.stringify(data));
+ queryClient.invalidateQueries({ queryKey: [PROVIDERS_KEYS_KEY] });
+ },
+ [queryClient],
+ );
+
+ const deleteLocalAIProviderData = useCallback(
+ (provider: PROVIDER_TYPE) => {
+ const config = PROVIDERS[provider] as LOCAL_PROVIDER_OPTION_TYPE;
+ localStorage.removeItem(config.lsKey);
+ queryClient.invalidateQueries({ queryKey: [PROVIDERS_KEYS_KEY] });
+ },
+ [queryClient],
+ );
+
+ const getLocalAIProviderData = useCallback((provider: PROVIDER_TYPE) => {
+ let retVal: LocalAIProviderData | undefined = undefined;
+ const config = PROVIDERS[provider] as LOCAL_PROVIDER_OPTION_TYPE;
+ const lsData = localStorage.getItem(config.lsKey);
+
+ if (lsData) {
+ const providerData: LocalAIProviderData | undefined = safelyParseJSON(
+ lsData,
+ true,
+ );
+
+ retVal = isEmpty(providerData) ? undefined : providerData;
+ }
+
+ return retVal;
+ }, []);
+
+ const extendWithLocalData = useCallback(
+ (data: { total: number; content: ProviderKey[] }) => {
+ const localData: ProviderKey[] = [];
+
+ PROVIDERS_OPTIONS.forEach((option) => {
+ if (option.locationType === PROVIDER_LOCATION_TYPE.local) {
+ const providerData: LocalAIProviderData | undefined =
+ getLocalAIProviderData(option.value);
+
+ if (providerData) {
+ localData.push({
+ id: option.lsKey,
+ keyName: option.apiKeyName,
+ created_at: providerData.created_at,
+ provider: option.value,
+ });
+ }
+ }
+ });
+
+ return localData.length > 0 &&
+ isNumber(data.total) &&
+ isArray(data.content)
+ ? {
+ content: [...data.content, ...localData],
+ total: data.total + localData.length,
+ }
+ : data;
+ },
+ [getLocalAIProviderData],
+ );
+
+ const getLocalIAProviderURL = useCallback(
+ (provider: PROVIDER_TYPE | "") => {
+ let retVal: string | undefined = undefined;
+
+ if (
+ provider &&
+ PROVIDERS[provider].locationType === PROVIDER_LOCATION_TYPE.local
+ ) {
+ const providerData = getLocalAIProviderData(provider);
+
+ if (providerData) {
+ retVal = providerData.url;
+ }
+ }
+
+ return retVal;
+ },
+ [getLocalAIProviderData],
+ );
+
+ return {
+ localModels,
+ extendWithLocalData,
+ getLocalAIProviderData,
+ setLocalAIProviderData,
+ deleteLocalAIProviderData,
+ getLocalIAProviderURL,
+ };
+};
+
+export default useLocalAIProviderData;
diff --git a/apps/opik-frontend/src/icons/integrations/ollama.svg b/apps/opik-frontend/src/icons/integrations/ollama.svg
new file mode 100644
index 0000000000..4ad94d2a46
--- /dev/null
+++ b/apps/opik-frontend/src/icons/integrations/ollama.svg
@@ -0,0 +1,17 @@
+
diff --git a/apps/opik-frontend/src/lib/charts.ts b/apps/opik-frontend/src/lib/charts.ts
index 0fcf975584..57bb185149 100644
--- a/apps/opik-frontend/src/lib/charts.ts
+++ b/apps/opik-frontend/src/lib/charts.ts
@@ -5,11 +5,14 @@ import { generateTagVariant } from "@/lib/traces";
export const getDefaultHashedColorsChartConfig = (
lines: string[],
labelsMap?: Record,
+ predefinedColorMap: Record = {},
) => {
return lines.reduce((acc, line) => {
acc[line] = {
label: labelsMap?.[line] ?? line,
- color: TAG_VARIANTS_COLOR_MAP[generateTagVariant(line)!],
+ color:
+ predefinedColorMap[line] ||
+ TAG_VARIANTS_COLOR_MAP[generateTagVariant(line)!],
};
return acc;
}, {});
diff --git a/apps/opik-frontend/src/lib/llm.ts b/apps/opik-frontend/src/lib/llm.ts
index 47522d9298..a6e6d3d968 100644
--- a/apps/opik-frontend/src/lib/llm.ts
+++ b/apps/opik-frontend/src/lib/llm.ts
@@ -1,7 +1,5 @@
import { LLM_MESSAGE_ROLE, LLMMessage } from "@/types/llm";
import { generateRandomString } from "@/lib/utils";
-import { PROVIDER_MODEL_TYPE, PROVIDER_TYPE } from "@/types/providers";
-import { PROVIDER_MODELS } from "@/constants/llm";
export const generateDefaultLLMPromptMessage = (
message: Partial = {},
@@ -13,27 +11,6 @@ export const generateDefaultLLMPromptMessage = (
id: generateRandomString(),
};
};
-export const getModelProvider = (
- modelName: PROVIDER_MODEL_TYPE,
-): PROVIDER_TYPE | "" => {
- const provider = Object.entries(PROVIDER_MODELS).find(
- ([providerName, providerModels]) => {
- if (providerModels.find((pm) => modelName === pm.value)) {
- return providerName;
- }
-
- return false;
- },
- );
-
- if (!provider) {
- return "";
- }
-
- const [providerName] = provider;
-
- return providerName as PROVIDER_TYPE;
-};
export const getNextMessageType = (
previousMessage: LLMMessage,
diff --git a/apps/opik-frontend/src/lib/playground.ts b/apps/opik-frontend/src/lib/playground.ts
index e4454dcfab..a80863525e 100644
--- a/apps/opik-frontend/src/lib/playground.ts
+++ b/apps/opik-frontend/src/lib/playground.ts
@@ -11,12 +11,14 @@ import {
PROVIDER_MODEL_TYPE,
PROVIDER_TYPE,
} from "@/types/providers";
-import { getDefaultProviderKey } from "@/lib/provider";
-import { PROVIDERS } from "@/constants/providers";
-import { generateDefaultLLMPromptMessage, getModelProvider } from "@/lib/llm";
+import { generateDefaultLLMPromptMessage } from "@/lib/llm";
+import {
+ ModelResolver,
+ ProviderResolver,
+} from "@/hooks/useLLMProviderModelsData";
export const getDefaultConfigByProvider = (
- provider: PROVIDER_TYPE,
+ provider?: PROVIDER_TYPE | "",
): LLMPromptConfigsType => {
if (provider === PROVIDER_TYPE.OPEN_AI) {
return {
@@ -39,59 +41,30 @@ export const getDefaultConfigByProvider = (
return {};
};
-const getDefaultModel = (
- lastPickedModel: PROVIDER_MODEL_TYPE | "",
- setupProviders: PROVIDER_TYPE[],
-) => {
- const lastPickedModelProvider = lastPickedModel
- ? getModelProvider(lastPickedModel)
- : "";
-
- const isLastPickedModelValid =
- !!lastPickedModelProvider &&
- setupProviders.includes(lastPickedModelProvider);
-
- if (isLastPickedModelValid) {
- return lastPickedModel;
- }
-
- const defaultProviderKey = getDefaultProviderKey(setupProviders);
-
- if (defaultProviderKey) {
- return PROVIDERS[defaultProviderKey]?.defaultModel || "";
- }
-
- return "";
-};
-
-const getDefaultModelConfigs = (model: PROVIDER_MODEL_TYPE | "") => {
- if (!model) {
- return {};
- }
-
- const modelProvider = getModelProvider(model);
-
- return modelProvider ? getDefaultConfigByProvider(modelProvider) : {};
-};
-
interface GenerateDefaultPromptParams {
initPrompt?: Partial;
setupProviders: PROVIDER_TYPE[];
lastPickedModel?: PROVIDER_MODEL_TYPE | "";
+ providerResolver: ProviderResolver;
+ modelResolver: ModelResolver;
}
export const generateDefaultPrompt = ({
initPrompt = {},
setupProviders = [],
lastPickedModel,
+ providerResolver,
+ modelResolver,
}: GenerateDefaultPromptParams): PlaygroundPromptType => {
- const modelByDefault = getDefaultModel(lastPickedModel || "", setupProviders);
+ const modelByDefault = modelResolver(lastPickedModel || "", setupProviders);
+ const provider = providerResolver(modelByDefault);
return {
name: "Prompt",
messages: [generateDefaultLLMPromptMessage()],
model: modelByDefault,
- configs: getDefaultModelConfigs(modelByDefault),
+ provider,
+ configs: getDefaultConfigByProvider(provider),
...initPrompt,
id: generateRandomString(),
};
diff --git a/apps/opik-frontend/src/lib/utils.ts b/apps/opik-frontend/src/lib/utils.ts
index 10c9adbb51..89c59b04b1 100644
--- a/apps/opik-frontend/src/lib/utils.ts
+++ b/apps/opik-frontend/src/lib/utils.ts
@@ -34,11 +34,11 @@ export const isValidJsonObject = (string: string) => {
return json && isObject(json);
};
-export const safelyParseJSON = (string: string) => {
+export const safelyParseJSON = (string: string, silent = false) => {
try {
return JSON.parse(string);
} catch (e) {
- console.error(e);
+ if (!silent) console.error(e);
return {};
}
};
diff --git a/apps/opik-frontend/src/main.scss b/apps/opik-frontend/src/main.scss
index 941ff09300..1eb8630f83 100644
--- a/apps/opik-frontend/src/main.scss
+++ b/apps/opik-frontend/src/main.scss
@@ -274,3 +274,11 @@
background: hsl(var(--secondary));
}
}
+
+.comet-markdown ul {
+ @apply list-disc pl-6;
+}
+
+.comet-markdown ol {
+ @apply list-decimal pl-6;
+}
diff --git a/apps/opik-frontend/src/plugins/comet/UserMenu.tsx b/apps/opik-frontend/src/plugins/comet/UserMenu.tsx
index 868ea652fc..a29295731c 100644
--- a/apps/opik-frontend/src/plugins/comet/UserMenu.tsx
+++ b/apps/opik-frontend/src/plugins/comet/UserMenu.tsx
@@ -94,6 +94,10 @@ const UserMenu = () => {
const organizationWorkspaces = workspaces.filter(
(workspace) => workspace.organizationId === organization?.id,
);
+ const teamNames = user.getTeams.teams.map((team) => team.teamName);
+ const organizationWorkspacesAsMember = organizationWorkspaces.filter(
+ (workspace) => teamNames.includes(workspace.workspaceName),
+ );
const isOrganizationAdmin =
organization?.role === ORGANIZATION_ROLE_TYPE.admin;
const workspacePermissions = userPermissions.find(
@@ -189,24 +193,25 @@ const UserMenu = () => {
", # required but unused
+ )
+
+ @app.post("/v1/chat/completions", response_model=None)
+ async def chat_completions(request: Request) -> Any:
+ try:
+ try:
+ body = await request.json()
+ except json.JSONDecodeError as e:
+ # Don't log the full traceback for expected errors
+ forward_logger.info(
+ f"[dim]{request.state.request_id}[/] Invalid JSON received: {str(e)}"
+ )
+ raise HTTPException(
+ status_code=400,
+ detail={
+ "error": f"Invalid JSON: {str(e)}",
+ "request_id": request.state.request_id,
+ },
+ )
+
+ stream = body.get("stream", False)
+
+ try:
+ if stream:
+ forward_logger.debug(
+ f"[dim]{request.state.request_id}[/] Starting streaming chat completion"
+ )
+ stream = await client.chat.completions.create(**body)
+ return StreamingResponse(
+ stream_generator(stream), media_type="text/event-stream"
+ )
+ else:
+ forward_logger.debug(
+ f"[dim]{request.state.request_id}[/] Starting chat completion"
+ )
+ response = await client.chat.completions.create(**body)
+ return JSONResponse(response.model_dump())
+ except APIConnectionError as e:
+ forward_logger.error(
+ f"[dim]{request.state.request_id}[/] Failed to connect to LLM server: {str(e)}"
+ )
+ raise HTTPException(
+ status_code=503,
+ detail={
+ "error": f"Failed to connect to LLM server. Is it running at {llm_server_host}?",
+ "request_id": request.state.request_id,
+ },
+ )
+ except APITimeoutError as e:
+ forward_logger.error(
+ f"[dim]{request.state.request_id}[/] LLM server timeout: {str(e)}"
+ )
+ raise HTTPException(
+ status_code=504,
+ detail={
+ "error": "LLM server took too long to respond",
+ "request_id": request.state.request_id,
+ },
+ )
+ except APIError as e:
+ forward_logger.error(
+ f"[dim]{request.state.request_id}[/] LLM server error: {str(e)}"
+ )
+ raise HTTPException(
+ status_code=e.status_code if hasattr(e, "status_code") else 500,
+ detail={
+ "error": f"LLM server error: {str(e)}",
+ "request_id": request.state.request_id,
+ },
+ )
+
+ except HTTPException:
+ raise
+ except Exception as e:
+ # Only log full traceback for unexpected errors
+ forward_logger.exception(
+ f"[dim]{request.state.request_id}[/] Error in chat completions endpoint"
+ )
+ raise HTTPException(
+ status_code=500,
+ detail={"error": str(e), "request_id": request.state.request_id},
+ )
+
+ @app.get("/v1/models", response_model=None)
+ async def models(request: Request) -> Dict[str, Any]:
+ """List available Ollama models."""
+ try:
+ try:
+ response = await client.models.list()
+ return JSONResponse(response.model_dump())
+ except APIConnectionError as e:
+ forward_logger.error(
+ f"[dim]{request.state.request_id}[/] Failed to connect to LLM server: {str(e)}"
+ )
+ raise HTTPException(
+ status_code=503,
+ detail={
+ "error": f"Failed to connect to LLM server. Is it running at {llm_server_host}?",
+ "request_id": request.state.request_id,
+ },
+ )
+ except APITimeoutError as e:
+ forward_logger.error(
+ f"[dim]{request.state.request_id}[/] LLM server timeout: {str(e)}"
+ )
+ raise HTTPException(
+ status_code=504,
+ detail={
+ "error": "LLM server took too long to respond",
+ "request_id": request.state.request_id,
+ },
+ )
+ except APIError as e:
+ forward_logger.error(
+ f"[dim]{request.state.request_id}[/] LLM server error: {str(e)}"
+ )
+ raise HTTPException(
+ status_code=e.status_code if hasattr(e, "status_code") else 500,
+ detail={
+ "error": f"LLM server error: {str(e)}",
+ "request_id": request.state.request_id,
+ },
+ )
+
+ except HTTPException:
+ raise
+ except Exception as e:
+ forward_logger.exception(
+ f"[dim]{request.state.request_id}[/] Unexpected error in list models endpoint"
+ )
+ raise HTTPException(
+ status_code=500,
+ detail={"error": str(e), "request_id": request.state.request_id},
+ )
+
+ @app.get("/health", response_model=None)
+ async def health() -> Dict[str, str]:
+ """Health check endpoint."""
+ return {"status": "ok"}
+
+ return app
diff --git a/sdks/python/src/opik/forwarding_server/utils.py b/sdks/python/src/opik/forwarding_server/utils.py
new file mode 100644
index 0000000000..14b205901d
--- /dev/null
+++ b/sdks/python/src/opik/forwarding_server/utils.py
@@ -0,0 +1,93 @@
+"""Utilities for the Opik forwarding server."""
+
+from rich.console import Console
+from rich.panel import Panel
+from rich.text import Text
+from rich import box
+from rich.align import Align
+import httpx
+from typing import List
+
+console = Console()
+
+
+def check_llm_server_running(url: str) -> bool:
+ """Check if the LLM server is accessible."""
+ try:
+ with httpx.Client() as client:
+ client.get(f"{url}", timeout=2.0)
+ return True
+ except Exception:
+ return False
+
+
+def get_model_list(local_url: str) -> List[str]:
+ """Get a list of available models."""
+ try:
+ with httpx.Client() as client:
+ response = client.get(f"{local_url}/v1/models", timeout=2.0)
+ res = response.json()
+ models = [model["id"] for model in res["data"]]
+ return models
+ except Exception:
+ return []
+
+
+def print_server_startup_message(
+ host: str,
+ port: int,
+ llm_server_type: str,
+ llm_server_host: str,
+) -> None:
+ """Print a nicely formatted server startup message."""
+ # Create main content with improved styling
+ local_url = f"http://{host if host != '0.0.0.0' else '127.0.0.1'}:{port}"
+ models = get_model_list(llm_server_host)
+
+ # Status section with improved formatting
+ content = Text()
+ if not check_llm_server_running(llm_server_host):
+ content.append("\n")
+ content.append(
+ f"⚠️ The {llm_server_type} server is not running at {llm_server_host}, all LLM calls will fail.\n",
+ style="bold red",
+ )
+
+ content.append(
+ "\n───────────────────────────────────────────────────────────────────────────────────────────────\n"
+ )
+
+ content.append("\n")
+ content.append(
+ "The Opik proxy server is now running, you can now navigate to Opik and setup up\n\n"
+ )
+ content.append("🚀 Proxy server running:\n")
+ content.append(f" - URL: {local_url}/v1/chat/completions\n")
+ if len(models) > 0:
+ content.append(f" - Models: {', '.join(models)}\n")
+ else:
+ content.append(" - No models available\n")
+ content.append("\n")
+ content.append("📚 Documentation:\n")
+ content.append(" - https://www.comet.com/docs/opik/playground\n")
+ content.append("\n")
+ content.append("Note:", style="bold yellow")
+ content.append(
+ f"\n This server is meant only for development purposes, for production we recommend deploying {llm_server_type}\n"
+ "in a scalable and secure way.\n"
+ )
+
+ # Create the main panel with rounded corners and title
+ main_panel = Panel(
+ Align.left(content),
+ box=box.ROUNDED,
+ border_style="cyan",
+ padding=(0, 2),
+ title="Opik Proxy Server",
+ title_align="center",
+ )
+
+ # Print everything with proper spacing
+ console.print()
+ console.print(main_panel)
+ console.print()
diff --git a/sdks/python/src/opik/id_helpers.py b/sdks/python/src/opik/id_helpers.py
index d14982686a..92953f57d6 100644
--- a/sdks/python/src/opik/id_helpers.py
+++ b/sdks/python/src/opik/id_helpers.py
@@ -1,5 +1,15 @@
from datetime import datetime
+from typing import Optional
import uuid
+import uuid6
+
+
+def generate_id(timestamp: Optional[datetime] = None) -> str:
+ if timestamp:
+ uuid4 = str(uuid.uuid4())
+ return str(uuid4_to_uuid7(timestamp, uuid4))
+
+ return str(uuid6.uuid7())
def uuid4_to_uuid7(user_datetime: datetime, user_uuid: str) -> uuid.UUID:
diff --git a/sdks/python/src/opik/integrations/aisuite/aisuite_decorator.py b/sdks/python/src/opik/integrations/aisuite/aisuite_decorator.py
index 560b9b8625..3e3878a9cc 100644
--- a/sdks/python/src/opik/integrations/aisuite/aisuite_decorator.py
+++ b/sdks/python/src/opik/integrations/aisuite/aisuite_decorator.py
@@ -136,12 +136,10 @@ def _end_span_inputs_preprocessor(
return result
- def _generators_handler(
+ def _streams_handler(
self,
output: Any,
capture_output: bool,
generations_aggregator: Optional[Callable[[List[Any]], str]],
) -> Optional[Union[Generator, AsyncGenerator]]:
- return super()._generators_handler(
- output, capture_output, generations_aggregator
- )
+ return super()._streams_handler(output, capture_output, generations_aggregator)
diff --git a/sdks/python/src/opik/integrations/anthropic/messages_create_decorator.py b/sdks/python/src/opik/integrations/anthropic/messages_create_decorator.py
index 267c40f347..c1c9aa581d 100644
--- a/sdks/python/src/opik/integrations/anthropic/messages_create_decorator.py
+++ b/sdks/python/src/opik/integrations/anthropic/messages_create_decorator.py
@@ -77,7 +77,7 @@ def _end_span_inputs_preprocessor(
return result
- def _generators_handler(
+ def _streams_handler(
self,
output: Any,
capture_output: bool,
diff --git a/sdks/python/src/opik/integrations/bedrock/converse_decorator.py b/sdks/python/src/opik/integrations/bedrock/converse_decorator.py
index 2cafbfadfe..db9d2013de 100644
--- a/sdks/python/src/opik/integrations/bedrock/converse_decorator.py
+++ b/sdks/python/src/opik/integrations/bedrock/converse_decorator.py
@@ -70,7 +70,7 @@ def _end_span_inputs_preprocessor(
return result
- def _generators_handler( # type: ignore
+ def _streams_handler( # type: ignore
self,
output: Any,
capture_output: bool,
diff --git a/sdks/python/src/opik/integrations/bedrock/invoke_agent_decorator.py b/sdks/python/src/opik/integrations/bedrock/invoke_agent_decorator.py
index b2aa5086c4..083b3ad2fc 100644
--- a/sdks/python/src/opik/integrations/bedrock/invoke_agent_decorator.py
+++ b/sdks/python/src/opik/integrations/bedrock/invoke_agent_decorator.py
@@ -62,7 +62,7 @@ def _end_span_inputs_preprocessor(
return result
- def _generators_handler( # type: ignore
+ def _streams_handler( # type: ignore
self,
output: Any,
capture_output: bool,
diff --git a/sdks/python/src/opik/integrations/crewai/crewai_decorator.py b/sdks/python/src/opik/integrations/crewai/crewai_decorator.py
index 5295d19b6c..b838a8724c 100644
--- a/sdks/python/src/opik/integrations/crewai/crewai_decorator.py
+++ b/sdks/python/src/opik/integrations/crewai/crewai_decorator.py
@@ -200,12 +200,10 @@ def _parse_outputs(
return model, provider, output_dict, usage
- def _generators_handler(
+ def _streams_handler(
self,
output: Any,
capture_output: bool,
generations_aggregator: Optional[Callable[[List[Any]], str]],
) -> Optional[Union[Generator, AsyncGenerator]]:
- return super()._generators_handler(
- output, capture_output, generations_aggregator
- )
+ return super()._streams_handler(output, capture_output, generations_aggregator)
diff --git a/sdks/python/src/opik/integrations/dspy/callback.py b/sdks/python/src/opik/integrations/dspy/callback.py
index 52b8fdc340..39e0746325 100644
--- a/sdks/python/src/opik/integrations/dspy/callback.py
+++ b/sdks/python/src/opik/integrations/dspy/callback.py
@@ -4,7 +4,7 @@
import dspy
from dspy.utils.callback import BaseCallback
-from opik import opik_context
+from opik import opik_context, types
from opik.api_objects import helpers, span, trace
from opik.api_objects.opik_client import get_client_cached
from opik.decorator import error_info_collector
@@ -238,7 +238,7 @@ def _callback_context_set(self, value: ContextType) -> None:
token = self._current_callback_context.set(value)
self._map_span_id_or_trace_id_to_token[value.id] = token
- def _get_span_type(self, instance: Any) -> span.SpanType:
+ def _get_span_type(self, instance: Any) -> types.SpanType:
if isinstance(instance, dspy.Predict):
return "llm"
elif isinstance(instance, dspy.LM):
diff --git a/sdks/python/src/opik/integrations/guardrails/guardrails_decorator.py b/sdks/python/src/opik/integrations/guardrails/guardrails_decorator.py
index fc3afd38fa..007389a23c 100644
--- a/sdks/python/src/opik/integrations/guardrails/guardrails_decorator.py
+++ b/sdks/python/src/opik/integrations/guardrails/guardrails_decorator.py
@@ -75,12 +75,10 @@ def _end_span_inputs_preprocessor(
return result
- def _generators_handler(
+ def _streams_handler(
self,
output: Any,
capture_output: bool,
generations_aggregator: Optional[Callable[[List[Any]], str]],
) -> Optional[Union[Generator, AsyncGenerator]]:
- return super()._generators_handler(
- output, capture_output, generations_aggregator
- )
+ return super()._streams_handler(output, capture_output, generations_aggregator)
diff --git a/sdks/python/src/opik/integrations/langchain/google_run_helpers.py b/sdks/python/src/opik/integrations/langchain/google_run_helpers.py
index 45e8d5896c..4f259ad417 100644
--- a/sdks/python/src/opik/integrations/langchain/google_run_helpers.py
+++ b/sdks/python/src/opik/integrations/langchain/google_run_helpers.py
@@ -1,8 +1,8 @@
import logging
-from typing import Any, Dict, Optional, TYPE_CHECKING, Tuple, cast
+from typing import Any, Dict, Final, Optional, TYPE_CHECKING, Tuple
from opik import logging_messages
-from opik.types import LLMUsageInfo, UsageDict
+from opik.types import LLMProvider, LLMUsageInfo, UsageDictVertexAI
from opik.validation import usage as usage_validator
if TYPE_CHECKING:
@@ -10,6 +10,8 @@
LOGGER = logging.getLogger(__name__)
+PROVIDER_NAME: Final[LLMProvider] = "google_vertexai"
+
def get_llm_usage_info(run_dict: Optional[Dict[str, Any]] = None) -> LLMUsageInfo:
if run_dict is None:
@@ -21,21 +23,30 @@ def get_llm_usage_info(run_dict: Optional[Dict[str, Any]] = None) -> LLMUsageInf
return LLMUsageInfo(provider=provider, model=model, usage=usage_dict)
-def _try_get_token_usage(run_dict: Dict[str, Any]) -> Optional[UsageDict]:
+def _try_get_token_usage(run_dict: Dict[str, Any]) -> Optional[UsageDictVertexAI]:
try:
+ provider, _ = _get_provider_and_model(run_dict)
+
usage_metadata = run_dict["outputs"]["generations"][-1][-1]["generation_info"][
"usage_metadata"
]
- token_usage = UsageDict(
+ token_usage = UsageDictVertexAI(
completion_tokens=usage_metadata["candidates_token_count"],
prompt_tokens=usage_metadata["prompt_token_count"],
total_tokens=usage_metadata["total_token_count"],
- )
- token_usage.update(usage_metadata)
-
- if usage_validator.UsageValidator(token_usage).validate().ok():
- return cast(UsageDict, token_usage)
+ **usage_metadata,
+ ) # type: ignore
+
+ if (
+ usage_validator.UsageValidator(
+ usage=token_usage,
+ provider=provider,
+ )
+ .validate()
+ .ok()
+ ):
+ return token_usage
return None
except Exception:
@@ -77,7 +88,7 @@ def _get_provider_and_model(
if invocation_params := run_dict["extra"].get("invocation_params"):
provider = invocation_params.get("_type")
if provider == "vertexai":
- provider = "google_vertexai"
+ provider = PROVIDER_NAME
model = invocation_params.get("model_name")
return provider, model
diff --git a/sdks/python/src/opik/integrations/langchain/openai_run_helpers.py b/sdks/python/src/opik/integrations/langchain/openai_run_helpers.py
index 898f7842fa..2301f76e5d 100644
--- a/sdks/python/src/opik/integrations/langchain/openai_run_helpers.py
+++ b/sdks/python/src/opik/integrations/langchain/openai_run_helpers.py
@@ -1,7 +1,7 @@
import logging
from typing import Any, Dict, Optional, TYPE_CHECKING, Tuple, cast
-from opik import logging_messages
+from opik import _logging as opik_logging, logging_messages
from opik.types import LLMUsageInfo, UsageDict
from opik.validation import usage as usage_validator
@@ -30,23 +30,36 @@ def _try_get_token_usage(run_dict: Dict[str, Any]) -> Optional[UsageDict]:
token usage info might be in different places, different formats or completely missing.
"""
try:
+ provider, _ = _get_provider_and_model(run_dict)
+
if run_dict["outputs"]["llm_output"] is not None:
token_usage = run_dict["outputs"]["llm_output"]["token_usage"]
- # streaming mode handling (token usage data will be available at the end of streaming)
- else:
- token_usage_full = run_dict["outputs"]["generations"][-1][-1]["message"][
- "kwargs"
- ]["usage_metadata"]
-
+ # streaming mode handling
+ # token usage data MAY be available at the end of streaming
+ # in async mode may not provide token usage info
+ elif token_usage_full := run_dict["outputs"]["generations"][-1][-1]["message"][
+ "kwargs"
+ ].get("usage_metadata"):
token_usage = UsageDict(
completion_tokens=token_usage_full["output_tokens"],
prompt_tokens=token_usage_full["input_tokens"],
total_tokens=token_usage_full["total_tokens"],
)
token_usage.update(token_usage_full)
+ else:
+ opik_logging.log_once_at_level(
+ logging_level=logging.WARNING,
+ message=logging_messages.WARNING_TOKEN_USAGE_DATA_IS_NOT_AVAILABLE,
+ logger=LOGGER,
+ )
+ return None
- if usage_validator.UsageValidator(token_usage).validate().ok():
+ if (
+ usage_validator.UsageValidator(usage=token_usage, provider=provider)
+ .validate()
+ .ok()
+ ):
return cast(UsageDict, token_usage)
return None
@@ -89,13 +102,23 @@ def _get_provider_and_model(
provider = None
model = None
+ # here will be always provider and model
+ # but model name may not be full (model+version)
if metadata := run_dict["extra"].get("metadata"):
provider = metadata.get("ls_provider")
model = metadata.get("ls_model_name")
+ # try to detect model+version more precise way if possible
+ # .invoke() mode
if llm_output := run_dict["outputs"].get("llm_output"):
model = llm_output.get("model_name", model)
+ # streaming mode
+ elif generation_info := run_dict["outputs"]["generations"][-1][-1][
+ "generation_info"
+ ]:
+ model = generation_info.get("model_name", model)
+ # provider: check base url
if base_url := run_dict["extra"].get("invocation_params", {}).get("base_url"):
if base_url.host != "api.openai.com":
provider = base_url.host
diff --git a/sdks/python/src/opik/integrations/langchain/opik_tracer.py b/sdks/python/src/opik/integrations/langchain/opik_tracer.py
index e4fde92a0c..0d87310689 100644
--- a/sdks/python/src/opik/integrations/langchain/opik_tracer.py
+++ b/sdks/python/src/opik/integrations/langchain/opik_tracer.py
@@ -5,15 +5,15 @@
from langchain_core.tracers import BaseTracer
from opik import dict_utils, opik_context
-from opik.api_objects import opik_client, span, trace
-from opik.types import ErrorInfoDict, LLMUsageInfo
+from opik.api_objects import span, trace
+from opik.types import DistributedTraceHeadersDict, ErrorInfoDict, LLMUsageInfo
from . import (
base_llm_patcher,
google_run_helpers,
openai_run_helpers,
opik_encoder_extension,
)
-from ...api_objects import helpers
+from ...api_objects import helpers, opik_client
if TYPE_CHECKING:
from uuid import UUID
@@ -34,6 +34,9 @@ def _get_span_type(run: Dict[str, Any]) -> SpanType:
if run.get("run_type") in ["llm", "tool"]:
return cast(SpanType, run.get("run_type"))
+ if run.get("run_type") in ["prompt"]:
+ return cast(SpanType, "tool")
+
return cast(SpanType, "general")
@@ -53,6 +56,7 @@ def __init__(
metadata: Optional[Dict[str, Any]] = None,
graph: Optional["Graph"] = None,
project_name: Optional[str] = None,
+ distributed_headers: Optional[DistributedTraceHeadersDict] = None,
**kwargs: Any,
) -> None:
super().__init__(**kwargs)
@@ -78,9 +82,9 @@ def __init__(
self._project_name = project_name
- self._opik_client = opik_client.Opik(
- _use_batching=True, project_name=project_name
- )
+ self._distributed_headers = distributed_headers
+
+ self._opik_client = opik_client.get_client_cached()
def _persist_run(self, run: "Run") -> None:
run_dict: Dict[str, Any] = run.dict()
@@ -137,6 +141,14 @@ def _process_start_span(self, run: "Run") -> None:
def _track_root_run(self, run_dict: Dict[str, Any]) -> None:
run_metadata = run_dict["extra"].get("metadata", {})
root_metadata = dict_utils.deepmerge(self._trace_default_metadata, run_metadata)
+
+ if self._distributed_headers:
+ self._attach_span_to_distributed_headers(
+ run_dict=run_dict,
+ root_metadata=root_metadata,
+ )
+ return
+
current_span_data = opik_context.get_current_span_data()
if current_span_data is not None:
self._attach_span_to_existing_span(
@@ -233,6 +245,27 @@ def _attach_span_to_existing_trace(
self._span_data_map[run_dict["id"]] = span_data
self._externally_created_traces_ids.add(current_trace_data.id)
+ def _attach_span_to_distributed_headers(
+ self,
+ run_dict: Dict[str, Any],
+ root_metadata: Dict[str, Any],
+ ) -> None:
+ if self._distributed_headers is None:
+ raise ValueError("Distributed headers are not set")
+
+ span_data = span.SpanData(
+ trace_id=self._distributed_headers["opik_trace_id"],
+ parent_span_id=self._distributed_headers["opik_parent_span_id"],
+ name=run_dict["name"],
+ input=run_dict["inputs"],
+ metadata=root_metadata,
+ tags=self._trace_default_tags,
+ project_name=self._project_name,
+ type=_get_span_type(run_dict),
+ )
+ self._span_data_map[run_dict["id"]] = span_data
+ self._externally_created_traces_ids.add(span_data.trace_id)
+
def _process_end_span(self, run: "Run") -> None:
run_dict: Dict[str, Any] = run.dict()
span_data = self._span_data_map[run.id]
diff --git a/sdks/python/src/opik/integrations/openai/openai_decorator.py b/sdks/python/src/opik/integrations/openai/openai_decorator.py
index f329f67e3c..e95a59fdbf 100644
--- a/sdks/python/src/opik/integrations/openai/openai_decorator.py
+++ b/sdks/python/src/opik/integrations/openai/openai_decorator.py
@@ -110,7 +110,7 @@ def _end_span_inputs_preprocessor(
return result
- def _generators_handler( # type: ignore
+ def _streams_handler( # type: ignore
self,
output: Any,
capture_output: bool,
diff --git a/sdks/python/src/opik/logging_messages.py b/sdks/python/src/opik/logging_messages.py
index d2a4b7037a..0ba7216d5b 100644
--- a/sdks/python/src/opik/logging_messages.py
+++ b/sdks/python/src/opik/logging_messages.py
@@ -56,3 +56,5 @@
PARSE_API_KEY_TOO_MANY_PARTS = "Too many parts (%d) found in the Opik API key: %r"
LLM_PROVIDER_RATE_LIMIT_ERROR_DETECTED_IN_EVALUATE_FUNCTION = "LLM provider rate limit error detected. We recommend reducing the amount of parallel requests by setting `task_threads` evaluation parameter to a smaller number"
+
+WARNING_TOKEN_USAGE_DATA_IS_NOT_AVAILABLE = "You didn't specify argument `stream_usage`=True during LLM initialization. Token usage data is not available for .stream() or .astream() methods."
diff --git a/sdks/python/src/opik/plugins/pytest/experiment_runner.py b/sdks/python/src/opik/plugins/pytest/experiment_runner.py
index e124a306e5..9afadb4d24 100644
--- a/sdks/python/src/opik/plugins/pytest/experiment_runner.py
+++ b/sdks/python/src/opik/plugins/pytest/experiment_runner.py
@@ -6,7 +6,7 @@
from opik.api_objects import opik_client
from opik.api_objects.experiment import experiment_item
from opik.api_objects.dataset import dataset_item
-from opik.api_objects import helpers
+from opik import id_helpers
from opik import datetime_helpers, dict_utils
from . import test_runs_storage, test_run_content
@@ -59,7 +59,7 @@ def run(client: opik_client.Opik, test_items: List[Item]) -> None:
dataset_item_id = dataset_item_id_finder(test_run_content)
if dataset_item_id is None:
- dataset_item_id = helpers.generate_id()
+ dataset_item_id = id_helpers.generate_id()
filtered_test_run_content_dict = dict_utils.remove_none_from_dict(
test_run_content.__dict__
)
diff --git a/sdks/python/src/opik/types.py b/sdks/python/src/opik/types.py
index 886fc01a4e..60e362a31f 100644
--- a/sdks/python/src/opik/types.py
+++ b/sdks/python/src/opik/types.py
@@ -1,7 +1,7 @@
import dataclasses
import sys
-from typing import Literal, Optional
+from typing import Literal, Optional, Union
from typing_extensions import TypedDict
if sys.version_info < (3, 11):
@@ -12,6 +12,7 @@
SpanType = Literal["general", "tool", "llm"]
FeedbackType = Literal["numerical", "categorical"]
CreatedByType = Literal["evaluation"]
+LLMProvider = Literal["openai", "google_vertexai"]
class UsageDict(TypedDict):
@@ -32,7 +33,32 @@ class UsageDict(TypedDict):
"""The total number of tokens used, including both prompt and completion."""
+class UsageDictVertexAI(UsageDict):
+ """
+ A TypedDict representing token usage information for Google Vertex AI.
+
+ This class defines the structure for token usage, including fields
+ for completion tokens, prompt tokens, and the total number of tokens used.
+ """
+
+ cached_content_token_count: NotRequired[int]
+ """The number of tokens cached."""
+
+ candidates_token_count: int
+ """The number of tokens used for the completion."""
+
+ prompt_token_count: int
+ """The number of tokens used for the prompt."""
+
+ total_token_count: int
+ """The total number of tokens used, including both prompt and completion."""
+
+
class DistributedTraceHeadersDict(TypedDict):
+ """
+ Contains headers for distributed tracing, returned by the :py:func:`opik.opik_context.get_distributed_trace_headers` function.
+ """
+
opik_trace_id: str
opik_parent_span_id: str
@@ -84,4 +110,4 @@ class ErrorInfoDict(TypedDict):
class LLMUsageInfo:
provider: Optional[str] = None
model: Optional[str] = None
- usage: Optional[UsageDict] = None
+ usage: Optional[Union[UsageDict, UsageDictVertexAI]] = None
diff --git a/sdks/python/src/opik/validation/usage.py b/sdks/python/src/opik/validation/usage.py
index cc43f31d24..e16a9de94d 100644
--- a/sdks/python/src/opik/validation/usage.py
+++ b/sdks/python/src/opik/validation/usage.py
@@ -1,20 +1,21 @@
-import pydantic
import dataclasses
+from typing import Any, Dict, Optional, Union
+
+import pydantic
-from typing import Any, Dict, Optional
-from ..types import UsageDict
-from . import validator, result
+from . import result, validator
+from ..types import UsageDict, UsageDictVertexAI
class PydanticWrapper(pydantic.BaseModel):
model_config = pydantic.ConfigDict(extra="forbid")
- usage: UsageDict
+ usage: Union[UsageDict, UsageDictVertexAI]
@dataclasses.dataclass
class ParsedUsage:
full_usage: Optional[Dict[str, Any]] = None
- supported_usage: Optional[UsageDict] = None
+ supported_usage: Optional[Union[UsageDict, UsageDictVertexAI]] = None
EXPECTED_TYPES = "{'completion_tokens': int, 'prompt_tokens': int, 'total_tokens': int}"
@@ -25,17 +26,23 @@ class UsageValidator(validator.Validator):
Validator for span token usage
"""
- def __init__(self, usage: Any):
+ def __init__(self, usage: Any, provider: Optional[str]):
self.usage = usage
-
+ self.provider = provider
self.parsed_usage = ParsedUsage()
def validate(self) -> result.ValidationResult:
try:
if isinstance(self.usage, dict):
- filtered_usage = _keep_supported_keys(self.usage)
+ filtered_usage = self.supported_keys
+
+ # run validation
PydanticWrapper(usage=filtered_usage)
- supported_usage = UsageDict(**filtered_usage) # type: ignore
+
+ if self.provider == "google_vertexai":
+ supported_usage = UsageDictVertexAI(**filtered_usage) # type: ignore
+ else:
+ supported_usage = UsageDict(**filtered_usage) # type: ignore
self.parsed_usage = ParsedUsage(
full_usage=self.usage, supported_usage=supported_usage
)
@@ -67,13 +74,18 @@ def failure_reason_message(self) -> str:
), "validate() must be called before accessing failure reason message"
return self.validation_result.failure_reasons[0]
+ @property
+ def supported_keys(self) -> Dict[str, Any]:
+ if self.provider == "google_vertexai":
+ supported_keys = UsageDictVertexAI.__annotations__.keys()
+ # `openai` and all other
+ else:
+ supported_keys = UsageDict.__annotations__.keys()
-def _keep_supported_keys(usage: Dict[str, Any]) -> Dict[str, Any]:
- supported_keys = UsageDict.__annotations__.keys()
- filtered_usage = {}
+ filtered_usage = {}
- for key in supported_keys:
- if key in usage:
- filtered_usage[key] = usage[key]
+ for key in supported_keys:
+ if key in self.usage:
+ filtered_usage[key] = self.usage[key]
- return filtered_usage
+ return filtered_usage
diff --git a/sdks/python/tests/e2e/test_tracing.py b/sdks/python/tests/e2e/test_tracing.py
index 1e8be094cb..bcde084604 100644
--- a/sdks/python/tests/e2e/test_tracing.py
+++ b/sdks/python/tests/e2e/test_tracing.py
@@ -1,3 +1,4 @@
+import time
import pytest
import uuid
@@ -516,6 +517,91 @@ def test_search_spans__happyflow(opik_client):
assert spans[0].id == matching_span.id, "Expected to find the matching span"
+def test_copy_traces__happyflow(opik_client):
+ # Log traces
+ unique_identifier = str(uuid.uuid4())[-6:]
+ ID_STORAGE = {}
+
+ project_name = f"e2e-tests-copy-traces-project - {unique_identifier}"
+ for i in range(2):
+ trace = opik_client.trace(
+ name="trace",
+ project_name=project_name,
+ input={"input": f"test input - {i}"},
+ output={"output": f"test output - {i}"},
+ feedback_scores=[
+ {
+ "name": "score_trace",
+ "value": i,
+ "category_name": "category_",
+ "reason": "reason_",
+ }
+ ],
+ metadata={"value": i},
+ tags=["a", "b"],
+ )
+ ID_STORAGE[f"trace-{i}-id"] = trace.id
+
+ trace.span(
+ name="span - 0",
+ input={"input": f"test input - {i} - 0"},
+ )
+
+ # Sleep so span timestamps are ordered due to uuid7
+ time.sleep(0.001)
+
+ trace.span(
+ name="span - 1",
+ input={"input": f"test input - {i} - 1"},
+ )
+
+ opik_client.flush()
+
+ new_project_name = project_name + "_v2"
+ opik_client.copy_traces(
+ project_name=project_name,
+ destination_project_name=new_project_name,
+ )
+
+ opik_client.flush()
+
+ traces = opik_client.search_traces(project_name=new_project_name)
+ for i, trace in enumerate(reversed(traces)):
+ verifiers.verify_trace(
+ opik_client=opik_client,
+ trace_id=trace.id,
+ name="trace",
+ input={"input": f"test input - {i}"},
+ output={"output": f"test output - {i}"},
+ feedback_scores=[
+ {
+ "id": trace.id,
+ "name": "score_trace",
+ "value": i,
+ "category_name": "category_",
+ "reason": "reason_",
+ }
+ ],
+ metadata={"value": i},
+ tags=["a", "b"],
+ project_name=new_project_name,
+ )
+
+ trace_spans = opik_client.search_spans(
+ project_name=new_project_name, trace_id=trace.id
+ )
+ for j, span in enumerate(reversed(trace_spans)):
+ verifiers.verify_span(
+ opik_client=opik_client,
+ span_id=span.id,
+ trace_id=trace.id,
+ name=f"span - {j}",
+ input={"input": f"test input - {i} - {j}"},
+ parent_span_id=span.parent_span_id,
+ project_name=new_project_name,
+ )
+
+
def test_tracked_function__update_current_span_used_to_update_cost__happyflow(
opik_client,
):
diff --git a/sdks/python/tests/library_integration/haystack/test_haystack.py b/sdks/python/tests/library_integration/haystack/test_haystack.py
index 4890b4c971..c9b9e6c53a 100644
--- a/sdks/python/tests/library_integration/haystack/test_haystack.py
+++ b/sdks/python/tests/library_integration/haystack/test_haystack.py
@@ -15,6 +15,7 @@
)
+@pytest.mark.skip
@pytest.fixture(autouse=True, scope="module")
def enable_haystack_content_tracing():
assert (
@@ -24,6 +25,7 @@ def enable_haystack_content_tracing():
yield
+@pytest.mark.skip
@pytest.mark.parametrize(
"project_name, expected_project_name",
[
diff --git a/sdks/python/tests/library_integration/langchain/test_langchain.py b/sdks/python/tests/library_integration/langchain/test_langchain.py
index 4808a8a7fb..3d689bf34d 100644
--- a/sdks/python/tests/library_integration/langchain/test_langchain.py
+++ b/sdks/python/tests/library_integration/langchain/test_langchain.py
@@ -1,3 +1,5 @@
+import asyncio
+
import langchain_google_vertexai
import langchain_openai
import pytest
@@ -9,6 +11,7 @@
from opik.api_objects import opik_client, span, trace
from opik.config import OPIK_PROJECT_DEFAULT_NAME
from opik.integrations.langchain.opik_tracer import OpikTracer
+from opik.types import DistributedTraceHeadersDict
from ...testlib import (
ANY_BUT_NONE,
ANY_DICT,
@@ -76,7 +79,7 @@ def test_langchain__happyflow(
spans=[
SpanModel(
id=ANY_BUT_NONE,
- type="general",
+ type="tool",
name="PromptTemplate",
input={"title": "Documentary about Bigfoot in Paris"},
output=ANY_DICT,
@@ -123,6 +126,149 @@ def test_langchain__happyflow(
assert_equal(EXPECTED_TRACE_TREE, fake_backend.trace_trees[0])
+def test_langchain__distributed_headers__happyflow(
+ fake_backend,
+):
+ project_name = "langchain-integration-test--distributed-headers"
+ client = opik_client.get_client_cached()
+
+ # PREPARE DISTRIBUTED HEADERS
+ trace_data = trace.TraceData(
+ name="custom-distributed-headers--trace",
+ input={
+ "key1": 1,
+ "key2": "val2",
+ },
+ project_name=project_name,
+ tags=["tag_d1", "tag_d2"],
+ )
+ trace_data.init_end_time()
+ client.trace(**trace_data.__dict__)
+
+ span_data = span.SpanData(
+ trace_id=trace_data.id,
+ parent_span_id=None,
+ name="custom-distributed-headers--span",
+ input={
+ "input": "custom-distributed-headers--input",
+ },
+ project_name=project_name,
+ tags=["tag_d3", "tag_d4"],
+ )
+ span_data.init_end_time().update(
+ output={"output": "custom-distributed-headers--output"},
+ )
+ client.span(**span_data.__dict__)
+
+ distributed_headers = DistributedTraceHeadersDict(
+ opik_trace_id=span_data.trace_id,
+ opik_parent_span_id=span_data.id,
+ )
+
+ # CALL LLM
+ llm = fake.FakeListLLM(
+ responses=["I'm sorry, I don't think I'm talented enough to write a synopsis"]
+ )
+
+ template = "Given the title of play, write a synopsys for that. Title: {title}."
+
+ prompt_template = PromptTemplate(input_variables=["title"], template=template)
+
+ synopsis_chain = prompt_template | llm
+ test_prompts = {"title": "Documentary about Bigfoot in Paris"}
+
+ callback = OpikTracer(
+ project_name=project_name,
+ tags=["tag1", "tag2"],
+ metadata={"a": "b"},
+ distributed_headers=distributed_headers,
+ )
+ synopsis_chain.invoke(input=test_prompts, config={"callbacks": [callback]})
+
+ callback.flush()
+
+ EXPECTED_TRACE_TREE = TraceModel(
+ id=ANY_BUT_NONE,
+ name="custom-distributed-headers--trace",
+ input={"key1": 1, "key2": "val2"},
+ output=None,
+ tags=["tag_d1", "tag_d2"],
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ project_name=project_name,
+ spans=[
+ SpanModel(
+ id=ANY_BUT_NONE,
+ name="custom-distributed-headers--span",
+ input={"input": "custom-distributed-headers--input"},
+ output={"output": "custom-distributed-headers--output"},
+ tags=["tag_d3", "tag_d4"],
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ project_name=project_name,
+ spans=[
+ SpanModel(
+ id=ANY_BUT_NONE,
+ name="RunnableSequence",
+ input={"title": "Documentary about Bigfoot in Paris"},
+ output=ANY_DICT,
+ tags=["tag1", "tag2"],
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ metadata={"a": "b"},
+ project_name=project_name,
+ spans=[
+ SpanModel(
+ id=ANY_BUT_NONE,
+ type="tool",
+ name="PromptTemplate",
+ input={"title": "Documentary about Bigfoot in Paris"},
+ output=ANY_DICT,
+ metadata={},
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ project_name=project_name,
+ spans=[],
+ ),
+ SpanModel(
+ id=ANY_BUT_NONE,
+ type="llm",
+ name="FakeListLLM",
+ input={
+ "prompts": [
+ "Given the title of play, write a synopsys for that. Title: Documentary about Bigfoot in Paris."
+ ]
+ },
+ output=ANY_DICT,
+ metadata={
+ "invocation_params": {
+ "responses": [
+ "I'm sorry, I don't think I'm talented enough to write a synopsis"
+ ],
+ "_type": "fake-list",
+ "stop": None,
+ },
+ "options": {"stop": None},
+ "batch_size": 1,
+ "metadata": ANY_BUT_NONE,
+ },
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ project_name=project_name,
+ spans=[],
+ ),
+ ],
+ )
+ ],
+ )
+ ],
+ )
+
+ assert len(fake_backend.trace_trees) == 1
+ assert len(callback.created_traces()) == 0
+ assert_equal(EXPECTED_TRACE_TREE, fake_backend.trace_trees[0])
+
+
@pytest.mark.parametrize(
"llm_model, expected_input_prompt, stream_usage",
[
@@ -193,7 +339,7 @@ def test_langchain__openai_llm_is_used__token_usage_is_logged__happyflow(
spans=[
SpanModel(
id=ANY_BUT_NONE,
- type="general",
+ type="tool",
name="PromptTemplate",
input={"title": "Documentary about Bigfoot in Paris"},
output={"output": ANY_BUT_NONE},
@@ -239,23 +385,22 @@ def test_langchain__openai_llm_is_used__streaming_mode__token_usage_is_logged__h
callback = OpikTracer(
tags=["tag3", "tag4"],
metadata={"c": "d"},
- project_name="878",
)
model = langchain_openai.ChatOpenAI(
max_tokens=10,
name="custom-openai-llm-name",
callbacks=[callback],
+ # THIS PARAM IS VERY IMPORTANT!
+ # if it is explicitly set to True - token usage data will be available
stream_usage=True,
)
chunks = []
- # for chunk in model.stream("what color is the sky?"):
for chunk in model.stream(
"Given the title of play, write a synopsys for that. Title: Documentary about Bigfoot in Paris."
):
chunks.append(chunk)
- # print(chunk.content, end="|", flush=True)
callback.flush()
@@ -330,6 +475,120 @@ def test_langchain__openai_llm_is_used__streaming_mode__token_usage_is_logged__h
assert_equal(EXPECTED_TRACE_TREE, fake_backend.trace_trees[0])
+def test_langchain__openai_llm_is_used__async_astream__no_token_usage_is_logged__happyflow(
+ fake_backend,
+ ensure_openai_configured,
+):
+ """
+ In `astream` mode, the `token_usage` is not provided by langchain.
+ For trace `input` always will be = {"input": ""}
+ """
+ callback = OpikTracer(
+ tags=["tag3", "tag4"],
+ metadata={"c": "d"},
+ )
+
+ model = langchain_openai.ChatOpenAI(
+ model="gpt-4o",
+ max_tokens=10,
+ name="custom-openai-llm-name",
+ callbacks=[callback],
+ # `stream_usage` param is VERY IMPORTANT!
+ # if it is explicitly set to True - token usage data will be available
+ # "stream_usage": True,
+ )
+
+ template = "Given the title of play, write a synopsys for that. Title: {title}."
+ prompt_template = PromptTemplate(input_variables=["title"], template=template)
+
+ chain = prompt_template | model
+
+ async def stream_generator(chain, inputs):
+ async for chunk in chain.astream(inputs, config={"callbacks": [callback]}):
+ yield chunk
+
+ async def invoke_generator(chain, inputs):
+ async for chunk in stream_generator(chain, inputs):
+ print(chunk)
+
+ inputs = {"title": "The Hobbit"}
+
+ asyncio.run(invoke_generator(chain, inputs))
+
+ callback.flush()
+
+ EXPECTED_TRACE_TREE = TraceModel(
+ id=ANY_BUT_NONE,
+ name="RunnableSequence",
+ input={"input": ""},
+ output=ANY_DICT,
+ tags=["tag3", "tag4"],
+ metadata={
+ "c": "d",
+ },
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ spans=[
+ SpanModel(
+ id=ANY_BUT_NONE,
+ name="RunnableSequence",
+ input={"input": ""},
+ output=ANY_BUT_NONE,
+ tags=["tag3", "tag4"],
+ metadata={
+ "c": "d",
+ },
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ type="general",
+ model=None,
+ provider=None,
+ usage=None,
+ spans=[
+ SpanModel(
+ id=ANY_BUT_NONE,
+ name="PromptTemplate",
+ input={"title": "The Hobbit"},
+ output=ANY_BUT_NONE,
+ tags=None,
+ metadata={},
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ type="tool",
+ model=None,
+ provider=None,
+ usage=None,
+ spans=[],
+ ),
+ SpanModel(
+ id=ANY_BUT_NONE,
+ name="custom-openai-llm-name",
+ input={
+ "prompts": [
+ "Human: Given the title of play, write a synopsys for that. Title: The Hobbit."
+ ]
+ },
+ output=ANY_BUT_NONE,
+ tags=None,
+ metadata=ANY_DICT,
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ type="llm",
+ model=ANY_STRING(startswith="gpt-4o"),
+ provider="openai",
+ usage=None,
+ spans=[],
+ ),
+ ],
+ )
+ ],
+ )
+
+ assert len(fake_backend.trace_trees) == 1
+ assert len(callback.created_traces()) == 1
+ assert_equal(EXPECTED_TRACE_TREE, fake_backend.trace_trees[0])
+
+
@pytest.mark.skip
@pytest.mark.parametrize(
"llm_model, expected_input_prompt, metadata_usage",
@@ -413,7 +672,7 @@ def test_langchain__google_vertexai_llm_is_used__token_usage_is_logged__happyflo
spans=[
SpanModel(
id=ANY_BUT_NONE,
- type="general",
+ type="tool",
name="PromptTemplate",
input={"title": "Documentary about Bigfoot in Paris"},
output={"output": ANY_BUT_NONE},
@@ -502,7 +761,7 @@ def test_langchain__openai_llm_is_used__error_occurred_during_openai_call__error
spans=[
SpanModel(
id=ANY_BUT_NONE,
- type="general",
+ type="tool",
name="PromptTemplate",
input={"title": "Documentary about Bigfoot in Paris"},
output={"output": ANY_BUT_NONE},
@@ -608,7 +867,7 @@ def f(x):
spans=[
SpanModel(
id=ANY_BUT_NONE,
- type="general",
+ type="tool",
name="PromptTemplate",
input={"title": "Documentary about Bigfoot in Paris"},
output={"output": ANY_BUT_NONE},
@@ -720,7 +979,7 @@ def f():
spans=[
SpanModel(
id=ANY_BUT_NONE,
- type="general",
+ type="tool",
name="PromptTemplate",
input={"title": "Documentary about Bigfoot in Paris"},
output=ANY_DICT,
@@ -826,7 +1085,7 @@ def f():
spans=[
SpanModel(
id=ANY_BUT_NONE,
- type="general",
+ type="tool",
name="PromptTemplate",
input={"title": "Documentary about Bigfoot in Paris"},
output={"output": ANY_BUT_NONE},
diff --git a/sdks/python/tests/unit/api_objects/test_opik_query_language.py b/sdks/python/tests/unit/api_objects/test_opik_query_language.py
index e3f4e11210..a96b2ed937 100644
--- a/sdks/python/tests/unit/api_objects/test_opik_query_language.py
+++ b/sdks/python/tests/unit/api_objects/test_opik_query_language.py
@@ -6,33 +6,76 @@
@pytest.mark.parametrize(
"filter_string, expected",
[
- ('name = "test"', {"field": "name", "operator": "=", "value": "test"}),
+ ('name = "test"', [{"field": "name", "operator": "=", "value": "test"}]),
(
"usage.total_tokens > 100",
- {"field": "usage.total_tokens", "operator": ">", "value": "100"},
+ [{"field": "usage.total_tokens", "operator": ">", "value": "100"}],
),
(
'tags contains "important"',
- {"field": "tags", "operator": "contains", "value": "important"},
+ [{"field": "tags", "operator": "contains", "value": "important"}],
),
(
'output not_contains "error"',
- {"field": "output", "operator": "not_contains", "value": "error"},
+ [{"field": "output", "operator": "not_contains", "value": "error"}],
),
(
"feedback_scores >= 4.5",
- {"field": "feedback_scores", "operator": ">=", "value": "4.5"},
+ [{"field": "feedback_scores", "operator": ">=", "value": "4.5"}],
+ ),
+ # New test cases for quoted identifiers
+ (
+ 'feedback_scores."Answer Relevance" < 0.8',
+ [
+ {
+ "field": "feedback_scores",
+ "key": "Answer Relevance",
+ "operator": "<",
+ "value": "0.8",
+ }
+ ],
+ ),
+ (
+ 'feedback_scores."Escaped ""Quote""" < 0.8',
+ [
+ {
+ "field": "feedback_scores",
+ "key": 'Escaped "Quote"',
+ "operator": "<",
+ "value": "0.8",
+ }
+ ],
+ ),
+ # Test cases for AND:
+ (
+ 'name = "test" AND tags contains "important" ',
+ [
+ {"field": "name", "operator": "=", "value": "test"},
+ {"field": "tags", "operator": "contains", "value": "important"},
+ ],
+ ),
+ (
+ 'feedback_scores."Escaped ""Quote""" < 0.8 and feedback_scores >= 4.5',
+ [
+ {
+ "field": "feedback_scores",
+ "key": 'Escaped "Quote"',
+ "operator": "<",
+ "value": "0.8",
+ },
+ {"field": "feedback_scores", "operator": ">=", "value": "4.5"},
+ ],
),
],
)
def test_valid_oql_expressions(filter_string, expected):
oql = OpikQueryLanguage(filter_string)
parsed = json.loads(oql.parsed_filters)
- assert len(parsed) == 1
- parsed_result = parsed[0]
+ assert len(parsed) == len(expected)
- for key, value in expected.items():
- assert parsed_result[key] == value
+ for i, line in enumerate(expected):
+ for key, value in line.items():
+ assert parsed[i][key] == value
@pytest.mark.parametrize(
@@ -46,6 +89,19 @@ def test_valid_oql_expressions(filter_string, expected):
r"When querying usage, invalid_metric is not supported.*",
),
('name = "test" extra_stuff', r"Invalid filter string, trailing characters.*"),
+ (
+ 'feedback_scores."Unterminated Quote < 0.8',
+ r'Missing closing quote for: "Unterminated Quote < 0.8',
+ ),
+ # Tests with AND
+ (
+ 'name = "test" and name = "other" extra_stuff',
+ r"Invalid filter string, trailing characters.*",
+ ),
+ (
+ 'name = "test" OR name = "other"',
+ r"Invalid filter string, OR is not currently supported",
+ ),
],
)
def test_invalid_oql_expressions(filter_string, error_pattern):
diff --git a/sdks/python/tests/unit/configurator/test_configure.py b/sdks/python/tests/unit/configurator/test_configure.py
index 18ef1806f4..bfbc9d80df 100644
--- a/sdks/python/tests/unit/configurator/test_configure.py
+++ b/sdks/python/tests/unit/configurator/test_configure.py
@@ -742,6 +742,34 @@ def test_get_workspace_accept_default(
'Do you want to use "default_workspace" workspace? (Y/n)'
)
+ @patch("opik.configurator.configure.opik.config.OpikConfig")
+ @patch(
+ "opik.configurator.configure.OpikConfigurator._get_default_workspace",
+ return_value="default_workspace",
+ )
+ @patch("opik.configurator.configure.ask_user_for_approval", return_value=True)
+ def test_get_workspace_accept_default__automatic_approvals_enabled__no_approval_questions_asked(
+ self, mock_ask_user_for_approval, mock_get_default_workspace, mock_opik_config
+ ):
+ """
+ Test that the user accepts the default workspace.
+ """
+ current_config = OpikConfig(workspace=OPIK_WORKSPACE_DEFAULT_NAME)
+ mock_opik_config.return_value = current_config
+
+ configurator = OpikConfigurator(
+ workspace=None,
+ force=False,
+ api_key="valid_api_key",
+ automatic_approvals=True,
+ )
+ needs_update = configurator._set_workspace()
+
+ assert configurator.workspace == "default_workspace"
+ assert needs_update is True
+ mock_get_default_workspace.assert_called_once_with()
+ mock_ask_user_for_approval.assert_not_called()
+
@patch("opik.configurator.configure.opik.config.OpikConfig")
@patch(
"opik.configurator.configure.OpikConfigurator._get_default_workspace",
@@ -1087,6 +1115,33 @@ def test_configure_local_uses_local_instance(
assert configurator.base_url == OPIK_BASE_URL_LOCAL
assert configurator.workspace == OPIK_WORKSPACE_DEFAULT_NAME
+ @patch("opik.configurator.configure.is_interactive", return_value=True)
+ @patch("opik.configurator.configure.ask_user_for_approval")
+ @patch(
+ "opik.configurator.configure.opik_rest_helpers.is_instance_active",
+ return_value=True,
+ )
+ @patch("opik.configurator.configure.OpikConfigurator._update_config")
+ def test_configure_local_uses_local_instance__automatic_approvals_enabled__no_approval_questions_asked(
+ self,
+ mock_update_config,
+ mock_is_instance_active,
+ mock_ask_user_for_approval,
+ mock_is_interactive,
+ ):
+ """
+ Test that the function configures the local instance when found and user approves.
+ """
+ configurator = OpikConfigurator(url=None, force=False, automatic_approvals=True)
+ configurator._configure_local()
+
+ mock_ask_user_for_approval.assert_not_called()
+ mock_update_config.assert_called_once_with()
+
+ assert configurator.api_key is None
+ assert configurator.base_url == OPIK_BASE_URL_LOCAL
+ assert configurator.workspace == OPIK_WORKSPACE_DEFAULT_NAME
+
@patch("opik.configurator.configure.is_interactive", return_value=False)
@patch("opik.configurator.configure.ask_user_for_approval", return_value=True)
@patch(
@@ -1111,6 +1166,33 @@ def test_configure_local_uses_local_instance__non_interactive(
mock_ask_user_for_approval.assert_not_called()
mock_update_config.assert_not_called()
+ @patch("opik.configurator.configure.is_interactive", return_value=False)
+ @patch("opik.configurator.configure.ask_user_for_approval")
+ @patch(
+ "opik.configurator.configure.opik_rest_helpers.is_instance_active",
+ return_value=True,
+ )
+ @patch("opik.configurator.configure.OpikConfigurator._update_config")
+ def test_configure_local_uses_local_instance__non_interactive__automatic_approvals_enabled__no_error_raised(
+ self,
+ mock_update_config,
+ mock_is_instance_active,
+ mock_ask_user_for_approval,
+ mock_is_interactive,
+ ):
+ """
+ Test that the function configures the local instance when found and user approves.
+ """
+ configurator = OpikConfigurator(url=None, force=False, automatic_approvals=True)
+ configurator._configure_local()
+
+ mock_ask_user_for_approval.assert_not_called()
+ mock_update_config.assert_called_once_with()
+
+ assert configurator.api_key is None
+ assert configurator.base_url == OPIK_BASE_URL_LOCAL
+ assert configurator.workspace == OPIK_WORKSPACE_DEFAULT_NAME
+
@patch("opik.configurator.configure.is_interactive", return_value=True)
@patch("opik.configurator.configure.ask_user_for_approval", return_value=False)
@patch("opik.configurator.configure.OpikConfigurator._ask_for_url")
diff --git a/sdks/python/tests/unit/decorator/test_tracker_outputs.py b/sdks/python/tests/unit/decorator/test_tracker_outputs.py
index 2cedc30640..2706d6b021 100644
--- a/sdks/python/tests/unit/decorator/test_tracker_outputs.py
+++ b/sdks/python/tests/unit/decorator/test_tracker_outputs.py
@@ -19,11 +19,11 @@
def test_track__one_nested_function__happyflow(fake_backend):
- @tracker.track(capture_output=True)
+ @tracker.track
def f_inner(x):
return "inner-output"
- @tracker.track(capture_output=True)
+ @tracker.track
def f_outer(x):
f_inner("inner-input")
return "outer-output"
@@ -137,16 +137,16 @@ def f(x):
def test_track__two_nested_functions__happyflow(fake_backend):
- @tracker.track(capture_output=True)
+ @tracker.track
def f_inner(z):
return "inner-output"
- @tracker.track(capture_output=True)
+ @tracker.track
def f_middle(y):
f_inner("inner-input")
return "middle-output"
- @tracker.track(capture_output=True)
+ @tracker.track
def f_outer(x):
f_middle("middle-input")
return "outer-output"
@@ -202,15 +202,15 @@ def f_outer(x):
def test_track__outer_function_has_two_separate_nested_function__happyflow(
fake_backend,
):
- @tracker.track(capture_output=True)
+ @tracker.track
def f_inner_1(y):
return "inner-output-1"
- @tracker.track(capture_output=True)
+ @tracker.track
def f_inner_2(y):
return "inner-output-2"
- @tracker.track(capture_output=True)
+ @tracker.track
def f_outer(x):
f_inner_1("inner-input-1")
f_inner_2("inner-input-2")
@@ -264,11 +264,11 @@ def f_outer(x):
def test_track__two_traces__happyflow(fake_backend):
- @tracker.track(capture_output=True)
+ @tracker.track
def f_1(x):
return "f1-output"
- @tracker.track(capture_output=True)
+ @tracker.track
def f_2(x):
return "f2-output"
@@ -327,7 +327,7 @@ def f_2(x):
def test_track__one_function__error_raised__trace_and_span_finished_correctly__outputs_are_None(
fake_backend,
):
- @tracker.track(capture_output=True)
+ @tracker.track
def f(x):
raise Exception("error message")
@@ -432,7 +432,7 @@ def f_inner():
def test_track__one_async_function__error_raised__trace_and_span_finished_correctly__outputs_are_None__error_info_is_added(
fake_backend,
):
- @tracker.track(capture_output=True)
+ @tracker.track
async def async_f(x):
await asyncio.sleep(0.01)
raise Exception("error message")
@@ -480,7 +480,7 @@ async def async_f(x):
def test_track__nested_calls_in_separate_threads__3_traces_in_result(fake_backend):
ID_STORAGE: Dict[str, str] = {}
- @tracker.track(capture_output=True)
+ @tracker.track
def f_inner(y, thread_id):
ID_STORAGE[f"f_inner-trace-id-{thread_id}"] = (
opik_context.get_current_trace_data().id
@@ -490,7 +490,7 @@ def f_inner(y, thread_id):
)
return f"inner-output-from-{thread_id}"
- @tracker.track(capture_output=True)
+ @tracker.track
def f_outer(x):
ID_STORAGE["f_outer-trace-id"] = opik_context.get_current_trace_data().id
ID_STORAGE["f_outer-span-id"] = opik_context.get_current_span_data().id
@@ -595,9 +595,9 @@ def f_outer(x):
def test_track__single_generator_function_tracked__generator_exhausted__happyflow(
fake_backend,
):
- @tracker.track(capture_output=True)
+ @tracker.track
def f(x):
- values = ["yielded-1", "yielded-2", "yielded-3"]
+ values = ["yielded-1", " yielded-2", " yielded-3"]
for value in values:
yield value
@@ -611,7 +611,7 @@ def f(x):
id=ANY_BUT_NONE,
name="f",
input={"x": "generator-input"},
- output={"output": "['yielded-1', 'yielded-2', 'yielded-3']"},
+ output={"output": "yielded-1 yielded-2 yielded-3"},
start_time=ANY_BUT_NONE,
end_time=ANY_BUT_NONE,
spans=[
@@ -619,7 +619,7 @@ def f(x):
id=ANY_BUT_NONE,
name="f",
input={"x": "generator-input"},
- output={"output": "['yielded-1', 'yielded-2', 'yielded-3']"},
+ output={"output": "yielded-1 yielded-2 yielded-3"},
start_time=ANY_BUT_NONE,
end_time=ANY_BUT_NONE,
spans=[],
@@ -683,23 +683,23 @@ def f(x):
assert_equal(EXPECTED_TRACE_TREE, fake_backend.trace_trees[0])
-def test_track__generator_function_tracked__generator_exhausted_in_another_tracked_function__trace_tree_remains_correct(
+def test_track__generator_function_tracked__generator_exhausted_in_another_tracked_function__generator_span_started_and_ended_with_generator_exhausting(
fake_backend,
):
- @tracker.track(capture_output=True)
+ @tracker.track
def f_inner(z, generator):
for _ in generator:
pass
return "inner-output"
- @tracker.track(capture_output=True)
+ @tracker.track
def gen_f(y):
- values = ["yielded-1", "yielded-2", "yielded-3"]
+ values = ["yielded-1", " yielded-2", " yielded-3"]
for value in values:
yield value
- @tracker.track(capture_output=True)
+ @tracker.track
def f_outer(x):
generator = gen_f("generator-input")
f_inner("inner-input", generator)
@@ -726,13 +726,79 @@ def f_outer(x):
spans=[
SpanModel(
id=ANY_BUT_NONE,
- name="gen_f",
- input={"y": "generator-input"},
- output={"output": "['yielded-1', 'yielded-2', 'yielded-3']"},
+ name="f_inner",
+ input={"z": "inner-input", "generator": ANY_BUT_NONE},
+ output={"output": "inner-output"},
start_time=ANY_BUT_NONE,
end_time=ANY_BUT_NONE,
- spans=[],
+ spans=[
+ SpanModel(
+ id=ANY_BUT_NONE,
+ name="gen_f",
+ input={"y": "generator-input"},
+ output={"output": "yielded-1 yielded-2 yielded-3"},
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ spans=[],
+ ),
+ ],
),
+ ],
+ ),
+ ],
+ )
+
+ assert len(fake_backend.trace_trees) == 1
+
+ assert_equal(EXPECTED_TRACE_TREE, fake_backend.trace_trees[0])
+
+
+def test_track__generator_function_tracked__generator_exhausted_in_another_tracked_function__generator_span_started_and_ended_with_generator_exhausting__span_from_tracked_function_inside_generator_attached_to_generator_span(
+ fake_backend,
+):
+ @tracker.track
+ def f_inner(z, generator):
+ for _ in generator:
+ pass
+
+ return "inner-output"
+
+ @tracker.track
+ def f_called_inside_generator():
+ return "f-called-inside-generator-output"
+
+ @tracker.track
+ def gen_f(y):
+ f_called_inside_generator()
+ values = ["yielded-1", " yielded-2", " yielded-3"]
+ for value in values:
+ yield value
+
+ @tracker.track
+ def f_outer(x):
+ generator = gen_f("generator-input")
+ f_inner("inner-input", generator)
+ return "outer-output"
+
+ f_outer("outer-input")
+ tracker.flush_tracker()
+
+ EXPECTED_TRACE_TREE = TraceModel(
+ id=ANY_BUT_NONE,
+ name="f_outer",
+ input={"x": "outer-input"},
+ output={"output": "outer-output"},
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ spans=[
+ SpanModel(
+ id=ANY_BUT_NONE,
+ name="f_outer",
+ input={"x": "outer-input"},
+ output={"output": "outer-output"},
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ spans=[
SpanModel(
id=ANY_BUT_NONE,
name="f_inner",
@@ -740,7 +806,29 @@ def f_outer(x):
output={"output": "inner-output"},
start_time=ANY_BUT_NONE,
end_time=ANY_BUT_NONE,
- spans=[],
+ spans=[
+ SpanModel(
+ id=ANY_BUT_NONE,
+ name="gen_f",
+ input={"y": "generator-input"},
+ output={"output": "yielded-1 yielded-2 yielded-3"},
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ spans=[
+ SpanModel(
+ id=ANY_BUT_NONE,
+ name="f_called_inside_generator",
+ input={},
+ output={
+ "output": "f-called-inside-generator-output"
+ },
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ spans=[],
+ ),
+ ],
+ ),
+ ],
),
],
),
@@ -755,7 +843,7 @@ def f_outer(x):
def test_track__single_async_function_tracked__happyflow(
fake_backend,
):
- @tracker.track(capture_output=True)
+ @tracker.track
async def async_f(x):
await asyncio.sleep(0.01)
return "the-output"
@@ -792,12 +880,12 @@ async def async_f(x):
def test_track__nested_async_function_tracked__happyflow(
fake_backend,
):
- @tracker.track(capture_output=True)
+ @tracker.track
async def async_f_inner(y):
await asyncio.sleep(0.01)
return "inner-output"
- @tracker.track(capture_output=True)
+ @tracker.track
async def async_f_outer(x):
await async_f_inner("inner-input")
return "outer-output"
@@ -841,18 +929,19 @@ async def async_f_outer(x):
assert_equal(EXPECTED_TRACE_TREE, fake_backend.trace_trees[0])
-def test_track__single_async_generator_function_tracked__generator_exhausted__happyflow(
+def test_track__top_level_single_async_generator_function_tracked__generator_exhausted__happyflow(
fake_backend,
):
- @tracker.track(capture_output=True)
+ @tracker.track
async def async_generator(x):
await asyncio.sleep(0.01)
- for item in ["yielded-1", "yielded-2", "yielded-3"]:
+ for item in ["yielded-1", " yielded-2", " yielded-3"]:
yield item
async def async_generator_user():
- async for _ in async_generator("generator-input"):
+ gen = async_generator("generator-input")
+ async for _ in gen:
pass
asyncio.run(async_generator_user())
@@ -863,7 +952,7 @@ async def async_generator_user():
id=ANY_BUT_NONE,
name="async_generator",
input={"x": "generator-input"},
- output={"output": "['yielded-1', 'yielded-2', 'yielded-3']"},
+ output={"output": "yielded-1 yielded-2 yielded-3"},
start_time=ANY_BUT_NONE,
end_time=ANY_BUT_NONE,
spans=[
@@ -871,7 +960,7 @@ async def async_generator_user():
id=ANY_BUT_NONE,
name="async_generator",
input={"x": "generator-input"},
- output={"output": "['yielded-1', 'yielded-2', 'yielded-3']"},
+ output={"output": "yielded-1 yielded-2 yielded-3"},
start_time=ANY_BUT_NONE,
end_time=ANY_BUT_NONE,
spans=[],
@@ -884,17 +973,74 @@ async def async_generator_user():
assert_equal(EXPECTED_TRACE_TREE, fake_backend.trace_trees[0])
+def test_track__top_level_async_generator_function_tracked__generator_has_another_tracked_function_inside__nested_function_attached_to_generator_span_and_trace(
+ fake_backend,
+):
+ @tracker.track
+ async def some_async_work():
+ await asyncio.sleep(0.001)
+
+ @tracker.track
+ async def async_generator(x):
+ await some_async_work()
+
+ for item in ["yielded-1", " yielded-2", " yielded-3"]:
+ yield item
+
+ async def async_generator_user():
+ gen = async_generator("generator-input")
+ async for _ in gen:
+ pass
+
+ asyncio.run(async_generator_user())
+
+ tracker.flush_tracker()
+
+ EXPECTED_TRACE_TREE = TraceModel(
+ id=ANY_BUT_NONE,
+ name="async_generator",
+ input={"x": "generator-input"},
+ output={"output": "yielded-1 yielded-2 yielded-3"},
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ spans=[
+ SpanModel(
+ id=ANY_BUT_NONE,
+ name="async_generator",
+ input={"x": "generator-input"},
+ output={"output": "yielded-1 yielded-2 yielded-3"},
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ spans=[
+ SpanModel(
+ id=ANY_BUT_NONE,
+ name="some_async_work",
+ input={},
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ spans=[],
+ )
+ ],
+ )
+ ],
+ )
+
+ assert len(fake_backend.trace_trees) == 1
+
+ assert_equal(EXPECTED_TRACE_TREE, fake_backend.trace_trees[0])
+
+
def test_track__async_generator_inside_another_tracked_function__happyflow(
fake_backend,
):
- @tracker.track(capture_output=True)
+ @tracker.track
async def async_generator(y):
await asyncio.sleep(0.01)
- for item in ["yielded-1", "yielded-2", "yielded-3"]:
+ for item in ["yielded-1", " yielded-2", " yielded-3"]:
yield item
- @tracker.track(capture_output=True)
+ @tracker.track
async def async_generator_user(x):
async for _ in async_generator("generator-input"):
pass
@@ -925,7 +1071,7 @@ async def async_generator_user(x):
id=ANY_BUT_NONE,
name="async_generator",
input={"y": "generator-input"},
- output={"output": "['yielded-1', 'yielded-2', 'yielded-3']"},
+ output={"output": "yielded-1 yielded-2 yielded-3"},
start_time=ANY_BUT_NONE,
end_time=ANY_BUT_NONE,
spans=[],
@@ -940,10 +1086,80 @@ async def async_generator_user(x):
assert_equal(EXPECTED_TRACE_TREE, fake_backend.trace_trees[0])
+def test_track__async_generator_inside_another_tracked_function__another_tracked_function_called_inside_generator_and_attached_to_its_span(
+ fake_backend,
+):
+ @tracker.track
+ async def some_async_work():
+ await asyncio.sleep(0.001)
+
+ @tracker.track
+ async def async_generator(y):
+ await some_async_work()
+
+ for item in ["yielded-1", " yielded-2", " yielded-3"]:
+ yield item
+
+ @tracker.track
+ async def async_generator_user(x):
+ async for _ in async_generator("generator-input"):
+ pass
+
+ return "generator-user-output"
+
+ asyncio.run(async_generator_user("generator-user-input"))
+
+ tracker.flush_tracker()
+
+ EXPECTED_TRACE_TREE = TraceModel(
+ id=ANY_BUT_NONE,
+ name="async_generator_user",
+ input={"x": "generator-user-input"},
+ output={"output": "generator-user-output"},
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ spans=[
+ SpanModel(
+ id=ANY_BUT_NONE,
+ name="async_generator_user",
+ input={"x": "generator-user-input"},
+ output={"output": "generator-user-output"},
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ spans=[
+ SpanModel(
+ id=ANY_BUT_NONE,
+ name="async_generator",
+ input={"y": "generator-input"},
+ output={"output": "yielded-1 yielded-2 yielded-3"},
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ spans=[
+ SpanModel(
+ id=ANY_BUT_NONE,
+ name="some_async_work",
+ input={},
+ output=None,
+ start_time=ANY_BUT_NONE,
+ end_time=ANY_BUT_NONE,
+ spans=[],
+ )
+ ],
+ )
+ ],
+ )
+ ],
+ )
+
+ assert len(fake_backend.trace_trees) == 1
+
+ assert_equal(EXPECTED_TRACE_TREE, fake_backend.trace_trees[0])
+
+
def test_track__distributed_tracing_with_headers__tracing_is_performed_in_2_threads__all_data_is_saved_in_1_trace_tree(
fake_backend,
):
- @tracker.track(capture_output=True)
+ @tracker.track
def f_remote(y, thread_id):
return f"f-remote-output-from-{thread_id}"
@@ -951,7 +1167,7 @@ def distributed_node_runner(y, thread_id, opik_headers):
f_remote(y, thread_id, opik_distributed_trace_headers=opik_headers)
return "result-from-node-runner"
- @tracker.track(capture_output=True)
+ @tracker.track
def f_outer(x):
distributed_trace_headers = opik_context.get_distributed_trace_headers()
t1 = threading.Thread(
@@ -1004,7 +1220,7 @@ def f_outer(x):
def test_track__trace_already_created_not_by_decorator__decorator_just_attaches_new_span_to_it__trace_is_not_popped_from_context_in_the_end(
fake_backend,
):
- @tracker.track(capture_output=True)
+ @tracker.track
def f(x):
return "f-output"
diff --git a/sdks/python/tests/unit/validation/test_usage_validator.py b/sdks/python/tests/unit/validation/test_usage_validator.py
index 91e509c127..48dfc1b2f0 100644
--- a/sdks/python/tests/unit/validation/test_usage_validator.py
+++ b/sdks/python/tests/unit/validation/test_usage_validator.py
@@ -53,7 +53,7 @@
],
)
def test_usage_validator(usage_dict, is_valid):
- tested = usage.UsageValidator(usage_dict)
+ tested = usage.UsageValidator(usage_dict, provider="some-provider")
assert tested.validate().ok() is is_valid, f"Failed with {usage_dict}"
diff --git a/sdks/typescript/package-lock.json b/sdks/typescript/package-lock.json
index 0e4bbff002..f2876eb78c 100644
--- a/sdks/typescript/package-lock.json
+++ b/sdks/typescript/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "opik",
- "version": "0.1.1",
+ "version": "0.2.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "opik",
- "version": "0.1.1",
+ "version": "0.2.0",
"license": "Apache-2.0",
"dependencies": {
"node-fetch": "^3.3.2",
diff --git a/sdks/typescript/package.json b/sdks/typescript/package.json
index a32deeed9f..97412740b6 100644
--- a/sdks/typescript/package.json
+++ b/sdks/typescript/package.json
@@ -1,7 +1,7 @@
{
"name": "opik",
"description": "Opik TypeScript and JavaScript SDK",
- "version": "0.1.1",
+ "version": "0.2.0",
"repository": {
"type": "git",
"url": "git+https://github.com/comet-ml/opik.git",
diff --git a/tests_end_to_end/page_objects/ExperimentItemsPage.py b/tests_end_to_end/page_objects/ExperimentItemsPage.py
index 6c3ae84bf7..9f63893e4f 100644
--- a/tests_end_to_end/page_objects/ExperimentItemsPage.py
+++ b/tests_end_to_end/page_objects/ExperimentItemsPage.py
@@ -25,7 +25,7 @@ def get_total_number_of_items_in_experiment(self):
def get_id_of_nth_experiment_item(self, n: int):
row = self.page.locator("tr").nth(n + 1)
- cell = row.locator("td").first
+ cell = row.locator("td").nth(1)
cell.hover()
cell.get_by_role("button").nth(1).click()
id = self.page.evaluate("navigator.clipboard.readText()")
@@ -35,7 +35,7 @@ def get_all_item_ids_on_current_page(self):
ids = []
rows = self.page.locator("tr").all()
for row_index, row in enumerate(rows[2:]):
- cell = row.locator("td").first
+ cell = row.locator("td").nth(1)
cell.hover()
cell.get_by_role("button").nth(1).click()
id = self.page.evaluate("navigator.clipboard.readText()")
diff --git a/tests_end_to_end/page_objects/ProjectsPage.py b/tests_end_to_end/page_objects/ProjectsPage.py
index ee1dd44df4..0475953067 100644
--- a/tests_end_to_end/page_objects/ProjectsPage.py
+++ b/tests_end_to_end/page_objects/ProjectsPage.py
@@ -18,7 +18,7 @@ def search_project(self, project_name):
def check_project_exists_on_current_page(self, project_name):
expect(
self.page.get_by_role("cell", name=project_name, exact=True)
- ).to_be_visible()
+ ).to_be_visible(timeout=3)
def check_project_not_exists_on_current_page(self, project_name):
expect(
@@ -39,7 +39,7 @@ def check_project_exists_on_current_page_with_retry(self, project_name, timeout)
)
def create_new_project(self, project_name):
- self.page.get_by_role("button", name="Create new project").click()
+ self.page.get_by_role("button", name="Create new project").first.click()
project_name_fill_box = self.page.get_by_placeholder("Project name")
project_name_fill_box.click()
@@ -57,3 +57,4 @@ def delete_project_by_name(self, project_name):
row.get_by_role("button").click()
self.page.get_by_role("menuitem", name="Delete").click()
self.page.get_by_role("button", name="Delete project").click()
+ self.check_project_not_exists_on_current_page(project_name)
diff --git a/tests_end_to_end/page_objects/PromptPage.py b/tests_end_to_end/page_objects/PromptPage.py
index af6c5e45bd..a0aadf78a0 100644
--- a/tests_end_to_end/page_objects/PromptPage.py
+++ b/tests_end_to_end/page_objects/PromptPage.py
@@ -31,7 +31,7 @@ def get_all_prompt_versions_with_commit_ids_on_page(self):
rows: list[Locator] = self.page.get_by_role("row").all()[1:]
versions = {}
for row in rows:
- prompt = row.get_by_role("cell").nth(1).inner_text()
+ prompt = row.get_by_role("cell").nth(2).inner_text()
commit_id = row.get_by_role("link").first.inner_text()
versions[prompt] = commit_id
diff --git a/tests_end_to_end/test_requirements.txt b/tests_end_to_end/test_requirements.txt
index 9c5d36765a..c4e25bf0d4 100644
--- a/tests_end_to_end/test_requirements.txt
+++ b/tests_end_to_end/test_requirements.txt
@@ -1,3 +1,3 @@
pytest
-playwright
+playwright==1.49.0
pytest-playwright
diff --git a/tests_end_to_end/tests/Datasets/conftest.py b/tests_end_to_end/tests/Datasets/conftest.py
index 280fe2ef3d..9425946e72 100644
--- a/tests_end_to_end/tests/Datasets/conftest.py
+++ b/tests_end_to_end/tests/Datasets/conftest.py
@@ -4,6 +4,6 @@
@pytest.fixture
-def insert_dataset_items_sdk(client: Opik, create_delete_dataset_sdk):
- dataset = client.get_dataset(create_delete_dataset_sdk)
+def insert_dataset_items_sdk(client: Opik, create_dataset_sdk):
+ dataset = client.get_dataset(create_dataset_sdk)
dataset.insert(TEST_ITEMS)
diff --git a/tests_end_to_end/tests/Datasets/test_dataset_items_crud_operations.py b/tests_end_to_end/tests/Datasets/test_dataset_items_crud_operations.py
index 9cd3d383f5..72e7f47493 100644
--- a/tests_end_to_end/tests/Datasets/test_dataset_items_crud_operations.py
+++ b/tests_end_to_end/tests/Datasets/test_dataset_items_crud_operations.py
@@ -23,7 +23,7 @@ class TestDatasetItemsCrud:
) # add insert_via_ui once flakiness is figured out
@pytest.mark.parametrize(
"dataset_creation_fixture",
- ["create_delete_dataset_sdk", "create_delete_dataset_ui"],
+ ["create_dataset_sdk", "create_dataset_ui"],
)
@pytest.mark.browser_context_args(permissions=["clipboard-read"])
@pytest.mark.sanity
@@ -82,7 +82,7 @@ def test_dataset_item_update(
request,
page: Page,
client: opik.Opik,
- create_delete_dataset_sdk,
+ create_dataset_sdk,
insert_dataset_items_sdk,
):
"""
@@ -93,7 +93,7 @@ def test_dataset_item_update(
4. Check that the new data is correct on both the UI and the SDK
"""
dataset = wait_for_dataset_to_be_visible(
- client=client, dataset_name=create_delete_dataset_sdk, timeout=10
+ client=client, dataset_name=create_dataset_sdk, timeout=10
)
wait_for_number_of_items_in_dataset(
@@ -127,7 +127,7 @@ def test_dataset_item_deletion(
request,
page: Page,
client: opik.Opik,
- create_delete_dataset_sdk,
+ create_dataset_sdk,
insert_dataset_items_sdk,
item_deletion,
):
@@ -139,7 +139,7 @@ def test_dataset_item_deletion(
4. Check that the item with that data no longer exists in both the SDK and the UI and that the length of the item list is updated
"""
dataset = wait_for_dataset_to_be_visible(
- client=client, dataset_name=create_delete_dataset_sdk, timeout=10
+ client=client, dataset_name=create_dataset_sdk, timeout=10
)
item_deleted = {}
@@ -186,7 +186,7 @@ def test_dataset_clear(
request,
page: Page,
client: opik.Opik,
- create_delete_dataset_sdk,
+ create_dataset_sdk,
insert_dataset_items_sdk,
):
"""
@@ -197,7 +197,7 @@ def test_dataset_clear(
4. Check that no items exist in the dataset when trying to get them via both the SDK and the UI
"""
dataset = wait_for_dataset_to_be_visible(
- client=client, dataset_name=create_delete_dataset_sdk, timeout=10
+ client=client, dataset_name=create_dataset_sdk, timeout=10
)
dataset.clear()
diff --git a/tests_end_to_end/tests/Datasets/test_datasets_crud_operations.py b/tests_end_to_end/tests/Datasets/test_datasets_crud_operations.py
index 6405b6ccd1..d17c009718 100644
--- a/tests_end_to_end/tests/Datasets/test_datasets_crud_operations.py
+++ b/tests_end_to_end/tests/Datasets/test_datasets_crud_operations.py
@@ -14,7 +14,7 @@
class TestDatasetsCrud:
def test_create_dataset_ui_add_traces_to_new_dataset(
- self, page: Page, create_delete_project_sdk, create_10_test_traces
+ self, page: Page, create_project, create_10_test_traces
):
"""
Basic test to check dataset creation via "add to new dataset" functionality in the traces page. Uses the UI after creation to check the project exists
@@ -24,7 +24,7 @@ def test_create_dataset_ui_add_traces_to_new_dataset(
4. If no errors raised and dataset exists, test passes
"""
dataset_name = "automated_tests_dataset"
- proj_name = create_delete_project_sdk
+ proj_name = create_project
projects_page = ProjectsPage(page)
projects_page.go_to_page()
projects_page.click_project(project_name=proj_name)
@@ -45,7 +45,7 @@ def test_create_dataset_ui_add_traces_to_new_dataset(
delete_dataset_by_name_if_exists(dataset_name=dataset_name)
@pytest.mark.parametrize(
- "dataset_fixture", ["create_delete_dataset_ui", "create_delete_dataset_sdk"]
+ "dataset_fixture", ["create_dataset_ui", "create_dataset_sdk"]
)
@pytest.mark.sanity
def test_dataset_visibility(
@@ -69,7 +69,7 @@ def test_dataset_visibility(
@pytest.mark.parametrize(
"dataset_fixture",
- ["create_dataset_sdk_no_cleanup", "create_dataset_ui_no_cleanup"],
+ ["create_dataset_sdk", "create_dataset_ui"],
)
def test_dataset_name_update(
self, request, page: Page, client: opik.Opik, dataset_fixture
@@ -114,7 +114,7 @@ def test_dataset_name_update(
@pytest.mark.parametrize(
"dataset_fixture",
- ["create_dataset_sdk_no_cleanup", "create_dataset_ui_no_cleanup"],
+ ["create_dataset_sdk", "create_dataset_ui"],
)
def test_dataset_deletion_in_sdk(
self, request, page: Page, client: opik.Opik, dataset_fixture
@@ -142,7 +142,7 @@ def test_dataset_deletion_in_sdk(
@pytest.mark.parametrize(
"dataset_fixture",
- ["create_dataset_sdk_no_cleanup", "create_dataset_ui_no_cleanup"],
+ ["create_dataset_sdk", "create_dataset_ui"],
)
def test_dataset_deletion_in_ui(
self, request, page: Page, client: opik.Opik, dataset_fixture
diff --git a/tests_end_to_end/tests/Experiments/conftest.py b/tests_end_to_end/tests/Experiments/conftest.py
index 5356699b42..c7804ceab1 100644
--- a/tests_end_to_end/tests/Experiments/conftest.py
+++ b/tests_end_to_end/tests/Experiments/conftest.py
@@ -10,8 +10,8 @@ def eval_task(item: dict):
@pytest.fixture()
-def mock_experiment(client: Opik, create_delete_dataset_sdk, insert_dataset_items_sdk):
- dataset = client.get_dataset(create_delete_dataset_sdk)
+def mock_experiment(client: Opik, create_dataset_sdk, insert_dataset_items_sdk):
+ dataset = client.get_dataset(create_dataset_sdk)
experiment_name = "test_experiment"
eval = evaluate(
experiment_name=experiment_name,
diff --git a/tests_end_to_end/tests/Projects/test_projects_crud_operations.py b/tests_end_to_end/tests/Projects/test_projects_crud_operations.py
index b109f318e4..9f4ad7ac15 100644
--- a/tests_end_to_end/tests/Projects/test_projects_crud_operations.py
+++ b/tests_end_to_end/tests/Projects/test_projects_crud_operations.py
@@ -10,9 +10,7 @@
class TestProjectsCrud:
- @pytest.mark.parametrize(
- "project_fixture", ["create_delete_project_ui", "create_delete_project_sdk"]
- )
+ @pytest.mark.parametrize("project_fixture", ["create_project", "create_project_ui"])
@pytest.mark.sanity
def test_project_visibility(self, request, page: Page, project_fixture):
"""
@@ -37,7 +35,7 @@ def test_project_visibility(self, request, page: Page, project_fixture):
@pytest.mark.parametrize(
"project_fixture",
- ["create_project_sdk_no_cleanup", "create_project_ui_no_cleanup"],
+ ["create_project", "create_project_ui"],
)
def test_project_name_update(self, request, page: Page, project_fixture):
"""
@@ -74,7 +72,7 @@ def test_project_name_update(self, request, page: Page, project_fixture):
)
except Exception as e:
- print(f"Error occured during update of project name: {e}")
+ print(f"Error occurred during update of project name: {e}")
raise
finally:
@@ -85,7 +83,7 @@ def test_project_name_update(self, request, page: Page, project_fixture):
@pytest.mark.parametrize(
"project_fixture",
- ["create_project_sdk_no_cleanup", "create_project_ui_no_cleanup"],
+ ["create_project", "create_project_ui"],
)
def test_project_deletion_in_sdk(self, request, page: Page, project_fixture):
"""
@@ -108,7 +106,7 @@ def test_project_deletion_in_sdk(self, request, page: Page, project_fixture):
@pytest.mark.parametrize(
"project_fixture",
- ["create_project_sdk_no_cleanup", "create_project_ui_no_cleanup"],
+ ["create_project", "create_project_ui"],
)
def test_project_deletion_in_ui(self, request, page: Page, project_fixture):
"""
diff --git a/tests_end_to_end/tests/Traces/test_trace_spans.py b/tests_end_to_end/tests/Traces/test_trace_spans.py
index aacf58225a..4e78d885ce 100644
--- a/tests_end_to_end/tests/Traces/test_trace_spans.py
+++ b/tests_end_to_end/tests/Traces/test_trace_spans.py
@@ -11,16 +11,14 @@ class TestTraceSpans:
["log_traces_with_spans_low_level", "log_traces_with_spans_decorator"],
)
@pytest.mark.sanity
- def test_spans_of_traces(
- self, page, request, create_delete_project_sdk, traces_fixture
- ):
+ def test_spans_of_traces(self, page, request, create_project, traces_fixture):
"""
Checks that every trace has the correct number and names of spans defined in the sanity_config.yaml file
1. Open the traces page of the project
2. Go through each trace and click it
3. Check that the spans are present in each trace
"""
- project_name = create_delete_project_sdk
+ project_name = create_project
_, span_config = request.getfixturevalue(traces_fixture)
projects_page = ProjectsPage(page)
projects_page.go_to_page()
@@ -42,7 +40,7 @@ def test_spans_of_traces(
)
@pytest.mark.sanity
def test_trace_and_span_details(
- self, page, request, create_delete_project_sdk, traces_fixture
+ self, page, request, create_project, traces_fixture
):
"""
Checks that for each trace and spans, the attributes defined in sanity_config.yaml are present
@@ -52,7 +50,7 @@ def test_trace_and_span_details(
4. Check the defined metadata is present
5. Go through each span of the traces and repeat 2-4
"""
- project_name = create_delete_project_sdk
+ project_name = create_project
trace_config, span_config = request.getfixturevalue(traces_fixture)
projects_page = ProjectsPage(page)
projects_page.go_to_page()
diff --git a/tests_end_to_end/tests/Traces/test_traces_crud_operations.py b/tests_end_to_end/tests/Traces/test_traces_crud_operations.py
index 03f51a2542..e14739e879 100644
--- a/tests_end_to_end/tests/Traces/test_traces_crud_operations.py
+++ b/tests_end_to_end/tests/Traces/test_traces_crud_operations.py
@@ -23,7 +23,7 @@ class TestTracesCrud:
)
@pytest.mark.sanity
def test_trace_creation(
- self, page: Page, traces_number, create_delete_project_sdk, create_traces
+ self, page: Page, traces_number, create_project, create_traces
):
"""Testing basic creation of traces via both decorator and low-level client.
Test case is split into 4, creating 1 and then 15 traces using both the decorator and the client respectively
@@ -32,7 +32,7 @@ def test_trace_creation(
2. Create the traces using one of the creation methods, following the naming convention of "test-trace-X", where X is from 1 to 25 (so all have unique names) - no errors should occur
3. In the UI, check that the presented number of traces in the project matches the number of traces created in the test case
"""
- project_name = create_delete_project_sdk
+ project_name = create_project
projects_page = ProjectsPage(page)
projects_page.go_to_page()
projects_page.click_project(project_name)
@@ -52,7 +52,7 @@ def test_trace_creation(
indirect=True,
)
def test_traces_visibility(
- self, page: Page, traces_number, create_delete_project_sdk, create_traces
+ self, page: Page, traces_number, create_project, create_traces
):
"""
Testing visibility within the UI and SDK of traces created via both the decorator and the client
@@ -67,7 +67,7 @@ def test_traces_visibility(
- Fetch all traces of the project via the API client (OpikApi.traces.get_traces_by_project)
- Check that the list of names present in the result is exactly equal to the list of names of the traces created (exactly the same elements on both sides)
"""
- project_name = create_delete_project_sdk
+ project_name = create_project
projects_page = ProjectsPage(page)
projects_page.go_to_page()
projects_page.click_project(project_name)
@@ -94,7 +94,7 @@ def test_traces_visibility(
indirect=True,
)
def test_delete_traces_sdk(
- self, page: Page, traces_number, create_delete_project_sdk, create_traces
+ self, page: Page, traces_number, create_project, create_traces
):
"""
Testing trace deletion via the SDK API client (v1/private/traces/delete endpoint)
@@ -106,7 +106,7 @@ def test_delete_traces_sdk(
4. Check in the UI that the deleted traces are no longer present in the project page
5. Check in the SDK that the deleted traces are no longer present in the fetch request (v1/private/traces endpoint, with project_name parameter)
"""
- project_name = create_delete_project_sdk
+ project_name = create_project
_ = create_traces
wait_for_traces_to_be_visible(project_name=project_name, size=traces_number)
@@ -148,7 +148,7 @@ def test_delete_traces_sdk(
indirect=True,
)
def test_delete_traces_ui(
- self, page: Page, traces_number, create_delete_project_sdk, create_traces
+ self, page: Page, traces_number, create_project, create_traces
):
"""
Testing trace deletion via the UI
@@ -160,7 +160,7 @@ def test_delete_traces_ui(
4. Check in the UI that the deleted traces are no longer present in the project page
5. Check in the SDK that the deleted traces are no longer present in the fetch request (v1/private/traces endpoint, with project_name parameter)
"""
- project_name = create_delete_project_sdk
+ project_name = create_project
projects_page = ProjectsPage(page)
projects_page.go_to_page()
projects_page.click_project(project_name)
diff --git a/tests_end_to_end/tests/conftest.py b/tests_end_to_end/tests/conftest.py
index 8eb539ad42..1e7f753351 100644
--- a/tests_end_to_end/tests/conftest.py
+++ b/tests_end_to_end/tests/conftest.py
@@ -3,7 +3,7 @@
import opik
from opik import track, opik_context
from tests.config import EnvConfig, get_environment_config
-from playwright.sync_api import Page, Browser
+from playwright.sync_api import Page, Browser, Playwright
from page_objects.ProjectsPage import ProjectsPage
from page_objects.TracesPage import TracesPage
from page_objects.DatasetsPage import DatasetsPage
@@ -11,13 +11,14 @@
from page_objects.PromptLibraryPage import PromptLibraryPage
from page_objects.FeedbackDefinitionsPage import FeedbackDefinitionsPage
from tests.sdk_helpers import (
- create_project_sdk,
+ create_project_api,
delete_project_by_name_sdk,
wait_for_number_of_traces_to_be_visible,
client_get_prompt_retries,
+ find_project_by_name_sdk,
+ wait_for_project_to_not_be_visible,
)
from utils import TEST_ITEMS
-import time
import re
import json
@@ -26,6 +27,17 @@ def pytest_configure(config):
config.addinivalue_line("markers", "sanity: mark test as a sanity test")
+@pytest.fixture(scope="session")
+def browser(playwright: Playwright, request) -> Browser:
+ browser_name = request.config.getoption("--browser", default="chromium")
+ if isinstance(browser_name, list):
+ browser_name = browser_name[0]
+ browser_type = getattr(playwright, browser_name)
+ browser = browser_type.launch(slow_mo=200)
+ yield browser
+ browser.close()
+
+
@pytest.fixture(scope="session")
def browser_context(browser: Browser, env_config: EnvConfig):
"""Create a browser context with required permissions and authentication"""
@@ -147,22 +159,50 @@ def experiments_page(page):
@pytest.fixture(scope="function")
-def create_project_sdk_no_cleanup():
- proj_name = "projects_crud_tests_sdk"
-
- create_project_sdk(name=proj_name)
+def create_project(page: Page):
+ """
+ Create a project via SDK and handle cleanup.
+ Checks if project exists before attempting deletion.
+ """
+ proj_name = os.environ["OPIK_PROJECT_NAME"]
+ if find_project_by_name_sdk(proj_name):
+ delete_project_by_name_sdk(proj_name)
+ create_project_api(name=proj_name)
yield proj_name
+ projects_page = ProjectsPage(page)
+ projects_page.go_to_page()
+ try:
+ projects_page.search_project(proj_name)
+ projects_page.check_project_not_exists_on_current_page(project_name=proj_name)
+ except AssertionError as _:
+ projects_page.delete_project_by_name(proj_name)
+ wait_for_project_to_not_be_visible(proj_name)
+
@pytest.fixture(scope="function")
-def create_project_ui_no_cleanup(page: Page):
- proj_name = "projects_crud_tests_ui"
+def create_project_ui(page: Page):
+ """
+ Create a project via UI and handle cleanup.
+ Checks if project exists before attempting deletion.
+ """
+ proj_name = os.environ["OPIK_PROJECT_NAME"]
+ if find_project_by_name_sdk(proj_name):
+ delete_project_by_name_sdk(proj_name)
projects_page = ProjectsPage(page)
projects_page.go_to_page()
projects_page.create_new_project(project_name=proj_name)
yield proj_name
+ projects_page.go_to_page()
+ try:
+ projects_page.search_project(proj_name)
+ projects_page.check_project_not_exists_on_current_page(project_name=proj_name)
+ except AssertionError as _:
+ projects_page.delete_project_by_name(proj_name)
+ wait_for_project_to_not_be_visible(proj_name)
+
@pytest.fixture(scope="function")
def log_traces_with_spans_low_level(client: opik.Opik):
@@ -284,43 +324,30 @@ def make_trace(x):
@pytest.fixture(scope="function")
-def create_delete_project_sdk():
- proj_name = "automated_tests_project"
- os.environ["OPIK_PROJECT_NAME"] = proj_name
- time.sleep(1)
- create_project_sdk(name=proj_name)
- yield proj_name
- delete_project_by_name_sdk(name=proj_name)
-
-
-@pytest.fixture
-def create_delete_project_ui(page: Page):
- proj_name = "automated_tests_project"
- projects_page = ProjectsPage(page)
- projects_page.go_to_page()
- projects_page.create_new_project(project_name=proj_name)
-
- yield proj_name
- delete_project_by_name_sdk(name=proj_name)
-
-
-@pytest.fixture(scope="function")
-def create_delete_dataset_sdk(client: opik.Opik):
+def create_dataset_sdk(client: opik.Opik):
dataset_name = "automated_tests_dataset"
client.create_dataset(name=dataset_name)
yield dataset_name
- client.delete_dataset(name=dataset_name)
+ try:
+ client.get_dataset(name=dataset_name)
+ client.delete_dataset(name=dataset_name)
+ except Exception as _:
+ pass
@pytest.fixture(scope="function")
-def create_delete_dataset_ui(page: Page, client: opik.Opik):
+def create_dataset_ui(page: Page, client: opik.Opik):
dataset_name = "automated_tests_dataset"
datasets_page = DatasetsPage(page)
datasets_page.go_to_page()
datasets_page.create_dataset_by_name(dataset_name=dataset_name)
yield dataset_name
- client.delete_dataset(name=dataset_name)
+ try:
+ client.get_dataset(name=dataset_name)
+ client.delete_dataset(name=dataset_name)
+ except Exception as _:
+ pass
@pytest.fixture(scope="function")
@@ -340,8 +367,8 @@ def create_dataset_ui_no_cleanup(page: Page):
@pytest.fixture
-def insert_dataset_items_sdk(client: opik.Opik, create_delete_dataset_sdk):
- dataset = client.get_dataset(create_delete_dataset_sdk)
+def insert_dataset_items_sdk(client: opik.Opik, create_dataset_sdk):
+ dataset = client.get_dataset(create_dataset_sdk)
dataset.insert(TEST_ITEMS)
@@ -381,8 +408,8 @@ def create_prompt_ui(client: opik.Opik, page: Page):
@pytest.fixture
-def create_10_test_traces(page: Page, client, create_delete_project_sdk):
- proj_name = create_delete_project_sdk
+def create_10_test_traces(page: Page, client, create_project):
+ proj_name = create_project
for i in range(10):
_ = client.trace(
name=f"trace{i}",
diff --git a/tests_end_to_end/tests/sdk_helpers.py b/tests_end_to_end/tests/sdk_helpers.py
index e7569726ee..6396b0f568 100644
--- a/tests_end_to_end/tests/sdk_helpers.py
+++ b/tests_end_to_end/tests/sdk_helpers.py
@@ -4,21 +4,30 @@
from typing import Optional
import json
import opik
+import os
-def create_project_sdk(name: str):
- client = OpikApi()
+def get_opik_api_client():
+ return OpikApi(
+ base_url=os.getenv("OPIK_URL_OVERRIDE", None),
+ workspace_name=os.getenv("OPIK_WORKSPACE", None),
+ api_key=os.getenv("OPIK_API_KEY", None),
+ )
+
+
+def create_project_api(name: str):
+ client = get_opik_api_client()
client.projects.create_project(name=name)
def find_project_by_name_sdk(name: str):
- client = OpikApi()
+ client = get_opik_api_client()
proj_page = client.projects.find_projects(name=name, page=1, size=1)
return proj_page.dict()["content"]
def delete_project_by_name_sdk(name: str):
- client = OpikApi()
+ client = get_opik_api_client()
project = find_project_by_name_sdk(name=name)
client.projects.delete_project_by_id(project[0]["id"])
@@ -29,7 +38,7 @@ def wait_for_project_to_be_visible(project_name, timeout=10, initial_delay=1):
while time.time() - start_time < timeout:
if find_project_by_name_sdk(project_name):
- return True
+ return
time.sleep(delay)
delay = min(delay * 2, timeout - (time.time() - start_time))
@@ -39,8 +48,24 @@ def wait_for_project_to_be_visible(project_name, timeout=10, initial_delay=1):
)
+def wait_for_project_to_not_be_visible(project_name, timeout=10, initial_delay=1):
+ start_time = time.time()
+ delay = initial_delay
+
+ while time.time() - start_time < timeout:
+ if not find_project_by_name_sdk(project_name):
+ return
+
+ time.sleep(delay)
+ delay = min(delay * 2, timeout - (time.time() - start_time))
+
+ raise TimeoutError(
+ f"{project_name} has not been deleted via API within {timeout} seconds"
+ )
+
+
def update_project_by_name_sdk(name: str, new_name: str):
- client = OpikApi()
+ client = get_opik_api_client()
wait_for_project_to_be_visible(name, timeout=10)
projects_match = find_project_by_name_sdk(name)
project_id = projects_match[0]["id"]
@@ -51,7 +76,7 @@ def update_project_by_name_sdk(name: str, new_name: str):
def create_traces_sdk(prefix: str, project_name: str, qty: int):
- client = OpikApi()
+ client = get_opik_api_client()
for i in range(qty):
client.traces.create_trace(
name=prefix + str(i),
@@ -66,7 +91,7 @@ def wait_for_traces_to_be_visible(project_name, size, timeout=10, initial_delay=
while time.time() - start_time < timeout:
if get_traces_of_project_sdk(project_name=project_name, size=size):
- return True
+ return
time.sleep(delay)
delay = min(delay * 2, timeout - (time.time() - start_time))
@@ -87,7 +112,7 @@ def wait_for_number_of_traces_to_be_visible(
project_name=project_name, size=number_of_traces
)
if len(traces) >= number_of_traces:
- return True
+ return
time.sleep(delay)
delay = min(delay * 2, timeout - (time.time() - start_time))
@@ -98,31 +123,31 @@ def wait_for_number_of_traces_to_be_visible(
def get_traces_of_project_sdk(project_name: str, size: int):
- client = OpikApi()
+ client = get_opik_api_client()
traces = client.traces.get_traces_by_project(project_name=project_name, size=size)
return traces.dict()["content"]
def delete_list_of_traces_sdk(ids: list[str]):
- client = OpikApi()
+ client = get_opik_api_client()
client.traces.delete_traces(ids=ids)
def update_trace_by_id(id: str):
- client = OpikApi()
+ client = get_opik_api_client()
client.traces.update_trace(
id=id,
)
def get_dataset_by_name(dataset_name: str):
- client = OpikApi()
+ client = get_opik_api_client()
dataset = client.datasets.get_dataset_by_identifier(dataset_name=dataset_name)
return dataset.dict()
def update_dataset_name(name: str, new_name: str):
- client = OpikApi()
+ client = get_opik_api_client()
dataset = get_dataset_by_name(dataset_name=name)
dataset_id = dataset["id"]
@@ -132,7 +157,7 @@ def update_dataset_name(name: str, new_name: str):
def delete_dataset_by_name_if_exists(dataset_name: str):
- client = OpikApi()
+ client = get_opik_api_client()
dataset = None
try:
dataset = get_dataset_by_name(dataset_name)
@@ -144,23 +169,23 @@ def delete_dataset_by_name_if_exists(dataset_name: str):
def get_experiment_by_id(exp_id: str):
- client = OpikApi()
+ client = get_opik_api_client()
exp = client.experiments.get_experiment_by_id(exp_id)
return exp
def delete_experiment_by_id(exp_id: str):
- client = OpikApi()
+ client = get_opik_api_client()
client.experiments.delete_experiments_by_id(ids=[exp_id])
def delete_experiment_items_by_id(ids: list[str]):
- client = OpikApi()
+ client = get_opik_api_client()
client.experiments.delete_experiment_items(ids=ids)
def experiment_items_stream(exp_name: str, limit: Optional[int] = None):
- client = OpikApi()
+ client = get_opik_api_client()
data = b"".join(
client.experiments.stream_experiment_items(
experiment_name=exp_name, request_options={"chunk_size": 100}
diff --git a/version.txt b/version.txt
index acd81d7f2c..04e0d3f14f 100644
--- a/version.txt
+++ b/version.txt
@@ -1 +1 @@
-1.4.13
+1.4.17
{description}
)}
diff --git a/apps/opik-frontend/src/constants/chart.ts b/apps/opik-frontend/src/constants/chart.ts
index 9eea2cb2eb..b4e48aeacc 100644
--- a/apps/opik-frontend/src/constants/chart.ts
+++ b/apps/opik-frontend/src/constants/chart.ts
@@ -1,5 +1,5 @@
export const DEFAULT_CHART_TICK = {
- stroke: "#94A3B8",
+ stroke: "#373D4D",
fontWeight: 200,
letterSpacing: "0.05rem",
fontSize: "10px",
diff --git a/apps/opik-frontend/src/constants/llm.ts b/apps/opik-frontend/src/constants/llm.ts
index c218dc8990..2cb67307f9 100644
--- a/apps/opik-frontend/src/constants/llm.ts
+++ b/apps/opik-frontend/src/constants/llm.ts
@@ -1,4 +1,3 @@
-import { PROVIDER_MODEL_TYPE, PROVIDER_TYPE } from "@/types/providers";
import {
LLM_JUDGE,
LLM_MESSAGE_ROLE,
@@ -6,127 +5,6 @@ import {
LLM_SCHEMA_TYPE,
} from "@/types/llm";
-type PROVIDER_MODELS_TYPE = {
- [key in PROVIDER_TYPE]: {
- value: PROVIDER_MODEL_TYPE;
- label: string;
- structuredOutput?: boolean;
- }[];
-};
-
-export const PROVIDER_MODELS: PROVIDER_MODELS_TYPE = {
- [PROVIDER_TYPE.OPEN_AI]: [
- // GPT-4.0 Models
- {
- value: PROVIDER_MODEL_TYPE.GPT_4O,
- label: "GPT 4o",
- structuredOutput: true,
- },
- {
- value: PROVIDER_MODEL_TYPE.GPT_4O_MINI,
- label: "GPT 4o Mini",
- structuredOutput: true,
- },
- {
- value: PROVIDER_MODEL_TYPE.GPT_4O_MINI_2024_07_18,
- label: "GPT 4o Mini 2024-07-18",
- structuredOutput: true,
- },
- {
- value: PROVIDER_MODEL_TYPE.GPT_4O_2024_08_06,
- label: "GPT 4o 2024-08-06",
- structuredOutput: true,
- },
- {
- value: PROVIDER_MODEL_TYPE.GPT_4O_2024_05_13,
- label: "GPT 4o 2024-05-13",
- },
-
- // GPT-4 Models
- {
- value: PROVIDER_MODEL_TYPE.GPT_4_TURBO,
- label: "GPT 4 Turbo",
- },
- {
- value: PROVIDER_MODEL_TYPE.GPT_4,
- label: "GPT 4",
- },
- {
- value: PROVIDER_MODEL_TYPE.GPT_4_TURBO_PREVIEW,
- label: "GPT 4 Turbo Preview",
- },
- {
- value: PROVIDER_MODEL_TYPE.GPT_4_TURBO_2024_04_09,
- label: "GPT 4 Turbo 2024-04-09",
- },
- {
- value: PROVIDER_MODEL_TYPE.GPT_4_1106_PREVIEW,
- label: "GPT 4 1106 Preview",
- },
- {
- value: PROVIDER_MODEL_TYPE.GPT_4_0613,
- label: "GPT 4 0613",
- },
- {
- value: PROVIDER_MODEL_TYPE.GPT_4_0125_PREVIEW,
- label: "GPT 4 0125 Preview",
- },
-
- // GPT-3.5 Models
- {
- value: PROVIDER_MODEL_TYPE.GPT_3_5_TURBO,
- label: "GPT 3.5 Turbo",
- },
- {
- value: PROVIDER_MODEL_TYPE.GPT_3_5_TURBO_1106,
- label: "GPT 3.5 Turbo 1106",
- },
- {
- value: PROVIDER_MODEL_TYPE.GPT_3_5_TURBO_0125,
- label: "GPT 3.5 Turbo 0125",
- },
- ],
-
- [PROVIDER_TYPE.ANTHROPIC]: [
- {
- value: PROVIDER_MODEL_TYPE.CLAUDE_3_5_SONNET_20241022,
- label: "Claude 3.5 Sonnet 2024-10-22",
- },
- {
- value: PROVIDER_MODEL_TYPE.CLAUDE_3_5_HAIKU_20241022,
- label: "Claude 3.5 Haiku 2024-10-22",
- },
- {
- value: PROVIDER_MODEL_TYPE.CLAUDE_3_5_SONNET_20240620,
- label: "Claude 3.5 Sonnet 2024-06-20",
- },
- {
- value: PROVIDER_MODEL_TYPE.CLAUDE_3_OPUS_20240229,
- label: "Claude 3 Opus 2024-02-29",
- },
- {
- value: PROVIDER_MODEL_TYPE.CLAUDE_3_SONNET_20240229,
- label: "Claude 3 Sonnet 2024-02-29",
- },
- {
- value: PROVIDER_MODEL_TYPE.CLAUDE_3_HAIKU_20240307,
- label: "Claude 3 Haiku 2024-03-07",
- },
- {
- value: PROVIDER_MODEL_TYPE.CLAUDE_3_5_SONNET_LATEST,
- label: "Claude 3.5 Sonnet Latest",
- },
- {
- value: PROVIDER_MODEL_TYPE.CLAUDE_3_5_HAIKU_LATEST,
- label: "Claude 3.5 Haiku Latest",
- },
- {
- value: PROVIDER_MODEL_TYPE.CLAUDE_3_OPUS_LATEST,
- label: "Claude 3 Opus Latest",
- },
- ],
-};
-
export const DEFAULT_OPEN_AI_CONFIGS = {
TEMPERATURE: 0,
MAX_COMPLETION_TOKENS: 1024,
diff --git a/apps/opik-frontend/src/constants/providers.ts b/apps/opik-frontend/src/constants/providers.ts
index ec0ef44ca7..2c9fc78585 100644
--- a/apps/opik-frontend/src/constants/providers.ts
+++ b/apps/opik-frontend/src/constants/providers.ts
@@ -1,23 +1,42 @@
import OpenAIIcon from "@/icons/integrations/openai.svg?react";
import AnthropicIcon from "@/icons/integrations/anthropic.svg?react";
+import OllamaIcon from "@/icons/integrations/ollama.svg?react";
-import { PROVIDER_MODEL_TYPE, PROVIDER_TYPE } from "@/types/providers";
+import {
+ PROVIDER_LOCATION_TYPE,
+ PROVIDER_MODEL_TYPE,
+ PROVIDER_TYPE,
+} from "@/types/providers";
type IconType = typeof OpenAIIcon;
-export type PROVIDER_OPTION_TYPE = {
+type PROVIDER_OPTION_TYPE = {
label: string;
value: PROVIDER_TYPE;
icon: IconType;
apiKeyName: string;
+ defaultModel: PROVIDER_MODEL_TYPE | "";
+ description?: string;
+};
+
+export type CLOUD_PROVIDER_OPTION_TYPE = PROVIDER_OPTION_TYPE & {
+ locationType: PROVIDER_LOCATION_TYPE.cloud;
apiKeyURL: string;
- defaultModel: PROVIDER_MODEL_TYPE;
+};
+
+export type LOCAL_PROVIDER_OPTION_TYPE = PROVIDER_OPTION_TYPE & {
+ locationType: PROVIDER_LOCATION_TYPE.local;
+ lsKey: string;
};
type PROVIDERS_TYPE = {
- [key in PROVIDER_TYPE]: PROVIDER_OPTION_TYPE;
+ [key in PROVIDER_TYPE]:
+ | CLOUD_PROVIDER_OPTION_TYPE
+ | LOCAL_PROVIDER_OPTION_TYPE;
};
+export const OLLAMA_LS_KEY = "provider_ollama";
+
export const PROVIDERS: PROVIDERS_TYPE = {
[PROVIDER_TYPE.OPEN_AI]: {
label: "OpenAI",
@@ -26,6 +45,7 @@ export const PROVIDERS: PROVIDERS_TYPE = {
apiKeyName: "OPENAI_API_KEY",
apiKeyURL: "https://platform.openai.com/account/api-keys",
defaultModel: PROVIDER_MODEL_TYPE.GPT_4O,
+ locationType: PROVIDER_LOCATION_TYPE.cloud,
},
[PROVIDER_TYPE.ANTHROPIC]: {
label: "Anthropic",
@@ -34,6 +54,18 @@ export const PROVIDERS: PROVIDERS_TYPE = {
apiKeyName: "ANTHROPIC_API_KEY",
apiKeyURL: "https://console.anthropic.com/settings/keys",
defaultModel: PROVIDER_MODEL_TYPE.CLAUDE_3_5_SONNET_LATEST,
+ locationType: PROVIDER_LOCATION_TYPE.cloud,
+ },
+ [PROVIDER_TYPE.OLLAMA]: {
+ label: "Ollama (Experimental)",
+ value: PROVIDER_TYPE.OLLAMA,
+ icon: OllamaIcon,
+ apiKeyName: "OLLAMA_LOCAL_EXPERIMENTAL",
+ description:
+ "All configuration for this provider is saved locally, and will not \nbe accessible in different browsers",
+ locationType: PROVIDER_LOCATION_TYPE.local,
+ lsKey: OLLAMA_LS_KEY,
+ defaultModel: "",
},
};
diff --git a/apps/opik-frontend/src/hooks/useLLMProviderModelsData.ts b/apps/opik-frontend/src/hooks/useLLMProviderModelsData.ts
new file mode 100644
index 0000000000..e227269784
--- /dev/null
+++ b/apps/opik-frontend/src/hooks/useLLMProviderModelsData.ts
@@ -0,0 +1,216 @@
+import { useCallback } from "react";
+import first from "lodash/first";
+import {
+ PROVIDER_LOCATION_TYPE,
+ PROVIDER_MODEL_TYPE,
+ PROVIDER_MODELS_TYPE,
+ PROVIDER_TYPE,
+} from "@/types/providers";
+import useLocalAIProviderData from "@/hooks/useLocalAIProviderData";
+import { getDefaultProviderKey } from "@/lib/provider";
+import { PROVIDERS } from "@/constants/providers";
+
+export type ProviderResolver = (
+ modelName?: PROVIDER_MODEL_TYPE | "",
+) => PROVIDER_TYPE | "";
+
+export type ModelResolver = (
+ lastPickedModel: PROVIDER_MODEL_TYPE | "",
+ setupProviders: PROVIDER_TYPE[],
+ preferredProvider?: PROVIDER_TYPE | "",
+) => PROVIDER_MODEL_TYPE | "";
+
+export const PROVIDER_MODELS: PROVIDER_MODELS_TYPE = {
+ [PROVIDER_TYPE.OPEN_AI]: [
+ // GPT-4.0 Models
+ {
+ value: PROVIDER_MODEL_TYPE.GPT_4O,
+ label: "GPT 4o",
+ structuredOutput: true,
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.GPT_4O_MINI,
+ label: "GPT 4o Mini",
+ structuredOutput: true,
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.GPT_4O_MINI_2024_07_18,
+ label: "GPT 4o Mini 2024-07-18",
+ structuredOutput: true,
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.GPT_4O_2024_08_06,
+ label: "GPT 4o 2024-08-06",
+ structuredOutput: true,
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.GPT_4O_2024_05_13,
+ label: "GPT 4o 2024-05-13",
+ },
+
+ // GPT-4 Models
+ {
+ value: PROVIDER_MODEL_TYPE.GPT_4_TURBO,
+ label: "GPT 4 Turbo",
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.GPT_4,
+ label: "GPT 4",
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.GPT_4_TURBO_PREVIEW,
+ label: "GPT 4 Turbo Preview",
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.GPT_4_TURBO_2024_04_09,
+ label: "GPT 4 Turbo 2024-04-09",
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.GPT_4_1106_PREVIEW,
+ label: "GPT 4 1106 Preview",
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.GPT_4_0613,
+ label: "GPT 4 0613",
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.GPT_4_0125_PREVIEW,
+ label: "GPT 4 0125 Preview",
+ },
+
+ // GPT-3.5 Models
+ {
+ value: PROVIDER_MODEL_TYPE.GPT_3_5_TURBO,
+ label: "GPT 3.5 Turbo",
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.GPT_3_5_TURBO_1106,
+ label: "GPT 3.5 Turbo 1106",
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.GPT_3_5_TURBO_0125,
+ label: "GPT 3.5 Turbo 0125",
+ },
+ ],
+
+ [PROVIDER_TYPE.ANTHROPIC]: [
+ {
+ value: PROVIDER_MODEL_TYPE.CLAUDE_3_5_SONNET_20241022,
+ label: "Claude 3.5 Sonnet 2024-10-22",
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.CLAUDE_3_5_HAIKU_20241022,
+ label: "Claude 3.5 Haiku 2024-10-22",
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.CLAUDE_3_5_SONNET_20240620,
+ label: "Claude 3.5 Sonnet 2024-06-20",
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.CLAUDE_3_OPUS_20240229,
+ label: "Claude 3 Opus 2024-02-29",
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.CLAUDE_3_SONNET_20240229,
+ label: "Claude 3 Sonnet 2024-02-29",
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.CLAUDE_3_HAIKU_20240307,
+ label: "Claude 3 Haiku 2024-03-07",
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.CLAUDE_3_5_SONNET_LATEST,
+ label: "Claude 3.5 Sonnet Latest",
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.CLAUDE_3_5_HAIKU_LATEST,
+ label: "Claude 3.5 Haiku Latest",
+ },
+ {
+ value: PROVIDER_MODEL_TYPE.CLAUDE_3_OPUS_LATEST,
+ label: "Claude 3 Opus Latest",
+ },
+ ],
+ [PROVIDER_TYPE.OLLAMA]: [
+ // the list will be full filled base on data in localstorage
+ ],
+};
+
+const useLLMProviderModelsData = () => {
+ const { localModels, getLocalAIProviderData } = useLocalAIProviderData();
+
+ const getProviderModels = useCallback(() => {
+ return { ...PROVIDER_MODELS, ...localModels };
+ }, [localModels]);
+
+ const calculateModelProvider = useCallback(
+ (modelName?: PROVIDER_MODEL_TYPE | ""): PROVIDER_TYPE | "" => {
+ if (!modelName) {
+ return "";
+ }
+
+ const provider = Object.entries(getProviderModels()).find(
+ ([providerName, providerModels]) => {
+ if (providerModels.find((pm) => modelName === pm.value)) {
+ return providerName;
+ }
+
+ return false;
+ },
+ );
+
+ if (!provider) {
+ return "";
+ }
+
+ const [providerName] = provider;
+
+ return providerName as PROVIDER_TYPE;
+ },
+ [getProviderModels],
+ );
+
+ const calculateDefaultModel = useCallback(
+ (
+ lastPickedModel: PROVIDER_MODEL_TYPE | "",
+ setupProviders: PROVIDER_TYPE[],
+ preferredProvider?: PROVIDER_TYPE | "",
+ ) => {
+ const lastPickedModelProvider = calculateModelProvider(lastPickedModel);
+
+ const isLastPickedModelValid =
+ !!lastPickedModelProvider &&
+ setupProviders.includes(lastPickedModelProvider);
+
+ if (isLastPickedModelValid) {
+ return lastPickedModel;
+ }
+
+ const provider =
+ preferredProvider ?? getDefaultProviderKey(setupProviders);
+
+ if (provider) {
+ if (PROVIDERS[provider].locationType === PROVIDER_LOCATION_TYPE.local) {
+ return (
+ (first(
+ (getLocalAIProviderData(provider)?.models || "").split(","),
+ )?.trim() as PROVIDER_MODEL_TYPE) ?? ""
+ );
+ } else {
+ return PROVIDERS[provider].defaultModel;
+ }
+ }
+
+ return "";
+ },
+ [calculateModelProvider, getLocalAIProviderData],
+ );
+
+ return {
+ getProviderModels,
+ calculateModelProvider,
+ calculateDefaultModel,
+ };
+};
+
+export default useLLMProviderModelsData;
diff --git a/apps/opik-frontend/src/hooks/useLocalAIProviderData.ts b/apps/opik-frontend/src/hooks/useLocalAIProviderData.ts
new file mode 100644
index 0000000000..9892d77910
--- /dev/null
+++ b/apps/opik-frontend/src/hooks/useLocalAIProviderData.ts
@@ -0,0 +1,147 @@
+import { useCallback } from "react";
+import { useQuery, useQueryClient } from "@tanstack/react-query";
+import isEmpty from "lodash/isEmpty";
+import isNumber from "lodash/isNumber";
+import isArray from "lodash/isArray";
+
+import { safelyParseJSON } from "@/lib/utils";
+import {
+ LOCAL_PROVIDER_OPTION_TYPE,
+ PROVIDERS,
+ PROVIDERS_OPTIONS,
+} from "@/constants/providers";
+import {
+ LocalAIProviderData,
+ PROVIDER_LOCATION_TYPE,
+ PROVIDER_MODEL_TYPE,
+ PROVIDER_MODELS_TYPE,
+ PROVIDER_TYPE,
+ ProviderKey,
+} from "@/types/providers";
+import { PROVIDERS_KEYS_KEY } from "@/api/api";
+
+const useLocalAIProviderData = () => {
+ const queryClient = useQueryClient();
+
+ // workaround to recalculate list of local models every type we have any change in PROVIDERS_KEYS_KEY
+ const { data: localModels } = useQuery({
+ queryKey: [PROVIDERS_KEYS_KEY],
+ queryFn: () => {
+ const retVal: Partial
- {sortBy(organizationWorkspaces, "workspaceName").map(
- (workspace) => (
- (
+
+
-
-
-
- {workspace.workspaceName}
-
-
-
-
- ),
- )}
+
+
+ {workspace.workspaceName}
+
+
+
+
+ ))}