Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: update prompts and invalidate cache #493

Open
wants to merge 15 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions integration-test/langfuse-integration-node.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -432,6 +432,26 @@ describe("Langfuse Node.js", () => {
version: expect.any(Number),
});
});

it.only("update a prompt", async () => {
hassiebp marked this conversation as resolved.
Show resolved Hide resolved
const promptName = "test-prompt" + Math.random().toString(36);
await langfuse.createPrompt({
name: promptName,
prompt: "This is a prompt with a {{variable}}",
isActive: true,
labels: ["john"],
});

const updatedPrompt = await langfuse.updatePrompt({
name: promptName,
version: 1,
newLabels: ["john", "doe"],
});
Comment on lines +445 to +449
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logic: Test should verify that old labels are preserved when updating with newLabels. Consider testing removal of labels and complete label replacement


const prompt = await langfuse.getPrompt(promptName);
expect(prompt.labels).toEqual(expect.arrayContaining(["john", "doe"]));
expect(updatedPrompt.labels).toEqual(expect.arrayContaining(["john", "doe"]));
});
});

it("link prompt to generation", async () => {
Expand Down
86 changes: 78 additions & 8 deletions langfuse-core/openapi-spec/openapi-server.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -723,6 +723,9 @@ paths:
Notes:


- Introduction to data model:
https://langfuse.com/docs/tracing-data-model

- Batch sizes are limited to 3.5 MB in total. You need to adjust the
number of events per batch accordingly.

Expand Down Expand Up @@ -1423,6 +1426,76 @@ paths:
schema: {}
security:
- BasicAuth: []
/api/public/v2/prompts/{promptName}/version/{version}:
patch:
description: Update labels for a specific prompt version
operationId: promptVersion_update
tags:
- PromptVersion
parameters:
- name: promptName
in: path
description: The name of the prompt
required: true
schema:
type: string
- name: version
in: path
description: Version of the prompt to update
required: true
schema:
type: integer
responses:
'200':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/Prompt'
'400':
description: ''
content:
application/json:
schema: {}
'401':
description: ''
content:
application/json:
schema: {}
'403':
description: ''
content:
application/json:
schema: {}
'404':
description: ''
content:
application/json:
schema: {}
'405':
description: ''
content:
application/json:
schema: {}
security:
- BasicAuth: []
requestBody:
required: true
content:
application/json:
schema:
type: object
properties:
newLabels:
type: array
items:
type: string
description: >-
New labels for the prompt version. Labels are unique across
versions. The 'latest' label is reserved and managed by
Langfuse.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logic: newLabels array should validate that 'latest' is not included since it's a reserved label

required:
- newLabels
/api/public/v2/prompts/{promptName}:
get:
description: Get a prompt
Expand All @@ -1446,7 +1519,7 @@ paths:
- name: label
in: query
description: >-
Label of the prompt to be retrieved. Defaults to "production" if no
Label of the prompt to be retrieved. Defaults to 'production' if no
label or version is set.
required: false
schema:
Expand Down Expand Up @@ -2754,7 +2827,6 @@ components:
description: Defaults to input+output if not set
unit:
$ref: '#/components/schemas/ModelUsageUnit'
nullable: true
inputCost:
type: number
format: double
Expand Down Expand Up @@ -2906,13 +2978,13 @@ components:
type: number
format: double
description: >-
The numeric value of the score. Equals 1 for "True" and 0 for
"False"
The numeric value of the score. Equals 1 for 'True' and 0 for
'False'
stringValue:
type: string
description: >-
The string representation of the score value. Is inferred from the
numeric value and equals "True" or "False"
numeric value and equals 'True' or 'False'
required:
- value
- stringValue
Expand Down Expand Up @@ -3183,7 +3255,7 @@ components:
to exact match, use `(?i)^modelname$`
startDate:
type: string
format: date
format: date-time
nullable: true
description: Apply only to generations which are newer than this ISO date.
unit:
Expand Down Expand Up @@ -3223,7 +3295,6 @@ components:
- id
- modelName
- matchPattern
- unit
- isLangfuseManaged
ModelUsageUnit:
title: ModelUsageUnit
Expand Down Expand Up @@ -4308,7 +4379,6 @@ components:
required:
- modelName
- matchPattern
- unit
Observations:
title: Observations
type: object
Expand Down
16 changes: 16 additions & 0 deletions langfuse-core/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ import {
type UpdateLangfuseGenerationBody,
type UpdateLangfuseSpanBody,
type GetMediaResponse,
UpdatePromptBody,
} from "./types";
import { LangfuseMedia, type LangfuseMediaResolveMediaReferencesParams } from "./media/LangfuseMedia";
import {
Expand Down Expand Up @@ -527,6 +528,15 @@ abstract class LangfuseCoreStateless {
);
}

async updatePromptStateless(
body: UpdatePromptBody & { name: string; version: number }
): Promise<LangfusePromptClient> {
return this.fetchAndLogErrors(
`${this.baseUrl}/api/public/v2/prompts/${encodeURIComponent(body.name)}/versions/${encodeURIComponent(body.version)}`,
this._getFetchOptions({ method: "PATCH", body: JSON.stringify(body) })
);
}

async getPromptStateless(
name: string,
version?: number,
Expand Down Expand Up @@ -1382,6 +1392,12 @@ export abstract class LangfuseCore extends LangfuseCoreStateless {
return new TextPromptClient(promptResponse);
}

async updatePrompt(body: { name: string; version: number; newLabels: string[] }): Promise<LangfusePromptClient> {
const newPrompt = await this.updatePromptStateless(body);
this._promptCache.invalidate(body.name);
return newPrompt;
}

async getPrompt(
name: string,
version?: number,
Expand Down
97 changes: 93 additions & 4 deletions langfuse-core/src/openapi/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,7 @@ export interface paths {
*
* Notes:
*
* - Introduction to data model: https://langfuse.com/docs/tracing-data-model
* - Batch sizes are limited to 3.5 MB in total. You need to adjust the number of events per batch accordingly.
* - The API does not return a 4xx status code for input errors. Instead, it responds with a 207 status code, which includes a list of the encountered errors. */
post: operations["ingestion_batch"];
Expand Down Expand Up @@ -338,6 +339,23 @@ export interface paths {
patch?: never;
trace?: never;
};
"/api/public/v2/prompts/{promptName}/version/{version}": {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
get?: never;
put?: never;
post?: never;
delete?: never;
options?: never;
head?: never;
/** @description Update labels for a specific prompt version */
patch: operations["promptVersion_update"];
trace?: never;
};
"/api/public/v2/prompts/{promptName}": {
parameters: {
query?: never;
Expand Down Expand Up @@ -949,12 +967,12 @@ export interface components {
/** @description Regex pattern which matches this model definition to generation.model. Useful in case of fine-tuned models. If you want to exact match, use `(?i)^modelname$` */
matchPattern: string;
/**
* Format: date
* Format: date-time
* @description Apply only to generations which are newer than this ISO date.
*/
startDate?: string | null;
/** @description Unit used by this model. */
unit: components["schemas"]["ModelUsageUnit"];
unit?: components["schemas"]["ModelUsageUnit"];
/**
* Format: double
* @description Price (USD) per input unit
Expand Down Expand Up @@ -988,7 +1006,7 @@ export interface components {
*/
ObservationLevel: "DEBUG" | "DEFAULT" | "WARNING" | "ERROR";
/** MapValue */
MapValue: (string | null) | (number | null) | (boolean | null) | (string[] | null) | undefined;
MapValue: (string | null) | (number | null) | (boolean | null) | (string[] | null);
hassiebp marked this conversation as resolved.
Show resolved Hide resolved
/**
* CommentObjectType
* @enum {string}
Expand Down Expand Up @@ -1477,7 +1495,7 @@ export interface components {
*/
startDate?: string | null;
/** @description Unit used by this model. */
unit: components["schemas"]["ModelUsageUnit"];
unit?: components["schemas"]["ModelUsageUnit"];
/**
* Format: double
* @description Price (USD) per input unit
Expand Down Expand Up @@ -3317,6 +3335,77 @@ export interface operations {
};
};
};
promptVersion_update: {
parameters: {
query?: never;
header?: never;
path: {
/** @description The name of the prompt */
promptName: string;
/** @description Version of the prompt to update */
version: number;
};
cookie?: never;
};
requestBody: {
content: {
"application/json": {
/** @description New labels for the prompt version. Labels are unique across versions. The "latest" label is reserved and managed by Langfuse. */
newLabels: string[];
};
};
};
responses: {
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["Prompt"];
};
};
400: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": unknown;
};
};
401: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": unknown;
};
};
403: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": unknown;
};
};
404: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": unknown;
};
};
405: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": unknown;
};
};
};
};
prompts_get: {
parameters: {
query?: {
Expand Down
8 changes: 8 additions & 0 deletions langfuse-core/src/prompts/promptCache.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,4 +50,12 @@ export class LangfusePromptCache {
public isRefreshing(key: string): boolean {
return this._refreshingKeys.has(key);
}

public invalidate(promptName: string): void {
for (const key of this._cache.keys()) {
if (key.startsWith(promptName)) {
this._cache.delete(key);
}
}
}
Comment on lines +54 to +60
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logic: The string prefix matching could match unintended keys if prompt names share common prefixes (e.g. 'prompt1' would match 'prompt10'). Consider using a more precise matching strategy like ${promptName}/ or a regex pattern.

Comment on lines +54 to +60
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logic: This method should also clear any pending refresh promises for the invalidated prompts from _refreshingKeys to prevent stale data from being re-cached.

}
3 changes: 3 additions & 0 deletions langfuse-core/src/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,9 @@ export type GetLangfuseDatasetRunsResponse = FixTypes<
export type CreateLangfusePromptBody = FixTypes<
paths["/api/public/v2/prompts"]["post"]["requestBody"]["content"]["application/json"]
>;
export type UpdatePromptBody = FixTypes<
paths["/api/public/v2/prompts/{promptName}/version/{version}"]["patch"]["requestBody"]["content"]["application/json"]
>;
export type CreateLangfusePromptResponse =
paths["/api/public/v2/prompts"]["post"]["responses"]["200"]["content"]["application/json"];

Expand Down
Loading