Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extracting RAI Metrics from Langkit #321

Open
ishachinniah-hds opened this issue Feb 11, 2025 · 0 comments
Open

Extracting RAI Metrics from Langkit #321

ishachinniah-hds opened this issue Feb 11, 2025 · 0 comments

Comments

@ishachinniah-hds
Copy link

I want to clarify the implementation of Langkit to extract the following metrics from my genAI application: 'prompt.injection', 'prompt.jailbreak_similarity', 'prompt.toxicity', 'response.hallucination', 'response.refusal_similarity', 'response.toxicity'.

Code:

    ## LANGKIT - Injections, jailbreak/refusal similarity, hallucination and toxicity
    response_hallucination.init(llm=langkit_azure_llm(), num_samples=3)
    profile = why.log({"prompt": query, "response": response},schema=text_schema).profile().view().to_pandas()
    # View the data
    print(profile)
    filepath = os.path.join(os.getcwd(),"src","evaluation", "RAI.csv")
    profile.to_csv(filepath, index=True)  # index=True includes the index column

RAI.csv output:

Image

In the above table, are the repeating scores under distribution/max, distribution/mean, distribution/median.. the accurate metric scores? Or is this not the correct way to get these scores?

Thank you for clarifying

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant