Klarity

Toolkit for LLM behavior analysis & uncertainty mitigation

🐳 Now with reasoning model support to analyse CoTs entropy and improve RL datasets

🎯 Overview

Klarity is a toolkit for inspecting and debugging AI decision-making processes. By combining uncertainty analysis with reasoning insights and visual attention patterns, it helps you understand how models think and fix issues before they reach production.

• Dual Entropy Analysis: Measure model confidence through raw entropy and semantic similarity metrics
• Reasoning Analysis: Extract and evaluate step-by-step thinking patterns in model outputs
• Visual Attention Analysis: Visualize and analyze how vision-language models attend to images
• Semantic Clustering: Group similar predictions to reveal decision-making pathways
• Structured Insights: Get detailed JSON analysis of both uncertainty patterns and reasoning steps
• AI-powered Report: Leverage capable models to interpret generation patterns and provide human-readable insights

VLM Analysis Example – Gaining insights into where your model focuses and examining related token uncertainty.

Reasoning Analysis Example - Understanding model's step-by-step thinking process

Entropy Analysis Example - Analyzing token-level uncertainty patterns

🚀 Quick Start Hugging Face

Install directly from GitHub:

pip install git+https://github.com/klara-research/klarity.git

🖼️ VLM Analysis Usage Example

For insights into where your model is focusing and to analyze related token uncertainty, you can use the VLMAnalyzer:

from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration, LogitsProcessorList
from PIL import Image
import torch
from klarity import UncertaintyEstimator
from klarity.core.analyzer import VLMAnalyzer
import os

os.environ["TOKENIZERS_PARALLELISM"] = "false"
# Initialize VLM model
model_id = "llava-hf/llava-onevision-qwen2-0.5b-ov-hf"
model = LlavaOnevisionForConditionalGeneration.from_pretrained(
    model_id,
    output_attentions=True,
    low_cpu_mem_usage=True
)

processor = AutoProcessor.from_pretrained(model_id)

# Create estimator with VLMAnalyzer
estimator = UncertaintyEstimator(
    top_k=100,
    analyzer=VLMAnalyzer(
        min_token_prob=0.01,
        insight_model="together:together:meta-llama/Llama-3.3-70B-Instruct-Turbo",
        insight_api_key="your_api_key",
        vision_config=model.config.vision_config, 
        use_cls_token=True
    ),
)
uncertainty_processor = estimator.get_logits_processor()

# Set up generation
image_path = "examples/images/plane.jpg"
question = "How many engines does the plane have?"
image = Image.open(image_path)

# Prepare input with image and text
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": question},
            {"type": "image"}
        ]
    }
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)

# Process inputs
inputs = processor(
    images=image,
    text=prompt,
    return_tensors='pt'
)

try:
    # Generate with uncertainty and attention analysis
    generation_output = model.generate(
        **inputs,
        max_new_tokens=200,
        temperature=0.7,
        top_p=0.9,
        do_sample=True,
        logits_processor=LogitsProcessorList([uncertainty_processor]),
        return_dict_in_generate=True,
        output_scores=True,
        output_attentions=True,
        use_cache=True
    )

    # Analyze the generation - Note the corrected parameter order
    result = estimator.analyze_generation(
        generation_output=generation_output,
        model=model,                    # Add model parameter
        tokenizer=processor,            # Changed from processor to tokenizer parameter
        processor=uncertainty_processor,
        prompt=question
    )

    # Get generated text using the processor instead of raw sequences
    input_length = inputs.input_ids.shape[1]
    generated_sequence = generation_output.sequences[0][input_length:]
    generated_text = processor.decode(generated_sequence, skip_special_tokens=True)
    
    print(f"\nQuestion: {question}")
    print(f"Generated answer: {generated_text}")

    # Token Analysis
    print("\nDetailed Token Analysis:")
    for idx, metrics in enumerate(result.token_metrics):
        print(f"\nStep {idx}:")
        print(f"Raw entropy: {metrics.raw_entropy:.4f}")
        print(f"Semantic entropy: {metrics.semantic_entropy:.4f}")
        print("Top 3 predictions:")
        for i, pred in enumerate(metrics.token_predictions[:3], 1):
            print(f"  {i}. {pred.token} (prob: {pred.probability:.4f})")

    # Attention Analysis (if available)
    if result.attention_data:
        print("\nAttention Analysis:")
        estimator.analyzer.visualize_attention(
            result.attention_data,
            image,
            save_path="examples/attention_maps/attention_visualization.png"
        )

    # Show comprehensive insight
    print("\nComprehensive Analysis:")
    print(result.overall_insight)

except Exception as e:
    print(f"Error during generation: {str(e)}")
    import traceback
    traceback.print_exc()  # Print full traceback for debugging

📝 Reasoning LLM Usage Example

For insights and uncertainty analytics into model reasoning patterns, you can use the ReasoningAnalyzer:

from transformers import AutoModelForCausalLM, AutoTokenizer, LogitsProcessorList
from klarity import UncertaintyEstimator
from klarity.core.analyzer import ReasoningAnalyzer
import torch

# Initialize model with GPU support
device = "cuda" if torch.cuda.is_available() else "cpu"

model_name = "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"
model = AutoModelForCausalLM.from_pretrained(model_name)
model = model.to(device)  # Move model to GPU
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Create estimator with reasoning analyzer and togetherai hosted model
estimator = UncertaintyEstimator(
    top_k=100,
    analyzer=ReasoningAnalyzer(
        min_token_prob=0.01,
        insight_model="together:meta-llama/Llama-3.3-70B-Instruct-Turbo",
        insight_api_key="your_api_key",
        reasoning_start_token="<think>", 
        reasoning_end_token="</think>"
    )
)
uncertainty_processor = estimator.get_logits_processor()

# Set up generation
prompt = "Your prompt <think>\n"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

# Generate with uncertainty analysis
generation_output = model.generate(
    **inputs,
    max_new_tokens=200,  # Increased for reasoning steps
    temperature=0.6,
    logits_processor=LogitsProcessorList([uncertainty_processor]),
    return_dict_in_generate=True,
    output_scores=True,
)

# Analyze the generation
result = estimator.analyze_generation(
    generation_output,
    tokenizer,
    uncertainty_processor,
    prompt  # Include prompt for better reasoning analysis
)

# Get generated text
generated_text = tokenizer.decode(generation_output.sequences[0], skip_special_tokens=True)
print(f"\nPrompt: {prompt}")
print(f"Generated text: {generated_text}")

# Show reasoning analysis
print("\nReasoning Analysis:")
if result.overall_insight and "reasoning_analysis" in result.overall_insight:
    analysis = result.overall_insight["reasoning_analysis"]
    
    # Print each reasoning step
    for idx, step in enumerate(analysis["steps"], 1):
        print(f"\nStep {idx}:")  # Use simple counter instead of accessing step_number
        print(f"Content: {step['step_info']['content']}")
        
        # Print step analysis
        if 'analysis' in step and 'training_insights' in step['analysis']:
            step_analysis = step['analysis']['training_insights']
            print("\nQuality Metrics:")
            for metric, score in step_analysis['step_quality'].items():
                print(f"  {metric}: {score}")
            
            print("\nImprovement Targets:")
            for target in step_analysis['improvement_targets']:
                print(f"  Aspect: {target['aspect']}")
                print(f"  Importance: {target['importance']}")
                print(f"  Issue: {target['current_issue']}")
                print(f"  Suggestion: {target['training_suggestion']}")

📝 Standard LLM Usage Example

To prevent most of common uncertainty scenarios and route to better models you can use our EntropyAnalyzer

from transformers import AutoModelForCausalLM, AutoTokenizer, LogitsProcessorList
from klarity import UncertaintyEstimator
from klarity.core.analyzer import EntropyAnalyzer

# Initialize your model
model_name = "Qwen/Qwen2.5-7B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Create estimator
estimator = UncertaintyEstimator(
    top_k=100,
    analyzer=EntropyAnalyzer(
        min_token_prob=0.01,
        insight_model=model,
        insight_tokenizer=tokenizer
    )
)

uncertainty_processor = estimator.get_logits_processor()

# Set up generation
prompt = "Your prompt"
inputs = tokenizer(prompt, return_tensors="pt")

# Generate with uncertainty analysis
generation_output = model.generate(
    **inputs,
    max_new_tokens=20,
    temperature=0.7,
    top_p=0.9,
    logits_processor=LogitsProcessorList([uncertainty_processor]),
    return_dict_in_generate=True,
    output_scores=True,
)

# Analyze the generation
result = estimator.analyze_generation(
    generation_output,
    tokenizer,
    uncertainty_processor
)

generated_text = tokenizer.decode(generation_output.sequences[0], skip_special_tokens=True)

# Inspect results
print(f"\nPrompt: {prompt}")
print(f"Generated text: {generated_text}")

print("\nDetailed Token Analysis:")
for idx, metrics in enumerate(result.token_metrics):
    print(f"\nStep {idx}:")
    print(f"Raw entropy: {metrics.raw_entropy:.4f}")
    print(f"Semantic entropy: {metrics.semantic_entropy:.4f}")
    print("Top 3 predictions:")
    for i, pred in enumerate(metrics.token_predictions[:3], 1):
        print(f"  {i}. {pred.token} (prob: {pred.probability:.4f})")

# Show comprehensive insight
print("\nComprehensive Analysis:")
print(result.overall_insight)

📊 Analysis Output

Klarity provides three types of analysis output:

VLM Analysis

Attention insights into where your model is focusing and related token uncertainty:

{
    "scores": {
        "overall_uncertainty": "<0-1>",
        "visual_grounding": "<0-1>",
        "confidence": "<0-1>"
    },
    "visual_analysis": {
        "attention_quality": {
            "score": "<0-1>",
            "key_regions": ["<region1>", "<region2>"],
            "missed_regions": ["<region1>", "<region2>"]
        },
        "token_attention_alignment": [
            {
                "token": "<token>",
                "attended_region": "<region>",
                "relevance": "<0-1>"
            }
        ]
    },
    "uncertainty_analysis": {
        "high_uncertainty_segments": [
            {
                "text": "<text>",
                "cause": "<reason>",
                "visual_context": "<what_model_looked_at>"
            }
        ],
        "improvement_suggestions": [
            {
                "aspect": "<what>",
                "suggestion": "<how>"
            }
        ]
    }
}

Reasoning Analysis

You'll get detailed insights into the model's reasoning process:

{
    "reasoning_analysis": {
        "steps": [
            {
                "step_number": 1,
                "step_info": {
                    "content": "",
                    "type": ""
                },
                "analysis": {
                    "training_insights": {
                        "step_quality": {
                            "coherence": "<0-1>",
                            "relevance": "<0-1>",
                            "confidence": "<0-1>"
                        },
                        "improvement_targets": [
                            {
                                "aspect": "",
                                "importance": "<0-1>",
                                "current_issue": "",
                                "training_suggestion": ""
                            }
                        ]
                    }
                }
            }
        ]
    }
}

Entropy Analysis

For standard language models you will get a general uncertainty report:

{
    "scores": {
        "overall_uncertainty": "<0-1>",  
        "confidence_score": "<0-1>",     
        "hallucination_risk": "<0-1>"    
    },
    "uncertainty_analysis": {
        "high_uncertainty_parts": [
            {
                "text": "",
                "why": ""
            }
        ],
        "main_issues": [
            {
                "issue": "",
                "evidence": ""
            }
        ],
        "key_suggestions": [
            {
                "what": "",
                "how": ""
            }
        ]
    }
}

🤖 Supported Frameworks & Models

Model Frameworks

Currently supported:

• ✅ Hugging Face Transformers -> Full uncertainty analysis with raw and semantic entropy metrics & vision attention monitoring

• ✅ vLLM -> Full uncertainty analysis with raw and semantic entropy metrics with max 20 logprobs per token

• ✅ Together AI -> Uncertainty analysis with raw log prob. metrics

Planned support:

• ⏳ PyTorch

Analysis Model (for the insights) Frameworks

Currently supported:

• ✅ Hugging Face Transformers
• ✅ Together AI API

Planned support:

• ⏳ PyTorch

Tested Target Models

Model	Type	Status	Notes
Qwen2.5-0.5B	Base	✅ Tested	Full Support
Qwen2.5-0.5B-Instruct	Instruct	✅ Tested	Full Support
Qwen2.5-1.5B-Instruct	Instruct	✅ Tested	Full Support
Qwen2.5-7B	Base	✅ Tested	Full Support
Qwen2.5-7B-Instruct	Instruct	✅ Tested	Full Support
Llama-3.2-3B-Instruct	Instruct	✅ Tested	Full Support
Meta-Llama-3-8B	Base	✅ Tested	Together API Insights
gemma-2-2b-it	Instruct	✅ Tested	Full Support
mistralai/Mistral-7B-Instruct-v0.3	Instruct	✅ Tested	Together API Insights
Qwen/Qwen2.5-72B-Instruct-Turbo	Instruct	✅ Tested	Together API Insights
DeepSeek-R1-Distill-Qwen-1.5B	Reasoning	✅ Tested	Together API Insights
DeepSeek-R1-Distill-Qwen-7B	Reasoning	✅ Tested	Together API Insights
Llava-onevision-qwen2-0.5b-ov-hf	Vision	✅ Tested	Together API Insights

Analysis Models

Model	Type	Status	JSON Reliability	Notes
Qwen2.5-0.5B-Instruct	Instruct	✅ Tested	⚡ Low	Consistently output unstructured analysis instead of JSON. Best used with structured prompting and validation.
Qwen2.5-7B-Instruct	Instruct	✅ Tested	⚠️ Moderate	Sometimes outputs well-formed JSON analysis.
Llama-3.3-70B-Instruct-Turbo	Instruct	✅ Tested	✅ High	Reliably outputs well-formed JSON analysis. Recommended for production use.

JSON Output Reliability Guide:

• ✅ High: Consistently outputs valid JSON (>80% of responses)
• ⚠️ Moderate: Usually outputs valid JSON (50-80% of responses)
• ⚡ Low: Inconsistent JSON output (<50% of responses)

🔍 Advanced Features

Custom Analysis Configuration

You can customize the analysis parameters:

analyzer = EntropyAnalyzer(
    min_token_prob=0.01,  # Minimum probability threshold
    semantic_similarity_threshold=0.8  # Threshold for semantic grouping
)

🤝 Contributing

Contributions are welcome! Areas we're looking to improve:

• Additional framework support
• More tested models
• Enhanced semantic analysis
• Additional analysis metrics
• Documentation and examples

Please see our Contributing Guide for details.

📝 License

Apache 2.0 License. See LICENSE for more information.

📫 Community & Support

• Website
• Discord Community for discussions & chatting
• GitHub Issues for bugs and features