Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

_DSBEvaluations tasks arg, unit tests, Coherence evaluator #39671

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
+256 −105
Open

_DSBEvaluations tasks arg, unit tests, Coherence evaluator #39671

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
defc630
Adding asks arg to dsb evals
slister1001 Feb 11, 2025
5bcadd2
adding unit tests and coherence evaluator
slister1001 Feb 12, 2025
b3bacaa
adding evaluation_name param
slister1001 Feb 12, 2025
9d39611
updates
slister1001 Feb 12, 2025
e5b42fa
adding more unit tests
slister1001 Feb 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_dsb_evaluation/_dsb_evaluation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@ async def _simulate_dsb(
max_conversation_turns: int = 1,
max_simulation_results: int = 3,
conversation_turns : List[List[Union[str, Dict[str, Any]]]] = [],
tasks: List[str] = [],
adversarial_scenario: Optional[Union[AdversarialScenario, AdversarialScenarioJailbreak]] = None,
slister1001 marked this conversation as resolved.
Show resolved Hide resolved
source_text: Optional[str] = None,
direct_attack: bool = False,
Expand All @@ -137,6 +138,8 @@ async def _simulate_dsb(
:type max_simulation_results: int
:param conversation_turns: Predefined conversation turns to simulate.
:type conversation_turns: List[List[Union[str, Dict[str, Any]]]]
:param tasks A list of user tasks, each represented as a list of strings. Text should be relevant for the tasks and facilitate the simulation. One example is to use text to provide context for the tasks.
:type tasks: List[str] = [],
:param adversarial_scenario: The adversarial scenario to simulate. If None, the non-adversarial Simulator is used.
:type adversarial_scenario: Optional[Union[AdversarialScenario, AdversarialScenarioJailbreak]]
:param source_text: The source text to use as grounding document in the simulation.
Expand Down Expand Up @@ -188,6 +191,7 @@ async def callback(
scenario=adversarial_scenario,
max_conversation_turns=max_conversation_turns,
max_simulation_results=max_simulation_results,
tasks=tasks,
conversation_turns=conversation_turns,
text=source_text,
target=callback,
Expand Down Expand Up @@ -445,6 +449,7 @@ async def __call__(
max_conversation_turns: int = 1,
max_simulation_results: int = 3,
conversation_turns : List[List[Union[str, Dict[str, Any]]]] = [],
tasks: List[str] = [],
source_text: Optional[str] = None,
data_path: Optional[Union[str, os.PathLike]] = None,
jailbreak_data_path: Optional[Union[str, os.PathLike]] = None,
Expand All @@ -465,6 +470,8 @@ async def __call__(
:type max_simulation_results: int
:param conversation_turns: Predefined conversation turns to simulate.
:type conversation_turns: List[List[Union[str, Dict[str, Any]]]]
:param tasks A list of user tasks, each represented as a list of strings. Text should be relevant for the tasks and facilitate the simulation. One example is to use text to provide context for the tasks.
:type tasks: List[str] = [],
:param source_text: The source text to use as grounding document in the evaluation.
:type source_text: Optional[str]
:param data_path: The path to the data file generated by the Simulator. If None, the Simulator will be run.
Expand Down Expand Up @@ -494,6 +501,7 @@ async def __call__(
max_conversation_turns=max_conversation_turns,
max_simulation_results=max_simulation_results,
conversation_turns=conversation_turns,
tasks=tasks,
source_text=source_text,
direct_attack=_DSBEvaluator.DIRECT_ATTACK in evaluators
)
Expand Down
Loading