You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Where EntitiesEvaluator is an implementation of IConfusionMatrixEvaluator and string-no-punctuation and default is syntactic sugar for other evaluator implementations.
We may want to have an option that allows you to evaluate all JSON properties in the expected and actual JSON values (not just the configured properties), in which case any unconfigured property value would just use the default evaluator.
The text was updated successfully, but these errors were encountered:
Turns out this is likely more challenging than I originally thought. Seems for NLU at least, that we need a custom comparer specific to NLU results for each property (intent, text and entities).
Change NLU.DevOps.ModelPerformance to a configurable JSON compare library such that any two JSON values can be compared.
We could have a simple interface like
IConfusionMatrixEvaluator
:E.g., for NLU today, we would create the following config:
Where
EntitiesEvaluator
is an implementation ofIConfusionMatrixEvaluator
andstring-no-punctuation
anddefault
is syntactic sugar for other evaluator implementations.We may want to have an option that allows you to evaluate all JSON properties in the expected and actual JSON values (not just the configured properties), in which case any unconfigured property value would just use the default evaluator.
The text was updated successfully, but these errors were encountered: