DynamicRunEvaluator#
- class langsmith.evaluation.evaluator.DynamicRunEvaluator(func: Callable[[Run, Example | None], EvaluationResult | EvaluationResults | dict | Awaitable[EvaluationResult | EvaluationResults | dict]], afunc: Callable[[Run, Example | None], Awaitable[EvaluationResult | EvaluationResults | dict]] | None = None)[source]#
A dynamic evaluator that wraps a function and transforms it into a RunEvaluator.
This class is designed to be used with the @run_evaluator decorator, allowing functions that take a Run and an optional Example as arguments, and return an EvaluationResult or EvaluationResults, to be used as instances of RunEvaluator.
- func#
The function that is wrapped by this evaluator.
- Type:
Callable
Initialize the DynamicRunEvaluator with a given function.
- Parameters:
Attributes
is_async
Check if the evaluator function is asynchronous.
Methods
__init__
(func[, afunc])Initialize the DynamicRunEvaluator with a given function.
aevaluate_run
(run[, example])Evaluate a run asynchronously using the wrapped async function.
evaluate_run
(run[, example])Evaluate a run using the wrapped function.
- __init__(func: Callable[[Run, Example | None], EvaluationResult | EvaluationResults | dict | Awaitable[EvaluationResult | EvaluationResults | dict]], afunc: Callable[[Run, Example | None], Awaitable[EvaluationResult | EvaluationResults | dict]] | None = None)[source]#
Initialize the DynamicRunEvaluator with a given function.
- Parameters:
func (Callable) – A function that takes a Run and an optional Example as
arguments –
ComparisonEvaluationResult. (and returns a dict or) –
afunc (Callable[[Run, Example | None], Awaitable[EvaluationResult | EvaluationResults | dict]] | None) –
- async aevaluate_run(run: Run, example: Example | None = None)[source]#
Evaluate a run asynchronously using the wrapped async function.
- This method directly invokes the wrapped async function with the
provided arguments.
- Parameters:
- Returns:
The result of the evaluation.
- Return type:
Union[EvaluationResult, EvaluationResults]
- evaluate_run(run: Run, example: Example | None = None) EvaluationResult | EvaluationResults [source]#
Evaluate a run using the wrapped function.
This method directly invokes the wrapped function with the provided arguments.
- Parameters:
- Returns:
The result of the evaluation.
- Return type:
Union[EvaluationResult, EvaluationResults]