DynamicRunEvaluator#

class langsmith.evaluation.evaluator.DynamicRunEvaluator(func: Callable[[Run, Example | None], EvaluationResult | EvaluationResults | dict | Awaitable[EvaluationResult | EvaluationResults | dict]], afunc: Callable[[Run, Example | None], Awaitable[EvaluationResult | EvaluationResults | dict]] | None = None)[source]#

A dynamic evaluator that wraps a function and transforms it into a RunEvaluator.

This class is designed to be used with the @run_evaluator decorator, allowing functions that take a Run and an optional Example as arguments, and return an EvaluationResult or EvaluationResults, to be used as instances of RunEvaluator.

func#

The function that is wrapped by this evaluator.

Type:

Callable

Initialize the DynamicRunEvaluator with a given function.

Parameters:
  • func (Callable) – A function that takes a Run and an optional Example as

  • arguments

  • ComparisonEvaluationResult. (and returns a dict or) –

  • afunc (Optional[Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]]) –

Attributes

is_async

Check if the evaluator function is asynchronous.

Methods

__init__(func[, afunc])

Initialize the DynamicRunEvaluator with a given function.

aevaluate_run(run[, example])

Evaluate a run asynchronously using the wrapped async function.

evaluate_run(run[, example])

Evaluate a run using the wrapped function.

__init__(func: Callable[[Run, Example | None], EvaluationResult | EvaluationResults | dict | Awaitable[EvaluationResult | EvaluationResults | dict]], afunc: Callable[[Run, Example | None], Awaitable[EvaluationResult | EvaluationResults | dict]] | None = None)[source]#

Initialize the DynamicRunEvaluator with a given function.

Parameters:
  • func (Callable) – A function that takes a Run and an optional Example as

  • arguments

  • ComparisonEvaluationResult. (and returns a dict or) –

  • afunc (Callable[[Run, Example | None], Awaitable[EvaluationResult | EvaluationResults | dict]] | None) –

async aevaluate_run(run: Run, example: Example | None = None)[source]#

Evaluate a run asynchronously using the wrapped async function.

This method directly invokes the wrapped async function with the

provided arguments.

Parameters:
  • run (Run) – The run to be evaluated.

  • example (Optional[Example]) – An optional example to be used in the evaluation.

Returns:

The result of the evaluation.

Return type:

Union[EvaluationResult, EvaluationResults]

evaluate_run(run: Run, example: Example | None = None) EvaluationResult | EvaluationResults[source]#

Evaluate a run using the wrapped function.

This method directly invokes the wrapped function with the provided arguments.

Parameters:
  • run (Run) – The run to be evaluated.

  • example (Optional[Example]) – An optional example to be used in the evaluation.

Returns:

The result of the evaluation.

Return type:

Union[EvaluationResult, EvaluationResults]