Hi all, having trouble with remote evaluators in the prompt playground and am not able to figure out the cause of the troubles. After some exploring and talking to the Alyx chat assistant I might have figured out a bug, would it be possible to fix it soonish?
"""
Bug Report: Remote evaluators silently fail to attach in the Prompt Playground "Add Evaluator" modal
Environment:
Arize AX Prompt Playground
What happened: I have 5 custom remote evaluators mapped to a dataset. When I open the "Add Evaluator" modal in the Prompt Playground and click on any of these remote evaluators, the modal closes but nothing gets added -- no evaluator appears in the Evaluators panel!
I observed there is a silent console error (from browser dev tools):
ERROR EvaluatorMultiSelector: no code evaluator found for eval id: RX***ZQ (I anonymized it)
Root cause (as I understand it): The EvaluatorMultiSelector component is trying to resolve the clicked evaluator as a "code evaluator" internally. Remote evaluators aren't found in that lookup, so the selection silently fails and the modal closes with no feedback to the user.
Expected behavior: Either (a) remote evaluators should be selectable in the playground and run automatically alongside the experiment, or (b) they should be visually excluded from the modal with a tooltip/label explaining why.
Impact: Cannot attach remote evaluators to playground experiment runs at all. The modal shows them as selectable options, giving the impression they should work, but they silently do nothing.
"""
