Skip to content

fix: grade None provider completions as empty answers instead of crashing - #44

Merged
lezama merged 1 commit into
trunkfrom
fix/none-completion-text
Jul 20, 2026
Merged

fix: grade None provider completions as empty answers instead of crashing#44
lezama merged 1 commit into
trunkfrom
fix/none-completion-text

Conversation

@lezama

@lezama lezama commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

What

First real multi-model pilot run (gemini-2.5-pro + flash on the hardened dataset) crashed with:

File .../wp_bench/utils.py, line 9, in strip_code_fences
    stripped = output.strip()
AttributeError: 'NoneType' object has no attribute 'strip'

litellm can hand back a choice whose message.content is None — providers do this on safety blocks, thinking-only responses, or truncation. ModelGeneration.text carried the None into parse_artifact, killing the entire run on one bad completion.

Fix

Normalize at the single extraction point in ModelInterface.generate_with_metadata: content or "". An empty completion is a model output, not a harness fault, so it should grade as an empty answer (score 0 for that test) and let the run continue. This also composes with run.continue_on_error for the remaining per-test failure modes.

Includes a regression test with a mocked None-content response.

🤖 Generated with Claude Code

…hing

Found by the first live multi-model pilot: gemini-2.5-pro returned a
response whose message content was None (providers do this on safety
blocks, thinking-only responses, or truncation). ModelGeneration.text
carried the None into parse_artifact -> strip_code_fences, which crashed
the whole run with AttributeError.

An empty completion is a model output, not a harness fault: normalize to
"" at the extraction point so the test grades as an empty answer
(score 0) and the run continues.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: lezama <migueluy@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

@lezama
lezama merged commit d20c0e7 into trunk Jul 20, 2026
5 checks passed
@lezama
lezama deleted the fix/none-completion-text branch July 20, 2026 14:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant