Try and get LLMs to return content matching the language of the original content

[dkotter]

What?

Closes #349, #351

Update some of our system instructions to prompt the LLM to return content in the same language as the original content they were given.

Why?

At the moment, you may get content returned from an LLM that isn't the same as your original content. For example, if you have a post that is written in French and you generate a title, the title may be in English (though in my testing, it always returned in French).

Ideal is to have the LLMs match the language of the content they were given. This supports the widest amount of use cases, like multi-lingual sites.

How?

Updates the system instructions for any Ability that generates user-facing text to tell the LLM to match the language of the content they were given.

For alt text generation, we may not have other content (say when generating alt text straight in the media library) so we pass the site locale to use as our default.

Worth noting it's still up to the LLM to follow these instructions so there's almost certainly edge cases that this won't work for, particularly if using a smaller model.

Use of AI Tools

None

Testing Instructions

1. Checkout this PR
2. Create some content in a language other than English
3. Turn on some Experiments (Title Generation, Excerpt Generation, Summarization, Alt Text Generation)
4. Run these Experiments and ensure the output returned matches the language of the content
5. Change your site locale to something other than English
6. Generate alt text for an image in the Media Library and ensure it matches the site locale

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: dkotter <dkotter@git.wordpress.org>
Co-authored-by: jeffpaul <jeffpaul@git.wordpress.org>
Co-authored-by: gziolo <gziolo@git.wordpress.org>
Co-authored-by: saarnilauri <laurisaarni@git.wordpress.org>
Co-authored-by: afercia <afercia@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[codecov]

Codecov Report

❌ Patch coverage is 25.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.75%. Comparing base (4f7a3ba) to head (3b25cf2).
⚠️ Report is 17 commits behind head on develop.

Files with missing lines	Patch %	Lines
includes/Abilities/Image/Alt_Text_Generation.php	0.00%	6 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             develop     #357      +/-   ##
=============================================
- Coverage      65.82%   65.75%   -0.07%     
  Complexity       763      763              
=============================================
  Files             53       53              
  Lines           3859     3863       +4     
=============================================
  Hits            2540     2540              
- Misses          1319     1323       +4     

Flag	Coverage Δ
unit	65.75% <25.00%> (-0.07%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[gziolo]

While reviewing this PR, I noticed a pre-existing bug in load_system_instruction_from_file(): it uses require_once for explicit filenames, which means the second call to get_system_instruction() with the same file silently returns an empty string (since require_once returns true instead of the file's return value on subsequent calls).

This doesn't block this PR, but it could affect alt text generation if called multiple times in a single request (e.g., batch processing images).

Opened #358 with a failing test and one-line fix (require_once → require).

[gziolo]

I tested this with a post written in Polish.

English site locale + Polish content:

Feature	Language	Expected
Excerpt	Polish ✅	Polish
Summary	English ❌	Polish
Alt text	English ❌	Polish (from content) or English (from locale)

Polish site locale + Polish content:

Feature	Language	Expected
Excerpt	Polish ✅	Polish
Summary	Polish ✅	Polish
Alt text	English ❌	Polish

Excerpt generation consistently respects content language, which is great. Summary only matched Polish when the site locale was also set to Polish — suggesting the LLM may not reliably follow the instruction when the default locale is English. Alt text returned English in all cases, even with Polish locale — the model seems to disregard the locale instruction entirely when working from an image alone.

(Page was refreshed before each action to ensure clean state.)

Aside, it might be not trivial to instruct LLM about the language for the alt image, as it largely depend on the context. In the Media library modal, it should probably will always default to the site's locale. However, in the post, you would need to pass some content so LLM can infer the language from it.

[dkotter]

I tested this with a post written in Polish.

Hmm.. thanks for thorough tests here @gziolo. Can you confirm what provider you were using? I've tested with OpenAI, both with French and Italian content and was always getting the results I would expected (and in fact, was getting the correct results prior to any changes made in this PR).

I know each model will behave differently so not sure there's a 100% way to ensure content matches the correct language but I was hoping results would be better than what you're seeing. Definitely open to suggestions on

Aside, it might be not trivial to instruct LLM about the language for the alt image, as it largely depend on the context. In the Media library modal, it should probably will always default to the site's locale. However, in the post, you would need to pass some content so LLM can infer the language from it.

We do already pass in the surrounding content (if any) when generating alt text for in-content images. And this PR updates the system instructions to instruct the LLM to match the returned alt text with the language in that content and if no content is provided, to match the passed in site locale. That's obviously not working in all scenarios, based on your testing, so open to refinements on that.

[gziolo]

I was using Claude. Maybe I'm missing some obvious details. I can investigate further tomorrow and include the instructions used and the responses produced.

[gziolo]

I did some more testing with Claude as the provider, using a post written in Polish and en_US site locale. I added debug logging to capture the exact system instructions, prompts, and responses.

Findings

Excerpt and Summary work correctly — they consistently return Polish because their instruction ("match the language of the content you are given") is straightforward, and the content is passed directly in <content> tags as the primary input.

Alt text consistently returns English despite Polish content being provided. Three compounding issues:

1. esc_html() mangles system instructions. load_system_instruction_from_file() wraps the result in esc_html(), so the LLM receives <additional-context> instead of <additional-context>, "Image of" instead of "Image of", etc. The instruction referencing "CONTENT in the <additional-context> tag" can't match the actual tag in the prompt.

2. English preamble wrapping the content — the context sent from generate-alt-text.ts wraps Polish content inside English instructions:

   What follows is the full article content, where the image has been
   replaced with the placeholder [[IMAGE_GOES_HERE]]. Use the surrounding
   text to understand the purpose, subject, and relevance of the image...
   CONTENT: {Polish text}
   

   This English preamble is a stronger language signal than the Polish content that follows, especially for a vision model primarily focused on describing the image.

3. Language instruction buried in a bullet point — for vision models, the image is the dominant input. A language instruction buried in the system instruction's bullet list doesn't carry enough weight, especially when mangled by esc_html().

What worked

I tested the following changes locally and alt text returned in Polish consistently:

Move the English preamble out of the context — send raw content only, move the usage instructions into the system instruction where they belong:

// Before
params.context = `What follows is the full article content, where the image
has been replaced with the placeholder ... CONTENT: \n\n${ contentWithPlaceholder }`;

// After
params.context = contentWithPlaceholder;

Update the system instruction — consolidate context usage and language guidance:

-- Consider context: If context is provided, ensure the alt text is relevant to the surrounding content
+- Consider context: If additional context is provided, it is the surrounding article content where the image location is marked with [[IMAGE_GOES_HERE]]. Use it to understand the purpose, subject, and relevance of the image. Describe only information not already conveyed in nearby text. Infer the language from this content and write the alt text in the same language
 - Plain text only: No markdown, quotes, or special formatting
-- If you are given CONTENT in the <additional-context> tag, ensure the alt text you return matches the language of that content...
+- If no additional context is provided, write the alt text in the language matching locale {$return_locale}

Put the language instruction in the user prompt — this bypasses the esc_html() issue entirely and gives the language instruction maximum weight:

// With context (editor) — match content language:
'Generate alt text for this image. Write the alt text in the same language as the surrounding article content.'

// Without context (Media Library) — use site locale:
'Generate alt text for this image. Write the alt text in the language matching locale pl_PL.'

Specific code changes:

diff --git a/includes/Abilities/Image/Alt_Text_Generation.php b/includes/Abilities/Image/Alt_Text_Generation.php
index 107bfb7..a487150 100644
--- a/includes/Abilities/Image/Alt_Text_Generation.php
+++ b/includes/Abilities/Image/Alt_Text_Generation.php
@@ -374,9 +374,17 @@ class Alt_Text_Generation extends Abstract_Ability {
 	protected function build_prompt( string $context = '' ): string {
 		$prompt = __( 'Generate alt text for this image.', 'ai' );
 
-		// If we have additional context, add it to the prompt.
+		// If we have additional context, instruct the LLM to match the content language.
 		if ( ! empty( $context ) ) {
+			$prompt .= ' ' . __( 'Write the alt text in the same language as the surrounding article content.', 'ai' );
 			$prompt .= "\n\n<additional-context>" . $context . '</additional-context>';
+		} else {
+			$locale = get_locale();
+			$prompt .= ' ' . sprintf(
+				/* translators: %s: locale code, e.g. pl_PL */
+				__( 'Write the alt text in the language matching locale %s.', 'ai' ),
+				$locale
+			);
 		}
 
 		return $prompt;
diff --git a/includes/Abilities/Image/alt-text-system-instruction.php b/includes/Abilities/Image/alt-text-system-instruction.php
index e05fc1b..624e27e 100644
--- a/includes/Abilities/Image/alt-text-system-instruction.php
+++ b/includes/Abilities/Image/alt-text-system-instruction.php
@@ -31,9 +31,9 @@ Requirements for the alt text:
 - Be objective: Describe what you see, not interpretations or assumptions
 - Avoid redundancy: Do not start with "Image of", "Picture of", or "Photo of"
 - Include relevant details: People, objects, actions, colors, and context when meaningful
-- Consider context: If context is provided, ensure the alt text is relevant to the surrounding content
+- Consider context: If additional context is provided, it is the surrounding article content where the image location is marked with [[IMAGE_GOES_HERE]]. Use it to understand the purpose, subject, and relevance of the image. Describe only information not already conveyed in nearby text. Infer the language from this content and write the alt text in the same language
 - Plain text only: No markdown, quotes, or special formatting
-- If you are given CONTENT in the <additional-context> tag, ensure the alt text you return matches the language of that content. For example, if the content is in English, the alt text should be in English. If the content is in Spanish, the alt text should be in Spanish. If you are not given CONTENT in the <additional-context> tag, ensure the alt text you return is in this locale: {$return_locale}
+- If no additional context is provided, write the alt text in the language matching locale {$return_locale}
 
 For images containing text, include the text in your description if it's essential to understanding the image.
 
diff --git a/src/utils/generate-alt-text.ts b/src/utils/generate-alt-text.ts
index a43f136..2c5b216 100644
--- a/src/utils/generate-alt-text.ts
+++ b/src/utils/generate-alt-text.ts
@@ -51,7 +51,7 @@ export async function generateAltText(
 				: content;
 
 		// Prepare the context.
-		params.context = `What follows is the full article content, where the image has been replaced with the placeholder ${ IMAGE_PLACEHOLDER }. Use the surrounding text to understand the purpose, subject, and relevance of the image within the article. Be sure to describe only information not already conveyed in nearby text. CONTENT: \n\n${ contentWithPlaceholder }`;
+		params.context = contentWithPlaceholder;
 	}
 
 	const response = await runAbility( 'ai/alt-text-generation', params );

Separate concern: esc_html() on system instructions

This is a pre-existing issue but worth noting: system instructions are sent to the LLM API as plain text, not rendered as HTML. Applying esc_html() corrupts them. This affects all abilities, not just alt text.

Summary

• Excerpt works as expected with Polish content when the user's local setting is set to English and Polish.
• Summary works as expected with Polish content when the user's local setting is set to English and Polish.
• With the changes described above, I was able to get into the state when alt text behaves as expected
  • Generates Polish text when the content is in Polish and the text is generated from the block's UI
  • Generates text in the user's local when it gets triggered from the Media Library without the content provided.

[dkotter]

Thanks for the great feedback @gziolo. I think I've addressed it all:

1. esc_html() mangles system instructions. load_system_instruction_from_file() wraps the result in esc_html(), so the LLM receives <additional-context> instead of <additional-context>, "Image of" instead of "Image of", etc. The instruction referencing "CONTENT in the <additional-context> tag" can't match the actual tag in the prompt.

I've removed the esc_html call now: 2069e6b. Open to other suggestions though if we still want some sort of sanitization/escaping here, though not sure it's needed.

2. English preamble wrapping the content — the context sent from generate-alt-text.ts wraps Polish content inside English instructions:
   This English preamble is a stronger language signal than the Polish content that follows, especially for a vision model primarily focused on describing the image.

The reason for this is we've tried to build the underlying Abilities here to be as general as possible, allowing others to build on top of those as desired. Thus the context field could be used for something other than surrounding content and so I added those instructions here instead of the system instructions.

That said, most important to support our own needs so I've removed these additional instructions from the context: 0137b11

And I've updated our system instructions with some of these same details: b2f4c6f

3. Language instruction buried in a bullet point — for vision models, the image is the dominant input. A language instruction buried in the system instruction's bullet list doesn't carry enough weight, especially when mangled by esc_html().
   Put the language instruction in the user prompt — this bypasses the esc_html() issue entirely and gives the language instruction maximum weight:

This makes sense and I've moved this instruction to the user prompt now: 300bfd9

Worth noting some fairly significant changes are coming to the system instructions for alt text generation (see #374) so will need to ensure the changes we make here are ported over to that (or vice versa depending on what gets merged first)

[gziolo]

Everything works as intended. I tested it exstensively and Claude consistently followed all the instructions for alt text, summary, and excerpt 🎉

[jeffpaul]

Holding on merge due to failing e2e tests

[dkotter]

@jeffpaul I see the milestone for this is 0.8.0 but this is ready to go (E2E tests are working now after #376). I don't think there's a reason to hold this until the next release