Enable <picture> element support

[adamsilverstein]

Current in progress PR: #73

Related trac ticket: https://core.trac.wordpress.org/ticket/42920

See this comment for a detailed description: #21 (comment)

---

Original description:

Use picture element to enable modern formats like AVIF with fallback to more widely supported format like WebP or JPEG, including responsive srcset markup like current images. Issue - adamsilverstein/modern-images-wp#1 . references: The anatomy of responsive images - JakeArchibald.com & markup: I just want an image on my page

• Support markup with one or more image fallbacks
• Enable serving WebP/AVIF with JPEG fallback, for more complete browser support.

A good initial test case for this feature would be generating WebP images and JPEG images on upload, then using the picture element to specify the WebP with JPEG as the fallback. Although this scenario isn't likely to be that useful, it will set the path for the same approach with other formats.

[adamsilverstein]

I've been digging into how images are stored and generated a bit further and will post an update here soon with a proposed path forward.

[adamsilverstein]

I started working on a POC in a branch, PR incoming.

Some good reading:

• Responsive images - https://web.dev/serve-responsive-images/
• Thepicture element: https://web.dev/learn/design/picture-element/

Some additional details about my findings and the approach:

Use cases for implementing the <picture> element

The picture element makes sense when the image has more than one mime type, or when the upload type does not match the original type and the user wants to provide a fallback. Picture element could also be used for art directed images.

Use Case 1: uploaded type converted for sub sizes (one mime type, different than original)

Description: User uploads JPEG, sub sized images are WebP. For maximum compatibility, user wants to display WebP in a picture element, falling back to the uploaded JPEG for browsers that don’t support WebP.

Expected output:

<picture>
	<source
		srcset="{WebP srcset}"
		sizes=""
	>
	<img
		alt=""
		src="{original image}"
	>
</picture>

Use Case 2: two mime types

Description: At some point in the future when a user uploads JPEG images, WordPress will be capable of converting these to AVIF images (AVIF support was added in PHP 8.1). To get the best performance and widest compatibility, the user wants to display a picture element showing first the AVIF format to supporting browsers, then falling back to JPEG images for other browsers.

Expected output:

<picture>
	<source
		type="image/avif"
		srcset="{AVIF srcset}"
		sizes=""
	>
	<img
		alt=""
		srcset="{JPEG srcset}"
		sizes=""
		src="{original image}"
	>
</picture>

Use Case 3: multiple mime types

Description: In this case the user wants to display a picture element with src sets for AVIF, falling back to WEBP when AVIF isn’t supported, and finally a fallback to JPEG when WEBP isn’t supported.

Expected output:

<picture>
	<source
		srcset="{AVIF srcset}"
		type="image/avif"
		sizes=""
	>
	<source
		srcset="{WebP srcset}"
		type="image/webp"
		sizes=""
	>
	<img
		alt=""
		srcset="{JPEG srcset}"
		sizes=""
		src="{original image}"
	>
</picture>

Sub sized image generation

When users upload an image to WordPress, the system generates sub sized images for front end display (a sub sized image is created for each available size smaller than the uploaded image, in the same mime type format as the upload). Data about the original image as well as each sub-sized image created is stored in post meta in an array with the key _wp_attachment_metadata. This meta data includes an array keyed on sizes that contains the sub-sized image data, including the file, width, height and mime type. In the future the sub sized images might not be in the same mime format as the original (as is the case when a JPEG is uploaded and WebP is output as the default), or may contain multiple mime type sub sized images, for example AVIF and JPEG images.

The default data for a typical image would look like this after uploading(image_meta truncated):

array (
  'width' => 1500,
  'height' => 1000,
  'file' => '2021/11/road-1.jpg',
  'sizes' =>
  array (
    'medium' =>
    array (
      'file' => 'road-1-300x200.jpg',
      'width' => 300,
      'height' => 200,
      'mime-type' => 'image/jpeg',
    ),
    'large' =>
    array (
      'file' => 'road-1-1024x683.jpg',
      'width' => 1024,
      'height' => 683,
      'mime-type' => 'image/jpeg',
    ),
    'thumbnail' =>
    array (
      'file' => 'road-1-150x150.jpg',
      'width' => 150,
      'height' => 150,
      'mime-type' => 'image/jpeg',
    ),
    'medium_large' =>
    array (
      'file' => 'road-1-768x512.jpg',
      'width' => 768,
      'height' => 512,
      'mime-type' => 'image/jpeg',
    ),
    'post-thumbnail' =>
    array (
      'file' => 'road-1-825x510.jpg',
      'width' => 825,
      'height' => 510,
      'mime-type' => 'image/jpeg',
    ),
  ),
  'image_meta' =>
 	array (  ),
  'original_image' => 'road-1.jpg',
)

Front end image display

WordPress automatically iterates through the post content and adds srcset attributes for images placed from the media library (images placed by URL don't work currently). It uses a regex to find media-library images, then calls ​​wp_get_attachment_image to get each image source. wp_get_attachment_image in turn calls wp_get_attachment_image_srcset to generate the srcset.

The image source code for a typical image looks like this (line breaks for clarity):

<img
	loading="lazy"
	width="1024"
	height="683"
	src="https://wpdev.localhost/wp-content/uploads/2021/12/road-1-1024x683.jpg"
	alt=""
	class="wp-image-2617"
	srcset=
		"https://wpdev.localhost/wp-content/uploads/2021/12/road-1-1024x683.jpg 1024w,
		https://wpdev.localhost/wp-content/uploads/2021/12/road-1-300x200.jpg 300w,
		https://wpdev.localhost/wp-content/uploads/2021/12/road-1-768x512.jpg 768w,
		https://wpdev.localhost/wp-content/uploads/2021/12/road-1.jpg 1500w"
	sizes="(max-width: 1024px) 100vw, 1024px"
>

Detour: image edits and sub sizes

The wp_calculate_image_srcset function, includes code to handle images changed by edits in the media library where each edited file name gets an edit hash appended (eg. “-e1640909072390”) to the filename. Sub sized images of the edited image get the same appended hash so they can be matched to the edited image.

For example if you rotate an image using the media library tools, the edit action creates a series of new files with hash extensions, and the post meta ‘sizes’ array is updated to reference the new files. The edited file urls will then be used for srcset generation. Note that the original_image meta data does not change, and the original image is never edited or removed, WordPress never changes your uploaded image!

Edited image data:

array (
  'width' => 2560,
  'height' => 1920,
  'file' => '2021/12/IMG_1893-1-1-scaled-e1640909072390.jpg',
  'sizes' =>
  array (
    'medium' =>
    array (
      'file' => 'IMG_1893-1-1-scaled-e1640909072390-300x225.jpg',
      'width' => 300,
      'height' => 225,
      'mime-type' => 'image/jpeg',
    ),
    'large' =>
    array (
      'file' => 'IMG_1893-1-1-scaled-e1640909072390-1024x768.jpg',
      'width' => 1024,
      'height' => 768,
      'mime-type' => 'image/jpeg',
    ),
    'thumbnail' =>
    array (
      'file' => 'IMG_1893-1-1-scaled-e1640909072390-150x150.jpg',
      'width' => 150,
      'height' => 150,
      'mime-type' => 'image/jpeg',
    ),
    'medium_large' =>
    array (
      'file' => 'IMG_1893-1-1-scaled-e1640909072390-768x576.jpg',
      'width' => 768,
      'height' => 576,
      'mime-type' => 'image/jpeg',
    ),
    '1536x1536' =>
    array (
      'file' => 'IMG_1893-1-1-scaled-e1640909072390-1536x1152.jpg',
      'width' => 1536,
      'height' => 1152,
      'mime-type' => 'image/jpeg',
    ),
    '2048x2048' =>
    array (
      'file' => 'IMG_1893-1-1-scaled-e1640909072390-2048x1536.jpg',
      'width' => 2048,
      'height' => 1536,
      'mime-type' => 'image/jpeg',
    ),
  ),
  'image_meta' =>
  array (  ),
  'original_image' => 'IMG_1893-1-1.jpg',
)

Note that cropping an image directly in the block editor works differently - a new image is created in the media library with the name “-cropped” and cropped dimensions, all sub-sizes are re-generated for that image.

Enabling additional mime types, a cookbook

How can we enable additional mime types in WordPress to support uses cases 2 & 3?

For example, in use case 3, WordPress would create sub sized images in several formats when users upload images (in any supported format). Sub sized images would be created in AVIF, WEBP and JPEG formats.

Proposed approach

• Internally, the sizes array keys can be any string, so when creating additional images beyond the first mime type - for each mime type we can prepend the mime type to the key names, so a WebP medium image would have the key images/webp-medium.
• wp_calculate_image_srcset would be updated to accept an optional mime type.
  • When a mime type is not included, the function would work as is, except it would filter out any additional mime type image sizes from the meta data.
• The function ​​wp_get_attachment_image would be updated to automatically output a picture element when alternate mime type sub sized images are found in the images post meta.
  • Mime type order and availability for picture element support would be filterable on a per-image basis.
  • When the original image doesn’t match the sub sized images, it could be used as a fallback
  • When multiple mime types are provided, they can be used as source elements for the picture element with the final type used for the fallback image srcset.
• Code must also consider image edits
  • As described above we need to ensure edited images are handled properly.
• Filters should be enabled so developers can adjust each part of the output, especially the source type, srsset and sizes elements. Existing filters should work to enable developer fine tuning, we should validate that and ensure all context is passed (eg mime type).
• Consider whether to require theme support of html5 eg with current_theme_supports( 'html5' )

For the purposes of the plugin we can test approaches by filter the post content.

[ddur]

1. New DOM element node <picture> may break theme or custom CSS selectors. How are you going to fix it?

2. Does <img> element CSS class/style applies only to <img> or to other <source> tags or to <picture> tag?

[adamsilverstein]

Hey @ddur, thanks for the questions:

New DOM element node may break theme or custom CSS selectors. How are you going to fix it?

Good point, we probably want to let themes opt into the behavior so they can adjust CSS accordingly. We could make opting in as simple as checking current_theme_supports( 'picture' ) (as suggested on the trac ticket). Maybe with core image blocks we can improve that by adjusting CSS?

Does element CSS class/style applies only to or to other tags or to tag?

Good question that needs more research; I'm guessing you would need to apply to the picture or both the img and picture elements, the element displayed may depend on browser support for the image format or for the picture element itself.

[adamsilverstein]

Reading up on the various possible uses for the picture element here: https://dev.opera.com/articles/responsive-images/ I created a demo pages with picture elements and linked elements for testing: preview & source

A few notes:

• Chrome only seems to use responsive images correctly when sizes are provided as part of the media query (first example), the second/third examples use srcset with sizes, specifying the source with the type attribute and work in firefox and safari, in chrome dev the responsive image fail to work with this approach, maybe I have something incorrect in my markup? Appreciate any help here!
• I created jpeg, webp and avif versions of images at various sizes.
• I added a "WebP" tag to the WebP images so they are easy to identify. I don't have an editor that works with AVIF to add the tags there.

[eclarke1]

@adamsilverstein is this something you are continuing to work on once back from the break? If so, could we assign the ticket to you please?

[ddur]

How about 'img' element and loading=lazy attribute? Does it apply to picture-source elements?