In encrypted room, full-size image downloaded immediately when thumbnail exists

[richvdh]

0. Open browser dev tools
1. Receive an encrypted m.image message with a regular image and a thumbnail; do not yet open the full-size image.
2. Note that the full-size image is downloaded as soon as the message appears on the timeline. This seems a bit redundant, when only the thumbnail is shown on-screen.

[t3chguy]

This is done intentionally when the mimetype may be animated (e.g. image/png), we have to load the blob to check if it is animated to render the GIF overlay. Related #26467

[richvdh]

This is done intentionally when the mimetype may be animated (e.g. image/png), we have to load the blob to check if it is animated to render the GIF overlay.

I'm afraid I don't really follow. Can you explain like I'm 5?

(naively: if clients have to download the full-sized image, maybe we don't need thumbnails?)

[t3chguy]

Design has asked that we show a "GIF" label in the corner of animated media, so that a user can hover/open it to play the animation.

When we encounter an image with a mimetype which may be animated ("image/gif", "image/webp", "image/png", "image/apng", "image/avif") we have to fetch the blob (because thumbnails of animated media are often not animated, e.g. Synapse) to inspect the file itself to see if it contains an animation rather than a static image to be able to do so. (Otherwise all PNGs would be shown as animations, because image/png can be animated)

[t3chguy]

FTR the codepath isn't specific to encrypted rooms:

element-web/src/components/views/messages/MImageBody.tsx

Lines 278 to 321 in 0899165

	let isAnimated = mayBeAnimated(content.info?.mimetype);
	// If there is no included non-animated thumbnail then we will generate our own, we can't depend on the server
	// because 1. encryption and 2. we can't ask the server specifically for a non-animated thumbnail.
	if (isAnimated && !SettingsStore.getValue("autoplayGifs")) {
	if (!thumbUrl || !content?.info?.thumbnail_info || mayBeAnimated(content.info.thumbnail_info.mimetype)) {
	const img = document.createElement("img");
	const loadPromise = new Promise((resolve, reject) => {
	img.onload = resolve;
	img.onerror = reject;
	});
	img.crossOrigin = "Anonymous"; // CORS allow canvas access
	img.src = contentUrl ?? "";
	try {
	await loadPromise;
	} catch (error) {
	logger.error("Unable to download attachment: ", error);
	this.setState({ error: error as Error });
	return;
	}
	try {
	const blob = await this.props.mediaEventHelper!.sourceBlob.value;
	if (!(await blobIsAnimated(content.info?.mimetype, blob))) {
	isAnimated = false;
	}
	if (isAnimated) {
	const thumb = await createThumbnail(
	img,
	img.width,
	img.height,
	content.info?.mimetype ?? "image/jpeg",
	false,
	);
	thumbUrl = URL.createObjectURL(thumb.thumbnail);
	}
	} catch (error) {
	// This is a non-critical failure, do not surface the error or bail the method here
	logger.warn("Unable to generate thumbnail for animated image: ", error);
	}
	}
	}

[richvdh]

This feels a bit of a cumbersome solution to the problem. How about having the sender flag the media as animated or not?

[t3chguy]

Sure that would be better, but for an indefinite transition period until the majority of bots, bridges, and clients do so we would need to keep this solution

[richvdh]

FTR the codepath isn't specific to encrypted rooms:

It really didn't seem to happen in plaintext rooms, though it's possible I was just failing at computers