AI-Generated Alt Text

Screen readers describe images using alternative text (alt text). In this class, you will evaluate the quality of websites’ existing alt text, and compare it to AI-generated alt text.

Preparation

First, read WebAIM’s article about the purpose of alt text, and best practices for authoring alt text.

Next, watch this video explaining how computer vision works.

Finally, read this description of Microsoft’s Image Analysis tool, which you will use in class.

Optional: Consider also reading/watching:

In Class

Try working with a partner, though you can complete the assignment individually if necessary.

Earlier in the course, you tested using screen readers on different websites. Screen readers can easily read a webpage’s text, but describing images is more difficult. To describe images, screen readers read an image’s “alt text.” For example, on the Wikipedia page for candy, the authors describe the candy pictured below as “Small, knobby pieces of candy, in different colors” (image credit: Midori).

Small, knobby pieces of candy, in different colors

You can see how this description is included in the HTML code for the image:

<img
alt="Small, knobby pieces of candy, in different colors"
src="//upload.wikimedia.org/wikipedia/commons/thumb/5/54/Kompeito_konpeito.JPG/120px-Kompeito_konpeito.JPG"
width="120"
height="90">

Unfortunately, some website creators don’t take the effort to write alt text, and without alt text, screen readers cannot describe images to visually impaired users.

Artificial intelligence may offer a partial solution to this issue. An AI algorithm can automatically generate descriptions of images. Among other companies, Microsoft offers AI captioning as a web service.

Step 1: Generate Alt Text Using AI

Microsoft offers a free demo of its image captioning service. Try uploading a few images to the service.

Discuss with your partner: Are the captions correct? Are the captions sufficiently detailed?

Step 2: Configure “AltToTitle” Bookmarklet

Normally, you will only encounter alt texts if you use a screen reader (or if you read a website’s code). To easily view alt texts, we will use some JavaScript code to modify the HTML of the websites you visit. To do this, we will use a feature called bookmarklets. Follow these instructions to install the “Alt to Title” bookmarklet in your browser.

Right-clicking on the bookmarks bar opens a menu, containing the option to add a page
In the resulting modal, specify the bookmark's name, and copy-paste the JavaScript code into the URL field
The bookmarklet link will appear in your bookmarks bar, with the name you specified
A screenshot from Wikipedia's 'Candy' page, showing a grid of different images of candy. After clicking the bookmarklet, hovering over an image will cause its alt text to be displayed in a tooltip.

Note: If an image doesn’t have alt text, the tooltip will show “Alt text was blank.”

Note: Browser plugins can interfere with the display of alt text. If the bookmarklet isn’t working, try disabling your browser plugins.

Note: If you are interested in how the JavaScript code works, view this version of the code. The version you copy-pasted has all the whitespace removed, which is necessary for the bookmarklet to run, but makes the code harder to read.

Note: The bookmarklet can only make changes to the local copy of a webpage stored by your browser. If you refresh a webpage, local changes will be lost. This means that the bookmarklet needs to be run again each time you reload a webpage.

Step 3: Compare Websites’ Alt Text to AI-Generated Alt Text

Next, evaluate the alt text on three websites. I recommend testing a news website (e.g., The New York Times, the BBC, etc.), a website of a large non-profit (e.g., Wikipedia, The Guggenheim, etc.), and another popular site of your choice (e.g., a shopping website, a restaurant’s website, another social media website, etc.).

For each website, record:

Finally, compare each website’s alt text to the AI-generated alt text. Discuss with your partner: Which alt text is better, and why?

Hint: Refer to WebAIM’s best practices for authoring alt text. Also, refer to the examples shown in the template below.

Note: If you run the bookmarklet but you don’t see any alt text tooltip (i.e., you don’t even see “Alt text was blank”), the website’s styling may prevent the display of tooltips. In this case, you can either test a different website, or you can view the source code of the website to view the alt text.

Submit

Upload a PDF containing:

Template

Reflection questions:

Alt text comparison:

Website Image Website's Alt Text AI-Generated Alt Text Which Alt Text is Better?
nytimes.com Representative Jim Jordan speaking to reporters. He is standing at a lectern in front of a seated group. A man standing at a podium with a microphone and people in the background The NYT alt text is better, because it explains who is in the photo.
nytimes.com An illustration shows sketches of people, some clustered together and some solo, in an urban landscape. A drawing of people on a rooftop The NYT alt text is better, because it explains the location and gives more detail. The AI-generated alt text gets the location wrong, and is less detailed.
nytimes.com Three photos show Nishad Singh holding a woman’s hand while walking on a city sidewalk, Gary Wang at a glass door and Caroline Ellison leaving a building. A collage of a man and a woman The NYT alt text is better, because it explains who is in the photos. The AI-generated alt text gets the number of people wrong.
wikipedia.org Small, knobby pieces of candy, in different colors A bowl of colorful candies Wikipedia's alt text is slightly better, because it describes the texture of the candy.
...............
...............
clarku.edu Photo by Clark University from facebook on Wed Oct 04 2023 A group of people sitting around a table Clark's alt text is just a placeholder, and doesn't explain the photo. The AI-generated alt text is better because it describes the photo accurately.
...............
...............

Learning Goals