Recitation 13: Artificial Intelligence

Preparation

Prepare by reading pages 265-276 from Chapter 9 of Blown to Bits.

At recitation, you will test Facbook’s “automatic alt text” feature. Prepare by reading these two articles, and watching the associated videos:

At Recitation

You can work either individually or with a partner.

Earlier in the course, you tested using screen readers on different websites. Screen readers can easily read a webpage’s text, but describing images is more difficult. To describe images, screen readers read an image’s “alt text.” For example, on the Wikipedia page for candy, the authors describe the candy pictured below as “Small, knobby pieces of candy, in different colors” (image credit: Midori).

Small, knobby pieces of candy, in different colors

You can see how this description is included in the HTML code for the image:

<img
alt="Small, knobby pieces of candy, in different colors"
src="//upload.wikimedia.org/wikipedia/commons/thumb/5/54/Kompeito_konpeito.JPG/120px-Kompeito_konpeito.JPG"
width="120"
height="90">

Unfortunately, many website creators don’t take the effort to write alt text, and without alt text, screen readers cannot describe images to visually impaired users.

Machine learning may offer a partial solution to this issue. A machine learning algorithm can process images, then generate descriptions. Facebook has trained an algorithm for this purpose, and has deployed it on their platform.

Display Alt Texts

Normally, you will only encounter alt texts if you use a screen reader (or if you read a website’s code). To easily view alt texts, we will use some JavaScript code to modify the HTML of the websites you visit. To do this, we will use a feature called bookmarklets. Follow these instructions to install the “Alt to Title” bookmarklet in your browser.

Right-clicking on the bookmarks bar opens a menu, containing the option to add a page
In the resulting modal, specify the bookmark's name, and copy-paste the JavaScript code into the URL field
The bookmarklet link will appear in your bookmarks bar, with the name you specified
A screenshot from Wikipedia's 'Candy' page, showing a grid of different images of candy. After clicking the bookmarklet, hovering over an image will cause its alt text to be displayed in a tooltip.

Note: If you are interested in how the JavaScript code works, view this version of the code. The version you copy-pasted has all the whitespace removed, which is necessary for the bookmarklet to run, but makes the code harder to read.

Inspect Facebook’s Automatic Alt Texts

Now that you have an easy way to view alt texts, load Facebook, and inspect the alt texts for the photos you and your friends have uploaded. Answer the questions below.

  1. What fraction of images have alt texts (e.g., out of ten images, how many have descriptions, and how many show “No photo description available” or “Alt text was blank”)?
  2. For the images that have alt texts, what fraction are accurate?
  3. For the images that have alt texts, how would you describe the quality of the algorithmic descriptions (e.g., how would they compare to human-written descriptions)?

Note: To enable the display of alt texts, you will need to click the bookmarklet each time a new image is added to the webpage. An easy way to view many alt texts at once is to load the “Your Photos” gallery from your Facebook profile page.

Inspect Other Websites’ Alt Texts

Next, evaluate the alt texts on other websites. For example, Wikipedia, news websites, and other popular sites. On websites like these, if alt texts are present, they are (usually) written by real people. For each website, record:

  1. The name of the website
  2. What fraction of images have alt texts (e.g., out of ten images, how many have descriptions, and how many show “Alt text was blank”)?
  3. For the images that have alt texts, how would you describe the quality of the descriptions?
  4. Do you think this website should introduce algorithmic alt texts like Facebook’s? Why or why not?

Note: If you run the bookmarklet but you still don’t see alt text tooltips, it could be that the website’s styling prevents the display of tooltips. In this case, test a different website.

Discuss Your Findings

After about 30 minutes, the instructor will gather the class to discuss your findings. Beyond your findings, discuss the promise and limitations of using machine learning to interpret images.

Submit

Ensure that all teammates submit to Moodle. You should both upload a PDF containing:

Template