Link Search Menu Expand Document
ENG 395

Algorithm Audit

  • Due Wednesday, May 5
    • You will submit part 1 of your algorithm audit plan by Tuesday, April 13
    • You will submit part 2 of your algorithm audit plan by Monday, April 26
  • 2000-2500 words (~7-8 pages double-spaced)
  • MLA/Chicago style
  • Turn in via the “Algorithm Audit” portal on Blackboard Assignments page

You will identify an algorithmic process you want to study and design an “audit” of this process, taking into consideration how the process functions, what data it uses, and its most important effects and consequences. Your paper will describe your selected algorithmic process, your audit design, an experimental proof-of-concept audit that you have conducted, and a discussion of your results. You will situate the algorithm you have chosen and your audit process in the context of the class and the major issues we have discussed.

Selecting your algorithm

You have a lot of freedom to select an algorithm or algorithmic process that interests you for this assignment. On page 408 of his article “Algorithmic Accountability,” Nicholas Diakopoulos lists several questions you might consider when selecting an algorithm to examine for this assignment, but the algorithm you select for this assignment does not necessarily have to be “newsworthy.” You will need to research this algorithm and/or the company or platform that created/utilizes it, so you should also keep that in mind when selecting your algorithm. A lot has been written about Google’s search algorithm, for example, while comparatively little has been written about TikTok.

Another aspect of selecting your algorithm is formulating a research question. This research question will form the basis of your algorithm audit (see “Plan part 1” below). An example of a research question might be, “How does the popularity of a tweet affect its position/appearance in followers’ timelines?” Your algorithm audit will be designed to try to answer this question.

Finally, a third important aspect of selecting your algorithm is simply selecting a process that you can design an audit about and that you feel you understand enough – or that you can understand enough, with some research – to investigate more. It’s unlikely that you will be able to access the technical processes underlying your selected algorithm, and even if you could, it’s unlikely that you will be able to understand all parts of it. However, you should select an algorithmic process that you can access in some way (as a user, most likely, though not exclusively), and that you are interested in and able to investigate more.

Designing your audit

You will actually be designing 2 different audits for this assignment. The first, which I refer to as the “blue sky audit,” is the audit you would conduct if you had all of the time, money, skills, and training you needed in order to conduct it. This is what you would consider a potentially “full” audit of your selected algorithmic process – i.e., what it would take, as near as you understand it, to actually approach an answer to your research question. The second audit is your “proof-of-concept audit.” This is the audit you can actually perform yourself as part of this assignment. While this proof-of-concept audit won’t necessarily definitively answer your research question, ideally it will make the case that further study of your selected algorithmic process is needed along the lines you describe (or, alternatively, you will learn through conducting your proof-of-concept audit that a different approach to auditing this algorithm is necessary).

Diakopoulos and Christian Sandvig et al describe methods for conducting algorithmic audits and types of algorithmic audits. Your blue sky and/or proof-of-concept audits may or may not fall into one of these categories, but examining these methods and types may give you ideas about how to design your audit. Additionally, both Diakopoulos and Sandvig et al cite a number of other studies of algorithms/algorithmic audits. You can mine their bibliographies to find further examples. Finally, if you’re looking for ideas or further resources, you may want to check out AlgorithmWatch, which is a non-profit research and advocacy organization that investigates algorithms based in the Netherlands. You can see their projects here.

The parts of this assignment

Algorithm audit plan part 1: due Tuesday, April 13

This is a 1-3 paragraph proposal for your audit. It should include the following things:

  1. A summary of the nature of your interest in this algorithm. I.e., what is your rationale for studying this algorithm? Why does it warrant further study?
  2. Your central research question (or questions: it’s ok if you have several at this point). This question should operationalize your interest in your selected algorithmic process in some way.
  3. A preliminary description of your blue sky audit. How would you go about answering this research question if you had all of the time, money, skills, etc required? This description is preliminary, but you should strive be as specific and concrete as possible when writing it. You are trying to develop a plan: what would the first step be, the next step, etc?

Algorithm audit plan part 2: due Monday, April 26 by class

This aspect of the assignment should include the following things:

  1. An annotated bibliography. It should include 5 sources about your algorithm, your audit design/audit designs in general, and/or related algorithmic processes or issues. These sources can be scholarly, journalistic, written by advocacy organizations, and/or written by the company/service that owns/created the algorithm (or other businesses like social media marketers). There are likely other kinds of sources you might cite, as well. The sources you cite should be geared toward helping you 1) learn about your selected algorithmic process and how it works; and/or 2) design your audits. Each source’s annotation should include 2-3 sentences describing the main argument/focus of that source, and 1-2 sentences describing how or why this source is important to your project.
  2. Revised research question and/or blue sky audit design. Have you changed your mind about what you want to investigate? About how you would structure your blue sky audit? Do you have new ideas about how such an audit might be done? Detail them here. (~1-2 paragraphs)
  3. A description of your proof-of-concept audit process. What is a scaled-down version of your blue sky audit that you can actually conduct? You should be as specific and concrete as possible when writing this description. What will you actually do? (~1-2 paragraphs)

Algorithm audit paper: due Wednesday, May 5

Your paper (2000-2500 words; ~7-8 pages) should include the following things:

  1. A description of your selected algorithmic process and its uses/functions. This description should include a rationale for conducting the audit (i.e., what is important or significant or interesting or noteworthy etc about this algorithmic process?) and the final version of your research question(s) (~1 page).
  2. A description of your blue sky audit design: How would you audit this algorithm in order to answer your research question if you had all the time/money/skills you needed? (~1 page)
  3. A description of your proof-of-concept audit. What did you do to audit this algorithm? Describe your process and observations/findings. (~2 pages)
  4. A general discussion: What did you discover or learn throughout this process and why is this important/interesting/significant/etc? You should tie this discussion to at least one salient issue we have covered in class. Notice that I wrote “issue” here and not “reading” – this is because the issue you discuss may relate to several of our readings from class. While there is no specific number of readings you must discuss, as always, I expect you to be specific in your discussion: tie the issue(s) you write about to specific readings and, if appropriate, to specific passages from those readings.