Assignment Guide: Critical Audit ofa Large Language Model (LLM)
Assignment Overview
In this assignment, you will conduct a critical audit of at least one large language model (LLM) such as Google Gemini, Microsoft Copilot, and/or ChatGPT. The goal is to apply sociological theories discussed in the course to evaluate the LLM’s responses to your devised prompts. This exercise will help you understand the intersection of technology and society, and how sociological theories can be used to critically analyze technological artifacts.
Objectives:
1. Apply sociological theories to a real-world technological issue.
2. Develop critical thinking and analytical skills.
3. Understand the capabilities and limitations of LLMs.
4. Reflect on the societal implications of AI technologies.
Due Date: October 31, by the end of the day (11:59pm)
Details:
● Select a Theory: Choose one or more theories covered in the course.
● Select one or more LLMs: Choose one or more LLMs that you are interested in
auditing (you could audit a single one, or compare more than one, depending on your interests).
● Devise Prompts: Create a set of 3-5 related prompts designed to elicit responses from the LLM(s) that will help you assess an application of the chosen theory or theories (see Example below). Ensure your prompts are clear and relevant to the theory. [Note: the 3- 5 prompts should all be connected to a single general line of inquiry. You are not devising 3-5 distinct assessments of your chosen LLM(s)].
● Interact with the LLM(s): Use the prompts to interact with the LLM(s). Document the responses you receive.
● Analysis: Analyze the LLM(s)’s responses using the chosen theory or theories. Consider the following questions:
o How does the LLM(s)’s response reflect or contradict the theory?
o What assumptions do/es the LLM(s) make about society, culture, or human behavior?
o How do/es the LLM(s) handle complex social issues or nuances?
o What are the potential societal implications of the LLM(s)’s responses?
● Write a Report: Compile your findings into a report. Your report should include:
o An introduction to the chosen theory.
o A detailed analysis of your interaction with the LLM(s), with concrete supporting examples from the prompts and responses.
o A conclusion reflecting on the implications of your findings.
o [Note: The report should be written primarily in prose, although bullet points or other formatting styles may be usedfor emphasis or clarity]
Requirements:
● Each memo should be a maximum of 1,000 words (not including appendices).
● You should attach appendices including all prompts and responses.
● Clearly identify the sociological theory/ies applied.
● Use specific examples from the prompts, responses, and readings to support your analysis.
● When appropriate, use in-text citations and bibliography in ASA format.
o Example in-text: (Bowker and Star 1999:210)
o Example bibliographic reference: Bowker, Geoffrey C., and Susan Leigh Star.
1999. Sorting Things Out: Classification and Its Consequences. Cambridge, MA: MIT Press.
Tips for Success
● Devise Relevant Prompts: Devise a series of related prompts that clearly test an aspect or implication of the sociological theory or theories you are applying (see Example below). Offer justification for your choice of prompts grounded in applying or testing your theory/ies in a particular context.
● Be Specific: Use specific examples from both the readings and the prompts/responses to support your points.
● Stay Focused: Keep your analysis concise and focused on the key points.
● Proofread: Check your work for clarity, coherence, and grammatical accuracy.
Submission Instructions
● Submit your report via the assignment page on Quercus by the due date.
● Ensure your report is in PDF format and properly labeled with your name and the assignment title.