Working with ChatGPT: Evaluation Essay Student Guide [Strategies]
10-15 minute read
Mary Landry
What You Will Learn in This Section
By the end of this tutorial, you will be able to utilize a specific formation of generative AI (GenAI)—the prominent Large Language Model (LLM) ChatGPT—as an aid within the evaluation essay writing process to
- develop specific assessment criteria
- maintain a professional, unbiased tone
- articulate the sociohistorical context of a subject
Additionally, you will be able to identify specific limitations with using ChatGPT for an evaluation essay, including its limited ability to perform evaluations itself.
What key terms should I know within artificial intelligence (AI) discourse?
Artificial Intelligence (AI)
Artificial Intelligence, often referred to as AI, encompasses computational systems created to imitate human behaviors and cognitive processes. In essence, computer scientists strive to design AI that can augment or replicate specific human cognitive functions, such as problem solving, experiential learning, pattern recognition, and decision making.
Generative AI (GenAI)
Generative AI is a branch of artificial intelligence trained on vast datasets (text, images, audio, code, etc.) in order to generate contextually relevant outputs. ChatGPT is a well-known formation of GenAI designed to generate human-like language in response to a prompt; in fact, the “G” in ChatGPT stands for “generative” to reflect this nature. However, GenAI covers a great variety of generators, such as image generators like Midjourney and DALL-E 2, or voice generators like Eleven Labs. Amid these various formations, ChatGPT belongs to a specific subcategory of GenAI called Large Language Models.
Large Language Models (LLMs)
Large Language Models are a specific subset of GenAI designed to process and generate human language. These models, trained on extensive textual data, can understand, generate, and manipulate text in ways that resemble human communication. LLMs include popular chatbots like OpenAI’s ChatGPT, Microsoft’s Bing, Google’s Bard, and more.
What do I need to know about evaluation as a skill before I begin this tutorial?
A Brief Overview
Evaluation is a skill used in every facet of life, whether you are assessing which career-path to pursue or which cappuccino to order. Despite the ubiquity of this skill, an assignment like an evaluation essay can prompt you to consider subtle details that often go unnoticed in daily decision-making. These subtleties include aspects like developing specific assessment criteria or contemplating a text’s broader socio-historical context. Paying attention to these subtleties within the writing process can contribute to stronger critical thinking and result in stronger essays. To achieve such a result, the following tutorial will explore three specific strategies that demonstrate how a GenAI tool like ChatGPT can assist you in mastering the art of writing an evaluation.
For further information on evaluation as a general skill and the process of writing one, see table 1.
Table 1
Open-Access Resources Related to Evaluation
- Writing Guide with Handbook[1]
- Chapter 7 “Evaluation or Review: Would You Recommend It?”
- Who Teaches Writing?[2]
- Chapter 14 “A Sociolinguist Teaches Evaluating Language”
- Chapter 15 “A Film Scholar Teaches Evaluative Writing”
- Chapter 16 “A Librarian Teaches Evaluating Information”
- Chapter 17 “A Sports Rhetorician Teaches Evaluation”
- How Arguments Work – A Guide to Writing and Analyzing Texts in College[3]
- Section 7.4 “Evaluation Arguments”
What are three ways ChatGPT can support evaluation essay composition?
Developing Specific Criteria
In today’s digital landscape, the thumbs-up symbol is a universal signal for liking something, be it an Instagram post, YouTube video, or Netflix movie. However, such likes are inherently based on individual preferences—one person may like a song for its lyrics while another may like it for its melody. As a result, likability lacks the persuasive power you may need when it comes to convincing someone of your opinion on a subject. Instead, creating an effective evaluation demands critical thinking about the criteria used for assessment.
Here, ChatGPT can provide assistance with brainstorming. It can help you move from genre to sub-genre characteristics and can reframe these characteristics as guiding questions for evaluation. For instance, in the following example under Strategy in Action #1, a user collaborates with ChatGPT to develop criteria for reviewing a horror movie, showcasing this particular type of evaluation skill. In this way, ChatGPT can help you elevate an evaluation from an instinctual judgment to focused, defensible assessment.
Strategy in Action #1
View the following example prompt and response with ChatGPT focused on developing specific criteria.
If you’d like to interact directly with ChatGPT in the above conversation, access the original chat through this link[4] and click the ‘Continue the conversation’ button in the opened dialogue window.
Maintaining a Professional, Unbiased Tone
It is human nature to possess biases. Each one of us is wrapped up in a motley quilt of experiences and beliefs that blankets the way we receive, perceive, and interpret information. Recognizing and navigating these internal biases requires concerted effort, especially when we’re aiming to maintain a professional and unbiased tone in an evaluation. Such a tone is essential for making evaluative writing more persuasive, as it adds credibility and fosters trust among readers. However, achieving an unbiased tone can be challenging due to the same factors that contribute to biases in the first place, such as preexisting beliefs, emotional attachments, and peer pressure.
As a Large Language Model (LLM), ChatGPT is specifically trained to generate language in various tones, making it an excellent resource for providing feedback on the tone of your work. By analyzing the language used in an essay, ChatGPT can offer valuable suggestions to address biased or emotional tones, helping you craft more objective and compelling writing. For example, see Strategy in Action #2 for an example where a user seeks feedback on the tone of an essay.
Strategy in Action #2
View the following example prompt and response with ChatGPT focused on maintaining a professional, unbiased tone.
If you’d like to interact directly with ChatGPT in the above conversation, access the original chat through this link[5] and click the ‘Continue the conversation’ button in the opened dialogue window.
Check out table 2 for additional tips and suggestions related to maintaining a professional, unbiased tone.
Table 2
Tips and Suggestions: Recognize the Internal Biases of ChatGPT Too
Have you ever heard of the concept “garbage in, garbage out”—GIGO for short? This phrase refers to the idea that the quality of a system’s output will directly mirror the quality of the input. In other words, if you were to plug the wrong number into a mathematical equation, you would receive the wrong answer in return.
When it comes to software like ChatGPT, you might want to consider how much “garbage” went into its training. For one, ChatGPT was trained on vast amounts of data from the Internet—including its most shadowy corners. To try and prevent ChatGPT from generating content from these shadowy corners, OpenAI employed Kenyan workers, earning less than $2 an hour, to identify and rate toxic content.[6] Even with these measures—which seem ethically dubious in terms of human rights—there’s still the question of who was deciding what is “toxic” content and what is not? What biases are influencing the human programmers and their choices?
These are only a few examples of the human biases inherent in ChatGPT’s operational design.[7] As such, it is crucial to constantly evaluate every response from ChatGPT for possible biases.
Articulating the Socio-Historical Context of a Subject
When writing an evaluation essay, it is helpful to draw parallels to professional review writing, which is a form of evaluation often encountered in contemporary culture. Much like crafting an academic evaluation essay, a professional critic’s central task involves evaluating a subject using specific criteria. However, the critic also has an important responsibility to guide the audience in comprehending the work within a broader cultural context. For example, reviews of the popular Netflix show Bridgerton—a steamy Regency era drama—have ranged widely between praising the show for having a notably diverse cast and criticizing it for “the negative stereotyping of its Black characters.”[8] Understanding this show against the backdrop of both the Regency era in which it is set (1811-1820) and the 21st century in which it was created helps highlight how this story is not just a period romance, but also a reflection of socio-historical attitudes around race. Consider how evaluating Lord Hastings—an abusive and absentee father—through both a past and present socio-historical lens reveals deeper layers to his portrayal; his obsession with having a suitable heir reflects the Regency-era emphasis on primogeniture[9] while his villainous character reflects the modern-day racist stereotype of being a Black man who abandons his child.
As such, being able to articulate the cultural context of a subject can help you create a more nuanced evaluation. To this end, ChatGPT can serve as a starting point for surveying prevailing cultural norms and historical events that may have influenced the creation or reception of a subject. As an example of this use, Strategy in Action #3 demonstrates a user collaborating with ChatGPT to articulate the significance of Zoom within its socio-historical context.
Strategy in Action #3
View the following example prompt and response with ChatGPT focused on articulating the socio-historical context of a subject.
If you’d like to interact directly with ChatGPT in the above conversation, access the original chat through this link[10] and click the ‘Continue the conversation’ button in the opened dialogue window.
Check out table 3 for additional tips and suggestions related to understanding the socio-historical context of a subject.
Table 3
Tips and Suggestions: Confirm the Accuracy of Socio-Historical Context
It’s important that ChatGPT is used only as a starting point for articulating the socio-historical context of a subject. It cannot be relied upon for accuracy due to its operational design. It was not designed to be a fact-finding search engine; it was designed to respond with the language that is statistically most likely to be correct within the context of a prompt.[11] In other words, it was designed to generate coherent and relevant output in response to your input.
This design is why it’s such a great starting point with this strategy. It can produce the language that is most commonly used to describe the socio-historical context you’re researching. Yet a statistically average response doesn’t reflect the full range of ideas and opinions about various socio-historical contexts, so it is important to confirm the accuracy of ChatGPT’s response by doing further research on your own.
What’s a pitfall to avoid when collaborating with ChatGPT on an evaluation essay?
Great for Skill Support, but Not so Great at the Skill Itself
While ChatGPT can provide help with writing an evaluation essay, it’s worth noting that ChatGPT is not very skilled at evaluation itself. This deficiency stems from the inherent subjectivity often involved with most forms of evaluation. As a computer software incapable of its own opinions, ChatGPT cannot offer such subjectivity in its responses. However, within its design as a LLM, ChatGPT can perform specific evaluation tasks.
Specifically, an LLM is engineered to identify patterns in language, which enables it to respond coherently to prompts. Within this process of pattern identification, ChatGPT will conduct sentiment analysis to determine whether the words in a user’s input correlate to patterns of positive, negative, or neutral sentiments. For example, ChatGPT would likely evaluate the use of words such as “fantastic” and “top-notch” as indicators of positive sentiment, and this evaluation would enable it to respond appropriately to such positive expressions. Thus, ChatGPT does possess evaluative capabilities, albeit within narrow confines.
With such limited skills in evaluation, it’s important to know that simply feeding an assignment prompt for an evaluation essay into ChatGPT likely won’t yield helpful results. This limitation is why this tutorial aims to upskill your writing with specific strategies for utilizing ChatGPT’s strengths while avoiding its weaknesses.
Learning Outcomes for ENGL 1302: How does this tutorial apply to state standards?
This tutorial is designed to support your success in ENGL 1302 (Composition II) by aligning with the student learning outcomes established by the Texas Higher Education Coordinating Board. As highlighted in Table 4 below, this tutorial directly addresses three key standards. By aligning with these specific learning outcomes, this tutorial not only provides you with skills and knowledge that transfer across diverse learning environments but also gives your education value outside your institution.
Table 4
Applicable State Learning Outcomes
- Demonstrate knowledge of individual and collaborative research processes. By collaborating with ChatGPT through any or all of the three strategies demonstrated in this tutorial, students will be to demonstrate knowledge of collaborative brainstorming, research, and editing within the writing process.
- Develop ideas and synthesize primary and secondary sources within focused academic arguments, including one or more research-based essays. In working with ChatGPT to understand a subject within its socio-historical context, students will be empowered to locate and synthesize relevant primary and secondary sources into more focused and well-supported evaluations.
- Write in a style that clearly communicates meaning, builds credibility, and inspires belief or action. Utilizing ChatGPT for support in maintaining a professional, unbiased tone will help students be able to compose an evaluation that is more persuasive and impactful.
Attribution:
Landry, Mary. “Working with ChatGPT: Evaluation Essay Student Guide.” Strategies, Skills and Models for Student Success in Writing and Reading Comprehension. College Station: Texas A&M University, 2024. This work is licensed with a Creative Commons Attribution 4.0 International License (CC BY 4.0).
- Robinson, Michelle Bachelor, et al. Writing Guide with Handbook. OpenStax, 21 Dec. 2021, https://openstax.org/details/books/writing-guide Accessed 21 Sept. 2023. ↵
- Branson, Tyler, et al. Who Teaches Writing? Oklahoma State University Libraries, 2022, https://doi.org/10.22488/okstate.22.000003. Accessed 21 Sept. 2023. ↵
- Mills, Anna. How Arguments Work - A Guide to Writing and Analyzing Texts in College. Humanities LibreTexts, 6 Oct. 2019, https://human.libretexts.org/Bookshelves/Composition/Advanced_Composition/How_Arguments_Work_-_A_Guide_to_Writing_and_Analyzing_Texts_in_College_(Mills). Accessed 21 Sept. 2023. ↵
- “Developing specific criteria" prompt. ChatGPT, OpenAI, 21 July 2023, https://chat.openai.com/share/4d48d0b2-bdf6-4eaf-b372-beea1a7d9ea3. ↵
- “Maintaining a professional tone" prompt. ChatGPT, OpenAI, 21 July 2023, https://chat.openai.com/share/544970e9-46e7-41f6-a7e5-bfe1e8ccdc4a ↵
- Perrigo, Billy. "OpenAI Used Kenyan Workers on Less Than $2 Per Hour." Time, 18 Jan. 2023, https://time.com/6247678/openai-chatgpt-kenya-workers/. Accessed 21 Sept. 2023. ↵
- Cousins, Barry. “Council Post: Uncovering The Different Types Of ChatGPT Bias.” Forbes, https://www.forbes.com/sites/forbestechcouncil/2023/03/31/uncovering-the-different-types-of-chatgpt-bias/. Accessed 21 Sept. 2023. ↵
- Hinds, Carolyn. “‘Bridgerton’ Sees Race Through a Colorist Lens.” Observer, 1 Jan. 2021, https://observer.com/2021/01/bridgerton-sees-race-through-a-colorist-lens/. Accessed 17 September 2023. ↵
- Primogeniture is a social practice, especially prevalent in the 18th and 19th centuries, where the eldest son inherits the entire estate or title of his parents upon their death. ↵
- “Articulating the socio-historical context of a subject" prompt. ChatGPT, OpenAI, 21 July 2023, https://chat.openai.com/share/0d1dd502-72dc-45e0-b459-386c5dc62d6f ↵
- @dsmerdon. “Why Does ChatGPT Make up Fake Academic Papers? By Now, We Know That the Chatbot Notoriously Invents Fake Academic References. E.g. Its Answer to the Most Cited Economics Paper Is Completely Made-up (See Image). But Why? And How Does It Make Them? A THREAD (1/n) 🧵 Https://T.Co/KyWuc915ZJ.” Twitter, 27 Jan. 2023, 9:42 PM, https://twitter.com/dsmerdon/status/1618816703923912704. Accessed 21 Sept. 2023. ↵