Developing Students’ AI Literacies as part of a Research-Focused Assessment Task

Developing Students’ AI Literacies as part of a Research-Focused Assessment Task

Teaching Context

Educator’s name: Dr Solange Glasser and Dr Julian Harris
Subject name: Music, Mind & Wellbeing
Discipline: Music
Faculty: Fine Arts and Music
Year level: Level 2 undergraduate breadth, elective, and core subject
Class size: 737 enrolments in Semester 2, 2023
Mode of delivery: Online only

Details of Assessment Redesign

What was the previous assessment design?

The subject previously included the following assessment tasks:

  1. A research report (individual task, due Week 1 of the Exam Period, 50%)
  2. Eight online annotations of readings (individual task, completed throughout the semester, 50%)

What were the main issues with the previous assessment design?

The research report task was considered to be vulnerable to academic misconduct through the use of generative AI tools. There were two main perceived reasons for this:

  1. Some students completing this subject might not have any prior experience of completing a research essay or report; and
  2. Students are often faced with significant assessment stress at the end of semester due to the prevalence of heavily weighted assignments at this time.

What was the main aim of the assessment redesign?

Aside from reducing the potential for academic misconduct, a key objective of the assessment redesign was to develop students’ understanding and skills pertaining to the use of generative AI tools, and ChatGPT in particular.

How was the assessment redesigned?          

For reasons of equity and privacy, students are still able to complete the original research report task, which is now called the ‘Conventional Research Report’. However, they are also now able to opt-in to an equivalent, process-oriented assessment task, called the ‘Contemporary Research Report (with AI-support)’, which comprises two core components:

  1. A Final Research Report submitted in Week 1 of the Exam Period (this has been reduced to 1250 words rather than 2000)
  2. A Final Reflective Journal, featuring three smaller written reflective activities that are completed in Weeks 4, 7, 9.

Both components are individual submissions, but students are required to collaborate with their peers and generative AI tools to complete the Final Reflective Journal activities.

Essentially, this redesigned task integrates generative AI use into the research process to strengthen students’ AI literacies and academic integrity practices. More specifically, a nested design incorporating a sequence of graded and ungraded weekly discussion activities is used to develop students’ generative AI prompt engineering awareness and strategies. The emphasis of the assignment is on process rather than product. For example, the Week 4 activity requires students to use Perusall to collaboratively read and annotate a resource on prompt engineering for generative AI. They are then asked to engineer their own ChatGPT prompts to help them develop an aspect of their Final Research Report. The final aspect of the activity is to write a brief 250-word critical reflection on whether the information in the prompt engineering resource was useful in helping them develop their ChatGPT prompts.

What are the benefits or impact of the new assessment design? Is there any evidence to support these benefits or impact?

The design of the AI-enhanced task is thought to better align with Advancing Students and Education strategic priorities around inclusive, student-centred, and innovative assessment design. It also provides students with a firsthand, scaffolded experience in using generative AI tools in appropriate and ethical ways. The team have ethics approval for a study which aims to evaluate the impact of this novel design.