Redesigning a final exam to reduce surface learning and academic misconduct

Redesigning a final exam to reduce surface learning and academic misconduct

Teaching Context

Educator’s name: Dr Eduardo Oliveira
Subject name: SWEN90007 - Software Design and Architecture
Discipline: Software Engineering
Faculty: Faculty of Engineering and Information Technology
Year level: Year 3 of the Master of Software Engineering program
Class size: 80 – 100 students
Mode of delivery: On-campus

Details of Assessment Redesign

What was the previous assessment design?
Prior to the redesign, the assessments for the subject were:

1. A multi-staged, pair-based project, completed throughout the semester (40%)
2. An  invigilated in-person written examination, completed during the examination period (60%)

What were the main issues with the previous assessment design?
The in-person invigilated exam transitioned to an open book online format in 2020 (i.e., during COVID-19) and there were concerns that students would be able to use online tools to cheat.

What was the main aim of the assessment redesign?
To improve the design of the exam so that it focuses more on assessment for learning rather than assessing the memorisation of facts.

How was the assessment redesigned?
After the redesign, the assessments for the subject were:

1. A multi-staged, pair-based project, completed throughout the semester (40%)
2. An open-book online written examination, completed during the examination period (60%)

As shown above, the weighting, timing, and type of assessments for the subject remained the same, but the exam shifted from an in-person delivery to open-book online delivery. In preparing for this shift, Eduardo and his team completely redesigned the exam so that any questions that were asking students to list, describe or explain various common concepts in software design and architecture were replaced with questions presenting imaginary scenarios, problems, or software designs.

According to Eduardo, ‘the [original exam] questions were about definitions, concepts, and terms that we teach, but they were not really [asking students] to apply that knowledge’. After the redesign, students were required to apply their knowledge from the subject to problem solve and design solutions in order to answer the questions. As Eduardo explained, ‘we really wanted [students] to see and do something rather than just memorize.’

What are the benefits or impact of the new assessment design? Is there any evidence to support these benefits or impact?
Eduardo explained that exam questions which require students to describe and define concepts make it easy for them to use AI or Google to seek their answers. More specifically, the original exam questions were asking about ‘really well-known principle[s]… [which makes it] kind of easy to memorize or to find the first link on Google or Wikipedia… so ChatGPT will know what we're talking about’. In contrast, the redesigned exam questions made it very difficult for students to cheat using AI or the internet because they were based around imaginary scenarios and presented unique problems that did not exist online or in ChatGPT’s training material. In other words, devising exam questions that ask students to apply, synthesise, or evaluate concepts to solve novel problems that don’t exist in the real world limits the potential that AI or Internet based search tools can be used for cheating or plagiarism.

Eduardo commented that the revised exam is ‘much better because we had less…collusion. For example to exchange the answers, you'd need to take photos, you’d need to write a lot, it's very personalized. There is not only one solution [either], you can design [answers] in different ways… it's very hard even to input all of the details that we provide the specificities of the system, which is a combination of criteria requirements and diagrams to ChatGPT’. To ensure that this was the case, Eduardo tested whether ChatGPT could complete the newer version of the exam questions. He found that ‘ChatGPT was scoring 10 out of 10 in our 2019/2020 exam, [but] in 2022 I couldn't even prompt my questions to ChatGPT, the way we redesigned [them]’.

Featured Assessment Strategies

By including more questions that focused on authentic (albeit imaginary) scenarios, problems and software designs, and asking students to apply their knowledge of software design and architecture, the redesigned exam in this case study provides an example of one of the assessment strategies: Incorporate more authentic, context-specific or personal assignments.