Rationales in real time: validating design decisions in Software Design and Architecture

Teaching context Educator Name(s): Dr Eduardo Araujo Oliveira and Dr Maria Rodriguez Subject Name: Software Design and Architecture (SWEN90007) Discipline: Software Engineering Faculty: Faculty of Engineering and Information Technology Year Level: Graduate coursework Class Size: Approximately 150 students Teaching Period: Semester 2 Mode(s) of Delivery: On-campus (each week involves a two-hour lecture and a one-hour workshop) Type of IOA: Artefact plus scenario, with marker-led IOA. Conducted in small groups.

Overview of the IOA

Background and Rationale

In-person IOAs were introduced into this project-based subject in Semester 2, 2025, as a response to ongoing discussions about assurance of learning and the increasing use of Generative Artificial Intelligence (GenAI) tools in Software Engineering education.

The IOAs simulate a professional client consultation relating to design artefacts, replicating a real-world workplace scenario. This approach evaluates not only students' technical reasoning but also their professional communication, collaboration skills, and project design artefacts. During the IOAs, students are required to demonstrate their understanding through artefacts such as software architecture diagrams, coded software patterns (and rationales behind their choices), testing approaches, and project reports. This allows markers to validate the rationale, coherence, and consistency between the team's proposed design and its implementation.

A structured and dialogic approach is used to support authentic assessment of both technical competence and professional readiness. It also encourages students to articulate their ideas clearly, justify their architectural decisions, and critically evaluate their work.

By simulating a realistic team-based environment, these IOAs prepare students for the challenges they will face in their future careers as software engineers.

Context and Scenario

These summative IOAs occurred at three different periods throughout the semester (Weeks 5, 9 and 12).

They were scheduled outside of class time and took place in meeting rooms at the Melbourne Connect building. In rare cases where students were unable to attend face-to-face, Zoom was used as an alternative.

During the IOA, students roleplay as members of a software design team, presenting and defending their architectural decisions to a ‘client’ (played by the marker). The IOAs were conducted in small groups but students were expected to answer individual questions within this team-based setting.

Each team was given a collection of questions (one at a time), and one student would volunteer to answer each question. Their response represented the group's collective understanding, as teams were expected to collaborate and distribute work evenly. Subsequent questions were directed to other members, ensuring all students had equal opportunity to demonstrate individual understanding.

If a student could not answer a particular question, that individual would receive 0 for that component, and another student would have the chance to answer it (representing the team).

Design Features

Each student participated for approximately 15 minutes, resulting in a 60-minute interactive session per team, followed by a 15-minute moderation period for assessors to complete notes and finalise feedback on a new marking sheet (i.e. 1 hour and 15 minutes per IOA session). This amounted to 45 minutes of face-to-face presentation per student for the entire semester, across the three sessions. The IOAs were collectively weighted at 92% of the overall mark for the subject (the remaining 8% of the weighting was attributed to a diagram design submitted in Week 3) Marking rubrics for the IOAs were co-designed with the entire teaching team.

To ensure consistency in assessment, all markers observed an IOA conducted by either the subject coordinator or head tutor. This initial session helped establish shared expectations for the process. When a marker led an IOA for the first time, a subject coordinator attended to provide feedback and support, fostering confidence and competence.

Implementation Details

The IOA was assessed by a single marker who documented feedback in a spreadsheet, which was later shared via the LMS to ensure equitable presentation time for all students. All IOAs were also recorded to support moderation processes.

The moderation process was comprehensive and multi-layered, designed to maintain fairness, consistency, and reliability across assessors and teams. Following each IOA, assessors had 15 minutes to finalise notes, cross-check scores against the rubric, and ensure alignment between verbal responses, documentation, and assessment criteria.

Cross-moderation meetings among teaching staff included:

1. Pre-IOA calibration discussions, where assessors reviewed sample responses and marking exemplars to standardise expectations
2. Spot checks of marking sheets by the subject coordinator, to identify any inconsistencies across tutors.
3. Post-assessment moderation of borderline cases or discrepant cases were reviewed collaboratively to ensure consistent interpretation of the criteria

This rigorous approach was designed to maintain assessment integrity and inter-rater reliability while fostering staff development in conducting and evaluating interactive oral assessments.

Reflections and outcomes

Following the redesign and implementation of IOAs in this subject, the overall End of Subject Survey score has improved—from 3.58 in 2024 to 4.42 in 2025. One student provided the following positive reflection in the survey, stating: “The group assessments were engaging and interesting, and I really enjoyed working through tasks with my team.”

Recommendations for other educators

For educators considering implementing IOAs, the coordinator provides some key recommendations. Firstly, co-designing the assessment rubric collaboratively with markers is essential, as it promotes shared understanding of criteria, aligns with learning outcomes, and clarifies performance evidence, ultimately strengthening inter-marker reliability.

Secondly, ensure that most markers observe at least one or two IOA sessions led by the subject coordinator or an experienced marker before conducting their own, as this helps calibrate standards and familiarise staff with the assessment process (e.g., timing, flow, expectations). After the IOAs have been implemented, conduct a ‘lessons learned’ reflection session with all markers to document feedback and refine the process for future iterations.

Finally, communicate expectations clearly and early with students, including practical reminders about equipment (e.g., bringing laptop chargers), punctuality, and presentation etiquette (e.g., do not dominate the meeting). These steps collectively contribute to smooth-running sessions and maintain fairness across teams.