Flexible assessments? We will lose our accreditation!

Introduction

In The Netherlands, Universal Design for Learning is gradually making its entrance into higher education. The UDL motto is: ‘firm goals, flexible means’. However one of the major questions in higher education is how to apply this motto to assessment. In other words: are flexible means of assessment possible if we adhere rigorously to assessment goals, in particular if quality assurance plays an important role? In this article we will deal with this question by using a case study of the Maastricht Science Programme where a variety of assessment forms are used in a very extended way. We will also share the feedback of the Accreditation Organisation for The Netherlands and Flanders (NVAO) on this case.

Firm goals, flexible means and quality assurance

For a number of years now we have assisted lecturers and educational staff to develop curricula that create an optimal learning experience for every learner by using Universal Design for Learning (UDL). In the last couple of years we have seen lecturers making great strides forward. Creating firm goals makes sense to them, providing multiple means of representation is also very readily understood, and even raising the engagement of students is easy once lecturers understand that they need to do more than only assuming that every student should motivate themselves. However, in every training and coaching session, we invariably encounter hesitance and even reluctance to work on flexible assessments.

We are often confronted with statements to the effect that ‘flexible assessments are not allowed, we will lose our accreditation if we make assessment flexible!’. It is true that the quality assurance system of the NVAO contains many standards, which are all taken into account in a quality assessment procedure. (The NVAO assures the quality of Higher Education in The Netherlands and Flanders.

It is an independent organisation established by a treaty between The Netherlands and Flanders. The NVAO works with a quality framework. To receive accreditation study programs need to meet the standards in this framework, www.nvao.com.)

However, the NVAO states clearly that the review is done using an ‘appreciative approach’, which means the starting point in applying the accreditation framework is the model chosen by the institution. In addition, Paul Zevenbergen (NVAO executive board member) states that ‘NVAO does not have specific guidelines with regard to assessment, except that assessments needs to be valid and reliable, need to have sufficient variation, and need to address the learning objectives’. This is because institutions must indicate their own final level of graduation, and therefore also what their assessment looks like. It is important to explain, motivate and comply, but there are many opportunities for flexibility’.

The assessment cycle: 7 steps to assure quality

The good news is that the accreditation system leaves enough room for study programs to create flexible assessments as an integral part of their curriculum. How now to ensure the quality of this flexible assessment? You can do so by going through the seven phases of the assessment cycle:

• basic design
• construct test matrix
• construct test establish norm
• conduct test
• grade process and analyse
• register result
• evaluate and improve

Phase 1: Basic design

The first step of this cycle is to create a basic design of the test. This test needs to be in line with the learning outcomes that are set for the course. The contents of the test, level and form must match the educational vision and the didactic concept of the institution. The basic design gives an initial idea of what you want to achieve with the test.

Phase 2: Construct test matrix

A test matrix is a table showing how the tasks are divided into the test, in relation to the stated learning outcomes. This is a very useful instrument in the design of a test. For each topic, one determines at what level it is tested, what types of test are the best for that topic and the number of questions to be asked about it. A test matrix helps to create a test that properly represents the course. The next step is to create a first test. https://toetsing.sites. uu.nl/modules/toetsmatrijs/theorie/

Phase 3: Construct the test, establish norm

If you have a rough draft of the test, the development of the test can take place like compiling questions, formulating tasks, collecting relevant supporting material, and describing case studies. Once the test is done, you can establish the norms to assess the outcomes of the test.

Phase 4: Conduct the test

In this phase it is important to make sure you create appropriate conditions, so that the result of the test actually reflects the abilities of the students. Be aware that, especially in UDL, assessment is an ‘essential, embedded feature of the learning process’ (Meyer, Rose and Gordon, 2014). According to Rose (op cit) in a UDL curriculum ‘the most important kind of assessment is formative assessment’.

Phase 5: Grade process and analyse

Once the test is made, it is assessed based on the set norms. This will provide insight into the difficulty, reliability and validity of the test and the test components. This analyses may give rise to the revision of the norms. Each student receives a final score. In The Netherlands, by law, the examination board determines the final test results. (The Dutch Law on Higher Education and Scientific Research (WHW) requires that every study program has an examination board that assures the quality of the tests and examinations.)

Phase 6: Register result

The test results are administered. The student will receive appropriate feedback on test result. Make sure that the feedback is understood by the student.

Phase 7: Evaluate and improve

In this phase you evaluate a number of tests on their quality in interdependence. You can for example look at difficulty, reliability and validity. An important element to assure quality is the confrontation of tests with empirical data (quantitative and / or qualitative), such as evaluations of the testing or analysis of the test results themselves. This helps you to improve the test and detect vulnerabilities. This in combination with peer consultation can help you to create the next test. With that new test you can start a new assessment cycle.

Flexible assessment in real life: a case study of the Maastricht Science Programme

The next step is to really create flexible assessment. What would that look like? Luckily, Dr. Roy Erkens, a lecturer at the Maastricht Science Programme was very curious about the answer to that question too. He followed a 10 week UDL online course at UDL Nederland and he decided to apply the concept of flexible assessment to his course ‘Tropical Ecology’. This is one of the third year courses he teaches at the Maastricht Science Programme. He gave his students total freedom to demonstrate their grasp of the learning goals of the course.

This means that he let them design their own test. He allowed every test format they came up with as long as they could explain to him how they demonstrated the learning goals by their chosen format (phase 1). He used test matrices of previous years to check the representativeness and reliability of the ideas his students came up with (phase 2). He asked the students to create the criteria which he should use to assess the quality of their work (phase 3). It is noteworthy that Dr. Erkens was pleasantly surprised by the high bar that the students set for themselves. The formats that the students chose were very diverse: a calendar, an exhibition for a museum, an information leaflet, and a written exam. Once Dr. Erkens and the students were satisfied with the format and criteria they signed a short contract to create transparency about the decisions they made and to be able to monitor to innovative journey that they were going to make together.

All tests worked towards a final product. However Dr. Erkens embedded formative assessment in the learning process by means of planned conversations with the students about their progress to give them feedback during the process (phase 4). The students realised during the process that they set quite high standards for themselves. It struck Dr. Erkens that they were very motivated to reach these standards; “we have raised the bar for ourselves, now we are going to get there no matter what!” In the end Dr. Erkens graded the test and gave the students feedback on their final results, registered the results and evaluated the process with the examination board. This led to the conclusion that he would continue with flexible assessment for this subject next year (phase 5, 6 and 7).

Important factors in the conclusion were the very high intrinsic motivation of the students, the students tendency to aim higher and the fact that the way of working caused a lot of extra joy in teaching the students for Dr. Erkens, but not a lot of extra work. In Dr. Erkens own words “for me it came down to more fun, less work”. We interviewed Dr. Roy Erkens on this work and you can watch the interview on the website of UDL Nederland (English subtitles).

Just before the release of this article Dr. Erkens informed us that he actually repeated the same format this year and he could tell us that the intrinsic motivation of the students was again very high and the results of this year were even more impressive. One student even created a scale model of a tropical rain forest complete with information on the subject (Image 3).

Response of the Accreditation Organisation of The Netherlands and Flanders (NVAO)

We were very curious what the opinion would be of the NVAO on this example of flexible assessment. We asked NVAO executive board member Paul Zevenbergen to analyse this case study. His spontaneous reaction was that this was a very inspiring example of flexible assessment. He encouraged Dr. Erkens to continue on this path and gave the following advice to assure quality:

1. Discuss with your peers. Preferably also from another field. ‘NVAO does not have deep expertise in any particular scientific field, and therefore relies on evaluation by peers.’
2. Without specific standards for an assignment, you need an extra pair of eyes for grading. Sometimes you need a fresh pair of eyes to put things into proper perspective. This is how you calibrate.
3. Build in checks and balances. This way you increase your critical assets.
4. Make your assessment imitable (not necessarily measurable but replicable/ traceable). What are you doing exactly and why? Explain!
5. Involve the Examination Board of your institute. They are accountable and they may provide the extra pair of eyes.

Conclusions

In this article we have shown that flexible assessment and assuring quality is possible. We even argue that flexible assessment improves the quality of the learning process, because it forces you to really assess the starting point of development of the learning process: it becomes an integral part of your course right from the start. In case full flexibility is a bridge too far yet, you can always start with giving choice between two options. Depending on the goal of the test this can be a choice in subject or method. Finally, the Assessment Cycle is a very helpful tool to keep you on track and to explain what you did and why during internal and external quality reviews.