PILNA cycles up to and including PILNA 2018 used only one version of each test booklet, one for literacy and one for numeracy. For PILNA 2021, there were multiple versions of these tests that students could receive. This was due to the new rotated booklet design, tested in the PILNA 2020 field study and then fully implemented in PILNA 2021.
Rotated booklet designs are those that use multiple versions of test booklets. They are created by establishing groups of questions, called blocks, and putting different sets of these blocks into the test booklets. This results in tests that measure performance in the same topics but use different questions to do so. Moreover, the blocks are inserted in the tests in different orders so as to control for the ‘order effect’ – students’ responses changing between questions close to the start of the test and those close to the end.
This design allowed for more questions to be asked across the sample of students so more information about student performance could be collected. Questions could be asked that covered more content and cognitive skills, had a wider level of difficulty, and elicited more types of responses. The design also enabled twice as many questions to be trialed in the PILNA 2020 field study than previous PILNA cycles.