Developing
an ARB resource
ARB resources are developed to assess aspects of the New Zealand English, Mathematics, and Science curricula at levels 2-5. The original purpose of ARB resources was summative assessment. However, particularly since 2003, the main focus has been for formative purposes.
The NZCER resource development team members are researchers who have expertise in
- at least one learning area (mathematics, English or science)
- assessment.
Most have been classroom teachers.
The development process
The assessment task
An assessment task is drafted by ARB development team members. The draft is then reviewed for:
- curriculum fit
- suitability for level
- clarity of instructions
- layout
- surface features, e.g., punctuation, paragraphing, spelling
- appropriateness of graphics and photographs
Changes are made based on review feedback. The resource may be rejected at this stage if reviewers decide it is unsuitable.
Pre-trial piloting (pen-and-paper resources)
Four to seven resources are grouped into a trial set.
The trial pack is piloted with a sample of 6-10 students. They complete the tasks and discuss each one with the researcher. Adjustments are made to the resources as a result of responses made by the students.
Particular attention is paid to:
- vocabulary used
- the clarity of instructions
- whether students can successfully complete the task
- student feedback.
The resource may be rejected or adapted at this point if students' work demonstrates it is unsuitable.
Trialling resources
There are two main ways resources are trialled.
Pencil-and-paper tasks are sent to approximately eight schools of different deciles from all over New Zealand. They are completed by about 200 students. The scripts are returned to NZCER. Researchers code student responses, and results are analysed and collated. This provides diagnostic data about the students' responses to the task.
Practical tasks are trialled face-to-face with groups or classes of students. Their responses and observations of them completing the task provide the diagnostic data.
Sometimes a combination of these trialling types is used. At other times NZCER researchers work with a class teacher and students.
Coding, marking, and analysing student data.
A code is devised to capture the sorts of responses students make when completing the task.
The student scripts are marked.
The responses are analysed by an NZCER statistician using a computer package called R. This allows the researchers to
- explore different patterns in the data
- identify how often a particular response is made
- establish a broad difficulty level for each part of the task.
Each trial set and its data are reviewed to decide whether
- tasks are neither too easy nor too hard
- the task is valid and reliable
- the order of the questions should be altered
- wording and formatting should be adjusted.
The resource may
- be accepted and published on the Banks after final changes
undergo further development and be retrialled
- be rejected as unsuitable.
Writing Teacher Pages
Teachers Pages are written to support teachers to use their students’ responses to inform next teaching and learning. This is informed by
- the trial data
- researchers’ knowledge of the learning area
- the New Zealand Curriculum.
This material goes through several reviews before the resource is published on the ARB website.
Revising and refreshing older ARB resources
Older resources go through a cycle of review to:
- check their relevance to current directions in teaching, learning and assessment
- check that the context relates to current students' experiences
- update their format.
The reviewed resources may:
- remain unchanged
- undergo minor changes that do not require retrialling
- undergo major changes that require retrialling
- be removed from the banks.
ARB
Home |
New Zealand Council for Educational Research. All rights reserved 
