An instrument for measuring Assessment for Learning (AfL) in the classroom

This article introduces educational researchers to the Assessment for Learning Measurement instrument (AfLMi). Based on research by Zita Lysaght and Michael O’Leary at Dublin City University, in partnership with Larry Ludlow at Boston College, the AfLMi consists of 20 statements relating to four key assessment for learning (AfL) strategies recommended in the assessment literature. As designed, the instrument is intended for use in intervention studies designed to measure the extent to which AfL is embedded in classroom practice prior to, during, and following professional development in AfL.

Across the world, Assessment for Learning (AfL), or formative assessment, is recognised as a central component of effective teaching and learning. However, tools to measure the extent to which teachers use AfL practices in classrooms, or to evaluate the extent to which programmes of professional development in AfL are successful in bringing about change, are in short supply. Researchers at Dublin City University and Boston College set out to fill these gaps. This article is focused specifically on the Assessment for Learning Measurement instrument, or AfLMi. The instrument can be used by educational researchers to measure change in classroom practices before and after professional development in AfL.

The Assessment for Learning Measurement instrument (AfLMi)
The AfLMi is based on an earlier instrument created by the researchers called the Assessment for Learning Audit Instrument (AfLAi). Developed using data from 594 teachers from 44 elementary schools in Ireland, the AfLAi consists of 58 statements describing different classroom practices across four key AfL strategies: sharing learning intentions/success criteria (LISC), questioning/classroom discussion (QCA), feedback (FB), and peer/self-assessment (PSA) (see Lysaght & O’Leary, 2017).

The AfLAi is used internationally by teachers and schools to self-evaluate the degree to which AfL practices are embedded in their classrooms based on a five-point scale:

5. Embedded = happens about 90% of the time;
4. Established = happens about 75% of the time;
3. Emerging = happens about 50% of the time;
2. Sporadic = happens about 25% of the time;
1. This Never happens

However, while the AfLAi provides comprehensive data, not all 58 statements are needed for measurement purposes in research contexts. Hence, the team developed a more streamlined instrument that captures the key elements of AfL classroom practices in a manner that meets the validity and reliability requirements of a high-quality measurement tool.

Devising the short-form instrument
In devising the shorter-form instrument (AFLMi), the researchers applied a Rasch model to data derived from the original administration of the AfLAi (58 statements) to determine item difficulties. Based on the outcomes of this analysis, and qualitative judgements to ensure content validity, 20 statements from across the four AfL strategies were selected for use in the AfLMi. These are shown in Table 1. The 20 statements, identified by their acronyms and numbers, are also included in the variable map that was derived from the Rasch analysis (Figure 1).

The AfLMi can be used for research purposes when tracking the ongoing adoption of AfL practices in the classroom.

The AfLMi Variable Map
The variable map in Figure 1 features two grouped distributions: one for the 20 statements (arranged to the right of the map’s central vertical line) and one on the left for the 594 teachers in the study (each # = 4 teachers).

At the bottom of the AfLMi measurement scale are those items that are easy to ‘embed’; in other words, those practices which teachers are likely to employ in their classrooms as a matter of routine. An example might be: Child-friendly language is used to share learning intentions with pupils (LISC3). As we climb up to the top of the scale, we find those classroom practices which are least likely to be embedded in a teacher’s arsenal, for example: Pupils are encouraged to record their progress using, for example, learning logs (PSA2). The researchers found that this arrangement was generally consistent both with extant theory as well as their own classroom observations (made predominantly in the Irish context).

The map also shows high-scoring (highly skilled) teachers in the top-left (ie, those with the most practices embedded in their usual classroom activities), while those less-skilled teachers appear in the lower-left corner of the map.

The result is an effective tool to track the professional development journey of elementary/primary school teachers.

Beside the variable map in the figure are details with respect to raw scores and logit scores (measure). It will be recalled from earlier in this article that a score of 5 on the AfLMi corresponds to a practice that is embedded, a desirable response that indicates an AfL practice that is being implemented frequently in a classroom. A score of 1, on the other hand, corresponds to a Never response, an undesirable outcome. Hence, the maximum raw score possible is 100 (ie, 20 x 5), while the minimum is 20 (ie, 20 x 1). A teacher’s total score can be obtained by summing their self-evaluative ratings (1 to 5) for each of the 20 AfL statements. In turn, group mean scores can be determined by summing the individual total scores and dividing by the number of teachers in the group of interest.

In Rasch analysis, raw (total) scores are converted to logit scores ranging in value from about +3 (items that are extremely difficult to embed/teachers with AFL most embedded) to -3 (items that are extremely easy to embed/teachers with AfL least embedded). The average value is 0.

The horizontal dotted lines in the figure mark cut points in a framework distinguishing between teachers with different levels of skill, and help the researcher to interpret the extent to which AfL practices are embedded in the classrooms of a cohort of interest (the 594 study participants in this case).

The value of the AfLMi to researchers is that it is relatively easy to track change, over time, in teachers’ use of AfL. Moreover, as the variable map derived from a Rasch analysis of AfLMi data (assuming c300 cases or more) facilitates the identification of specific practices that range from least to most embedded, bespoke programmes of professional development in AfL can be designed and evaluated.

Conclusions and future uses
The AfLMi is proposed for use by researchers employing experimental and/or mixed methods designs to evaluate intervention studies in AfL that include teacher professional development as a key component. Because the instrument can be adapted easily for use on on-line survey platforms, it is also an attractive option for use in large-scale, system-wide evaluations. Please refer to Lysaght, O’Leary and Ludlow (2017) for further details.