Assessing Spoken Language Ability: A Many-Facet Rasch Analysis

Abstract

Assessing speaking in a useful way has been attended with some concerns, such as scoring subjectivity and test bias. Most often, scoring the speaking performance might result in some unfairness that could possibly emanate from these two issues. This, in turn, could harm the life of test-takers and many other stakeholders. This article investigated the assessment of speaking among learners of English in an ESP program in an EFL context. To this end, 213 test-takers were assessed on their speaking ability by 12 raters using a five-rubric scale (task achievement, fluency, grammar, vocabulary and phonology). Each candidate was assessed by two raters, totaling six pairs of two raters each). The speaking exam included four parts only of which two parts were graded, thus excluding the opening and closing sections. All the exam questions were pre-formulated and teachers were instructed to stick to the frame of the exam. The results of the study showed that generally the test-takers’ speaking ability was scored more leniently than harshly and that raters were biased towards the speaking rubrics, which indicated a fuzzy idea about such rubrics. Assuredly, the different statistical tests of the FACETS showed that the speaking exam was neither valid nor reliable. The study had relevant implications for rater training in how to score speaking in an objective way and for recommending writing a list of test specifications (specs) to design useful and fair speaking exams in similar-related contexts.

Appendices

Appendix 1 Notes for Oral Assessors

There are two distinct tasks and it is important that each candidate gets a fair chance at both. Timings and question types should be adhered to strictly.

The mark sheet is designed to permit separate task-by-task marking. This is the preferred method as the tasks have different focuses. However, this does place a strain on the assessor who has to make four separate assessments (Candidate A task 1, Candidate B task 1, Candidate A task 2, Candidate B task 2). If assessors find this unwieldy they may mark holistically, i.e., give marks for the two assessed tasks overall.

The speaking topics are arranged so that the paired discussion questions are related to the individual question boxes to the immediate left of them.

Appendix 2 Paired Format

1.1 Interlocutor

1.1.1 Assessor

Phase A	Introduction	Not assessed
Phase B	Task 1	Assessed
Phase C	Task 2	Assessed
Phase D	Closing	Not assessed

Phase A	Introduction
Interlocutor • Good morning/afternoon • In this interview, we are going to ask each of you some questions relating to the topics you have studies so far this term. You should direct your answers to your partner. After that, we will introduce a similar topic for discussion, and we would like you to discuss it together. You are mainly going to be speaking together and my colleague and I will be listening to you • Any questions? The candidates come into the room. The assessors greet and invite them to sit down, smiling and putting them at their ease

Phase B	Task 1 (individual)	2 min each
Interlocutor • In this part of the speaking test, we would like each of you to answer a question that relates to one of the topics you have studied so far • Your question is … [from question bank provided—Task 1] This phase offers the students a chance to talk to their partner to express their opinion on their own before they discuss together in Phase C The assessors should make sure marks for this task are recorded on mark sheets

Phase C	Task 2 (paired discussion)	3–4 min
Interlocutor • In this part of the test, we are going to give you a question relating to one of the questions you have just answered, and we would like you to discuss it together. You are encouraged to interact with each other by asking questions and responding appropriately. Ok, your question is … (from question bank provided—Task 2) • Is that clear? Ok you have about 3–4 min for this, so please begin • Thank you The assessor should make sure marks for this task are recorded on the mark sheets

Phase D	Closing
Interlocutor and Assessor • That is the end of the test • Thank you and goodbye Candidates exit and Interlocutor and Assessor finalize marks using the mark sheets and assessment criteria provided

1. *Interlocutor has primary responsibility for managing the interaction. Assessor should take no part in this but concentrate on assessing and completing score sheets. The Interlocutor should be prepared to offer confirmation/querying of Assessor’s marks

Appendix 3: Speaking Test Assessment Criteria

Score	Task Achievement (Task 1)	Interactive communication (Task 2)	Fluency	Grammar	Vocabulary	Phonology
5	Task accomplished fully and effectively	Able to initiate and respond with ease in the interaction Takes and gives turns appropriately	Able to sustain flow of language necessary to accomplish the tasks with occasional pauses to think	Candidate has the range of grammar necessary to accomplish the tasks and is generally accurate	Candidate has the range of vocabulary necessary to accomplish the tasks	L1 features do not intrude
4	Task accomplished adequately	Able to initiate and respond sufficiently to keep the interaction going. Takes and gives turns appropriately	Able to sustain flow of language to accomplish the tasks but with occasional pauses to think and pauses to search for words	Candidate lacks either the full range necessary to accomplish the tasks or lacks necessary accuracy	Candidate lacks the full range of vocabulary necessary but achieves communication through paraphrase	L1 features present but do not obstruct understanding
3	Task accomplished to a limited degree	Able to initiate and respond with difficulty. Struggles to keep the interaction going. Takes and gives turns not appropriately	Pauses to search for words are frequent and the flow of language necessary to accomplish the tasks is not always sustained	Candidate lacks both the range and accuracy necessary to accomplish the tasks	Candidate lacks the range of vocabulary necessary to accomplish the tasks and has limited ability to paraphrase	L1 features present and occasionally obstruct understanding
2	Task attempted but not accomplished	Able to respond but not initiate. Relies on partner to keep the interaction going	Speech seems a little disconnected and sometimes difficult to follow	Candidate has range and accuracy sufficient to attempt but not to accomplish the tasks	Candidate has sufficient range to accomplish the tasks and cannot paraphrase	L1 features present and often obstruct understanding
1	Task not attempted	Limited responses and no initiations	Speech is disconnected and difficult to follow	Range and accuracy inadequate for the test	Range wholly inadequate for the test	L1 features constantly obstruct understanding
0				No adequate sample of language

Appendix 4

Speaking test score sheet: English 1111/ 1222

Student name
Student ID

Criterion	Task achievement/Interactive communication	Fluency	Grammar	Vocabulary	Phonology	Total
Score Task 1
Score Task 2

1. • Enter a mark 0–5 for each of the criteria
2. • Enter grand total in box to the right

Appendix 5

Bias interaction rater by rubrics