How did the Plenary Speakers interpret the "Bridging Borders" main Conference Theme?
Bringing the Law of Comparative Judgment into the methodological toolbox of learner corpus researchers
Magali Paquot FNRS – UCLouvain (Centre for English Corpus Linguistics)
Thurstone’s (1927) Law of Comparative Judgment (CJ) asserts that “a person perceiving some phenomenon will assign to it some instantaneous ‘value’ […], and that when we ask them to choose the ‘better’ of two such phenomena it is these values that are compared” (Pollitt, 2012a:159). This has been reformulated as the assumption that people are able to compare two performances more easily and reliably than to assign a score to an individual performance (Lesterhuis et al., 2017). The CJ technique relies on the consensus of a panel of judges who are asked to compare two performances (e.g. geography essays, academic writing, mathematical problem solving exercises). From many such pairwise comparisons, a scaled distribution of performances is then produced.
CJ has developed into a popular methodology in various domains of research and practice, and most particularly in educational assessment, where it is increasingly being valued for “making reliable the assessment of skills that are currently problematic” (Pollit, 2012b: 292). This includes the assessment of writing quality, a multidimensional and complex construct for which CJ tasks have already been reported to produce reliable and valid results (e.g., Lesterhuis et al., 2017; Lesterhuis et al., 2018; van Daal et al., 2019).
The main objective of this talk is to introduce the method of CJ to the LCR community and demonstrate its usefulness for enriching L2 data with proficiency scores that are often absent from, or measured in unreliable ways in currently available learner corpora. I will report on a number of studies that we have conducted within the framework of the Crowdsourcing Language Assessment Project (CLAP). Collectively, these studies investigate whether a crowd of judges is able to assess learner texts in a CJ task with high reliability (Paquot et al., 2022; Thwaites & Paquot, 2023; Thwaites et al., in preparation). More specifically, they seek to answer the following research questions:
RQ1. To what extent do specific characteristics of learner texts (topic, length, homogeneity in terms of proficiency) have an effect on the reliability of an ACJ task?
RQ2. To what extent do specific characteristics of judges (language assessment training and expertise) have an effect on the reliability of an ACJ task?
RQ3. To what extent do specific characteristics of learner texts and characteristics of judges have an effect on the validity of an ACJ task?
So far, we have only used CJ to enrich learner corpora with proficiency scores but as I will argue by the end of my presentation, CJ may also serve to elicit scores and rankings of productions for other multidimensional constructs (e.g., complexity, accuracy, fluency) that have proved difficult to define and operationalize in the field of L2 research.
References
Lesterhuis, M., Verhavert, S., Coertjens, L., Donche, V., & De Maeyer, S. (2017). Comparative judgement as a promising alternative to score competences. In E. Cano & G. Ion (Eds.), Innovative practices for higher education assessment and measurement (pp. 119–138). Hershey, PA: IGI Global.
Lesterhuis, M., van Daal, T., Van Gasse, R., Coertjens, L., Donche, V., & De Maeyer, S. (2018). When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality. L1 Educational Studies in Language and Literature, 18, 1–22. https://doi.org/10.17239/L1ESLL-2018.18.01.02
Paquot, M., Rubin, R. & Vandeweerd, N. (2022). Crowdsourced Adaptive Comparative Judgment: A community-based solution for proficiency rating. Language Learning, 72(3), 53-885. https://onlinelibrary.wiley.com/doi/10.1111/lang.12498
Pollitt, A. (2012a). Comparative judgement for assessment. International Journal of Technology and Design Education, 22, 157–170. https://doi.org/10.1007/s10798-011-9189-x
Pollitt, A. (2012b). The method of adaptive comparative judgement. Assessment in Education: Principles, Policy & Practice, 19, 281–300. https://doi.org/10.1080/0969594X.2012.665354
Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273–86.
Thwaites, P. & Paquot, M. (2023). Comparative judgement of L2 writing in learner corpora: does a narrow proficiency range reduce test reliability? Paper to be presented at the ALTE 8th International Conference, 26-28 April 2023, Madrid, Spain.
Thwaites, P., Vandeweerd, N., & Paquot, M. (in preparation). Crowdsourcing language assessment: how reliable are laypeople’s assessments of L2 writing?
van Daal, T., Lesterhuis, M., Coertjens, L., Donche, V., & De Maeyer, S. (2019). Validity of comparative judgement to assess academic writing: Examining implications of its holistic character and building on a shared consensus. Assessment in Education: Principles, Policy & Practice, 26, 59–74. https://doi.org/10.1080/0969594X.2016.1253542
Bridging research and practice: Spoken learner corpora for language teaching
Raffaella Bottini, Lancaster University
Speaking is a core communicative skill, one of the most challenging in second language (L2) learning (Gablasova et al., 2017). Spoken communication involves a range of cognitive, social, and linguistic resources, is characterised by real-time processing and limited possibilities to edit the language (Gablasova & Bottini, 2022). Also, L2 proficiency includes the ability to tailor spoken language production to different communicative contexts, registers, and audiences. Language teaching materials generally include a narrow focus on speaking skills, often limiting learners’ exposure to examples of interactions which are not based on authentic data (e.g., Hughes & Reed, 2017). Textbooks rarely compare different spoken registers or raise learners’ awareness of the importance of tailoring language production to different audiences and of the pragmatic role of features of speech such as interjections, pauses, and repetitions. Spoken learner corpora are a unique resource to inform language learning and to develop language teaching materials that reflect authentic language use in spoken communication (e.g., Chambers, 2019; Forti & Spina, 2019; Meunier, 2016). This talk will provide an example of how learner corpus research can inform language teaching. First, I will give an overview of the major pedagogical applications of spoken learner corpora. Next, I will focus on a recent study on the effect of task interactivity on lexical complexity (i.e., diversity, density and sophistication) in L2 English speech (Bottini, 2022). The study used the 4.2-million-word Trinity Lancaster Corpus (Gablasova et al., 2019) based on the Graded Examinations in Spoken English – a high-stakes exam of L2 English administered by Trinity College London – and consisting of transcripts of learners’ spoken performance (CEFR B1 to C2) across different tasks. The study has practical implications for language teaching and offers an opportunity to reflect on challenges of bridging research and practice and possible solutions, as well as directions for future research.
References
Bottini, R. (2022). Lexical complexity in L2 English speech: Evidence from the Trinity Lancaster Corpus. PhD thesis, Lancaster University.
Chambers, A. (2019). Towards the corpus revolution? Bridging the research–practice gap. Language Teaching, 52(4), 460–475.
Forti, L., & Spina, S. (2019). Corpora for linguists vs. corpora for learners: Bridging the gap in Italian L2 learning and teaching. EL.LE, 8(2), 349–362.
Gablasova, D., & Bottini, R. (2022). Spoken learner corpora to inform teaching. In R. R. Jablonkai & E. Csomay (Eds.), The Routledge Handbook of Corpora in English Language Teaching and Learning (pp. 296–310). Routledge.
Gablasova, D., Brezina, V., McEnery, T., & Boyd, E. (2017). Epistemic stance in spoken L2 English: The effect of task and speaker style. Applied Linguistics, 38(5), 613–637.
Gablasova, D., Brezina, V., & McEnery, T. (2019). The Trinity Lancaster Corpus: Development, description and application. International Journal of Learner Corpus Research, 5(2), 126–158.
Hughes, R., & Reed, B. S. (2017). Teaching and researching speaking. Taylor & Francis.
Meunier, F. (2016). Learner corpora and pedagogical applications. In F. Farr & L. Murray (Eds.), The Routledge handbook of language learning and technology (pp. 493–507). Taylor & Francis.