Despite the profusion of studies into the factor structure of language tests, limited research is currently available on the relationship between test performance and language use in TLU domains. Utilizing the structural equation modeling approach, this study set out to investigate the factor structure of a high-stakes university-based English proficiency test, and then modeled the relationship between test takers’ performance and their ability to use language in TLU domains. A self-assessment (SA) inventory was developed and validated to capture test takers’ ability to use language in TLU domains. The results showed that the higher-order factor model best fit both the test and SA data. Structural regression analysis indicated a moderately strong relationship between students’ general EFL ability and the SA latent factor. The results of this study lend empirical support to the construct and predictive validity of the test, both of which are deemed crucial to its overall validity argument. The results also shed light on the utility of SA to capture language use in TLU domains as well as the use of SA for test validation purposes.
Keywords: Factor structure Language ability Language use Self-assessment Structural equation modelingAlderson, J. C., & Banerjee, J. (2002). Language testing and assessment (Part 2). Language Teaching, 35(02), 79–113.
Bachman, L. F., & Palmer, A. S. (1989). The construct validation of self-ratings of communicative language ability. Language Testing, 6(1), 14–29.
Bentler, P. M., & Wu, E. J. (2005). EQS 6.1 for windows. Encino, CA: Multivariate Software.
Blanche, P., & Merino, B. J. (1989). Self-assessment of foreign language skills: Implications for teachers and researchers. Language Learning, 39(3), 313–338.
Bond, T., & Fox, C. M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences. New York: Routledge, Taylor & Francis Group.
Butler, Y. G., & Lee, J. (2006). On-task versus off-task self-assessments among Korean elementary school students studying English. The Modern Language Journal, 90(4), 506–518.
Byrne, B. M. (2006). Structural equation modeling with EQS: Basic concepts, applications, and programming (2nd ed.). Mahwah, NJ: Psychology Press.
Chapelle, C. A., Enright, M. K., & Jamieson, J. M. (2008). Building a validity argument for the Test of English as a Foreign Language. New York: Routledge, Taylor & Francis Group.
Cho, Y., & Bridgeman, B. (2012). Relationship of TOEFL iBT® scores to academic performance: Some evidence from American universities. Language Testing, 29(3), 421–442.
Enright, M. K., Bridgeman, B., Eignor, D., Lee, Y.-W., & Powers, D. E. (2008). Prototyping measures of listening, reading, speaking, and writing. In C. A. Chapelle, M. K. Enright, & J. M. Jamieson (Eds.), Building a validity argument for the Test of English as a Foreign Language (pp. 145–186). New York: Routledge, Taylor & Francis Group.
Fan, J. (2016). The construct and predictive validity of a self-assessment scale. Papers in Language Testing and Assessment, 5(2), 69–100.
Fan, J., & Bond, T. (2016). Using MFRM and SEM in the validation of analytic rating scales of an English speaking assessment. In Q. Zhang (Ed.), Pacific rim objective measurement symposium (PROMS) 2015 conference proceedings (pp. 29–50). Singapore: Springer.
Fan, J., & Ji, P. (2014). Test candidates’ attitudes and their test performance: The case of the Fudan English Test. University of Sydney Papers in TESOL, 9, 1–35.
FDU Testing Team. (2014). The Fudan English test syllabus. Shanghai: Fudan University Press.
Gu, L. (2014). At the interface between language testing and second language acquisition: Language ability and context of learning. Language Testing, 31(1), 111–133.
In’nami, Y., & Koizumi, R. (2011). Factor structure of the revised TOEFL test: A multi-sample analysis. Language Testing, 29(1), 131–152.
In’nami, Y., Koizumi, R., & Nakamura, K. (2016). Factor structure of the Test of English for Academic Purposes (TEAP®) test in relation to the TOEFL iBT® test. Language Testing in Asia, 6(1), 1–23.
Kunnan, A. J. (1995). Test taker characteristics and test performance: A structural modeling approach. Cambridge: Cambridge University Press.
Llosa, L. (2007). Validating a standards-based classroom assessment of English proficiency: A multitrait-multimethod approach. Language Testing, 24(4), 489–515.
Ockey, G. J., & Choi, I. (2015). Structural equation modeling reporting practices for language assessment. Language Assessment Quarterly, 12(3), 305–319.
Oller, J. W. (1976). Evidence for a general language proficiency factor: An expectancy of grammar. Die neueren sprachen, 75(2), 165–174.
Oscarson, M. (2013). Self-assessment in the classroom. In A. J. Kunnan (Ed.), The companion to language assessment (Vol. 2, pp. 712–729). New York: Wiley-Blackwell.
Powers, D. E., & Powers, A. (2015). The incremental contribution of TOEIC® listening, reading, speaking, and writing tests to predicting performance on real-life English language tasks. Language Testing, 32(2), 151–167.
Powers, D. E., Roever, C., Huff, K. L., & Trapani, C. S. (2003). Validating Language™ courseware scores against faculty ratings and student self-assessments. Retrieved from http://www.ets.org/Media/Research/pdf/RR-03-11-Powers.pdf.
Ross, S. (1998). Self-assessment in second language testing: A meta-analysis and analysis of experiential factors. Language Testing, 15(1), 1–20.
Sackett, P. R., Borneman, M. J., & Connelly, B. S. (2008). High stakes testing in higher education and employment: Appraising the evidence for validity and fairness. American Psychologist, 63(4), 215.
Sang, F., Schmitz, B., Vollmer, H. J., Baumert, J., & Roeder, P. (1986). Models of second language competence: A structural equation approach. Language Testing, 3(1), 54–79.
Satorra, A., & Bentler, P. M. (2001). A scaled difference Chi square test statistic for moment structure analysis. Psychometrika, 66(4), 507–514.
Sawaki, Y., Stricker, L. J., & Oranje, A. H. (2009). Factor structure of the TOEFL Internet-based test. Language Testing, 26(1), 5–30.
Shin, S. K. (2005). Did they take the same test? Examinee language proficiency and the structure of language tests. Language Testing, 22(1), 31–57.
Suzuki, Y. (2015). Self-assessment of Japanese as a second language: The role of experiences in naturalistic acquisition. Language Testing, 32(1), 63–81.
Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics, 5th. Needham Height, MA: Allyn & Bacon.
Zheng, Y., & Cheng, L. (2008). Test review: College English Test (CET) in China. Language Testing, 25(3), 408–417.