This article reports the development of the Test Takers’ Metacognitive Awareness Reading Questionnaire (TMARQ) which measures test takers’ metacognition in reading comprehension tests. The TMARQ comprises seven subscales: planning strategies, evaluating strategies, monitoring strategies, strategies for identifying important information, inference-making strategies, integrating strategies, and supporting strategies. In this article, a validity argument is laid out for the questionnaire by presenting content-referenced, substantive, and structural evidence of validity, which is primarily yielded through Rasch measurement and structural equation modeling.
Keywords: Metacognitive awareness Reading test Rasch measurement Structural equation modeling ValidityAnderson, N. (1991). Individual differences in strategy use in second language reading and testing. Modern Language Journal, 75, 460–472.
Anderson, R. C., & Pearson, P. D. (1984). A schema-theoretic view of basic processes in reading. In R. Barr, M. L. Kamil, P. Mosenthal, & P. D. Pearson (Eds.), Handbook of reading research (Vol. 2, pp. 255–292). White Plains, NY: Longman.
Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43, 561–573.
Arbuckle, J. L. (2011). IBM SPSS Amos 20.0 [computer program]. New York: IBM.
Aryadoust, V. (2013). Building a validity argument for a listening test of academic proficiency. Newcastle: Cambridge Scholars.
Aryadoust, V., Goh, C., & Lee, O. K. (2011). An investigation of differential items functioning in the MELAB listening test. Language Assessment Quarterly, 8(4), 361–385.
Aryadoust, V., Goh, C., & Lee, O. K. (2012). Developing an academic listening self-assessment questionnaire: A study of modeling academic listening. Psychological Test and Assessment Modeling, 54(3), 227–256.
Bachman, L. F. (1990). Fundamental consideration in language testing. Oxford: Oxford University Press.
Bachman, L. F., & Cohen, A. D. (1998). Interface between second language acquisition and language testing research. Cambridge: Cambridge University Press.
Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice. Oxford: Oxford University Press.
Bachman, L. F., & Palmer, A. S. (2010). Language testing in practice. Oxford: Oxford University Press.
Baker, L., & Brown, A. L. (1984). Metacognitive skills and reading. In R. Barr, M. L. Kamil, P. Mosenthal, & P. D. Pearson (Eds.), Handbook of reading research (Vol. 2, pp. 353–394). White Plains, NY: Longman.
Bandalos, D. L., & Finney, S. J. (2001). Item parceling issues in structural equation modeling. Mahwah, NJ: Lawrence Erlbaum.
Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum Associates.
Brown, A. L. (1978). Knowing when, where, and how to remember: A problem of metacognition. In R. Glaser (Ed.), Advances in instructional psychology (Vol. 1, pp. 77–165). Hillsdale, NJ: Lawrence Erlbaum Associates.
Byrne, B. M. (2011). Structural equation modeling with Mplus: Basic concepts, applications, and programming. New York: Routledge.
Cain, K., Oakhill, J., & Bryant, P. (2004). Children’s reading comprehension ability: Concurrent prediction by working memory, verbal ability, and component skills. Journal of Educational Psychology, 96, 31–42.
Carrell, P. L. (1989). Metacognitive awareness and second language reading. Modern Language Journal, 73(2), 121–131.
Chapelle, C. A., Chung, Y.-R., Hegelheimer, V., Pendar, N., & Xu, J. (2010). Towards a computer-delivered test of productive grammatical ability. Language Testing, 27(4), 443–469.
Cohen, A. D. (2006). The coming age of research on test-taking strategies. Language Assessment Quarterly, 3(4), 307–331.
Cohen, A.D., & Upton, T.A. (2006). Strategies in responding to the new TOEFL reading tasks (TOEFL Monograph Series Report No. 33). Princeton, NJ: Educational Testing Service. Retrieved July 20, 2012 from http://www.ets.org/Media/Research/pdf/RR-06-06.pdf.
Dörnyei, Z. (2003). Questionnaires in second language research: Construction, administration and processing. Mahwah, NJ: Erlbaum.
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Rev ed.). Cambridge, MA: MIT Press.
Flavell, J. H. (1978). Metacognitive development. In J. M. Scadura & C. J. Brainerd (Eds.), Structural Process theories of complex human behavior. Alphen aan den Rijn: Sijthoff and Noordhoff.
Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive- developmental inquiry. American Psychologist, 34, 906–911.
Gagnè, E. D., Yekovich, C. W., & Yekovich, F. R. (1993). The cognitive psychology of school learning. New York: Harper Collins.
Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2010). Multivariate data analysis (6th ed.). Upper Saddle River, NJ: Pearson Education.
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternative. Structural Equation Modeling, 6(1), 1–55.
Jacobs, J. E., & Paris, S. G. (1987). Children’s metacognition about reading: Issues in definition, measurement, and instruction. Educational Psychologist, 22, 255–278.
Kintsch, W., & van Dijk, (1978). Toward a model of text comprehension and production. Psychological Review, 85, 363–394.
Kline, R. B. (2011). Principles and practices of structural equation modeling (2nd ed.). New York: Guilford.
Kunnan, A. J. (1998). An introduction to structural equation modeling for language assessment research. Language Testing, 15(3), 295–332.
Lee, C. B. (2013). Examining intentional knowing among secondary school students: Through the lens of metacognition. The Asia-Pacific Education Researcher, 22(1), 79–90.
Linacre, J. M. (2010a). A users’ guide to WINSTEPS ® MINISTEPS Rasch-model computer programs. Chicago, IL: Winsteps.
Linacre, J. M. (2010b). WINSTEPS: Rasch model computer programs. Chicago, IL: Winsteps.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (pp. 13–103). New York: Macmillan.
Mokhtari, K., Sheorey, R., & Reichard, C. A. (2009). Measuring the reading strategies of first and second language readers. In K. Mokhtari & R. Sheorey (Eds.), Reading strategies of first- and second language learners. Norwood, MA: Christopher-Gordon Publishers, Inc.
Paris, S. G., Lipson, M. Y., & Wixson, K. K. (1983). Becoming a strategic reader. Contemporary Educational Psychology, 8, 248–293.
Paris, S. G., & Winograd, P. (1990). How metacognition can promote academic learning and instruction. In B. F. Jones & L. Idol (Eds.), Dimensions of thinking and cognitive instruction (pp. 15–51). Hillsdale, NJ: Erlbaum.
Park, G. (1997). Language learning strategies and English proficiency in Korean university students. Foreign Language Annals, 30, 211–221.
Pearson, P. D. (2009). The roots of reading comprehension instruction. In S. E. Isreal & G. G. Duffy (Eds.), Handbook of research on reading comprehension. New York: Routledge.
Phakiti, A. (2003). A closer look at the relationship of cognitive and metacognitive strategy used to EFL reading achievement test performance. Language Testing, 20(1), 26–56.
Phakiti, A. (2008). Construct validation of Bachman and Palmer’s (1996) strategic competence model over time in EFL reading tests. Language Testing, 25(2), 237–272.
Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1993). Reliability and predictive validity of the motivated strategies for learning questionnaire (MSLQ). Educational and Psychological Measurement, 53, 801–803.
Pressley, M., & Afflerbach, P. (1995). Verbal protocols of reading: The nature of constructively responsive reading. Hillsdale, NJ: Erlbaum.
Purpura, J. E. (1997). An analysis of the relationships between test takers’ cognitive and metacognitive strategy use and second language test performance. Language Learning, 47, 289–294.
Purpura, J. E. (1999). Learner strategy use and performance on language tests: A structural equation modeling approach. Cambridge: Cambridge University Press.
Roeschl-Heils, A., Schneider, W., & van Kraayenoord, C. E. (2003). Reading, metacognition, and motivation: A follow-up study of German students 7 and 8. European Journal of Psychology of Education, 18, 75–86.
Smith, E. V. Jr. (2005). Effect of redundancy on Rasch item and person estimates. Journal of Applied Measurement, 6(2), 147–163.
Song, X. (2005). Language learner strategy use and English proficiency on the Michigan English Language Assessment Battery. Spaan Fellow Working Papers in Second or Foreign Language Assessment, 3, 1–26.
Song, X., & Cheng, L. (2006). Language learner strategy use and test performance of Chinese learners of English. Language Assessment Quarterly, 3(3), 243–266.
Tarricone, P. (2011). The taxonomy of metacognition. New York: The Psychology Press.
van Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic.
Wang, W. C. (2004). Direct estimation of correlation as a measure of association strength using multidimensional item response models. Educational and Psychological Measurement, 64, 937–955.
Wen, Q., & Johnson, R. (1997). L2 learner variables and English achievement: A study of tertiary-level English majors in China. Applied Linguistics, 18, 27–48.
Wenden, A. L. (1998). Metacognitive knowledge and language learning. Applied Linguistics, 19(4), 515–537.
Wolfe, E. W., & Smith, E. V. Jr. (2007a). Instrument development tools and activities for measure validation using Rasch models: Part I—Instrument development tools. Journal of Applied Measurement, 8, 97–123.
Wolfe, E. W., & Smith, E. V. Jr. (2007b). Instrument development tools and activities for measure validation using Rasch models: Part II—Validation activities. Journal of Applied Measurement, 8, 204–233.
Wright, B. D., & Linacre, J. M. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8, 370.
Wu, M. L., Adams, R., Wilson, M., & Haldane, S. A. (2007). ACER ConQuest, version 2.0 [computer program]. Melbourne: Australian Council for Educational Research.
Zhang, L. J. (2010). A dynamic metacognitive systems account of Chinese university students’ knowledge about EFL reading. TESOL Quarterly, 44(2), 320–353.
Zhang, L. (in press). A structural equation modeling approach to investigating test takers’ strategy use and reading test performance. Asian EFL Journal.
Zhang, L., & Zhang, L. J. (2013). Relationships between Chinese college test takers’ strategy use and EFL reading test performance: A structural equation modelling approach. RECL Journal, 44(1), 35–57.
Zheng, Y., & Cheng, L. (2008). Test review: College English test (CET) in China. Language Testing, 25(3), 408–417.