AER Banner
  • Keyword
  • Volume/Page
  • DOI
  • Citation
  • Advanced
 
   
 
 
Astronomy Education Review / Volume 10 / Issue 1 / RESEARCH AND APPLICATIONS

2011, AER, 10 (1), 010107, doi:http://dx.doi.org/10.3847/AER2011030

A Study of General Education Astronomy Students’ Understandings of Cosmology. Part II. Evaluating Four Conceptual Cosmology Surveys: A Classical Test Theory Approach

Published 15 December 2011

Colin S. Wallace and Edward E. Prather

Center for Astronomy Education (CAE), Steward Observatory, University of Arizona, Tucson, Arizona 85721

Douglas K. Duncan

Department of Astrophysical and Planetary Sciences, University of Colorado at Boulder, Boulder, Colorado 80309

View MapView Map
Full Text: Read Online (HTML) | Download PDF | View Cart
This is the second of five papers detailing our national study of general education astronomy students’ conceptual and reasoning difficulties with cosmology. This article begins our quantitative investigation of the data. We describe how we scored students’ responses to four conceptual cosmology surveys, and we present evidence for the inter-rater reliability of those scores. We devote the bulk of this article to a classical test theory analysis of the data. We calculate difficulties and discriminations for each item, and we compute Cronbach’s α as a measure of the reliability of the surveys. We also discuss the implications this analysis has for the validity of the surveys.

© 2011 The American Astronomical Society

EDITORIALLY RELATED

    Related Articles

  1. A Study of General Education Astronomy Students' Understandings of Cosmology. Part III. Evaluating Four Conceptual Cosmology Surveys: An Item Response Theory Approach
    Colin S. Wallace et al.
    AER 11, 010103 (2012)AERSCZ000011000001010103000001
  2. A Study of General Education Astronomy Students' Understandings of Cosmology. Part IV. Common Difficulties Students Experience with Cosmology
    Colin S. Wallace et al.
    AER 11, 010104 (2012)AERSCZ000011000001010104000001
  3. A Study of General Education Astronomy Students' Understandings of Cosmology. Part I. Development and Validation of Four Conceptual Cosmology Surveys
    Colin S. Wallace et al.
    AER 10, 010106 (2011)AERSCZ000010000001010106000001

KEYWORDS and PACS

Keywords

cosmology, education, College non-majors, Cosmology, Research into teaching/learning, Assessment, 01.40.-d, 98.80.-k, 01.40.kF, 01.40.G-

PACS

PUBLICATION DATA

ISSN

1539-1515 (online)

Publisher

$abstract.society.value is a Member of CrossRef The American Astronomical Society

ARTICLE DATA

History
Received 21 September 2011
Accepted 01 November 2011
Published online 15 December 2011
Digital Object Identifier
http://dx.doi.org/10.3847/AER2011030

    References

  1. Bardar, E. M., Prather, E. E, Brecher, K., and Slater, T. F. 2007, “Development and Validation of the Light and Spectroscopy Concept Inventory,” Astronomy Education Review, 5, 103.
  2. Cohen, J. 1960, “A Coefficient of Agreement for Nominal Scales,” Educational and Psychological Measurement, 20, 37.
  3. Cohen, J. 1968, “Weighted Kappa: Nominal Scale Agreement with Provision for Scaled Disagreement or Partial Credit,” Psychological Bulletin, 70, 213. [MEDLINE]
  4. Ding, L., and Beichner, R. 2009, “Approaches to Data Analysis of Multiple-Choice Questions,” Physical Review Special Topics–Physics Education Research, 5, 020103.
  5. Ding, L., Chabay, R., Sherwood, B., and Beichner, R. 2006, “Evaluating an Electricity and Magnetism Assessment Tool: Brief Electricity and Magnetism Assessment,” Physical Review Special Topics– Physics Education Research, 2, 010105.
  6. Fleiss, J. L., Levin, B., and Paik, M. C. 2003, Statistical Methods for Rates and Proportions, 3rd ed., Hoboken, NJ: John Wiley and Sons, Inc.
  7. George, D. and Mallery, P. 2009, SPSS for Windows Step by Step: A Simple Guide and Reference, Boston, MA: Pearson Education, Inc.
  8. Hambleton, R. K., and Jones, R. J. 1993, “Comparison of Classical Test Theory and Item Response Theory and Their Applications to Test Development,” Educational Measurement: Issues and Practice, 12, 253.
  9. Landis, J. R., and Koch, G. G. 1977, “The Measurement of Observer Agreement for Categorical Data,” Biometrics, 33, 159. [MEDLINE]
  10. Lord, F. M., and Novick, M. R. 1968, Statistical Theories of Mental Test Scores, Reading, MA: Addison-Wesley.
  11. Maloney, D. P., O'Kuma, T. L., Hieggelke, C. J., and van Heuvelen, A. 2001, “Surveying Students' Conceptual Knowledge of Electricity and Magnetism,” American Journal of Physics, 69, S12AJPIAS0000690000S1000S12000001.
  12. McDermott, L. C., Rosenquist, M. L., and van Zee, E. H., 1987, “Student Difficulties in Connecting Graphs and Physics: Examples From Kinematics,” American Journal of Physics, 55, 503AJPIAS000055000006000503000001.
  13. Otero, V. K., and Harlow, D. B. 2009, “Getting Started in Qualitative Physics Education Research,” in Reviews in PER Vol. 2: Getting Started in PER, ed. C. Henderson and K. A. Harper, College Park, MA: American Association of Physics Teachers, 1.
  14. Schmitt, N. 1996, “Uses and Abuses of Coefficient Alpha,” Psychological Assessment, 8, 350.
  15. Thompson, B. 2003, “Understanding Reliability and Coefficient alpha, Really,” in Score Reliability, ed. B. Thompson, Thousand Oaks, CA: SAGE Publications, 3.
  16. Wallace, C. S., and Bailey, J. M. 2010, “Do Concept Inventories Actually Measure Anything?,” Astronomy Education Review, 9, 010116AERSCZ000009000001010116000001.
  17. Wallace, C. S., Prather, E. E., and Duncan, D. K. 2011a, “A Study of General Education Astronomy Students' Understandings of Cosmology. Part I. Development and Validation of Four Conceptual Cosmology Surveys,” Astronomy Education Review, 10, 010106.
  18. Wallace, C. S., Prather, E. E., and Duncan, D. K. 2011b, “A Study of General Education Astronomy Students' Understandings of Cosmology. Part III. Evaluating Four Conceptual Cosmology Surveys: An Item Response Theory Approach,” Astronomy Education Review (submitted).
  19. Wallace, C. S., Prather, E. E., and Duncan, D. K. 2011c, “A Study of General Education Astronomy Students' Understandings of Cosmology. Part IV. Common Difficulties Students Experience with Cosmology,” Astronomy Education Review (submitted).
  20. Wallace, C. S., Prather, E. E., and Duncan, D. K. 2011d, “A Study of General Education Astronomy Students' Understandings of Cosmology. Part V. The Effects of a New Suite of Cosmology Lecture-Tutorials on Students' Conceptual Knowledge,” (in preparation).
  21. Wilson, M. 2005, Constructing Measures: An Item Response Modeling Approach, Mahwah, NJ: Lawrence Erlbaum Associates.


Tables

Table I. The scoring rubric for Item 1 on Form B (fall 2010 version)

View Table
Table II. A sample of student responses to Item 1 on Form B and their overall scores and reasoning element codes. The student responses contain their original spellings, grammar, and punctuation

View Table
Table III. The pre-instruction difficulties (P-values) and discriminations of the items on Forms A–D

View Table
Table IV. The post-instruction difficulties (P-values) and discriminations of the items on Forms A-D

View Table
Table V. Cronbach’s α (pre-instruction and post-instruction) for Forms A–D

View Table


Close
ADVERTISEMENT

Copyright © 2001 - American Astronomical Society. All rights reserved.

close