AER Banner
  • Keyword
  • Volume/Page
  • DOI
  • Citation
  • Advanced
 
   
 
 
Astronomy Education Review / Volume 11 / Issue 1 / RESEARCH AND APPLICATIONS

2012, AER, 11 (1), 010103, doi:http://dx.doi.org/10.3847/AER2011031

A Study of General Education Astronomy Students’ Understandings of Cosmology. Part III. Evaluating Four Conceptual Cosmology Surveys: An Item Response Theory Approach

Published 2 February 2012

Colin S. Wallace and Edward E. Prather

Center for Astronomy Education (CAE), Steward Observatory, University of Arizona, Tucson, Arizona 85721

Douglas K. Duncan

Department of Astrophysical and Planetary Sciences, University of Colorado at Boulder, Boulder, Colorado 80309

View MapView Map
Full Text: Read Online (HTML) | Download PDF | View Cart
This is the third of five papers detailing our national study of general education astronomy students’ conceptual and reasoning difficulties with cosmology. In this paper, we use item response theory to analyze students’ responses to three out of the four conceptual cosmology surveys we developed. The specific item response theory model we use is known as the partial credit model. Since readers may be unfamiliar with the partial credit model, we provide a pedagogical introduction to this model. We use the partial credit model to assess the reliabilities of the four survey forms and to determine the probabilities of students achieving different scores on survey items.

© 2012 The American Astronomical Society

EDITORIALLY RELATED

    Related Articles

  1. A Study of General Education Astronomy Students' Understandings of Cosmology. Part IV. Common Difficulties Students Experience with Cosmology
    Colin S. Wallace et al.
    AER 11, 010104 (2012)AERSCZ000011000001010104000001
  2. A Study of General Education Astronomy Students' Understandings of Cosmology. Part I. Development and Validation of Four Conceptual Cosmology Surveys
    Colin S. Wallace et al.
    AER 10, 010106 (2011)AERSCZ000010000001010106000001
  3. A Study of General Education Astronomy Students' Understandings of Cosmology. Part II. Evaluating Four Conceptual Cosmology Surveys: A Classical Test Theory Approach
    Colin S. Wallace et al.
    AER 10, 010107 (2011)AERSCZ000010000001010107000001

KEYWORDS and PACS

Keywords

astronomical surveys, cosmology, education, physics education, probability

PACS

  • 01.50.-i

    Educational aids

  • 02.50.Cw

    Probability theory

  • 95.80.+p

    Astronomical catalogs, atlases, sky surveys, databases, retrieval systems, archives, etc.

  • 98.80.Jk

    Mathematical and relativistic aspects of cosmology

PUBLICATION DATA

ISSN

1539-1515 (online)

Publisher

$abstract.society.value is a Member of CrossRef The American Astronomical Society

ARTICLE DATA

History
Received 24 October 2011
Accepted 23 December 2011
Published online 02 February 2012
Digital Object Identifier
http://dx.doi.org/10.3847/AER2011031

    References

  1. Baker, F. B., and Kim, S. 2004, Item Response Theory: Parameter Estimation Techniques, 2nd ed., New York, NY: Marcel Dekker, Inc.
  2. Embretson, S. E., and Reise, S. P. 2000, Item Response Theory for Psychologists, Mahwah, NJ: Lawrence Erlbaum Associates.
  3. Hambleton, R. K., and Jones, R. J. 1993, “Comparison of Classical Test Theory and Item Response Theory and Their Applications to Test Development,” Educational Measurement: Issues and Practice, 12, 253.
  4. Harris, D. 1989, “Comparison of 1-, 2-, and 3-Parameter IRT Models,” Educational Measurement: Issues and Practice, 8, 157.
  5. Kennedy, C. A., Wilson, M., Draney, K., Tutunciyan, S., and Vorp, R. 2006, ConstructMap Software, Berkeley, CA: Berkeley Evaluation and Assessment Research (BEAR) Center. Available at: http://bearcenter.berkeley.edu/ConstructMap.
  6. Masters, G. N. 1982, “A Rasch Model for Partial Credit Scoring,” Psychometrika, 47, 149.
  7. Rasch, G. 1980, Probabilistic Models for Some Intelligence and Attainment Tests, Chicago, IL: University of Chicago Press. Originally published in 1960, Copenhagen, DK: The Danish Institute for Educational Research.
  8. Schmitt, N. 1996, “Uses and Abuses of Coefficient Alpha,” Psychological Assessment, 8, 350.
  9. Thompson, B. 2003, “Understanding Reliability and Coefficient alpha, Really,” in Score Reliability, ed. B. Thompson, Thousand Oaks, CA: SAGE Publications, 3.
  10. Wallace, C. S., and Bailey, J. M. 2010, “Do Concept Inventories Actually Measure Anything?” Astronomy Education Review, 9, 010116AERSCZ000009000001010116000001.
  11. Wallace, C. S., Prather, E. E., and Duncan, D. 2011a, “A Study of General Education Astronomy Students' Understandings of Cosmology. Part I. Development and Validation of Four Conceptual Cosmology Surveys,” Astronomy Education Review, 10, 010106.
  12. Wallace, C. S., Prather, E. E., and Duncan, D. 2011b, “A Study of General Education Astronomy Students' Understandings of Cosmology. Part II. Evaluating Four Conceptual Cosmology Surveys: A Classical Test Theory Approach,” Astronomy Education Review, 10, 010107AERSCZ000010000001010107000001.
  13. Wallace, C. S., Prather, E. E., and Duncan, D. 2012a, “A Study of General Education Astronomy Students' Understandings of Cosmology. Part IV. Common Difficulties Students Experience with Cosmology,” Astronomy Education Review, 11, 010104.
  14. Wallace, C. S., Prather, E. E., and Duncan, D. 2012b, “A Study of General Education Astronomy Students' Understandings of Cosmology. Part V. The Effects of a New Suite of Cosmology Lecture-Tutorials on Students' Conceptual Knowledge” (in preparation).
  15. Wilson, M. 2005, Constructing Measures: An Item Response Modeling Approach, Mahwah, NJ: Lawrence Erlbaum Associates.
  16. Yen, W. M. 1993, “Scaling Performance Assessments: Strategies for Managing Local Item Dependence,” Journal of Educational Measurement, 30, 187.

Figures (12) Tables (3)

Figures (click on thumbnails to view enlargements)

FIG.1
The CRCs for Form B, Item 1 from the fall 2009. Each curve shows the probability of a particular score as a function of ability. The blue dashed and dotted line correspond to a score of 0, the green dotted line to a score of 1, the red dashed line to a score of 2, the purple short dashed line to a score of 3, and the solid black line to a score of 4. The x-axis shows the range of estimated student abilities as measured by the fall 2009 version of Form B.

FIG.1 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

FIG.2
The CRCs for Form C, Item 3 from the fall 2009. Each curve shows the probability of a particular score as a function of ability. The blue dashed and dotted line correspond to a score of 0, the green dotted line to a score of 1, the red dashed line to a score of 2, and the purple short dashed line to a score of 3. The x-axis shows the range of estimated student abilities as measured by the fall 2009 version of Form C.

FIG.2 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

FIG.3
The CRCs for Form A, Item 3 from the fall 2009. Each curve shows the probability of a particular score as a function of ability. The blue dashed and dotted line corresponds to a score of 0, the green dotted line to a score of 1, the red dashed line to a score of 2, and the purple short dashed line to a score of 3. The x-axis shows the range of estimated student abilities as measured by the fall 2009 version of Form A.

FIG.3 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

FIG.4
Form A’s Wright map for the fall 2009. A histogram of students’ abilities (proficiencies) is shown on the left. On the right are the Thurstonian thresholds for each item. Blue corresponds to βi0 (i.e., the ability at which one has equal probability of earning a score < 1 and a score ≥ 1), green to βi1 (i.e., the ability at which one has equal probability of earning a score < 2 and a score ≥ 2), etc.

FIG.4 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

FIG.5
Form B’s Wright map for the fall 2009. A histogram of students’ abilities (proficiencies) is shown on the left. On the right are the Thurstonian thresholds for each item. Blue corresponds to βi0 (i.e., the ability at which one has equal probability of earning a score < 1 and a score ≥ 1), green to βi1 (i.e., the ability at which one has equal probability of earning a score < 2 and a score ≥ 2), etc.

FIG.5 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

FIG.6
Form C’s Wright map for the fall 2009. A histogram of students’ abilities (proficiencies) is shown on the left. On the right are the Thurstonian thresholds for each item. Blue corresponds to βi0 (i.e., the ability at which one has equal probability of earning a score < 1 and a score ≥ 1), green to βi1 (i.e., the ability at which one has equal probability of earning a score < 2 and a score ≥ 2), etc.

FIG.6 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

FIG.7
Form A’s Wright map for the spring 2010. A histogram of students’ abilities (proficiencies) is shown on the left. On the right are the Thurstonian thresholds for each item. Blue corresponds to βi0 (i.e., the ability at which one has equal probability of earning a score < 1 and a score ≥ 1), green to βi1 (i.e., the ability at which one has equal probability of earning a score < 2 and a score ≥ 2), etc.

FIG.7 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

FIG.8
Form B’s Wright map for the spring 2010. A histogram of students’ abilities (proficiencies) is shown on the left. On the right are the Thurstonian thresholds for each item. Blue corresponds to βi0 (i.e., the ability at which one has equal probability of earning a score < 1 and a score ≥ 1), green to βi1 (i.e., the ability at which one has equal probability of earning a score < 2 and a score ≥ 2), etc.

FIG.8 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

FIG.9
Form C’s Wright map for the spring 2010. A histogram of students’ abilities (proficiencies) is shown on the left. On the right are the Thurstonian thresholds for each item. Blue corresponds to βi0 (i.e., the ability at which one has equal probability of earning a score < 1 and a score ≥ 1), green to βi1 (i.e., the ability at which one has equal probability of earning a score < 2 and a score ≥ 2), etc.

FIG.9 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

FIG.10
Form A’s Wright map for the fall 2010. A histogram of students’ abilities (proficiencies) is shown on the left. On the right are the Thurstonian thresholds for each item. Blue corresponds to βi0 (i.e., the ability at which one has equal probability of earning a score < 1 and a score ≥ 1), green to βi1 (i.e., the ability at which one has equal probability of earning a score < 2 and a score ≥ 2), etc.

FIG.10 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

FIG.11
Form B’s Wright map for the fall 2010. A histogram of students’ abilities (proficiencies) is shown on the left. On the right are the Thurstonian thresholds for each item. Blue corresponds to βi0 (i.e., the ability at which one has equal probability of earning a score < 1 and a score ≥ 1), green to βi1 (i.e., the ability at which one has equal probability of earning a score < 2 and a score ≥ 2), etc.

FIG.11 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

FIG.12
Form C’s Wright map for the fall 2010. A histogram of students’ abilities (proficiencies) is shown on the left. On the right are the Thurstonian thresholds for each item. Blue corresponds to βi0 (i.e., the ability at which one has equal probability of earning a score < 1 and a score ≥ 1), green to βi1 (i.e., the ability at which one has equal probability of earning a score < 2 and a score ≥ 2), etc.

FIG.12 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

Tables

Table I. The step difficulty bij and Thurstonian threshold βj parameters for the items on Forms A–C for the fall 2009. All values are in logits

View Table
Table II. The step difficulty bij and Thurstonian threshold βj parameters for the items on Forms A–C for the spring 2010. All values are in logits

View Table
Table III. The step difficulty bij and Thurstonian threshold βj parameters for the items on Forms A–C for the fall 2010. All values are in logits

View Table


Close
ADVERTISEMENT

Copyright © 2001 - American Astronomical Society. All rights reserved.

close