AER Banner
  • Keyword
  • Volume/Page
  • DOI
  • Citation
  • Advanced
 
   
 
 
Astronomy Education Review / Volume 11 / Issue 1 / RESEARCH AND APPLICATIONS

2012, AER, 11 (1), 010107, doi:http://dx.doi.org/10.3847/AER2012010

A Classical Test Theory Analysis of the Light and Spectroscopy Concept Inventory National Study Data Set

Published 22 June 2012

Wayne M. Schlingman, Edward E. Prather, Colin S. Wallace, and Gina Brissenden

Center for Astronomy Education (CAE), Steward Observatory, University of Arizona, Tucson, Arizona 85721

Alexander L. Rudolph

Department of Physics and Astronomy, California State Polytechnic University, Pomona, California, 91768

View MapView Map
Full Text: Read Online (HTML) | Download PDF | View Cart
This paper is the first in a series of investigations into the data from the recent national study using the Light and Spectroscopy Concept Inventory (LSCI). In this paper, we use classical test theory to form a framework of results that will be used to evaluate individual item difficulties, item discriminations, and the overall reliability of the LSCI. We perform an analysis of individual students’ normalized gains, providing further insight into the prior results from this data set. This investigation allows us to better understand the efficacy of measuring student achievement using the LSCI. Future papers will discuss our investigation of the data from the recent national study using item response theory (IRT).

© 2012 The American Astronomical Society

KEYWORDS and PACS

Keywords

astronomy, education, reliability, Research into teaching/learning, Assessment

PACS

  • 01.40.Fk

    Research in physics education

PUBLICATION DATA

ISSN

1539-1515 (online)

Publisher

$abstract.society.value is a Member of CrossRef The American Astronomical Society

ARTICLE DATA

History
Received 19 March 2012
Accepted 14 May 2012
Published online 22 June 2012
Digital Object Identifier
http://dx.doi.org/10.3847/AER2012010

    References

  1. Allen, M. J., and Yen, W. M. 1979, Introduction to Measurement Theory, Long Grove, Illinois: Waveland Press, Inc.
  2. Bardar, E. M., Prather, E. E., Brecher, K., and Slater, T. F. 2007, “Development and Validation of the Light and Spectroscopy Concept Inventory,” Astronomy Education Review, 5, 103.
  3. Crocker, L. M., and Algina, J. 1986, Introduction to Classical and Modern Test Theory, New York, NY: Holt, Rinehart, and Winston.
  4. Deming, G., and Hufnagel, B. 2001, Who's taking ASTRO 101?, Physics Teacher, 39(6), 368PHTEAH000039000006000368000001.
  5. diSessa, A. 1983, “Phenomenology and Evolution of Intuition,” in Mental Models, ed. A. L. Stevens, Hillsdale, NJ: Lawrence Erlbaum Associates, 25.
  6. diSessa, A. 2000, Changing Minds: Computers, Learning, and Literacy, Boston, MA: MIT Press, 90.
  7. George, D., and Mallery, P. 2003, SPSS for Windows Step by Step: A Simple Guide and Reference, 4th ed., 11.0 Update, Boston, MA: Allyn & Bacon/Pearson,
  8. Hake, R. R. 1998, “Interactive-Engagement versus Traditional Methods: A Six-Thousand-Student Survey of Mechanics Test Data for Introductory Physics Courses,” American Journal of Physics, 66, 64.
  9. Hambleton, R. K., and Jones, R. J. 1993, “Comparison of Classical Test Theory and Their Applications to Test Development,” Education Measurement: Issues & Practices, 12, 253.
  10. Lord, F. M., and Novick, M. R. 1968, Statistical Theories of Mental Test Scores, Reading, MA: Addison-Wesley.
  11. Prather, E. E., Rudolph, A. L., Brissenden, G., and Schlingman, W. M. 2009, “A National Study Assessing the Teaching and Learning of Introductory Astronomy. Part I. The Effect of Interactive Instruction,” American Journal of Physics, 77, 320AJPIAS000077000004000320000001.
  12. Rudolph, A. L., Prather, E. E., Brissenden, G., Consiglio, D., and Gonzaga, V. 2010, “A National Study Assessing the Teaching and Learning of Introductory Astronomy. Part II: The Connection between Student Demographics and Learning,” Astronomy Education Review, 9, 010107AERSCZ000009000001010107000001.
  13. Thompson, B. 2003, “Understanding Reliability and Coefficient alpha, Really,” in Score Reliability, ed. B. Thompson, Thousand Oaks, CA: SAGE Publications, 3.
  14. Wallace, C. S., and Bailey, J. M. 2010, “Do Concept Inventories Actually Measure Anything?,” Astronomy Education Review, 9, 010116AERSCZ000009000001010116000001.

Figures (8) Tables (1)

Figures (click on thumbnails to view enlargements)

FIG.1
The pre-instruction difficulty and post-instruction difficulty for each item of the LSCI are plotted along the horizontal and vertical axes, respectively. Items that become easier to answer after instruction will lie below the diagonal line

FIG.1 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

FIG.2
The pre-instruction discrimination and post-instruction discrimination for each item of the LSCI are plotted along the horizontal and vertical axes respectively. Items that increase in discrimination after instruction will lie above the diagonal line

FIG.2 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

FIG.3
The difference in difficulty, post- minus pre-, for each item of the LSCI, is plotted on the horizontal axis. Items that become easier post-instruction will be farther to the left along the horizontal axis. The difference in discrimination, post- minus pre-, for each item, is plotted on the vertical axis. Items with an increased discrimination value post-instruction will be farther up the vertical axis. The items showing improvement due to instruction are shown above the horizontal line and to the left of the vertical line

FIG.3 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

FIG.4
The post-instruction score versus pre-instruction score for the 1881 students with matched data (blue circles). The different shades of blue represent the number of students that have the pre-instruction post-instruction score combination at a given point. The line provided identifies the location of equal pre-instruction and post-instruction scores

FIG.4 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

FIG.5
The normalized gain versus pre-instruction score for the 1881 students with matched data (blue triangles). The horizontal line is the zero normalized gain line. Any point below this line corresponds to students that did worse post-instruction than they did pre-instruction

FIG.5 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

FIG.6
Item 3 from the LSCI

FIG.6 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

FIG.7
Item 21 from the LSCI

FIG.7 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

FIG.8
Item 25 from the LSCI

FIG.8 Download High Resolution Image (.zip file) | Export Figure to PowerPoint

Tables

Table I. The LSCI’s items’ difficulties and discriminations, calculated for both pre-instruction and postinstruction student responses. Bolded items are outside the conventionally accepted parameter ranges for item difficulty and/or discrimination

View Table


Close
ADVERTISEMENT

Copyright © 2001 - American Astronomical Society. All rights reserved.

close