The survey instrument used in this study has been stable, using the same questions, since 1988 and has two parts. The first is based on, and has a substantial degree of overlap with, the instrument used by the National Science Foundation (NSF) in the biennial survey of public knowledge of science and technology. It is designed to measure general scientific knowledge using a set of 21 knowledge-based questions, four of which are open-ended with short written answers and 17 of which require true/false or multiple choice responses. The second part is original and is designed to measure attitudes about science and technology, perceptions of and susceptibility to some forms of pseudoscience, and certain aspects of faith and religious belief. It utilizes a set of 24 statements where the responses are on a five-point Likert scale. For a more detailed discussion of the instrument, its reliability and completion rate, plus a description of the data entry and coding process, see Impey et al. (2011)17
The survey is administered each year to students in large astronomy lecture courses that consist mostly of freshmen and sophomores and satisfy the natural sciences General Education requirement at the University of Arizona. Administered in the first week of class, it is voluntary and does not count for any part of a student's grade. Ten to fifteen minutes are allowed for its completion. Although the survey is anonymous, a modest amount of demographic information is collected: gender, major, grade-point average, class standing, and the number of college science courses already completed. The survey is administered on paper, and subsequent data entry and coding are done manually. General characteristics of the sample are that the over 11 000 respondents are equally divided into men and women; the fractions of freshmen, sophomores, juniors, and seniors are 47%, 31%, 12%, and 10%; the largest three categories of major are business and professional (33%), social and behavioral sciences (21%), and fine arts and humanities (16%); and 41% of the sample had not previously taken any college science classes, while 12% had completed the General Education requirement of three one-semester long science classes intended for freshmen and sophomores (and not requiring labs).
The survey measures some aspects of science literacy with 17 forced-choice knowledge-based questions that cover a broad range of scientific subjects, from physics (“Which travels faster, light or sound?”) and astronomy (“Does the Earth go around the Sun, or does the Sun go around the Earth?”) to biology (“The oxygen we breathe comes from plants, true or false?”). In the analysis presented here, we used 15 of the 17 forced-choice questions to have maximum overlap with the NSB Science Indicators survey; the 15 questions used here include 9 that are consistently part of the NSB survey. From the answers to these questions, each student gets a score on a scale from 0 to 15, with each correct question worth one point. Four open-ended questions ask for definitions of the scientific method, DNA, radiation, and computer software; they were not used for the analysis in this paper.
Students also responded to a set of 24 statements on a five-point Likert-scale of strongly agree, agree, no opinion, disagree, and strongly disagree. A sampling of the items gives a sense of the range of topics addressed. Statements like “Pure science should be funded regardless of its lack of immediate benefit to society” and “Genetic engineering is a good idea” probe general attitudes towards science and technology. “We should make a concerted effort to search for life on other planets” addresses expectations in astrobiology, while a related item “Some ancient civilizations were visited by extraterrestrials” alludes to belief in unidentified flying objects (UFOs). Others such as “There are some circumstances when medical science should not be used to prolong life” address ethical issues. Another set of items probes superstitions and beliefs in pseudoscience; examples include “Some numbers are especially lucky for some people” and “The positions of the planets have an effect on everyday life.” There are also statements that relate to religious belief, such as “The Biblical story of creation should be taught alongside evolution in our schools” and “Faith healing is a valid alternative to conventional medicine.” The survey is shown in Appendix A.
Some caveats and qualifications are in order. It is not possible to capture the nuances of any individual's science beliefs and knowledge in a fifteen-minute survey. Nor is it possible to combine such results into an “identikit” picture of the beliefs and knowledge of a population; the variance of responses on any particular survey item is large, so information is lost in forming means or looking for trends. Moreover, items in the survey are not uniformly strong and do not have equal diagnostic power. This is not as much of a problem with the forced-choice and open-ended science knowledge questions, which have been used and validated by the NSB for over two decades, as it is for some of the attitudinal Likert-scale statements. For example, “The universe was created in an enormous explosion billions of years ago” might elicit student disagreement because they did not agree with the scientific story of creation or because they did not agree with the overly declarative phrasing (or because they recognized that an explosion is a poor analogy for the instantaneous creation of all space-time). Similarly, a student might concur with the statement “UFOs are real and should be investigated” because they think alien space ships have visited the Earth or because they think that unidentified aerial phenomena are worthy of further study. Disagreement with positive statements about genetic engineering and nuclear power might arise from lack of information and antipathy or from a balanced judgment based on a high degree of information. Nonetheless, we contend that the survey has utility in the aggregate, and for example, as we will show in Section 4, the items that relate to pseudoscience have diagnostic power as a group.