1. The term mental model was first coined to refer to complex conceptual representations, such as understanding the workings of an electrical grid or an automobile engine. We extend the use here to motor skills, referring to what are sometimes called motor schemas.
2. The data about student study strategies come from a survey by J. D. Karpicke, A. C. Butler, & H. L. Roediger, Metacognitive strategies in student learning: Do students practice retrieval when they study on their own?, Memory 17 (2010), 471–479.
3. Peter Brown interview of Matt Brown, March 28, 2011, Hastings, MN. All quotes of Matt Brown are from this interview.
4. Find this advice online at http://caps.gmu.edu/educationalprograms/pamphlets/StudyStrategies.pdf, accessed November 1, 2013.
5. Find this advice online at www.dartmouth.edu/~acskills/docs/study_actively.doc, accessed November 1, 2013.
6. The study advice cited from the St. Louis Post-Dispatch is distributed by Newspapers in Education and can be seen online in “Testing 1, 2, 3! How to Study and Take Tests,” p14, at http://nieonline.com/includes/hottopics/Testing%20Testing%20123.pdf, accessed November 2, 2013.
7. The studies showing the futility of mere repetition in recalling the details of what a penny looks like or where a fire extinguisher is located in a building are in R. S. Nickerson & M. J. Adams, Long term memory of a common object, Cognitive Psychology 11 (1979), 287–307, and A. D. Castel, M. Vendetti, & K. J. Holyoak, Inattentional blindness and the location of fire extinguishers, Attention, Perception and Performance 74 (2012), 1391–1396.
8. The experiment referred to by Tulving was reported in E. Tulving, Subjective organization and the effects of repetition in multi-trial free recall learning, Journal of Verbal Learning and Verbal Behavior 5 (1966), 193–197.
9. The experiment on how rereading does not produce much benefit in later retention is from A. A. Callender & M. A. McDaniel, The limited benefits of rereading educational texts, Contemporary Educational Psychology 34 (2009), 30–41.
10. The survey showing that students prefer to reread as a study strategy is from Karpicke et al., Metacognitive strategies. Data were also taken from J. McCabe, Metacognitive awareness of learning strategies in undergraduates, Memory & Cognition 39 (2010), 462–476.
11. Illusions of knowing will be a theme throughout this book. A general reference is Thomas Gilovich, How We Know What Isn’t So: The Fallibility of Human Reason in Everyday Life (New York: Free Press, 1991).
12. R. J. Sternberg, E. L. Grigorenko, & L. Zhang, Styles of learning and thinking matter in instruction and assessment, Perspectives on Psychological Science 3 (2008), 486–506.
13. The project at Columbia Middle School is reported in M. A. McDaniel, P. K. Agarwal, B. J. Huelser, K. B. McDermott, & H. L. Roediger (2011). Test-enhanced learning in a middle school science classroom: The effects of quiz frequency and placement. Journal of Educational Psychology, 103, 399–414.
14. The concept of testing as a learning tool is described in detail in Chapter 2. A general reference on material in this chapter (and other educational applications of cognitive psychology to education) is M. A. McDaniel & A. A. Callender, Cognition, memory, and education, in H. L. Roediger, Cognitive Psychology of Memory, vol. 2 of Learning and Memory: A Comprehensive Reference (Oxford: Elsevier, 2008), pp. 819–844.
1. Peter Brown interview of Michael Ebersold, December 31, 2011, Wabasha, MN. All quotes from Ebersold are from this interview.
2. The early work on forgetting curves was published by Hermann Ebbinghaus in 1885 in a book translated into English as On Memory in 1913. The most recent version is H. Ebbinghaus, Memory: A contribution to experimental psychology (New York: Dover, 1964). Ebbinghaus is often viewed as the “father” of the scientific study of memory.
3. The quotes from Aristotle and Bacon are from H. L. Roediger & J. D. Karpicke, The power of testing memory: Basic research and implications for educational practice, Perspectives on Psychological Science 1 (2006), 181–210.
4. Benedict Carey, “Forget what you know about good study habits,” New York Times, September 7, 2010. The study reported in this article was H. L. Roediger & J. D. Karpicke, Test-enhanced learning: Taking memory tests improves long-term retention, Psychological Science 17 (2006), 249–255.
5. A. I. Gates, Recitation as a factor in memorizing, Archives of Psychology 6 (1917) and H. F. Spitzer, Studies in retention, Journal of Educational Psychology 30 (1939), 641–656. These two large-scale studies with children in elementary and middle school were among the first to document that taking a test or reciting material appearing in didactic texts improved retention for that material.
6. The study involving repeated testing versus repeated studying was E. Tulving, The effects of presentation and recall of material in free-recall learning, Journal of Verbal Learning and Verbal Behavior 6 (1967), 175–184. The study involving amounts of forgetting being reduced from testing is M. A. Wheeler & H. L. Roediger, Disparate effects of repeated testing: Reconciling Ballard’s (1913) and Bartlett’s (1932) results, Psychological Science 3 (1992), 240–245.
7. The positive effects of generation appear in L. L. Jacoby, On interpreting the effects of repetition: Solving a problem versus remembering a solution, Journal of Verbal Learning and Verbal Behavior 17 (1978), 649–667. This laboratory experiment demonstrated that generation of target information does not have to be exceptionally challenging in order for generation to produce better retention relative to reviewing information to be learned.
8. Two papers describing the research at Columbia Middle School are H. L. Roediger, P. K. Agarwal, M. A. McDaniel, & K. McDermott, Test-enhanced learning in the classroom: Long-term improvements from quizzing, Journal of Experimental Psychology: Applied 17 (2011), 382–395, and M. A. McDaniel, P. K. Agarwal, B. J. Huelser, K. B. McDermott, & H. L. Roediger, Test-enhanced learning in a middle school science classroom: The effects of quiz frequency and placement, Journal of Educational Psychology 103 (2011), 399–414. These companion papers were the first to report well-controlled experiments on the benefits of quizzing for middle school students’ performances on classroom exams in social studies and science. The findings demonstrated that quizzing produced a significant improvement relative to no-quizzing or directed review of target concepts on unit exams and on cumulative semester and end-of-year exams. In addition, in some cases a single well-placed review quiz produced benefits on the exams that were as robust as several repeated quizzes. For an interesting view of this project by one of the lead researchers, the first teacher and the first principal involved, see P. K. Agarwal, P. M. Bain, & R. W. Chamberlain, The value of applied research: Retrieval practice improves classroom learning and recommendations from a teacher, a principal, and a scientist. Educational Psychology Review 24 (2012), 437–448.
9. Peter Brown interview of Roger Chamberlain, October 27, 2011, Columbia Middle School, Illinois. All quotes from Chamberlain are from this interview.
10. Peter Brown interview of Andrew Sobel, December 22, 2011, St. Louis, Missouri. All quotes from Sobel are from this interview.
11. The experiments described here are by H. L. Roediger & J. D. Karpicke, Test-enhanced learning: Taking memory tests improves long-term retention, Psychological Science 17 (2006), 249–255. Experiments showing that recall of studied prose passages produced better 2-day and one-week retention than did restudy of the passages. For an earlier study with the same outcome using word lists, see C. P. Thompson, S. K. Wenger, & C. A. Bartling, How recall facilitates subsequent recall: A reappraisal. Journal of Experimental Psychology: Human Learning and Memory 4 (1978), 210–221. This experiment showed that massing study was better than practicing retrieval on an immediate test but not a delayed test.
12. Many studies exist on the effects of feedback. One is A. C. Butler & H. L. Roediger, Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing. Memory & Cognition 36 (2008), 604–616. The experiments show that feedback strengthens the effects of testing alone, and that feedback may be more beneficial when it’s slightly delayed. The authors also showed that that feedback enhances the positive effects and reduces the negative effects of multiple-choice testing. For motor skills, a classic reference is A. W. Salmoni, R. A. Schmidt, and C. B. Walter, Knowledge of results and motor learning: A review and critical reappraisal. Psychological Bulletin 95 (1984), 355–386. The authors proposed the guidance hypothesis of feedback effects on motor learning: Frequent immediate feedback can be detrimental to long-term learning—even though it helps immediate performance—because it provides a crutch during practice that is no longer present on a delayed test.
13. The open-book test study was P. K. Agarwal, J. D. Karpicke, S. H. K. Kang, H. L. Roediger, & K. B. McDermott, Examining the testing effect with open- and closed-book tests, Applied Cognitive Psychology 22 (2008), 861–876.
14. Studies comparing the types of tests are S. H. Kang, K. B. McDermott, H. L. Roediger, Test format and corrective feedback modify the effect of testing on long-term retention. European Journal of Cognitive Psychology 19 (2007), 528–558, and M. A. McDaniel, J. L. Anderson, M. H. Derbish, & N. Morrisette, Testing the testing effect in the classroom. European Journal of Cognitive Psychology 19 (2007), 494–513. These parallel experiments, one conducted in the laboratory and one in a college course, showed that a short-answer quiz with feedback produced better gains on final tests than a recognition quiz with feedback. The implication is that the testing effect is more robust when more effort is required for retrieval, as it typically is for short-answer questions than for multiple choice questions. However, some studies have shown that multiple choice tests, especially when given repeatedly, can have as much positive effect in the classroom as a short-answer test; see K. B. McDermott, P. K. Agarwal, L. D’Antonio, H. L. Roediger, & M. A. McDaniel, Both multiple-choice and short-answer quizzes enhance later exam performance in middle and high school classes, Journal of Experimental Psychology: Applied (in press).
15. These studies examined students’ use of testing as a study strategy: J. D. Karpicke, A. C. Butler, & H. L. Roediger, III, Metacognitive strategies in student learning: Do students practice retrieval when they study on their own?, Memory 17 (2009), 471–479, and N. Kornell & R. A. Bjork, The promise and perils of self regulated study, Psychonomic Bulletin & Review 14 (2007), 219–224. These studies reported the surveys of college students’ use of retrieval practice as study technique.
16. Taking a test—even when one fails to correctly recall information on it—enhances learning from a new study episode. See K. M. Arnold & K. B. McDermott, Test-potentiated learning: Distinguishing between the direct and indirect effects of tests, Journal of Experimental Psychology: Learning, Memory and Cognition 39 (2013), 940–945.
17. This is a study of frequent low-stakes testing: F. C. Leeming, The exam-a-day procedure improves performance in psychology classes, Teaching of Psychology 29 (2002), 210–212. The author found that in sections in which he gave students a short test at the start of every class the students attended class more often and felt that they studied more and learned more than students in classes with only four tests throughout the semester. Final test performance for the different sections (quiz a day or no quiz a day) confirmed students’ impressions. Another interesting study conducted in a classroom is K. B. Lyle & N. A. Crawford, Retrieving essential material at the end of lectures improves performance on statistics exams, Teaching of Psychology 38 (2011), 94–97.
Two reviews of research on retrieval practice and testing appear in H. L. Roediger & J. D. Karpicke, The power of testing memory: Basic research and implications for educational practice, Perspectives on Psychological Science 1 (2006), 181–210. This paper represents a comprehensive review of laboratory and classroom studies over nearly one hundred years of research, showing that testing can be a powerful learning tool. A more recent review points to many benefits of frequent testing in addition to the direct benefit from retrieval practice: H. L. Roediger, M. A. Smith, & A. L. Putnam, Ten benefits of testing and their applications to educational practice, in J. Mestre & B. H. Ross (eds.), Psychology of Learning and Motivation (San Diego: Elsevier Academic Press, 2012). This chapter provides a summary of the host of potential benefits of using testing as a learning technique.
1. The report of the beanbag study can be found in R. Kerr & B. Booth, Specific and varied practice of motor skill, Perceptual and Motor Skills 46 (1978), 395–401.
2. Many well-controlled experiments conducted with a variety of materials and training tasks provide solid evidence that massed practice (doing the same thing over and over repeatedly, a strategy often preferred by learners) is inferior to spacing and interleaving of practice for learning and retention. A review of the literature on the spacing effect in memory can be found in N. J. Cepeda, H. Pashler, E. Vul, J. T. Wixted, & D. Rohrer, Distributed practice in verbal recall tasks: A review and quantitative synthesis, Psychological Bulletin 132 (2006), 354–380.
3. The surgery study is C-A. E. Moulton, A. Dubrowski, H. MacRae, B. Graham, E. Grober, & R. Reznick, Teaching surgical skills: What kind of practice makes perfect?, Annals of Surgery 244 (2006), 400–409. This study randomly assigned surgical residents to either a normal daylong intensive lesson on a surgical procedure or to an experimental lesson that spaced four short periods of instruction over several weeks. The findings, showing better retention and application of the surgical techniques after spaced instruction, prompted the medical school to reexamine their standard instructional procedure of cramming instruction on a particular surgical technique into one intensive session.
4. The study showing the benefit of interleaving in mathematics problems is D. Rohrer & K. Taylor, The shuffling of mathematics problems improves learning, Instructional Science 35 (2007), 481–498. The standard practice in mathematics textbooks is to cluster practice problems by problem type. This laboratory experiment demonstrated that this standard practice produced inferior performance on a final test in which new problems of each problem type were given relative to a practice procedure in which the practice problems from different problem types were shuffled (interleaved).
5. The study relating differences in practice strategies to differences in motor-memory consolidation was by S. S. Kantak, K. J. Sullivan, B. E. Fisher, B. J. Knowlton, & C. J. Winstein, Neural substrates of motor memory consolidation depend on practice structure, Nature Neuroscience 13 (2010), 923–925.
6. The anagram study was by M. K. Goode, L. Geraci, & H. L. Roediger, Superiority of variable to repeated practice in transfer on anagram solution, Psychonomic Bulletin & Review 15 (2008), 662–666. These researchers gave subjects practice on solving anagrams for a set of words: one group was given the same anagram for a particular target word on every practice trial (massed practice), whereas another group was given a different anagram for a particular target word on each practice trial (varied practice). Surprisingly, varied practice produced better performance on a final trial in which the anagrams were the very ones that were practiced in the other group that had practiced the tested anagram repeatedly.
7. The study about learning of artists’ styles was by N. Kornell & R. A. Bjork, Learning concepts and categories: Is spacing the “enemy of induction”?, Psychological Science 19 (2008), 585–592. In these experiments, college students attempted to learn the painting style of a number of relatively unknown artists. Students learned the styles better when the paintings of the artists were interleaved compared to when each artist’s paintings were massed during learning. Yet, at odds with the objective learning outcomes, most of the learners insisted that they learned better with the massed presentations. Another informative study is S. H. K. Kang & H. Pashler, Learning painting styles: Spacing is advantageous when it promotes discriminative contrast, Applied Cognitive Psychology 26 (2012), 97–103, which showed that mixing the examples of paintings helped to highlight the differences among painters’ styles (what we are calling discriminative contrast).
8. The finding that improving discrimination among examples contributes to conceptual learning is from L. L. Jacoby, C. N. Wahlheim, & J. H. Coane, Test-enhanced learning of natural concepts: effects on recognition memory, classification, and metacognition, Journal of Experimental Psychology: Learning, Memory, and Cognition 36 (2010), 1441–1442.
9. Peter Brown interview of Doulas Larsen, December 23, 2011, St. Louis, MO. All quotes from Larsen are from this interview.
10. Doug Larsen’s work can be found in D. P. Larsen, A. C. Butler, & H. L. Roediger, Repeated testing improves long-term retention relative to repeated study: a randomized controlled trial. Medical Education 43 (2009), 1174–1181; D. P. Larsen, A. C. Butler, A. L. Lawson, & H. L. Roediger, The importance of seeing the patient: Test-enhanced learning with standardized patients and written tests improves clinical application of knowledge, Advances in Health Science Education 18 (2012), 1–17; and D. P. Larsen, A. C. Butler, & H. L. Roediger, Comparative effects of test-enhanced learning and self-explanation on long-term retention, Medical Education 47, 7 (2013), 674–682.
11. Peter Brown interview of Vince Dooley, February 18, 2012, Athens, GA. All quotes of Dooley are from this interview.
12. Psychologists interested in learning have long distinguished between momentary performance and underlying learning (as measured after a delay with intervening reminders). As a simple example, someone might tell you that James Monroe was the fifth US president. You would probably be able to answer correctly if asked about the fifth president for the rest of the day or the week. That would be due to having just heard it (thus boosting the momentary strength or what the psychologists Robert and Elizabeth Bjork call retrieval strength). However, if someone asks you a year later about the fifth president, this would be a measure of habit strength or, as the Bjorks call it, storage strength. See R. A. Bjork & E. L. Bjork, A new theory of disuse and an old theory of stimulus fluctuation, in A. F. Healy, S. M. Kosslyn, & R. M. Shiffrin (eds.), From learning processes to cognitive processes: Essays in honor of William K. Estes (vol. 2, pp. 35–67) (Hillsdale, NJ: Erlbaum, 1992). For a recent discussion, see N. C. Soderstrom & R. A. Bjork, Learning versus performance, in D. S. Dunn (ed.), Oxford Bibliographies online: Psychology (New York: Oxford University Press, 2013) doi 10. 1093/obo/9780199828340-0081.
1. All quotes of Mia Blundetto are from telephone conversations between Peter Brown, in Austin, TX, and Blundetto, at Camp Fuji, Japan, on February 9 and March 2, 2013.
2. The phrase “desirable difficulties in learning” originated in the article R. A. Bjork & E. L. Bjork, A new theory of disuse and an old theory of stimulus fluctuation, in A. F. Healy, S. M. Kosslyn, & R. M. Shiffrin (eds.), From learning processes to cognitive processes: Essays in honor of William K. Estes (vol. 2, pp. 35–67) (Hillsdale, NJ: Erlbaum, 1992). The idea seems counterintuitive—how can making a task more difficult lead to it’s being learned better and retained longer? The rest of this chapter explains this puzzle and why it seems to arise.
3. Psychologists distinguish among three stages in the learning/memory process: Encoding (or acquisition of information); storage (persistence of information over time); and retrieval (later use of the information). Any time you successfully remembered an event, all three stages were intact. Forgetting (or the occurrence of false memories—retrieving a wrong “memory” of some event but believing it to be right) can occur at any stage.
4. For a classic article on consolidation, see J. L. McGaugh, Memory—a century of consolidation, Science 287 (2000), 248–251. For a somewhat more recent and lengthy review, see Y. Dudai, The neurobiology of consolidations, or, how stable is the engram?, Annual Review of Psychology 55 (2004), 51–86. For evidence that sleep and dreaming helps with memory consolidation, see E. J. Wamsley, M. Tucker, J. D. Payne, J. A. Benavides, & R. Stickgold, Dreaming of a learning task is associated with enhanced sleep-dependent memory consolidation, Current Biology 20 (2010), 850–855.
5. Endel Tulving emphasized the critical role of retrieval cues in remembering by stressing that remembering is always a product of both the information stored (the memory trace) and the cues in the environment that might remind you of the information. With stronger cues, even weaker traces become accessible for recall. See E. Tulving, Cue dependent forgetting, American Scientist 62 (1974), 74–82.
6. Robert Bjork has emphasized the role of forgetting of an original event to some degree as aiding the amount of learning from a second presentation of the same event. The power of spacing of events on memory (the spacing effect) is one example. For examples see N. C. Soderstrom & R. A. Bjork, Learning versus performance, in D. S. Dunn (ed.), Oxford Bibliographies in Psychology (New York: Oxford University Press, in press).
7. The problem of old learning interfering with new learning is called negative transfer in psychology. For evidence on how forgetting of old information can help in learning of new information, see R. A. Bjork, On the symbiosis of remembering, forgetting, and learning, in A. S. Benjamin (ed.), Successful Remembering and Successful Forgetting: A Festschrift in Honor of Robert A. Bjork (pp. 1–22) (New York: Psychology Press, 2010).
8. The situation where information still exists in memory yet cannot be actively recalled has been emphasized as a key problem in remembering (Tulving, Cue dependent forgetting). Stored information is said to be available, whereas retrievable information is accessible. The instance we give in this chapter of an old address that a person cannot recall but could easily recognize among several possibilities is an example of the power of retrieval cues in making available memories accessible to conscious awareness. Recognition tests usually provide more powerful cues than recall tests.
9. The study of baseball players practicing hitting was reported in K. G. Hall, D. A. Domingues, & R. Cavazos, Contextual interference effects with skilled baseball players, Perceptual and Motor Skills 78 (1994), 835–841.
10. “Reload” is the term the Bjorks use to indicate reconstruction of a concept or skill after some delay. A good, accessible source for these ideas is E. L. Bjork & R. A. Bjork, Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning, in M. A. Gernsbacher, R. W. Pew, L. M. Hough, & J. R. Pomerantz (eds.), Psychology and the real world: Essays illustrating fundamental contributions to society (pp. 56–64) (New York: Worth, 2009).
11. The term reconsolidation has several different uses in psychology and neuroscience. The core meaning is the reviving of an original memory and then having it consolidate again (as in retrieval practice). However, the original memory can be changed by reconsolidation if new information is introduced when the original memory is revived. Reconsolidation has been studied by both neurobiologists and cognitive psychologists. Some entry points into this literature are D. Schiller, M. H. Monfils, C. M. Raio, D. C. Johnson, J. E. LeDoux, & E. A. Phelps, Preventing the return of fear in humans using reconsolidation update mechanisms, Nature 463 (2010), 49–53, and B. Finn & H. L. Roediger, Enhancing retention through reconsolidation: Negative emotional arousal following retrieval enhances later recall, Psychological Science 22 (2011), 781–786.
12. For the research on interleaving, see M. S. Birnbaum, N. Kornell, E. L. Bjork, & R. A. Bjork, Why interleaving enhances inductive learning: The roles of discrimination and retrieval, Memory & Cognition 41 (2013), 392–402.
13. Several studies have shown that although making text more difficult to read by leaving out letters or using an unusual typography may slow reading, readers remember more. See M. A. McDaniel, G. O. Einstein, P. K. Dunay, & R. Cobb, Encoding difficulty and memory: Toward a unifying theory, Journal of Memory and Language 25 (1986), 645–656, and C. Diemand-Yauman, D. Oppenheimer, & E. B. Vaughn, Fortune favors the bold (and the italicized): Effects of disfluency on educational outcomes, Cognition 118 (2010), 111–115. The study in which the outline either matched or mismatched the chapter is S. M. Mannes & W. Kintsch, Knowledge organization and text organization, Cognition and Instruction 4 (1987), 91–115.
14. Studies showing that generation can improve retention include L. L. Jacoby, On interpreting the effects of repetition: Solving a problem versus remembering a solution, Journal of Verbal Learning and Verbal Behavior 17 (1978), 649–667, and N. J. Slamecka & P. Graf, The generation effect: Delineation of a phenomenon, Journal of Experimental Psychology: Human Learning and Memory 4 (1978), 592–604. More recently, the act of generation before a learning episode has also been shown to enhance performance; see L. E. Richland, N. Kornell, & L. S. Kao, The pretesting effect: Do unsuccessful retrieval attempts enhance learning? Journal of Experimental Psychology: Applied 15 (2009), 243–257.
15. The cited study of write-to-learn is K. J. Gingerich, J. M. Bugg, S. R. Doe, C. A. Rowland, T. L. Richards, S. A. Tompkins, & M. A. McDaniel, Active processing via write-to-learn assignments: Learning and retention benefits in introductory psychology, Teaching of Psychology, (in press).
16. B. F. Skinner had many influential and interesting ideas about learning in schools as well as on other topics in American society. His important book Science and Human Behavior can be downloaded at no cost from the website of the B. F. Skinner Foundation. See also B. F. Skinner, Teaching machines, Science 128 (1958), 969–977. Errorless learning does seem important in teaching memory-impaired people, but for most educational situations, errors (so long as they are corrected with feedback) do not hurt and may even aid learning. For example, see B. J. Huelser & J. Metcalfe, Making related errors facilitates learning, but learners do not know it, Memory & Cognition 40 (2012), 514–527.
17. The French study on schoolchildren solving anagrams appears in F. Autin & J. C. Croziet, Improving working memory efficiency by reframing metacognitive interpretation of task difficulty, Journal of Experimental Psychology: General 141 (2012), 610–618. For a story on the Festival of Errors, see Lizzy Davis, “Paris Stages ‘Festival of Errors’ to Teach French Schoolchildren How to Think,” Guardian, July 21, 2010, http://www.guardian.co.uk/world/2010/jul/21/france-paris-festival-of-errors, accessed October 22, 2013.
18. Peter Brown telephone interview of Bonnie Blodgett, March 10, 2013, St. Paul, MN. All quotes of Blodgett are from this interview.
19. The quote from the Bjorks comes from E. L. Bjork & R. A. Bjork, Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning, in M. A. Gernsbacher, R. W. Pew, L. M. Hough, and J. R. Pomerantz (eds.), Psychology and the real world: Essays illustrating fundamental contributions to society (pp. 56–64) (New York: Worth, 2009).
1. The field of metacognition—what we know about what we know and how we assess our performance—is a burgeoning one in psychology. A good general reference about metacognition is John Dunlosky and Janet Metcalfe, Metacognition (Los Angeles: Sage, 2009). Daniel Kahneman, Thinking Fast and Slow (New York: Farrar, Strauss and Giroux, 2011), also includes a discussion of many illusions to which the mind falls prey. For an earlier discussion of many illusions, see Thomas Gilovich, How We Know What Isn’t So: The Fallibility of Human Reason in Everyday Life (New York: Free Press, 1991). For a briefer review, see H. L. Roediger, III, & A. C. Butler, Paradoxes of remembering and knowing, in N. Kapur, A. Pascual-Leone, & V. Ramachandran (eds.), The Paradoxical Brain (pp. 151–176) (Cambridge: Cambridge University Press, 2011).
2. Peter Brown interview of David Garman, December 12, 2011, Minneapolis, MN. All quotes of Garman are from this interview.
3. The China Airlines incident is reported in: National Transportation Safety Board, “Aircraft Accident report–China Airlines Boeing 747-SP N4522V, 300 Nautical Miles Northwest of San Francisco, California, February 19, 1985,” March 29, 1986, and can be found at http://www.rvs.uni-bielefeld.de/publications/Incidents/DOCS/ComAndRep/ChinaAir/AAR8603.html, accessed October 24, 2013.
The report of the National Transportation Safety Board’s investigation into the Carnahan accident is reported by: D. A. Lombardo, “ ‘Spatial disorientation’ caused Carnahan crash,” Aviation International News, AINonline, July 2002, and can be found at: http://www.ainonline.com/aviation-news/aviation-international-news/2008-04-16/spatial-disorientation-caused-carnahan-crash, accessed October 24, 2013.
The report of the National Transportation Safety Board’s investigation into the J. F. Kennedy Jr. accident is reported by: N. Sigelman, “NTSB says spatial disorientation caused Cape Air crash,” Martha’s Vineyard Times, mntimes.com, and can be found at http://www.mvtimes.com/ntsb-says-spatial-disorientation-caused-cape-air-crash-960/, accessed October 24, 2013.
4. E. Morris, “The anosognosic’s dilemma: Something’s wrong but you’ll never know what it is” (pt. 5), New York Times, June 24, 2010.
5. L. L. Jacoby, R. A. Bjork, & C. M. Kelley, Illusions of comprehension, competence, and remembering, in D. Druckman & R. A. Bjork (eds.), Learning, remembering, believing: Enhancing human performance (pp. 57–80) (Washington, DC: National Academy Press, 1994).
6. The Carol Harris/Helen Keller study is reported in R. A. Sulin & D. J. Dooling, Intrusion of a thematic idea in retention of prose, Journal of Experimental Psycholog 103 (1974), 255–262. For an overview on memory illusions, see H. L. Roediger & K. B. McDermott, Distortions of memory, in F. I.M. Craik & E. Tulving (eds.), The Oxford Handbook of Memory (pp. 149–164) (Oxford: Oxford University Press, 2000).
7. Imagination inflation has been shown both in studies of memories from early life and in laboratory studies. Two of the original references for each type of study are M. Garry, C. G.Manning, E. F. Loftus, & S. J. Sherman, Imagination inflation: Imagining a childhood event inflates confidence that it occurred, Psychonomic Bulletin & Review 3 (1996), 208–214, and L. M. Goff & H. L. Roediger, Imagination inflation for action events: Repeated imaginings lead to illusory recollections, Memory & Cognition 26 (1998), 20–33.
8. The leading questions experiment is E. F. Loftus & J. C. Palmer, Reconstruction of automobile destruction: An example of the interaction between language and memory, Journal of Verbal Learning and Verbal Behavior 13 (1974), 585–589.
9. One article on the dangers of hypnosis on memory is P. A. Register & J. F. Kihlstrom, Hypnosis and interrogative suggestibility, Personality and Individual Differences 9 (1988), 549–558. For an overview of issues in memory relevant to legal situations, see H. L. Roediger & D. A. Gallo, Processes affecting accuracy and distortion in memory: An overview, in M. L. Eisen, G. S. Goodman, & J. A. Quas (eds.), Memory and Suggestibility in the Forensic Interview (pp. 3–28) (Mahwah, NJ: Erlbaum, 2002).
10. The story about Don Thomson can be found in B. Bower, Gone but not forgotten: Scientists uncover pervasive unconscious influences on memory, Science News 138, 20 (1990), 312–314.
11. The curse of knowledge, hindsight bias, and other topics are covered in Jacoby, Bjork, & Kelley, Illusions of comprehension, competence, and remembering, and in many other places. A relatively recent review of the effects of fluency can be found in D. M. Oppenheimer, The secret life of fluency, Trends in Cognitive Science 12 (2008), 237–241.
12. Social contagion of memory: H. L. Roediger, M. L. Meade, & E. Bergman, Social contagion of memory, Psychonomic Bulletin & Review 8 (2001), 365–371
13. Two important reviews of the false consensus effect are found in L. Ross, The false consensus effect: An egocentric bias in social perception and attribution processes, Journal of Experimental Social Psychology 13 (1977), 279–301, and G. Marks, N. Miller, Ten years of research on the false-consensus effect: An empirical and theoretical review, Psychological Bulletin 102 (1987), 72–90.
14. Flashbulb memories of 9/11: J. M. Talarico & D. C. Rubin, Confidence, not consistency, characterizes flashbulb memories, Psychological Science 14 (2003), 455–461, and W. Hirst, E. A. Phelps, R. L. Buckner, A. Cue, D.E. Gabrieli & M.K. Johnson Long-term memory for the terrorist attack of September 11: Flashbulb memories, event memories and the factors that influence their retention, Journal of Experimental Psychology: General 138 (2009), 161–176.
15. Eric Mazur material comes from his YouTube lecture “Confessions of a converted lecturer,” available at www.youtube.com/watch?v=WwslBPj8GgI, accessed October 23, 2013.
16. The curse of knowledge study about guessing tunes tapped out is from L. Newton, Overconfidence in the communication of intent: Heard and unheard melodies (Ph.D. diss., Stanford University, 1990).
17. The Dunning-Kruger effect originated with Justin Kruger & David Dunning, Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments, Journal of Personality and Social Psychology 77 (1999), 1121–1134. Many later experimental studies and articles have been based on this one. See D. Dunning, Self-Insight: Roadblocks and Detours on the Path to Knowing Thyself (New York: Psychology Press, 2005).
18. Stories on student-directed learning: Susan Dominus, “Play-Dough? Calculus? At the Manhattan Free School, Anything Goes,” New York Times, October 4, 2010, and Asha Anchan, “The DIY Approach to Education,” Minneapolis StarTribune, July 8, 2012.
19. Studies showing that students drop flashcards sooner than they should for long-term learning include N. Kornell & R. A. Bjork, Optimizing self-regulated study: The benefits—and costs—of dropping flashcards, Memory 16 (2008), 125–136, and J. D. Karpicke, Metacognitive control and strategy selection: Deciding to practice retrieval during learning, Journal of Experimental Psychology: General 138 (2009), 469–486.
20. Eric Mazur has published Peer Instruction: A User’s Manual, about his approach to teaching. (Upper Saddle River, NJ: Prentice-Hall, 1997). In addition, he exemplifies his approach in an engaging YouTube lecture, “Confessions of a converted lecturer,” described in Note 15. Again, it is http://www.youtube.com/watch?v=WwslBPj8GgI, accessed October 23, 2013.
21. The Dunning quote comes from E. Morris, “The anosognosic’s dilemma: Something’s wrong but you’ll never know what it is” (pt. 5), New York Times, June 24, 2010.
22. Peter Brown interview of Catherine Johnson, December 13, 2011, Minneapolis, MN.
23. Much of this chapter is about how to regulate one’s learning while avoiding various illusions and biases based on fluency, hindsight bias, and the like. An excellent recent article on self-regulated learning that would prove useful to anyone seeking more knowledge on these topics is R. A. Bjork, J. Dunlosky, & N. Kornell, Self-regulated learning: Beliefs, techniques, and illusions, Annual Review of Psychology 64 (2013), 417–444.
1. Francis Bacon (1561–1626) was an English philosopher and statesman. The full quote is “All rising a to great place is by a winding stair; and if there be factions, it is good to side a man’s self, whilst he is in the rising, and to balance himself when he is placed.” From Bacon’s essay Of Great Place.
2. Peter Brown interview of Bruce Hendry, August 27, 2012, St. Paul, MN. All quotes of Hendry are from this interview.
3. Betsy Morris, Lisa Munoz, and Patricia Neering, “Overcoming dyslexia,” Fortune, May, 2002, 54–70.
4. Annie Murphy Paul, “The upside of dyslexia,” New York Times, February 4, 2012. The work by Geiger and Lettvin is described in G. Geiger & J. Y. Lettvin, Developmental dyslexia: A different perceptual strategy and how to learn a new strategy for reading, Saggi: Child Development and Disabilities 26 (2000), 73–89.
5. Survey is listed in F. Coffield, D. Moseley, E. Hall, Learning styles and pedagogy in post-16 learning, a systematic and critical review, 2004, Learning and Skills Research Centre, London; the quote by the student (“there’s no point in me reading a book”) is from same source, p. 137. The quote “a bedlam of contradictory claims” is from Michael Reynolds, Learning styles: a critique, Management Learning, June 1997, vol. 28 no. 2, p. 116.
6. The material about learning styles is drawn largely from H. Pashler, M. A. McDaniel, D. Rohrer, & R. A. Bjork, Learning styles: A critical review of concepts and evidence, Psychological Science in the Public Interest 9 (2009), 105–119. This article reviewed the published evidence bearing on whether learning is improved when the instructional method is matched to students’ learning styles relative to when the instructional method is not matched. Two important findings were that (1) there are very few studies that adopted the gold standard of performing controlled experiments, and (2) the few published experiments consistently found that matching instruction to learning style did not improve learning. One key conclusion is that more experimental research on this issue is needed, but at the moment there is little evidence for the existence of commonly postulated learning styles.
7. An excellent text on classic views of intelligence is Earl Hunt, Human intelligence (Cambridge: Cambridge University Press, 2010).
8. Howard Gardner’s theory is described in his book Multiple Intelligences: New Horizons (New York: Basic Books, 2006), among other venues.
9. The material on work by Robert Sternberg, Elena Grigorenko, and their colleagues comes from several sources. For a nice presentation of the theory, see R. J. Sternberg, Grigorenko, E. L., & Zhang, L., Styles of learning and thinking in instruction and assessment, Perspectives on Psychological Science (2008) 486–506. Another interesting study by Sternberg, Grigorenko and colleagues identified college students who showed much higher skill in either analytical, creative, or practical ability (relative to the other two abilities), and assigned them to different classes that focused on analytic instruction, creative instruction, or practical instruction. Students receiving instruction that matched their strongest ability tended to perform better on certain class-performance assessments than students who received mismatched instruction; see R. J. Sternberg, E. L. Grigorenko, M. Ferrari, & P. Clinkenbeard, A triarchic analysis of an aptitude–treatment interaction, European Journal of Psychological Assessment 15 (1999), 1–11.
10. The study of Brazilian children was T. N. Carraher, D. W. Carraher, & A. D. Schliemann, Mathematics in the streets and in the schools, British Journal of Developmental Psychology 3 (1985), 21–29. This fascinating study focused on five children from very poor backgrounds who were working on street corners or markets in Brazil. Performance was compared for similar multiplication problems presented in different contexts: the natural context in which the child was expert (e.g., selling coconuts, but role-played in the experiment), word problems phrased within a different context (e.g., selling bananas), or formal math problems without context. The children solved nearly 100 percent of the problems when presented in the natural context, fewer in the different context, and only about a third when presented as a formal problem. A key point is that the children used concrete grouping strategies to solve the natural context problems, but then switched to school-taught strategies (not yet well learned) when presented with the formal problems. The mathematical strategies the children had developed were not evident on an academically oriented test.
11. The study of race handicappers is S. J. Ceci & J. K. Liker, A day at the races: A study of IQ, expertise, and cognitive complexity, Journal of Experimental Psychology: General 115 (1986), 255–266. This study sampled harness racing fans, with some classified as expert and some as less expert. The expert group and less expert group were evenly matched on IQ, yet the expert group showed much better success at predicting outcomes of actual races and experimenter-contrived races. The experts’ success was related to their using an extremely complex system of weighting and combining the range of information related to the horses and the race conditions.
12. Dynamic testing: Robert Sternberg and Elena Grigorenko discuss this concept in Dynamic Testing: The Nature and Measurement of Learning Potential (Cambridge: Cambridge University Press, 2002).
13. The fundamental work on structure building was begun by M. A. Gernsbacher, K. R. Varner, & M. E. Faust, Investigating differences in general comprehension skills, Journal of Experimental Psychology: Learning, Memory, and Cognition 16 (1990), 430–445. This article provides some of the elegant experimental work that contributed to the development of the structure-building theory—the idea that good comprehenders are able to construct a coherent, organized representation of a narrative from many sources (either read, listened to, or seen in pictures), whereas less able comprehenders tend to construct many, somewhat fractionated representations of the narratives. This research further suggested that poor structure-builders, but not good structure-builders, have trouble inhibiting irrelevant information, which likely contributes to their fractionated (ineffective) representations. Another relevant article is A. A. Callender & M. A. McDaniel, The benefits of embedded question adjuncts for low and high structure builders, Journal of Educational Psychology 99 (2007), 339–348. They demonstrated that low structure-builders achieve less learning from standard school materials (textbook chapters) than do high structure-builders. However, embedding questions into chapters to focus the low structure-builders on the important concepts (and requiring them to answer the questions) boosted the low structure-builders to levels of learning enjoyed by high structure-builders.
14. The discussion of learning concepts here relies on two studies: T. Pachur, & H. Olsson, Type of learning task impacts performance and strategy selection in decision making, Cognitive Psychology 65 (2012), 207–240. The typical approach to studying conceptual learning in the laboratory is to provide one example at a time, with learners attempting to learn the likely classification of this example (e.g., given a case with a particular set of symptoms, what is the likely disease?). This experiment modified that procedure by presenting two examples simultaneously (e.g., two cases) and requiring learners to select which of the two would be most likely to reflect a particular classification. This comparative approach stimulated less focus on memorizing the examples and better extraction of the underlying rule by which the examples were classified. A similar theme to the one above, except that the focus was on transfer in problem solving, appears in M. L. Gick & K. J. Holyoak, Schema induction and analogical transfer, Cognitive Psychology 15 (1983), 1–38. Learners either studied one example of how to solve a particular problem or were required to contrast two different kinds of problems to figure out the common elements of their solutions. The learners who contrasted two problems were more likely to extract a general solution scheme and transfer that scheme to successfully solve new problems than were the learners who studied only one problem.
15. The reference on rule learners and example learners is M. A. McDaniel, M. J. Cahill, M. Robbins, & C. Wiener, Individual differences in learning and transfer: Stable tendencies for learning exemplars versus abstracting rules, Journal of Experimental Psychology: General 143 (2014). Using laboratory learning tasks, this novel study revealed that some people tend to learn concepts by focusing on memorizing the particular examples and responses associated with the examples that are used to illustrate the concept (termed exemplar learners), whereas other learners focus on the underlying abstraction reflected in the particular exemplars used to illustrate the concept (termed abstractors). Further, a particular individual’s concept-learning tendency persisted across quite different laboratory concept-learning tasks, suggesting that individuals may have a fairly stable predisposition toward exemplar learning versus abstraction across a range of conceptual-learning tasks. Of interest, an initial result was that the abstractors on average achieved higher grades in an introductory college chemistry course than did the exemplar learners.
1. A good introduction to Walter Mischel’s classic research on delay in gratification in children is W. Mischel, Y. Shoda, & M. L. Rodriguez, Delay of gratification in children, Science 244 (1989), 933–938. For an accessible introduction for nonpsychologists, see Jonah Lehrer, “Don’t! The secret of self-control,” New Yorker, May 18, 2009, 26–32. For a 2011 update, see W. Mischel & O. Ayduk, Willpower in a cognitive-affective processing system: The dynamics of delay of gratification, in K. D. Vohs & R. F. Baumeister (eds.), Handbook of Self-Regulation: Research, Theory, and Applications (2nd ed., pp. 83–105) (New York: Guilford, 2011).
2. Accounts of Carson are reprinted at the website maintained by historian Bob Graham, whose antecedents were among the original American settlers in California, www.longcamp.com/kit_bio.html, accessed October 30, 2013, and are drawn from material published originally in the Washington Union in the summer of 1847 and reprinted in Supplement to the Connecticut Courant, July 3, 1847. Hampton Sides, Blood and Thunder (New York: Anchor Books, 2006), 125–126, relates Fremont’s directing Carson on this journey.
3. Research on brain plasticity: J. T. Bruer, Neural connections: Some you use, some you lose, Phi Delta Kappan 81, 4 (1999), 264–277. The Goldman-Rakic quote comes from Bruer’s article, which quotes from remarks she made before the Education Commission of the States. Further research on brain plasticity, with an emphasis on treatment of brain damage, may be found in D. G. Stein & S. W. Hoffman, Concepts of CNS plasticity in the context of brain damage and repair, Journal of Head Trauma Rehabilitation 18 (2003), 317–341.
4. H. T. Chugani, M. E. Phelps, & J. C. Mazziotta, Positron emission tomography study of human brain function development, Annals of Neurology 22 (1987), 487–497.
5. J. Cromby, T. Newton, and S. J. Williams, Neuroscience and subjectivity, Subjectivity 4 (2011), 215–226.
6. An accessible introduction to this work is Sandra Blakeslee, “New tools to help patients reclaim damaged senses,” New York Times, November 23, 2004.
7. P. Bach-y-Rita, Tactile sensory substitution studies, Annals of the New York Academy of Sciences 1013 (2004), 83–91.
8. For work on myelination, see R. D. Fields, White matter matters, Scientific American 298 (2008), 42–49, and R. D. Fields, Myelination: An overlooked mechanism of synaptic plasticity?, Neuroscientist 11 (December 2005), 528–531. For a more popular exposition, see Daniel Coyle, The Talent Code (New York: Bantam, 2009).
9. Some references on neurogenesis: P. S. Eriksson, E. Perfilieva, T. Björk-Eriksson, A. M. Alborn, C. Nordborg, D. A. Peterson, & F. H. Gage, Neurogenesis in the adult human hippocampus, Nature Medicine 4 (1998), 1313–1317; P. Taupin, Adult neurogenesis and neuroplasticity, Restorative Neurology and Neuroscience 24 (2006), 9–15.
10. The quote comes from Ann B. Barnet & Richard J. Barnet, The Youngest Minds: Parenting and Genes in the Development of Intellect and Emotion (New York: Simon and Schuster, 1998), 10.
11. The Flynn effect is named for James Flynn, who first reported on the trend for increased IQs in the twentieth century in developed nations in J. R. Flynn, Massive IQ gains in 14 nations: What IQ tests really measure, Psychological Bulletin 101 (1987), 171–191.
12. This section draws heavily on Richard E. Nisbett, Intelligence and How to Get It (New York: Norton, 2009.)
13. The study cited is J. Protzko, J. Aronson, & C. Blair, How to make a young child smarter: Evidence from the database of raising intelligence, Perspectives in Psychological Science 8 (2013), 25–40.
14. The cited study is S. M. Jaeggi, M. Buschkuehl, J. Jonides, & W. J. Perrig, Improving fluid intelligence with training on working memory, Proceedings of the National Academy of Sciences 105 (2008), 6829–6833.
15. The failure to replicate the working memory training result appears in T. S. Redick, Z. Shipstead, T. L. Harrison, K. L. Hicks, D. E. Fried, D. Z. Hambrick, M. J. Kane, & R. W. Engle, No evidence of intelligence improvement after working memory training: A randomized, placebo-controlled study, Journal of Experimental Psychology: General 142, 2013), 359–379.
16. Carol Dweck’s research on growth mindsets is summarized in many places. See a nice summary by Marina Krakovsky, “The effort effect,” Stanford Magazine, March/April 2007. For two articles by Dweck, see H. Grant & C. S. Dweck, Clarifying achievement goals and their impact, Journal of Personality and Social Psychology 85 (2003), 541–553, and C. S. Dweck, The perils and promise of praise, Educational Leadership 65 (2007), 34–39. She also has a book, Mindset: The New Psychology of Success (New York: Ballantine Books, 2006).
17. Dweck quote is from Krakovsky, “Effort effect.”
18. The Dweck quotes are from Po Bronson, “How not to talk to your kids,” New York Times Magazine, February 11, 2007.
19. Paul Tough, How Children Succeed (New York: Houghton Mifflin Harcourt, 2012).
20. Anders Ericsson’s work on deliberate practice has been described in many places, including Malcolm Gladwell, Outliers: The Story of Success (New York: Little, Brown, 2008). For accessible introductions to the work by Ericsson, see K. A. Ericsson & P. Ward, Capturing the naturally occurring superior performance of experts in the laboratory: Toward a science of expert and exceptional performance, Current Directions in Psychological Science 16 (2007), 346–350.
21. Mental imagery and its power as an aid to learning and memory has been appreciated since the time of the ancient Greeks. However, psychologists only began studying the topic in experimental studies in the 1960s. Allan Paivio’s research showed the power of imagery in controlled studies. A summary of his early research appears in A. Paivio, Imagery and Verbal Processes (New York: Holt, Rinehart, and Winston, 1971).
22. Mark Twain, “How to Make History Dates Stick,” Harper’s, December 1914, available at www.twainquotes.com/HistoryDates/HistoryDates.html, accessed October 30, 2013.
23. In the history of mnemonic devices (and psychologists’ and educators’ attitudes toward them), they have suffered various reversals of fortune over the centuries. They were valued from Greek and Roman times and throughout the Middle Ages by educated people who needed to remember large amounts of information (e.g., to make a two-hour speech in the Roman Senate). In recent years, educators have dismissed them as useful merely for rote learning. However, as we show in this chapter, this charge is not fair. Mnemonics, as used by James Paterson and his students, can serve (as they did for the ancient Greeks and Romans) as organizing systems for retrieving information. To put it simply, mnemonic devices are not necessarily good for comprehending complex information, but using a mnemonic system to help to retrieve learned information can be invaluable. James Worthy and Reed Hunt provide an excellent introduction to the history of and psychological research on mnemonic devices in their book Mnemonology: Mnemonics for the 21st Century (New York: Psychology Press, 2011).
24. James Paterson is a “memory athlete,” partaking in a growing sport in Europe, China, and to some extent the United States. Joshua Foer wrote about this emerging subculture in his bestselling book Moonwalking with Einstein: The Art and Science of Remembering Everything (New York: Penguin, 2011). How long might it take a person to remember a shuffled deck of cards in order? For you, a long time. For a memory athlete in the top rungs, under two minutes. A video of Simon Reinhard memorizing a deck of cards in 21.9 seconds is available at www.youtube.com/watch?v=sbinQ6GdOVk, accessed October 30, 2013. This was a world record at the time, but Reinhard has since broken it (21.1 seconds is the record as of this writing). Reinhard has broken twenty seconds in practice sessions but not yet in a timed public event (Simon Reinhard, personal communication in the form of a conversation over dinner in St. Louis, MO, on May 8, 2013, with Roddy Roediger and several other people).
25. Michela Seong-Hyun Kim’s description of her experience using mnemonics was relayed to Peter Brown by James Paterson in private correspondence, February 8, 2013.
26. Peter Brown and Roddy Roediger interview of James Paterson, January 4, 2013, St. Louis, MO.
27. Peter Brown interview of Karen Kim, April 18, 2013, St. Paul, MN.
1. Peter Brown telephone interview of Michael Young, May 21, 2013. All quotes of Young are from this interview.
2. Peter Brown telephone interview of Stephen Madigan, May 20, 2013.
3. Peter Brown interview of Nathaniel Fuller, April 29, 2013, Minneapolis, MN.
4. John McPhee, “Draft no. 4,” New Yorker, April 29, 2013, 32–38.
5. Peter Brown interview of Thelma Hunter, April 30, 2013, St. Paul, MN.
6. Peter Brown interview of Mary Pat Wenderoth, May 7, 2013, Seattle, WA.
7. The empirical studies aimed at testing the effects of high-structure classes in reducing student attrition in gateway science classes are S. Freeman, D. Haak, & M. P. Wenderoth, Increased course structure improves performance in introductory biology, CBE Life Sciences Education 10 (Summer 2011), 175–186; also S. Freeman, E. O’Connor, J. W. Parks, D. H. Cunningham, D. Haak, C. Dirks, & M. P. Wenderoth, Prescribed active learning increases performance in introductory biology, CBE Life Sciences Education 6 (Summer 2007), 132–139.
8. Peter Brown telephone interview of Michael Matthews, May 2, 2013.
9. Peter Brown telephone interview of Kiley Hunkler, May 21, 2013.
10. Peter Brown interview of Kathleen McDermott, June 20, 2013, Folly Beach, SC.
11. Peter Brown telephone interview of Kathy Maixner, July 18, 2013.
12. Peter Brown telephone interview of Kenneth Barber, July 1, 2013.
13. Peter Brown telephone interview of Richard Wynveen, July 17, 2013.
14. Peter Brown telephone interview of Erik Isaacman, June 2, 2013.