Repackaging the Introductory IS Survey Course: An Information Technology Solution Sharen Bakke missives@sharenbakke.com Robert Faley rfaley@kent.edu Management and Information Systems Department, Kent State University Kent, OH 44240 USA Geoff Steinberg gsteinb@kent.edu Management and Information Systems Department, Kent State University Kent, OH 44240 USA Abstract Both instructors and students dread large introductory courses. IS introductory courses are no exception as they are often too large, the material too dry and the atmosphere too impersonal. This presentation describes a unique curriculum for delivering introductory IS survey courses that keep students interested and engaged while producing high-quality learning outcomes. In designing this curriculum three objectives were met: 1) the classroom experience is enjoyable; 2) the students have greater control over the learning process; and 3) most of the administration of the class as well as the grading and testing is accomplished electronically. Quiz scores for students after the implementation of the new curriculum were found to be significantly higher than quiz scores obtained before the new curriculum was adopted. Keywords: large introductory classes, pedagogical issues, curriculum design, active learning, learning environment 1. INTRODUCTION Organizations have readily used information systems to improve the efficiency and effectiveness of their business processes, provide information for more effective decision-making, and enable workgroup collaboration among employees and partners throughout the world (O'Brien and Marakas, 2006). Because they play a meaningful role in improving organizational productivity and profitability, the study of information systems has become an essential discipline in undergraduate and graduate business programs. In most undergraduate business programs students are first exposed to information systems (IS) using an introductory survey course. The typical curriculum covers a vast array of subjects including data and information characteristics, operating systems, information system development, database theory and design, hardware and software concepts, the use of information systems in decision-making and information systems in e-commerce and Internet security. While the objectives of this approach are laudable, they are often not realized: introductory classes are usually too large, focus too broadly on most topics, are populated with students with a variety of backgrounds and interests, and are often perceived as impersonal. Students often enroll in these courses primarily to satisfy graduation requirements rather than to satisfy an inherent interest in the subject. Hence, neither students nor professors are satisfied with the learning experience and, more importantly, graduates from business schools do not master fundamental IS skills before entering the professional world. This is not a desirable situation for students, the institutions granting their diplomas, or their potential employers. We discuss a unique curriculum for delivering introductory IS survey courses that keep students interested and engaged while producing high-quality learning outcomes. In designing this curriculum three objectives were met: 1) the classroom experience is enjoyable; 2) the students have greater control over the learning process; and 3) most of the administration of the class as well as the grading and testing is accomplished electronically. The remainder of this presentation contains three sections. The first section examines the state of the art of the current pedagogical literature on curriculum design, the second section describes how these design considerations were met by the IS curriculum described above and the third section provides empirical results demonstrating the effectiveness of the curriculum. 2. PEDAGOGICAL CURRICULUM DESIGN CONSIDERATIONS Factors to consider when designing a curriculum include creating a positive learning environment, incorporating active learning techniques, giving students a choice, providing practice opportunities and administering formative assessments. Positive Learning Environment According to Dewey (1916/1966), ”We never educate directly, but indirectly by means of the environment “(p. 19). The learning environment has been described by Hiemestra (1991) as all the physical surroundings, psychosocial or emotion conditions and social or cultural influences present in a learning situation. Fulton (1991) found that both the physical and social aspects of a learning environment influence student participation and satisfaction. Entwistle (2005) concluded that student learning occurs within a holistic teaching-learning environment that consists of various types of teaching, e-learning, assessment criteria and procedure, assignments, feedback and workload. The quality of the learning that students achieve is affected by the interaction of the various components. He further notes students’ perceptions of the teaching and assessment procedures affect their learning more than the actual environment. This perception is derived from accessibility and thoroughness of explanations, relevancy of material, enthusiasm shown for the subject, empathy shown for students’ difficulties and the quality of support provided for these difficulties (Checkland and Scholes, 1999). Zimbardo (2005) offers four basic premises that professors can follow to create an optimal learning environment: (1) make it memorable, (2) right, (3) relevant, and (4) better next time. This is supported by Auster and Wylie’s (2006) four dimensions of the teaching process: context setting, class preparation, class delivery, and continuous improvement. Active Learning Techniques Active learning theory suggests that students become an integral part of the learning process by studying ideas, solving problems and applying what they learn. Active learning helps students to hear, see, ask questions about issues and problems and have the opportunity to discuss them with others (Bean, 1996; Bonwell and Eison, 1991; Silberman, 1996). There is consensus among pedagogical researchers that active learning techniques may have a positive impact upon students' learning (Astin, 1984; Bonwell and Eison, 1991; Colleges, 1986; House, 2002; Kvam, 2002; McClanahan and McClanahan, 2002; Miller, 1988; Udovic, Morris, Dickman, Postlethwait and Wetherwax, 2002). Researchers have found active learning is related to higher levels of confidence in their discipline based knowledge (Anderman and Young, 1994; House, 2002); learning involvement and motivation (House, 2002; Udovic, Morris, Dickman, Postlethwait and Wetherwax, 2002); problem solving abilities (Zoller, 1987); developing independent learning skills and the ability to apply knowledge (Sivan, Leung, Woon and Kember, 2000); and increasing retention for students whose performance is average or below average (Kvam, 2002). Udovic et al. (2002) found that students involved in this approach developed a deeper conceptual understanding of the material, logical reasoning, and a greater appreciation for the discipline than did students in a comparison section of the course taught with passive methods. Educational games have been identified as an effective active learning technique. They are characterized as being enjoyable (Lawson, 1995), motivating (Watson, Kessler, Kalla, Kam and Ueki, 1996), help students increase their confidence with class material (Townsend, Moore, Tuck and Wilton, 1998) and improve their higher order thinking and reasoning skills (Hogle, 1996). Games that are considered fun are those that are intrinsically motivating and offer the right amount of challenge (Allen, 2003; Lepper and Malone, 1987; Malone, 1980; Malone and Lepper, 1987; Malouf, 1988). Giving Students a Choice Zimbardo (1969) and other researchers (Lepper and Malone, 1987; Liao and Tai, 2006; Malone, 1980; Malone and Lepper, 1987) have shown giving people a choice, or even a perception of choice can increase their motivation to do a task. This applies equally to the learning process. Students can be more motivated to complete homework and practice exercises when given a choice to regulate their learning process (Csikszentmihaly, 2000; Zimbardo, 2005). Practice Opportunities Lepper and Malone’s research (1987) suggests that students spend more time on activities that engage their interest. This leads to better learning of the course material and more sustained interest in future encounters with the subject matter. The longer students spend on particular tasks and the increased interest leads to more practice, more automaticity of pattern recognition, more efficient retrieval of concepts, and better use of basic knowledge (Trabasso, 1987). Kritch and Bostow (1998) found that learning was strengthened in those students who practiced with more questions thus supporting the argument that frequent questions is a critical design feature for effective computer-based instruction. Requiring more frequent responses takes students more time but appears to improve performance: the process of responding is more critical than the time spent on the activity. Results in their study highlight the importance of instructional techniques that require learners to repeatedly and overtly practice the desired behavior. Answering more questions increased not only the performance on quizzes but also the extent to which students could subsequently accomplish a relevant applied skill. This supports the notion that increased interactivity produces increased learning (Fletcher, 1990; Schaffer and Hannafin, 1986). Students’ interest tends to be more stimulated when practice evokes misconceptions about newly learned information (Smith and Ragan, 1993). Hence, designers should consider ways in which learners might misunderstand lesson content, then design practice experiences that allow learners to discover misconceptions and correct them. Administering Formative Assessment Formative assessment is used to provide feedback to students rather than to evaluate them for course grades. It allows students to assess their own progress and understanding of the course material (Brown and Knight, 1994; Seale, Chapman and Davey, 2000; William and Black, 1996) and to evaluate their progress against an established goal (Smith and Ragan, 1993; Wiliam, 2006). Having received feedback on their performance, students may then take steps indicated by that feedback to remedy whatever weaknesses the assessment has exposed. The function of formative assessment is to assist learners in closing the gap between actual and desired levels of performance (William and Black, 1996). Bransford et al. (2000) noted that formative assessment designs should be sufficiently interesting to capture student attention and motivate them to improve their performance. Bostow et al. (1995) found that technology can provide the meaningful interaction between student and instructional materials - an essential component of successful pedagogy. Typical assessment tools include those activities that help students learn such as short multiple-choice, true or false or fill in the blank tests and quizzes, and questions and answers within the lesson. Formative assessment must be provided at an appropriate point in the learning process and must contain some prescription for action concerning what must be done for students to improve their performance (Brown and Knight, 1994; William and Black, 1996). Students need both concrete steps for how to address their weaknesses and sufficient time to act upon the feedback. Formative assessment administered through technology enables students to tailor their use of the assessments to their own learning style. Online technology provides students with several advantages such as having the freedom to access the learning material at times and places of their own choosing (Bostow, Kritch and Tomkins, 1995), repeatedly taking the same test to assess the improvement in performance after study, working at their own pace, providing individualized feedback in a flexible and cost-effective manner and reducing student anxiety before summative tests (Zakrzewski and Bull, 1999). Clariana (1997) has shown that some individuals learn faster than others from computer-based materials, implying that people differ in the amount of computer time they require. Buchanan (2000) showed that a web-based formative assessment strategy improves student learning interest and student scores. The more diverse formative assessment strategies are embedded in the online learning environment, the greater the learning effect obtained by the students (Darling-Hammond, 2000). Buchanan (2000) found students who are exposed to a balanced curriculum, consisting of the appropriate selection of teaching, learning and assessment methods are likely to benefit from the experience. 3. ADDRESSING DESIGN CONSIDERATIONS IN THE NEW CURRICULUM Enjoyable Classroom Experience This introductory class averages between 400 and 450 students per semester and meets for 1¼ hours twice a week in a large auditorium. The atmosphere in the auditorium is controlled by the professor. The lights are dimmed while soft music is played and pictures of various pieces of art are displayed on the main viewing screen. The dimming of the lights and the rate at which the artwork is displayed has an obvious calming effect. The music and artwork is changed periodically to maintain interest. This mood is maintained until the class begins. Developing a rapport with approximately 400 students is a much more difficult feat. Typically, open-ended questions are bantered around between the students and the professor before class officially starts. The questions are very topical and often lead to a stimulating dialogue. Anyone can participate as long as their remarks and behavior are not offensive. Topics such as upcoming quizzes, and tests or group performances on previous exercises and exams are discussed. Individual questions are reserved for after class. One of the primary goals of the early interaction between professor and students is to establish a trusting, safe, and supportive environment that facilitates learning. While delivering the course material on PowerPoint slides the professor makes use of active learning techniques such as guided lectures, discussions and educational games. Game show activities pop up on the screen at random times accompanied by specific theme music and colorful graphics. Randomly selected contestants (students alone or in teams) come to the front of the auditorium where they are asked questions pulled from a test bank containing all the course-related material presented to that date. The goal of the game show is to acquire tokens (see Figure 1) that can be redeemed to purchase “gifts” from the Online Gift Catalog located in ORION, the online classroom management application. Figure 1: Token distributed in class. The value of the token is circled (.1). Three types of games are available. Game 1 requires one student to answer one question correctly within ten seconds. The second game involves two teams with two students on each team. Each team is required to answer three questions correctly within 15 seconds. The team with the most correct answers wins the tokens. The third game consists of one team of four students with the active assistance of the entire class. The intent of the third game is to foster camaraderie among all the students by encouraging them to actively support the contestants who can earn the rest of the class tokens also. The team is given 30 seconds to answer ten questions correctly. If the team answers five or more correctly then all students in attendance that day are given a token. Regulating the Learning Process Students have significant control over when they want to complete homework assignments and quizzes and how often they want to submit homework assignments. They can collect extra-credit points by selecting the Early Bird Special and by answering pop questions correctly. Students can also extend homework and quiz deadlines, expunge grades and submit additional homework assignments by using their tokens to purchase these gift catalog options. The course contains twelve regular homework assignments. These assignments are application-oriented and cover topics such as working with an Excel spreadsheet (e. g. importing data, manipulating cells using a macro, and creating charts), creating a simple web page, and querying information from a database. ORION grades these assignments automatically; sufficient hints are provided for students’ to correct their mistakes and resubmit their answers. Each assignment must be completed by the due date and may be submitted three times before the due date. The grade recorded is the value earned on the last attempt. There are five objective quizzes in formats such as multiple-choice, true or false or fill in the blank. Each quiz is administered in the College of Business lab during a specified “quiz period” consisting of 5 weekdays only. Each quiz contains material covered in class during the three weeks prior to the beginning of the quiz period. Quizzes are open notes and students are permitted to use other software while taking a quiz. The quiz can be taken only once. To enhance the learning process, specific practice exercises must be successfully completed before quizzes can be taken. These practice exercises, referred to as prerequisites, can be attempted as often as necessary to attain the required grade of 100%. Students keep track of their prerequisite practice exercises (see Figure 2) through ORION. Figure 2: This screen displays the status of students’ prerequisite exercises Earning Extra-Credit Points Students can make use of an Early-Bird Special option to gain extra percentage points for turning in homework assignments or for attempting quizzes ahead of the final quiz due date. The professor determines the number of percentage points awarded, the minimum grade that must be attained to earn the points and the number of days ahead for different award levels. The Early Bird Special set up for Quiz 2 has three award levels. If the student attempts the quiz on the first open day and attains a score greater than 60% the final score is increased by four percentage points. Students attempting the quiz on the second day, earning a score greater than 70% earn two percentage points and those students attempting the quiz on the third day in the quiz period, earning more than 80% increase their score by 1 percentage point. In Figure 3 the student attempted Quiz 2 on the first day, scored more than the requisite 60% and was awarded four additional percentage points. Figure 3: This student earned 4 additional percentage points for Quiz 2: the score was greater than 60% and the quiz was attempted on the first day. Whenever students are logged into ORION they will be given the opportunity to answer questions about course content. Each correct answer will result in an award of 0.10 points. During the semester they will receive up to 45 extra credit pop questions for a possible 4.5 points. These points can be redeemed in the Gift Catalog. Purchasing Gifts from the Gift Catalog Tokens earned in class can be redeemed in ORION’s Gift Catalog for quiz and homework privileges. Available gifts include due-date extensions for quizzes or homework, permission to retake a quiz and permission to submit an additional homework assignment one additional time. The gifts and the number of tokens necessary for each gift are shown in Figure 4. Figure 4: Gifts available in the gift catalog The Online Gift Catalog is accessed through ORION. After logging in, students select the Gifts/Token option from the Home page and use one of the options in Figure 5 to manage (e.g. redeem tokens, view recorded tokens, status, purchases etc.) their tokens. Figure 5: This screen allows students to redeem their tokens and manage their activity A Rewind option allows students to turn in overdue assignments. Students have the opportunity to submit late assignments but are assessed a late fee. Electronic Administration of Class ORION allows the bulk of this class to be administrated electronically. The ORION application is written in ASP.NET and housed on a Windows 2003 Server containing a SQL 2000 database. The ORION system has a student and an administrative “view”. The student view enables students to access the syllabus, complete the homework and quizzes, view their status in the course and verify they have all the prerequisites, access the study and practice exercises, manage their tokens and access the gift catalog, and contact the professor as well as communicate with other students. The administrative view enables the professor to create and manage the entire class. Four major sections (shown in Figure 6) are addressed in the administrative view: 1) the calendar, 2) the students, 3) communication, and 4) grades. The calendar section allows the professor to manage the test bank, specify due dates, establish the grading rules and determine the number of points for homework and quizzes. Details associated with the Early Bird Specials such as the number of percentage points received, the minimum grade that must be earned to attain the points and the number of days before the deadline are specified here. Included in this section is the Extra Credit Management option where test bank questions, game shows and tokens are managed. Reports can be generated that analyze the student’s performance on the questions and indicate the students’ purchasing behavior for gifts, extra credit points and penalties. The number of tokens redeemed and by whom is accessible as well. The game show setup allows the professor to specify the type of games to be played, the number of questions, the amount of time for each game and the music associated with each game. Lastly, the prerequisite exercises for each quiz are specified in this section. Figure 6: Administrators manage the calendar, student information, communication and grades from this screen. The Student section allows the professor to view the homework, prerequisite and quiz status report for one or many students and obtain partial or full portions of the roster. The Communication section is used to publish the syllabus, send e-mails, post messages, visit the chat room, and post handouts. The Grades section allows the professor to view the grades for every assignment for one or all students. Grades can be exported for submission to another information system from this section. The ability to generate and process tokens distributed in the classroom makes the ORION system very different from other classroom management applications. Once the tokens are created from the Administrative side they can be redeemed from the Student side and become part of the student’s Extra Credit history record. This provides ample opportunity for the professor to analyze the effectiveness of the tokens, the games, and classroom attendance; the desired information can be obtained through simple database queries. An unstated but necessary component of any teaching environment is personal access to the professor, who is available during specified office hours. The office is simply furnished to accommodate as many students as possible and to encourage student interaction. Students come for individual help but end up talking with each other and answering their own questions. The professor facilitates rather than controls the learning exchange during office hours. The professor also maintains a constant presence on Instant Messenger and quickly responds to all e-mails. Although the class is administered electronically, students have ample opportunity for personal interaction if they wish. 4. EMPIRICAL RESULTS SHOWING EFFECTIVENESS OF THE NEW CURRICULUM The effectiveness of the new curriculum was measured by comparing quiz scores obtained before and after the new curriculum was implemented. Student data was collected from large introductory IS courses in a small mid-western public university for five semesters before the implementation of the IS curriculum and for five semesters after. The majority of the students enrolled in these classes were freshmen. Quiz scores from the fifth quiz for ten semesters were selected for comparison to ensure the students received the full benefit of the new curriculum. Scores from the first quiz were subtracted from the fifth quiz scores to account for individual student differences. An independent-sample t-test was conducted to evaluate the following research hypothesis (in alternative form): Ha: Quiz scores for students after the implementation of the new curriculum are higher than quiz scores obtained before the new curriculum was adopted. The test was significant t(3317) = -8.378. p = 0.00) confirming the research hypothesis: quiz score differences between quiz one and quiz five after the curriculum was implemented (M= -3.8532, SD = 13.9) on the average are higher than quiz score differences between quiz one and quiz five before the curriculum was implemented (M= 0.3422, SD = 14.88). 5. CONCLUSION This curriculum meets the three goals established at the beginning of this presentation. First, positive changes in student evaluations since the reformulation of the course substantiate that the class is fun and entertaining compared to the earlier traditional approach. Second, the students successfully self-regulate the learning process: they choose when to submit their homework and take their exams as well as the location from which they submit their assignments. Student comments such as “ORION lets me do things when I want to – I like that” and “I like the feedback ORION gives me – it helps me get my work done” strongly suggest that students prefer to have greater control over the learning experience. Third, the class is almost completely administered electronically through ORION, the online classroom management application. This frees up time for the professor to design and test new ideas that might further enhance both student enjoyment and performance. The significant difference in test scores suggests the new IS curriculum is more effective for student learning than the previous curriculum. The feature that appears to have the greatest impact on increasing test scores was the prerequisite exercises that must be completed successfully before quizzes can be attempted. Future research would confirm the belief that requiring students to score 100% on prerequisite exercises forces them to learn the material thoroughly before continuing. Other topics to explore in the future include differences in quiz scores based on gender and ethnicity. 6. REFERENCES Allen, M. W. (2003): Michael Allen's Guide to E-Learning. Hoboken, N. J.: John Wiley & Sons, Inc. Anderman, E. M., and A. J. Young (1994): "Motivation and Strategy Use in Science," Journal of Research in Science Teaching, 31, 811 - 831. Astin, A. W. (1984): "Student Involvement: A Developmental Theory for Higher Education," Journal of College Student Personnel, 22, 297 - 308. Auster, E. R., and K. K. Wylie (2006): "Creating Active Learning in the Classroom: A Systematic Approach," Journal of Management Education, 30, 333 - 353. Bean, J. C. (1996): The Professors' Guide to Integrating Writing, Critical Thinking, and Active Learning in the Classroom. San Francisco: Jossey-Bass. Bonwell, C. C., and J. A. Eison (1991): Active Learning: Creating Excitement in the Classroom. Washington, D.C.: George Washington University, School of Education and Human Development. Bostow, D. E., L. M. Kritch, and B. F. Tomkins (1995): "Computers and Pedagogy: Replacing Telling with Interactive Computer-Programmed Instruction," Behavior Research Methods, Instruments & Computers, 27, 297 - 300. Bransford, J., A. Brown, and C. R. (2000): How People Learn: Mind, Brain, Experience and School, Expanded Edition. Washington, DC: National Academy Press. Brown, S. A., and P. Knight (1994): Assessing Learners in Higher Education. London: Kogan Page. Buchanan, T. (2000): "The Efficacy of a World-Wide Web Mediated Formative Assessment," Journal of Computer Assisted Learning, 16, 193 - 200. Checkland, P. B., and J. Scholes (1999): Soft Systems Methodology in Action. London: Wiley. Clariana, R. (1997): "Pace in Mastery-Based Computer-Assisted Learning," British Journal of Educational Technology, 28, 135 - 137. Colleges, A. o. A. (1986): "A New Vitality in General Education," Task Group on General Education. Csikszentmihaly, M. (2000): Beyond Boredom and Anxiety, 25th Anniversary Edition. San Francisco: Jossey-Bass Publishers. Darling-Hammond, L. (2000): "Teacher Quality and Student Achievement: A Review of State Policy and Evidence," Education Policy Analysis Archives, 8. Dewey, F. (1916/1966): Democracy and Education. New York: The Free Press. Entwistle, H. (2005): "Enhancing Teaching-Learning Environments in Undergraduate Courses in Electronic Engineering: An Introduction to the Etl Project," International Journal of Electrical Engineering Education, 42, 1 - 7. Fletcher, J. D. (1990): Effectiveness and Cost of Interactive Videodisc Instruction in Defense Training and Education. Alexandria, VA: Institute for Defense Analyses. Fulton, R. D. (1991): "A Conceptual Model for Understanding the Physical Attributes of Learning Environments," San Francisco, CA: Jossey-Bass, 13 - 22. Hiemstra, R. (1991): "Aspects of Effective Learning Environments," San Francisco, CA: Jossey-Bass, 5 - 12. Hogle, J. G. (1996): "Considering Games as Cognitive Tools: In Search of Effective "Edutainment."" Educational Resource Information Center, ED 425 737, 1 - 28. House, J. D. (2002): "The Motivational Effects of Specific Teaching Activities and Computer Use for Science Learning: Findings from the National Mathematics and Science Study," International Journal of Instructional Media, 29, 423 - 439. Kritch, K. M., and D. E. Bostow (1998): "Degree of Constructed-Response Interaction in Computer-Based Programmed Instruction," Journal of Applied Behavioral Analysis, 31, 387 - 398. Kvam, P. H. (2002): "The Effect of Active Learning Methods on Student Retention in Engineering Statistics," The American Statistician, 54, 136 - 140. Lawson, T. J. (1995): "Active-Learning Exercises for Consumer Behavior Courses," Teaching of Psychology, 25, 200 - 202. Lepper, M., and T. W. Malone (1987): "Intrinsic Motivation and Instructional Effectiveness in Computer-Based Education," Hillsdale, NJ: Erlbaum, 255 - 286. Liao, W., and W. Tai (2006): "Organizational Justice, Motivation to Learn, and Training Outcomes," Social Behavior and Personality, 34, 545 - 556. Malone, T. W. (1980): "What Makes Things Fun to Learn? A Study of Intrinsically Motivating Computer Games.," Unpublished dissertation, Stanford University. Malone, T. W., and M. Lepper (1987): "Making Learning Fun: A Taxonomy of Intrinsic Motivations for Learning," Hillsdale, N.J.: Earlbaum. Malouf, D. B. (1988): "The Effect of Instructional Computer Games on Continuing Student Motivation," Journal of Special Education, 21, 27 - 38. McClanahan, E. B., and L. L. McClanahan (2002): "Active Learning in Non-Majors Biology Class," College Teaching, 50, 92 - 94. Miller, G. E. (1988): The Meaning of General Education: The Emergence of a Curriculum Paradigm. Columbia University, New York: Teachers College Press. O'Brien, J., and G. M. Marakas (2006): Management Information Systems. New York: McGraw-Hill Irwin. Schaffer, L., and M. J. Hannafin (1986): "The Effects of Progressively Enriched Interaction on Learning from Interactive Video," Educational Communication and Technology Journal, 34, 89 - 96. Seale, J., J. Chapman, and C. Davey (2000): "The Influence of Assessments on Students' Motivation to Learn in a Therapy Degree Course," Medical Education, 34. Silberman, M. (1996): Active Learning: 101 Strategies to Teach Any Subject. Needham Heights, MA: Allyn and Bacon. Sivan, A., R. W. Leung, C. Woon, and D. Kember (2000): "An Implementation of Active Learning and Its Effect on the Quality of Student Learning," College Teaching, 37, 381 - 389. Smith, P. L., and T. J. Ragan (1993): "Designing Instructional Feedback for Different Learning Outcomes," Englewood Hills, NJ: Educational Technology Publications, 75 - 103. Townsend, M. A. R., D. W. Moore, B. F. Tuck, and K. M. Wilton (1998): "Self-Concept and Anxiety in University Students Studying Social Science Statistics within a Co-Operative Learning Structure," Educational Psychology, 18. Trabasso, T. (1987): "Discussion," Hillsdale, NJ: Earlbaum. Udovic, D., D. Morris, A. Dickman, J. Postlethwait, and P. Wetherwax (2002): "Workshop Biology: Demonstrating the Effectiveness of Active Learning in an Introductory Biology Course," Bioscience, 52, 272 - 281. Watson, D. L., D. A. Kessler, S. Kalla, C. M. Kam, and K. Ueki (1996): "Active Learning Exercises Are More Motivating Than Quizzes for Underachieving College Students," Psychological Reports, 78. Wiliam, D. (2006): "Formative Assessment: Getting the Focus Right," Educational Assessment, 11, 283 - 289. William, D., and P. Black (1996): "Meanings and Consequences: A Basis for Distinguishing Formative and Summative Functions of Assessment," British Educational Research Journal, 22, 537 - 548. Zakrzewski, S., and J. Bull (1999): "The Mass Implementation and Evaluation of Computer-Based Assessments," Assessment and Evaluation in Higher Education, 23, 141 - 152. Zimbardo, P. G. (1969): The Cognitive Control of Motivation: The Consequences of Choice and Dissonance. Glenview, Illinois: Scott, Foresman and Company. Zimbardo, P. G. (2005): "Optimizing the Power and Magic of Teaching," Journal of Social and Clinical Psychology, 24, 11 - 21. Zoller, U. (1987): "The Fostering of Question-Asking Capability: A Meaningful Aspect of Problem Solving in Chemistry," Journal of Chemical Education, 64, 510 -512.