Assessment of Student Outcomes in Management Information Systems Online Course Participation Dr. Adnan Omar aomar@suno.edu Mr. Daff Kalulu dkalulu.004@hotmail.com Department of MIS, Southern University New Orleans New Orleans, LA 70126, USA Dr. M. Khurrum S. Bhutta khurrum.bhutta@nicholls.edu Ohio State University Abstract Most universities view e-learning as a step to the future. In past years, universities regarded e-learning as a strategy to increase their student enrollment, retention, and quality while lowering tuition. However, after Hurricane Katrina, several Gulf Coast universities resorted to e-learning as a means of providing basic education to their students. In fact, Southern University of New Orleans (SUNO) has chosen e-learning as an element of its quality enhancement program (QEP), to enhance the quality of education and instruction especially for first year freshmen. However, despite the administration of pre-mastery tests at the beginning of every semester along with an extensive workshop by the e-learning department, students still do not get motivated in their daily performance in a timely fashion. The reason behind this phenomenon is most likely the lack of good oriented learning, a deficiency in mentoring from K-12, and social-economic constraints on the students’ time. As a result, administrators and scholars at SUNO have spent countless hours and resources addressing reasons for this lack of student participation. Pre-tests and post-tests were administrated to measure students’ learning outcomes. Data was collected to analyze the lack of student involvement. The findings of this study will provide faculty members teaching online courses with ways of structuring their online courses. Keywords: Assessment, E-Learning, Process, Participation, Student, Faculty INTRODUCTION The latest educational research indicates that a university can achieve its educational objectives through the use of e-learning as effectively as it does through traditional classroom instruction. According to such research, the subject matter of most university courses can be successfully conveyed to students through the implementation of e-learning tools. Not only can e-learning convey knowledge, but it can also enhance interactivity between student and teacher, which is a hallmark of higher learning. Furthermore, some theorists even claim that e-learning offers advantages over classroom instruction, namely: greater convenience, improved pacing, and higher levels of communication between instructor and learners, instruction and instructors, and student and student (Siemens, 2004; Soloman & Schrum, 2007, and Reynard, 2008). Students and faculty are increasingly turning to online education and the Internet to supplement or even replace traditional approaches to classroom learning and teaching (Alavi and Leidner, 2001; Altbach, Gumport, and Johnstone, 2001; Hanna, Glowacki-Dudka, and Conceicao-Runlee, 2000; Palloff and Pratt, 2001). Advancements in computer and communications technologies, the Internet, and online education are attractive and powerful new tools for teaching and learning. Some scholars even argue that these technologies have the potential to revolutionize higher education with increased access to educational services for students and a wider reach in the educational marketplace for academic institutions (Hollenbeck, Zinkhan, and French, 2005; Medlin, Vannoy, and Dave, 2004). Wireless networks, course management systems, multimedia, and other technologies add new dimensions of richness and complexity to the learning experience. While technology offers a wide range of learning possibilities, it also presents a new set of challenges. To use e-learning effectively, institutions must adapt their pedagogy, enhance the technical proficiency of users, and develop a reliable and robust technology infrastructure (Arabasz and Baker, 2003). Despite the unquestionable benefits of e-learning technologies, the number of special education teachers who are trained to use technology in classrooms remains low. While the availability of computers, internet, and various types of assistive technologies has continued to grow, most school personnel find themselves limited in the understanding of how to operate, utilize, and implement the functions of the available hardware (Birnbaun, 2000). The presence of complex sets of factors makes motivating students a difficult challenge for faculty. Lack of participation on the first day of school is one factor that the university must address. This paper discusses the process of teaching online, including teacher skills in course organization and planning, teaching guidelines as well as new software tools, implementation of new ideas, mentoring relationships, means of student motivation, and measurement of outcome, and focuses on assessment of student performance and course evaluation. LITERATURE REVIEW The rapid growth of distant learning and the increasing pace of technological innovation is a challenge to course designers. With all the research dedicated to e-learning, it is clear that more efforts are needed from both the teacher and the student. However, not much is being discussed about how these kinds of changes should affect the recognition and assessment of academic value of the skills that are being developed in the learning process. Additionally, a review of literature on e-learning and student motivation suggests that self motivation from the student as well as the instructor is critical for success in online classes (Cheng, 2008; Reynard, 2008). The intensive use of technology challenges students’ participation in online classes. Although many students believe that their success in the online orientation proves their online communication skills, some do not have sufficient technology experience to use communication technologies such as accessing course materials on the Blackboard Software, sending and receiving emails, browsing the Internet or searching for information online. Students lacking computer skills hardly concentrate on the learning activities. Instead, they spend their time fearing how they would successfully communicate using a computer (Lee, 2000). Fear, lack of confidence, and low self-esteem usually undermine online students’ participation and performance. Thus, the burden of motivating online students in order to increase their participation and reduce the drop-out rates rests on the shoulders of the instructors. A syllabus or webpage consisting of a detailed course description, prerequisites, learning objectives, work assignments as well as estimated time it will take to complete course work would help students to set aside adequate time for studying, writing and submitting assignments in order to meet expectations (Hofmann, 2003). However, it can be argued that even if a detailed syllabus or website is published, students may be reluctant to fully participate in online classes if they have inadequate computer skills. In this case, instructors should be prepared to spend time during the first week of an online class helping students to access and navigate the Blackboard because it is unlikely that all students will participate successfully during the first week of the semester, and the inability to use the technologies would ensure students’ failure in the long run. As information and communication technology advances, colleges and universities are increasingly offering online classes worldwide. However, this phenomenon is accompanied by a high drop-out of online students compared to the traditional classroom students. A survey conducted on 35 students who had taken online classes showed that 90% confirmed that lack of self discipline and inadequate skill in new technology were the main problems students encounter in online classes. Many students do not set aside specific and adequate time for studying and writing assignments. Without frequent interaction with other online students or an instructor, online students may easily lose their interest and motivation mid or late in the online course of study (Roper, 2007). METHODOLOGY Online learning programs continue to grow in popularity, due mainly to the increasing number of adults who aspire to earn a college degree but are unable to do so because their full-time jobs or for other personal or professional commitments prevent them from attending on-campus, daytime classes. Online courses are fast becoming both economical and practical, because the technological infrastructure needed to address the growing interest in online education is readily available (Totaro, Tanner, Noser, Fitzgerald, & Birch, 2005). Southern University at New Orleans (SUNO) established the department of e-learning in January 2006 and has set policies and procedures concerning faculty support, standards, course approval and coordination, faculty training, course development and ownership, teaching load, enrollment cap, student services, and student tuition and fees. The e-learning department is responsible for meeting standards set forth by the Board of Regents, Southern Region Education Board, and the Western Interstate Commission for Higher Education. With students displaced from New Orleans and scattered across the nation post-Katrina, implementing e-learning on a full scale directly helped SUNO retain and graduate many of its students. Recently, students in California, Georgia, Mississippi, and Texas completed degree work through online curricula, an accomplishment that was impossible pre-Katrina. SAMPLE AND COLLECTION OF DATA SUNO is an open admission institution with a predominantly African American student body, the vast majority of who come from economically-challenged homes in the Greater New Orleans area. SUNO services approximately 2,600 students per semester in all degree areas with approximately 50% employed full-time. Furthermore, the Departments of Criminal Justice, Early Childhood Education, and General Studies currently offer on-line undergraduate degree programs. An on-line Master’s Degree Program in Museum Studies is also available. Table 1 gives a summary of the grade distribution for the past 5 semesters plus the pre-test and post-test statistics as shown in the Appendix. COURSE ASSESSMENT Focusing on the assessment of a course entitled “Personal Productivity”, which is Microsoft Office 2003, the concept can be further distinguished in summative assessment, performed at the end of an asset of learning activities. As an important component of modern teaching and learning processes in face-to-face courses as well as in e-learning environments, assessment provides valuable feedback to teachers and students, which facilitates the revision and adaptation of teaching and learning activities. Furthermore, assessment activities and results can also be utilized for building and strengthening metacognitive skills. Many research studies that focus on learning use students’ test score improvements to measure their learning performance. In this study, students taking “Personal Productivity” in Management Information Systems (MIS) were subjected to a pre-test and a post-test analysis. The Online Course Assessment in Table 2 (Appendix) was used as an instrument for pre-test and post-test analysis and to measure the score difference between these two tests. To delve deeper into the matter, the grades were coded and analyzed using appropriate statistical techniques. Table 3 (Appendix) served as grading scales that were used to formulate the salient statistics. Based on the coding system adopted, some of the salient statistics are given in Table 4 (Appendix). To analyze any statistical difference in the scores between the semesters, a hypothesis test was carried out; H0 = student performance in online classes will be the same across semesters i.e.; µ1=µ2=µ3=µ4=µ5 Where: µ = Mean Grade in semester A single Factor ANOVA was conducted to test the null hypothesis. F-test indicated that there is not enough evidence at the 0.05 confidence level to reject the null hypothesis, i.e., the mean grades across the semesters are equal. NEW APPROACH The implementation of e-learning after Katrina has allowed SUNO to keep its doors open as well as to move forward with its mission of providing higher education to students from diverse backgrounds. Furthermore, e-learning will enable the University both to recover and to play a vital role in preparing individuals to meet the labor needs of the city (Omar, 2008). Despite the above mentioned short term success with e-learning, the university should ensure the ongoing educational improvement process by requiring one hour seminar/workshop prior to enrollment for all students who wish to participate in an e-course. This seminar/workshop should address: networking, managing time, academic skill, study habits, peer group influence, family responsibility, financial problems, support services and extra-curricular activities. The instructor must notify the Recruitment and Retention Office if the student does not participate and/or contact the professor within the first 2 weeks of school. Finally, a book voucher should be issued to students in the form of a debit card in the first week so that they can purchase books based on their allotted financial aid. Educational institutions offering online courses are responsible for providing a quality education. E-learning is having a great impact on higher education. Review modification is planning to implement alternative models of teaching and learning by installing advanced software and hardware and creating multimedia based learning modules in order to enhance e-learning as well as onsite learning outcomes. In the meantime, the e-Learning department at SUNO is providing training sessions to assist its faculty members in using advanced technology. In addition, instructional techniques and strategies for promoting interactivity should be adapted to address students’ varied needs and styles and enhance student success (Omar, Liu & Koong, 2008). Clearly, distance learning, a particularly powerful addition to a growing array of delivery options in higher education, is having a very real impact on higher education and creating alternative models of teaching and learning. As technology continues to change the way that people work and play, it has challenged institutions of higher education to redesign their course delivery methods. E-teaching in the virtual classroom can present pedagogical and technological challenges for faculty members to address students’ learning styles. Research shows that online learning modules that are static provide little interactivity for learners (Cheng, 2008). The content taught in the classroom, the tools used to deliver it and enhance learning, and the ways in which courses are delivered have changed. For example, Camtasia Studio software has allowed instructors to become more involved in “teaching” distance courses (Creighton, Kilcoyne, & McDonald 2008). Software such as Adobe Breeze Presenter with Microsoft PowerPoint software as well as Adobe Captivate 2 empowers faculty to create effective, engaging presentations through voice and animations, delivered on the web (Wyrostek, 2008). Another powerful tool to increase online participation is student-to-student interaction. It has been observed that students who communicate with each other regarding class activities become part of the academic group, which lessens their feeling of isolation. Furthermore, Lee states that when a learning task is accomplished, students who participate in teamwork get higher self esteem than those who work individually. Therefore, it can be argued that online student interaction with each other, minimizes the chance of drop-out, and results increase levels of motivation. In addition, he explains that communication through online threaded discussions enables online students to know each other by recognizing the writing style and expression of thoughts and ideas rather than by physical attributes. As a result, many online students develop meaningful connections with each other which may result in enhanced career networking opportunities in years to come (Lee, 2000 & Roper, 2007). Instructors can motivate online students by awarding points to the processes online students use in order to arrive at the final answer. Such processes include thinking, interaction, collaboration, communication, and application (Reynard, 2008). Instructors should encourage all online students to show innovation and demonstrate critical thinking and application. Online students’ efforts and skills to perform on a higher level than answering multiple choice questions should gain points towards the final course grade. Instructors should reward online students based on each student’s learning process. Instructors offering online courses or face-to-face traditional classes can motivate students’ participation and enhance the learning outcome by supporting and facilitating the learning process in Figure 1 as shown in the Appendix. Figure 1 illustrates future developments for assessing students’ learning processes with the help of an online instructor as a motivator to enhance an outcome. In the “Instructor” column, the instructor enhances online learning by implementing new software in order to redesign the delivery of online courses (1A), by creating effective presentations with voice and animations (1B), and by learning how to use new tools to organize, prepare, teach and monitor the online class (1C). These processes enable the instructor to establish and encourage online students’ learning outcomes through innovation, collaboration and implementation of new ideas. In the “Online Student” column, assessment is based on the student’s demonstration of an innovative method (2A), an illustration of collaborative effort (2B), and the implementation of new ideas (2C). A student who follows these learning processes should be able to write required information and add new information (3A), follow required format and implement new designs (3B), show required learning application and new suggestions (3C), and demonstrate learning ability that is different from other students (3D). In the “Outcome” column, the student benefits from enhanced learning and is graded accordingly. This process should be replicated in such a way that both students and faculty advance their intellectual learning skills. Implementing such a technique should improve the student’s learning outcome. Applying this process should definitely improve the student’s outcome. CONCLUSION It can be argued that without the physical presence of an instructor and face-to-face interaction between student and instructor and student and student, online students may lose interest and motivation. This may be particularly true of students whose motivation and management skills are inadequate, and an instructor’s best efforts to motivate these students may not succeed in an online environment. Thus, as technology advances, it becomes incumbent on the instructor to develop and possess excellent course management skills, such as recording and posting lectures on the board using Interactive Java Applet, so that online students can access lectures and answer questions following the lecture. The knowledge gained from this study provides faculty members with insights to further explore innovative use of advanced technology to address students’ learning styles, preferences, and outcomes. The outcome of this study shows that although there is a very good improvement in pre-test and post-test, there is no significant difference between the semesters. F-test indicated that there is not enough evidence at the 0.05 confidence level to reject the null hypothesis. REFERENCES Alavi, M., and Leudner, D. E. (2001), “Research Commentary: Technology-Mediated Learning—A Call for Greater Depth and Breadth of Research.” Information Systems Research, Vol. 12(1), pp. 1-10. Altbach, P. G., Gumport, P. J., and John stone, D. B. (2001), In Defense of American Higher Education. Baltiore: Johns Hopkins University Press Arabasz, P. and Baker, B. (2003). Evolving Campus Support Models for E-Learning Courses. Educause Center for Applied Research. Retrieved June 3, 2008, from http://net.educause.edu/ir/library/pdf/ecar_so/ers/ers0303/ekf0303.pdf Cheng, J. (2008). Virtual Tutor For The 21st Cyberclassroom. University of Central Oklahoma. The IABPAD Conference Proceedings. Creighton, W., Kilcoyne, M., & McDonald, J. (2008). Integrating Multi-media Technology into Business Courses. Association of Business Information Systems. Federation of Business Disciplines Conference. Houston, TX. Hanna, D. E., Glowacki-Dudka, M., and Conceicao-Runlee, S. (2000), 147 Practical Tips for Teaching Online Groups: Essentials of Web-based Education. Madison, WI: Atwood Publishing. Hofmann, J. (2003). Motivating Online Learners. Learning Circuits. ASTD’s Online Magazine All About E-Learning. Retrieved May 9, 2008, from http://www.learningcircuits.org/2003/aug2003/hofmann.htm Hollenbeck, C. R., Zinkhan, G., and French, W. (2005), “Distance Learning Trends and Benchmarks: Lessons from an Online MBA Program.” Marketing Education Review Vol. 15 (2), pp. 39-52. Lee, C. (2000). Student Motivation in the Online Learning Environment. Journal of Educational Media & Library Sciences, Vol. 37, Iss. 4; p. 367-375. Retrieved May 13, 2008, from http://homepages.findlay.edu/clee/research_interests/motivation_online.htm Medlin, B. D., Vannoy, S. A., and Dave, D. S. (2004), “An Internet-based Approach to the Teaching of Information Technology: A Study of Student Attitudes in the United States.” International Journal of Management, Vol. 21 (4), pp. 427-434. Omar, A. (2008). Hurricane Katrina’s Silver Lining: E-Learning at Southern University atNew Orleans, Journal of Research in Business Information Systems, Vol.1, Iss. 1; p. 23-27. Omar, A. Liu L, & Koong K. (2008). From disaster recovery to mobile learning: A Case Study. Int. J. Mobile Learning and Organization, Vol. X, No. X. PP 1-14. Osika, E. R. (2006). Assessing Student Learning Online: It’s More Than Multiple Choice. Online Cl@ssroom. Ideas for Effective Online Instruction. Retrieved May 29, 2008, from http://www.magnapubs.com/products/0606ocff.html Palloff, R. and Pratt, K. (2001),Lessons from the Cyberspace Classroom: The Realities of Online Teaching. San Francisco: Jossey-Bass. Reynard, R. (2008). 21st Century Teaching and Learning: Assessing New Knowledge. T.H.E Journal. Retrieved May 1, 2008, from http://www.thejournal.com/articles.22531 Roper, A. R. (2007). How Students Develop Online Learning Skills. EQ Educause Quarterly. The IT Practitioner’s Journal, Vol. 30, Iss. 1. Retrieved May 13, 2008, from http://connect.educause.edu/Library/EDUCAUSE+Quarterly/HowStudentsDevelopOnlineLearningSkill.htm Siemens, G. (2004, Dec. 12). Connectivism: A learning theory for the digital age. Soloman, G; Schrum, L. (2007). Web 2.0: new tools, new schools. International Society for Technology in Education OR, USA. Totaro, M., Tanner, T., Fitzgerald, J., & Birch, R. (2005). Faculty perceptions of distance education courses: a survey. Journal of College Teaching & Learning, 2 (7), 13-20. Wyrostek, F. (2008). Bridging The Onsite/Online Gap For Students/Faculty: Integrating Adobe Breeze Software To Teach Managerial Finance. Association of Business Information Systems. Federation of Business Disciplines Conference. Houston, TX APPENDIX Table 1: Online Course Assessment (MGIS 166) Grades Distribution Avg. Pre-Test Score A B C D *F **FX W Avg. Post-Test Score Spring, 06 40 6 3 11 4 4 14 14 68 Fall, 06 42 6 7 11 - 3 10 5 70 Spring, 07 40 5 2 7 5 2 7 4 69 Fall, 07 45 5 8 7 5 3 7 10 68 Spring, 08 48 3 3 10 1 3 10 9 71 *F: Academically Fail *FX: Excessive Absence Table 2: Online Course Assessment (MGIS166) Grades Distribution Avg. Pre-Test Scores Avg. Post-Test Scores Percentage Improvement Spring, 06 40 68 28% Fall, 06 42 70 28% Spring, 07 40 69 29% Fall, 07 45 68 23% Spring, 08 48 71 23% Table 3 illustrates the coding of the grades. Grade A B C D F/FX Code 5 4 3 2 1 Table 4 shows some of the salient statistics Spring, 06 Fall, 06 Spring, 07 Fall, 07 Spring, 08 Mean 2.053571 2.627907 2.71875 2.4 2.076923 Standard Error 0.187983 0.233001 0.277643 0.221108 0.218523 Standard Deviation 1.406732 1.527888 1.570584 1.48324 1.364679 Sample Variance 1.978896 2.334441 2.466734 2.2 1.862348 Count 56 43 32 45 39 Table 5: ANOVA: Single Factor Groups Count Sum Average Variance Spring, 06 56 115 2.053571 1.978896 Fall, 06 43 113 2.627907 2.334441 Spring, 07 32 87 2.71875 2.466734 Fall, 07 45 108 2.4 2.2 Spring, 08 39 81 2.076923 1.862348 ANOVA Source of Variation SS df MS F F crit Between Groups 15.60645 4 3.901614 1.817023 2.414642 Within Groups 450.9238 210 2.147256 Total 466.5302 214