Evaluation of Posttest Scores for a Web-Based Tutorial Authoring Tool that Encompasses Pedagogy in the Development Process 

Thomas Janicki
University of North Carolina - Wilmington
Wilmington, NC  26403

Geoffrey Steinberg
Kent State University
Kent, OH  44242

ABSTRACT

Educational content on the Internet is rapidly increasing.  Academicians and businesses are placing more course material on-line to supplement classroom and business training situations.  In addition significant increases in undergraduate enrollments in Information System courses and the rapid pace of new knowledge in the field leads researches to call for new innovative approaches to learning.  Prior researchers have reported that this new web-based training technology (which has it foundation in computer-based training) has not integrated sound pedagogical practices into the authoring process when developing new tutorials.  This paper summarizes an experiment to evaluate the effect on posttest scores of a web-based authoring tool that includes learning theory in the development process for the author.  Early results indicate that the tool is more effective then traditional HTML authoring tools and that the number of exercises affects posttest scores in a positive manner.

Keywords: Learning Theory, Computer-Based Training, Web-Based Tutorials, Distance Learning


INTRODUCTION 

Educational content on the Internet is rapidly increasing. Academicians are placing more course material on-line to supplement and sometimes replace classroom instructions. Recognizing potential new marketing opportunities, universities are placing entire courses on the web to attract new students from around the world. The typical user and usage of the web is also changing from technical users to educational users.  Professionals from computer technical fields comprised 31.4% of all web users in 1995, while individual users for educational purposes totaled 23.7%.  In just one year computer occupation users dropped to 29.6% while educational users jumped to 27.8% of the web's overall users [Robin & McNeil, 1997]. 

Hamalainen et.al. [1996] and Robin & McNeil [1997] discuss that education has the potential to be the key application in electronic commerce.  However they warn that new technology alone will not make these new web-based tutorials and learning modules more effective. Hamalainen et. al. predict a gloomy forecast for learning advances in that we can expect only marginal improvements in student performance if web developers continue to re-implement traditional and conventional models borrowed from the classroom.  Their prediction is based on a review of the current offerings of web-based educational content that are mostly tutorials that passively transmit information or data.  By itself, more technology will not make education more efficient.  Robin & McNeil [1997] also support this opinion and call for new innovation modules of production, presentation and delivery that take advantage of the Internet's power that emphasize the capability of the learners to participate.

At the current time there does not seem to be consensus on what comprises effective learning modules.  This paper will briefly review the literature in the arena of computerized learning modules and synthesize a model we call Computer Supported Learning System (CSLS).  This model will assist in the development of tutorials and learning modules based on accepted learning concepts while utilizing the power of the computer.
PROBLEM STATEMENTS

Many educators, students, and employers intuitively feel that the integration of the computer with its interactive capabilities into a classroom or learning experience will enhance learning and the student's ability to apply knowledge and skills to future problem solving situations [Alavi, 1994].  Despite this belief, Alavi states that this new technology has not integrated sound pedagogical practices into the development of new learning modules either as a stand-alone lesson or combined with a classroom setting.  Robin & McNeil [1997] and Hamalainen et al [1996] express that new paradigms of education are needed.  These new paradigms must take advantage of the interactivity of the web.  Bork [1986] warns that when developers apply the computer to learning situations that they often tend to merely transpose books and lectures into web based materials.  

Schank [1998] also promotes the idea that new web-based courses must be different from the current course offerings, as the current offerings are not very good.  He has found that general methodology and the instructional pedagogy of the current material are poor.  A reason that these modules have not achieved their full educational potential is that information is not training, and that many of these systems present information, but do not necessarily teach [Schank, 1993].  He proposes that more one-on-one conversation and teaching based on learning concepts needs to occur.  New innovated models that take advantage of the inherent power of the platform are necessary. He envisions new software that emulates good teaching, and allows a designer to build in a virtual teacher into the module.  

Current web and computer based offerings are generally built by individuals with a background in HTML (HyperText Markup Language - the language of the Internet browsers) skills.  Early efforts to place course materials on the web required the developers have a minimal amount of programming knowledge.  Thus in general the early adopters of this new medium to transfer educational information (the World Wide Web) were individuals skilled more in programming applications than in educational principles.  

Murray [1998] reviewed the early web educational offerings as well as the current web courseware and summarized that they are difficult to build, and also costly as they generally are started from scratch.  Each new course starts anew and does not use a template from prior classes.  Also, he discusses the point that the developers of these modules primary build them based on their own teaching or learning style, not necessarily based on pedagogy.  Murray continues that most university professors never had a course in educational theory, so do they really know how to incorporate learning into an electronic module?  Also, he notes that many professors learn to teach from many years of experience, but the question is can they develop this experience into a methodology and pedagogy usable for computer based tutorials?

As discussed the number of educational offerings on the web will continue to rapidly grow.  However the overall effectiveness and quality of the materials published has been questioned.  Therefore, the following types of questions emerge: What is the pedagogy for web-based learning modules and does this pedagogy differ from traditional classrooms?  What are the design and navigation features that should be incorporated into web based learning modules?
DEVELOPMENT AND IMPLEMENTATION OF THE LEARNING MODEL

To provide a foundation of pedagogy for learning, a series of instructional design concepts were included in a framework for the model.  They were chosen as they repetitively surface as the keys to effective learning in research.  In addition these components may utilize the web's interactivity in their implementation.  Briefly they are a combination of learning theories from the behavioral psychology, cognitive theory and resource based theories of learning.  The instructional design concepts shown in Figure 1 consolidates the instructional design activities of Dear [1987], the events of instruction proposed by Wager [1982] Gagne, Briggs & Wager [1988] and Rojas [1989], the design guidelines of Hannafin and Peck [1988] and the strategies of instructional design by Merrill [1997].

Figure 1:Summary of Instructional Design Concepts

Next effective web-based design concepts were researched.  Table 1 summarizes the web-design concepts as offered by Jonassen et. al. [1995], Schank [1998], Murray [1996B], Ward &  Lee [1995], Leinder & Jarvenpaa [1993],Tennyson [1989]  Bugbee [1996], Anderson & Reiser [1985], and Martin [1995].  Similar to the instructional design principles, the concepts shown in Table 1 repetitively surfaced as positive influences on learning.  
Table 1:	Effective Web-Design Concepts

* Instructor as a facilitator
* Learner controls the pace
* Clear Navigation	
* Testing and Prompt Feedback
* Variety of Presentation Styles
* Multiple exercises
* Hands-On problems
* Consistent Layout
* Help Screens

Figure 2 merges the two schools of research (instructional technology and web-design to provide a model for more effective web-based tutorials.  As can been seen in Figure 2, the prototype authoring system named WeBTAS (Web-Based Tutorial Authoring System)  has two major functional tasks.  The first functional task is shown on the right side of Figure 2 and manages the creation of the HTML programming code, file tracking and the learner logging processes.  The system facilitates a consistent layout to the screens, incorporates help menus and also the administration of the test taking, grading, and feedback links.  The second functional area (shown in the left column of Figure 2) prompts the author for the actual learning content using for its foundation the instructional design concepts in Figure 1.

Figure 2: WeBTAS (Web-Based Tutoring Authoring System)

Using the concepts in Figure 2 as a guide, a prototype tutoring authoring system was created in the Visual Basic and Java Script programming languages and uses CGI (Common Gateway Interface) for the client/server processes.  This prototype model has three major operational components.  The first component assists the author or developer of web-based tutorial material.  This component guides a tutorial author through a series of prompts based on pedagogy.  (Verification of the adherence of five learning principles in the model occurs in Chapter Four).  The second operational component of the system transforms the data entered by the author to an HTML format and creates the necessary files and hyperlinks between the resulting files.  The second component also transfers all of the files, images, and databases to a web-server.  The third operational component manages the use of the system by a learner.  It provides an interactive learning environment comprised of lesson content, examples and exercises.  Test results and enhanced feedback links are also managed by the system.  

Creation of web-based tutorials by volunteer instructors

Ten Ph.D. graduate business students were recruited to develop web-based tutorials on two different topics using two different authoring systems. Each of the volunteer graduate students created a web-based tutorial for one of the topics (either equity or expectancy motivation theory) by utilizing any web-authoring tool they choose.  For the other topic the developers utilized the WeBTAS authoring system.  The instructors were randomly assigned which authoring system they would use on a particular topic, with half of the instructors creating WeBTAS tutorials on a topic and half created another topic using WeBTAS.  PowerPoint was chosen by eighty percent of the authors as the alternative web building tool due to their familiarity of the software.  

A panel of experts from a graduate school of education was surveyed to determine the learning concepts used in the design of the system.  The survey results indicate that the WeBTAS system does promote the inclusion of five learning principles proposed in its development.  These learning principles are a listing of learning objectives, list of pre-requisites, a variety of presentation styles, learner self-control through the lesson and feedback and testing.  In addition the survey also reports that the authoring tool incorporates effective web-design concepts.

An analysis of the tutorial content pages also occurred.  First it was observed that the developmental time to create the tutorials using the WeBTAS system were equivalent to the time it took to create the tutorials using PowerPoint.  An observed advantage of the WeBTAS system was that in the same amount of developmental time, authors were able to create an average of two additional screens of materials (approximately 20% more) for the learner.  Another advantage was that the WeBTAS created tutorials included interactive exercises and examples for the learner that was not present in the PowerPoint developed tutorials.

MODEL OVERVIEW

Once an author has built the tutorials via input boxes and WYSIWIG word processing screens, the WeBTAS system automatically translates the author's work to HTML and programming code and places it on a web-server.    Figure 3 provides an overview of the WeBTAS system from a learner's viewpoint, as well as detailing the different learning theories incorporated into the authoring tool.

Figure 3:	Flowchart of the learner side of the WeBTAS system


EXPERIMENTAL PROCESS
Once the graduate students developed the tutorials, they were ready for testing by actual learners.  A computer laboratory was reserved for each of the experimental sessions to reduce outside noise and interference.  Subjects were permitted to take notes during the reading and review of the web-based materials (lesson), but were not allowed any other outside materials such as textbooks or class notes.  Subjects were requested to complete all five steps of the experiment (Read Lesson 1, Quiz on Lesson 1, Read Lesson 2, Quiz on Lesson 2, and Survey of Learning Tool), and the computer tracked and timed their completion through all five steps.  Prior to the actual experiment, a pilot test was conducted. This pilot test demonstrated that the system had been programmed effectively and could support sixteen users simultaneously without any difficulties or degradation of speed and performance.  
The students were randomly assigned to initially receive a WeBTAS created learning module or a module created by a Microsoft Office HTML authoring tool. If they received a WeBTAS created module first then they received a non-WeBTAS created module later and vice versa. The order of presentation of the WeBTAS tutorial versus non-WeBTAS tutorial in the experiment was also random.  Also random was which topic they would receive for the WeBTAS module versus non -WeBTAS module.  In order to provide a control group 10% of the subjects were not given a web-based tutorial, but instructed to "surf" the World Wide Web on the appropriate subject (equity or expectancy theory).  

Undergraduate volunteers were recruited from three sections of the same class (Principles of Management) which was offered during the 1999 summer term.  Three different instructors taught the sections, and all instructors agreed not to present any course material on both equity and expectancy motivation theories (the tutorial topics chosen for development) prior to the actual experiment.  The three sections had a total enrollment of 75 subjects, with 68 students in class on the day of the experiment. Sixty-three of the students agreed to participate in the experiment.  
EXPERIMENT RESULTS
Table 2 details the means and standard deviations for the six different treatment groups.  Treatment Groups R1 and R4 received the same authoring systems for each tutorial topic, the only difference was the order of presentation.  For example R1 received the non-WeBTAS created tutorial on expectancy theory and then the WeBTAS created tutorial on equity theory, whereas R4 received the equity theory tutorial creating using WeBTAS first and the expectancy theory tutorial second.  Similarly R2 & R5, and R3 & R6 received the same treatments just in reverse order.

Two observations on the control group.  The control groups were told to use search engines and "surf" the web for information on the two topics.  The first observation was that two subjects, who were randomly assigned to the control group, entered the wrong registration number into the system, and therefore the system placed them into treatment groups At first glance the mean posttest scores for the control group who were the web "surfers" (66.67 and 58.34) appear to be significant and above the treatment groups.  After running a t test  between the control groups and the other treatment groups, no significance was found between the total quiz control group scores and the scores of each of the other treatment groups.  
Table 2:	Raw Test Scores

Expectancy Tutorial
Equity Tutorial

Non WeBTAS
WeBTAS
Cntl

Non WeBTAS
WeBTAS
Cntl 
# Sub
jects
29
30
4
30
29
4
Mean Score
59.54
67.74
66.67
54.22
54.25
58.34
SD
14.58
12.33
18.05
14.4
15.71
12.62
More research and testing is necessary to determine if the control group would repeat these scores.  Therefore the remainder of this analysis will contrast the WeBTAS treatment groups versus the non-WeBTAS treatment groups.
Key Results 

In the background section of this paper, several key questions emerged.  These questions revolved around the effectiveness of adding pedagogy to the authoring process of web-based tutorials, in addition to measuring the affect of other variables such as the number of examples, exercises, and comfort level of the learning subjects with the Internet and the Web.  These questions were framed for investigation as hypotheses for experimentation.
H1: A web-based tutorial authoring system that incorporates learning principles from the instructional technology and web-based design fields will be more effective (in terms of student learning) than web-based tutorials using Microsoft Office's HTML authoring tools.

The analysis was completed in two manners.  First the test scores for all WeBTAS created quizzes were compared to the test scores for all non-WeBTAS quizzes. Table 3 details the results of the posttest scores after a subject received a WeBTAS created tutorial versus a non-WeBTAS created tutorial.  Mean posttest scores for the WeBTAS tutorials was 62.05 versus a 56.84 score for the non-WeBTAS created tutorial.  
Table 3:Comparison of Means, WeBTAS Treatment versus Non-WeBTAS
 
Mean
N
Std. Dev
Std. Error Mean
 WeBTAS  Quiz Scores
62.05
59
15.97
2.0789
 Non WeBTAS Quiz Scores
56.84
59
14.61
1.9025
 
To determine if there is a difference between the two tutorial developmental methods this study used a matched pair or matched sample comparison. An alpha level of .10 was chosen as Bostrom et. al. [1990] report that for exploratory research (often with small sample sizes) alpha levels of .10 may be used to indicate the level of acceptance or rejection of the hypothesis. 

The analysis shown in Table 5 yields the following results:
H0:	(WeBTAS =  (Non-WeBTAS
Ha:	(WeBTAS < >  (Non-WeBTAS
At an (= .10 and 58 d.f.
And therefore reject H0 if t > 1.67;    t is 1.811, 
therefore we reject the hypotheses that the WeBTAS posttest scores are equal to the Non-WeBTAS scores.  In addition since the mean test scores for the WeBTAS treatment group was 62 versus 57 (see Table 3) for the non treatment group, it is supported that the WeBTAS scores are greater than the Non WeBTAS scores at and alpha level = .10 with  58 degrees of freedom and with p = .075.
Table 4:	Paired Samples Test of Posttest Scores, WeBTAS versus Non-WeBTAS treatment groups - The treatment groups are not the same at a .075 significance level
Comparison of the difference between posttest scores
Std. Deviation
t
df
Sig. (2-tailed)
 
 
 
 
 
 
22.1149
1.811
58
.075
 H2: The learner's comfort level with the World Wide Web and the Internet will have a positive effect on posttest scores.
Prior studies by Shlechter [1990] reported that the comfort level of the subjects with computers had an affect on posttest scores.  The demographic survey asked the subjects to rate their comfort with the World Wide Web.  Table 5, reports the average means for the different comfort levels with the Internet and Web as surveyed.  The scale is 1 with a extremely high comfort level with the Web to 5 indicating being uncomfortable with the Web.  One subject did not answer this question (Comfort = 0).
Table 5:	Mean Posttest Scores by Comfort Level with the Web
COMFORT
Mean
N
Std. Deviation
 1.00
117.7792
12
22.2636
 2.00
123.7506
16
18.3720
 3.00
116.1525
24
21.2800
 4.00
115.5567
6
28.1775
5.00
0.00
0
0.00
 Total
118.8875
58
21.1666
 Scale:  1= High level of comfort with the web, 3= neutral, and 5= no comfort.

After completing the linear regression of comfort level with the World Wide Web H2 was not supported.  Therefore no significant affect on posttest scores from the subjects' level of comfort with the Web (p= .40,  F= .721 
H3: The time a learner spends on a tutorial lesson will have a positive effect on posttest results.

Researchers [Dyck and Mayer, 1989] have reported an effect correlating the amount of time a learner invested in a lesson versus overall performance.  The results do not support this hypothesis as the p value is .29, again above the acceptable level indicating no support for this variable on the posttest scores, and the hypothesis is not supported.

For the last two hypotheses, only the posttest scores for the WeBTAS created tutorials was used in this analysis.  The unit of measurement in the WeBTAS modules was the number of web-pages developed by the system.  For the non-WeBTAS system there the same unit of measurement was not available.  The quantification of only the WeBTAS examples and exercises was an objective exercise not a subjective as it would have been with the non-WeBTAS modules.
H4: Including examples in the lesson content of web-based tutorials will have a positive effect on posttest results.

No significance was supported for this hypothesis.  Shown in Table 6 is the analysis of variable and mean scores.  P values equaled .658 from this analysis, and provide no evidence from the effect of the number of exercises on posttest scores.
Table 6:Mean Scores by the Number of Examples in the Tutorial - no significance supported
Number of Examples
Mean
N
Std. Deviation
 .00
62.2233
6
11.6747
 1.00
65.0008
12
13.3713
 2.00
61.0505
20
15.0512
 3.00
61.2695
21
19.6209
 Total
62.0512
59
15.9684
 
H5: Including exercises and mini-quizzes in the lesson content of web-based tutorials that will have a positive effect on posttest results

The final hypothesis was to determine the affect of mini quizzes or exercises on the overall test scores.  This hypothesis was found to be significant at an alpha level of .10, with a p value = .092.  The p values and analysis of variance results are shown in Table 7.  

Table 7:	Mean Scores by the Number of Exercises in the Tutorial
Number of exercises or mini quizzes
Mean
N
Std. Deviation
 .00
53.3333
18
16.8052
 1.00
69.6973
11
14.4105
 2.00
66.2323
13
12.9051
 3.00
63.1371
17
15.1139
 Total
62.0512
59
15.9684
 
t
Sig.
 

 17.669
.000
 1.716
.092
 a  Dependent Variable: WEBSCOR

An observation of mean posttest scores in Table 8 yields questions for future research.  At first glance as the number of exercise increases from one to three, the mean posttest scores decreased and this needs to be investigated.  This research used as the criteria the number of exercise web pages developed, and did not count the number of questions on each page.  For some developers they included multiple questions in one exercise, while other developers included one or two questions per mini-quiz.  Therefore future research should consider not only the number of web pages developed, but also the total number of questions developed.
SUMMARY
This research proposed a model for a web-based authoring tool based on learning principles.  The model consolidates the theories of instructional technology and web-design researches into one framework for an authoring tool.  A prototype system was written and an experiment was conducted to measure its effectiveness versus authoring tools that do no include a pedagogical component in the design process.

A pilot test of student subjects was conducted to verify the robustness and soundness of the prototype system.  This test by sixteen subjects indicated the system to be reliable and able to handle multiple learners at the same time. Also the logging of subject movements through the WeBTAS tutorial by individual page visited was verified.  Based on posttest interviews and surveys the instructions given to the subjects were modified prior to the actual experiment. 

Next an experiment was conducted in a laboratory setting, employing undergraduate subjects from three different sections of the same class. The goal of the experiment was to create the necessary tool and data collection techniques to examine five different hypotheses and also explore any other factors effecting subject performance on the posttest scores.

The posttest analysis supported two of the hypotheses. The first hypothesis concerned the learning affect on subjects based on the completion of web-based tutorials that had been created using a prototype authoring system.  As noted earlier, the authoring system was evaluated by a panel of educational experts and found to encourage the developer to create the tutoring based on learning principles and effective web design principles.  The question is: "would this inclusion of pedagogy have an effect on a learner's posttest results?"

The first hypothesis supported was that web-based tutorials built having a pedagogical component, as part of its developmental process would be more effective in terms of subject learning.  The analysis of the experiment reported that the posttest results for those tutorials that had been developed using the WeBTAS authoring tool had a positive affect on quiz scores when compared to the tutorials created using a traditional HTML tool. 

Two additional hypotheses were included in the study to measure the effect that a presenting the lesson content would affect learning.  Previous researches noted that learners have different learning preferences, and that some learners like examples, while others prefer hands-on exercises.  An analysis of the posttest scores indicated mixed results.  One hypothesis was accepted and one hypothesis was rejected in the analysis based on a variation of learning presentation styles

Supported was the hypothesis that proposed that the inclusion of exercises or mini quizzes would affect posttest scores in a positive manner.  The hypothesis concerning a second learning style was not supported.  This hypothesis proposed that the inclusion of examples in a web-based tutorial would affect posttest scores in a positive manner.  The posttest results did not report any significant difference in mean posttest scores based on the number of examples	Two other hypotheses could not be supported.  These hypotheses were developed for testing as a result of the literature review. Stephenson [1996] reported the level of comfort of the subjects to the computers affected posttest scores.  Subjects were asked to rank their level of comfort with computers and the World Wide Web.  Regression analysis did not support the hypotheses that the comfort level of subjects to the Internet and web would affect their test scores.  The last hypothesis was based on a study by Dyck and Mayer, which reported that test scores were affected by the amount of time the subjects spent with the tutorials.  The affects of time spend reading and participating with the tutorials was not shown to be significant to overall posttest scores.
FUTURE RESEARCH
The initial results indicate the WeBTAS affects learning in a positive manner.  However it is only one study, more study is necessary.  There is a need to develop additional tutorials by a larger pool of developers.  These tutorials should be created on a variety of subjects and cover many disciplines.  These tutorials should be built not only for educational purposes but also corporate training needs.  It would be interesting to note that the motivation factor of employees to learn a new skill in the workplace might be different from students in a class setting.  This research should occur over a longer period of time so that the same developer can build more than one tutorial utilizing the WeBTAS system.  This would assist to measure if any learning curve of the system features occurred.	

Another interesting study would be to contrast the overall learning affect based on different levels of instructor experience.  Early results indicate that WeBTAS assists novice instructors to create more effective tutorials.  Would WeBTAS assist "expert" instructors to create more effective tutorials? 

Another thrust of future research would be evaluating the WeBTAS tutorials versus other systems that claim to assist developers in building web-based tutorials.  Some of these other tools are Director, AuthorWare, TopClass and WebCT.  These tools provide different features and the measurement of their learning impact is warranted.

One last area for research would be evaluating each of the learning theories incorporated into the model, to determine if they have a positive effect on the learner.  Are there any other principles that could be incorporated into a web-based system that may influence posttest results?

REFERENCES

Alavi, M. (1994). Computer mediated collaborative learning: an empirical evaluation. MIS Quarterly, 18 (2), 159-174.

Anderson, J. and Reiser, B.(1985) The LISP tutor. Byte Magazine, 10 (4), 159-175.

Barrett-Murie, D. (1995). Distance Learning spurs growth in VSAT Market. Satellite Communication, 19 (9), 87-90.

Bork, A. (1986). Advantages of computer based learning. Journal of Structured Learning, 9 (1), 63-76. 

Bostrom, R., Olfman, L., and Sein, M. [1993]. Learning Styles and End-User Training: A First Step. MIS Quarterly, 17 (1), 118-120.

Brandt, S. (1997). Constructivism: Teaching for Understanding of the Internet. Communications of the ACM, 40 (10), 112-117.

Bugbee, A. (1996). Equivalence of paper and pencil and computer based testing. Journal of Research on Computing in Education, 28 (3), 282-299.

Collins, A. and Brown, J.S. (1987). Cognitive apprenticeship: Teaching students in the craft of reading, writing and mathematics. Cognition and Instruction. Hillsdale NJ: Lawrence Erlbaum.

Collis, B. (1996). The Internet as an Educational Innovation: Lessons from experience with computer implementation. Educational Technology, 27 (11), 21-28.

Dear, B. (1987). AI and the Authoring Process. IEEE Expert, 2 (4), 17-23.

Dick, W. and Carey, L. (1996). The Systematic Design of Instruction. New York: Harper Collings

Dyck, J. and Mayer, R. (1989). Teaching for transfer of computer program comprehension skill. Journal of Educational Psychology, 81 (1), 16-24.

Gagne, R.; Briggs, L. and Wager, W. (1988). Principles of Instruction Design Third Edition. New York: Holt, Reinhard, & Winston.

Gagne, R.; Wager, W. and Rojas, A. (1981). Planning and Authoring Computer-Based Instruction Lessons. Educational Technology, 21 (9), 17-26.

Hamalainen, M.; Whintson, A. and Vishik, S. (1996). Electronic markets for learning: education brokerages on the Internet. Communications of the ACM, 39 (6), 51-58.

Hannafin, M. and Peck, K. (1988). The Design Development and Evaluation of Instructional Software. New York: MacMillian Publishing.

Jonassen, D.; Davidson, M.; Collins, M.; Campbell J.and Haag, B. (1995). Constructivism and Computer Mediated Communication in Distance Education. American Journal of Distance Education, 9 (2), 7-25.

Leidner, D. and Jarvenpaa, S. (1993). The Information Age Confronts Education: Case Studies on Electronic Classrooms. Information Systems Research, 4 (1), 24-51.

Martin, T. (1995). The development of interactive WWW courseware for students of engineering and technology at deakin university. Proceedings of the Australian World Wide Web Conference. Lismore Australia: AUSWEB.

McGee, M. (1998). Save on Training, Information Week, n708, 141-146.

Merrill, M.D. (1997). On Instructional Strategies. Instructional Technology Magazine, 2 (1).

Merrill, M. D. (1988). The Role of Tutorial and Experiential Models in Intelligent Tutoring Systems. Educational Technology, 28 (7), 7-13.

Molina, C. (1995). Transitioning to CBT. Performance & Instruction, 35 (9), 26-33.

Murray, T. (1998). Authoring Knowledge Based Tutors: Tools for Content, Instructional Strategy, Student Model, and Interface Design. Journal of the Learning Sciences, 7(1) 5-64. 

Murray, T. (1996A). From Story Boards to Knowledge Bases, the First Step in Making CAI 'Intelligent'. In Carlson & Makedon (Eds.) Educational Multimedia and Hypermedia, Proceedings, (pp. 509-514). Charlottesville VA: AACE. 

Murray, T. (1996B). Special Purpose Ontologies and the Representation of Pedagogical Knowledge. Proceedings of the International Conference for the Learning Sciences. Charlottesville VA: AACE.

Niederman, F.; Brancheau, J. and Wetherbe, J. (1991). Information systems management issues for the 1990's. MIS Quarterly, 15 (4), 475-500.

Robin, B. and McNeil, S. (1997). Creating a course-based web site in a university environment. Computer & Geosciences,  23 (5), 563-572.

Schank, R. (1993). Learning Via Multimedia Computers. Communications of the ACM, 36 (5), 54-56.

Schank, R. (1998). Horses for courses. Communications of the ACM, 41 (7), 23-25.

Shlechter, T. (1990). Relative Instructional Efficiency of Small Group CBT. Journal Educational Computing Research, 6 (3), 329 - 341.

Stephenson, S.(1992). The Role of the Instructor in Computer-Based Training. Performance & Instruction, 32 (7), 23-26.

Tennyson R. (1989). Cognitive Science and Instructional Technology: Improvements in Higher Order Thinking Strategies. Proceedings of the Association for Educational Communication & Technology, Dallas TX: AECT.

Woolf, B. (1987). Survey of Intelligent Tutoring Systems. Proceedings of the American Association of Artificial Intelligence.
181