Development and Preliminary Evaluation of a 
Rubric for Assessing Educational and Workplace Computing Competence

Teresa A. Wagner
wagnerta@muohio.edu
Farmer School of Business, Miami University
Oxford, Ohio  45056

David D. Langan 
dlangan@usouthal.edu
School of Computer and Information Systems, University of South Alabama
Mobile, Alabama  36688

Steven A. Corman
steven.corman@case.edu
Weatherhead School of Management, Case Western Reserve University
Cleveland, Ohio  44106

 Jeffrey P. Landry
jlandry@usouthal.edu

Herbert E. Longenecker, Jr.
LongeneckerB@gmail.com

School of Computer and Information Systems, University of South Alabama
Mobile, Alabama  36688

Abstract

Academic programs are expected to produce graduates who satisfy key learning objectives as defined by measurable competencies.  This paper describes the development and validation of a rubric used to assess discipline specific skills that enable direct validation of program outcomes and objectives.  A case study of a small sample of year-long, full-time externs provides support for the use of such a rubric for assessing the increasing educational and professional maturity of students in four-year, undergraduate computing programs.  Additional work is needed to further validate the Computing Competence Metric.

Keywords: accreditation, certification, program outcomes, rubrics


1.  INTRODUCTION
Regardless of any additional aims, academic programs exist to develop “confident and competent” students (Davis, 2002).  Increasingly, regional and national level accrediting agencies are requiring academic programs to prove that they have met the aforementioned goal in order to maintain their accreditation status (Boff et al., 2009).  Programs can and must demonstrate their success in the production of qualified students at two levels: program outcomes and program objectives (Rogers, 2004).  Program outcomes “describe what students are expected to know and be able to do by the time of graduation” (p. 6). Program objectives “describe the expected accomplishments of graduates during the first few years after graduation” (p. 6), or in other words, the transfer of learning from the classroom to the application that knowledge in the professional world.  
Program objectives can be defined in terms of retained knowledge post graduation or more importantly (as is the focus of this paper) in terms of demonstrated professional development.  Although defining program objectives and developing a corresponding measurement method are important to the assessment process, these steps are often difficult given the individualized and nebulous aims of different academic programs. Additionally, even if it is possible to define representative objectives and design a measurement process, it is often difficult to collect data at a quality level appropriate for analysis.
In an attempt to define the skill base underlying the computing disciplines, Landry et al. (2000) successfully developed a set of skill definitions by utilizing both qualitative and quantitative methodologies.  These skill definitions were utilized in Colvin’s survey of recent (1-3 years) graduates in which the graduates gave perceptual ratings of the skill set needed for their position (Colvin, 2007).  The survey results support the work of Landry et al. (2000), producing skill sets and skill depths within a few percentage points of the original work.  
Given the increasing emphasis on measureable and successful program outcomes and objectives, this paper aims to develop a rubric utilizing the skill list of Landry et al. (2000). The ultimate success of such an instrument could potentially advance the study of computing education as well as employment practices.  If successful, the rubric would be a valuable tool for employers seeking a method to rapidly assess employee performance, as well as for university program administrators who are trying to assert that their academic programs are appropriately aligned and attaining program objectives.  The rubric may also prove useful in a variety of personnel decisions, such as determining which employee to assign to which task by matching an employee’s skills to the Knowledge, Skills, and Abilities (KSAs) required for the task.  The rubric most likely would not replace an employee performance review, but rather would supplement it.
2.  DEFINITION OF SKILLS
Before researchers can effectively compile a list of which skills are necessary for a position, family of positions, or entire industry, one must define what exactly is meant by “skill.”  Defining the concept of “skill” in terms of job requirements has been researched extensively since the 1930s, beginning with the Department of Labor’s Directory of Occupational Titles (DOT) which outlined the activities of a specific job, and continuing to the present day with constant advancements in the description and measurement of a job and its worker (termed job analysis).  In particular, job analysis refers to the process of defining job characteristics and worker characteristics for specific jobs, job families, or industries (Standards for Educational and Psychological Testing, 1985; Harvey, 1991).
Industrial Organizational Psychologists, Psychometricans, and other applied psychologists have acknowledged the need and value of defining the characteristics of a job (i.e., job specifications); as well as determining which individual characteristics a person who is successful in the job must possess (i.e., worker specifications).  These needs can be addressed using job analysis.  In fact, job analysis is the basis for most managerial and human resource functions, including performance appraisal and hiring decisions (Fine and Cronshaw, 1999).  In addition, the use of worker specifications is key in the development of licensure and accreditation examinations (Wang, Schnipke, and Witt, 2005).  Particularly of interest to this paper is the ability of job analysis to derive KSA levels that are needed for success in a specific job.  Under the vernacular of job analysis, “skills” refer to specific aptitudes that one can develop through long term effort in relation to the data, people, and things used or encountered in a job (Harvey, 1991).  This structure was proposed by Sidney Fine in 1955 and is still mirrored today in the literature of popular business (Business Dictionary, 2009a).  In fact, according to BusinessDictionary.com (2009b), skill is defined as: “[The] ability and capacity acquired through deliberate, systematic, and sustained effort to smoothly and adaptively carryout complex activities or job functions involving ideas (cognitive skills), things (technical skills), and/or people (interpersonal skills).”  
It is reasonable to infer that the skills supported empirically in research (Landry et al., 2000; Davis, 2002; Colvin, 2007) are compatible with this definition.  Prior validation studies derived 38 sub-skill groupings (Landry et al., 2000).  As a result an applied measure of these sub-skills, The Center for Computing Education Research (CCER) developed an assessment exam termed the “Information Systems Analyst (ISA)” for use in academic institutions (McKell et al., 2003, 2004).  The exam uses a pass/fail metric allowing those who pass to apply to the ICCP for certification.  The ICCP ISA certificate is based on the attainment of the 38 sub-skills.  In addition to the thousands of students who have completed the exam, institutions have used the exam in their program outcome assessment process by simultaneously referring both to detailed sub-skill achievements, as well as performance on Learning Units specified within the national model curriculum (Davis et al., 2002).
Of concern in this paper is the further development of this instrument such that it can be more readily utilized for the evaluation of program objectives.  Although the authors agree that many programs find the current exam structure useful, it is often the case that employers either do not need the detail provided by the ISA exam or are not willing to spend the time administering the exam, and/or figuring out how to interpret the results.  Rather, employers often want an easily interpretable list of competences that can be used in a quick and reliable check for management purposes.  
3.  DEFINITION OF COMPETENCE
A competence under job analysis is a cluster of KSAs needed to perform a job task or general work activity.  As was the case with the skill definition, other less research based sources (e.g., Business Dictionary, 2009a) mirror the definition advanced by job analysis literature.  We observe that BusinessDictionary.com (2009a) describes the word competence as follows: General: (plural competences) cluster of related abilities, commitments, knowledge, and skills that enable a person (or an organization) to act effectively in a job or situation.” 
Given the fact that employers might be more willing to use a greatly simplified tool, our rubric yields a simplified result for use in the evaluation process.  We suggest that it may be more appropriate to look at the aggregate of the sub-skill groupings, or competencies.  In accordance with the results of Landry et al. (2000), the 38 sub-skills aggregate to eight primary KSA groupings.  In the IS’95 (Longenecker et al., 1994, 1995) and IS’97 Model Curriculum (Couger et al., 1997) these eight groupings were referred to as “Exit Sub-Areas” of the curriculum.  They were the same areas that guided the confirmatory factor analysis of Landry et al. (2000).  By examining the skill clusters at the competence level of aggregation we can form item production guidelines for an instrument which is grounded in empirical research.  Later in the paper we will propose a specific rubric for this purpose.
4.  SKILL DEVELOPMENT IN THE COMPUTING CURRICULUM
Although research has shown that the overall skill levels and KSAs needed by different specialties housed under the computing job family are similar (e.g., Landry et al., 2000; Colvin, 2007), the goals of Information Systems are very different from those of Computer Science, Information Technology, and Software Engineering. These differences may lead to different courses with very different content emphases.  For example, the goal of a Software Engineering program might be that graduates primarily produce “high quality defect free software completely compliant with given requirements.”  For an Information Systems Program, the focus might be to ensure that graduates are able to “assist their clients in achieving higher productivity through the application of information technology.”  In both cases, software production is involved; for the Software Engineer, the software is the focus, and for the Information Systems professional, the people are the focus.  Yet, the eight competences are entirely appropriate for both programs and in fact, IS2002 (Davis et al., 2002) is based on these KSA groupings.  What differentiates the two is the conceptual framework under which the respective faculty and post-graduation professionals operate.
Despite the differences that exist between the programs IS, IT, CS, and SE, students from each program have taken the ISA exam and passed.  Interestingly, factor analysis on exam items has shown that there is a prevailing strong first factor upon which all items load.  This factor presumably represents the overall discipline of computing and the commonality among the disciplines (Wagner et al., 2009).  This is not to suggest that the disciplines are the same by any means – they are certainly not! However, there are clearly shared aspects of competences.
5.  THE SKILL METRIC AND SETTING LEVELS OF COMPETENCE
For the purposes of this paper a rubric is defined as a two-dimensional subjective measurement tool designed and utilized as a way to reduce subjective bias that might otherwise accompany ad hoc or arbitrary methods of evaluation.  The dimensions in the rubric are intended to be a logical break-down which results in the components of that to be evaluated.  For each of the eight competences a multi-level scale was created.  The scale is a 5-point behaviorally anchored scale which includes behavioral manifestations of what is required to obtain each rating (see Appendix 1 for a list of behaviors corresponding with each anchor).  Typically, competences to the left require less achievement, whereas those to the right require considerable more accomplishment as well as performance of all of the ones to the left.  The rubric is defined such that the first three competences would be characteristic of undergraduate behavior, whereas the fourth and fifth would indicate significantly advanced experience.  In that the sub-skills required a qualitative assessment of meaning of the job ad words, to build this rubric involved not only the qualitative combination of the sub-skills into the higher level competence representing the sub-skills, but the experience/educational level factors as well.  
Development of the Rubric Dimensions
As stated above, the KSA groupings selected for this rubric come from Landry et al. (2000) and Colvin (2007).  The skills analyzed were initially defined by subject matter experts, (SMEs) as is dictated by the job analysis process.  Each of the competences represents one of the eight factors listed below: 
* Individual and Team Skills
* Project Management
* Business Fundamentals
* Organizational Systems Development/SAD
* Database/Data Management
* Software Development
* Web Development
* Systems Integration/Networking
In addition to the support garnered through initial factor analyses, research being conducted presently is further supporting this structure via additional factor analysis and path analysis (Wagner et al., 2009)
Given the rather extensive psychometric and statistical basis for both the development of (e.g., Colvin, 2007) and measurement of (e.g., Wagner et al., 2009) the skill set defined by Landry, it was deemed acceptable for use as the basis for a successful rubric.
6.  PRELIMINARY APPLICATION OF SKILL RUBRIC TO AN EXTERNSHIP: A CASE STUDY
Overall Method and Participants
The Computing Competence Maturity Metric was pilot tested in a case study method that utilized both observational and qualitative data.  The case study sampled students participating in an externship program, a joint academic/industry partnership in which the students worked for a local firm while being hired and supervised by a university faculty member (Langan & Barnett, 2003).  In this instance, the local firm was a telephone service provider, and the supervising faculty member was one of the researchers co-authoring this paper.  The use of this method was deemed reasonable based on the work of Ford and Voyer (2000) and Dale (2000) who followed participants documenting their specific experiences in an organizational setting.  Further the case study method has been used as a way for studies to address exploratory, descriptive and descriptive questions (Stake, 1995; Yin, 1994).   
Procedure and Analysis
The study utilized a pre/post design that included the ratings of students at two points in time—the point at which they were hired and at the termination of the externship, usually one year later, although some students participated for a longer span of time.  The Computing Competence rubric was used for ratings, with the students completing a pre/post self-rating with the corroboration of the researcher on each dimension of the rubric.  Post externship a 360 degree feedback was used incorporating multiple sources of subjective forms of assessment from all levels, including student self reports.  
Results and Discussion
For each dimension of the rubric, three mean performance indicators were computed and reported: 1. pre-externship, 2. expected, and 3. post-externship.  The pre/post means were computed from the student self-ratings.  The expected score of 60, on a 100-point rating scale, represents an “exit” level of competence that students should be expected to achieve by the completion of a four-year degree program.  The 60% level was at the third of five progressive levels of the rubric.  It was thought that the highest two levels required more practical experience than what most graduating seniors would have been able to attain.  We hypothesized that the rubric, representing a progression combining educational and professional maturity, would be able to discern the value-added provided by the externship experience.     
The results indicated that student self-perceptions of computing competence increased from the pre- to post-externship, and in all eight competence dimensions, the post-externship rating exceeded the expected score (see Appendix 2 for a graph of the results).  The pre-externship scores, which ranged from 32 to 54 across the various competences, were predictably low, as students are not yet at the graduating, or exit point in the curriculum.  The fact that all of the means are greater than 60 after the externship indicates that the students benefitted from the externship in a measurable way.  The high post-externship scores confirm the hypothesis that externships provide a value-added experience, that is, additional professional maturity the complements that developed in the educational setting.  
These results are encouraging for two reasons.  The rubric was able to capture student maturity growth in a measurable way.  Furthermore, it was able to capture growth over a single year of experience.  This evidence lends support to the rubric being used as an indicator of maturity over the course of a four-year degree program.
7.  PROPOSED FUTURE RESEARCH AND CONCLUSIONS
The results obtained in the above study with a small sample are only the first step in the validation process of the rubric.  Although the results are encouraging, it is important to take the exploration of the rubric to the next level.  The case study method is typically used as a basis for determining whether future research should examine outcomes in larger samples (Johnson and Christensen, 2004).  As such, further studies are needed to replicate these results in larger samples and different populations, as well as to determine the accuracy needed for a rubric utilized in such a wide-scale manner.
One of the future steps in the validation process is the illustration of the level of accuracy and predictability necessary for a “high-stakes” certification instrument (Uniform Guidelines, 1978).  Arguably some of the more pressing issues are the validation of the structure with a large sample of employers and other SMEs, allowing for use in the workplace.  In fact, The Uniform Guidelines in Testing (1978), which sets the legal precedent for the use of employment testing, requires a job analysis based validation in applied settings.  Clearly, the empirical linking of skills and competences is critical.
Another important step is to explore the degree to which the rubric is appropriate for use in the different sub-disciplines.  This is arguably an important question, but requires a large and diverse sample which covers the breadth of the sub-disciplines adequately.  It may very well be the case that various aspects of the rubric will be more successful in different sub-disciplines or at different levels of academic or job experience.  
In general, the case study test of the rubric was only the first step in the validation process for the promising rubric.  Although much work is left to be done before one can say with assurance that the process of validation is complete, the results of the case study and other results provide sufficient evidence that the validation process should be continued.
8.  REFERENCES
Boff, G., G. DeLorenzo, L. Kovalchick, and P. Sible (2009). “Leveraging Academic Resources in the ABET Accreditation Process: A Case from California University of Pennsylvania.” Information Systems Education Journal, 7(79).
BusinessDictionary.com. (2009a). “Definition: Skill.” Retrieved July 31, 2009, from http://www.businessdictionary.com /definition/competence.html 
BusinessDictionary.com. (2009b). “Definition: Competence.” Retrieved July 31, 2009, from http://www.businessdiction ary.com/definition/skill.html 
Colvin, R. (2007). “Information Systems Skills and Career Success.” Unpublished master’s thesis, University of South Alabama.
Couger, J. D., H. E. Longenecker, D. L. Feinstein, G. B. Davis, J. T. Gorgone, D, Dologite, G. M. Kasper, J. C. Little, J. S. Valacich, and A. M. Jenkins. (1995). “IS’95: Guidelines for Undergraduate IS Curriculum.” MIS Quarterly, 19(3), 341-359.
Couger, J. D., G. B. Davis, D. L. Feinstein, J. T. Gorgone, and H. E. Longenecker. (1997). “IS’97: Model Curriculum and Guidelines for Undergraduate Degree Programs in Information Systems.” Data Base, 26(1), 1-94.
Davis, G. B., J. T. Gorgone, J. D. Couger, D. L. Feinstein, and H. E. Longenecker. (1997). “IS ‘97 Model Curriculum and Guidelines for Undergraduate Degree Programs in Information Systems.” ACM, New York, NY and AITP, Park Ridge, IL.
Fine, S. A., and S. F. Cronshaw. (1999). Functional Job Analysis: A Foundation for Human Resources Management. Lawrence Erlbaum Associates, Mahwah, NJ.
Fine, S. (1955). “A Structure of Worker Functions.” Personnel & Guidance Journal, 34, 66-73. 
Gorgone, J. T., G. B. Davis, J. S. Valacich, H. Topi, D. L. Feinstein, and H. E. Longenecker. (2002). “IS 2002 Model Curriculum and Guidelines for Undergraduate Degree Programs in Information Systems.” ACM, New York, NY, AIS, and AITP, Park Ridge, IL.
Harvey, R. J. (1991). “Job Analysis.” In M. D. Dunnette & L. Hough (Eds.), Handbook of Industrial and Organizational Psychology (2nd Edition), Consulting Psychologists Press, Palo Alto, CA.
Langan D.D, and L. Barnett. (2003). “An On Campus Faculty-Directed Team Project: An Alternative  Industry – Academic Partnership Model (Joint Perspective).” Proceedings of ASEE Regional Conference, Macon, Georgia, Spring 2003. 
Landry, J. P., H. E. Longenecker, B. Haigood, and D. L. Feinstein. (2000). “Comparing Entry-Level Skill Depths Across Information Systems Job Types: Perceptions of IS Faculty.” Proceedings of the Americas Conference on Information Systems, August 10-13.
Landry, J. P., J. H. Pardue, H. E. Longenecker, and D. L. Feinstein. (2003). “A Common Theme for IT Degree Programs.” Communications of the ACM, 46(11), 117-120.
Longenecker, H. E., J. D. Clark, J. D. Couger, D. L. Feinstein, J. T. Gorgone. (1994).  “Development of IS’95:  A Joint Activity of DPMA, ACM, ICIS, AIS.”  Proceedings of ISECON’94, pp. 1.
Longenecker, H. E., D. L. Feinstein, J. D. Couger, G. B. Davis, and J. T. Gorgone. (1995).  “Information Systems ’95:  A Summary of the Collaborative IS Curriculum Specification of the Joint DPMA, ACM, AIS Task Force.”  Journal of Information Systems Education, 6(4), 174-187
McKell, L. J., J. H. Reynolds, H. E. Longenecker, and J. P. Landry. (2003). “Aligning ICCP Certification with the IS2002 Model Curriculum: A New International Standard.” International Business & Economics Research Journal, 2(9), 87-91.
McKell, L. J., J. H. Reynolds, H. E. Longenecker, J. P. Landry, and J. H. Pardue. (2004). “Information Systems Analyst (ISA): A Professional Certification Based on the IS2002 Model Curriculum.” Proceedings of the European Applied Business Research Conference, June 14-18.
McKell, L. J., J. H. Reynolds, H. E. Longenecker, J. P. Landry, and J. H. Pardue. (2005). “The Center for Computing Education Research (CCER): A Nexus for IS Institutional and Individual Assessment.” Proceedings of the Information Systems Educators Conference 2005, Columbus.
McKell, L. J., H. E. Longenecker, J. P. Landry, and J. H. Pardue. (2007). “Integrating Institutional And Individual Information Systems Assessment Through The Center For Computing Education Research.” Proceedings of Americas Conference on Information Systems.
McNurlin, B. C. and R. H. Sprague. (1998). Information Systems Management in Practice, (4th Edition). Prentice-Hall, London.
Pardue, J. H., J. P. Landry, H. E. Longenecker, and L. McKell. (2006). “Computing Program Curriculum Assessment:  The Emergence of a Community of Practice.” Journal of Informatics Education Research, 8(2), 105-119.
Wagner, T. A., S. A. Corman, and H. E. Longenecker. (2009). “A Qualitative Validation of an Accreditation Exam for Computing in the Workplace.” Manuscript in preparation.
Rogers, G. (2002). “The Language of Assessment: Humpty Dumpty Had a Great Fall…” Communications Link, Summer. 
Wang, N., D. Schnipke, and E. A. Witt. (2005). “Use of Knowledge, Skill, and Ability Statements in Developing Licensure and Certification Examination.” Educational Measurement: Issues and Practice, 25(2), 15-22.

9.  APPENDICES

Appendix 1


Appendix 2