Paper Presentation
END-USER ACCEPTANCE OF A LEARNING MANAGEMENT SYSTEM IN TWO HYBRID LARGE-SIZED INTRODUCTORY UNDERGRADUATE COURSES: A CASE STUDY
Cheng-Chang
“Sam” Pan
University of Central Florida, Orlando, FL, USA
Glenda Gunter
University of Central Florida, Orlando, FL, USA
Stephen Sivo
University of Central Florida, Orlando, FL, USA
Richard Cornell
University of Central Florida, Orlando, FL, USA
Abstract
Drawing from the Technology Acceptance Model, this correlational study was intended to determine student users' adoption of WebCT and its relationship to their final grade. In acknowledging the relevance of human-computer interaction, the author attempted to investigate a belief-attitude-behavior relationship in a WebCT setting. Given a social learning approach, two external variables affecting the relationship were taken into account. The quantitative inquiry involved two large-sized undergraduate hybrid courses with a total of 469 students participating. To model both direct and indirect causal effects as well as measurement error, path analysis was conducted, using structural equation modeling. Implications for practitioners and researchers are addressed.
Introduction
Background
As technology continues to evolve, learning can occur regardless of geographical or temporal differences. To respond to this type of learning experience, cyber education has been introduced for years (Cornell, 2001). In gaining more acceptance, cyber education has experienced phenomenal growth around the globe and it has become a top priority in higher education and other organizations (Shelly, Cashman, Gunter, & Gunter, 2002). Of all potential advantages that come with the growth and acceptance from Web-based instruction, the most significant factors are increased enrollment, better finances, and a higher academic reputation. The advantages Web-based instruction includes are not exclusively utilized by universities and institutions of higher learning. Prospective and current students enrolled in coursework in both education and training environments are able to enjoy advantages or collateral benefits such as high-end computer hardware and groundbreaking software systems used to further advance the growth of learner support. All these are true under some assumptions. One of them pertains to end-user acceptance of the course management system adopted. That is the center of our study.
Purposes
The purpose of this structural equation modeling study was to investigate the causal relationship between student attitude toward the use of WebCT and their actual system use. In doing so, we intended to further replicate the Technology Acceptance Model (TAM).
This hypothetic model is represented in the following diagram with arrowed solid lines denoting the original TAM and arrowed broken lines representing the expanded TAM (see Figure 1).
Figure 1. The TAM Hypothesized.
Seven variables involved in this study are defined as follows:
- Perceived ease of use: the degree to which the individual believes that using the target system would be free of mental and physical efforts (Davis, 1993, p. 477).
- Perceived usefulness: the degree to which an individual believes that use of the target system could enhance the job performance (Davis, 1993, p. 477).
- Attitude toward use of target system: the degree to which an individual evaluates and associates the target system with his or her job (Davis, 1993, p. 476).
- Actual system use: a behavioral response, measured by the individual’s action in reality (Davis, 1989).
- Computer self-efficacy: the degree to which the individual would be able to control the power given by the computer (Lee, 2002).
- Subjective norms: the degree to which an individual complies with correct behavior favored by a group of people (Liker & Sindi, 1997).
- Grades: we used students’ end-of-class final grades as an outcome variable.
Research Questions
- To what degree is our hypothetical model fitted to our datasets collected on two occasions?
- To what degree do four dichotomous variables; gender (male vs. female), employment (full-time students vs. part-time students), timeliness (turning in homework on time vs. not turning in homework on time), and course type (psychology vs. engineering); impact the model hypothesized as a whole?
Relevance of the Study
Many resources have been invested in the adoption of WebCT. The intent of this study is to assist UCF in offering an alternative educational medium and tailor customized instruction for the purpose of better suiting the wide variety of UCF students with diverse backgrounds. Due to limited financial and human resources, it is incumbent upon the university to make learned decisions when implementing an information infrastructure project. A smart marketing approach can take full advantage of money spent on this initiative. The significance of this study may provide administrators with insights into student perception about the system employed and their individual traits, which may mediate the acceptance of such technology. Combining these components with the findings from previous WebCT-related studies (e.g., Dziuban & Moskal, 2001; Moskal & Dziuban, 2001) can assist decision makers in analyzing the influence of WebCT on the various UCF campuses, including physical and virtual ones.
Literature Review
Background
Eugene and Robert (2000) warned that annually almost half of new information systems projects fail. While the decision to adopt a questionable course management system occurs easily, adopting an information technology (IT) system based on subjective, superficial, and trivial considerations can be problematic and can increase total costs (Finkelstein & Scholz, 2000). To prevent such poor adoption, the best policies in technology adoption involve the development and implementation of a well-deigned IT investment plan. Understanding behavioral and cognitive patterns of end-users should be part of the strategic plan. Utility, such as the usefulness of systems based on product design, is the principle issue of users. As a result of becoming the principal concern of end users, usefulness has become a focus of many research studies (Heilman, Bowman, & Wright, 2000; Rubenzer, 1992; Spitzer, Kroenke & Williams, 1999).
Technology Acceptance Model
One of primary reasons the TAM was adapted from the literature is that the model involved merely four generic variables, which makes it more dynamic and more flexible for us to apply it to different contexts and to further the model (Venkatesh, 1998). Concerning the effect and power of the TAM in predicting user acceptance in the past two decades, a meta-analysis of the Technology Acceptance Model by Legris, Ingham, and Collerette (2003) reported three major findings:
- Varied settings were employed in terms of the target system and the TAM has been cross-examined and compared to other theories.
- Researchers adapted the TAM to a varying degree and they focused on different relationships among the variables.
- External variables used differed from one study to another.
- Only 40% variance of the Use variable was explained.
Legris, et al (2003) cited the target systems used in the TAM studies could be grouped into three categories: office automation tool, software development tool, and business application tool. Their review indicated that those TAM researchers either compared the TAM to the theory of reasoned action (TRA) or the theory of planned behavior (TPB) and some researchers added subjective norms constructs to their expanded TAM-based model, a variable examined in the TRA. With respect to the versions of the TAM, the twenty-two TAM-based studies reviewed by Legris et al. delineated a high proportion of congruent findings on each of the ten possible linear relationships among variables.
Regarding the external variables measured in the TAM studies, Legris, et al. observed that variables such as computer self-efficacy and subjective norms were added to the hypothetic model together with others, which allowed for a fuller picture of the model. Though the antecedents accounted for trivial increases on the explained variances of the outcome variable, the presence of those external variables afforded strategic plans that would result in greater likelihood of user acceptance. Furthermore, attitudes toward the system use and behavioral intention to the system use appeared to be either used interchangeably or overlooked in the literature, while some studies included both variables. Actual system use, the outcome variable in the TAM, was either measured, using frequency and duration of system use, or disregarded by the researchers.
Methodology
Design
This is a structural equation modeling study where causal paths, both directly and indirectly, and measurement errors and disturbances of both dependent and independent variables are simultaneously modeled, which is different from classic regression techniques. Specifically, this co-relational inquiry was made using path analysis. Two time occasions were involved: Time 1 and Time 2, with Time 2 as a replication.
Sample
In the present study, 469 out of 1,219 students from two large-sized Web-enhanced hybrid courses, Psychology and Engineering, participated. Specifically, 239 were from the entry-level Psychology course and 230 from the introductory Engineering class. Both classes were chosen because of the fundamental differences between the social science-oriented class (i.e., Psychology) and its hard science counterpart (i.e., Engineering).
Measures
Five instruments were adopted and adapted. The Usability Instrument (including Perceived Ease of Use Scales and Perceived Usefulness Scales) had 12 items, measured on a seven-point Likert scale. The Attitude Instrument comprised five items, measured on a sever-point bipolar (Semantics) scale. The System Use Instrument was composed of two distinct variables, both measured on a five-point ordinal scale. The Computer Self-Efficacy Instrument included five subscales with a total of 27 variables, measured on a seven-point Likert scale. The Subjective Norms Instrument contained three indicators, measured on a seven-point Likert scale. Prior to our data collection, all instruments were re-validated and internal consistency was secured on all but the System Use Instrument. Thus, two indicators from System Use Instrument, Frequency and Intensity remained for analysis as two separate outcome variables because of their significance to our study.
Data Collection
Two online questionnaires were administered throughout spring 2003, one in the 2nd week and the other in the 13th week, using ColdFusion (to integrate HTML and MS Access), JavaScript (to avoid missing data), HTML (to create Web surveys), CSS (to keep interface consistency), and MS Access (to house the data). To increase the response rate, in-class announcements, e-mail notices, WebCT’s Tip of the Day, and forum postings were used.
Data Analysis
We imported datasets from MS Access to MS Excel. After modification in Notepad and MS Word, the data were opened in SAS v. 8 for Windows for further analysis. We then analyzed the datasets one by one on a scale level, using such procedures as SAS FACTOR, SAS CORR, and SAS CALIS for T-test for independent variables and structural equation modeling as well as examining model fitting and model modification.
Results
Question 1: To what degree is our hypothetical model fitted to our datasets collected on two occasions?
By attempting to fit our original TAM and the expanded TAM to our Time 1 and Time 2 data, we found the expanded TAM on Time 2 was fitted to our data best. The results are represented in Table 1.
Table 1. Square Multiple Correlations on the TAM and the Expanded TAM by Period
Variables |
R-Square |
|||
TAM |
Expanded
TAM |
|||
| Time One | ||||
|
.0813 |
.0972 |
||
|
.4508 |
.4882 |
||
|
.3403 |
.3999 |
||
|
N/A |
.2142 |
||
|
.0250 |
.0250 |
||
| Time Two | ||||
|
.1289 |
.1289 |
||
|
.0289 |
.0550 |
||
|
.3862 |
.4348 |
||
|
.3405 |
.4283 |
||
|
N/A |
.4748 |
||
|
.0414 |
.0436 |
||
Although we successfully replicated and expanded the TAM on both time occasions, the expanded TAM on Time 2 was kept for further analysis because, the more variance of each variable explained, the better the model.
Question 2: To what degree do four dichotomous variables; gender (male vs. female), works (full-time students vs. part-time students), timeliness (turning in homework on time vs. not turning in homework on time), and course type (psychology vs. engineering); impact the model hypothesized as a whole?
To answer Question 2, a T-Test for independent variables was conducted. One of the four dichotomous variables, group type, was found significantly impacting the model as a whole (see Figure 2).
Figure 2. Path Diagram of the Expanded TAM with Course Type on Time 2.
We decided to further pursue this issue by conducting a covariance structure analysis on a scale level. We compared both class by randomly selecting the same amount of student number (n=230) on each of the instruments. Interestingly enough, we found the two classes responded to two instruments: Computer Self-Efficacy and Subjective Norms in a dissimilar way (see Table 2).
Table 2. Fit Results for Rival Models by Student Group [a]
Fit
Index |
Value |
| Perceived Usefulness | |
|
0.9342 |
|
0.0485 |
|
174.9573 |
|
60 |
|
<.0001 |
| Perceived Ease of Use |
|
|
0.9569 |
|
0.0288 |
|
165.9631 |
|
60 |
|
<.0001 |
| Attitude |
|
|
0.9976 |
|
0.0255 |
|
44.0726 |
|
4040 |
|
0.3034 |
| Self-Efficacy |
|
|
0.6153 |
|
0.0988 |
|
6799.5032 |
|
1404 |
|
<.0001 |
| Subjective Norms |
|
|
0.8235 |
|
0.0670 |
|
67.4634 |
|
24 |
|
<.0001 |
[a] Covariance Structure Analysis: Maximum Likelihood Estimation
By examining the fit indices in accordance with SEM traditions, we found each CFI value of Computer Self-Efficacy Instrument and Subjective Norms Instrument did not exceed .90, a customary cut-off point. Neither did their SRMR values (> .05). Having said that, the dissimilar manner in which both classes responded to the two scales is confirmed.
Conclusions and Implications
Conclusions
This structural equation modeling inquiry was intended to replicate the Technology Acceptance Model (TAM) and to further expand the TAM using undergraduate students in two large sized (n= 469) WebCT-enhanced hybrid courses at the University of Central Florida (UCF), U.S.A. Causality was examined among four core latent factors; perceived ease of use, perceived usefulness, attitude toward WebCT, and actual use of WebCT; and two outside latent variables, computer self-efficacy and subjective norms. Additionally, the students' end-of-course grade was measured as a third outcome variable together with Frequency and Intensity. During the spring 2003 semester, this co-relational study was conducted to answer the two questions, using six instruments: (1) the Usability Instrument, (2) the Attitude Instrument, (3) the Computer Self-efficacy Instrument, (4) the Subjective Norms Instrument, (5) the System Use Instrument, and (6) the Student Demographic Instrument. Data were collected on two occasions. The Time 2 model was used to replicate Time 1 model.
Data were analyzed using path analysis and t-test for independent variables in SAS v8 for Windows. Findings of path analysis indicated that the Technology Acceptance Model was successfully tested, which suggested that both perceived ease of use and perceived usefulness are determinants of students’ attitude toward WebCT, which, in turn, determined their WebCT use. Results also suggested that student attitudes toward WebCT predicted their end-of-course grade. Analysis of t-test findings indicated that none of four dichotomous variables, other than course type, can impact their perception, attitude, or use of WebCT in this U.S. southeastern metropolitan university.
Implications
Based on the findings, the students' perception of WebCT with regard to ease of use influenced their attitude toward WebCT instruction. Their attitude, in turn, impacted their WebCT use. This suggests that a majority of the students in the two participating classes perceived that WebCT was easy to use and useful to their coursework. Most of the students had a more favorable attitude toward WebCT, indicating that they liked the online learning system that was incorporated within the course they were taking. Most of them may have been competent and comfortable with WebCT related skills, which made these goal driven students login and logout of the course quickly. It may also have been that they did not feel WebCT was sufficiently playful.
Having said that, we may be able to predict a success in WebCT adoption in both psychology and engineering courses once they become fully Web-based classes.
One of the values of the expanded Technology Acceptance Model lies in the exploration of external variables, such as computer self efficacy and subjective norms. We went beyond a perception-attitude-behavior relationship and examined how computer self- efficacy and subjective norms influenced students' perception in terms of ease of use and usefulness of WebCT. In acknowledging the large measurement errors, we suggested that more external variables be added to the model for further examination. Second, in conjunction with research from faculty and school administrators’ viewpoints, this inquiry with a student perspective may reinforce and strengthen the justification of any learned decisions the policy makers might subsequently consider.
Limitations
Due to the scope of the present study, we did not include time and effort for interviews with participants. The quantitative study may be better enhanced by adopting a mixed method design to present a fuller picture in regard to end-user acceptance of a course management system. Due to the nature of the case study, extra care should be taken when applying the findings to other similar settings.
Further Recommendations
Additional research endeavors should be devoted to measurement of the System Use, campus wide longitudinal data collection, and temporal cross-lagged effects of variables, in the hope to further analyze students’ acceptance or rejection of WebCT in this type of courses at UCF.
References
Cornell, R. A. (2001). Scaling the cyberheights: How far will we go? In L. R. Vandervert, L. V. Shavinina, & R. A. Cornell (Eds.), Cybereducation: The future of long distance learning (pp. 3-11). Larchmont, NY: Mary Ann Liebert.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319-340.
Davis, F. D. (1993). User acceptance of information technology: System characteristics, user perceptions, and behavioral impacts. International Journal of Man Machine Studies, 38, 475-487.
Dziuban, C., & Moskal, P. (2001). Evaluating distributed learning in metropolitan universities. Metropolitan Universities, 12(1), 41-49.
Eugene, S., & Robert, D. F. (2000). An examination of organizational factors influencing new product success in internal and alliance-based processes. Journal of Marketing, 64(1), 31-49.
Finkelstein, M., & Scholz, B. W. (2000). What do we know about information technology and the cost of collegiate teaching and learning? In M. J. Finkelstein, C. Frances, F. I. Jewett, & B. W. Scholz (Eds.), Dollars, distance, and online education: The new economics of college teaching and learning (pp. 3-34). Phoenix, AZ: Oryx Press.
Heilman, C. M., Bowman, D., & Wright, G. P. (2000). The evolution of brand references and choice behaviors of consumers new to a market. Journal of Marketing Research, 37(2), 39-155.
Lee, C.-Y. (2002). The impact of self-efficacy and task value on satisfaction and performance in a Web-based course. Unpublished doctoral dissertation, University of Central Florida, Orlando.
Legris, P., Ingham, J., & Collerette, P. (2003). Why do people use information technology? A critical review of the technology acceptance model. Information & Management, 40, 191-204.
Liker, J. k., & Sindi, A. A. (1997). User acceptance of expert systems: A test of the theory of reasoned action. Journal of Engineering and Technology Management, 14, 147-173.
Moskal, P. D., & Dziuban, C. D. (2001). Present and future directions for assessing cybereducation: The changing research paradigm. In L. R. Vandervert, L. V. Shavinina, & R. A. Cornell (Eds.), Cybereducation: The future of long distance learning (pp. 157-184). Larchmont, NY: Many Ann Liebert.
Rubenzer, S. (1992). A comparison of traditional and computer-generated psychological reports in an adolescent inpatient setting. Journal of Clinical Psychology 1992, 48(6), 817-827.
Spitzer, R. L., Kroenke, K., & Williams, J. B. W. (1999). Validation and utility of a selfreport version of PRIME-MD: The PHQ Primary Care Study. Journal of the American Medical Association, 282(18), 1737-1744.
Venkatesh, V. (1998). User acceptance of information technology: A unified view. Unpublished doctoral dissertation, University of Minnesota, Minneapolis.
TCC Online Conferences
Kapi`olani Community College
University of Hawai`i
Honolulu, HI