Below is a collection of recent papers on online learning in alphabetical order (references to earlier studies can be found in these papers, in particular, in the U.S. Department of Education study). The main focus of this list is the pedagogial and education effects of online learning, in particular but not only, the comparison of semester-long college-level courses of different formats, as well as the faculty perspective on online learning. Abstracts or short extracts summarizing the main message/focus of the papers are included.
Suggestions for additions of thorough recent studies, important discussion pieces, or key earlier studies (not quoted in studies listed here) for inclusion on this list are welcome. Email firstname.lastname@example.org with comments or suggestions.
Allen, I.E., Seaman, J., Lederman, D., Jaschik, S., Conflicted: Faculty and Online Education, 2012.
[Please read the acknowledgement/disclaimer and check the authors' affiliations.]
Faculty report being more pessimistic than optimistic about online learning. Academic technology administrators, on the other hand, are extremely optimistic about the growth of online learning, with over 80 percent reporting that they view it with “more excitement than fear.” [...] Nearly two-thirds [of professors] say they believe that the learning outcomes for an online course are inferior or somewhat inferior to those for a comparable face-to-face course. [...] Faculty members with a greater exposure to online education have a less pessimistic view than their peers. [...] About one-third of faculty members report they think that their institution is pushing too much instruction online, compared to fewer than 10 percent of administrators. Over all, fewer than one half of all professors believe that their institution has good tools in place to assess the quality of in-person instruction, while only one-quarter say the institution has good tools for assessing online instruction.
APLU - Sloan National Commission on Online Learning, Volume II: The Paradox of Faculty Voices: Views and Experiences with Online Learning, Results of a National Faculty Survey, Part of the Online Education Benchmarking Study, 2009.
This survey was conducted in fall 2008 and winter 2009. The findings contained in this volume are based on responses from more than 10,700 faculty from 69 colleges and universities across the country. [... F]aculty are not uniform in their opinions toward online learning. Faculty with experience developing or teaching online courses have a much more positive view towards online instruction than those without such experience. Faculty with no online experience remain relatively negative about online learning outcomes. [...] Driving faculty concerns is the pervasive belief that teaching or developing an online course requires more time and effort than for a comparable face-to-face offering. Faculty rate this issue as the most important barrier to teaching and developing online programs. Faculty also report that they have serious reservations about the quality of online learning outcomes, and they believe that their institutions are below average in providing support and incentives. Approximately one-third of all faculty have taught an online course, with around one-quarter currently teaching online. When asked why they teach online, faculty consistently provide student centered reasons. The survey results show that, even with their reservations about online learning, a majority of faculty members have recommended online courses to students, a rate that jumps to well over 80 percent among faculty with experience developing or teaching an online course. The views of the faculty suggest that significant challenges must be resolved before online learning is universally accepted across the academy. However, the paradoxes evidenced by the survey results also suggest considerable opportunity for campus leaders to engage the faculty in constructive dialogue about the quality, support, and overall role of online at their respective institutions.
Webinar: Selected Faculty Survey Result Slides.
Bowen, W.M., Chingos, M.M., Lack, K.A., Nygren, T.I., Interactive Learning Online at Public Universities: Evidence from Randomized Trials, Ithaka S+R, 2011.
Online learning is quickly gaining in importance in U.S. higher education, but little rigorous evidence exists as to its effect on student learning outcomes. In "Interactive Learning Online at Public Universities: Evidence from Randomized Trials," we measure the effect on learning outcomes of a prototypical interactive learning online (ILO) statistics course by randomly assigning students on six public university campuses to take the course in a hybrid format (with machine-guided instruction accompanied by one hour of face-to-face instruction each week) or a traditional format (as it is usually offered by their campus, typically with 3-4 hours of face-to-face instruction each week).
We find that learning outcomes are essentially the same—that students in the hybrid format "pay no price” for this mode of instruction in terms of pass rates, final exam scores, and performance on a standardized assessment of statistical literacy. These zero-difference coefficients are precisely estimated. We also conduct speculative cost simulations and find that adopting hybrid models of instruction in large introductory courses have the potential to significantly reduce instructor compensation costs in the long run.
Euzent, P.J., Martin, T.L., Moskal, P., Moskal, P., Teaching Principles to the Masses: Assessing Student Performance in Lecture Capture vs. Face-to-Face Course Delivery, SSRN, 2011.
This study examined student performance and student perceptions in two large sections (N>300) of an introductory Economics course. One section employed traditional face-to-face instruction, and the other employed LC over the Internet. This study took place over two consecutive semesters. Students selected their course section (delivery format) during course registration. The instructional methods, exams, and instructor were the same for each section over both semesters. Students who agreed to participate allowed the authors to use their exam and homework scores, final grade, and some demographic data to compare student performance across the two delivery approaches. At the end of each semester, participating students also were asked to complete an online survey about their perceptions of their course section. The results indicated that there were no significant differences in student performance across the two delivery formats. Our results did show a higher withdrawal rate in the LC sections compared to the face-to-face sections (5.1% to 1.9%). LC courses require more discipline because students must complete coursework on their own, and it is likely that freshman and sophomores, in particular, may find this medium more difficult. Finally, student perceptions of LC were quite positive. Seventy-two percent perceived that they had more control over their learning than in a traditional face-to-face class. Forty-three percent also responded that they felt LC enhanced their performance in the course, while only 28% thought that it did not. Eighty percent indicated that LC was as good as or better than the traditional large lecture class experience, and 73% reported that they would choose to take another LC course. The flexibility and convenience of LC were what students liked the most about taking the course this way. Thus, it does appear that, indeed, we are “doing no harm” with LC course delivery, and that many students prefer this medium.
Figlio, D. N., Rush, M., Yin, L., Is it Live or is it Internet? Experimental Estimates of the Effects of Online Instruction on Student Learning, NBER Working Paper 16089, June 2010.
This paper presents the first experimental evidence on the effects of live versus internet media of instruction. Students in a large introductory microeconomics course at a major research university were randomly assigned to live lectures versus watching these same lectures in an internet setting, where all other factors (e.g., instruction, supplemental materials) were the same. Counter to the conclusions drawn by a recent U.S. Department of Education meta-analysis of non-experimental analyses of internet instruction in higher education, we find modest evidence that live-only instruction dominates internet instruction. These results are particularly strong for Hispanic students, male students, and lower-achieving students. We also provide suggestions for future experimentation in other settings.
"In summary, none of the studies cited in the widely-publicized meta-analysis released by the U.S. Department of Education included randomly-assigned students taking a full-term course, with live versus online delivery mechanisms, in settings that could be directly compared (i.e.,similar instructional materials delivered by the same instructor.) The evidence base on the relative benefits of live versus online education is therefore tenuous at best. From a public and university policy standpoint, the current state of this research is dismaying: More students are being exposed to internet classes yet there is no satisfactory research demonstrating whether such changes help, hinder, or have no effect on student learning. This paper aims to fill this important gap by reporting on an experiment in which students were randomly assigned to either an online or a live section of a course taught by one instructor and for which the ancillaries for the class, such as the web page, problem sets and TA support, as well as the exams, were identical between the sections.The only difference between these sections is the method of delivery of the lectures: Some students viewed the lectures live, as would be the case in traditional classes, while other students viewed the lectures on the internet. Thus we are able to determine how online delivery of lectures compares with live delivery." (p 4-5)
Gose, B., Learning the Art of Virtual Instruction: Traditional colleges offer training, along with incentives, for wary professors, The Chronicle, 2010.
Everywhere Carol Maxson looked, colleges were delivering courses online. Everywhere, it seemed, except at Trevecca Nazarene University, where Ms. Maxson is associate provost and dean of academic affairs. Trevecca's faculty members were not eager to teach online, and many doubted that such courses could create the same sort of community found in classes on the Nashville campus. Ms. Maxson knew she had to do something. "The train had passed us by, and I was hoping that we could grab on to the caboose," she says. So last fall, Ms. Maxson signed up for "Introduction to Online Teaching," a six-week virtual course offered by Nazarene Bible College, in Colorado Springs. Since then, about 30 Trevecca faculty members have taken the course, galvanizing a grass-roots movement at Trevecca toward online education. In October, Trevecca offered its first online course, for its first online-degree programa master's in organizational leadership. While some for-profit institutions, like the University of Phoenix, have a finely honed process for training professors to teach online, at nonprofit institutions skeptical faculty members drive much of the academic decision making, and the process isn't nearly so neat.
Jenkins, R., Online Classes and College Completion, The Chronicle, 2012.
Perhaps what the most at-risk students really need, instead of being herded into online courses, is the "personal touch." Maybe they need more face-to-face interaction with instructors and other students; more conferences in their professors' offices; more private, one-on-one tutoring sessions; more hanging out with their peers in the student center between classes.
Maybe allowing those students to sit at home, alone in front of their computers, with little in the way of emotional support—not to mention, in many cases, educational support—is actually a bad idea. Maybe instead of doing everything we can to encourage them to take as many of classes as possible online, we should be welcoming them to our campuses and into our classrooms.
If future funds for higher education really do follow completion rates, rather than just enrollment figures, then this approach could even end up being more cost-effective for colleges in the long run.
Milliron, M.D., Online Education vs. Traditional Learning: Time to End the Family Feud, The Chronicle, October 2011.
Online learning tools and techniques including fully online courses, blended learning, mobile learning, game-based learning, and social networking are some of the newest and rowdiest children in the family of higher-education resources. They hold the promise of expanding, improving, and deepening learning for our students. A quick exploration of Carnegie Mellon's Open Learning Initiative, or the Monterey Institute for Technology and Education's National Repository of Online Courses, or Florida Virtual School's Conspiracy Code(a history course in a game) gives you sense of what's possible and what's coming.
Online Learning 2010: Virtual Education Goes Mainstream, The Chronicle, October 2011.
Will the classroom of the future be a virtual one, a face-to-face one, or a hybrid of the two? Yes to all of those.
Online Learning by the Numbers The Chronicle, 2010.
Ragan, L., 10 Principles of Effective Online Teaching: Best Practices in Distance Education, Faculty Focus, Special Report, 2012.
In the traditional college classroom today, faculty and students arrive with a certain set of expectations, shaped largely by past experiences. And although students may need the occasional(or perhaps frequent) reminder of what’s required of them, there’s usually something very familiar about the experience for both faculty and students alike.
In the online classroom, an entirely new set of variables enters the equation. It’s a little like trying to drive in a foreign country. You know how to drive, just like you know how to teach, but it sure is hard to get the hang of driving on the left side of the road, you’re not quite sure how far a kilometer is, and darn it if those road signs aren’t all in Japanese.
Roscoe, D.D., Comparing Student Outcomes in Blended and Face-to-Face Courses, Journal of Political Science Education, 8/1, 1-19, 2012.
This article reports on a study of student outcomes in a pair of matched courses, one taught face-to-face and one taught in a blended format, in which students completed most of the work online but met several times face-to-face. Learning objectives, course content, and pedagogical approaches were identical but the mode of instruction was different. The data suggest academic performance was not influenced by the mode of instruction. Additionally, the data show few differences between the sections in terms of attitudinal outcomes, although students in the blended section reported a lower sense of community. However, one notable outcome was the tendency of students to drop out of the online activities after having a face-to-face session. Lessons for the design of blended courses are discussed.
Rose, R., 6 Tips for the Successful Online Teachers, the journal: Transforming Education through Technology, 8/1, 06/18/12.
While online teaching offers many rewards for instructors, it takes a special set of skills and attitudes to excel at it. And these are emphatically not the same skills and attitudes that make an exceptional classroom teacher. Here's the mindset it takes to be a successful online teacher.
Smith Jaggars, S., Bailey, T., Effectiveness of Fully Online Courses for College Students: Response to a Department of Education Meta-Analysis, Community College Research Center, July 2010.
Proponents of postsecondary online education were recently buoyed by a meta-analysis sponsored by the U.S. Department of Education suggesting that, in many cases, student learning outcomes in online courses are superior to those in traditional face-to-face courses. This finding does not hold, however, for the studies included in the meta-analysis that pertain to fully online, semester-length college courses; among these studies, there is no trend in favor of the online course mode. What is more, these studies consider courses that were taken by relatively well-prepared university students, so their results may not generalize to traditionally underserved populations. Therefore, while advocates argue that online learning is a promising means to increase access to college and to improve student progression through higher education programs, the Department of Education report does not present evidence that fully online delivery produces superior learning outcomes for typical college courses, particularly among low-income and academically underprepared students. Indeed some evidence beyond the meta-analysis suggests that, without additional supports, online learning may even undercut progression among low-income and academically underprepared students.
In keeping with the notion of improved student access as a strongly emphasized rationale for online learning, we first narrowed our focus to the 28 studies included in the Department of Education meta-analysis that compared fully online courses to face-to-face courses. [...] When considering only those studies conducted with undergraduate or graduate students in semester-long online courses, the set of 28 studies is reduced to 7. [...] Lack of consistent differences in outcomes between online and face-to-face. [...] More than half of the studies targeted courses that explicitly taught technology or electronic communication concepts, perhaps because these topics were thought to be particularly well suited for online teaching and learning. [...] All seven studies were conducted at mid-sized or large universities, with five rated as 'selective' or 'highly selective' by U.S. News and World Report, and all seemed to involve relatively well-prepared students. [...] Most institutions place a strong value on increasing access for underserved students. [...]Does online learning meet these goals? For well-prepared and motivated students, perhaps it does [...] For low-income and underprepared students, however, an expansion of online education may not substantially improve access and may undercut academic success and progression through school." (p 3, 4, 7, 8, 11)
U.S. Department of Education, Office of Planning, Evaluation, and Policy Development, Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online LearningStudies, Washington, D.C., (revised version) 2010.
A systematic search of the research literature from 1996 through July 2008 identified more than a thousand empirical studies of online learning. Analysts screened these studies to find those that (a) contrasted an online to a face-to-face condition, (b) measured student learning outcomes, (c) used a rigorous research design, and (d) provided adequate information to calculate an effect size. As a result of this screening, 50 independent effects were identified that could be subjected to meta-analysis. The meta-analysis found that, on average, students in online learning conditions performed modestly better than those receiving face-to-face instruction. The difference between student outcomes for online and face-to-face classes—measured as the difference between treatment and control means, divided by the pooled standard deviation—was larger in those studies contrasting conditions that blended elements of online and face-to-face instruction with conditions taught entirely face-to-face. Analysts noted that these blended conditions often included additional learning time and instructional elements not received by students in control conditions. This finding suggests that the positive effects associated with blended learning should not be attributed to the media, per se. An unexpected finding was the small number of rigorous published studies contrasting online and face-to-face learning conditions for K–12 students. In light of this small corpus, caution is required in generalizing to the K–12 population because the results are derived for the most part from studies in other settings (e.g., medical training, higher education).
Ward, D., ProfHacker: Pushing through the Perils of Teaching Online, The Chronicle of Higher Education, 08/27/2012.
As online and hybrid classes grow in popularity, more and more teachers are being asked to transfer class material to a digital format. That is easier for some, perhaps impossible for others. As Jason has written in his series on teaching online for the first time, an online course comes with its share of surprises, frustrations, and unexpected roadblocks. (None of that ever happens with classroom teaching, right?) In this post, I take a longer view, reflecting on some of the mistakes I made in my first venture online.
Xu, D., Smith Jaggars, S., The Effectiveness of Distance Education Across Virginia’s Community Colleges: Evidence From Introductory College-Level Math and English Courses, Educational Evaluation and Policy Analysis, 2011.
Although online learning is rapidly expanding in the community college setting, there is little evidence regarding its effectiveness among community college students. In the current study, the authors used a statewide administrative data set to estimate the effects of taking one’s first college level math or English course online rather than face to face, in terms of both course retention and course performance. Several empirical strategies were used to minimize the effects of student self selection, including multilevel propensity score. The findings indicate a robust negative impact of online course taking for both subjects. Furthermore, by comparing the results of two matching methods, the authors conclude that within-school matching on the basis of a multilevel model addresses concerns regarding selection issues more effectively than does traditional propensity score matching across schools.