I Love/Hate the U.S. News Online Program Rankings: The Tightrope Between Innovation and Conservatism

By Richard Garrett
Vice President & Principal Analyst, Online Higher Education

With surprisingly little media coverage, the second U.S. News & World Report rankings of online programs recently appeared. Last year, the first rankings declined to actually rank schools or programs overall, instead ranking only by sub-category, such as faculty training or student services. But this time, schools are ranked outright for Bachelor’s programs, plus rankings for Master’s in Business, Education, Engineering, IT and Nursing. So has the mystery been solved, and we now know which schools have the best online programs?

Distinguishing Great From Good? 

In any industry, rankings are tantalizing. In theory, rankings objectively distinguish the great from the good, help buyers make smart decisions and suppliers stand out from the crowd. In any young industry, like online higher education, the prospect of ranking embodies the longing for quantitative measurement of value so hard to grasp amid the rhetoric and anecdote of immaturity. The big dilemma for U.S. News is whether online higher education is ready to be ranked. Is quality sufficiently defined and visible to be self-reported by schools against a survey instrument?

In my view, while U.S. News has been nothing if not diligent and nuanced in its methodology, this year’s rankings leave as many questions as answers. The rankings cover “Faculty Credentials & Training”, “Student Services & Technology” and “Student Engagement”, plus “Admissions Selectivity” for graduate programs. Beneath each heading is a host of sub-criteria, and U.S. News has made a valiant attempt to include not just input metrics but process and outcomes, too. It’s great to see schools reporting completion rates for specific online programs.

Random Rather Than Intuitive

I think that the initial reaction of many people is that the results feel odd or random rather than intuitive. This may reflect the very immaturity of online higher education, where conventional proxies for quality or prestige may not apply. The problem for U.S. News is that it is perhaps more difficult than it should be to understand why a school ranks in the top 5. The methodology is laid out clearly and exhaustively, but the absence of a published combined score obscures the distance between one school and another.

The graduate rankings make assumptions about the role of selectivity and tradition in an online context. There is a tension between policy and commercial interest in using online learning to scale up high quality, low cost programming and the U.S. News stance that small class sizes, admission test scores, PhD and tenured faculty and cases of tradition-bound specialized accreditation strengthen one online program over another. Granted, U.S. News does not use selectivity in ranking online Bachelor’s programs, and there is no doubt a place for selective online programs, but the strong selectivity emphasis does feel like convention curtailing innovation. For graduate programs, 25% of total score derives from peer reputation, which feels far from robust given the novelty of online programming at most schools.

Compared to Year 1, the second iteration of the rankings relies less on school self-reporting of practices and policies, the efficacy or implementation of which may vary widely. But even in Year 2, I calculate that about 50% of Bachelor’s points relate to self-reported practices/policies or rather conservative faculty arrangements (PhD, tenure); and at graduate level the ratio is just as high, and higher still if peer reputation is included.

The Right Kind of Transparency 

On balance, U.S. News should be commended for trying to make sense of the messy, complex world of online, but online appears far from ready. The sector will benefit from the transparency ranking ushers in, but risks suffering from the wrong kind of transparency and stubborn obscurity on other kinds of value. The rankings need to be simplified, in order to focus on the most objective measures of value. Would time to completion, graduation rate, student debt and a third party student assessment (CLA?), all controlled for student demographics and incoming credit, fit the bill? That way, online quality would be defined by performance that all players could compete on, whether large or small, tenured or otherwise, selective or open admission.

Be On The Lookout

In 2013, Eduventures will publish a report on online higher education and accountability, an attempt to advance the conversation on how to distinguish the value of online programs in general, and one from another, from both a consumer and policymaker perspective.