By Senior Analyst Brian Fleming
PayScale.com has taken higher ed by storm with its now yearly cadence of “College ROI” reports, which offer compelling, controversial, and without a doubt un-conventional ways to think about outcomes. PayScale has received plenty of positive nods from employers and policy makers including President Obama. Newly released 2014 data has sparked debate about its proper use and impact among a critical mass of higher education leaders who are willing and ready to pick apart its every conclusion. All the while, the public at large seems to be embracing it—evidenced by an impressive 4.6 million “Likes” on Facebook, 35,000+ hits on its site a day, and general sense that PayScale (or things like it) will only grow in popularity for years to come.
What is PayScale?
PayScale is basically a crowdsourced data aggregator that provides ranked and sortable lists by “ROI” metrics, mainly how much money graduates go on to make after college. It is segmented by variables such as region, school type, and ROI by major, even some of Princeton Review’s more subjective categories such as “Party” and “Sober Schools.” Prospective students, parents, and policy makers use this as a tool to determine what schools are actually producing students that go on to high paying jobs and jobs of “high meaning” (their words, not ours). Unlike the often debated but massively popular U.S. News rankings, PayScale is much less obfuscated by prestige and prominence and more, they claim, based on hard data. Georgia Tech, for instance, is ranked #6 based on costs to 20 year net ROI whereas Williams College is ranked #31.
So what makes PayScale so controversial?
PayScale does what higher ed has arguably still yet to do in any comprehensive way: quench the public’s thirst for aggregated and quantifiable outcomes data linked to earning potential. While typically assumed to be a key driving value of a college degree, this kind of data is either difficult for most institutions to produce, for many legitimate reasons, or deemed too subjective to have any real value. While many people would argue that something like PayScale has some merit, not everyone is impressed. Common critiques include:
- PayScale uses student reported data, which is unreliable and potentially misleading. This is salient critique. To many people, it’s not the idea of ranking by ROI, but the crowdsourcing element of all this that is so troubling. It feels, some say, like the inmates running the asylum. For instance, what is to stop people over-reporting salary metrics or simply reporting false or skewed data? Of course, in theory, ranking systems like U.S. News have long suffered a similar critique based on institutions potentially reporting dubious or misleading data, though some argue that external rankings systems are more reliable than something like PayScale.
- The value of a college experience is hard to quantify, or simply un-quantifiable. A familiar objection, which extends far beyond that of PayScale, is that any ROI assessment deceivingly assigns ultimate value to a college degree using measures foreign to the fundamental mission of higher education. Andrew Delbanco, professor of history at Columbia and frequent critic of these kind of things, asks, “Should returns in dollars be the only measure of educational value? What does that say about the traditional mission of college to educate young people for engaged citizenship, and to provide opportunities for self-fulfillment in ways that do not necessarily line up with income and status?” In other words, college is about more than just workforce development but personal mentoring, study abroad trips, and a bunch of other coming of age “self-fulfilling” experiences that can hardly be reduced down to something as pedestrian as ROI.
- PayScale’s methodology is approximate at best. This is where the arguments against PayScale either fall into the abyss of endless academic debate (rarely popular with the general public) or or simply add energy to the ethereal debate about the value of higher education in the 21st century. Few sensible minds argue that things like PayScale have no place, but many suggest it should exist alongside other ROI measures that seek, however imperfectly, to achieve more precise measures. What those measures are, and how to quantify them in distributive and compelling ways, is curiously muted in this debate, and we have no reason to expect much consensus in the near-term. PayScale, of course, acknowledges much of this already and has made admirable attempts to justify its methodology, remind critical masses of its limitations, and clearly spell out what it aims and does not aim to accomplish in its rankings.
What about adult learners?
Plenty of people have endeavored to comment on the impact of PayScale. We only wish to extend this discussion in relation to a concurrent debate about the place of non-traditional, online, and adult-serving forms of higher education in these kind of ROI/outcomes-type assessments, which PayScale seems to obscure. Our thoughts are, if PayScale is to keep pace with the rising numbers of adult learners, projected to make up 42% of all higher education enrollments by 2020, it will need to extend its purview into the breadth of the many types of higher education out there today, taking into account some of the following:
- The “value” of higher education for non-traditional learners. Such heavy focus on categories such as “research universities,” “private universities,” and “Ivy League Schools” and even “Party” and “Sober Schools,” alone is evidence enough that PayScale assigns its primary value to very traditional perceptions of higher education. PayScale, for instance, does not do much to profile the work of online providers, or for-profit education. PayScale also lacks clear weighting to metrics that measure ROI based on outcomes that resonate with adult learners, giving significant weight to things like whether one’s major “changes the world” or “makes the world a better place,” which, while compelling, might not resonate as clearly with the pragmatism of adult education.
- Diverse school/employment scenarios. A persistent challenge for any calculation of outcomes is delayed graduation, e.g. students working part time through school and as a result, slowing down, stopping out, and taking, on average, upwards of 7 years to finish their degree. This is a common scenario for adult learners which PayScale does not fully recognize. In seeking to account for delayed graduation, PayScale weights forgone earnings based on time spent in school without including potential earnings from part-time, or even full-time, employment. This assumption alone seems to suggest that “real employment” comes only after they graduate, a realty that may not reflect the circumstances of adult learners seeking to advance their careers, transition into other careers, or seeking a bachelor’s degree for any number of professional development purposes.
- The benefits of non-traditional delivery modes. Part of a now raging debate about the future of higher education are a groundswell of innovative delivery models, namely in online and blended education, which promise to, if done right, speed up time to completion, reduce costs, and bolster employment outcomes. Part of this is also the rising popularity of competency-based education, which promises to ensure enhanced workforce alignment with faster responsiveness to professional demand. Assuming these innovations move further into the mainstream of higher education, tools like PayScale will need to expand our notion of ROI to include metrics related to very unconventional measures of the impact of higher education.