Education, Testing and the Problem with PISA


February 22, 2012 by Jackie Modeste

In her March 2012 article in the Harvard Business Review, “Rethinking School” Stacey Childress makes several observations and suggestions for improving the US k-12 system (and beyond as a matter of consequence). Childress refers to a comprehensive study by Eric Hanushek and Ludger Woessmann, The Role of Cognitive Skills in Economic Development that compares a country’s GDP with academic test scores. In a nutshell, the study found that improved test scores led to increased GDP. The study uses the period 1960 – 2000 and makes the claim that had the US closed the “education achievement gap with better-performing nations, GDP in 2010 could have been [8 – 12 percentage points] higher.” There is plenty of room for delving into the study and asking questions that consider the impact of history on the findings, such as: how did the study account for newly integrated schools in the US, the Civil Rights movement and Viet Nam? Did the rise in dual income households impact test scores? We could also ask questions about the veracity of testing in general; and specifically, the implications of using an international test such as PISA to not only level disparities inherent in collecting data but on collecting cross-cultural aggregate cognitive ability data! But for now and for the sake of initiating a discussion on the exhaustive and thoughtful study that was actually funded and conducted, I will accept the assumptions in the Hanushek and Woessmann study.


  1. The use of technology in schools – Childress notes that technology is now being incorporated in schools in really innovative ways that “blend the best of teacher and computer-delivered instruction” for a more “personalized” learning experience. As she notes, the successful integration of technology in one low-performing district in California led to the students eventually performing at the same level as their peers in an affluent area.
  2. Teacher quality as the “biggest factor in boosting that student’s performance.”
  3. Test scores as the primary indicator of student performance.


  1. Technology is a great way to compound the learning experience and make it more effective — to an extent. First, technology is costly and states strapped with budgetary concerns are at a loss to implement such technology broadly. Alliances with the private sector are fraught with challenges, one of which is achieving the “right” balance between educational and profit-driven interests in advocating for students. Second, while “adaptive software” may personalize the leaning experience for students and so approximate the teacher-student relationship, the lack of human interaction remains problematic. Humans are necessary and should have a primary actual — not virtual — leadership role in educating students. Third, when students in the “low-performing” group catch up to the scores of students in the affluent group, how far will students in the affluent group have progressed? We are naive in assuming their lives will remain stagnant during the interim. Since they already have the “good test scores” they can focus on expanding their horizons in ways far beyond areas measured in assessment data.
  2. Teacher quality – there really needs to be an effective way of accessing teacher effectiveness without castigating teachers and demeaning the profession.
  3. Test scores as an indicator of student performance — there should be a holistic, more personalized and differentiated approach to assessing student performance. Too many kinds of intelligence are ignored in favor of those areas experts measure quantitatively, in this case: science, math, reading.

Back to the Hanushek & Woessmann study…

The authors’ focus on cognitive ability is a strategic move to shift discussions on improving student performance from an increase in resources and “years in school” to the “best policies to improve skills.” (658) Aligning cognitive ability with GDP is sure to attract the associated financial investments in support of educational initiatives geared towards improving skills. As Hanushek and Woessmann note, research “strongly suggests that getting the substantial improvements in the quality of schools that are necessary requires structural changes in schooling institutions.” (659). The authors offer three suggestions: (1) strong accountability systems that accurately measure student performance; (2) local autonomy that allows schools to make appropriate educational choices; and (3) choice and competition in schools so parents can enter into determining the incentives that schools face.

I “get” the idea of changing the direction of the discussion by appealing to the desire of Americans to raise GDP, especially in these recessionary times. My concern is that in aligning cognitive ability with GDP, we narrow our focus to the three areas tested by the researchers — science, math and reading — to the exclusion of other areas where cognitive ability can be demonstrated. Areas of culture that require cognitive ability include: language training, music and visual arts; sports; combination skills such as those required to build a bridge, boat or house, or identify medicinal herbs. Creativity, resilience and social adaptation require cognitive ability as well but these would not be captured in a standardized test in science, math or reading. The narrow focus on measurements emanating from developed Western nations misses a lot that’s happening in the rest of the world and so risks leaving a vast amount of cognitive ability untapped and so a vast amount of GDP unrealized… think Jeremy Lin.

In terms of job creation, business leaders consistently identify skills beyond the scope of science, math and reading as crucial for success in our global economy and recognize the value of humans to guide technology. In “Skills that will Remain in Demand in a Computer-Rich World” Leslie Brokaw refers to a study conducted by Erik Brynjolfsson and Andrew McAfee that cites six skills that computers can’t do. Haushek and Woessmann’s study identifying cognitive ability is just fine but it’s acute focus on ability obscures practical application. My concern here is that in our blind pursuit of increased cognitive ability as determined through science, math and reading; we train a cadre of doers and not thinkers and create a society that lacks the ability to interact, engage, and innovate. A thoughtless society is ripe for dictatorship. The ability to think critically is crucial to democracy.

Despite the nation’s lackluster scores on PISA, the US continues to champion test scores and international comparisons as valid tools for measuring student performance. The Brown Center Report issued by Brookings calls into questions such comparisons. However, the US continues to have a highly desirable system of educating. “Bucking Cultural Norms, Asia Tries Liberal Arts” is an article by Karen Fischer that champions liberal arts education precisely because it is not narrowly focused and so is thought to hold the “secrets” of innovation. Similarly, in his article, “Why do we Continue to Isolate Ourselves by only Speaking English?” Will Hutton laments the fact that so few students bother learning foreign languages and says, “To acquire another language is to open yourself up to the world and to increase vastly your employability.” Language training, like learning an instrument (I will sing this song forever), requires discipline, dedication, and endless practice. I am certain these skills are transferable to other fields like science, math and reading. I am certain these skills are beneficial to the workplace. I am certain these skills are relevant to a global economy. In each country considered in the Hanushek and Woessmann study, the populations are multilingual. Several Western nations also have Ministers of Culture who ensure the nation’s culture is integrated into the society and its policies at every level and yet these points are omitted in their influential study. The problem with PISA is that it does not accurately reflect the best of who we are as a nation.

So, why trumpet science, math and reading to the exclusion of creative forms of expression? I think Hanushek and Woessmann are correct in noting the very structure of the educational system must change and in linking their study to US capitalist desires (cognitive ability – GDP), the authors “sell” an idea they know will attract funding; it is then left to researchers, educators, policy gurus, and private sector allies to determine the methods of change. The study is so influential that it has trickled down to the level of educators and policy makers who have run rampant with the idea of science, math and reading as THE indicators of cognitive ability. We have seen a steady erosion of arts and physical education in the public education system and a steady increase in testing along with a steady rise in diagnoses of ADD and ADHD, childhood obesity and Type 2 diabetes. This may be so practical as to defy common sense but having kids sit on their butts all day (and for extended days with double and triple periods of science, math and reading!) and then diagnosing them with ADHD and ADD and Type 2 diabetes is an evil game.

So back to “Rethinking School” — some points to consider: (1) providing access to and increasing the human-centered, responsible and interactive use of technology in the learning experience; (2) eliminating the various forms of bias that have excluded and otherwise discouraged so many from the classroom; (3) mandating creative and performing arts, sports and language training k-16; (4) having an interdisciplinary cadre of “assessors” who evaluate student performance; (5) reevaluating and increasing teacher compensation, it is necessary and it is long overdue; (6) increasing student exchange programs, both in country and international.

One thought on “Education, Testing and the Problem with PISA

  1. Very nice article thank you so much as a teacher and having been involved in education directly or indirectly internationally for nearly forty years we never do get a clear picture of as you indicated teacher quality and student performance. Measuring a cross section of high school students in US to a cross section of high school students in any country other than US and you get radical differences. Most other countries literally all number one remove special education students before high school inclusion is a US term. Technical oriented stduents are gone and you are now “testing” in all other countries college bound high school students. We do not test this way. We test a cross section and may have cognitive levels from near mentally impaired to college bound. In Georgia our ninth grade literature test susposed end of course test is a reading test only. Nothing on the test is in the ninth grade literature curriculum. A student could take at the beginning of the semester and do about the same as the end of semester but we say it is an end of course test. Standarized testing is only a measure of a point in time in a specific category of material on test and not a measure of achievemnt or knowledge gained. Unless we have before and after tests. As for teachers there are measures out there that do give a good window into a teachers abilities. Charlotte Danielson’s efforts and rubrics can show some significant points but still teaching is a difficult item to measure quantitatively. I am one who likes portfolios for students and teachers alike a complilation of materials across a period of time showing growth and evolution. Far too many teachers become simply a teacher and never go beyond. A teacher must always be a learner first prior to being a teacher and continue being a learner through their life. Sorry for wandering but your thoughts are great in your blog thank you. bird

Add your voice

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Enter your email address to follow this blog and receive notifications of new posts by email.

RSS Take the “A” Train

%d bloggers like this: