PISA Shangai

Most folks I know working on improving education for USA kids don't think too much about the "international comparisons" like PISA, TIMMS, etc.  I.e., the focus is on the achievement gap within the USA; the international tests essentially shows the gap between suburban American kids and some other nations. 

Still, it's relevant to our work.  A big picture question tends to go like this: should we focus on the within USA gaps (which means help the struggling Americans), or should we focus on the international gap (which means help median Americans)?  The bigger the second gap, the more resources might be allocated towards the burbs versus the high-poverty communities. 

Shanghai has been receiving high praise on the recent PISA.  Google and you'll get a ton of hits.  Lots of "lessons" to learn. 

Until now.  One of my mentors, Tom Loveless of Brookings, examined the Shangai PISA data, and found it woefully misleading. 

I haven't seen a data taketown like this since around 2001, when Jay Greene, US News journalists, and some others showed how the high school dropout rate in the USA was more than double in real life what the official stats suggested.   This led to massive revision over the whole decade to the reasonably accurate data we have today.

Tom writes:

And now for three years running, the OECD and PISA continue to promote a distorted picture of Shanghai’s school system by remaining silent on the plight of Chinese migrant children and what is one of the greatest human rights calamities of our time.

You have to read the whole article unless you already understand the hukou system.  But here's the key part:

In 2010, Andreas Schleicher of the OECD revealed that the 2009 PISA was conducted in 12 provinces in China. The data from mainland provinces other than Shanghai have never been released, and OECD’s list of participants in the 2009 PISA continues to omit them. A Chinese website leaked purported scores from other provinces, but the scores have never been confirmed by PISA officials in Paris.

This shroud of secrecy is peculiar in international assessment. Now the world has new data from the 2012 PISA. The OECD has not disclosed if other Chinese provinces secretly took part in the 2012 assessment. Nor have PISA officials disclosed who selected the provinces that participated. Did the Chinese government pick the provinces? Does the Chinese government decide which scores will be released? In 2012, the BBC reported that the Chinese government did not “allow” the OECD to publish PISA 2009 data on provinces other than Shanghai. There is a lack of transparency surrounding PISA’s relationship with China.

Shanghai is portrayed as a paragon of equity in PISA publications. A 2010 OECD publication,Strong Performers and Successful Reformers in Education, highlights model systems that the world should emulate. Shanghai is singled out for praise. One section on Shanghai is entitled, “Ahead of the pack in universal education.” The city is described as an “education hub,” and the only discussion that even remotely touches upon migrants is the following:

“Graduates from Shanghai’s institutions are allowed to stay and work in Shanghai, regardless of their places of origin. For that reason, many ’education migrants now move to Shanghai mainly to educate their children.”

That description is surreal. PISA’s blindness to what is really going on in Shanghai was also evident in the official release of PISA’s latest scores. The 2012 data appear in volumes organized by themes. Volume II is entitled, PISA 2012 Results: Excellence through Equity, Giving Every Student the Chance to Succeed. Shanghai is named as one jurisdiction where schools “achieve high mathematics performance without introducing greater inequities in education outcomes (p. 28)” and one with “above average socio-economic diversity (p. 30).” In the 336 pages of this publication on equity, the word “migrant” appears only once, in a discussion of Mexico. The word “hukou” does not appear at all.

Is it possible that PISA officials are simply unaware of the hukou system and the media coverage cited above? That’s doubtful, but even if it were the case, PISA’s own data shout out that something is wrong with Shanghai’s enrollment numbers. PISA reports that 90,796 of Shanghai’s 15 year-olds are enrolled in school in grade 7 or above, out of a total population of 108,056 15 year-olds, producing an enrollment rate of about 84%. That’s comparable to other PISA participants.  Shanghai appears as inclusive as any other PISA participant.

...How is it that Shanghai, with a population two to four times that of these ten countries, yields a similar number of 15 year-olds? A back of the envelope calculation suggests that a jurisdiction with 24 million people should yield a minimum of 230,000 15 year-olds. The missing population in Shanghai exceeds the recognized one. Where did all of Shanghai’s 15 year-olds go?

It will be interesting to see a response from PISA, and thoughts from Amanda Ripley, who wrote about PISA in her recent book.  One thing we all remember from Loveless classes at the Kennedy School of Government; that guy is really sharp with his data.