Comments, observations and thoughts from two bloggers on applied statistics, higher education and epidemiology. Joseph is an associate professor. Mark is a professional statistician and former math teacher.
Starting a new thread on Common Core over at the teaching blog. I'll be cross-posting the highlights but if you want to follow the whole thing, click on the link above.
Adrian: Once again we've got our friend from military intelligence. Can you tell us what you've found out about the enemy since you've been here?
Adrian as Gomer: We found out that we can't find them. They're out there, and we're having a major difficulty in finding the enemy.
Adrian: Well, what do you use to look for them?
Adrian as Gomer: Well, we ask people, 'Are you the enemy? And whoever says yes, we shoot them.
There is no profession anywhere in the country that has such astonishing rules. Good lord-- even if your manager at McDonalds decides you're not up to snuff, he doesn't blackball you from ever working in any fast food joint ever again! Yes, every profession has means of defrocking people who commit egregious and unpardonable offenses. But-- and I'm going to repeat this because I'm afraid your This Can't Be Real filter is keeping you from seeing the words that I'm typing-- Massachusetts proposes to take your license to teach away if you have a couple of low evaluations.
and
One version of the plan even allows for factoring in student evaluations of teachers; yes, teachers, your entire career can be hanging by a thread that dangles in front of an eight-year-old with scissors.
Wow. Isn't it a good thing that we are sure that this power will not be abused? I wish I could get Mark to give a perspective on this one, as I think it would have major implications.
Obviously this sort of scheme would only be workable for major offenses or where the barriers to entry to the profession are exceptionally low. Adam Smith pointed out that the less security that you give workers, the more expensive they get (for the same quality of worker). Now removing teacher tenure might or might not be a huge blow depending on the employment regulations in a state. There are professions (e.g. hedge fund trader) where people can move around a lot during a career. Teaching isn't well set up for this, but one can at least imagine making it work.
But losing one's license for bad test scores is a massive penalty. Heck, is it even the case that you'd lose a teaching license (as opposed to merely being fired) for tampering with the tests? Think carefully because employees will be able to work this one out . . .
A few months ago, Meredith Broussard ran a great expose of the role of textbook companies in the education reform movement (it's about other things as well, but that will get us started). I plan on discussing it more at length but the queue is getting long and I'd like to get this into the conversation as soon as possible.
The end result is that Philadelphia’s numbers simply don’t add up. Consider the eighth grade at Tilden Middle School in Southwest Philadelphia. According to district records, Tilden uses a reading curriculum called Elements of Literature, published by Houghton Mifflin. In 2012–2013, Tilden had 117 students in its eighth grade, but it only had 42 of these eighth-grade reading textbooks, according to the (admittedly flawed) district inventory system. Tilden’s eighth grade students largely failed the state standardized test: Their average reading score was 29.4 percent, compared with 57.9 percent districtwide.
One problem is that no one is keeping track of what these students need and what they actually have. Another problem is that there’s simply too little money in the education budget. The Elements of Literature textbook costs $114.75. However, in 2012–2013, Tilden (like every other middle school in Philadelphia) was only allocated $30.30 per student to buy books—and that amount, which was barely a quarter the price of one textbook, was supposed to cover every subject, not just one. My own calculations show that the average Philadelphia school had only 27 percent of the books required to teach its curriculum in 2012-2013, and it would have cost $68 million to pay for all the books schools need. Because the school district doesn’t collect comprehensive data on its textbook use, this calculation could be an overestimate—but more likely, it’s a significant underestimate.
This mentality also ruins human interaction. There is a tremendous variety of human beings within each political party. To judge human beings on political labels is to deny and ignore what is most important about them. It is to profoundly devalue them. That is the core sin of prejudice, whether it is racism or partyism.
There are some important points (and some questionable ones) made in this column. I'm not sure the 'ism' adds much to the discussion, but it's a topic that deserves more attention (and more pieces like this 2012 episode of This American Life). It's also a topic for another post.
For now let's stick with the con(firmation) artists thread. That's my somewhat cumbersome term for a school of journalists who get away with cliched, factually-challenged reporting because they appeal to the prejudices of their target audiences and, more importantly, of their peers and superiors. It is a group strongly associated with the New York Times and best exemplified by Brooks.
Much of Brooks' success comes from his ability to craft readable, scholarly sounding pieces that use statistics and anecdotes to reinforce class stereotypes, particularly those held by people on the top have toward people on the bottom. Frequently, when he could not find suitable support for his arguments, Brooks has used facts that aren't actually true. This might simply be the product of sloppiness, but it should be noted that the sloppiness always seems to occur in one direction.
At the risk of oversharpening, if you take the paragraph above and substitute the concept of class in for party, every criticism Brooks makes applies to much if not most of his own work. He has made a remarkably successful career out of understating the "tremendous variety of human beings" and judging them based on class and other crude demographic labels. What's worse, he continues to resort to statements that simply aren't true (like this claim about vaccines) in order to reinforce various stereotypes.
Does this behavior qualify as "the core sin of prejudice"? I'm not sure, but Brooks' partyism column certainly demonstrates a stunning lack of self-awareness.
Put simply, any teacher who wants his or her students to pass the tests has to give out books from the Big Three publishers. If you look at a textbook from one of these companies and look at the standardized tests written by the same company, even a third grader can see that many of the questions on the test are similar to the questions in the book. In fact, Pearson came under fire last year for using a passage on a standardized test that was taken verbatim from a Pearson textbook.
The issue often has as much to do with wording as it does with facts or figures. Consider this question from the 2009 PSSA, which asked third-grade students to write down an even number with three digits and then explain how they arrived at their answers. Here’s an example of a correct answer, taken from a testing supplement put out by the Pennsylvania Department of Education:
Here’s an example of a partially correct answer that earned the student just one point instead of two:
This second answer is correct, but the third-grade student lacked the specific conceptual underpinnings to explain why it was correct. TheEveryday Math curriculum happens to cover this rationale in detail, and the third-grade study guide instructs teachers to drill students on it: “What is one of the rules for odd and even factors and their products? How do you know that this rule is true?” A third-grader without a textbook can learn the difference between even and odd numbers, but she will find it hard to guess how the test-maker wants to see that difference explained.
One of these days, I need to post a nice long thread on targeted marketing, benefits vs. limitations, promise vs. hype. In the meantime, I'll collect as many examples as I can of targeting done well, targeting done badly and, in this case, of targeting not done at all.
In a comment to Ken Levine's previously mentioned traffic report prank.
In Canada, Closed Captioning is sponsored and announced during the broadcast. Frequently, these sponsors are new album releases - Katy Perry, Rihanna, Coldplay.
Assuming that the language of the captioning is the same as the language of the broadcast and is not used for translation (not an absolute given, I suppose, in Canada), think about the target demographic of captioning.
We’re all familiar with the critiques of standardized tests and other common measures used for high-stakes decisions. Recently, somebody in my circle has started going on about measures of “grit” and their predictive power. I am willing to believe that “grit” is an excellent predictor of all sorts of things. But I wonder if much of the predictive power of “grit” comes from the fact that these measures are currently low-stakes, so people have few incentives to game them.
I really think that this is the heart of the measurement problem. Insofar as there is a way to do better on a test, in a way that is less work than just be really good at it, then it is probable that much of your signal will be gaming. Studying the form of the question, for example, is likely to improve performance (by less confusion, if nothing else) but access to these approaches may vary by context.
Even worse, some of the test prep may have nothing to do with the underlying measure. So the score starts to measure things like "willingness to sacrifice learning time for test prep time".
This is a very good insight and likely to be eternally problematic in education.
I realize we've hammered this point quite a bit but, for their own good as well as the public's, charter schools have got to come up with an aggressive plan to deal with the self-dealing, price gouging and general looting that is becoming widespread in places like Michigan, Florida and Ohio.
The charter school Imagine School for the Arts is paying rent of nearly $1 million a year on a downtown building with the education funding it gets from the state, prompting criticism from a progressive advocacy group that studied charter-school finances around the state.
The complicated financial arrangement also involves a school-affiliated trust company spending more than $7 million last year to buy a building valued at less than $2 million.
The liberal advocacy group ProgressOhio attacked the size of the rent payments at charter schools operated in Toledo and other Ohio cities by Imagine Schools Monday as excessive. Imagine is a national for-profit educational management company.
According to ProgressOhio, Imagine’s subsidiary, Schoolhouse Finance, collected at least $14.4 million in public money last year for the company’s 17 Ohio schools. Of that, $8.9 million covered rent for long-term leases to Schoolhouse Finance. The $5.5 million balance went to pay “indirect costs” to Imagine to provide management services.
Both the tech and financial sector have embraced the idea that economic rewards are directly correlated to work and worth. It's a strange mixture of efficient market theorem and social Darwinism, often with more than a bit of Randianism.
Because rich people spend their money on useless stuff. Not far from where I live, there is a new house going up. It will be over 10,000 square feet when it is complete. 2,500 of those square feet will be a closet that has two separate floors, one for regular clothes and one for formal wear. If that is what you are spending your money on, then yes, I believe raising your taxes to fund education, infrastructure, and health spending is a net gain for society.
Don’t poor people spend money on stupid stuff? Of course they do. Isn’t the government an inefficient provider of some of these goods, like education? Maybe. But even if both those things are true, public investment and/or transfers to poor people will result in some net investment that I’m not currently getting from the mega-closet family. I’m happy to talk about alternative institutional settings that would ensure a greater proportion of the funds get spent on actual investments.
And,
Because I’m not afraid that some embattled, industrious core of “makers” will decide to “go Galt” and drop out of society, leaving the rest of us poor schleps to fend for ourselves. Oh, however will we figure out how to feed ourselves without hedge fund managers around to guide us?
I think that this points out that the notion that people deserve the actual income/wealth that they currently have, in some sort of fundamental way, is a true measure of worth/contribution. It is true that rich people do pay taxes, but consider all of the benefits they get? We have a whole society of laws devoted to reducing kidnapping, murder of near kin, and robbery, and the wealthy definitely benefit from this.
So I am not saying that the current level of taxes is too low, too high, or just right (on any specific segment of the market). I am saying that an open discussion should consider issues like "everybody spends money to satisfy desires and that these desires rarely pass the scrutiny of outside parties". Or that the whole idea of "going Galt" is often rather silly. In academics, we have many good people for every position. The marginal loss of any one person is sad but does not destroy the whole enterprise.
Similarly, I suspect that there are a number of possible hedge fund managers in the world who could do a relatively comparable job (at least insofar as society as a whole matters -- losing your superstar investor could be a private tragedy). Otherwise we'd expect the market to collapse when one of these key people dies of old age (and the market has been pretty robust to replacements via death so far).
A couple of quick caveats: I am never entirely comfortable focusing too much on "additional years of learning" -- it's a problematic metric -- and the link to the original report seems to be dead.
The good folks at Vox are very excited by the results of this recent experiment in merit pay. Perhaps I'm missing something obvious, but that jump between 2010 and 2009 seems really big.
For me, one of the most interesting stories in politics these days is the way that information has come to flow in the the conservative movement. And sometimes the most interesting part of that story is the way information fails to flow.
Rep. Jason Chaffetz (R-Utah) says Ron Klain is "off to a bad start" in his new role as the president's Ebola response coordinator, and that the U.S. Surgeon General should be the one leading the effort. But what Chaffetz doesn't seem to realize is that there hasn't been a surgeon general for more than a year.
“Why not have the surgeon general head this up?" Chaffetz asked in a Wednesday appearance on Fox News. "I think that’s a very legitimate question. At least you have somebody who has a medical background whose been confirmed by the United States Senate.”
“It begs the question, what does the surgeon general do?" he added. "Why aren’t we empowering that person?”
After this broke, Chaffetz tried to moonwalk his way back from the statement but there is simply no way to frame this so that the man comes off as both well-informed and honest. His problem is that he is trying to follow an official party line that makes consistency almost impossible (you can't block relevant nominations and gut relevant funding while plausibly complaining about the government doing too little to address an epidemic).
I suspect the root of the problem is that the leadership of the conservative movement fell in love with the appealing but doubly flawed idea that you can create optimal messaging by controlling the process. Fox News has always been a hothouse for ideas and arguments crafted to appeal to the base. Conflicting data and effective counter-arguments were largely kept out of the environment.
This approach can work for a while but at some point you lose control. The system is too complex to fine-tune. Eventually you find yourself saddled with a bunch of ideas that the base is committed to even though they can't hope to survive in the outside world.
I've been going through some old posts on incentive pay and they got me thinking about how education reformers, fresh water economists and libertarians often fall back on a strongly linear model. That, in turn, got me thinking about a Heinlein story I read years ago.
"—We Also Walk Dogs" is a science fiction short story by Robert A. Heinlein. One of his Future History stories, it was first published in Astounding Science Fiction (July 1941 as by Anson MacDonald)....
'General Services', a very successful company that provides various personal services ... is asked to ... enable an interplanetary conference to be held on Earth, whose strong gravity is inhospitable to many of the native races of other planets in the solar system.
Much of the action of the story is ... about how to persuade the world’s leading physicist to undertake the job.
As memory serves, that really is the gist of the story. Find the right person, offer him or her sufficient incentive, get what you want. Add to that an entrepreneur hero...
Dean Dad had a great post a while back on trying to improve classroom scheduling. In it, he had a great example of the perils of overfitting. The context is that the Gates Foundation had suggested that a more efficient algorithm for classroom management could improve costs. The problem is that they aren't testing this in prospective environments -- leading to some issues:
Finally, and I can’t stress this enough, enrollment is a moving target. Patterns change unevenly, so in any given year you’ll get a fresh new crop of anomalies. When enrollment continues until just before classes start, as is typical at community colleges, you’re left making predictions based on partial information. And you have to make those predictions early enough for alternate plans to be realistic. That’s never a perfect science -- every year, someone complains that his course would have filled if we had just given it more time, which is unprovable either way -- but it’s inherent in being “responsive.” If we locked down enrollments months in advance, we would have time to squeeze out some efficiencies. With numbers changing until the last minute, it’s harder. You’ll never capture that if you only look at “snapshots.”
It's also worth noting that the staffing side is also an issue. Paying for "Just in time" staffing for a community college would almost certainly reduce the qualifications of the instructors, do strange things to the workload of permanent staff, or increase prices. After all, if you do not know if you are going to be taking a class until the first day of classes (when enrollments are likely close enough to know) then you will likely charge more or look for another job. Landlords are notoriously unsympathetic to "most of the time this works and I can pay rent" pleas.
So the report might be interesting as an estimate of the maximum possible benefit. But in the real world you never get 100% efficiency in any process, especially one that has ebbs and flows. Even McDonalds, good as they are (and they are really good at this) cannot completely eliminate wait times due to the lunch rush.
/cdn1.vox-cdn.com/uploads/chorus_asset/file/2386382/Screen_Shot_2014-10-23_at_5.04.16_PM.0.png)