Your Good Mother, Part the Third

Your Good Mother has just figured out another situation you should listen to. The Gallup folks are not happy about being fingered as Romney Borgs, so now they're crying wolf. If the "aggregators" manage to get elections (and, by extension, other polling efforts such as used in, say, advertising) right, does this mean that pollsters will be put out of business, until there's only one left? Could be. That's Gallup's assertion.

One one hand, Gallup is right: the cheap drives out the dear. On the other, in this case, aggregation does identify the perpetually wrong, who will in time fade into Chapter 7. There's nothing wrong with that, given the Social Darwinist imperative. If, over time, aggregation finds the perpetually correct, and that one pollster dominates, so what? It may be that many (not widely admitted) social goods exist in our economy. In other words, Adam Smith may be largely wrong in his point of view: much of commerce devolves to one (or a few) producers. And having one (or a few) producers, means socialism. Otherwise we get monopoly, and restricted access and concentration of power. Remember, Smith's archetype was straight pin manufacture.

Organizations that traditionally go to the expense and effort to conduct individual polls could, in theory, decide to put their efforts into aggregation and statistical analyses of other people's polls in the next election cycle and cut out their own polling. If many organizations make this seemingly rational decision, we could quickly be in a situation in which there are fewer and fewer polls left to aggregate and put into statistical models.

Welcome to the world of oligopoly. And here's the coup de grace:

The aggregators that came closest to Obama's overall winning margin were the ones that attempted to account for pollsters' house effects.

In other words, making reasoned decisions on bias does the best job of removing bias. Imagine that: brains beat data. Read through the piece. Once again, lemmings end up killing each other.

The APL of My Eye

When I found R, I found an artifact of a language I dabbled in decades ago (the use of <- for assignment). That language is APL (A Programming Language, by Iverson). Turns out that I'm not the only, or first, to notice this connection. An older post from the folks at Revolution Analytics. I actually had my hands on a real APL keyboard, lo those many days ago. APL is still around, but I don't know about the keyboards.

Reading through the wiki piece, the similarity between the matrix operation paradigm of APL and the set operations of the RM came to mind once again. Turns out Codd wrote this paper on using the two together.

And there were the legendary one liner contests:

There were soon contests to see who could find a one-line version of an algorithm previously thought to require many. Sometimes this improved clarity. Sometimes it had the opposite effect, and brevity trumped readability.

The point was about the same as Obfuscated C contests, I suppose.

RStudio maps alt- to <-, so it's no burden.

Eight Miles High

Financial services isn't the only field of endeavour where blind allegiance to quants leads to disaster. In fact, another area where disaster, induced by quant blindness, is far more common (although, from a macro-economic view, thankfully localized) is drug development. For those not paying attention to drugs, the process of taking a compound (either new or reformulated in some way) from inception of consumers is said to take a decade and $1 billion. For myself, most of the money goes to line the pockets of non-productive management, so the true cost is somewhat lower.

As to the quants. This week brought yet another example of a failed trial. Once a compound is deemed fit for testing on humans, there are three general phases. The first phase is intended to confirm safety, the second to determine baseline efficacy in a small well defined sample population, and finally phase three, where a much larger population is used. This third phase (and often two PIII trials are required) is the basis for FDA approval.

This week brought yet another failed PIII trial in a cancer drug. The company was Oncothyreon, and the drug Stimuvax. Previous trials led many, even large funds, to believe that Stimuvax (a cancer "vaccine") was a slam dunk. The data convinced them. In the aftermath, many articles have been written, and I expect more yet. This one struck my as particularly apt, to this endeavour. This isn't a Monday morning quarterback, by the way. The author had laid out his thesis last year. This article is a short recap (pardon the expression).

The telling point:

A drug's mechanism of action is central to its effect. Do not ignore this! Clinical data can be misleading, innocently biased, meaningless, manipulated and sometimes even downright doctored. However, the question, "How does the drug work?" is always critical.

Much like the lemmings who dove over the housing cliff, those that chose to ignore the basic metric of house price and median income data which were in plain sight, quants will ignore the most basic question: what does this shit do?

It's the QE, Stupid

Spend some time following R-bloggers, and each day, on average, is another posting detailing how to get rich with yet another "trading" algorithm/strategy. Never mind that Galbraith got it right decades ago: "Financial genius is a rising stock market". Today brings yet more evidence that he's right.

What's disappointing with today's piece: it ignores the true motivator for the inflation of the stock market, Greenspan's (now, Bernanke's) flood of moolah to stock trading firms. Free money makes pushing stocks to new highs essentially risk free behavior. And simply explains (Occam's Razor, and all that) the data. A couple of quotes are kind of fun though.

Their conclusion was that none of these factors -- which investors often cite when explaining their moves -- come remotely close to forecasting accurately how stocks will perform in the coming year. "One-year forecasts of the market are practically meaningless," Mr. Aliaga-Díaz says.

Yet, posting after posting explain how to predict tomorrow's prices using very short-term data. My, my.

...despite all the storm clouds hanging over this economy, professional investors appear willing to look past the poor data. In fact, money managers say they are more bullish about domestic blue-chip stocks than about stocks in emerging markets or the rest of the developed world, according to a recent survey by Russell Investments.

They're not looking past poor data. They're looking past the near-zero opportunity cost of "safe" (e.g. Treasuries) placements. They have to participate in the Wall Street Ponzi scheme, if they want to "earn" their beloved bonuses. Their are myriad more (perhaps) unintended consequences of Greenspan's stupidity. This was Greenspan's ploy, and Bernanke follows suit. Greenspan did this out of monetarist's zeal; Bernanke because the Right Wingnuts have emasculated fiscal policy leaving him no other option.

Do Be a SAP

In the process of doing some research on a piece suggested, but not yet contracted, on the subject of R and other databases (those without PL/R, which is most of them, sort of), I came across, again, pieces about SAP HANA and R. This one had the following to say:

SAP's long-term ambition with its in-memory technology is for it to serve as both the transactional database and the analytic database, cutting out layers of hardware, related management software and cost while also improving performance. That will cut out Oracle (as well as IBM DB2, Microsoft SQL Server, and Sybase ASE) as the database that runs SAP apps as well as app-related analytics.

The piece was aimed at the friction twixt SAP and Oracle, but the notion that database manipulation belongs with/in the engine sounds kind of familiar. We are not alone.

Et Tu, Brute?

With all the hoopla around Apple these days, I find myself submitting comments here and there; which comments amount to the observation that Apple puts its effort not into innovation, but honing design and hype to an existing tech. The iPod was a late entry to MP3. The iPad was hardly the first tablet. And so on.

Here in the AnandTech review of the iPad 4 we find this observation:

Apple has been quick to dismiss Microsoft's attempts to bring touch to notebooks, but there's a lot of history around Apple laughing at something only to bring forth its own take later on.

It's going to be amusing to see whether Apple can continue to sell me-too products at such premium prices.

Green Grow the Rushes

There's been a bit of angst on display through R-bloggers. First one Matt Asher showed his simple analysis, "disproving" global warming. Lots of comments, mostly disagreeing.

Then, Ian posted an explicit rebuttal. More comments, generally agreeing with Ian.

I confess, I commented a bit. I just returned from looking at the comment streams, still ongoing, and just have to share a bit of one of them (from one dhogaza, no link):

CO2 forcing is real. If you think you can disprove this basic physical fact with statistical analysis, I invite you to stare into the business end of a CO2 laser, hit the "on" switch, and report back afterwards ...

Ouch!

This all is relevant, beyond peeing on the global warming folks, due to the quants' penchant for ignoring physical reality, and assuming they can simulate a truer reality (they don't always say it so bluntly, but that's what they mean). These quants crashed the global economy be the simplistic assumption that the data describing the US housing markets for a short period of time would be true for all time going forward. They ignored the historic record of the ratio of house price to median income. They ignored the divergence of this metric as it happened. They ignored the process (fiddling by mortgage companies, and thence in competitive response, banks) which propelled the divergence. Their Monte Carlo models told them they were right. Sure.

Ignorance is bliss. If you've scarfed up all the money already.