Your Good Mother, Part the Third

Your Good Mother has just figured out another situation you should listen to. The Gallup folks are not happy about being fingered as Romney Borgs, so now they're crying wolf. If the "aggregators" manage to get elections (and, by extension, other polling efforts such as used in, say, advertising) right, does this mean that pollsters will be put out of business, until there's only one left? Could be. That's Gallup's assertion.

One one hand, Gallup is right: the cheap drives out the dear. On the other, in this case, aggregation does identify the perpetually wrong, who will in time fade into Chapter 7. There's nothing wrong with that, given the Social Darwinist imperative. If, over time, aggregation finds the perpetually correct, and that one pollster dominates, so what? It may be that many (not widely admitted) social goods exist in our economy. In other words, Adam Smith may be largely wrong in his point of view: much of commerce devolves to one (or a few) producers. And having one (or a few) producers, means socialism. Otherwise we get monopoly, and restricted access and concentration of power. Remember, Smith's archetype was straight pin manufacture.

Organizations that traditionally go to the expense and effort to conduct individual polls could, in theory, decide to put their efforts into aggregation and statistical analyses of other people's polls in the next election cycle and cut out their own polling. If many organizations make this seemingly rational decision, we could quickly be in a situation in which there are fewer and fewer polls left to aggregate and put into statistical models.

Welcome to the world of oligopoly. And here's the coup de grace:

The aggregators that came closest to Obama's overall winning margin were the ones that attempted to account for pollsters' house effects.

In other words, making reasoned decisions on bias does the best job of removing bias. Imagine that: brains beat data. Read through the piece. Once again, lemmings end up killing each other.

The APL of My Eye

When I found R, I found an artifact of a language I dabbled in decades ago (the use of <- for assignment). That language is APL (A Programming Language, by Iverson). Turns out that I'm not the only, or first, to notice this connection. An older post from the folks at Revolution Analytics. I actually had my hands on a real APL keyboard, lo those many days ago. APL is still around, but I don't know about the keyboards.

Reading through the wiki piece, the similarity between the matrix operation paradigm of APL and the set operations of the RM came to mind once again. Turns out Codd wrote this paper on using the two together.

And there were the legendary one liner contests:

There were soon contests to see who could find a one-line version of an algorithm previously thought to require many. Sometimes this improved clarity. Sometimes it had the opposite effect, and brevity trumped readability.

The point was about the same as Obfuscated C contests, I suppose.

RStudio maps alt- to <-, so it's no burden.

Eight Miles High

Financial services isn't the only field of endeavour where blind allegiance to quants leads to disaster. In fact, another area where disaster, induced by quant blindness, is far more common (although, from a macro-economic view, thankfully localized) is drug development. For those not paying attention to drugs, the process of taking a compound (either new or reformulated in some way) from inception of consumers is said to take a decade and $1 billion. For myself, most of the money goes to line the pockets of non-productive management, so the true cost is somewhat lower.

As to the quants. This week brought yet another example of a failed trial. Once a compound is deemed fit for testing on humans, there are three general phases. The first phase is intended to confirm safety, the second to determine baseline efficacy in a small well defined sample population, and finally phase three, where a much larger population is used. This third phase (and often two PIII trials are required) is the basis for FDA approval.

This week brought yet another failed PIII trial in a cancer drug. The company was Oncothyreon, and the drug Stimuvax. Previous trials led many, even large funds, to believe that Stimuvax (a cancer "vaccine") was a slam dunk. The data convinced them. In the aftermath, many articles have been written, and I expect more yet. This one struck my as particularly apt, to this endeavour. This isn't a Monday morning quarterback, by the way. The author had laid out his thesis last year. This article is a short recap (pardon the expression).

The telling point:

A drug's mechanism of action is central to its effect. Do not ignore this! Clinical data can be misleading, innocently biased, meaningless, manipulated and sometimes even downright doctored. However, the question, "How does the drug work?" is always critical.

Much like the lemmings who dove over the housing cliff, those that chose to ignore the basic metric of house price and median income data which were in plain sight, quants will ignore the most basic question: what does this shit do?

It's the QE, Stupid

Spend some time following R-bloggers, and each day, on average, is another posting detailing how to get rich with yet another "trading" algorithm/strategy. Never mind that Galbraith got it right decades ago: "Financial genius is a rising stock market". Today brings yet more evidence that he's right.

What's disappointing with today's piece: it ignores the true motivator for the inflation of the stock market, Greenspan's (now, Bernanke's) flood of moolah to stock trading firms. Free money makes pushing stocks to new highs essentially risk free behavior. And simply explains (Occam's Razor, and all that) the data. A couple of quotes are kind of fun though.

Their conclusion was that none of these factors -- which investors often cite when explaining their moves -- come remotely close to forecasting accurately how stocks will perform in the coming year. "One-year forecasts of the market are practically meaningless," Mr. Aliaga-Díaz says.

Yet, posting after posting explain how to predict tomorrow's prices using very short-term data. My, my.

...despite all the storm clouds hanging over this economy, professional investors appear willing to look past the poor data. In fact, money managers say they are more bullish about domestic blue-chip stocks than about stocks in emerging markets or the rest of the developed world, according to a recent survey by Russell Investments.

They're not looking past poor data. They're looking past the near-zero opportunity cost of "safe" (e.g. Treasuries) placements. They have to participate in the Wall Street Ponzi scheme, if they want to "earn" their beloved bonuses. Their are myriad more (perhaps) unintended consequences of Greenspan's stupidity. This was Greenspan's ploy, and Bernanke follows suit. Greenspan did this out of monetarist's zeal; Bernanke because the Right Wingnuts have emasculated fiscal policy leaving him no other option.

Do Be a SAP

In the process of doing some research on a piece suggested, but not yet contracted, on the subject of R and other databases (those without PL/R, which is most of them, sort of), I came across, again, pieces about SAP HANA and R. This one had the following to say:

SAP's long-term ambition with its in-memory technology is for it to serve as both the transactional database and the analytic database, cutting out layers of hardware, related management software and cost while also improving performance. That will cut out Oracle (as well as IBM DB2, Microsoft SQL Server, and Sybase ASE) as the database that runs SAP apps as well as app-related analytics.

The piece was aimed at the friction twixt SAP and Oracle, but the notion that database manipulation belongs with/in the engine sounds kind of familiar. We are not alone.

Et Tu, Brute?

With all the hoopla around Apple these days, I find myself submitting comments here and there; which comments amount to the observation that Apple puts its effort not into innovation, but honing design and hype to an existing tech. The iPod was a late entry to MP3. The iPad was hardly the first tablet. And so on.

Here in the AnandTech review of the iPad 4 we find this observation:

Apple has been quick to dismiss Microsoft's attempts to bring touch to notebooks, but there's a lot of history around Apple laughing at something only to bring forth its own take later on.

It's going to be amusing to see whether Apple can continue to sell me-too products at such premium prices.

Green Grow the Rushes

There's been a bit of angst on display through R-bloggers. First one Matt Asher showed his simple analysis, "disproving" global warming. Lots of comments, mostly disagreeing.

Then, Ian posted an explicit rebuttal. More comments, generally agreeing with Ian.

I confess, I commented a bit. I just returned from looking at the comment streams, still ongoing, and just have to share a bit of one of them (from one dhogaza, no link):

CO2 forcing is real. If you think you can disprove this basic physical fact with statistical analysis, I invite you to stare into the business end of a CO2 laser, hit the "on" switch, and report back afterwards ...

Ouch!

This all is relevant, beyond peeing on the global warming folks, due to the quants' penchant for ignoring physical reality, and assuming they can simulate a truer reality (they don't always say it so bluntly, but that's what they mean). These quants crashed the global economy be the simplistic assumption that the data describing the US housing markets for a short period of time would be true for all time going forward. They ignored the historic record of the ratio of house price to median income. They ignored the divergence of this metric as it happened. They ignored the process (fiddling by mortgage companies, and thence in competitive response, banks) which propelled the divergence. Their Monte Carlo models told them they were right. Sure.

Ignorance is bliss. If you've scarfed up all the money already.

Shared What?!

It can't be any fun to be an actuary any more. I read up the first exams' study guide years ago, but decided not to do the tests. McElhone's mantras, "the world is not linear" and "you don't estimate beyond the data", kept ringing in my ears. These days, when we get a once in a century event every couple of years? At the subatomic level, the universe is probabilistic. At the macro level, not so much; causes have effects, if only we pay attention.

There's a wealth of material to ponder in today's NYT "Sunday Review" and "Sunday Business"; I'll limit my musing to a piece on home insurance. The authors are from the Wharton School, and thus should have face validity. Not so much.

Here's where they go off the rails, so far as I'm concerned (a frequent theme here): they ignore community. Their theme is essentially this, household insurance in places of known higher risk should *alone bear the burden*. When I took the course in grad school, the title was "Risk and Insurance", and was conducted from the economics department, not the B school. The point of the course was to study the implementation of shared risk. The MBA crowd is solely concerned (modulo the occasional and optional Business Ethics course) with profit. That makes a difference.

First, premiums should reflect risk. This makes transparent the magnitude of the hazards one faces and could limit new construction in high-risk areas. Residents would be encouraged to reduce risk by getting discounts for, say, elevating a house or strengthening the roof.

This sounds fair, on its face. But there is a problem: if insurance companies (or public providers, for that matter) are allowed to exercise market segmentation at will, the process is no longer insurance, but prepaid consumption. The result of unfettered segmentation is to shift all of the cost (and source of profit) for any risk solely to those who are subject to that risk. Recall the recent "Your Good Mother" essay? The rich get richer and the poor whither. Insurance, from the point of view of economics, is all about shared risk, not individual risk. Individual risk is just prepaid consumption.

The justification for this predatory behavior is stated:

Those who live in nonrisky areas are subsidizing the choices of others. Take residential flood insurance. Most insurers refuse to cover the risk, so the National Flood Insurance Program, run by the Federal Emergency Management Agency, was established in 1968 with subsidized rates for those then living in flood-prone areas.

This approach is little different from allowing health insurance companies to ding subscribers with pre-existing conditions. If you live on the Atlantic beaches, you're far more likely to get hit by a hurricane. But those in Fly Over Land are far more likely to get hammered by baseball sized hail storms and tornadoes. Neither group can, just to avoid the condition, simply move to Safe Land.

The authors bury the point in an otherwise benign paragraph:

... the need to provide shareholders with an attractive return.

But shared risk is socialistic, or tribal, or communal. We all have some sort of risk associated with where we live. We can't just load up the SUV and head out for Safe Land.

How, then, to persuade folks from building in risky places? Is there sufficient acreage for a burgeoning population to live only where natural (and man made) disaster is less than average? Well, of course not. The arithmetic can't work. If all the folks in hurricane land and tornado land and earthquake land moved to Safe Land, the price of housing in Safe Land (wherever that may be, and likely in my neighborhood, as it happens) goes through the roof, so to speak. Not to mention the cratering of wages, with the influx of those needing employment. Back in the 18th and 19th centuries, there was more than enough room for the population, mostly subsistence farmers. We have since industrialized, urbanized, de-industrialized, and urbanized some more. The USofA population has more than doubled since WWII, those halcyon days of 90% maximum marginal tax rates. I suppose we could engage in the sort of back to the land re-education that Mao imposed. Doing so would certainly mitigate the risks of these Gray Swan events. And mitigate the rent seeking behavior of insurance companies.

Pulling up stakes and moving house (as the Brits put it) used to be just a matter of loading one's few possessions into a conestoga wagon and heading for Indian Territory to steal a few acres. Those days are long gone, and acting as if such action is the solution to any current problem is just plain evil.

We do need to make it more expensive to live in known dangerous places. But just as we have, nominally, a progressive tax paradigm, we should have a progressive insurance risk penalty. Those with McMansions on the beach should subsidize those living in shotgun shacks, not the other way round. Those who've been living in some risky place for some number of years, say 20, don't get dinged so much, but those who've entered recently (or in the future) when the risks have been clearly documented pay more. Local zoning and building codes are the major lever to pull to make building in vulnerable areas more, even wildly so, expensive. I don't know about you, but I don't see the commissioners of Miami Beach voluntarily curbing the building push. It was housing in Florida which was the epicenter of The Great Recession. Since increasing housing activity is viewed, even by Lefties who should know better, as the solution to The Great Recession, what to do? I highly recommend Carl Hiaasen's thinly veiled novels on the evil of development. "The trailers imploded, exploded, popped off the tie-downs and took off like fucking aluminum ducks." That's from "Stormy Weather", written in the aftermath of hurricane Andrew. You must read it, if you haven't.

It can be argued, of course, that wind insurance and flood insurance are sops to the slothful. That's the authors' point. Too simplistic by half.

Still, Don't Be A Dupe

Yet another dustup over SandForce and deduplication over at AnandTech. The fisticuffs are in the comments. What's still amusing is that the combatants still haven't figured out that dedupe is on-the-fly/ad-hoc 5NF storage. Ask them about relational databases, and they roll their eyes.

For example, if you have two very similar photos which are 5MB each, the controller may not write 10MB. Instead, it will only write let's say 8MB to the NAND because some of the data is duplicate and the whole idea of deduplication is to minimize NAND writes.

If you go and delete one of these photos, the OS sends a TRIM command that tells the LBA is no longer in use and it can be deleted. What makes SandForce more complicated is the fact that the photos don't necessarily have their own LBAs, so what you need to do is to check that the LBA you're about to erase is not mapped to any other LBA. Otherwise you might end up erasing a portion of the other photo as well.

While it's still up in the air whether SandForce controllers actually do dedupe, rather than just compression, the observation is correct. Not that any of these folks is able to make the mental leap. Kids! They never figure anything out until it's too late.

Your Good Mother, Part The Second

Along similar motivation of the Dee Feat series, herewith the second (for the sake of settling on a number) in our Good Mother series. This report in today's news reminded me of this other device, reported on by "60 Minutes" among others.

So, as Your Good Mother constantly reminds you: "what would the world be like if everybody behaved like you?" (She isn't necessarily literate enough with to know which preposition to use!) Once again, particularly from the Hawaii story, it's quite possible to bring overall degradation to an economy (and its society) by going all in on Rand.

At what point is the individual, especially acting in consort with some number of others, obliged to temper actions based on externalities? For those who worship at the feet of Adam Smith (the real one), the question is a bit more piquant, since his most famous text assumes that there are no externalities! Quants, working as they almost always do, for individual actors (in the context of some market, not humans) are described by micro-economic arguments. Again, externalities are assumed away. Not a good, or honest, thing to do. One might strongly argue that The Great Recession came about just because the quants (at the behest of The Suits) took explicit advantage of the fact that their companies would be protected from meeting the demands of the externalities of what they did. The micro point of view, of course, denies culpability for externalities: "those on the other side of the transaction are obligated to caveat emptor". Again, if everybody does "it", there's no way to mitigate the externalities except by taxing the community at large. And the bad actors get to keep their gains.

Another example, charter schools. Evidence is piling up that charter/private schools, particularly those catering to "inner city" students, don't do any better at educating, overall, than the public schools from which they rob moolah. As with spontaneous cancer remissions (every type has some, small, number of long term survivors), a few success stories of individual students can be cited. I'm always reminded of my local nutball, Linda McMahon, who propounded that, since "she" (actually, her equally nutball hubby) had become a millionaire, everybody else could, too. What such folks neatly elide: if everyone were a millionaire, a loaf of bread would cost everybody $10,000 (give or take a few thousand). Just because one can cite a single success from some policy, doesn't make the policy intelligent for the community. Not that the Rand-ians care about the community.

In all three cases, we have the problem of: divide and conquer (the Big Uns surreptitious way to win), where only those who can afford the individualized alternative get the good/service. The earliest example, in recent history, was White Flight in the 1960s. Suburban communities morphed into wealth by denying the common good; one might even see this as the continuation of Manifest Destiny (a stretch, but not by much). Many Christian schools are rooted in such communities; they don't get the irony, of course.

Now, I'm not a fan of The Power Grid, and never was. The first great blackout happened when I was a child, and I lived in it. To my surprise, the wiki piece does mention the point germane to this missive: Holyoke, an abutting town, had its own standalone powerplant (you can see it from Route 5 or I-91) and got along just fine. The justification for The Grid is load/demand balancing across time/space. I'm not sure of the physics, but transmission loss is not insignificant (the Bloom Energy piece states that avoiding said loss is a meaningful part the advantage). So far as Your Good Mother is concerned: what should be the unit of community where externalities exist? For power, it would seem to be the community (city/town). Holyoke wasn't, and isn't, like the Hamptons with hordes of upper-middle class folks fleeing the hoi polloi. Mostly hoi polloi, in fact.

One could even extend a power grid analysis to the au courant meme: The Cloud. I'll leave that as an exercise to The Reader.

Inbreeding Isn't a Strength

Well, the story is starting to emerge. Here's a version from The Atlantic. Toward the end of the piece we get this:

"A lot of the software is kind of late. It's looking ugly and I go out on this Field call," Kunesh remembered. "And people are like, 'Man, we should fire your bosses man... We gotta get the guys from the DNC. They don't know what the hell you're doing.' I'm sitting there going, 'I'm gonna get another margarita.'"

The problem: it was the DNC guys who screwed the pooch in 2010, and did little to stop the tsunami in state legislatures then or this year. We now have Yankees and Johnny Rebs, in spades. It's just now Coasts versus Crew Cuts. The lackadaisical DNC gave us this. I met with those DNC guys. They let 2010 happen. They aren't the ones to follow.

And then they provided a separate way for the Analytics people, who had very specific needs, to get the data in a different form.

Of course, if they'd used Postgres and PL/R, they'd have had it all together.

Recall from Triage, the observation that getting the response data would be the most costly (and time consuming)?

With Davidsen's help, the Analytics team built a tool they called The Optimizer, which allowed the campaign to buy eyeballs on television more cheaply. They took set-top box (that is to say, your cable or satellite box or DVR) data from Davidsen's old startup, Navik Networks, and correlated it with the campaign's own data. This occurred through a third party called Epsilon: the campaign sent its voter file and the television provider sent their billing file and boom, a list came back of people who had done certain things like, for example, watched the first presidential debate. Having that data allowed the campaign to buy ads that they knew would get in front of the most of their people at the least cost.

And so it was.

This is near the end of the piece. Not the last words, but the last ones that matter:

And losing, they felt more and more deeply as the campaign went on, would mean horrible things for the country. They started to worry about the next Supreme Court Justices while they coded.

If that doesn't connect with you, shame on you.

Such A Touching Tribute!

A bit more than two years ago, along with the first iPad, I considered the future of data stores and tablets. We now have Windows 8, in its three(?) suits of clothes. Back then I mused:

BCNF schemas, doled out in "just the right size" to Goldilocks, is the way forward. Very cool.

So, today we get this view of Win8, and it does sound rather familiar! This writer is new to SA, and on balance does pump pieces. On the other hand, the transition to pickable form factors is inevitable. As I pointed out back in 2010, the pick list implementation on tablets has been around for at least two decades. It may be new to most consumers, and Fortune X00 office drones, but it isn't new to a large swath of software developers.

The victors will be those that embrace the core meme of this endeavor: multi-processor/core/SSD machines on *nix running an industrial strength 5NF database. Only such systems have the capability to serve up small bites of coherent bytes of pickable data, without a tsunami of client (and only for one language) code to sort out the alternative from bloated flat-files, database resident or not.

The Apple of My Eye?

And now for something completely different. In which an attempt is made to forecast the future of Apple Computer. The share price has been vying with Lindsey Vonn for fastest time. Why might that be? And what might it mean?

Apple's "growth" has been driven by "new" products, what Vance Packard termed "planned obsolescence". They've been good at it in the Jobs-ian past. There's no indication that they'll continue to do so. Just what "new" product is waiting to enthrall the Apple Fan Boys? If the answer is, "we ain't got none", then Apple's growth goes poof.

Apple invented none of the platforms (MP3, laptop, smartphone, tablet, desktop) that have driven this growth. They did tweak their products in those platforms. All existed before Apple's entries. In some of those cases, Apple (purposely?) lagged the cutting edge to decipher the profitable high-end, MP3 for sure. Apple gambled that wireless network hardware would catch up soon enough, just as MicroSoft gambled that the next PC hardware would be just fast enough to support the next bloated version of Office.

What *existing* platform(s) are out there, for which Apple doesn't have an entry? If the answer is "none", then Apple has no option but to be MicroSoft/HP. They could invent a new platform, but that's something *not even Jobs did* (Woz made the machines). Right now, the answer is "none" to both questions. Feel free to name any.

Moreover, Apple's products have, since the original Mac, been intentionally high-end and small shift. That Samsung, et al, shift more parts isn't paradigm breaking; it's the norm. Samsung is MicroSoft/IBM et al.

Apple has benefited from a psychotic patent system which permits any random thought to be moated. And we now see Apple morphing from maker to taker; suing everybody on the planet. A patent on rectangle with rounded corners, indeed.

In sum, there is reason to believe that Apple has gone to level flight. Since it eschews high volume, low unit profit for low volume, high unit profit, can it continue to grow? No. While population growth continues on the steep end of the hockey stick, the number of households which fit Apple's economic criteria isn't. Continuing wealth and income concentration take a big bite out of what would otherwise be a burgeoning upper-middle class, in both the USofA and Europe. Japan continues in its lost decades (yes, now plural), and China's macro-policies of exporting poverty don't help Apple much either. Apple needs an expanding, global, upper-middle class to support its growth. Even if it does stumble onto a new platform. Whatever happened to iTV? What additional computerized activity can be made mobile? Address those questions with positive answers, and Apple has new parts it can shift at its traditional very high margins.

Blind Date

This endeavor has talked about high normal form on multi-processor/core/SSD machines from the beginning. On occasion, the complaint that such "complicated" schemas are not practically updatable, given existing client code, comes up. One hears this from Agile Zealots, who still want to write dumb data/smart code just as their granddaddies did. My answer has always been that "real" base tables are hidden, with the named "tables" being views. Updating, through joins for example, is handled with stored procs.

Turns out that SQL Server, at least, implements triggers which get around the updatable view problem. Michael Swart talks about it.

But When Would I Use This?

I thought of the perfect use case. This strategy helps with SCHEMA REFACTORING.

With the I/O capability of these current machines, the cost of maintenance drops considerably. Think of it as OO inheritance for databases. Or as Bill Gates was famous for saying (he did, didn't he?), "let the new hardware fix it".

For Your Listening Pleasure

Regular Reader is well aware by now that I've less than sanguine view of Kiddie Koders. Too many. Doing the wrong thing. And such.

Revolution Analytics runs webinars, which this week is a presentation from Deere. I was expecting a commercial for Revolution's closed source R implementation, but it isn't. What it is, is a surprisingly sober discussion of stats, analytics, and forecasting. There's no R code discussed directly, rather Derek Hoffman discusses the use of R results in Deere. He has a, in my view, jaundiced take on data science/analytics hype, which runs through his discussion and the Q&A at the end (some pointed remarks about Wall Street quants that could just as easily have been read here). You should run the slide show first; the YouTube video is way too fuzzy to be legible.

A by the way: Deere is a DB2 shop (with other less worthy databases). He doesn't say whether it's z/OS DB2, but I suspect it is.

Jim

We begin, again, with lyrics:

You don't tug on Superman's cape
You don't spit into the wind
You don't pull the mask off that old Lone Ranger
And you don't mess around with Jim
-- Jim Croce

More dirty laundry explaining the Romney Bubble (Yahoo! feed):

"As a result," says Crawford, "they believed that the public/media polls were skewed" in Obama's favor, and rejiggered them to show Romney with "turnout levels more favorable to Romney." In essence, Romney "unskewed" the polls, mirroring widely mocked moves by conservatives to show their candidate with a lead, epitomized by the now-infamous website UnskewedPolls.com.

Much the same as The Great Recession implosion, the Boys In Power decided to invent their own reality. They believed their propaganda. They were convinced that they could re-make the world into their image. They learned from Karl Rove.

That's not the way the world really works anymore. We're an empire now, and when we act, we create our own reality.

Can there be anything like a Right Wing quant? Put another way, is it possible for those attached to the Right Wing (including, obviously, Wall Street) to be based in the data? The evidence, so far, is No. These Wall Street quants ignored the most basic of data driving their MBS, CDO, CDS, and other derivatives (income/mortgage ratio). Incompetence of the highest order, but The Boys In Power wanted what they wanted, and their quants were happy to say Yes. Empires fall. The Right Wingnuts have fallen. Could they have won? Not without turning their back on the Tea Party and Rand-ians who have co-opted the party.

Reagan co-opted the so-called blue collar middle class, through the brilliant stratagem of convincing enough of them that, while they were Chosen People, their neighbors were freeloaders who deserved to starve. Steve Jobs was often accused of having a reality distortion field; those that fudge data in order to appease The Boys In Power today, sometimes end up on the losing end of the bargain. Not always, of course, only when the data are important. If they are truly attached to the agenda of the clients. In the usual scheme of things, the quants get their cut up front, of course. I suspect that the devisers of Orca made out very well. If they had built a system which accurately reflected reality (in the data), they would have lost the contract, since it would have contradicted the desires of The Boys In Power. Just as the Wall Street quants were only following orders.

Ann's Horse Didn't Win, Either

My, my. A recent missive here asserted that Gallup was wrong. Gallup was wrong right up to the end, although according to 538, they had inched toward Obambi at the end. Didn't quite get there. But that's not today's topic.

Turns out, Willard had his version of Triage!! I know nothing, nothing I say, about it. Except what I read in the newspapers.

The success, though, would have to depend on volunteer troops united by a Web-based smartphone app.

Is this the Literary Digest, all over again? Deja vu? I haven't found the original (I'm not on their mailing list, so...), here's the PR (from yet another Right Wing organ) referenced above. Figures that the Right Wingnuts would chose a predator as the project's nickname. I suppose, in their over-weening adolescence, they thought it was cool.

OTOH, there's this (kind of like Triage, don't you think):

For example: if we happen to be down in a state at lunch time, we can pinpoint exactly what is causing it. So, if we know we're going to win X state by 3 points, let's move our resources to Y state, county. In sum, Project ORCA will give us an enormous advantage by being able to know the current result of a state.

As described in the Triage piece, the key to making these kinds of systems work is timeliness of data. I suspect that these webby quants had never worked campaigns, spent time in a political city (my first two were Boston and DC), or understood that "dashboards" are cute, but GIGO.

In the primary, we learned it was difficult to be working from Boston and really affect voter turnout in the states. It was disappointing to receive data later and realize if we had access to that data earlier, we could have done something differently and affected the outcome.

Kind of late in the game, though. It's not quite as silly as my own state's Linda McMahon, who managed to get on the ballot more than once (as did Murphy). It seems that there were other parties officially on the ballot, which motivated McMahon to run ads in the last couple of days with talking head "real people" saying that they'd vote for Obama, and ..... Linda (as she termed herself, I suppose to offset the images of burly half naked men clobbering each other with metal chairs).

Another key component to Project ORCA is state-of-the-art dashboard. For the past several months, a "brain" has been built into this dashboard and it will take in, analyze and recommend actions on the millions of pieces of incoming data. In the fast-paced environment of an Election Day command center, having this programmed "brain" will alert decision-makers to key findings and suggest reallocation of resources.

Deja vu. That's the essence of Triage. But far too late in the game.

Here's the original, quoted liberally in the Yahoo! feed. It must have hurt. When I lived in Washington, there were a bunch of local papers in the Virginia and Maryland burbs. Thanks to WikiPedia, now I know why I hadn't heard of the Examiner: a mini-Koch bought them up some years ago and turned them into a freebie Washington Times. Mein Gott! Dat's gotta hurt.

Later, another aide said Orca had pretty much crashed in the heat of the action. "Somebody said Orca is lying on the beach with a harpoon in it," said the aide.

You know, if only they'd built Orca with R. Open source and quite powerful. Kind of a community organized to analyze data. Turns out a community, once motivated, can kick Roark's butt.

The Big Uns Win

Decades ago, after grad school and in a sort of job, I found much more time on my hands in the world of work than I'd had at university. Grad school is very much like being an indentured servant. I had lots of time, but not a lot of moolah. I ended up pursuing various phys. ed. activities, the likes of which never occurred to me earlier. One was the study of karate, with a Greek fellow in East Boston. One of his observations has stuck with me ever since: "the good big uns always beat the good little uns".

In recent months, and recent discussions in various places, I've asserted that OCZ syndrome is endemic to consumer SSD business. Just as there are 2.x HDD companies, SSDs (in the consumer venue) will devolve into the same oligopoly. The Big Uns will drive out the Little Uns in the race to the bottom; there is no meaningful performance gain from bespoke parts in a consumer SSD that Joe Six Pack ("It boots Windoze 8 in only 30 seconds!") will deign to pay for, so the big silicon houses will triumph. Intel might be a Big Un, but it has just revealed that it, too, wants to move into The Enterprise Space.

AnandTech reveals Intel's new controller, and its interest in being an Enterprise SSD vendor. Those who've followed the SSD saga, here and/or elsewhere, for half a decade or longer may remember the X-25E: an SLC "enterprise" SSD. Didn't work out so hot. STEC was riding high, and serious about enterprise. STEC is, perhaps, circling the drain. The whole notion of Enterprise SSD is in danger of disappearing. The first, in permanent place, quote on this endeavor may be coming true, some years later than expected.

The direct flash appliance, from IBM/Oracle/whoever, may soon obsolete out the spinning rust semantics. Intel's new SSD may be too little too late. On the other hand, when the Sandforce and G3 (really, just tweaked G2) "new" drives appeared about a year or so ago, many speculated that Intel had folded its tent. Guess not.

But based on this story, Intel isn't getting past STEC, despite its troubles, or the appliance approach from IBM/Oracle/whoever. This is a prosumer drive. It specs as a good database developer machine drive; an economical surrogate for the performance an application would get on Real Iron (well, sand). I was just about to pick up a Samsung 830, cheap, since the 840 is starting to appear. Not so much. A 400 may be too much to resist.

Danger, Will Robinson

Revolution Analytics, in the person of David Smith, has an active blogging presence; this link is to a recent presentation (yes, toward the end it's a commercial). I've just added a quote from the intro. For the record, run the slides, and note 8. In the database business, in large companies, there exists the not-too-functional marriage of business analyst and data geek. What happens, all too often, is that the BAs have far too much authority. They are in the Danger Zone! Then, of course, have a gander at 24. Sound familiar? One might interpret the imperative as "keep the logic in the data, not the code". That's my point of view, of course. A particular implementation, not mentioned in the presentation, is PL/R with postgres. Postgres isn't mentioned at all (if you don't count databases grown from PG).

Treat, no Trick

Talk about a Treat!!! There is a "preview" build of R Studio (note the switch to .com) and now supports vi(m)!!! Adults use either emacs (the demented side of the family) or vi(m) (the sane and smart side). Woo Hoo!!

I Vant to Biite Your Nheck

R version 2.15.2 is out, aptly name "Trick or Treat". Just in time for Sandy. Note that you need to build from source, and run configure with the --enable-R-shlib switch in order to use R with Postgres through PL/R. Works like a charm.

Merry Quant

The return of the mini (and micro-mini) skirt does my heart good. Once again, we wait for the heaven sent Sharon Stone moment. Ah, not so often. I do wonder: do our Merry Quants know they're the descendants of such a creative mind? Or, are they just in it for the money? Just the money I guess. But that begs the question: why did financialization overwhelm our economy, and other countries'? In the 1960s, Uruguay started down that road, and ended up with guerrilla warfare and military coup. While I've long since lost my thesis, this paper provides a more recent appraisal. It's not any prettier.

So, why did/do the quants hold sway, and play only in fiduciary instruments? Why financialize (conversely, de-industrialize) an economy? Is there a Maslow's pyramid of economic development? No, I think not. What's happened since Reagan is a shift of power, aided and abetted by Washington's Right Wing tilt, to the easy, fast buck. Dr. Smith used the phrase of "the quick buck versus the slow deuce". While something of a Right Wingnut (he considered himself an entrepreneur stuck in a small New England college; his signature idea was a concrete Sunfish. I'm not joking.), he had no use for the quick buck artists. Yet, we've allowed them to take over the economy. Why?

The answer, by my lights, is obvious: fiduciary capital is easier; there's no messy physics (in all of its manifestations) to muck up the plan. That which exists only as bytes in computers can be controlled (or so the thinking goes) by programs in computers. The Merry Quants, and their handlers, ignore the truth; at some point a fiduciary attaches to a real object. In the case of The Great Recession, that object was a McMansion in Boca. It was attached to a subprime, fixed reset, ARM sold to a minimum wage janitor. And so the dominoes fell.

Last Sunday's NY Times "Sunday Business" has a long piece describing the impetus for the flight to fiduciaries. When one makes direct investment in physical capital, one is tied to the market for the widgets involved. To earn that return, it's not merely a matter of scarfing some of the money flow between saver and borrower, as the vampire squid do their voodoo so well. When you build it, you're betting they will come. And buy your widgets. Which widgets you've made for cheaper due to your investment. And, if you've been really clever, they buy more widgets, to boot.

This endeavor has oft quoted Your Good Mother: "what would the world be like if everybody behaved like you?", in the context of opprobrium for bad behaviour. Turns out, if everybody tries to make and sell the same widget at the same time, nobody makes any return on the invested capital. Thus Warren Buffett's observation about moats.

In the shale gas case, a gold rush meme doesn't work out. Gold is craved for no good reason, so having it, when and where others not so much, is always profitable (aside: it was the merchants who served the miners that made out selling eggs and the like for obscene prices), while natural gas is only valuable in use. Explode the supply, and well, you get it.

But while the gas rush has benefited most Americans, it's been a money loser so far for many of the gas exploration companies and their tens of thousands of investors.

Not that the Banksters didn't try for a nice slice of the pie.

Like the recent credit bubble, the boom and bust in gas were driven in large part by tens of billions of dollars in creative financing engineered by investment banks like Goldman Sachs, Barclays and Jefferies & Company.

After the financial crisis, the natural gas rush was one of the few major profit centers for Wall Street deal makers, who found willing takers among energy companies and foreign financial investors.

Not all were convinced.

"He is like the bartender serving drinks for people who can't handle it," said Fadel Gheit, a managing director at Oppenheimer & Company, about Mr. Eads [one of the money men profiled]. "And the whole gas industry has gotten a rude awakening, a hangover, with gas prices plummeting. The investment bankers were happy to help with a smile and get their cut."

To the predictable conclusion.

In hindsight, it should have been clear to everyone that a bust was likely to occur, with so many new wells being drilled and so much money financing them.

But, there is a silver lining; well for those of us with an eye-for-an-eye streak.

Some local landowners, having spent their initial lease bonuses, are now deeply in debt. Local restaurants and other businesses are suffering steep losses now that so many drillers have left town.

From the point of view of our Merry Quants, they flew just a bit too close to the sun. Best to stick with fiduciaries. How about re-insurance in Bermuda?

The Next Time Bomb

Our Regular Reader knows my fascination with Bermuda; there've been more than a few missives about the country. Most reflect my belief that Bermuda represents a real life, real time social petri dish. As Bermuda goes, so will the USofA. The reason is straightforward: Bermuda abandoned its natural strengths in tourism, to becoming a tax evasion haven (Bermudians dislike that characterization, but so what?). The government rewrote its laws (much as North Dakota did for the credit card industry) to support tax evasion. The result of this was to import a significant number of foreigners, at high salaries. The result of this increased skewing of income distribution has led to two inflation forces: demand pull, due to the high salaried imports bidding up housing, in particular; and cost push, due to the increased demand for imported goods caused by those same high salaried minority. The predictable reaction: native Bermudians demand wage increases to "keep up with prices", as such folks always do. They'd done nothing to produce the higher prices, and didn't buy the argument that they deserve to have their incomes de facto reduced. Income is always relative, never absolute.

Last month, the NY Times ran a piece on Bermuda's next foray into manipulating the financial services sector of the global economy. Further revealing the hollowness of Bermuda protestations of innocence.

We have another mine. We need a new canary. Herewith, some eye opening quotes.

The hedge fund industry has been rushing headlong to open Bermuda-based reinsurers. ... And while the hedge funds are likely to profit, the question is: Who's watching to make sure this doesn't lead to another financial calamity?

And, of course, few if any strings attached:

Yet while they are partly hedge funds, these new companies are regulated as reinsurers. And Bermuda requires only minimal capital requirements and disclosure of financial positions, and it does not strictly regulate how these companies invest their money.

As the incentives which led to The Great Recession, hedgies are looking for better than risk-free return (Treasuries, for example), damn the torpedoes.

Let's close with one more quote (you really should read the whole piece; especially if you're the Stephen King sort of weeny):

In other words, the reinsurance market is starting to look like many of the markets before the financial crisis -- lightly regulated and interconnected in ways that policy makers can't see, with banks potentially left with the wreckage. The industry may be right that reinsurance is different and has its own checks and balances, but we've also heard that before. It behooves United States regulators to make sure.

Given that Wall Street runs DC, not the other way around, do you think that likely? Will the Merry Quants blow up the world again? How many of you, Dear Readers, have even heard of re-insurance? How many would know to even go looking? QED

A Horse's Ass?

It would appear the Gallup is trotting down the same bridle path as The Literary Digest. While likely not as well known as the Chicago Tribune "Dewey Defeats Truman" headline, it is actually more significant today.

In retrospect, the polling techniques employed by the magazine were to blame. Although it had polled 10 million individuals (only about 2.4 million of these individuals responded, an astronomical sum for any survey), it had surveyed firstly its own readers, a group with disposable incomes well above the national average of the time (shown in part by their ability still to afford a magazine subscription during the depths of the Great Depression). The magazine also used two other readily available lists: that of registered automobile owners and that of telephone users. While such lists might come close to providing a statistically accurate cross-section of Americans today, this assumption was manifestly untrue in the 1930s. Both groups had incomes well above the national average of the day, which resulted in lists of voters far more likely to support Republicans than a truly typical voter of the time.

Today, Gallup is doing something similar, by relying on landlines. They say they've decreased the sampling using such, but when one poll is the outlier among many, it's wrong.

Come Into My Parlor, Said The Spider

A minor, but recurring, theme of this endeavor is the fragility of advert driven business, web division. As most know, virtually everything on the web is either advert based, or shopping (well, by most accounts porn is the majority by far). Deriving one's revenue from folks who may or may not want to buy something else has been debated for decades. I first got a glimpse when I read "The Hidden Persuaders" by Vance Packard, either high school or junior high (now often called Middle School). It was written in the late 50s, and I read it sometime later.

Over my Panera coffee this morning, there was this article about click based adverts in my dead trees version. Made some notes, thought about posting about it, but... nah. Then, kaboom, Google pooped the bed!

What to make of all this? First and foremost: the advert based interTubes was always fragile. The 2000 dotbomb wasn't the end of things, although I do miss the Pets.com sock puppet. (Aside: I read some wag who opined that CEOs are so overpaid, because these MegaCorps could be run by a sock puppet. That was before 2008, of course.) As Mr. Telford found out, there always other, and oft times cheaper, ways to get people to buy your widgets. As I've written long before, Google is always one "innovation" away from irrelevance: a more effective (at least to the advert buyers) advert platform will do to Google what Google did to newspapers. Of note, Mr. Udell (another small businessman):

"The cost to get someone just to visit your Web site has, in some cases, become prohibitive. Something that cost $3 might be a no-brainer, but at $20 it becomes absurd. It's basic math, and if it doesn't add up, we won't do it." He said he planned to redirect some of his advertising dollars to print, television and radio. [my emphasis]

Google is trying to do, successfully, what Sun didn't do with java: leverage one product to sell something else. Sun expected java to increase its shift of servers. Google is trying much the same with Android; which is not java. We'll see. Google may have overpriced its advert platform. On the other hand, Yelp is reported to be tracking fake positive reviews, and other business espionage.

One wonders: will Google tweak its "algorithm" yet again to degrade organic matches? Do they really have any choice? For myself, I always use AdBusters, which I found when I was on dialup as a way to reduce bandwidth hogging. AdBusters causes some controversy, but I don't see the issue. If one has no intention of following ads, as I don't, why object to AdBusters? It isn't as though I'll start madly clicking away, while page loading slows to a crawl!

Make a widget and sell it.

Sherlock Holmes Plays One

Yet another dive into the deep end of the pool. Most folks interested in enterprise level SSD know about Violin, but because it's been private (unlike Texas Memory, it's newly minted, relatively speaking) the moolah folks and general public haven't heard of them much.

Get out your checkbooks. Hopefully, not another OCZ fiasco! No, I don't think so.

Lefty Loosy, Righty Tighty [picture fixed]

It's not often that I send regular reader off to some other post, but this is the exception. I'm going to explore the data referenced, just as I hope you will. There's more to quant than predicting the next 10 bagger.

For what it's worth is.R() is new to R-bloggers and already my preferred reference. You could do worse than spend time there. But don't leave Here, of course.

Just a note for Ubuntu users: you'll need to symlink gfortran-X.X to gfortran, if Hmisc isn't already installed (it requires fortran, of course and Hmisc will error out complaining it can't find the compiler). The build scripts I've run across don't look for these Ubuntu's (Debians?) versioned compilers. Also, and I haven't worked out the fix, on my 10.04/2.15.0 machine, the party legend doesn't fill, and the two legends are flipped. We'll see. Since the version in the post doesn't expand, it's really hard to read.

We'll Need a Bigger Boat

Why did/do flunked out math and physics PhD candidates go into finance/business admin/economics? The quants. Follow the money. They started in earnest in the 1970s, and became the Mongol hoards (or a school of squid?) which cratered the global economy a couple of years ago (we are past that, right?). Now that the pillaging is done with, Wall Street has turned off the money spigot, yes? No. See here. Not all that newsy, in that the previous year wasn't much different.

Between 2009 and 2011, compensation in the securities industry grew at an average annual rate of 8.7 percent, outpacing 5.3 percent for the rest of the private sector.

Note that the main data refers to 2011; we don't yet know what the numbers will be for 2012, but...

Some 48 percent of 911 Wall Street employees surveyed by eFinancialCareers.com said they felt their bonuses this year would higher than in 2011.

OK, so they've done so much better by the world's economy, they need yet more moolah. But here's the real problem:

Financial jobs accounted for nearly a quarter of all private sector wages paid in [New York City] last year, even though they accounted for just a fraction, 5.3 percent, of the city's private sector jobs.

One of the real conundra of the financial services sector is the assumption that it's driven by computers and quants and superior smarts. Put another way, in the industrial sector, wages have been falling in deference to capital. Earlier posts have discussed the fall in labor as input to production. In the finance world, not so much. With all the talk of double secret probationed HFT computers, it's the Boys in the Boiler Room who get most of the moolah. Moolah that, could, be used to buy plant and machinery for physical production. All that money just to partner savers with borrowers? My, my.

Nearly half of all revenue on Wall Street is earmarked for compensation; in 2009, Morgan Stanley, which was hit harder during the crisis than most of its rivals, found itself paying out a record 62 percent of its net revenue in compensation and benefits. That number has since come down.

I'm Melting!

Well, it really did happen, OCZ has crashed and burned. Time to muse on the effect, if any, on the thesis of this endeavor, that SSD on multi-processor/core/thread machines will take over the world. And that high NF databases will be the data store of choice.

Hmm. OCZ was never, despite its assertions, a player in the Enterprise Space, so there shouldn't be any direct effect. STEC slumped this morning pre-market when OCZ did its face plant, but has recovered. STEC does have its own CEO problem, of course, but does have a track record of producing Enterprise Space SSD that OEMs and end buyers really want. Fusion-io keeps on truckin'. The niche/private Enterprise Space continues on. The Enterprise Flash Appliance Space looks to be going well, at least according to Forbes reportage.

The loss, if it comes to that, of OCZ will most likely be felt in the Postgres/MySql world of SMB/web startups. These entities are largely populated by pimply faced boys who've never heard of Dr. Codd, or those that have, dismiss him.

In the months leading up to Peterson's "leaving" (and it's now reported that he's on the dole of OCZ for rather a while), OCZ touted its imminent release of "the first TLC" SSD. Didn't happen. Samsung's 840 (not the 840 Pro) is about to deliver, and has been tested, a bit, by AnandTech. The consumer SSD market is, it has always seemed to me, coelescing with the Big Boys, those that can survive and thrive in a race to the bottom. But then, I did predict that a while back. The boutique houses, that only assemble Other People's Parts, will either have to provide unique capabilities (STEC has it's own controller, so it doesn't count, either) or die. Darwin was right.

[Yes, if you read this earlier, the title is changed. I'd allocated that to a piece not yet completed. Too busy.]

Cheap at Twice The Price

High normal form relational databases provide the minimum byte load, and guaranteed integrity. Why have these qualities been so clearly ignored over the years? One excuse has been that normal form is too slow on conventional hard drives and uni-processors. Now that we have high quality SSD and multi-processor/core/thread machines, the technical complaints should disappear.

But my experience says that there is still an uphill slog in view. Why? Coders will remain obstructionist, since they're in self-preservation mode. Another piece today provides yet more examination of how this happens. Were humans rational, this sort of behavior wouldn't be tolerated.

It's an unfortunate reality that efficiency often goes unrewarded in the workplace. I had that feeling a lot when I was a partner in a Washington law firm. Because of my expertise, I could often answer a client's questions quickly, saving both of us time. But because my firm billed by the hour, as most law firms do, my efficiency worked against me.

Sound familiar? An unexceptional relationalist can build rings around a hoard of coders, but doing so threatens not only the hoard, but the (bureaucratic) bosses of the hoard.

...a measurement system based on hours makes no sense for knowledge workers. Their contribution should be measured by the value they create through applying their ideas and skills.

But, still, the coders circle the wagons, and insist that the olde ways are better. The bureaucrats who, ostensibly, manage them are happier with the bloat: the bigger the budget the greater the importance. Progress is wonderful.

The solution, it seems to me, has always been clear: engage in fixed price contracts, whether internal or external.

When Good News Happens to Bad People

Back in February, there was "Lies, Damn Lies, and the BLS", which you can find in any of the various incarnations of this endeavor. At the time, the numbers came out better than the pundits predicted. It happened again this week (for September), but since it is the second but last run of the numbers before the election (Election Day is Tuesday the 6th, while the numbers day is Friday the 2nd) with enough time to get all piqued up.

And did the Right Wingnuts get all piqued!!! To repeat: the data come from sample surveys, two not a single integrated survey. And if you read the fine print, as I suggested back in February, then you'll see that the estimates have plenty of room to waffle. I don't recall any articles in February that went into the details. Well, this time there is. Worth reading up. The on-line version is graphier, and therefore more useful.

As to the GOP stroking out: the BLS folks who make the decisions are Supergrades (here for explanation), who, in all likelihood, got to these positions of authority under BushII. If there's any conspiracy, it's among geeks who live in Fairfax County with a stay at home wifey and three kids in Christian School. I spent a decade working with those sorts (not at BLS, for the record).

Let's Have a Party!!

It qualifies as something of an open secret that lots of financial service (even actuarial) analysis is done in Excel. The revelation appears as snide comments in texts with some regularity.

ggplot2 has become (to my mind, anyway) the de facto graphing package in R. Recently, Hadley released an updated version, with themes getting the most attention. Today, David Smith (of Revolution Computing, and an Aussie, if memory serves) posted some comments on themes (referencing theme work done by Jeffrey Arnold), including this beaut:

or if you're an R user on the down-low, with that "classic ugly look and feel" of Microsoft Excel. (A better idea would be to just come out!)

More often than not, Americans use the phrase (in TV dramas quite noticeably) way, way out of context. One might wonder what they call Excel in Gay Paree?

Many Happy Returns

Today brings not one, but two, cautionary tales. The reason for doing quants, either for anonymous blogging or mucho dinero, is to deal with issues which can only be answered with data, not policy. I'm on record that policy trumps data every time, but with the proviso that the policy can be enforced *despite* the eventual collapse. Greenspan's crashing of interest rates was a policy which motivated The Great Recession; he almost fully admitted it after the fact. There was clear data that the collapse was in the making, but ignored by both policy makers (who wants to admit error?) and participants (who wants to be first to miss the boat?). Capital's need for real return, and the consequent need for monopoly, appears in disparate stories today.

To recap. The justification for real physical investment is to make, and sell, either more product at constant price or current/less product at lower cost. This productivity delta is the real return. The Great Recession(s) come about when fiduciary capital is placed in other fiduciary instruments, rather than physical investment. Returns on real estate, whether residential or commercial, can only come out of the income streams of the underlying entities, households or businesses. Residential housing provides no financial returns, in use; paying the vig either comes out of rising incomes (there weren't any during Bush II) or consumption shifts (the volume of moolah needed couldn't be supported by that, although some apologists asserted so). In almost all commercial cases, the same is true. One might argue that a Park Avenue address will attract more business than one in Alphabet City (do they still call it that? and is it still a slum?), in greater proportion to the rent differential; but I'll consign that to outlier status.

So, real return to real capital requires making more and better stuff. Banksters tend to ignore that. AnandTech has another Haswell piece posted today.

If all mainstream client computing moves to smartphones, and Intel doesn't take a dominant portion of the smartphone market, it will be left in the difficult position of having to support fabs that no longer run at the same capacity levels they once did. Without the volume it would become difficult to continue to support the fab business. And without the mainstream volume driving the fabs it would be difficult to continue to support the enterprise business.

There's more background in the text, but it amounts to this: Intel needs to keep its fabs running full blast to get the return on the cost of the fabs. In order to do that, it needs to produce chips which move like hotcakes. You sorta have to get it right.

On the other side of the world, we get the Chinese solar problem. The title: "Strategy of Solar Dominance Now Poses a Threat to China" in my dead trees copy, the title on-line is different.

But now China's strategy is in disarray. Though worldwide demand for solar panels and wind turbines has grown rapidly over the last five years, China's manufacturing capacity has soared even faster, creating enormous oversupply and a ferocious price war.

Trying to generate that real return? You betcha. Does it work, by default? Not hardly.

In the solar panel sector, "If one-third of them survive, that's good, and two-thirds of them die, but we don't know how that happens," said Li Junfeng, a longtime director general for energy and climate policy at the National Development and Reform Commission, the country's top economic planning agency.

We have to do something about that Ruinous Competition!!! Wind turbines? Same thing.

The Chinese government also wants to see the country's more than 20 wind turbine manufacturers, many of which are losing money, consolidate to five or six. "Wind does not need so many manufacturers," said Mr. Li, who in addition to drafting renewable energy policies is the president of the Chinese Renewable Energy Industries Association.

Capitalists continually assume that they deserve outsize returns, but every time they try it, chaos results. Will they never learn?

The modest cutbacks in production barely put a dent in China's overcapacity problem. GTM Research, a renewable energy consulting firm in Boston, estimates that Chinese companies have the ability to manufacture 50 gigawatts of solar panels this year, while the Chinese domestic market is on track to absorb only 4 to 5 gigawatts. Exports will take another 18 or 19 gigawatts.

The enormously expensive equipment in solar panel factories needs to be run around the clock, seven days a week, to cover costs.

Both Intel and the Chinese alt energy sectors are the poster children for fiduciary "investing": while fiduciary "investing" is faux, the return is largely controlled, in the short run (which is all they care about), by policy. The residential home builders made out like bandits, literally, while mortgage companies, banksters, and MacMansion buyers got the shaft. As always, one needs to follow the money. The Chinese alt energy companies (and the government) can't, or won't, find buyers for its shiny new toys. One might argue, and the government surely did, that investing in some fiduciary capital in product producing entities is better than investing only in infrastructure. Infrastructure, as MacMansions, is difficult in the same way: how does one impute (much less collect) real returns? For infrastructure, the return is explicitly societal. PhD candidates have been writing dissertations on the problem for decades. Eisenhower's "National Defense Highway System" was the earliest in my lifetime. The official name became "Dwight D. Eisenhower National System of Interstate and Defense Highways" under Bush I. What's it worth? Well, Ike wanted it because he saw the difficulty (to the Allies) caused by Germany's Autobahn; it was intended to be a network to move men and materiel during the coming wars. Just as DARPAnet was all about the military and turned into a commercial enterprise we call The Web. Who gets the return?

The Best Laid Plans

For those interested in how real world databases and operating systems get along (or not) in the post single-cpu, sorta parallel regime of today, there is a piece on lwn.net which is instructive. I had originally inserted the link, but it's not that simple. The link I used was supplied by a post on the Postgres/Performance email group. For better or worse, lwn.net is by subscription, but it appears that subscribers are free to link to content. So, go subscribe to the Performance email group (you should anyway, here), and you'll get the link in today's posts. It's got "20% performance drop on PostgreSQL 9.2 from kernel 3.5.3 to 3.6-rc5 on AMD chipsets" in/as title.

The discussion in the post revolves (that's a pun) around the inherent conflict in scheduling between an (multi-client) operating system and a (multi-client) database engine. While flagged by the PG folks, any engine which attempts its own scheduling would have gotten caught, to some degree, by the patch. Who said parallel was easy?

Here's a quote which describes the issue:

So what is different about PostgreSQL that caused it to slow down in response to this change? It seems to come down to the design of the PostgreSQL server and the fact that it does a certain amount of its own scheduling with user-space spinlocks. Carrying its own spinlock implementation does evidently yield performance benefits for the PostgreSQL project, but it also makes the system more susceptible to problems resulting from scheduler changes in the underlying system. In this case, restricting the set of CPUs on which a newly-woken process can run increases the chance that it will end up preempting another PostgreSQL process. If the new process needs a lock held by the preempted process, it will end up waiting until the preempted processes manages to run again, slowing things down. Possibly even worse is that preempting the PostgreSQL dispatcher process -- also more likely with Mike's patch -- can slow the flow of tasks to all PostgreSQL worker processes; that, too, will hurt performance.

Spinlocks have been around for at least two dogs' ages. And the description reminds me of IBM's AS/400 (now called something else) solution, and M$'s abortive (so far) effort to replicate it, winFS; in which the database engine and the operating system are merged. Those with even longer memories could be reminded of PICK.

Carnac Wins Again

A few musings ago, I speculated that the folding smartphone was the next big thing. It's just been reported that Apple has patented the basic notion; perhaps without need of an external hinge. Will the iPhone 6 have the form factor of the original Motorola Razr? Reactionary capitalism, at its finest.

Money For Nuthin, The Picture

So, what's the data look like? Among Real Quants, pie charts are viewed as stinky doo, whilst Suits love 'em. Figures. I'm not a big fan of them, not so surprisingly, but these data are simple enough, and the chart stark enough (whilst still being honest) that it can't be passed up.

You'll likely need to expand it to see the labels, although the two that mean something are legible.

Money For Nuthin

With all the moaning about the "fairness" of capital gains taxation, it's about time that this endeavor took a look, don't you think? There's both quant aspects and macro aspects.

On the quant side, is: what numbers matter? From the point of those who assert that capital gains is totally unearned income (as opposed to those who get itchy with the word 'unearned'), the bull stock market since March 2009 is the exemplar. Here's what happens.
- you buy shares of CNO (Conseco Insurance, though the name has been changed) in March, 2009 at $.26.
- you ride the rebound to, at least, March, 2010 and sell.
- you sell the shares, which reached $10.18 last friday.
- you pay, modulo other shenanigans, 15%.

That's a gain of $9.92/share, or a bit more than 38 times gain. For which neither you, nor that money, did ABSOLUTELY ANYTHING. One can say, "but my money was at risk. CNO could have stayed down to $.20 or even gone bankrupt." And that's true. You GAMBLED that the person who sold you the stock was stupid to sell so cheap. S/he, on the other hand, gambled that you were a fool to pay so much. NONE of the money went to CNO to run its business. You did nothing more than a horse plunger. You look at historical information. You assessed where CNO's (or whatever company) prospects were going. You looked at the general market. And you read J.K. Galbraith, "Financial genius is a rising market."

There are other forms of capital gains. After much exploration, here is table of the various asset categories reported by the IRS for 2007 (the latest). More than half is purely financial transactions. Gains on residences is, wait for it, 1% of the total. Read that again, 1%.

On the macro side, we have to assert either that the USofA is based on a progressive tax system, or not. If we are, then capital gains, especially with the likes of Bain shifting "income" into "capital gains" deserve no special treatment, particularly given how little capital gain is due to physical investment. It's just a financial manipulation.

Is that so difficult to figure out?

Listen to the Bald Headed Guy

As regular reader must have figured out by now, while I dearly love stats/quant, I've been skeptical of mathy staty economics/business ever since grad school when I was subjected to a bunch of flunked out math and physics Ph.D. re-purposed into assistant professors of economics. This was the mid-1970s, ancient history to most alive today. The Great Recession's seeds were sown that long ago.

In the movie "Taras Bulba", Yul Brynner in the title role tells his son that he must go live with the Poles to understand how they think so that the Cossacks can win back what they've lost. I feel much the same about "financial engineering", an oxymoron ranking with "happily married" as the apotheosis of irony. So, in a desultory way, I've been reading David Ruppert's "Statistics and Data Analysis for Financial Engineering". Today was the chapter on copulas (which, if one uses them, must be termed copulation, yes?). Wait for it: we sure got copulated by Wall Street.

Each chapter has a Bibliographic Notes, which is usually a bunch of reference to papers in the professional literature. However, here is a reference to a piece in Wired, by Felix Salmon. Boy Howdy! It was written in the early days of the Great Recession, Spring 2009, just before or after this endeavor went public. I was unaware of it until now. It is eerie, reading it now. I do disagree that the quant invasion happened in the 80s; it was a decade earlier.

To reiterate: my issue with Wall Street quants is that many (most? all?) have little understanding of either macro or micro economics. The micro folks believe that each actor is independent while the macro folks understand that such a view is naive' at best.

One of my favorite aphorisms (attributed to any Good Mother, and presented a few times already): "what would the world be like if everybody behaved like you?". This was said to misbehaving kids.

...the real danger was created not because any given trader adopted it but because every trader did. In financial markets, everybody doing the same thing is the classic recipe for a bubble and inevitable bust.

Conspiracy nuts would, perhaps did at the time, have fun with the fact that the inventor of this particular copula (there are a host of specific ones; copula is a general definition, so there are many variations) is a Chinese named Li. Was he sent here to crash the American economy? Only the time will tell. As of the date of the piece he had returned to China and was working in banking.

What Salmon doesn't get into is the specifics of why CDOs and CDSs came to be so loved by the Wall Street folks. Here, the answer is Greenspan and Dubya: they had crashed interest rates on Treasuries, leaving all that Chinese money and American pension money and what have you looking for greater "risk free" returns. As I've said more than once here, returns (real world variety) can only come from better production of and sufficient demand for the increased production of goods; there really isn't much point in making physical investment just to produce what you already do (modulo firing most of your employees, see Mother's Advice above). Home mortgages provide no such. Some within the economics profession have talked, for decades, about "psychic utility" and its measure in housing. Here's a piece which skewers it well and truly. If it sounds familiar, just delve into the early musings here and you'll find his arguments and more.

Returning to Salmon's piece.

...because an unlimited number of credit default swaps can be sold against each borrower, the supply of swaps isn't constrained the way the supply of bonds is, so the CDS market managed to grow extremely rapidly.

In other words, while, in my opinion Wall Street investing is really just gambling twixt buyers and sellers of stocks and bonds, this was very much a step further into wagering.

What Li, and any of the quants who bought his story, relied on was the truth of The Efficient Market Hypothesis. That is, those pricing both CDOs and CDSs were always the Smartest Guys in The Room. The problem here is simple: the financial engineering folks rely wholly and explicitly on some amount of historical data, and almost all of that data is some single time series. Imagine your local weather person saying that the day was dry and sunny, because her forecasting model said so, but never bothered to look out at the downpour in the parking lot. Such was the simplicity of the error made by the Wall Street quants. They wanted to, vampire squid style, suck ever more moolah from the saver-to-borrower stream, and there was all that Chinese money just itching for some place to sit. It was not in their individual best interest (see Mother's Advice above) to question the wisdom of the plan. One need look, as early as 2002, no further than the (median house price / median income) metric to know that someone was lying. House prices were soaring, but median income was flat. No amount of utility shifting could account for the divergence.

And Li didn't just radically dumb down the difficulty of working out correlations; he decided not to even bother trying to map and calculate all the nearly infinite relationships between the various loans that made up a pool.

It isn't as if all these mortgage based CDSs and CDOs were built on new and better production. They weren't. All that held them up was the incomes of the home buyers. Nothing else. Said home buyers might derive a whole lot of psychic utility from a McMansion they'd never dreamed they'd ever be living in. Psychic utility doesn't pay the mortgage. Some academic economists, going back at least to when I was in school, have questioned American's perverse "investment" in housing. Europe doesn't waste capital that way. We shouldn't, either.

The numbers are staggering:

The CDS and CDO markets grew together, feeding on each other. At the end of 2001, there was $920 billion in credit default swaps outstanding. By the end of 2007, that number had skyrocketed to more than $62 trillion. The CDO market, which stood at $275 billion in 2000, grew to $4.7 trillion by 2006.

God may not play dice with the world, but Wall Street is more than happy to play dice with other people's money. "Abandon hope all ye who enter here."

So there you have it. Just as I've described, but with some contemporary reportage. The results of TARP and the QEs? A soaring stock market in the face of economic stagnation. How can that be? Now that all that Chinese money and American pension money doesn't have AAA rated bonds (and their derivatives) to buy, where else to go? Yup, the squid. It's deja vu all over again.

Talk Like a Duck

Could the SSD revolution already be over? Last week, Western Digital (actually the Hitachi folks they recently bought) announced the imminent release of helium filled hard drives. This story doesn't mention size, price, or availability. The only numbers: rather than five platters, these drives will have seven. That's a 40% increase, and for less power.

Dum da dum dum. Short stroking a hoard of these might be just as cost effective as equivalent SSD. Don't know yet. But could be. Does it matter? In a sense no, in that the point of SSD, so far as I've been concerned, is that SSD was an obvious finger pointing to high normal form databases as the solution to both byte bloat and application performance. For single disk machines, RAID/10 isn't an option, of course, but the point has been database servers, where a RAID set versus SSD is a feasible trade off.

Like a Ton of Bricks

Somewhere in this canon (or the other one; way too lazy to go looking), I mused that the future of eTailing was retailing. The reasoning is straightforward: the cost of fuel, air and ground, will only rise. We either run out of crude or we substantially reduce how much we use so that we have enough air to breath. In the near term, it's unambiguous that rail transport is an order of magnitude cheaper than air, and nearly so for highway ground.

The advantage of brick & mortar retail is inventory; it can be moved in bulk, thus by rail and at substantial savings. At the time I mused, I didn't consider the shift back to brick & mortar to be all that swift. I didn't count on the folks at Amazon to have figured it out already. Here's a recent piece on Amazon's brick & mortar effort. Rather than you taking a short drive to Target, Amazon takes an almost short drive to you. Who said size doesn't matter? The losers? UPS and FedEx and the Post Office. They still do the last mile, so to speak, but not the bulk of the delivery.

Amazon's delivery of everyday objects needs to be fast enough and cheap enough to wean customers from their local stores. Yet it also must be economically feasible for the retailer, which is investing so heavily in the warehouses that it is barely profitable.

Some residents want more. "They want to be able to order something and then drive down the street to the warehouse and pick it up," said Rod Butler, the city manager. Here, just like everywhere else, shoppers dream of same-day delivery.

And what makes this better? The stated impetus was having to collect sales tax, but I just don't buy it. The cost of transport is significant. Just for yucks, one could use Amazon's build out as a test of the infamous Traveling Salesman Problem. For this instance, one need analyze the goods sold by region (cluster analysis sounds about right; one need define both the region and its goods' list concurrently), the size of the warehouse needed to supply those goods by rail (again, a region need have sufficient shipments, by value, to justify its boundaries), which would have to account for volume of goods (rail is effective for large movements). The most significant parameter is meeting demand lead time on the list of goods to be kept in the regional warehouses. The goal would be to meet some percentage, say 99.5%, of value shipped out of a warehouse to the region. Since these buildings, and if you read the links you'll find tax breaks abounding, still don't move much, definition of "region" is critical. Low volume/value would be shipped out of "central" stores, likely three; each coast and the midwest. The regional warehouses would be stocked by rail. I'd bet, but haven't got data, that the warehouses are quite near, if not on sight to, railheads.

Surprise, surprise. A quick search turns up this story.

Given the prime location in the Midwest with 4,700 miles of mainline rail track, three international airports and more than 11,000 total highway miles, it's no surprise that Amazon has already invested heavily in placing fulfillment centers there.

If you follow the Delaware link in that story, and then search for railheads in Middletown, Delaware, there's Norfolk Southern. Brick & mortar wins after all.

The Real Work is in the Microcodd

A while back, I noted that Intel is implementing transactional memory in its new processor. AnandTech has a piece explaining (no algebra!) how it works. What's amusing is that the piece adopts a RDBMS metaphor!

The root of the locking problems is that locking is a trade-off. Suppose you have a shared data structure such as a small table or list. If the developer is time constrained, which is often the case, the easiest way to guarantee consistency is to let a certain thread lock the entire data structure (e.g. the table lock of MySQL MyISAM).

According to Intel, using an application that previously used a coarse grained lock (like the older MyISAM storage engines of MySQL) together with a TSX enabled library should improve scaling spectacularly.

I wonder if Dr. Codd is smiling in his grave? When early RDMBS engines were being written, mainstream multi-processors were as rare as hen's teeth (this was about the same time as Connection Machine and such, but much of relational databasing was done on unix/AIX/HP-UX/etc. vanilla minis; well and a bit on the 370). I doubt many engines were parallelized, they just spun a lot (read the piece).

So, we have transactional cpu with large memories with fast SSD primary store. Can high normal form databases be around the corner? Enquiring minds want to know. They should be, of course.

Another Brick in The Wall

We don't need no education...

The Left Wing, and a bit of the Right Wing (sort of), are playing the education tune as the panacea to the "middle class is dead" problem. Alas, it can't work. The flawed assumption is that our (global) economy can absorb millions of folks doing well paid middle class work. The issue is the same as the robot ruminations of late: high value work is high value just because it's highly leveraged. That doesn't necessarily mean guild-like restrictions, but more fundamentally input-output; no matter the price, there just isn't the need for millions of middle class workers. Here's a quote about the iPhone folks:

Apple sent the entire 16-member design team to the award presentation and the entourage followed Ive on stage to receive the award.

So, to be clear, Apple moves 2 million 5s in 24 hours. 16 people created that device. Now, that's LEVERAGE. And it means that education isn't the solution.

It's the Distribution, Stupid!

Filching the Teacher's Apple

Continuing. How, then, to staunch Apple's crusade for world domination? A company run, until recently, by an avowed Buddhist? Somehow, the two can't really be true. But I digress.

You read it here first. Or, at the least, I've not read it anywhere.

The issued "solved" by the 5 is to put a reasonable form factor back into mobile phones, a conclusion not yet realized by mainstream pundits. They're all gaga about the iTunes and such. You do know that you never *own* any of those songs, yes?

What would provide a useful alternative? Well, a flip phone that's still all screen when opened. I believe it can be done. All that's required is a bit of smarter mechanical engineering (my uncle was an MIT ME, and claimed that all problems in products were solved through the MEs' efforts). There are lots of hinges in the real world that match up to within a ten-thousandth's of an inch. Corning would have to figure out Gorilla Glass such that the line between the two plates is a pixel or so. That shouldn't be a big deal. The bigger deal will be the boards. With two part chassis, some volume is required to accommodate the ribbon connector. With the solid phone, board layout is not so encumbered. But it would work out. The convenience of the self-protecting flip phone is not to be ignored.

May be Nokia?

New Day Just Like the Old Day

An earlier post, on the Apple/Samsung tiff, also mentioned my bewilderment at the original iPhone, specifically its size and shape. Five years on and, using the wide screen excuse (16:9 has become the de facto standard), the 5 is pretty much the dimensions of my old flip phones. I think that's what's called reactionary.

I Demand Transactional Immunity for My Testimony!

Intel's slides from the Haswell announcement are at AnandTech, here. One of the more amusing aspects of Intel's continuing CPU evolution is the increasing congruence with real relational database semantics. One wonders whether MicroSoft's integrated file system, cum RDBMS (just like the AS/400 of decades ago), might not be dead; just re-emerging as another Wintel effort? See, slide 14, 16. I wonder how the "eventual consistency" peanut gallery will swallow this?

There's also the ARM stuff; how the RISCy business will put Intel in a bind. Once again, Haswell documents that the real processor inside that CPU is a RISC machine. Has been for years. That Intel hasn't split out the hard core is something of a puzzlement. The knee jerk answer is that Intel has lots of secret sauce it doesn't want exposed.

Mail Time, Part 1

In the spirit of PTI, a basso voice from above calls, "Mail Time!" In today's episode, a month or so old article from ComputerWorld, dealing with tests of SSD and related issues.

...when targeted at specific applications, such as ... online relational databases, the costs to achieve the same performance with flash compared with HDDs can be vastly lower...

Now, if these so-called tech reporters knew enough about the RM, they'd press folks on the data reduction advantage of High NF and why they did, or more likely didn't, go that route. But, of course they don't. Just as reporters in any other specialty area, they're still generalists who don't know enough to avoid the BS.

Carnac, the Magnificent

Do you ever get that deja vu feeling before it happens? So today has gone. Early this AM, I checked my email, and a new discussion popped up from one of the LinkedIn groups I follow (Insurance, blah blah). The OP stated, among other things, that in the data warehouse world 1) the relational database was bad and 2) data storage was too much. This led me to comment, as one might expect.

Here is the text:

A- with multi-processor/core/thread/massive memory/SSD machines cheaper than a Starbucks' latte, stars and snowflakes need no longer be the crutch of DW. High normal form databases, in orders of magnitude fewer bytes than flatfiles (still the typical structure in Insurance, alas), stars or snowflakes, are just far more capable than folks who haven't accepted Codd as their saviour will admit. Get rid of the bloat, and the database just flies

B- IBM, vendor to the legacy stars, now has SPSS and Texas Memory in-house. SPSS is the SQL friendly stat pack (so is R, but that's another episode), and Texas Memory by far and away the most experienced and capable developer of SSD. PREDICTION: IBM/DB2 will now find Codd and promote high normal form databases, for the simple reason that they now can make mucho dinero with that meme.

That bit of wisdom was posted a couple of hours ago. So what shows up in my inbox a couple of minutes ago, but a posting on a Database Developers group:

Next DB2 Tech Talk: Get smart on Realtime Operational Warehousing

Join us September 11 to learn more about the benefits and advantages of realtime operational warehousing as compared to non-realtime warehouses and standard database apps. You will also learn about new features in DB2 10 and InfoSphere Warehouse 10, including query performance enhancements and Continuous Data Ingest that are designed with the realtime operational warehouse in mind.

Now, I have to concede that neither this flyer nor the announcement itself mentions SSD or Texas Memory or SPSS. But for a two hour return on predictive investment, and I'll certainly argue now that SSD and SPSS will soon enough be part of the discussion, that ain't bad.