Shared What?!

It can't be any fun to be an actuary any more. I read up the first exams' study guide years ago, but decided not to do the tests. McElhone's mantras, "the world is not linear" and "you don't estimate beyond the data", kept ringing in my ears. These days, when we get a once in a century event every couple of years? At the subatomic level, the universe is probabilistic. At the macro level, not so much; causes have effects, if only we pay attention.

There's a wealth of material to ponder in today's NYT "Sunday Review" and "Sunday Business"; I'll limit my musing to a piece on home insurance. The authors are from the Wharton School, and thus should have face validity. Not so much.

Here's where they go off the rails, so far as I'm concerned (a frequent theme here): they ignore community. Their theme is essentially this, household insurance in places of known higher risk should *alone bear the burden*. When I took the course in grad school, the title was "Risk and Insurance", and was conducted from the economics department, not the B school. The point of the course was to study the implementation of shared risk. The MBA crowd is solely concerned (modulo the occasional and optional Business Ethics course) with profit. That makes a difference.

First, premiums should reflect risk. This makes transparent the magnitude of the hazards one faces and could limit new construction in high-risk areas. Residents would be encouraged to reduce risk by getting discounts for, say, elevating a house or strengthening the roof.

This sounds fair, on its face. But there is a problem: if insurance companies (or public providers, for that matter) are allowed to exercise market segmentation at will, the process is no longer insurance, but prepaid consumption. The result of unfettered segmentation is to shift all of the cost (and source of profit) for any risk solely to those who are subject to that risk. Recall the recent "Your Good Mother" essay? The rich get richer and the poor whither. Insurance, from the point of view of economics, is all about shared risk, not individual risk. Individual risk is just prepaid consumption.

The justification for this predatory behavior is stated:

Those who live in nonrisky areas are subsidizing the choices of others. Take residential flood insurance. Most insurers refuse to cover the risk, so the National Flood Insurance Program, run by the Federal Emergency Management Agency, was established in 1968 with subsidized rates for those then living in flood-prone areas.

This approach is little different from allowing health insurance companies to ding subscribers with pre-existing conditions. If you live on the Atlantic beaches, you're far more likely to get hit by a hurricane. But those in Fly Over Land are far more likely to get hammered by baseball sized hail storms and tornadoes. Neither group can, just to avoid the condition, simply move to Safe Land.

The authors bury the point in an otherwise benign paragraph:

... the need to provide shareholders with an attractive return.

But shared risk is socialistic, or tribal, or communal. We all have some sort of risk associated with where we live. We can't just load up the SUV and head out for Safe Land.

How, then, to persuade folks from building in risky places? Is there sufficient acreage for a burgeoning population to live only where natural (and man made) disaster is less than average? Well, of course not. The arithmetic can't work. If all the folks in hurricane land and tornado land and earthquake land moved to Safe Land, the price of housing in Safe Land (wherever that may be, and likely in my neighborhood, as it happens) goes through the roof, so to speak. Not to mention the cratering of wages, with the influx of those needing employment. Back in the 18th and 19th centuries, there was more than enough room for the population, mostly subsistence farmers. We have since industrialized, urbanized, de-industrialized, and urbanized some more. The USofA population has more than doubled since WWII, those halcyon days of 90% maximum marginal tax rates. I suppose we could engage in the sort of back to the land re-education that Mao imposed. Doing so would certainly mitigate the risks of these Gray Swan events. And mitigate the rent seeking behavior of insurance companies.

Pulling up stakes and moving house (as the Brits put it) used to be just a matter of loading one's few possessions into a conestoga wagon and heading for Indian Territory to steal a few acres. Those days are long gone, and acting as if such action is the solution to any current problem is just plain evil.

We do need to make it more expensive to live in known dangerous places. But just as we have, nominally, a progressive tax paradigm, we should have a progressive insurance risk penalty. Those with McMansions on the beach should subsidize those living in shotgun shacks, not the other way round. Those who've been living in some risky place for some number of years, say 20, don't get dinged so much, but those who've entered recently (or in the future) when the risks have been clearly documented pay more. Local zoning and building codes are the major lever to pull to make building in vulnerable areas more, even wildly so, expensive. I don't know about you, but I don't see the commissioners of Miami Beach voluntarily curbing the building push. It was housing in Florida which was the epicenter of The Great Recession. Since increasing housing activity is viewed, even by Lefties who should know better, as the solution to The Great Recession, what to do? I highly recommend Carl Hiaasen's thinly veiled novels on the evil of development. "The trailers imploded, exploded, popped off the tie-downs and took off like fucking aluminum ducks." That's from "Stormy Weather", written in the aftermath of hurricane Andrew. You must read it, if you haven't.

It can be argued, of course, that wind insurance and flood insurance are sops to the slothful. That's the authors' point. Too simplistic by half.

Still, Don't Be A Dupe

Yet another dustup over SandForce and deduplication over at AnandTech. The fisticuffs are in the comments. What's still amusing is that the combatants still haven't figured out that dedupe is on-the-fly/ad-hoc 5NF storage. Ask them about relational databases, and they roll their eyes.

For example, if you have two very similar photos which are 5MB each, the controller may not write 10MB. Instead, it will only write let's say 8MB to the NAND because some of the data is duplicate and the whole idea of deduplication is to minimize NAND writes.

If you go and delete one of these photos, the OS sends a TRIM command that tells the LBA is no longer in use and it can be deleted. What makes SandForce more complicated is the fact that the photos don't necessarily have their own LBAs, so what you need to do is to check that the LBA you're about to erase is not mapped to any other LBA. Otherwise you might end up erasing a portion of the other photo as well.

While it's still up in the air whether SandForce controllers actually do dedupe, rather than just compression, the observation is correct. Not that any of these folks is able to make the mental leap. Kids! They never figure anything out until it's too late.

Your Good Mother, Part The Second

Along similar motivation of the Dee Feat series, herewith the second (for the sake of settling on a number) in our Good Mother series. This report in today's news reminded me of this other device, reported on by "60 Minutes" among others.

So, as Your Good Mother constantly reminds you: "what would the world be like if everybody behaved like you?" (She isn't necessarily literate enough with to know which preposition to use!) Once again, particularly from the Hawaii story, it's quite possible to bring overall degradation to an economy (and its society) by going all in on Rand.

At what point is the individual, especially acting in consort with some number of others, obliged to temper actions based on externalities? For those who worship at the feet of Adam Smith (the real one), the question is a bit more piquant, since his most famous text assumes that there are no externalities! Quants, working as they almost always do, for individual actors (in the context of some market, not humans) are described by micro-economic arguments. Again, externalities are assumed away. Not a good, or honest, thing to do. One might strongly argue that The Great Recession came about just because the quants (at the behest of The Suits) took explicit advantage of the fact that their companies would be protected from meeting the demands of the externalities of what they did. The micro point of view, of course, denies culpability for externalities: "those on the other side of the transaction are obligated to caveat emptor". Again, if everybody does "it", there's no way to mitigate the externalities except by taxing the community at large. And the bad actors get to keep their gains.

Another example, charter schools. Evidence is piling up that charter/private schools, particularly those catering to "inner city" students, don't do any better at educating, overall, than the public schools from which they rob moolah. As with spontaneous cancer remissions (every type has some, small, number of long term survivors), a few success stories of individual students can be cited. I'm always reminded of my local nutball, Linda McMahon, who propounded that, since "she" (actually, her equally nutball hubby) had become a millionaire, everybody else could, too. What such folks neatly elide: if everyone were a millionaire, a loaf of bread would cost everybody $10,000 (give or take a few thousand). Just because one can cite a single success from some policy, doesn't make the policy intelligent for the community. Not that the Rand-ians care about the community.

In all three cases, we have the problem of: divide and conquer (the Big Uns surreptitious way to win), where only those who can afford the individualized alternative get the good/service. The earliest example, in recent history, was White Flight in the 1960s. Suburban communities morphed into wealth by denying the common good; one might even see this as the continuation of Manifest Destiny (a stretch, but not by much). Many Christian schools are rooted in such communities; they don't get the irony, of course.

Now, I'm not a fan of The Power Grid, and never was. The first great blackout happened when I was a child, and I lived in it. To my surprise, the wiki piece does mention the point germane to this missive: Holyoke, an abutting town, had its own standalone powerplant (you can see it from Route 5 or I-91) and got along just fine. The justification for The Grid is load/demand balancing across time/space. I'm not sure of the physics, but transmission loss is not insignificant (the Bloom Energy piece states that avoiding said loss is a meaningful part the advantage). So far as Your Good Mother is concerned: what should be the unit of community where externalities exist? For power, it would seem to be the community (city/town). Holyoke wasn't, and isn't, like the Hamptons with hordes of upper-middle class folks fleeing the hoi polloi. Mostly hoi polloi, in fact.

One could even extend a power grid analysis to the au courant meme: The Cloud. I'll leave that as an exercise to The Reader.

Inbreeding Isn't a Strength

Well, the story is starting to emerge. Here's a version from The Atlantic. Toward the end of the piece we get this:

"A lot of the software is kind of late. It's looking ugly and I go out on this Field call," Kunesh remembered. "And people are like, 'Man, we should fire your bosses man... We gotta get the guys from the DNC. They don't know what the hell you're doing.' I'm sitting there going, 'I'm gonna get another margarita.'"

The problem: it was the DNC guys who screwed the pooch in 2010, and did little to stop the tsunami in state legislatures then or this year. We now have Yankees and Johnny Rebs, in spades. It's just now Coasts versus Crew Cuts. The lackadaisical DNC gave us this. I met with those DNC guys. They let 2010 happen. They aren't the ones to follow.

And then they provided a separate way for the Analytics people, who had very specific needs, to get the data in a different form.

Of course, if they'd used Postgres and PL/R, they'd have had it all together.

Recall from Triage, the observation that getting the response data would be the most costly (and time consuming)?

With Davidsen's help, the Analytics team built a tool they called The Optimizer, which allowed the campaign to buy eyeballs on television more cheaply. They took set-top box (that is to say, your cable or satellite box or DVR) data from Davidsen's old startup, Navik Networks, and correlated it with the campaign's own data. This occurred through a third party called Epsilon: the campaign sent its voter file and the television provider sent their billing file and boom, a list came back of people who had done certain things like, for example, watched the first presidential debate. Having that data allowed the campaign to buy ads that they knew would get in front of the most of their people at the least cost.

And so it was.

This is near the end of the piece. Not the last words, but the last ones that matter:

And losing, they felt more and more deeply as the campaign went on, would mean horrible things for the country. They started to worry about the next Supreme Court Justices while they coded.

If that doesn't connect with you, shame on you.

Such A Touching Tribute!

A bit more than two years ago, along with the first iPad, I considered the future of data stores and tablets. We now have Windows 8, in its three(?) suits of clothes. Back then I mused:

BCNF schemas, doled out in "just the right size" to Goldilocks, is the way forward. Very cool.

So, today we get this view of Win8, and it does sound rather familiar! This writer is new to SA, and on balance does pump pieces. On the other hand, the transition to pickable form factors is inevitable. As I pointed out back in 2010, the pick list implementation on tablets has been around for at least two decades. It may be new to most consumers, and Fortune X00 office drones, but it isn't new to a large swath of software developers.

The victors will be those that embrace the core meme of this endeavor: multi-processor/core/SSD machines on *nix running an industrial strength 5NF database. Only such systems have the capability to serve up small bites of coherent bytes of pickable data, without a tsunami of client (and only for one language) code to sort out the alternative from bloated flat-files, database resident or not.

The Apple of My Eye?

And now for something completely different. In which an attempt is made to forecast the future of Apple Computer. The share price has been vying with Lindsey Vonn for fastest time. Why might that be? And what might it mean?

Apple's "growth" has been driven by "new" products, what Vance Packard termed "planned obsolescence". They've been good at it in the Jobs-ian past. There's no indication that they'll continue to do so. Just what "new" product is waiting to enthrall the Apple Fan Boys? If the answer is, "we ain't got none", then Apple's growth goes poof.

Apple invented none of the platforms (MP3, laptop, smartphone, tablet, desktop) that have driven this growth. They did tweak their products in those platforms. All existed before Apple's entries. In some of those cases, Apple (purposely?) lagged the cutting edge to decipher the profitable high-end, MP3 for sure. Apple gambled that wireless network hardware would catch up soon enough, just as MicroSoft gambled that the next PC hardware would be just fast enough to support the next bloated version of Office.

What *existing* platform(s) are out there, for which Apple doesn't have an entry? If the answer is "none", then Apple has no option but to be MicroSoft/HP. They could invent a new platform, but that's something *not even Jobs did* (Woz made the machines). Right now, the answer is "none" to both questions. Feel free to name any.

Moreover, Apple's products have, since the original Mac, been intentionally high-end and small shift. That Samsung, et al, shift more parts isn't paradigm breaking; it's the norm. Samsung is MicroSoft/IBM et al.

Apple has benefited from a psychotic patent system which permits any random thought to be moated. And we now see Apple morphing from maker to taker; suing everybody on the planet. A patent on rectangle with rounded corners, indeed.

In sum, there is reason to believe that Apple has gone to level flight. Since it eschews high volume, low unit profit for low volume, high unit profit, can it continue to grow? No. While population growth continues on the steep end of the hockey stick, the number of households which fit Apple's economic criteria isn't. Continuing wealth and income concentration take a big bite out of what would otherwise be a burgeoning upper-middle class, in both the USofA and Europe. Japan continues in its lost decades (yes, now plural), and China's macro-policies of exporting poverty don't help Apple much either. Apple needs an expanding, global, upper-middle class to support its growth. Even if it does stumble onto a new platform. Whatever happened to iTV? What additional computerized activity can be made mobile? Address those questions with positive answers, and Apple has new parts it can shift at its traditional very high margins.

Blind Date

This endeavor has talked about high normal form on multi-processor/core/SSD machines from the beginning. On occasion, the complaint that such "complicated" schemas are not practically updatable, given existing client code, comes up. One hears this from Agile Zealots, who still want to write dumb data/smart code just as their granddaddies did. My answer has always been that "real" base tables are hidden, with the named "tables" being views. Updating, through joins for example, is handled with stored procs.

Turns out that SQL Server, at least, implements triggers which get around the updatable view problem. Michael Swart talks about it.

But When Would I Use This?

I thought of the perfect use case. This strategy helps with SCHEMA REFACTORING.

With the I/O capability of these current machines, the cost of maintenance drops considerably. Think of it as OO inheritance for databases. Or as Bill Gates was famous for saying (he did, didn't he?), "let the new hardware fix it".

For Your Listening Pleasure

Regular Reader is well aware by now that I've less than sanguine view of Kiddie Koders. Too many. Doing the wrong thing. And such.

Revolution Analytics runs webinars, which this week is a presentation from Deere. I was expecting a commercial for Revolution's closed source R implementation, but it isn't. What it is, is a surprisingly sober discussion of stats, analytics, and forecasting. There's no R code discussed directly, rather Derek Hoffman discusses the use of R results in Deere. He has a, in my view, jaundiced take on data science/analytics hype, which runs through his discussion and the Q&A at the end (some pointed remarks about Wall Street quants that could just as easily have been read here). You should run the slide show first; the YouTube video is way too fuzzy to be legible.

A by the way: Deere is a DB2 shop (with other less worthy databases). He doesn't say whether it's z/OS DB2, but I suspect it is.

Jim

We begin, again, with lyrics:

You don't tug on Superman's cape
You don't spit into the wind
You don't pull the mask off that old Lone Ranger
And you don't mess around with Jim
-- Jim Croce

More dirty laundry explaining the Romney Bubble (Yahoo! feed):

"As a result," says Crawford, "they believed that the public/media polls were skewed" in Obama's favor, and rejiggered them to show Romney with "turnout levels more favorable to Romney." In essence, Romney "unskewed" the polls, mirroring widely mocked moves by conservatives to show their candidate with a lead, epitomized by the now-infamous website UnskewedPolls.com.

Much the same as The Great Recession implosion, the Boys In Power decided to invent their own reality. They believed their propaganda. They were convinced that they could re-make the world into their image. They learned from Karl Rove.

That's not the way the world really works anymore. We're an empire now, and when we act, we create our own reality.

Can there be anything like a Right Wing quant? Put another way, is it possible for those attached to the Right Wing (including, obviously, Wall Street) to be based in the data? The evidence, so far, is No. These Wall Street quants ignored the most basic of data driving their MBS, CDO, CDS, and other derivatives (income/mortgage ratio). Incompetence of the highest order, but The Boys In Power wanted what they wanted, and their quants were happy to say Yes. Empires fall. The Right Wingnuts have fallen. Could they have won? Not without turning their back on the Tea Party and Rand-ians who have co-opted the party.

Reagan co-opted the so-called blue collar middle class, through the brilliant stratagem of convincing enough of them that, while they were Chosen People, their neighbors were freeloaders who deserved to starve. Steve Jobs was often accused of having a reality distortion field; those that fudge data in order to appease The Boys In Power today, sometimes end up on the losing end of the bargain. Not always, of course, only when the data are important. If they are truly attached to the agenda of the clients. In the usual scheme of things, the quants get their cut up front, of course. I suspect that the devisers of Orca made out very well. If they had built a system which accurately reflected reality (in the data), they would have lost the contract, since it would have contradicted the desires of The Boys In Power. Just as the Wall Street quants were only following orders.

Ann's Horse Didn't Win, Either

My, my. A recent missive here asserted that Gallup was wrong. Gallup was wrong right up to the end, although according to 538, they had inched toward Obambi at the end. Didn't quite get there. But that's not today's topic.

Turns out, Willard had his version of Triage!! I know nothing, nothing I say, about it. Except what I read in the newspapers.

The success, though, would have to depend on volunteer troops united by a Web-based smartphone app.

Is this the Literary Digest, all over again? Deja vu? I haven't found the original (I'm not on their mailing list, so...), here's the PR (from yet another Right Wing organ) referenced above. Figures that the Right Wingnuts would chose a predator as the project's nickname. I suppose, in their over-weening adolescence, they thought it was cool.

OTOH, there's this (kind of like Triage, don't you think):

For example: if we happen to be down in a state at lunch time, we can pinpoint exactly what is causing it. So, if we know we're going to win X state by 3 points, let's move our resources to Y state, county. In sum, Project ORCA will give us an enormous advantage by being able to know the current result of a state.

As described in the Triage piece, the key to making these kinds of systems work is timeliness of data. I suspect that these webby quants had never worked campaigns, spent time in a political city (my first two were Boston and DC), or understood that "dashboards" are cute, but GIGO.

In the primary, we learned it was difficult to be working from Boston and really affect voter turnout in the states. It was disappointing to receive data later and realize if we had access to that data earlier, we could have done something differently and affected the outcome.

Kind of late in the game, though. It's not quite as silly as my own state's Linda McMahon, who managed to get on the ballot more than once (as did Murphy). It seems that there were other parties officially on the ballot, which motivated McMahon to run ads in the last couple of days with talking head "real people" saying that they'd vote for Obama, and ..... Linda (as she termed herself, I suppose to offset the images of burly half naked men clobbering each other with metal chairs).

Another key component to Project ORCA is state-of-the-art dashboard. For the past several months, a "brain" has been built into this dashboard and it will take in, analyze and recommend actions on the millions of pieces of incoming data. In the fast-paced environment of an Election Day command center, having this programmed "brain" will alert decision-makers to key findings and suggest reallocation of resources.

Deja vu. That's the essence of Triage. But far too late in the game.

Here's the original, quoted liberally in the Yahoo! feed. It must have hurt. When I lived in Washington, there were a bunch of local papers in the Virginia and Maryland burbs. Thanks to WikiPedia, now I know why I hadn't heard of the Examiner: a mini-Koch bought them up some years ago and turned them into a freebie Washington Times. Mein Gott! Dat's gotta hurt.

Later, another aide said Orca had pretty much crashed in the heat of the action. "Somebody said Orca is lying on the beach with a harpoon in it," said the aide.

You know, if only they'd built Orca with R. Open source and quite powerful. Kind of a community organized to analyze data. Turns out a community, once motivated, can kick Roark's butt.

The Big Uns Win

Decades ago, after grad school and in a sort of job, I found much more time on my hands in the world of work than I'd had at university. Grad school is very much like being an indentured servant. I had lots of time, but not a lot of moolah. I ended up pursuing various phys. ed. activities, the likes of which never occurred to me earlier. One was the study of karate, with a Greek fellow in East Boston. One of his observations has stuck with me ever since: "the good big uns always beat the good little uns".

In recent months, and recent discussions in various places, I've asserted that OCZ syndrome is endemic to consumer SSD business. Just as there are 2.x HDD companies, SSDs (in the consumer venue) will devolve into the same oligopoly. The Big Uns will drive out the Little Uns in the race to the bottom; there is no meaningful performance gain from bespoke parts in a consumer SSD that Joe Six Pack ("It boots Windoze 8 in only 30 seconds!") will deign to pay for, so the big silicon houses will triumph. Intel might be a Big Un, but it has just revealed that it, too, wants to move into The Enterprise Space.

AnandTech reveals Intel's new controller, and its interest in being an Enterprise SSD vendor. Those who've followed the SSD saga, here and/or elsewhere, for half a decade or longer may remember the X-25E: an SLC "enterprise" SSD. Didn't work out so hot. STEC was riding high, and serious about enterprise. STEC is, perhaps, circling the drain. The whole notion of Enterprise SSD is in danger of disappearing. The first, in permanent place, quote on this endeavor may be coming true, some years later than expected.

The direct flash appliance, from IBM/Oracle/whoever, may soon obsolete out the spinning rust semantics. Intel's new SSD may be too little too late. On the other hand, when the Sandforce and G3 (really, just tweaked G2) "new" drives appeared about a year or so ago, many speculated that Intel had folded its tent. Guess not.

But based on this story, Intel isn't getting past STEC, despite its troubles, or the appliance approach from IBM/Oracle/whoever. This is a prosumer drive. It specs as a good database developer machine drive; an economical surrogate for the performance an application would get on Real Iron (well, sand). I was just about to pick up a Samsung 830, cheap, since the 840 is starting to appear. Not so much. A 400 may be too much to resist.

Danger, Will Robinson

Revolution Analytics, in the person of David Smith, has an active blogging presence; this link is to a recent presentation (yes, toward the end it's a commercial). I've just added a quote from the intro. For the record, run the slides, and note 8. In the database business, in large companies, there exists the not-too-functional marriage of business analyst and data geek. What happens, all too often, is that the BAs have far too much authority. They are in the Danger Zone! Then, of course, have a gander at 24. Sound familiar? One might interpret the imperative as "keep the logic in the data, not the code". That's my point of view, of course. A particular implementation, not mentioned in the presentation, is PL/R with postgres. Postgres isn't mentioned at all (if you don't count databases grown from PG).