Where Have All the Returns Gone?

Not to belabor the point too, too much; but if you still doubt that real returns in anything compute related are a vanishing species, just have a look at this Apple slide. Read the whole page. Poof!

From a later page:

That's right, I still have my old PowerMac G5 Dual 2.5GHz (upgrade from my original 2.0 model). It's interesting to note that single threaded performance has only improved by 2.8x over that 2.5GHz dual G5 machine from around a decade ago.

And now a word from Amdahl:

The first thing I noticed while running this test is how much the workload can impact CPU core utilization. Even though I was dealing with a substantial 4K project, only portions could spawn enough work to keep all 12 cores/24 threads busy.

All that investment of billions and billions of dollars just to ... what? As the Red Queen said, "Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!" (Pulled from the wiki)

An Epidemic of Kissing Disease

How old would you think the word 'monoculture' is? My guess, before I went out to discover, was at least since Stephen J. Gould. Turns out, that's wrong, that is, off by a bunch.

According to here, it appeared in the OED for the first time in 1901. Which likely means it was in use during the 19th century.

Those in the *nix community have been vocal in their warning that M$ DOS/Windows dominance leads to the usual casualties of inbred stock. What they've been less vocal about is the monoculture at the hardware level. While one can build linux from source, and if you can verify that all the source is clean, then you're somewhat outside the box. But that chip is still X86.

So, have a read of the latest paranoid's jeremiad. One might wonder how the 'strict constructionists' abide such? Just because the Floundering Fathers didn't have PCs and innterTubes, does it follow that only the means of communication and domicile that existed in 1789 are protected from Government (and their corporate minions)? One might conclude that, in the case of personal freedom, the Right would eagerly adopt a 'living document' viewpoint. Or could it be that they believe in the sanctity of government, after all? Obama has, just on the basis of admitted behavior, done what the Right Wing wants. The crunch will come when any level of protest gets one labeled 'terrorist'. We've been there before, and it wasn't pleasant.

Who is being protected from whom?

Where's a Robo Cop when you need one?

Apple, My Eye!!

Here's a piece about the new Mac Pro, with this sound byte:

There, Philip Schiller, Apple's top marketing executive, seemed to go off script when, after showing the Mac Pro's striking design, he said, "Can't innovate anymore, my ass."

What Schiller seems to forget, or is too young to know, is that Apple tried this "unusual" design aesthetic game before, and called it The Cube. As Steve said, "steal". This time, Tim is recycling Steve.

And you'd better like it:

With a height of 9.9 inches and a diameter of 6.6 inches, the 2013 Mac Pro has no room for expansion via PCI cards or internal storage devices.

At least The Cube had a real cpu. Bombed like a three-pounds-of-chili fart, though.

Dig a Hole to China

As mentioned in these endeavors over the last couple years, on occasion, it is obvious that the Chinese experiment in social Darwinism must needs assault its economy in much the same way it did the West's. That Giant Pool of Money is still out there, getting bigger, and still demanding high return on minimal risk. That there ain't no such thing as a free lunch matters not to those who wish to live well on moolah alone. Daddy Warbucks, at least, actually made some stuff. Today's banksters merely suck moolah from the fire house aimed by the savers at the borrowers. Never forget that it was a Chinese, Li, who foisted the Gaussian Copula on us. Financial quants are driven by visions of finding the Ultimate Loophole in the system, that crack in the dam holding back all that moolah from their hungry maws, and Li provided, what looked like, that loophole. No, it is way too Byzantine to conclude that Beijing sent him in, "Bond, James Bond" style, to wreak havoc on the hated West. Or is it....?

With the West's fitful (can you say: "neutering Volker"?) attempts to bring the banksters to heel, all that Chinese money is now being directed inwardly. With the expected result. I've mentioned the "60 Minutes" report (revealing the ongoing real estate fiasco) from a few months ago, and much earlier stories from print sources.

Today brings us another. Don't be surprised.

I long ago forgot where I read/heard it (you can find it in older essays), but the following semi-quote about sums up the cynic's view:
"One hears from CEO types about how much work it is to run these corporations, but they never seem to want to take on failing companies. The ones they do run could be just as successfully run by a sock puppet."

Which brings us to:

Yao Jingyuan, the former chief economist at the state statistics agency, said ... "With this kind of operational model banks will continue making money even if all the bank presidents go home to sleep and you replaced them by putting a small dog in their seats."

The piece ledes with the punchline, as any cub reporter has been taught to do:

China's financial system is in danger of becoming too big to bail out.

My, my. All that money, and little to do with it, besides build yet more condos.

Official bank lending has more than doubled since the global financial crisis, growing nearly twice as fast as the overall economy.

Without all those Western mortgages to soak up the moolah, China is generating such internally. Good luck with that.

"The chains of lending and borrowing can be long, just like the securitized subprime mortgages. The result can be devastating."...said Yu Yongding, a senior fellow at the Institute of World Economics and Politics of the Chinese Academy of Social Sciences...

Stating the obvious:

Savers have had few alternatives to banks until very recently: Real estate prices are already stratospheric relative to incomes...

Ya think??? Time to copulate.

Cheap at Half the Price

An interesting position (not very many, alas, these days) came by. It's at a .org in NYC (yummy!! Letterman four nights a week!). But they say they prefer SAS. Which got me to wondering, yet again, why? This is an old .org with, possibly, some deep pocket benefactors. But, still, why SAS? Coming up with an exact price for SAS/seat doesn't seem to be available on the SAS site (or it is well hidden), but various posters conclude that base SAS plus graphics and such is $15K.

What you get for the $15K, that standard R doesn't give you, is the ability to do RBAR stats from a dumb-pick set of menus. (But they ask for a programming sample? Cognitive dissonance, in the extreme.) BFD. For $15K (or say, $45K for a three year machine), one can have 128G/xeon machine ( here's one for a bit under $12K ). Needs a monitor, and linux is basically free, along with R, and even a decent SQL editor (SQuirreL), although I've bought and used both AQT/windows and Aqua/linux.

I have a suspicion that this .org has an agenda. We'll see.

Mikey Likes It

With all too infrequent frequency, one finds a coder-centric babbler who gets RDBMS. This one (and it's earlier entry) are well worth reading (not hot off the presses, but a link in this Artima post). Nearly all of it is old hat to relationalists, but, I guess, still novel to kiddie koders. Oh well.

In the earlier set, is "2. Processing data in Java memory" wherein he advises:

... So in fact, by moving OLAP to the database, you gain two things:

Simplicity. It's probably easier to write correctly in SQL than in Java

Performance. The database will probably be faster than your algorithm. And more importantly, you don't have to transmit millions of records over the wire.

I have always added: "and doing the work in the database means the data is agnostic to client platform, OS, language, or anything else; only that it supports sending SQL requests (for any definition of 'send')".

A number of his 20 recommendations boil down to this: do as much processing on the database, and leave only formatting and input capture to the client. With (nearly) universal 4G bandwidth, even running VT-X00 style client/server apps is now feasible; populate them thar pick-lists in realtime, pardner. Talk about disruptive technology!!

They, the koders, won't do it of course, since they view the world through Dollar Colored LoC glasses, and ceding to the database that which is the database's reduces their LoC landscape. "You'll have to pry this keyboard from my cold, dead hands!" Yeah, I will.

Do You Do Windows?

Regular reader may not remember, but a few years ago there was a bit of a stink over the fact that bank ATMs were being switched out from OS2 based to Windows based.

So, now we know that Target's card customers just got siphoned up. Guess what OS that POS (in both senses of the acronym) runs? Yup.

"Each of our server endpoints and each of our 5,400 POS registers has a System Center Operations Manager agent installed. That way, we can ensure the checkout experience for our guests remains fast and efficient."

That's a M$ page, and it's sort of surprising they haven't taken it down.

Numbskulls On Parade

All you need to know about American Intelligence: the NSA runs Windows. And worries about computer problems. Talk about positive feedback.

What Hath Quant Wrought?

Today's Business section of the NYT has a host of malfeasance on display. A cornucopia of greed and punishment. But that's not what I came here to talk about. Jesse Eisinger has a DealB%k piece, and he's got some reporting of research which illuminates.

What was the proximate cause of The Great Recession? The answer largely depends on which end of the political/economic spectrum one sits.

The Right tale goes: "It was all those poor folks tired of living in shotgun shacks (and wanting McMansions) who came to the overly solicitous and naive` mortgage companies and banks demanding oddly structured subprime ARMs, the parameters of which they dictated to the naive` mortgage companies and banks. Who, being naive`, reluctantly devised such loans. Of course, the loans eventually went South, victimizing the mortgage companies and banks."

The Left tale goes: "The Banksters (mortgage companies, banks, rating agencies, and the Trilateral Commission) set out to gut the 99% and get rich in the process by creating mortgages which enticed the naive` poor folks into believing they could leave their shotgun shacks for McMansions. When said mortgages eventually collapsed, the Banksters kept their ill-gotten gains, and the poor folks retreated to their shotgun shacks, now paying more in rent than they did before the whole sorry tale happened." (Aside: private-equity, hedge funds, and God knows who else are slurping up housing and becoming absentee corporate landlords. That will not end well for communities.)

The truth, to the extent that anyone can be objective, lies mostly with the finance industry/sector growing into Jabba The Hut. One of the earliest themes of these endeavors is that Greenspan is Patient Zero in the epidemic. By crashing interest rates, he set in motion the effort to generate other vehicles of "risk free, high return". There are no such vehicles, of course, but since finance was about as unregulated by 2001 as it's ever been, there was no adult driving the train. "Let's see how fast we can make the choo-choo go, Mary!!" And it went fast.

While I can't claim to be the first to suggest it (although my recollection is that finding others, in the pundit class, who expressed the notion followed my coming to the conclusion), the problem with the financial engineering brigade, i.e. raptor-quants, is that they don't want finance to be simple and boring. Convoluted and opaque is better. And the reason it's better is that profit from finance comes not from value added, but sucked out of the moolah stream twixt savers and borrowers. Better to hide the shenanigans.

Which brings us to Mr. Eisinger today. He reviews some of Jack Lew's, the newish Treasury secretary, earlier pronouncements, along with data produced by outside researchers. It is these concrete facts which are of interest.

The way to really solve "too big to fail" is not by tinkering with the existing system, which leaves the great and fundamental problem still with us. The economy has become overly "financialized."

GE's profit percentage from finance had reached 50%. Other companies saw a quick buck, and took to shuffling paper and sucking moolah from the, what looked like, tidal wave of moolah to be processed.

Historically, finance's share of the economy has been at about 4 percent. Today, it's about twice that. And the peak occurred not in pre-bubble 2007, but in post-crash 2010, at just under 9 percent, according to research from Thomas Philippon of New York University. That represents a shift of more than $600 billion of wealth a year, as Wallace C. Turbeville, a former investment banker-turned-financial reformist, has pointed out.

The result is obvious:

Despite technological innovation, finance costs more than it used to, even though prices have fallen for things like trading stocks.

The Banksters suck their profits from the stream. Even Eisinger has the gonads to be plain:

The financial sector has become a self-sustaining perpetual motion machine that extracts money from the rest of the economy. Shouldn't it be a goal of society -- Mr. Lew's focus -- to restore the financial industry to its traditional role as an intermediary between companies that need capital and savers who have it?

In simple words: finance should be simple, dumb, and transparent. And cheap. It is after all, little more than Marrying Sam, putting savers and borrowers together. All the fancy quant does is extract ever more from the stream for the quants and their bosses. The quants would be more productive in marine biology and such.

Research from Professor Philippon shows that financial activities have gone up in the deregulatory era, and now cost about the same as in 1900, the last Gilded Age. In other industries, like retail, technological innovation has led to lower prices and therefore decreased the size of the sector. In finance, the opposite happened.

The tail is wagging the dog.

(Go to the web page to follow the links to the underlying research.)

Unite Us All

Perhaps another shoe has dropped. Rambus and Micron have signed a deal.

Here's an excerpt:

and (ii) a seven year term patent license to other memory and semiconductor products.

It was just about two years ago that Rambus bought Unity Semiconductor, whose only significant asset was CMOx memory, a non-volatile type which is (said to be) more robust than NAND/NOR. Unity and Micron had agreements at the time. Not much out of Rambus in the intervening time. Looks like the effort isn't for naught. If CMOx does work as advertised, life could be quite different. Lots more coders will learn to use mmap(). And the distinction between in-memory and persistent data gets lots fuzzier.

Enough to add a buck and a half to Rambus share pre-market. A bit less now.

Maker's Mark

They ads say, it's a really good hootch. Could be, but I don't often get shit faced, so I wouldn't know. But in the course of commenting on SA, I wordsmithed (not the first time), so I'll take this opportunity to mark my words.

Thus we have the fabless companies, seeking to avoid [having to make and sell physical chips]. But, of course, someone has to have the fabs. Well, until we're all implanted with The Personality and Economic Function Chip as we slip down the chute. And that fab with be owned by Big Brother (who may be government or RoboCop Corporation; as things stand now, RoboCop is more likely).

The Personality and Economic Function Chip™

Hey, if the tobacco companies can claim dope names, I can claim science fiction names. (And, no, I've never bothered to confirm that urban legend. Were it not true, such would spoil all the fun.)

Holy One Track Bat Computer Mind, Batman

I knew nothing of Adobe Analytics, until this post appeared on R-bloggers (and not much more as I type, of course). What motivated this post is the following:

I saw companies that were spending six-or-seven-figures annually on their analytics infrastructure, multi-millions in salary for employees and yet the only way to understand what data they were collecting was to inspect their JavaScript code.

Eek!!! JS hell.

And, let us not forget that the folks at simple-talk, where I spend entirely too much time, would ban me were I to forward similar language in a piece (I have been known to sneak such into comments, hehe). From the AA site:

Adobe Analytics helps you create a holistic view of your business by turning customer interactions into actionable insights.

I'll leave it to the reader to infer why. Hint: it's just one word.

Which has been cataloged with the sigs, and will likely find a place at the top of site. Soon.

(According to the Wiki, this missive's title is a true Robin-ism.)

Isaac Singer's Revenge

Thomas Friedman's column today impels me to work out some ideas that have been bubbling away for some time.

Let's start with a sig from the archives,

Like everything else in technology, the cost of starting a startup has decreased dramatically. Now it's so low that it has disappeared into the noise. The main cost of starting a Web-based startup is food and rent. Which means it doesn't cost much more to start a company than to be a total slacker.
-- Paul Graham/2005

Nearly a decade ago, Graham defined in the new era of the sewing machine. While the sewing machine enabled industrialists to build factories, it also forced (pre safety net; and post, of course) women into slave wage piece work at home. These days, the cheap PC with 16gig, i7,and SSD is much like the sewing machine of 1850: anyone can build a web site. Or, with an R/RStudio installation, claim to be a high-powered quant. And, since anyone can, it's an exercise of low value, and most of the profit redounds to those few who can ensnare the many who do the work. Say hello to Oliver Twist; "More? Not on my watch, buddy. Keep typing, that ACA mod is due today and important people need their healthcare". Have to find some way to scrounge up $3,000/month for a cheap flat in that City By The Bay. I mean, California could demand all those starry eyed, but welfare dependent, tweens to spend 20 hours a week maintaining the DMV in exchange for the coin to pay for the latte`. Imprisoned drug dealers make the license plates, hungry tween coders run the agency. Birds of a feather, and all that. I wonder whether the Indians could still compete with jail labor? Just a thought.

Here's the difference between today's software "entrepreneur" and Hewlett and Packard in the 1950s (read up the Wiki piece):

Of the many projects they worked on, their very first financially successful product was a precision audio oscillator, the Model HP200A. Their innovation was the use of a small incandescent light bulb (known as a "pilot light") as a temperature dependent resistor in a critical portion of the circuit, the negative feedback loop which stabilized the amplitude of the output sinusoidal waveform. This allowed them to sell the Model 200A for $54.40 when competitors were selling less stable oscillators for over $200. The Model 200 series of generators continued until at least 1972 as the 200AB, still tube-based but improved in design through the years.

Real engineering, making a real product that had constructive usefulness. Today's tweens make twitter and instagram and sundry diverting entertainments. Could there be a greater gulf? I think not. But twitter and instagram are the sort of toys that one can make with little capital and little intelligence. Kind of like sewing blouses. Hewlett and Packard were Stanford EEs, for crying out loud. Today's generic tween is a PHP hacker who never took a math or database course, assuming said tween even spent time in higher education. Most of the tweens disparage education, if you scratch their very thin surfaces. One in 10,000,000 get to be Zuck (who may have cadged the whole thing from Those Other Guys).

Which brings us to the misguided, as he usually is, Friedman.

We're now in an era in which globalization and the information technology revolution have merged to drastically shrink what was the basis of our middle class for so many years: the "high-wage, middle-skilled" job.

Yes, but, this happened as a result of policy decisions, not the least of which was the "opening" of Chinese labor to American corporations. Economists of the micro- persuasion (and quants, who mostly work to maximize revenue for their corporation), mostly (always?) argue that the field is ignorant, and rightly so, of value judgment. Whatever the math says is the right answer, even if the answer is that oligarchs get 99.44% of wealth and the rest of the society starves in their service. Adam Smith (the real one) didn't agree, by the way. The difference between 1776 (when Smith's most famous book was written) and today is that the oligarchs have the NSA, et al, protecting them from the guillotine.

What Friedman, either out of ignorance or deceptive intent, doesn't talk about is that all of the human condition is relative. The USofA, post World War II, had a blue collar middle class because policy made it possible. After all, the war ended the Depression (thus proving the point that massive government spending will lift an economy out of collapse, a fact which disturbs the Right Wingnuts), and was still within the life experience of most of the country's citizens. The laissez faire policies which led to The Great Depression could have been allowed to return. They weren't, and until 1973 when the Arabs had enough of Israel and its enabler (us), life went well.

Time for another quote.

As mass production has to be accompanied by mass consumption; mass consumption, in turn, implies a distribution of wealth -- not of existing wealth, but of wealth as it is currently produced -- to provide men with buying power equal to the amount of goods and services offered by the nation's economic machinery.
-- Marriner Eccles

That quote has permanent place at the start of one version of this endeavor, and it is key to both quant and policy. It is key to quant because it refutes the nihilism of the quant ethos. It is key to policy because it provides the logical foundation for equitable decision making; you don't miss your water until the well runs dry, and you don't feel hungry until you've killed the golden goose. In both cases, the goal is growth of the society and economy without having to resort to the guillotine.

So, Friedman continues,

In today's hyperconnected world without walls -- when more Indians, Chinese, computers, robots and software can perform more average blue-collar and white-collar jobs -- the only high-wage jobs are increasingly high-skill jobs.

Wherein he blithely ignores a key factor in the collapse of the American white collar middle class: the invasion of cheap Indian (and others, to be fair) white collar labor. Tom, old sot, you can't have it both ways. Especially in just one sentence. I guess The Times doesn't edit your copy? Moreover, the number of high-wage/high-skill jobs is necessarily limited, since they're the explicit result of pyramiding the workforce. To put it plainly: more high-skill workers will only drive down the value of high-skill work while relentless automation diminishes the number required. Remember what the Indians are doing to us? D'oh!!!!!

In any case, there was only one Einstein. Establishing an educational system which winnows out the participants, until there's just an Einstein, leaving the rest to starve can't win in the long run. That's North Korea, in case anyone's still not getting the point. And more than a few white-flight suburban gated housing tracts; "so glad to get away from all those SNAWT".

He then goes on to quote one Andreas Schleicher, who runs the PISA (Program for International Student Assessment)

"Since the link between skills, jobs and growth is becoming ever tighter, it will be harder and harder for governments to address inequalities through redistribution"

Which is exactly wrong. Education, as well as innate intelligence and height and weight and beauty, is all relative. Telling kids that getting some level of education is the key to comfortable adulthood is just a lie when there isn't demonstrated demand for such skills (all that Arab Spring). The 20-somethings hoping to score the next instagram know that already, hanging out in those Bay Area coffee shops, furiously sewing away on their laptops. "Just one more blouse..." The cure for maldistribution is... wait for it... a theory of distribution which is inclusive rather than the Right Wing's exclusive. Robinson Cano` just got $240 million (over ten years) from the Mariners. Is that sufficient justification for middle schools to institute bayesball as top of the required curriculum? Not even the nuttiest of the Right Wingnuts would go that far (Codd, I hope so).

The real world of production has less and less demand for increasing high skill as automation continues on its exponential expansion. There was a time when one had to be a grizzled EE in order to design a cpu. Today? You just need to know a bit about EDA Lego machines. In a nutshell, it really has devolved to Lucy on the assembly line (if that doesn't sound familiar, do the search), with the exception of the folks who make the tools. The number who make the tools (and need the high-skills) is vastly smaller than those who use the tools (and don't need the high-skills, or get the big wages); as it always is in a Social Darwinist world. As Graham observed, not much of anything is needed to make a toy program.

There is a canary, at least one. It is called Apple, which has long since abandoned making computers as its modus vivendi. Some are convinced that it can't continue to grow revenue and profit by catering only to the 10% (or thereabouts) of the US/EU. Others believe it will score a deal with China Mobile. Even if the deal happens, on even generous (to Apple) terms, China's long adopted strategy of currency manipulation (I know, some Really Important Pundits insist that doesn't happen) may make for a Phyrric victory. It's by no means sure that there are enough Chinese who can afford a $700 phone to make much difference even if the deal happens. It's easy to grow 10%/annum when you're small; not so easy when you've eaten nearly the entire planet.

Finally, we get to the Supremes. You can (if you've not yet) read up Vaughan-Nichols' piece here. He's in the 'patents on software is stupid' crowd, as am I. The notion of software, and worse, business method, patents is destructive. If the Floundering Fathers wanted abstract ideas patentable, they would have said so. Algorithms were a known entity at the time, so don't prattle on about modern times.

In sum: patents grant to the smart (or, as often, lucky) an effective perpetual monopoly on some construct. The length of this perpetuity has been lengthened by Congress many times since the Founding (not that Congress bowed to pressure or anything, of course). Congress has done even worse with copyright, of course. Walt's heirs will own Mickey until we blow up the planet. Carving out high living by law (which is to say, policy) to the few won't help get us out of collapse; it will just accelerate the call for the guillotine. The data show us how Western economies were constructed when they did well, at the macro and household levels. The intelligent approach is to build such a construction.

Larry, No Curly or Moe

If you're in the mood for a compact refresher, or crash, course in probability and statistics (using R, but not overtly so), then Larry Wasserman's "All of Statistics" is where to go. A couple of pieces of the Preface are worth the price of admission. Alas, Amazon has banned copy from the Look Inside. Which makes no sense, since about the only thing you can see is the Copyright page and ToC. Silly.

Anyway, here are the two bits.

(my ten fingers):

Using fancy tools like neural nets, boosting, and support vector machines without understanding basic statistics is like doing brain surgery before knowing how to use a band-aid.

Amen, Brother! Now, could you smite those xml database zealots?

The Preface ends with a Rosetta Stone for stats/CS, and has been transcribed (by a student at Carnegie Tech) here (doesn't display worth a damn on Dr. Codd's page...). Whether both camps fully agree I don't know, but it rings true to my ears.

It's in the Bag

Thanks, once again, to Artima for a precious linked piece. I haven't followed javaworld, on purpose anyway, in some years.

Why this piece? Well, as sometimes mentioned on various versions of this endeavor, I'm a Bermuda-phile. I've only been there once, alas, but I've stayed attached through various on-line versions of its newspapers. For most of the time since universal sufferage (not so long ago as one might assume), the government was of the (generally accepted) black/native party, the PLP. Until the last election, when the (generally accepted) white/foreigners party, the OBA, won election. Since that time, the drumbeat for punishing the lower classes, in particular blaming the lower classes for the oppressive (generally accepted) price level, has gotten very much louder. Never mind that prices on a group of tiny islands (as a child, I was taught that the place was named The Bermuda Islands, not the more common singular of today; it is in fact an archipelago remains from the cone of a volcano) are driven by transport costs and the brutal income skew of those uppity white folks.

Couple that with recent reporting on crime and cost of living problems in the Dakotas' oil patch, and an age old theme emerges.

Which brings us to the techies, which term among the Kiddie Koders is presented as insult.

As the San Francisco Chronicle's Nellie Bowles reports, the word "techie" has become an insult, comparable to being called a yuppie in the late 1980s, only with less developed personal grooming habits.

That quote includes a link to the Chronicle story. But no matter. The javaworld piece is suitably tongue-in-cheek and a bit satiric. And then there's this:

Take San Francisco's Mission District, where cheap burrito joints, secondhand clothing and appliance stores, and mom-and-pop tiendas still abound. The average rental for a one-bedroom apartment there is now nearly $2,700. In South of Market, the former warehouse district where many of these bubble companies are based (full disclosure: as is InfoWorld), rents are approaching $3,500 a month.

One can never be too rich or too thin.

And I'll close this missive with:

They're actually Digital Bohemians and Aspiring Glassholes -- or D-BAGs for short.

I assume that those who created the acronym (or backronym) are aware what a real D-BAG is?

Schemas Don't Lie, But... Well, You Know

Well, the reactionaries are reacting. To be expected. The Big Lie of NoSql/xml is that such datastores are "schemaless", and thus able to handle any sort of "unstructured" data. RDBMS, on the other hand according to these snake oil salesmen, is trapped in the table/column straitjacket of the RM. The thing about Big Lies: they have to be big in order to flummox the ignorant. An Organic Normal Form™ relational schema is orthogonal by definition nearly immune to side effects from mods (outside the client side code uninterested in the change), while a hierarchy (IMS/xml/HL7/whatever) change propagates at least all the way down. And how does one find related data? The xml zealots are still adding knobs and whistles.

Dave Kellogg, cashiered CEO of MarkLogic (among many) is quoted in an article thus:

I met many Oracle-DBA-lifers during my time working with the government. And I'm OK with their personal decision to stop learning, not refresh their skills, not stay current on technology, and to want to ride a deep expertise in the Oracle DMBS into a comfortable retirement. I get it. It's not a choice I'd make, but I can understand.

Of course, that's bullshit. MarkLogic, initially and for some years, explicitly promoted itself as The xml Database. And, as you know gentle reader, xml is just IMS in, mostly, plain text. xml, by definition, is a hierarchical datastore (when used as such). MarkLogic is just IMS done badly, and perhaps a bit cheaper. Each and every xml definition is a rigid hierarchy. That .xsd (and given the label schema) came along later is part and parcel of the kludge. The xml folks have been sneaking in relationality at least since IDREF, with limited success.

He goes on to say:

It had never occurred to me, for example, that in a $630M project -- where MarkLogic might get maybe $5 to $10M -- that someone would try to blame failure of what appears to be one of the worst-managed projects in recent history on a component that's getting say 1% of the fees.

Not to be all too macabre, but the Challenger o-ring was nowhere near that much of the cost of a shuttle. The point, which Kellogg attempts to redefine out of existence, is not how much of the total bill goes to MarkLogic, but how much of a bottleneck it is. And it is the bottleneck. OLTP systems have no business in NoSql and a colossus of code. That's what today's Kiddie Koders' grandpas did in the 1960s with COBOL/VSAM/IMS. The problem with MarkLogic isn't that retiring Oracle developers are unfamiliar with new tech, but that MarkLogic isn't new tech and is crap for OLTP. The "core database". Yeah right.

The current CEO (well, this week anyway) Gary Bloom, in The Wall Street Journal (of course):

Mr. Bloom said CMS needed to process non-standard data types from multiple vendors.
...
If CMS had elected to use a SQL system, its programmers would have had to build a common data model, or schema, to describe the disparate data sources within the application.

Ok, so what are these "non-standard data types"? Plutostrings and minervaintegers?

So, MarkLogic just automagically slurps up any old byte stream and figures it out? Not really:

The trick is that content not in XML must be normalized; that is, converted to XML. MarkLogic has developed some proprietary methods to perform its data management operations.

In other words: MarkLogic does an ETL exercise, just as any sql RDBMS would do in the same design. Not only that, but any benighted xml data source has to be munged to look "just so" for MarkLogic's structure. This schemaless prattle is just stinky poo. (Although I object to calling MarkLogic's exercise "normalized".) Well, unless the sql developers were smart and federated the databases and read (writing takes a bit more work) from the original. While there are provisions for user defined data types in all the industrial strength RDBMSs, I've not seen them used much. The fact is: one can read a string, an integer, or a float from any foreign database and not have a problem. The principal source of angst would be word size of the machines.

Back to Kellogg:

Oracle was non-standard in 1983. Thirty years later it's too standard (i.e., part of an oligopoly) and not adapted to the new technical challenges at hand. All because some bright group of people wanted to try something new, to meet a new challenge, that cost probably a fraction of what Oracle would have charged, the naysayers and Oracle lifers will challenge it endlessly saying it's "different."

While I agree that Oracle is rapacious, and a fair number of its clients think so too, it's worth noting (as done in an earlier missive) that ten years on, Oracle was a profitable $584 million company. Ten years on and MarkLogic is still sucking at the VC teat; since it's private, we don't know whether there's been any profit, but needing more VC money indicates, not much. While the likes of Kellogg and Bloom and the rest of the xml snake oil peddlers continue to bray about new, disruptive tech; most folks who've been in the real world for a while know the real story. And the real story is that xml, as datastore or data transfer, is a dirty bung hole from the mid-60s.

Oh, and that oligopoly? It's called ANSI. The SQL standard, which each vendor commercial or otherwise amends, extends, and bends to stay "different and better" than vanilla SQL. There's a reason that applications get locked into Oracle or SQL Server or (less often, alas) DB2. Although on IBM's z machines, DB2 is just about the only relational database one can sanely run. Calling ANSI an oligopoly is, to be charitable, mistaken. Were I in a ranting mood, I'd likely say, bloody lie. And their user code (stored procedure language) isn't standard. Yet.

If you're interested in a case study, then here is one. Note that XQuery is an bung hole kludge compared to sql. I guess that's one reason coders like it: a moat of obscurity around their jobs. RBAR's revenge. Just what coders who can't grok simple set theory can rally around.

Positive Feedback

You know that screech that happens when the roadies have messed up the mike/speaker configuration? Positive feedback.

Well, the folks at DataMind have produced a ranking of Coursera stats-with-R offerings. I suppose it shouldn't be surprising that social networking is the number one course, since the ranking is based on Facebook and Twitter. One wonders whether Einstein could have produced two theories of relativity, which didn't even win him the Nobel (his work on the photoelectric effect), as well as a raft of others if he'd been of the self-absorbed generation?

Unhand Her, You Cad; The Map

Well, there's been a tonne of ink spilled about Bezos' spiders' web. Everyone concludes that it's just nuts.

Have a look at this map. Long range octocopters, perhaps? For the 10 mile radius that Bezos mentioned, and given that most of the centers are more or less in the middle of nowhere (downtown Boston is sort of expensive real estate), delivery coverage isn't much. I saw a comment in my journey which postulated that Amazon would send out semi-trailers loaded up with flying spiders, find a central point, jack-in-the-box the top of the trailer, and spew thousands of spiders. Snakes on a plane seems tranquil by comparison.

Who Will Predict Us From the Predictors?

Revolutionary Dave posts 14 predictions from a webinar run, in part, by his company. Part of my job is to keep the fires burning for high quality prediction, so I would be remiss if I didn't fulfill my function of predicting the predictors. Herewith my review of each prediction.

1 - Analysts will matter more than data scientists

Well, they always have. From the London Whale all the way back to Long Term Capital (wanna buy a tulip?), with a Great Recession diversion in between, whenever policy and data collide, data loses. More's the pity.


2 - R will replace legacy SAS solutions and go mainstream

Perhaps. Revolution sells a RBAR version of R, and appears (to me, at least) attempting to encroach on SAS's meme: "your all in one data platform" (my instant quote, but SAS said much the same in the Goodnight Old Days). By the mid-80s SAS was selling itself as a database, not just a stat pack.

This paper examines the database features of the Version 6 SAS System [1986] and compares them to the services offered by several popular DBMSs. The conclusion is that the SAS System can provide a cost-effective alternative to a commercial DBMS for the storage of data.

I don't follow SAS marketing these days to know whether they're still pushing that notion.


3 - Big Data will bring its "A game" in sports marketing

Can't say much about that. Nate Silver started out in bayesball, so it could be. Not that I care even a little bit.


4 - Hadoop moves from curiosity to critical

Codd, I hope not. All this NoSql stupidity has been revealed by MarkLogic and healthcare.gov. It is true that stat types have always viewed data as flatfile images. Have I mentioned earning kopecks in the basement of the UMass graduate center typing 059 input for BMDP? Stat types remain enamoured of flat. PL/R has shown them the way, and both Oracle and SAP are providing similar functionality. Still, they persist in viewing the computer as a Friden adding machine.


5 - Gartner's prediction that the line-of-business will drive analytics spend will happen

Again, this is not new. "Mad Men" have been using stats since Hector was a pup. The only difference is that R (and before it, 1-2-3 and Excel) with i7 chips and 32gig PCs make it possible to do such analyses without a server, much less a mainframe. Beware: never let the children play with the sharp cutlery.


6 - Visual analytics continues to grow but users need more

Well, "users" remain in love with pie charts, so I'm not so sanguine. Between lattice and ggplot2, R has more than enough built-in graphics to give Tufte a massive MI. Don't do that to him; he's a nice guy.


7 - Analysts lives get more complex, but also easier

Again, beware children and sharp cutlery. The London Whale made a mess just with Excel. Hmm. Progress requires change, but change isn't necessarily progress.


8 - Predictive analytics will no longer be a specialist subject

And, in due time, the same will happen with neurosurgery. Utopia will proliferate.


9 - Customer analytics is the next big marketing role

Next? What does anyone think Google and Amazon and ... is doing with all that personal data the sheep provide? Curing cancer?


10 - A new analytics stack will emerge

Revolution R, I suppose he means.


11 - Location meets big data analytics

Again, old wine, new bottles. While a graduate student, I did some stat work with one of my undergraduate professors. He called his product "Shift and Share Analysis", wherein we took the smallest grain SMSA data to predict the best location for local bank branches. Buffalo Bob (yeah, that's what we called him when he wasn't looking) was by no means the first to try this. Big data doesn't help here, much.


12 - NoSQL meets analytics

Analysts have always hated RDBMS, so this is nothing to predict.


13, 14 he doesn't list.

A Baker's Dozen

The original MarkLogic implementation called for one server and one backup. Bad idea, but these are xml Kiddies. Now we have the White House self-report on progress to normalcy.

Note page 5, where the slide shows 12, count 'em 12, dedicated servers. And sundry other hardware. Ain't bloated flatfiles so much fun? Of course, we (at least, I) have yet to see a report of who, by name and rank, insisted on MarkLogic. My guess is it won't be a political payoff (much as certain segments of the political spectrum wish it to be so), but rather a Supergrade who's been in love with HL7 and other diagnostic data processing aspects of health care. I have mentioned that I spent some years writing (Progress based) a pre-qual database appication? HL7 was even then much loved. Bah. It's a fact that MarkLogic turned to health care when the first few avenues for peddling that scurvy xml "database" bore no fruit. Mixing metaphors, I know, but the colossal incompetence of this pisses me off. The Right Wing has been served up a watermelon pitch by a bunch of knuckleheads. Be rich or die. Yeah, we're so much more civilized now than 100 years ago.

Thanks for the Memories

Well, it appears that the OCZ saga is finally, really, definitely ova. Toshiba is paying $35 million for the carcass. Which isn't much, and likely won't be much more than the debt (Yahoo! list this as $17 million, but that's as of 31 August; Hercules had all assets under its lending). Two points of interest here. First, during the day the share bottomed at $.10 and went to $.26, probably on shorts covering and fools speculating. During AH it came back a bit, then tanked to $.13 as I type after the announcement.

Second, and of more interest, is the fact that Violin Memory is attached at the hip to Toshiba. Methinks that Toshiba has a plan.

Unhand Her, You Cad

There's that line, sort of, from 1930's cliffhanger serials; "Unhand her, you cad!!" I was watching something else, and I'm not a fan of Bezos anyway, so I missed the piece about the drones. Helpfully, it's now made print (well, sort of). As predicted in these endeavors some time back (I'll leave the spelunking to the reader), Amazon is transforming itself into a Brick & Mortar outfit, just with some Emperor's New Clothes.

In order for 86% (<= 5 lbs.) of deliveries to be made with flying spiders, there have to be warehouses, aka stores, within a fairly small distance of 100% of Amazon's customers. Given technology (battery capacity or internal combustion engine miniaturization) a SWAG on delivery radius is on the order of 30 minutes (60 minutes round trip) flight time. So, 100 mile radius requires ~ 200 mph. A sky full of these flying spiders at faster than (all but raptor style firing platforms) helicopter speed? Not likely. So, lots more warehouse, aka stores, will be needed.

Densely populated areas provide the best bang for the buck, but how does delivery work for apartment buildings? Will a storage box, with one way top hatch have to be built? Who pays for that? How will it be engineered to allow packages, but not rain/snow/sleet/rodents? "Bob, we've got RATS!!!!"

Publicity stunt. Amazon still makes no money, and continues to morph into Wal-Mart, without the parking lot, but lots of humans scurrying around getting widgets. Silly.

Flying Blind

Another, not so, happy coincidence. It's been a while since I visited Nick Carr's blog, and we get two very unique and related posts.

First, the FAA admits pilots don't fly much, anymore.

Second, programmers don't write code much, anymore.

The first shouldn't be too surprising, while the second impinges more directly on this endeavor. I've recounted recollections (without a cite I can claim, alas) of studies done in the, I'll guess, early 90s that GUI-fied applications (Word, you loathsome bastard) deflect the user's attention from content to eye-candy. And, thus, earn my enmity. The comment stream is nearly as interesting as the post and the linked article of Nick's.

What I conclude from my experience, the post, and the comments:

1) Even coders who acknowledge that there might be a problem with automating coding don't get that the code structure is a function of the code generator. While the discussion was largely directed at IDEs, the more insidious problem is with application frameworks: RoR, Struts, and the like. IDE/framework is a crutch. The application of "higher level thinking" isn't in the use of such crutches, but in their construction. Tool maker versus tool user, once again.

2) None see the extension to The Great Recession, which is that "higher level thinking" is just codeword for "let's stop doing real brain work, and just all get rich doing finance thingees". Possibly a larger indictment of the quant squad, but it's the coders' fault, too (much of Wall Street runs on C++). One must not forget the Li issue, as told by Salmon. Recent times, last 3 decades, have been largely about devolving from productive employment (actual rocket scientists) to leech employment (automated financial despoilation), all in the name of "higher level thinking". Other than process size improvements, largely an engineering issue, not science or discovery, not much can be said about "higher level thinking" having moved civilization forward. Are Apple's round cornered rectangle patents progress? Are such "inventions" what we should be showering with the maximal rewards (enforced by the legal system, of course)? One comment mentions chip layout, but skips over the relevant part: one can still find photos of chips being taped out (1970s), with real mylar and real hands; these days one uses builders which offer up ever larger blocks of circuits which an "engineer" (for those few still doing such) Lego blocks (verb form) together. How those circuits actually work, and whether there's any alternative, is gone. Whether this Lego blocking truly implements "higher level thinking" is not obvious, to me. There's a reason most everything on a platform looks the same. Apple, in particular, asserted as a marketing meme that a "seamless experience" was justification for uniformity in applications. As biologists well know: monoculture leads to doom. This is not to assert we should all revert to various assemblers (and not nearly as many as there were in 1980, say), but it's equally true that most OO coding (mentioned by more than one comment) isn't OO at all, but olde style FORTRAN function/data with cuter syntax. Even the OO revolution was a sham.

3) Much of the defense of the status quo involved citing the likes of Instagram! Facebook, twitter, Google (even) and such are toys, not tools or anything actually useful. The great minds of the younger generation worry about keeping "in touch" with friends. Somehow, I don't think Einstein had that as a priority. Nor did Crick and Watson.

4) Humans have run up against an immovable object: there's little left to discover in the real world. We've found all the land, resources, and physical laws. We're on the edge of miniaturization of nano-object construction. There is no warp drive or time travel. There is no new continent to exploit (modulo, possibly, Africa), at least in the previously unknown sense. We haven't had a plague in centuries, so we're well past the heel of the population hockey stick, again. Again, the rocket scientists of today are in finance devising ever more opaque methods for extracting moolah for themselves from the productive economy.

5) Oddly, none see the explosion of healthcare.gov as indicative of bloat and inefficiency engendered by such automatons, which by their nature are code-centric rather than data-centric. (There's this disputed NYT report that the system, per se, is 500 million lines. Not likely. There could well be that much including all the foreign systems it connects to, though.) Back in the old days, it was the WinTel monopoly: Intel needed MicroSoft to make ever more greedy software to justify buying a new machine with a fancier X86 chip, while MicroSoft needed ever more bounteous cpus to run the clusterfuck it called Windows. Whether ARM or Intel or whoever will find a needy compatriot is an open question. That we can see the physical limit of node shrinking is not in question (and no, nanotech isn't the answer; once you get to the single atom as we nearly are, the next step is inside the atom which straps your ass to quantum mechanics and that's nowhere you want to be). Given my predilection, the continued ascendence of coding, as opposed to DRI in RDBMS, is the highest mark of IT's regression. While the Kiddies have all their new languages, they've reverted to the paradigm of COBOL/VSAM development that Granddaddy used. Where COBOL was invented, intentionally, so that (non-programmer) managers could read and understand what the application did, today's IDE/framework puts the coder in the same frame of ignorance. Just jam a bunch of them code blocks together. This is intellectual regression, not progression. And they haven't a clue that they're flying blind.

6) None mentioned Knuth.

Angry Birds

When these endeavors began, the first post asserted "Why the Stimulus Will Fail"

In 2009, who are the unemployed? Not, by and large, workers in factories that will make goods for American consumers. The deindustrialization of the economy, in progress since the 1970's, makes any stimulus program a low probability gamble. Will the stimulus program re-employ the leeches in the financial services industry that sent us over the edge in the first place? It is important to realize that this sector of the economy had grown to a very large proportion; possibly unprecedented. Even if the Federal government chose to reward them with new employment, what is it that the Federal government could buy from the sector? Variable annuities? Would those who are re-employed as a result of the stimulus program spend their newly increased incomes in the financial services sector, thus re-employing all those folks? Would that be rational?

In later postings, which don't come to mind (and given the allusive nature of post titling used ...), this notion was made more specific. Any economic recovery from depression/recession faces a fork in the road early on: either re-inflate the businesses which had been employing those made redundant by the collapse (even, and especially in the current case, those who caused the collapse), or move the economy into new (and by intent anyway, less silly) modes of production. I've returned to them in both versions of this endeavor on occasion. Re-inflation is easier to do, since it rewards the wealthy miscreants, and that's exactly what the QE exercise has done. Total employed has barely changed over the course. Capital gains, for those with capital naturally, have exploded. Such a country!!

In sum: the raptor-quants intend to snip the gonads off the only part of Dodd-Frank that deals with the root cause of The Great Recession. They want all mortgages to be classified as exempt from the stricter rules (which aren't all that strict, particularly in historical context). They want to go back to 2004 when they could slurp jeroboams of moolah from the stream going from the saving class to the borrowing class. A zero-sum profit machine.

As Floyd Norris reports today, the Banksters, in harmony with builders and even some titular Left Wingnuts, have been hard at work to "Make it So", putting the oomph back into Banksterism.

The rules on qualified mortgages are meant to assure that consumers can afford them, and the requirements are rather low. Lenders must go to the trouble of verifying a borrower's income, and the total monthly debt obligation must be no more than 43 percent of pretax income. There are no requirements for down payments, or limits on how much is lent relative to the value of the property.

He later quotes the figure I've generally used: 35% as the operational rule of thumb. The new rules define three classes of mortgages, with the best having no restrictions. Naturally, the Banksters want all mortgages to magically qualify. "Welcome to Lake Homebegon, where all the Banksters are strong, all the men are feckless, and all the mortgages are above average."

So, the part of Dodd-Frank that's under assault is that which intends to rein in profligacy by home mortgage issuers (be they banks or mortgage companies). Both don't want to go back to the Good Old Days (which they always do, mostly) when savings' banks lent most mortgages and held them to term. Not much moolah to be slurped up that way. Better to make loosey-goosey ARMs and such, and sell them off for a quick few bucks. The builders don't want the new rules, since such rules reduce the number of high value qualifying houses; higher price thus higher profit. Just ask Apple how that works.

The raptor-quants are starting to feel hungry and need a hearty meal. I suppose it's just a coincidence that we hear about their hunger pangs at Thanksgivanukkah.

Another One Bytes the Dust

OCZ has filed bankruptcy. No STEC, no OCZ, Fusion-io in the tank, ditto Violin Memory. What's a fast relationalist supposed to do? I know, NoSql flat files. Yeah, that's the ticket.

By the end of 2014, SSD will have coalesced into Samsung, Intel, and may be SanDisk. Such a disappointing revolution.

Snake Bit

Bitcoin is in the news again. Twice in the NYT in a few days (once, and twice).

What none of the pundit class has discussed, in the cyberink that comes my way, is what a fixed amount of currency (21 millions "units", as I understand it) would have on a global economy? If the "exchange rate" (e.g. $35/ounce as gold once was) is fixed, then dee-flation is the rule. It has to happen, since as commerce expands both across nations (growth by accretion) and within nations (growth by procreation) prices have to aggregate to the amount of currency available.

If one looks at 19th century USofA, when specie money was the law, that's exactly what happened. There was localized inflation in mining areas (shopkeepers charged steep prices for necessities, and more for amenities), but national deflation since the economy was expanding along with its territory. With a nearly fixed amount of specie, relative to economic activity, aggregate prices had to multiply out to that specie hoard. Holders of specie made out, but the rest of the economy suffered in debt that grew in real terms. Not a process that aided Main Street. Not to mention tech/science was creating truly new forms of goods and commerce. And, no, today's cybershit isn't remotely as revolutionary to this era as coal fired steam traction and petroleum introduction and air flight and telegraphy and on and on to its.

Should bitcoin, and any number of other fiat currencies (why, one might wonder, is it OK for private issue fiat currency to exist, but not sovereign?), displace national currencies, batten down the hatches, Bro! The Dark Ages will have returned. The Winklevossen will be the liege, and the rest of us serfs.

Life and Death on the Nile

One might ask, "why is it that CMS would opt for a twelve year old, still private, still VC slurping outfit to 'run' healthcare.gov?" Why would such a tactic make sense? I mean, they could go belly-up at any moment; although having insinuated themselves into CMS might get them to Too Critical To Fail status. Well, the answer appears to be, inertia.

More than a couple of decades ago I worked for Optimed Medical Systems (no longer exists separately, and last I knew had been inhaled by a competitor after stops elsewhere). This was a Progress shop, and was known for the cadillac pre-qualification software of the time. VT-X00s connected to *nix and the Progress "database" run C/S in-a-box style. We made fitful attempts at a sort of AI module. Prolog would have been the proper language, but that's another episode. HL7 was then relatively recent, and beginning to be widely adopted.

What I found odd, and never reconciled, was the medical community's love affair with hierarchy. The embodiment of this affair is HL7. The Wiki article is reasonably comprehensive. Here's a cautionary tale. IBM flowchart support diagrams, (an example on page 29 that looks just like a pyramid!) dating from the 1950s, are perhaps the earliest existing example of hierarchy blessed by The Smartest Guys in the Room. Dr. Codd faced significant opposition, and IBM paid him back by appointing an IMS cowboy (Chamberlin) to devise the query language.

As with any hierarchy biased datastore, finding *relations* among all that data is hard. And it's relations that we care about. R folks, and quants generally, take a probabilistic view of data relations. There's X% of people who buy diapers and whiskey together, so let's use an animated baby in our ad for Uncle George's Fine Moonshine. And get Uncle George stocked in grocery stores, and diapers in liquor dispensaries. RDBMS folks view relations as deterministic: order lines *will have* an associated order. The pyramid folks get all bent out of shape with this kind of assertion, blithely ignoring that their hierarchical datastore forces all data into *but one* deterministic structure. Some people...

From a diagnostician's point of view, the IBM 1401 flowchart (or its railroad equivalent) establishes a reasonable model for decision making: is the patient breathing? And so forth. As a datastore "model", not so much. The medical community had a serviceable hammer, and made all else into nails.

So, in a world of folks who insist that all data be pyramids, forcing MarkLogic on the developers isn't all that surprising. As usual, policy beats the crap out of data when there's a disagreement.

Marky Mark and the Funky Bunch

So, who or what, the hell is MarkLogic? For one thing, it's sometimes Mark Logic and other times MarkLogic. Since I left my Clark Kent memorial fedora, complete with press pass, at Superman's phone booth, what follows (modulo my speculation) is gleaned from the InnterTubes. In sum: not so sure I'd bet the farm on these guys for an HIE style application. Somebody, whom I've not identified, is responsible for choosing them to run that "core" of healthcare.gov. Them new duds the Emperor's got sure impressed somebody.

Its site says it's been around since 2001, yet slurped $25 million more in VC money recently. Twelve years in being the greatest thing since high button shows, and still living off the VCs? That and the named managers, from CEO on down, turn over about as fast as "want fries with that?" voices at Mickey D's. Something's not quite right. Still private, so financials aren't available. By comparison (and MarkLogic's promotional lit goes after them), when Oracle was ten years out from its first release (1989) it was a public company with $584 million (in 1989 dollars, mind) revenue. "And you don't mess around with Jim."

Let's start with some promotional slides, circa 2011, here. The slides are marked "confidential". Hehe.

Slide 23 - they're touting the CMS Health Insurance Exchange: "Massive amounts of data at high velocity; highly transactional" not so much, when the rubber hit the road

Slide 34 - they admit (though likely not bright enough to know it) that they've re-invented IMS: "Information is stored as XML (hierarchical)" Kiddies just can't stop building pyramids; perhaps we should rename these Madoff Information Systems (MIS, rebooted from the 70s)

What the Kiddies don't get is that hierarchies (whether in semi-binary as IMS, or clear text as xml) are built to support *a single*, hard coded, optimized access path. And that's for reading. Any other path may be impossible, and will be at least difficult. Changing the structure of the hierarchy requires changing any code which uses said hierarchy, since some (even most) Stations of the Cross have been moved. Dr. Codd figured out the problem: with an easily fungible relational structure (and sensible queries) adding tables or columns will be transparent to existing queries; those that don't need the added data may ignore it. Orthogonal is a very good thing. Too bad the Kiddies never figured that part out. The Kiddies insist that re-inventing IMS is progress. Yeah, right.

The tenor of the deck (hard not to insert the obligatory r) is MarkLogic as a Master Data Management implementation, although that term is kind of passe' most places these days.

Another of their buzz sentences: "Database and search engine are the same" as if this were any different from real RDBMS, or IMS for that matter

Since I'm willing to risk being accused of ad hominum behaviour, here's what the founder, Christopher Lindblad, did before MarkLogic

Chris founded MarkLogic while working as an architect on Ultraseek Server, Infoseek's enterprise search application, which is now one of Autonomy's principal products following the acquisition of Verity.

Here's the dirt: Infoseek was crap, Ultraseek got passed around like a plate of week old baked beans until Autonomy ultimately ended up with it, and who was the mess that H-P way, way overpaid for. And, "principal product"?, no it isn't: "Under Autonomy, Ultraseek continues to be developed and marketed as Autonomy's entry-level keyword-based site search offering." Not something to brag about, no. "Try not! Do, or do not, there is no try" The point: HIE is an OLTP application, not a Google. If the only tool you have is a hammer, everything tends to look like a nail. Not do OLTP from a doc store; there is no try.

As I posited in an earlier post, Kiddies assume that search *is* computing and that hierarchies are the bee's knees of datastores; and they aren't. While there may be some part of ACA implementation that will benefit from blob/clob data bits, the guts are OLTP/POS. And they ain't nothing better than an industrial strength RDBMS for that. If it starts in Organic Normal Form™, morphing the schema as you go isn't such a big deal. Lots of schema migration apps out there to help, if you feel the need. Remember: orthogonal is your friend. Cleave to orthogonal; as the adverts used to say, "Calgon, take me away!" Hierarchies ain't, they're as mixed up as a Christmas fruit cake.

The really big deal here is the federation with Federal, state, and insurance company systems. That's the crunchy part. There are multi-database federation protocols out there, including in SQL-2003; it's called Foreign Data Wrappers in PG terms, SQL/MED (Management of External Data) officially. DB2 provides the most robust, cross-vendor federation (likely because it has to!). Oracle does too, of course using its own semantics (took them a while to wean themselves off CONNECT BY for CTE syntax, of course). Progress (you remember it, right?) offered federation in the early 90s.

This is a classic distributed database application. Full stop. No need for NoSql. There is no try.

Never Send A Boy to Do a Man's Job

About a month ago, I mused that the Kiddies were the cause of healthcare.gov's abject failure.

I haven't yet found more than vague descriptions of the architecture of healthcare.gov (herein, hc.gov), but I'm willing to bet a quid that it's a labyrinth of xml data streams exploding all across the InnterTubes.

Turns out to be worse than even I surmised. CGI has been an Oracle based application developer for a long time. Turns out that some part (don't know the extent from this reporting) of the site is run on MarkLogic, a *xml/NoSql* "database". Here's their site. Nailed it!!

Another sore point was the Medicare agency's decision to use database software, from a company called MarkLogic, that managed the data differently from systems by companies like IBM, Microsoft and Oracle. CGI officials argued that it would slow work because it was too unfamiliar. Government officials disagreed, and its configuration remains a serious problem.

Come on guys!! One knows that the NYT isn't a techy journal, but a bit more specificity would help citizens understand how bad this idea was. CGI has a marginal reputation, but this just tied both hands behind their backs. To repeat: reporting so far doesn't describe where in healthcare.gov the MarkLogic "database" sat. We also don't yet, from reporting, know whether this MarkLogic store was the central repository; although "yes, yes it is" seems a reasonable inference.

According to this reporting before 1 Oct:

It is also the core back-end database powering the new federal healthcare exchange that is set to open October 1. That system has been called the world's largest information integration project, says Aaron Rosenbaum, MarkLogic's director of product management.

No definition of what "core" means here, but since we're talking NoSql, I'll assert that "weakist link" covers most of what matters. I'd gloat, but hc.gov is way too important for that. It also means that Kiddies are easily flummoxed by Emperor's New Clothes. Back before the Triage piece, I was invited to the DNC IT folks to talk about databases and stats. This was preceded by phone conversations about databases and stats. When I got there, I suffered through a couple of 20-somethings who only wanted to jabber about Ruby on Rails. Didn't go well, and I wrote up Triage in a fit of pique. DNC proceeded to blow it in the state elections just as they did in 2010, although the absolute number wasn't as bad. Given the Obama coattails, I'd argue that DNC blew it worse. Off years are always difficult, but presidential years such as 2012 should be a springboard. Not 2012.

Then, this:

"We're doing grown-up, dull sorts of things," Todd says, "but also cool things that other NoSQL players aren't able to start thinking about because they're still playing catch up on the enterprise stuff."

Um. No, not grownups. Never send a boy to do a man's job.

Party Like It's VT-100 Time

Those handful of folks who've followed this desultory philippic know that I've been banging a drum for Organic Normal Form™ databases running on multi-processor/SSD boxes from the beginning. As connectivity and bandwidth have progressed from dial-up to today, the infrastructure for VT-100s connected to *nix databases over a kind of long RS-232/RJ14 wire gets ever closer to reality. Whether COBOL indentured servants or Kiddie Koders (for the same reason, but different paths to that reason) will ever embrace the paradigm still isn't a lead pipe cinch.

Which brings us to the news of the ECMAScript development plan.

Citing other Web-related developments, Eich said WebRTC, a specification equipping browsers with real-time communications apps via JavaScript APIs and HTML5, would become a standard. The work of Google, Mozilla, and the IETF will make this standardization happen, said Eich, currently CTO at Mozilla. "It's really cool, and you can use it for data communications, too," he said, noting that browser support for WebRTC varies, with Mozilla's Firefox a bit behind Google Chrome.

While progress requires change, not all change is progress. In particular, the dumb data on a client with smart code (also on the client; fie!) paradigm embraced by both COBOL dinosaurs and java/PHP/fooBar Kiddie developers. Resistance is futile. You will be assimilated. Render unto Server that which is Server's.

The Hard Way

Just saw this post from r-bloggers. Since Heroku is Postgres based, I've left the following comment:

Why not just use PL/R? Would be a lot less work. I've built a small project using PG and PL/R, and it works as advertised.

Not Such a Nobel Man

Lars Peter Hansen is the Other Guy who got the Nobel in economics this year. Mostly been quiet about it, although it was generally understood from the outset that he was somewhere in between Shiller and Fama (get it?) on the ideological spectrum. Well, today Jeff Sommer in the NYT has a bit of an interview. I'm disappointed. This is a Nobel Man? He labours at Univ. of Chicago, which makes him, a priori, a Freshwater economist. (That's another way of saying Right Wing.)

As with most econometricians of late, he ignores policy (or, as some do, provides cover for his preferred version), and toes the stats line.

Earlier in his career, along with Professor Sims and especially with Professor Sargent, he provided some of the mathematical underpinnings for what is known as the rational expectations theory -- the notion that people use all available information in making economic decisions.

Nonsense. Capital, in particular, makes decisions based on gaming the rules of engagement. To posit otherwise is foolish. If they really were "rational" there'd never have been a Great Recession, since no rational Bankster or CDO insurer would have touched those mortgages with a ten-foot pole. The mortgage companies wouldn't have made them, either, since such instruments were bound to fail. While the macro folks in New York and Washington might have willfully ignored the fantastic unsticking of median house price to income ratio, local real estate agents couldn't. The numbers were in their faces every day. Early on in the run up to the crash, I read/saw some reporting in which the reporter asked one of these how the buyer could possibly carry the mortgage he just sold them. His response: "I don't care". He didn't care because he'd gotten his gelt, and from then on the problem belonged to someone else. Such a country!!

One might, and some have, argue that everyone from the local real estate agents on up to the CEOs of Merrill and AIG and so on rationally expected to get a Washington bailout when it all imploded. But that's both specious and tautologous. No one who hews to the homo economicus myth can argue that it is rational to depend on welfare.

It suggests that people may not "be fooled by policy makers" into making decisions against their own self-interest, he said -- for example, by spending all the proceeds of a one-time tax cut if they understand that the windfall is only temporary.

This is an astonding assertion, and is contradicted by actual behaviour. Folks used these "one time" gains in house appreciation as ATM withdrawls, particularly as median income continued to stagnate. And people do so every spring, with that tax refund they get. Never mind that 99.44% of those getting a refund have no clue that they've lent the money to Uncle Sugar at 0% interest. Come on!!

Prevailing economic models do not adequately explain the financial crisis, the severe recession or the weak global recovery, he said. "Systemic risk" is a buzzword for politicians and financial regulators, he said, but "the truth is, we really don't know how to measure it or what exactly it is."

Again, more willful ignorance. The data were out there and available to modelers. But, just as the Banksters took the notion of musical chairs ("while the music plays, we all have to keep dancing"), or Russian roulette, with the real economy, so too did a large proportion of quants who claimed to be analyzing the economy objectively. Since the behaviour didn't square with both their bias and models, i.e. house prices are always correct, they made no attempt to incorporate conflicting data. And so we got a black swan event. Except that it wasn't.

As my Momma used to say, "what would the world be like if everybody behaved like you?" Ignorance of this question remains the flaw in the "macro is just the aggregate of all the micros" approach to all of economics, both theory and quant. It ain't, just as a local water supply can't be sustained if all residents treat all the water as all theirs. The fact that this remains unspoken by the micro folks just means it will happen again. Most quants, be they econometricians or hedgies, earn their pay from entities which ignore or actively dissemble the externalities they create. If Adam Smith (the real one) were right, then there would be no unpaid externalities. No fracking way, of course.

Obambi, Meet Gresham

Gresham's Law is long and widely known in econ circles, though not mentioned often enough among the punditocracy. Obambi faces it now, and once again, he's blinked (as of the time I'm typing).

In essence, in the absence of symmetric true information, bad goods drive out good goods (hard to say, and it's not just coinage). There was a reason that the scam health insurance policies, almost all aimed at uninsured individuals, weren't grandfathered into the AHA: stupid poor-ish people are easily duped into buying same. They look only at the monthly premium, not at the expected value vs. the premium. Such policies will drive those most in need of real insurance into faux insurance.

As I type, the news states only existing (and those canceled, allegedly, due to AHA) policies will be allowed to continue. New buying is not permitted. We'll see the Right Wingnuts crying: "let the market decide!!!" once again. As if health is a market good. Ayn Rand would be proud.

RIP, Bob

Bob's dad died.

You might remember Bob, that yakking paperclip that was supposed to make Windows easier to use? His pop was Clifford Nass, and the creation of Bob should be enough to ignore his passing. I mean Bob???

On the other hand, according to the recitation in the Times obit, his principle occupation was at Stanford, and he had, at best, a gingerly view of human/computer interaction.

I mean, who wouldn't applaud this:

Clifford Nass, a Stanford professor whose pioneering research into how humans interact with technology found that the increasingly screen-saturated, multitasking modern world was not nurturing the ability to concentrate, analyze or feel empathy

While the obit doesn't include a cite, I expect he'd agree with the legendary studies which showed that GUI-fied applications, Word in particular, distract users from content to focus on the easy facade. Playing is always more fun than working, so it's not surprising to discover that Word users would expend more time and energy on font selection than on more mind taxing content.

Nass' research centered on the effects of multi-tasking. Again, I'm a member of the choir.

"We all bet high multitaskers were going to be stars at something," he said in an interview with the PBS program "Frontline." "We were absolutely shocked. We all lost our bets. It turns out multitaskers are terrible at every aspect of multitasking. They're terrible at ignoring irrelevant information; they're terrible at keeping information in their head nicely and neatly organized; and they're terrible at switching from one task to another."

...

"We worry that it may be creating people who are unable to think well and clearly."

...

Dr. Nass found that people who multitasked less frequently were actually better at it than those who did it frequently. He argued that heavy multitasking shortened attention spans and the ability to concentrate.

Is it any wonder that some of us question the au courant meme of Agile, for example? Read Nick Carr in memorium.

Kumite', Again

Anthony Bourdain is among the most interesting of reality show personalities. He's now on his third show, and third network: "Parts Unknown" on CNN, Sunday at 9 PM. (As with his previous shows, the title is both metaphor and allusion: he goes to places tourists seldom do, and indulges in eating animal bits that suburban Americans wouldn't, and likely don't even know about.) Somehow, I missed last week's episode, but I saw a promo for it, in which he talks about doing the episode. It's even edgier than usual. Which is more than usual for basic cable.

The episode is Tokyo, but not the touristy parts. It repeats again Sunday, the 10th, at 8 PM. You should see it. This Tokyo is disturbing.

Well, it's Tokyo and Tony visits with a sushi chef he'd known for years in New York, Yasuda. At about 20 minutes into the piece, we see that Yasuda has been practicing karate for some years, and we see a few minutes of footage from the dojo he uses in Tokyo. An earlier missive in this endeavor made metaphor of karate and kumite'; not all practitioners have the skill and discipline to fight as they practice. There's some kumite' on display, and some, but not all, of the roundhouse kicks are by the book.

We find that Yasuda does, both in kumite' and sushi.

Yasuda, not too surprisingly, is a stickler for doing sushi the right way. His knuckles remind me rather a bit of George's.

The Human Pretzel Factory

Never been a big fan of Agile, which should come as no surprise, since any relationalist subscribes to the notion that one needs first decide on the nature of persistent data. There are lots of tools for schema migration these days, so don't give me any shit about that. Changing schema is no more difficult than changing code. And, lest we forget, an Organic Normal Form™ schema is the constraints of record, and thus can be used to generate client side code. Lots of folks use xml monsters to do just that.

So, I came across this piece, via Artima. There's a link to the prequel in the piece.

As he concludes, there is no silver bullet. Well, there is, it just isn't some codified process. Carpenters live by it: "measure twice, cut once". The whole agile thing asserted that it isn't possible to understand an application a priori, so don't bother and just code sling until the cows come home. Eventually, if you don't wear out your debugger in the process, you'll get something to work. It should come as no surprise that healthcare.gov got built this way. Here's the autopsy. For the record: the previous sentence's contents, I just discovered, while the one next previous I just winged it: seemed obvious given the Kiddie Koders who've been the public facing talking heads.

Measure twice, cut once. Repeat until you understand that.

Frankly My Dear, I Don't Give a Damn

Let's, for the moment, assume that Obamacare works much as it was designed: provide affordable healthcare. Next, let's follow Einstein's lead and conduct a thought experiment. He devised the notion, although not the algebra, for the theory of special relativity whilst sitting on a tram looking at a receding clock tower. He imagined that, were the tram moving fast enough, the clock would look to him, on the tram, to have stopped. And the rest is history and a Nobel. The proceeds went to his estranged wife, whom he didn't like all that much. Oddly, still more: she did most of the algebra underpinning special relativity. Albert was a bit of an ingrate. Didn't believe in quantum theory, either.

What, then, can we imagine would stop if Obamacare works as designed?

Mostly, bad marriages. Very Einstein, don't you think? And why would one come to such a conclusion? It's been well documented that having access to healthcare is a powerful force, keeping otherwise disgruntled people in bad marriages and bad jobs. House ownership, too. But Obamacare doesn't deal with that. Yet anyway.

So, we should see an increase in divorces. Whether that increase will be above sampling noise, is another issue. But unintended consequences have a habit of occurring. Just ask Greenspan; he crashed interest rates without much of a clue what would ensue.

Bookmark this post, and come back in five years. If the globe hasn't exploded, of course. Never know what the Jews and Arabs might do.

Pretty Boy Floyd

Floyd Norris' column this Saturday was unusual, in that there was no cite to the QWERTY Analytics (or some such), just his byline, NYT, and BLS data. Let's see.

Start with the title:

Changes in Labor Force Mask Gains in the Jobs Situation

With such a title, one would rationally expect that he's going to present data which contradicts both the Left Wing and Right Wing Nuts, who point to the continuing drop in the size of the labour force as the main reason the stated unemployment rate has fallen. The lack of progress by Obambi is something both camps agree on. They disagree as to why, of course. The Left asserts that the Tea Party House (and the Senate, but less egregiously) eviscerated the stimulus effort; they got to spike the recovery and shift the blame to Obambi. The Right disclaims any responsibility for watering down the stimulus, claiming it wouldn't work at any amount of money; the key is to get all those welfare cheats back to work at starvation wages, and then all will be well.

The record is clear: the Right did spike the stimulus. What might have happened with either a more focused stimulus, or none at all can't be known now. We do have much historical data to demonstrate that robust stimulus works. It's called World War II. It would be helpful to get the stimulus effect without all the dead bodies and razed cities, of course.

So, he continues:

But the actual employment picture may be better than those statistics would indicate. Over the last few years, the labor force has changed in important ways because of demographics.

The accompanying charts attempt to adjust for those changes. They do not show a strong recovery, but they do indicate that the overall employment situation has improved.

One might reasonably infer from this that, despite a drop in the size of the measured labour force (usually referenced as the Labour Force Participation Rate, rather the count), unemployment has really gone down, i.e. employment has really gone up. Let's see.

The makeup of the working age population has changed substantially in only a few years. When the recession began at the end of 2007, 54 percent of the people considered to be working age were in the prime working age range of 25 to 54. Now, the figure is 51 percent. The proportion over 55 went to 34 percent, from 30 percent. With more people at or over traditional retirement age, it should be no surprise that fewer are working.

While I don't have a cite immediately to hand, data since The Great Recession began have measured an *increase* of older workers working past the "average" retirement age of the years previously. In other words, to the extent that older workers make up the demographic, there is data indicating that they've not gone off to the Life of Riley Hammock, sipping gin. And if we look at the linked graphs, we see that old folks continued to work. If you examine the middle cohort, in middle dense blue, you can see that the unemployment rate and participation rate track exactly in The Great Recession period, lending credence to both the Left and Right that things ain't all that much better.

In sum: the oldest cohort has stuck to the labour force in greater numbers than in the past, from which those with rose colored glasses conclude that the unemployment situation is getting better, overall. For those not so festooned, the middle cohort has, even with Norris' "adjusted" numbers taken it the gut. The youngest cohort, worse still.

As I read the data, the oldest cohort retains due to coercion, not choice. Their positive effect on overall employment status isn't really a positive: they can't afford to retire and leave their jobs to younger folks. This is not progress. Note that the Right Wing Nuts constantly bray about forcing the old farts to work until 70 or 75; after all in 1935, 65 was the average life expectancy so we should obviously raise today's retirement age to average life expectancy. Makes sense to me.

First Thing We Do Is, Kill All the Biostats

Sequenom is a diagnostic/testing company that devised a way to test for Down syndrome non-invasively. They've been in a patent fight over their tests for some time. Today's Times has some details. What's both interesting, and perhaps disturbing, about the case is exemplified in some contradictory text in the article.

On the one hand,

The new tests offered by Sequenom and several competitors take advantage of the fact that some DNA from the fetus can be found in a pregnant woman's blood and can be analyzed with new genetic techniques.

Note the "new genetic techniques". But later on

In her ruling, Judge Illston said that the presence of the DNA in the pregnant woman's blood was a novel discovery but could not be patented because it was a natural phenomenon. And claims in the patent over how to detect that DNA were not eligible for a patent because they involved conventional genetic techniques.

So, we have it both ways. At least in this write-up.

But more to the point, as quoted in the article

"It's hard to imagine patents on diagnostics surviving if that approach is taken," Christopher M. Holman, a law professor at the University of Missouri-Kansas City, said of Wednesday's ruling.

And, that I would agree. All those biostats out on the sidewalk, destitute, having to hit the dusty trail to New York to work as a financial quant. One question: how large a part of quants generally are the biostats? This piece from a few years ago says, about 1/3, if the data have the biostats counted with the general stats, 1/4 if they're really disjoint. And half of the remaining 2/3 work in the vampire squid colony in lower Manhattan.

While I'm by no means a fan of software patents, and most bio patents, in the latter a case can be made that such diagnostics/tests can be novel physical inventions, and thus meet the criteria for patenting as set down by The Founding Fathers. Unlike Myriad, in my opinion, what Sequenom has is different from attempting to patent the "natural phenomenon". One of the arguments against a patent is the notion of obviousness to a competent practitioner. Where that exists in statute, I can't find, but is repeated endlessly. Let's suppose that it is really in law. Then, in the case of medical diagnostics (and the jobs of all those poor biostats), the issue is obviousness in how the "conventional genetic techniques" (if truly "conventional") were structured. Ultimately, all new genetic techniques derive from conventional techniques, at some arm's length. Standing on the shoulders of giants, and all that. By this logic, no patent on a VOM could be awarded, since, after all, electrical current/resistance/voltage are all natural phenomena and lots of conventional meters existed. If Apple can get a patent for a rectangle with rounded corners...