Where Have All the Returns Gone?

Not to belabor the point too, too much; but if you still doubt that real returns in anything compute related are a vanishing species, just have a look at this Apple slide. Read the whole page. Poof!

From a later page:

That's right, I still have my old PowerMac G5 Dual 2.5GHz (upgrade from my original 2.0 model). It's interesting to note that single threaded performance has only improved by 2.8x over that 2.5GHz dual G5 machine from around a decade ago.

And now a word from Amdahl:

The first thing I noticed while running this test is how much the workload can impact CPU core utilization. Even though I was dealing with a substantial 4K project, only portions could spawn enough work to keep all 12 cores/24 threads busy.

All that investment of billions and billions of dollars just to ... what? As the Red Queen said, "Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!" (Pulled from the wiki)

An Epidemic of Kissing Disease

How old would you think the word 'monoculture' is? My guess, before I went out to discover, was at least since Stephen J. Gould. Turns out, that's wrong, that is, off by a bunch.

According to here, it appeared in the OED for the first time in 1901. Which likely means it was in use during the 19th century.

Those in the *nix community have been vocal in their warning that M$ DOS/Windows dominance leads to the usual casualties of inbred stock. What they've been less vocal about is the monoculture at the hardware level. While one can build linux from source, and if you can verify that all the source is clean, then you're somewhat outside the box. But that chip is still X86.

So, have a read of the latest paranoid's jeremiad. One might wonder how the 'strict constructionists' abide such? Just because the Floundering Fathers didn't have PCs and innterTubes, does it follow that only the means of communication and domicile that existed in 1789 are protected from Government (and their corporate minions)? One might conclude that, in the case of personal freedom, the Right would eagerly adopt a 'living document' viewpoint. Or could it be that they believe in the sanctity of government, after all? Obama has, just on the basis of admitted behavior, done what the Right Wing wants. The crunch will come when any level of protest gets one labeled 'terrorist'. We've been there before, and it wasn't pleasant.

Who is being protected from whom?

Where's a Robo Cop when you need one?

Apple, My Eye!!

Here's a piece about the new Mac Pro, with this sound byte:

There, Philip Schiller, Apple's top marketing executive, seemed to go off script when, after showing the Mac Pro's striking design, he said, "Can't innovate anymore, my ass."

What Schiller seems to forget, or is too young to know, is that Apple tried this "unusual" design aesthetic game before, and called it The Cube. As Steve said, "steal". This time, Tim is recycling Steve.

And you'd better like it:

With a height of 9.9 inches and a diameter of 6.6 inches, the 2013 Mac Pro has no room for expansion via PCI cards or internal storage devices.

At least The Cube had a real cpu. Bombed like a three-pounds-of-chili fart, though.

Dig a Hole to China

As mentioned in these endeavors over the last couple years, on occasion, it is obvious that the Chinese experiment in social Darwinism must needs assault its economy in much the same way it did the West's. That Giant Pool of Money is still out there, getting bigger, and still demanding high return on minimal risk. That there ain't no such thing as a free lunch matters not to those who wish to live well on moolah alone. Daddy Warbucks, at least, actually made some stuff. Today's banksters merely suck moolah from the fire house aimed by the savers at the borrowers. Never forget that it was a Chinese, Li, who foisted the Gaussian Copula on us. Financial quants are driven by visions of finding the Ultimate Loophole in the system, that crack in the dam holding back all that moolah from their hungry maws, and Li provided, what looked like, that loophole. No, it is way too Byzantine to conclude that Beijing sent him in, "Bond, James Bond" style, to wreak havoc on the hated West. Or is it....?

With the West's fitful (can you say: "neutering Volker"?) attempts to bring the banksters to heel, all that Chinese money is now being directed inwardly. With the expected result. I've mentioned the "60 Minutes" report (revealing the ongoing real estate fiasco) from a few months ago, and much earlier stories from print sources.

Today brings us another. Don't be surprised.

I long ago forgot where I read/heard it (you can find it in older essays), but the following semi-quote about sums up the cynic's view:
"One hears from CEO types about how much work it is to run these corporations, but they never seem to want to take on failing companies. The ones they do run could be just as successfully run by a sock puppet."

Which brings us to:

Yao Jingyuan, the former chief economist at the state statistics agency, said ... "With this kind of operational model banks will continue making money even if all the bank presidents go home to sleep and you replaced them by putting a small dog in their seats."

The piece ledes with the punchline, as any cub reporter has been taught to do:

China's financial system is in danger of becoming too big to bail out.

My, my. All that money, and little to do with it, besides build yet more condos.

Official bank lending has more than doubled since the global financial crisis, growing nearly twice as fast as the overall economy.

Without all those Western mortgages to soak up the moolah, China is generating such internally. Good luck with that.

"The chains of lending and borrowing can be long, just like the securitized subprime mortgages. The result can be devastating."...said Yu Yongding, a senior fellow at the Institute of World Economics and Politics of the Chinese Academy of Social Sciences...

Stating the obvious:

Savers have had few alternatives to banks until very recently: Real estate prices are already stratospheric relative to incomes...

Ya think??? Time to copulate.

Cheap at Half the Price

An interesting position (not very many, alas, these days) came by. It's at a .org in NYC (yummy!! Letterman four nights a week!). But they say they prefer SAS. Which got me to wondering, yet again, why? This is an old .org with, possibly, some deep pocket benefactors. But, still, why SAS? Coming up with an exact price for SAS/seat doesn't seem to be available on the SAS site (or it is well hidden), but various posters conclude that base SAS plus graphics and such is $15K.

What you get for the $15K, that standard R doesn't give you, is the ability to do RBAR stats from a dumb-pick set of menus. (But they ask for a programming sample? Cognitive dissonance, in the extreme.) BFD. For $15K (or say, $45K for a three year machine), one can have 128G/xeon machine ( here's one for a bit under $12K ). Needs a monitor, and linux is basically free, along with R, and even a decent SQL editor (SQuirreL), although I've bought and used both AQT/windows and Aqua/linux.

I have a suspicion that this .org has an agenda. We'll see.

Mikey Likes It

With all too infrequent frequency, one finds a coder-centric babbler who gets RDBMS. This one (and it's earlier entry) are well worth reading (not hot off the presses, but a link in this Artima post). Nearly all of it is old hat to relationalists, but, I guess, still novel to kiddie koders. Oh well.

In the earlier set, is "2. Processing data in Java memory" wherein he advises:

... So in fact, by moving OLAP to the database, you gain two things:

Simplicity. It's probably easier to write correctly in SQL than in Java

Performance. The database will probably be faster than your algorithm. And more importantly, you don't have to transmit millions of records over the wire.

I have always added: "and doing the work in the database means the data is agnostic to client platform, OS, language, or anything else; only that it supports sending SQL requests (for any definition of 'send')".

A number of his 20 recommendations boil down to this: do as much processing on the database, and leave only formatting and input capture to the client. With (nearly) universal 4G bandwidth, even running VT-X00 style client/server apps is now feasible; populate them thar pick-lists in realtime, pardner. Talk about disruptive technology!!

They, the koders, won't do it of course, since they view the world through Dollar Colored LoC glasses, and ceding to the database that which is the database's reduces their LoC landscape. "You'll have to pry this keyboard from my cold, dead hands!" Yeah, I will.

Do You Do Windows?

Regular reader may not remember, but a few years ago there was a bit of a stink over the fact that bank ATMs were being switched out from OS2 based to Windows based.

So, now we know that Target's card customers just got siphoned up. Guess what OS that POS (in both senses of the acronym) runs? Yup.

"Each of our server endpoints and each of our 5,400 POS registers has a System Center Operations Manager agent installed. That way, we can ensure the checkout experience for our guests remains fast and efficient."

That's a M$ page, and it's sort of surprising they haven't taken it down.

Numbskulls On Parade

All you need to know about American Intelligence: the NSA runs Windows. And worries about computer problems. Talk about positive feedback.

What Hath Quant Wrought?

Today's Business section of the NYT has a host of malfeasance on display. A cornucopia of greed and punishment. But that's not what I came here to talk about. Jesse Eisinger has a DealB%k piece, and he's got some reporting of research which illuminates.

What was the proximate cause of The Great Recession? The answer largely depends on which end of the political/economic spectrum one sits.

The Right tale goes: "It was all those poor folks tired of living in shotgun shacks (and wanting McMansions) who came to the overly solicitous and naive` mortgage companies and banks demanding oddly structured subprime ARMs, the parameters of which they dictated to the naive` mortgage companies and banks. Who, being naive`, reluctantly devised such loans. Of course, the loans eventually went South, victimizing the mortgage companies and banks."

The Left tale goes: "The Banksters (mortgage companies, banks, rating agencies, and the Trilateral Commission) set out to gut the 99% and get rich in the process by creating mortgages which enticed the naive` poor folks into believing they could leave their shotgun shacks for McMansions. When said mortgages eventually collapsed, the Banksters kept their ill-gotten gains, and the poor folks retreated to their shotgun shacks, now paying more in rent than they did before the whole sorry tale happened." (Aside: private-equity, hedge funds, and God knows who else are slurping up housing and becoming absentee corporate landlords. That will not end well for communities.)

The truth, to the extent that anyone can be objective, lies mostly with the finance industry/sector growing into Jabba The Hut. One of the earliest themes of these endeavors is that Greenspan is Patient Zero in the epidemic. By crashing interest rates, he set in motion the effort to generate other vehicles of "risk free, high return". There are no such vehicles, of course, but since finance was about as unregulated by 2001 as it's ever been, there was no adult driving the train. "Let's see how fast we can make the choo-choo go, Mary!!" And it went fast.

While I can't claim to be the first to suggest it (although my recollection is that finding others, in the pundit class, who expressed the notion followed my coming to the conclusion), the problem with the financial engineering brigade, i.e. raptor-quants, is that they don't want finance to be simple and boring. Convoluted and opaque is better. And the reason it's better is that profit from finance comes not from value added, but sucked out of the moolah stream twixt savers and borrowers. Better to hide the shenanigans.

Which brings us to Mr. Eisinger today. He reviews some of Jack Lew's, the newish Treasury secretary, earlier pronouncements, along with data produced by outside researchers. It is these concrete facts which are of interest.

The way to really solve "too big to fail" is not by tinkering with the existing system, which leaves the great and fundamental problem still with us. The economy has become overly "financialized."

GE's profit percentage from finance had reached 50%. Other companies saw a quick buck, and took to shuffling paper and sucking moolah from the, what looked like, tidal wave of moolah to be processed.

Historically, finance's share of the economy has been at about 4 percent. Today, it's about twice that. And the peak occurred not in pre-bubble 2007, but in post-crash 2010, at just under 9 percent, according to research from Thomas Philippon of New York University. That represents a shift of more than $600 billion of wealth a year, as Wallace C. Turbeville, a former investment banker-turned-financial reformist, has pointed out.

The result is obvious:

Despite technological innovation, finance costs more than it used to, even though prices have fallen for things like trading stocks.

The Banksters suck their profits from the stream. Even Eisinger has the gonads to be plain:

The financial sector has become a self-sustaining perpetual motion machine that extracts money from the rest of the economy. Shouldn't it be a goal of society -- Mr. Lew's focus -- to restore the financial industry to its traditional role as an intermediary between companies that need capital and savers who have it?

In simple words: finance should be simple, dumb, and transparent. And cheap. It is after all, little more than Marrying Sam, putting savers and borrowers together. All the fancy quant does is extract ever more from the stream for the quants and their bosses. The quants would be more productive in marine biology and such.

Research from Professor Philippon shows that financial activities have gone up in the deregulatory era, and now cost about the same as in 1900, the last Gilded Age. In other industries, like retail, technological innovation has led to lower prices and therefore decreased the size of the sector. In finance, the opposite happened.

The tail is wagging the dog.

(Go to the web page to follow the links to the underlying research.)

Unite Us All

Perhaps another shoe has dropped. Rambus and Micron have signed a deal.

Here's an excerpt:

and (ii) a seven year term patent license to other memory and semiconductor products.

It was just about two years ago that Rambus bought Unity Semiconductor, whose only significant asset was CMOx memory, a non-volatile type which is (said to be) more robust than NAND/NOR. Unity and Micron had agreements at the time. Not much out of Rambus in the intervening time. Looks like the effort isn't for naught. If CMOx does work as advertised, life could be quite different. Lots more coders will learn to use mmap(). And the distinction between in-memory and persistent data gets lots fuzzier.

Enough to add a buck and a half to Rambus share pre-market. A bit less now.

Maker's Mark

They ads say, it's a really good hootch. Could be, but I don't often get shit faced, so I wouldn't know. But in the course of commenting on SA, I wordsmithed (not the first time), so I'll take this opportunity to mark my words.

Thus we have the fabless companies, seeking to avoid [having to make and sell physical chips]. But, of course, someone has to have the fabs. Well, until we're all implanted with The Personality and Economic Function Chip as we slip down the chute. And that fab with be owned by Big Brother (who may be government or RoboCop Corporation; as things stand now, RoboCop is more likely).

The Personality and Economic Function Chip™

Hey, if the tobacco companies can claim dope names, I can claim science fiction names. (And, no, I've never bothered to confirm that urban legend. Were it not true, such would spoil all the fun.)

Holy One Track Bat Computer Mind, Batman

I knew nothing of Adobe Analytics, until this post appeared on R-bloggers (and not much more as I type, of course). What motivated this post is the following:

I saw companies that were spending six-or-seven-figures annually on their analytics infrastructure, multi-millions in salary for employees and yet the only way to understand what data they were collecting was to inspect their JavaScript code.

Eek!!! JS hell.

And, let us not forget that the folks at simple-talk, where I spend entirely too much time, would ban me were I to forward similar language in a piece (I have been known to sneak such into comments, hehe). From the AA site:

Adobe Analytics helps you create a holistic view of your business by turning customer interactions into actionable insights.

I'll leave it to the reader to infer why. Hint: it's just one word.

Which has been cataloged with the sigs, and will likely find a place at the top of site. Soon.

(According to the Wiki, this missive's title is a true Robin-ism.)

Isaac Singer's Revenge

Thomas Friedman's column today impels me to work out some ideas that have been bubbling away for some time.

Let's start with a sig from the archives,

Like everything else in technology, the cost of starting a startup has decreased dramatically. Now it's so low that it has disappeared into the noise. The main cost of starting a Web-based startup is food and rent. Which means it doesn't cost much more to start a company than to be a total slacker.
-- Paul Graham/2005

Nearly a decade ago, Graham defined in the new era of the sewing machine. While the sewing machine enabled industrialists to build factories, it also forced (pre safety net; and post, of course) women into slave wage piece work at home. These days, the cheap PC with 16gig, i7,and SSD is much like the sewing machine of 1850: anyone can build a web site. Or, with an R/RStudio installation, claim to be a high-powered quant. And, since anyone can, it's an exercise of low value, and most of the profit redounds to those few who can ensnare the many who do the work. Say hello to Oliver Twist; "More? Not on my watch, buddy. Keep typing, that ACA mod is due today and important people need their healthcare". Have to find some way to scrounge up $3,000/month for a cheap flat in that City By The Bay. I mean, California could demand all those starry eyed, but welfare dependent, tweens to spend 20 hours a week maintaining the DMV in exchange for the coin to pay for the latte`. Imprisoned drug dealers make the license plates, hungry tween coders run the agency. Birds of a feather, and all that. I wonder whether the Indians could still compete with jail labor? Just a thought.

Here's the difference between today's software "entrepreneur" and Hewlett and Packard in the 1950s (read up the Wiki piece):

Of the many projects they worked on, their very first financially successful product was a precision audio oscillator, the Model HP200A. Their innovation was the use of a small incandescent light bulb (known as a "pilot light") as a temperature dependent resistor in a critical portion of the circuit, the negative feedback loop which stabilized the amplitude of the output sinusoidal waveform. This allowed them to sell the Model 200A for $54.40 when competitors were selling less stable oscillators for over $200. The Model 200 series of generators continued until at least 1972 as the 200AB, still tube-based but improved in design through the years.

Real engineering, making a real product that had constructive usefulness. Today's tweens make twitter and instagram and sundry diverting entertainments. Could there be a greater gulf? I think not. But twitter and instagram are the sort of toys that one can make with little capital and little intelligence. Kind of like sewing blouses. Hewlett and Packard were Stanford EEs, for crying out loud. Today's generic tween is a PHP hacker who never took a math or database course, assuming said tween even spent time in higher education. Most of the tweens disparage education, if you scratch their very thin surfaces. One in 10,000,000 get to be Zuck (who may have cadged the whole thing from Those Other Guys).

Which brings us to the misguided, as he usually is, Friedman.

We're now in an era in which globalization and the information technology revolution have merged to drastically shrink what was the basis of our middle class for so many years: the "high-wage, middle-skilled" job.

Yes, but, this happened as a result of policy decisions, not the least of which was the "opening" of Chinese labor to American corporations. Economists of the micro- persuasion (and quants, who mostly work to maximize revenue for their corporation), mostly (always?) argue that the field is ignorant, and rightly so, of value judgment. Whatever the math says is the right answer, even if the answer is that oligarchs get 99.44% of wealth and the rest of the society starves in their service. Adam Smith (the real one) didn't agree, by the way. The difference between 1776 (when Smith's most famous book was written) and today is that the oligarchs have the NSA, et al, protecting them from the guillotine.

What Friedman, either out of ignorance or deceptive intent, doesn't talk about is that all of the human condition is relative. The USofA, post World War II, had a blue collar middle class because policy made it possible. After all, the war ended the Depression (thus proving the point that massive government spending will lift an economy out of collapse, a fact which disturbs the Right Wingnuts), and was still within the life experience of most of the country's citizens. The laissez faire policies which led to The Great Depression could have been allowed to return. They weren't, and until 1973 when the Arabs had enough of Israel and its enabler (us), life went well.

Time for another quote.

As mass production has to be accompanied by mass consumption; mass consumption, in turn, implies a distribution of wealth -- not of existing wealth, but of wealth as it is currently produced -- to provide men with buying power equal to the amount of goods and services offered by the nation's economic machinery.
-- Marriner Eccles

That quote has permanent place at the start of one version of this endeavor, and it is key to both quant and policy. It is key to quant because it refutes the nihilism of the quant ethos. It is key to policy because it provides the logical foundation for equitable decision making; you don't miss your water until the well runs dry, and you don't feel hungry until you've killed the golden goose. In both cases, the goal is growth of the society and economy without having to resort to the guillotine.

So, Friedman continues,

In today's hyperconnected world without walls -- when more Indians, Chinese, computers, robots and software can perform more average blue-collar and white-collar jobs -- the only high-wage jobs are increasingly high-skill jobs.

Wherein he blithely ignores a key factor in the collapse of the American white collar middle class: the invasion of cheap Indian (and others, to be fair) white collar labor. Tom, old sot, you can't have it both ways. Especially in just one sentence. I guess The Times doesn't edit your copy? Moreover, the number of high-wage/high-skill jobs is necessarily limited, since they're the explicit result of pyramiding the workforce. To put it plainly: more high-skill workers will only drive down the value of high-skill work while relentless automation diminishes the number required. Remember what the Indians are doing to us? D'oh!!!!!

In any case, there was only one Einstein. Establishing an educational system which winnows out the participants, until there's just an Einstein, leaving the rest to starve can't win in the long run. That's North Korea, in case anyone's still not getting the point. And more than a few white-flight suburban gated housing tracts; "so glad to get away from all those SNAWT".

He then goes on to quote one Andreas Schleicher, who runs the PISA (Program for International Student Assessment)

"Since the link between skills, jobs and growth is becoming ever tighter, it will be harder and harder for governments to address inequalities through redistribution"

Which is exactly wrong. Education, as well as innate intelligence and height and weight and beauty, is all relative. Telling kids that getting some level of education is the key to comfortable adulthood is just a lie when there isn't demonstrated demand for such skills (all that Arab Spring). The 20-somethings hoping to score the next instagram know that already, hanging out in those Bay Area coffee shops, furiously sewing away on their laptops. "Just one more blouse..." The cure for maldistribution is... wait for it... a theory of distribution which is inclusive rather than the Right Wing's exclusive. Robinson Cano` just got $240 million (over ten years) from the Mariners. Is that sufficient justification for middle schools to institute bayesball as top of the required curriculum? Not even the nuttiest of the Right Wingnuts would go that far (Codd, I hope so).

The real world of production has less and less demand for increasing high skill as automation continues on its exponential expansion. There was a time when one had to be a grizzled EE in order to design a cpu. Today? You just need to know a bit about EDA Lego machines. In a nutshell, it really has devolved to Lucy on the assembly line (if that doesn't sound familiar, do the search), with the exception of the folks who make the tools. The number who make the tools (and need the high-skills) is vastly smaller than those who use the tools (and don't need the high-skills, or get the big wages); as it always is in a Social Darwinist world. As Graham observed, not much of anything is needed to make a toy program.

There is a canary, at least one. It is called Apple, which has long since abandoned making computers as its modus vivendi. Some are convinced that it can't continue to grow revenue and profit by catering only to the 10% (or thereabouts) of the US/EU. Others believe it will score a deal with China Mobile. Even if the deal happens, on even generous (to Apple) terms, China's long adopted strategy of currency manipulation (I know, some Really Important Pundits insist that doesn't happen) may make for a Phyrric victory. It's by no means sure that there are enough Chinese who can afford a $700 phone to make much difference even if the deal happens. It's easy to grow 10%/annum when you're small; not so easy when you've eaten nearly the entire planet.

Finally, we get to the Supremes. You can (if you've not yet) read up Vaughan-Nichols' piece here. He's in the 'patents on software is stupid' crowd, as am I. The notion of software, and worse, business method, patents is destructive. If the Floundering Fathers wanted abstract ideas patentable, they would have said so. Algorithms were a known entity at the time, so don't prattle on about modern times.

In sum: patents grant to the smart (or, as often, lucky) an effective perpetual monopoly on some construct. The length of this perpetuity has been lengthened by Congress many times since the Founding (not that Congress bowed to pressure or anything, of course). Congress has done even worse with copyright, of course. Walt's heirs will own Mickey until we blow up the planet. Carving out high living by law (which is to say, policy) to the few won't help get us out of collapse; it will just accelerate the call for the guillotine. The data show us how Western economies were constructed when they did well, at the macro and household levels. The intelligent approach is to build such a construction.

Larry, No Curly or Moe

If you're in the mood for a compact refresher, or crash, course in probability and statistics (using R, but not overtly so), then Larry Wasserman's "All of Statistics" is where to go. A couple of pieces of the Preface are worth the price of admission. Alas, Amazon has banned copy from the Look Inside. Which makes no sense, since about the only thing you can see is the Copyright page and ToC. Silly.

Anyway, here are the two bits.

(my ten fingers):

Using fancy tools like neural nets, boosting, and support vector machines without understanding basic statistics is like doing brain surgery before knowing how to use a band-aid.

Amen, Brother! Now, could you smite those xml database zealots?

The Preface ends with a Rosetta Stone for stats/CS, and has been transcribed (by a student at Carnegie Tech) here (doesn't display worth a damn on Dr. Codd's page...). Whether both camps fully agree I don't know, but it rings true to my ears.

It's in the Bag

Thanks, once again, to Artima for a precious linked piece. I haven't followed javaworld, on purpose anyway, in some years.

Why this piece? Well, as sometimes mentioned on various versions of this endeavor, I'm a Bermuda-phile. I've only been there once, alas, but I've stayed attached through various on-line versions of its newspapers. For most of the time since universal sufferage (not so long ago as one might assume), the government was of the (generally accepted) black/native party, the PLP. Until the last election, when the (generally accepted) white/foreigners party, the OBA, won election. Since that time, the drumbeat for punishing the lower classes, in particular blaming the lower classes for the oppressive (generally accepted) price level, has gotten very much louder. Never mind that prices on a group of tiny islands (as a child, I was taught that the place was named The Bermuda Islands, not the more common singular of today; it is in fact an archipelago remains from the cone of a volcano) are driven by transport costs and the brutal income skew of those uppity white folks.

Couple that with recent reporting on crime and cost of living problems in the Dakotas' oil patch, and an age old theme emerges.

Which brings us to the techies, which term among the Kiddie Koders is presented as insult.

As the San Francisco Chronicle's Nellie Bowles reports, the word "techie" has become an insult, comparable to being called a yuppie in the late 1980s, only with less developed personal grooming habits.

That quote includes a link to the Chronicle story. But no matter. The javaworld piece is suitably tongue-in-cheek and a bit satiric. And then there's this:

Take San Francisco's Mission District, where cheap burrito joints, secondhand clothing and appliance stores, and mom-and-pop tiendas still abound. The average rental for a one-bedroom apartment there is now nearly $2,700. In South of Market, the former warehouse district where many of these bubble companies are based (full disclosure: as is InfoWorld), rents are approaching $3,500 a month.

One can never be too rich or too thin.

And I'll close this missive with:

They're actually Digital Bohemians and Aspiring Glassholes -- or D-BAGs for short.

I assume that those who created the acronym (or backronym) are aware what a real D-BAG is?

Schemas Don't Lie, But... Well, You Know

Well, the reactionaries are reacting. To be expected. The Big Lie of NoSql/xml is that such datastores are "schemaless", and thus able to handle any sort of "unstructured" data. RDBMS, on the other hand according to these snake oil salesmen, is trapped in the table/column straitjacket of the RM. The thing about Big Lies: they have to be big in order to flummox the ignorant. An Organic Normal Form™ relational schema is orthogonal by definition nearly immune to side effects from mods (outside the client side code uninterested in the change), while a hierarchy (IMS/xml/HL7/whatever) change propagates at least all the way down. And how does one find related data? The xml zealots are still adding knobs and whistles.

Dave Kellogg, cashiered CEO of MarkLogic (among many) is quoted in an article thus:

I met many Oracle-DBA-lifers during my time working with the government. And I'm OK with their personal decision to stop learning, not refresh their skills, not stay current on technology, and to want to ride a deep expertise in the Oracle DMBS into a comfortable retirement. I get it. It's not a choice I'd make, but I can understand.

Of course, that's bullshit. MarkLogic, initially and for some years, explicitly promoted itself as The xml Database. And, as you know gentle reader, xml is just IMS in, mostly, plain text. xml, by definition, is a hierarchical datastore (when used as such). MarkLogic is just IMS done badly, and perhaps a bit cheaper. Each and every xml definition is a rigid hierarchy. That .xsd (and given the label schema) came along later is part and parcel of the kludge. The xml folks have been sneaking in relationality at least since IDREF, with limited success.

He goes on to say:

It had never occurred to me, for example, that in a $630M project -- where MarkLogic might get maybe $5 to $10M -- that someone would try to blame failure of what appears to be one of the worst-managed projects in recent history on a component that's getting say 1% of the fees.

Not to be all too macabre, but the Challenger o-ring was nowhere near that much of the cost of a shuttle. The point, which Kellogg attempts to redefine out of existence, is not how much of the total bill goes to MarkLogic, but how much of a bottleneck it is. And it is the bottleneck. OLTP systems have no business in NoSql and a colossus of code. That's what today's Kiddie Koders' grandpas did in the 1960s with COBOL/VSAM/IMS. The problem with MarkLogic isn't that retiring Oracle developers are unfamiliar with new tech, but that MarkLogic isn't new tech and is crap for OLTP. The "core database". Yeah right.

The current CEO (well, this week anyway) Gary Bloom, in The Wall Street Journal (of course):

Mr. Bloom said CMS needed to process non-standard data types from multiple vendors.
...
If CMS had elected to use a SQL system, its programmers would have had to build a common data model, or schema, to describe the disparate data sources within the application.

Ok, so what are these "non-standard data types"? Plutostrings and minervaintegers?

So, MarkLogic just automagically slurps up any old byte stream and figures it out? Not really:

The trick is that content not in XML must be normalized; that is, converted to XML. MarkLogic has developed some proprietary methods to perform its data management operations.

In other words: MarkLogic does an ETL exercise, just as any sql RDBMS would do in the same design. Not only that, but any benighted xml data source has to be munged to look "just so" for MarkLogic's structure. This schemaless prattle is just stinky poo. (Although I object to calling MarkLogic's exercise "normalized".) Well, unless the sql developers were smart and federated the databases and read (writing takes a bit more work) from the original. While there are provisions for user defined data types in all the industrial strength RDBMSs, I've not seen them used much. The fact is: one can read a string, an integer, or a float from any foreign database and not have a problem. The principal source of angst would be word size of the machines.

Back to Kellogg:

Oracle was non-standard in 1983. Thirty years later it's too standard (i.e., part of an oligopoly) and not adapted to the new technical challenges at hand. All because some bright group of people wanted to try something new, to meet a new challenge, that cost probably a fraction of what Oracle would have charged, the naysayers and Oracle lifers will challenge it endlessly saying it's "different."

While I agree that Oracle is rapacious, and a fair number of its clients think so too, it's worth noting (as done in an earlier missive) that ten years on, Oracle was a profitable $584 million company. Ten years on and MarkLogic is still sucking at the VC teat; since it's private, we don't know whether there's been any profit, but needing more VC money indicates, not much. While the likes of Kellogg and Bloom and the rest of the xml snake oil peddlers continue to bray about new, disruptive tech; most folks who've been in the real world for a while know the real story. And the real story is that xml, as datastore or data transfer, is a dirty bung hole from the mid-60s.

Oh, and that oligopoly? It's called ANSI. The SQL standard, which each vendor commercial or otherwise amends, extends, and bends to stay "different and better" than vanilla SQL. There's a reason that applications get locked into Oracle or SQL Server or (less often, alas) DB2. Although on IBM's z machines, DB2 is just about the only relational database one can sanely run. Calling ANSI an oligopoly is, to be charitable, mistaken. Were I in a ranting mood, I'd likely say, bloody lie. And their user code (stored procedure language) isn't standard. Yet.

If you're interested in a case study, then here is one. Note that XQuery is an bung hole kludge compared to sql. I guess that's one reason coders like it: a moat of obscurity around their jobs. RBAR's revenge. Just what coders who can't grok simple set theory can rally around.

Positive Feedback

You know that screech that happens when the roadies have messed up the mike/speaker configuration? Positive feedback.

Well, the folks at DataMind have produced a ranking of Coursera stats-with-R offerings. I suppose it shouldn't be surprising that social networking is the number one course, since the ranking is based on Facebook and Twitter. One wonders whether Einstein could have produced two theories of relativity, which didn't even win him the Nobel (his work on the photoelectric effect), as well as a raft of others if he'd been of the self-absorbed generation?

Unhand Her, You Cad; The Map

Well, there's been a tonne of ink spilled about Bezos' spiders' web. Everyone concludes that it's just nuts.

Have a look at this map. Long range octocopters, perhaps? For the 10 mile radius that Bezos mentioned, and given that most of the centers are more or less in the middle of nowhere (downtown Boston is sort of expensive real estate), delivery coverage isn't much. I saw a comment in my journey which postulated that Amazon would send out semi-trailers loaded up with flying spiders, find a central point, jack-in-the-box the top of the trailer, and spew thousands of spiders. Snakes on a plane seems tranquil by comparison.

Who Will Predict Us From the Predictors?

Revolutionary Dave posts 14 predictions from a webinar run, in part, by his company. Part of my job is to keep the fires burning for high quality prediction, so I would be remiss if I didn't fulfill my function of predicting the predictors. Herewith my review of each prediction.

1 - Analysts will matter more than data scientists

Well, they always have. From the London Whale all the way back to Long Term Capital (wanna buy a tulip?), with a Great Recession diversion in between, whenever policy and data collide, data loses. More's the pity.


2 - R will replace legacy SAS solutions and go mainstream

Perhaps. Revolution sells a RBAR version of R, and appears (to me, at least) attempting to encroach on SAS's meme: "your all in one data platform" (my instant quote, but SAS said much the same in the Goodnight Old Days). By the mid-80s SAS was selling itself as a database, not just a stat pack.

This paper examines the database features of the Version 6 SAS System [1986] and compares them to the services offered by several popular DBMSs. The conclusion is that the SAS System can provide a cost-effective alternative to a commercial DBMS for the storage of data.

I don't follow SAS marketing these days to know whether they're still pushing that notion.


3 - Big Data will bring its "A game" in sports marketing

Can't say much about that. Nate Silver started out in bayesball, so it could be. Not that I care even a little bit.


4 - Hadoop moves from curiosity to critical

Codd, I hope not. All this NoSql stupidity has been revealed by MarkLogic and healthcare.gov. It is true that stat types have always viewed data as flatfile images. Have I mentioned earning kopecks in the basement of the UMass graduate center typing 059 input for BMDP? Stat types remain enamoured of flat. PL/R has shown them the way, and both Oracle and SAP are providing similar functionality. Still, they persist in viewing the computer as a Friden adding machine.


5 - Gartner's prediction that the line-of-business will drive analytics spend will happen

Again, this is not new. "Mad Men" have been using stats since Hector was a pup. The only difference is that R (and before it, 1-2-3 and Excel) with i7 chips and 32gig PCs make it possible to do such analyses without a server, much less a mainframe. Beware: never let the children play with the sharp cutlery.


6 - Visual analytics continues to grow but users need more

Well, "users" remain in love with pie charts, so I'm not so sanguine. Between lattice and ggplot2, R has more than enough built-in graphics to give Tufte a massive MI. Don't do that to him; he's a nice guy.


7 - Analysts lives get more complex, but also easier

Again, beware children and sharp cutlery. The London Whale made a mess just with Excel. Hmm. Progress requires change, but change isn't necessarily progress.


8 - Predictive analytics will no longer be a specialist subject

And, in due time, the same will happen with neurosurgery. Utopia will proliferate.


9 - Customer analytics is the next big marketing role

Next? What does anyone think Google and Amazon and ... is doing with all that personal data the sheep provide? Curing cancer?


10 - A new analytics stack will emerge

Revolution R, I suppose he means.


11 - Location meets big data analytics

Again, old wine, new bottles. While a graduate student, I did some stat work with one of my undergraduate professors. He called his product "Shift and Share Analysis", wherein we took the smallest grain SMSA data to predict the best location for local bank branches. Buffalo Bob (yeah, that's what we called him when he wasn't looking) was by no means the first to try this. Big data doesn't help here, much.


12 - NoSQL meets analytics

Analysts have always hated RDBMS, so this is nothing to predict.


13, 14 he doesn't list.

A Baker's Dozen

The original MarkLogic implementation called for one server and one backup. Bad idea, but these are xml Kiddies. Now we have the White House self-report on progress to normalcy.

Note page 5, where the slide shows 12, count 'em 12, dedicated servers. And sundry other hardware. Ain't bloated flatfiles so much fun? Of course, we (at least, I) have yet to see a report of who, by name and rank, insisted on MarkLogic. My guess is it won't be a political payoff (much as certain segments of the political spectrum wish it to be so), but rather a Supergrade who's been in love with HL7 and other diagnostic data processing aspects of health care. I have mentioned that I spent some years writing (Progress based) a pre-qual database appication? HL7 was even then much loved. Bah. It's a fact that MarkLogic turned to health care when the first few avenues for peddling that scurvy xml "database" bore no fruit. Mixing metaphors, I know, but the colossal incompetence of this pisses me off. The Right Wing has been served up a watermelon pitch by a bunch of knuckleheads. Be rich or die. Yeah, we're so much more civilized now than 100 years ago.

Thanks for the Memories

Well, it appears that the OCZ saga is finally, really, definitely ova. Toshiba is paying $35 million for the carcass. Which isn't much, and likely won't be much more than the debt (Yahoo! list this as $17 million, but that's as of 31 August; Hercules had all assets under its lending). Two points of interest here. First, during the day the share bottomed at $.10 and went to $.26, probably on shorts covering and fools speculating. During AH it came back a bit, then tanked to $.13 as I type after the announcement.

Second, and of more interest, is the fact that Violin Memory is attached at the hip to Toshiba. Methinks that Toshiba has a plan.

Unhand Her, You Cad

There's that line, sort of, from 1930's cliffhanger serials; "Unhand her, you cad!!" I was watching something else, and I'm not a fan of Bezos anyway, so I missed the piece about the drones. Helpfully, it's now made print (well, sort of). As predicted in these endeavors some time back (I'll leave the spelunking to the reader), Amazon is transforming itself into a Brick & Mortar outfit, just with some Emperor's New Clothes.

In order for 86% (<= 5 lbs.) of deliveries to be made with flying spiders, there have to be warehouses, aka stores, within a fairly small distance of 100% of Amazon's customers. Given technology (battery capacity or internal combustion engine miniaturization) a SWAG on delivery radius is on the order of 30 minutes (60 minutes round trip) flight time. So, 100 mile radius requires ~ 200 mph. A sky full of these flying spiders at faster than (all but raptor style firing platforms) helicopter speed? Not likely. So, lots more warehouse, aka stores, will be needed.

Densely populated areas provide the best bang for the buck, but how does delivery work for apartment buildings? Will a storage box, with one way top hatch have to be built? Who pays for that? How will it be engineered to allow packages, but not rain/snow/sleet/rodents? "Bob, we've got RATS!!!!"

Publicity stunt. Amazon still makes no money, and continues to morph into Wal-Mart, without the parking lot, but lots of humans scurrying around getting widgets. Silly.