Stockholm Syndrome [update]

Shiller gets to go to Stockholm and bloviate, while I'm stuck here in Arctic New England. Life ain't fair. At least, he's provided a precis of the proceedings. And, in the process, elevated my view of Left Wingnuts.

There are two major points: first, quants mostly get it wrong; and second, homo economicus is a myth. The first occurs because the second is true.

Does it make sense to suppose that economic decisions or market prices can be modeled in the precise way that mathematical economists have traditionally favored? Or is there some emotionality in all of us that defies such modeling?

Of course. The field was originally named 'Political Economics', and there's little in human existence that's more attached to emotion than politics. How else to explain Kansas? All those gun loving, God fearing folks voting for the corporate manipulators eager to keep them poor. Well, except for those few who get fat farm subsidizes. Not that such moolah is welfare, of course.

It is hard to sum up all this discussion, however, because of a basic problem: defining "rational."

Aye, matey, thar's the rub. A frequent quote from the housing bubble amounted to: "we all have to keep dancing as long as the band plays; all the others are dancing". Another way to put (from J.K. Galbraith, variously phrased) it: "financial genius is a rising market". Thus Li was able to foist a mechanistic formula on human behavior, which is to say, where they make the rules rather than Mother Nature. The ultimate basis of quants is that the rules are, more or less, stable and *not under the control of the analyst*. Such analysis works well with microarray data, for example. This is the crux of Shiller's point, too.

He closes:

The question is not simply whether people are rational. It's about how best to describe their complex behavior. A broader notion of irrationality may someday be reconciled with one of rationality, and account for actual human behavior. My bet is that real progress will come from outside economics -- from other social sciences, and even from information sciences and computer engineering.

In other words, don't assume that one can model human decisions using only "normal" data. If policy (those musical chairs during the housing bubble) trumps data whenever enough people decide that the policy is more fun, then data loses. The best that financial quants can hope to do is front run changes in money flows. Which is not to assert that data is useless in bubbles; those that noted the massive unsticking of house prices and incomes made money going up and coming down. But not so many. And that means Joe Sixpack with his PC and standard data and R doesn't stand much of a chance to beat the pros with their HPC machines and proprietary data. You're better off tracking Briefing.com, with an eye to spotting grey swans; watch for events.

[update]
I've spent the last few hours looking for more views on the subject, and found this one. Very interesting. Could have been written by me (well, in a dream, may be).

From page 20:

The very complexity of the mathematics used to measure and manage risk, moreover, made it increasingly difficult for top management and boards to assess and exercise judgement over the risks being taken. Mathematical sophistication ended up not containing risk, but providing false assurance that other prima facie indicators of increasing risk (e.g. rapid credit extension and balance sheet growth) could be safely ignored.

And, from page 24 (the punch line, in the gut):

New generations of students will have to use the tools and techniques of QRM [quantitative risk management] wisely in a world where the rules of the game will have been changed.

Protect the Children: End Graphic Violence

In the beginning was Bachman
He looked upon the Firmament of Data, and saw Chaos
And lo unto Him, he created IDS, which begat IDMS
These holies were as the spider's Web, one scion goeth to all kith and kin
And they to the scion, each to many, as the tendrils of the spider
It was good

Thence came the Business Machine of all nations, and lo, it would not pay tribute to Bachman
The Machine pruned the spider's web, each from one and only one, which was called IMS
Lo, the Machine was prideful, having banished the spider's web from its machines
It was good for the Business Machine of all nations

Darkly, a spawn in the Business Machine looked upon this new firmament of data and saw tyranny
He preached of relations and their power to erect any monument, whether like the spider or the serpent
No longer shackled to the rack of pointers and hard links, the people were free expand their data as need be
This prophet was called Codd, lo many spake of him as God, yet many in the Business Machine were fearful of him
Mighty lords of the Business Machine took heed of this fear and consecrated the Chamberpot to write the text
Only when the Oracle child did propagate boldly would the Business Machine acknowledge its patrimony

So went the years, for many of those years, when the apostate youth, innumerate as wide as the heavens, looked blindly at themselves
We do not understand the Set and our relations are carnal, verily we must have a data supine before our code
Ye, they consorted with satanic harlots, producing an evil spawn which they called Graph, which was only Bachman in sack cloth

And so, the great ouroboros has deceived the youth, who must wait for the second coming of the Codd
May they not bow before a false Codd as lo they have done for these many years

Major Kong Falls Over Switzerland

Net neutrality is dead!!! Long live Robo Cop!!! Or, what does the future hold now that net neutrality is gone?? The problem for quants is that, while data driven analysis makes perfect sense in the real world, i.e. the ones driven by forces of nature, the process gets squishy when the rules of the game change at the whim of humans. Most often, those rule changes are made by those who benefit from the change, and which change is often hidden from the public at large. And that group includes many of the quants.

When there is massive policy change such as this, the job of analysis is to consider how behavioural incentives have changed. What behaviours, previously forbidden, and which can turn a profit, are now legal? Who wins and who loses? The swan may not be black, but gray enough as to not matter.

Verizon, the entity which initiated the court action says:

"Verizon has been and remains committed to the open Internet, which provides consumers with competitive choices and unblocked access to lawful websites and content when, where and how they want," the company said in a statement. "This will not change in light of the court's decision."

By way of comparison, after the voting rights act ruling, those who made the ruling claimed that the law had worked, and nothing would change, because, well, the law had worked. Within 48 hours, those purely democratic Southern states which would never, ever return to past bad behaviour, set about undoing voting rights. You can look it up. Follow the incentive.

Anyone who actually believes that is a fool. The whole point of killing neutrality is to segment the market, and dump any segments which aren't "profitable enough".

In 2002, the agency said Internet service should not be subject to the same rules as highly regulated utilities, which are governed by regulations on matters like how much they can charge customers and what content they can agree to carry.

So, who was running the FCC in 2002?

Michael K. Powell, who was F.C.C. chairman in 2002 when the agency set up its Internet governance structure, said, "Today's historic court decision means that the F.C.C. has been granted jurisdiction over the Internet."

Then, who is Powell? Well, he's Colin's son; appointed by Clinton; and toadied for Bush. He now heads up the cable lobby.

Who, in the current world of Darwinist Capitalism, is the master of market segmentation? That would be Apple. Market segmentation means not only varying price to capture consumers' surplus (the notional version of the gambit) but also to remove market segments from supply. There has been in the common lexicon for some years the term 'digital divide', wherein the less wealthy have less of the digital domain. If the less wealthy don't have access to the innterTubes on a level playing field, this becomes yet another case of not wearing an Old School Tie (the Brits will get that; Yanks may be not so much). To the extent that current affairs reporting becomes beholden to innterTubes for dissemination, we can expect a few changes. First, the ever more concentrated control over the innterTubes will provide only 'good news'. And, second, the less wealthy will get 'good news' which portrays them as the cause of all that is bad with society. Fascism, as defined by Mussolini, is government by and for capitalists. As the information superhighway becomes the only path for information, and is controlled by a handful of capitalists, what's the incentive to not mold the information?

Remember: the 'free' parts of the innterTubes run on adverts (Wikipedia being an exception), and Apple has shown that segregating out the non-buyers from its sphere is more profitable. With the carriers now free to segregate their customers, why would they not? Why would they not make deals with Netflix, et al, to provide highspeed connections to wealthy neighborhoods/towns/cities, but not to South Buttfuck? Of course they will. There are sites already which catch my use of Adblock Plus, which is more to preserve bandwidth than to avoid ads (although I've never clicked on one and never will). Some won't let me in, others can be fooled. Market segmentation at work.

Of course, there is no such incentive. The incentive is to, Darwinist/Rand fashion, crush the weak under the boot heel. And, there is no punishment for being a bad actor.

The allure of innterTube ads is that they're more focused than print ads. But net neutrality limits the ability to segment aggressively. The time will come when sites will not only block those that don't view or click ads, but don't buy. Who better to enforce this segmentation than the carriers? Rather than each advert owner having to keep track, the carriers offer up a throttle: they'll keep track of those who buy and those who don't. Those who don't get blocked from some sites, and get reduced bandwidth in the bargain. In due time, the innterTubes will become like the Apple ecosystem: of, for, and by the top 20% of the wealth curve. That Old School Tie will be adorned with the Verizon Hyperspeed tie tack.

Listen to Henry

I've not attempted to verify the following:

Samsung is in a unique position in the SSD market. It's the only company in the consumer SSD business with a fully vertically integrated business model and zero reliance on other companies. The controller, NAND, DRAM, firmware and software are all designed and produced by Samsung...

but I've certainly seen similar versions of that statement over the years. The immediate response was, that's what Henry Ford built. The Ford Motor Company for many decades had a simple structure: raw materials in one end, Fords out the other. Keep all the profit from each step. Exactly the opposite of today's out sourcing meme.

A signature which waxes and wanes in the prologue of these endeavors:

A business that makes nothing but money is a poor business.
-- Henry Ford

Big Dig, Big Data, Big Deal?

Among the largest old city rehab efforts in the history of the country was The Big Dig in Boston. It finally finished, late and over budget. But it includes one of the prettiest bridges on this side of The Pond. Why is it that any European country manages to do civil engineering with greater beauty in its homeliest structures than the USofA does in its best? Why is it that virtually every "innovation" in automobiles since Henry Ford was created by some European company? Just asking.

Recently, this endeavor mused on the Death of Big Data. Or, perhaps, high morbidity. Watson has been getting ink recently on blogs, so it's not a surprise that IBM would take the opportunity to discuss the machine. And from what I can't tell is whether Watson is sui generis, or a model shippable in quantity. From the wiki description, it's built from off the shelf parts. Except, of course, for the software. What's even more interesting: Watson doesn't make it to the top 500 of supercomputers, and appears to be I/O bound by *hard drives*:

According to John Rennie, Watson can process 500 gigabytes, the equivalent of a million books, per second. IBM's master inventor and senior consultant Tony Pearson estimated Watson's hardware cost at about $3 million. Its performance stands at 80 TeraFLOPs which is unfortunately not enough to place it at Top 500 Supercomputers list. According to Rennie, the content was stored in Watson's RAM for the game because data stored on hard drives are too slow to access.

I guess these smart folks never heard of SSD!!!

What's even more interesting: some of that software is Prolog:

We required a language in which we could conveniently express pattern matching rules over the parse trees and other annotations (such as named entity recognition results), and a technology that could execute these rules very efficiently. We found that Prolog was the ideal choice for the language due to its simplicity and expressiveness.

Some background, some my own, on Prolog.
- it was created within months of Codd's relational model paper
- it uses what amounts to being a normalized, in memory, database. most Prologs refer to this data as "database".
- while at OMS in the early 90s, a couple of my colleagues attempted to build an AI sub-system for the main product, medical pre-qualification, in its database/4GL (Progress). never got very far, if only because Progress has never been particularly relational or normal in application
- while at CSC I had to endure a Prolog mutant called GraphTalk, which CSC had bought up a few years before from France. of course. my colleagues at CSC turned this mutant into COBOL/VSAM coding. Yum.
- one of the current uses of Watson is in medical diagnosis. hmm. twenty years too soon was I.

There's still a commercial version of Prolog/datastore called Amzi! (yes, the ! is part of the name just as Yahoo!). And guess what? It's major market is business rule and decision support implementations. As it happens, Prolog syntax/semantics is more alien to C inculcated coders than even R or SQL. But, according to its zealots, Prolog systems are orders of magnitude more compact than imperative (e.g., java/C/FORTRAN) equivalents. Kind of like what relational zealots say about RM/SQL databases versus flat files.

So, today's Times has a puff piece from IBM on the use and future of Watson. As others have concluded, but with suspicion, IBM sees Watson as central to its commercial success.

IBM's elevation of Watson is the biggest illustration yet of the technology industry's faith that so-called Big Data holds promise for the economy -- and the failure so far to meet that promise.

Big data is just descriptive statistics, since one has all the numbers. Look at any Baby Stat book, and an early chapter (and likely the shortest in the book) will cover all one needs to know about descriptive statistics. I know, I know. Big Data is really about correlation and finding the correct distribution. Mostly, for commercial uses, it's about finding a few golden correlation needles in a haystack of choices by millions of people. So you can spit more enticing ads at a few of them. I wonder how many of these Big Data projects were ever subjected, a priori that is, to a rigorous cost/benefit analysis? While Watson is a multi-million dollar machine, most Big Data can be handled using R or PL/R on a pumped up Dell. So, in such cases, one need only find a few silver needles.

The apostates are beginning to crawl out:

Likewise, IBM will have to sharpen its focus and what it delivers, said Henry D. Morris, an analyst at the consulting company IDC. "Big Data by itself isn't value, it has to deliver recommendations about what to do," he said. "They have to show people not just analysis, but action. They understand that there are challenges ahead."

By the way, I'd love to get invited to the Watson party (fat chance, of course): the staff will be located in the East Village. If you have to ask where the East Village is, you're so uncool.

R, How to Write It, and Some Bad Quant

The stream of R books continues. Amazon sent this one along this morning. This is the chapter 10 Title:

Loops, The Un-R Way to Iterate

A step in the right direction. Which brings us to a couple of posts that popped up on R-bloggers, also this morning.

"R as a second language".

In most languages if one wants to do something many times the obvious way is using a loop (coded like, for() or while()). It is possible to use a for() loop in R but many times is the wrong tool for the job
...
There are some complications with some of the design decisions in R, especially when we get down to consistency which begets memorability. A glaring example is the apply family of functions and here is where master opportunist (in the positive sense of expert at finding good opportunities) Hadley Wickham made sense out of confusion in his package plyr.

Both statements ring true: the apply() functions are set-oriented (fits the RDBMS mind-set), and their lineage makes the syntax a Google-able construct which in turn makes the plyr package so much better.

Hot on that post's heels is this one, with a similar point to make.

... in R it is much simpler to take advantage of the R idioms to get there a lot faster. With this approach there is no need for loops or conditional branches. There is also no need for iterative array construction. Instead everything is done in one shot using a set-theoretic approach combined with function transformations.

Again, we read about set. While I've always been irritated by the fanboi need to cloak R in OOD/OOP/functional mystique (rather than actual syntax/semantics/structure), the (nearly) declarative approach to syntax (and, thence semantics) is comforting. R is just FORTRAN's function/data, and folks should just let it go at that. Iteration belongs on the metal, modulo true array processors (which don't explicitly need it); the continuing bottom of the brain stem memory of assembler (and, face it, C) likely accounts for loops in so-called higher level languages. Bah.

So, now we're on to concerns of quants. It seems that Big Data is Dead?

Google Trends shows searches of the term "Big Data" peaked in October, ending a nearly ceaseless climb that began three years earlier.

Huge flatfiles of un-normalized bytes may finally be seen as The Emperor's New Clothes by The Deciders out there:

In the case of Big Data, this probably means less focus on back-end technologies like new types of storage or database frameworks, and a rethinking about how best to integrate human knowledge, algorithms and diverse sets of data.

Can you say: "I want my PL/R"? And have I mentioned: the free-as-in-beer DB2/LUW can accommodate 15 terabytes? You don't get all the really cool bells and whistles, but that's pretty big data.

And now, for a little night music. Jenny Lind was the "Swedish Nightingale", which is close enough to a canary for government work. "We have our canary!" cried some. I'd noticed that house prices had been on a run, but not enough to put fingers to keyboard. This writer is from the AEI, so he blames The Damn Gummint, of course:

Both this bubble and the last one were caused by the government's housing policies, which made it possible for many people to purchase homes with very little or no money down.

Which, of course, is baloney. The bubble was motivated, widely known by anyone not hanging out with Mussolini's ghost, by the financial sector seeking high (but risk free) returns. So, they invented 'interesting' mortgages in order to sell more houses/mortgages which could be packaged together to make high yielding, but low risk (housing is always safe, isn't it?), securities. It wasn't the $10/hour bus drivers who went to CountryWide and said, "make me a mortgage that looks like this". Not the way it happened. The author is clearly confused, because he says:

In 1997, housing prices began to diverge substantially from rental costs. Between 1997 and 2002, the average compound rate of growth in housing prices was 6 percent, exceeding the average compound growth rate in rentals of 3.34 percent. This, incidentally, contradicts the widely held idea that the last housing bubble was caused by the Federal Reserve's monetary policy. Between 1997 and 2000, the Fed raised interest rates, and they stayed relatively high until almost 2002 with no apparent effect on the bubble, which continued to maintain an average compound growth rate of 6 percent until 2007, when it collapsed.

But, later on:

They claim that people will not be able to buy homes. What they really mean is that people won't be able to buy expensive homes. When down payments were 10 to 20 percent before 1992, the homeownership rate was a steady 64 percent -- slightly below where it is today -- and the housing market was not frothy. People simply bought less expensive homes.

Ya can't have it both ways, Jake. The moolah, whether Chinese, German, or American, flooded the mortgage process, pushing up housing costs, while the Banksters went about making mortgages for this housing stock available to the larger number of buyers needed to "clear the market" (cute econ speak, what?). Affordability was swindled in order to conjure the securities. In the short term (and we're definitely in the short term with the Bubble/Recession), only so many units could be built, yet to accommodate the Giant Pool's valuation (Banksters chasing mortgages with ever more bulging pockets) prices had to rise to "clear" the Pool's value. The builders were the ultimate winners:

When anyone suggests that down payments should be raised to the once traditional 10 or 20 percent, the outcry in Congress and from brokers and homebuilders is deafening.

Couldn't agree more; said it before.

I disagree with the use of rental payments as the measure of bubbles, however. There's no material difference between rent and (full load) mortgage; just go read up on what's going on in the Oil Plains States to see the effect of localized inflation of housing. The measure that matters is (local, for some definition of local) median income. No, what drove the Bubble/Recession was the flood of moolah. Without that flood, no amount of fiddling with mortgage lending rules would make a difference; there'd be no reason to generate ever more mortgages made possible by fiddling the rules, since there'd be no increased demand for the resulting securities; no demand means no supply, Laffer be damned. One could argue, even, that had the chronology happened the other way (relaxation of rules, first), no flood of money would have been conjured, since such mortgages would be clearly wonky to those who *need convincing* to part with more moolah, which they either didn't have at all or would have to move from other instruments. It was at the behest of the Giant Pool that the mortgages were conjured. Causation makes a difference, both in assessing blame and determining regulation from here.

In the end, it wasn't The Gummint which drove the The Great Recession. Without the flood of moolah seeking better than Fed rates (so, I guess the author should be pointing the j'accuse finger at Greenspan?), the housing bubble couldn't have happened. Without the quants figuring out how to abuse copulas, it wouldn't have happened. Without the collapse of demand for technical brains in technical professions (thus being 'freed up' to pursue finance, in the sense of automation 'freeing' farmers to work in Detroit after the turn of the 20th century), it wouldn't have happened. Without the decline in median income, and thus the willingness to spend unearned (and unexpected, largely) equity on consumption, it wouldn't have happened.

The author's most egregious lie:

By 1994, Fannie was accepting down payments of 3 percent and, by 2000, mortgages with zero-down payments. Although these lenient standards were intended to help low-income and minority borrowers, they couldn't be confined to those buyers. Even buyers who could afford down payments of 10 to 20 percent were attracted to mortgages with 3 percent or zero down. By 2006, the National Association of Realtors reported that 45 percent of first-time buyers put down no money. The leverage in that case is infinite.

Yet, he contradicts himself with regard to the driving enemy:

In effect, then, borrowing was constrained only by appraisals, which were ratcheted upward by the exclusive use of comparables in setting housing values.

And who drove the appraisals? I guess it must have been The Gummint? For what it's worth: Fairfax County Virginia was using stat analysis of home sales (as opposed to on-site inspection) to do appraisals by, at least, the mid-1970s (I was there). Such software is a big seller, and has been for a long time. Here's one example:

Based on a 20-year proven relational data model and common value approaches (Cost, Market, Comparable Sales and Income), the CAMA module produces reliable and accurate results which also apply if or when an assessment must be defended through the appeal process. Comprehensive comp sheets, ratio analysis and a CAMA valuation sheet that lists all items on a property and the dollar results of each component help automate the document management process.

For the finance types, infinity is (sorta, kinda) accurate. But not for a bus driver earning $10/hour facing an ARM reset. Not even close. The mortgagee is just a homeowner, not a titan of Wall Street. There's no 'real return' to be earned by the homeowner; the homeowner doesn't use the house to build either more or better (or both) 'psychic return' home widgets which he then sells to the market (a la Herbalife). Doesn't work that way. A house isn't an assembly line, or one robot, or even a simple milling machine; it generates no real return to anyone, not even the finance guys (they only get moolah). Leverage is just a red herring.

So, clearly there's more moolah flowing (not, yet, flooding) to the mortgage market. Recent stories have shown that hedge funds (at least) have gotten into the landlord business, buying up large tracts all to once. A more cogent analysis (assuming there's sufficiently granular data) would measure separately for owner-occupied and investor-owned units. It's pretty obvious that median income measures (which are still stagnant, at best) don't support increasing prices. I'd bet on deep pockets looking to monopolize SMAs fighting amongst themselves. Lots of money to be made being the major/sole source of shelter in an SMA.

It's Not a Joke, It's a Pun

It's now 2014, and the (amateur) punditocracy finally show some signs of understanding Amdahl. Here's exhibit A. It's been a bit less than a decade since Intel started with multi-core (distinct from multi-chip package) chips. One would think that's plenty of time to have figured out teh issue. It's not a new problem, what with the likes of Thinking Machines having found a market wanting when it tried to build and sell such machines as servers back in the 1980s.

However, there's a catch. While there are some "embarrassingly parallel" workloads out there, they're usually not all that common in the consumer space. Why? The truth is that many applications and algorithms aren't well suited to parallelization (i.e. being coded in a fashion that allows the work to be distributed across a huge number of chips), so at the end of the day the vast majority of consumer applications (whether it's PC games or mobile apps) benefit far more from having more performance on 1 or 2 cores than less performance per core but more cores.

Gee, ya think?

Single core with ever increasing clocks was the key to superior performance for decades, until Intel and the rest discovered that physics made it clear that The End of the Road (a great unkown novel, by the way; if you don't know Barth, you've missed out majorly) had been reached. So far as meaningful, i.e. linear with respect to historical increases, new performance was concerned not so much.

Which brings us to New Year's Predictions. No Cringely am I (not bothered to follow him, either one, for years), but what the hell. I've been goaded so here's two.

From Revolutionary guys, I was moved to comment

Nope. The big thing in 2014 will be embracing of in-database analytics with PL/R-ish functionality in all the major databases. SAP/HANA is the prototype. Oracle is a bit behind, but does claim to have something that sounds like PL/R. DB2 is still doing SPSS from its side, alas; although they do have a Netezza (nee, Postgres) implementation for the Z machines.

As corps begin to realize that Big Data is a limited phenomenon, they'll also see the advantage of providing analytics in their applications, and that is most easily done from within the database engine. While it's been a bit more than two years since the Triage piece, and I despair every now and again that the deciders (in the C suites and pundit thrones) will ever get it, reality appears to be bopping dem boys upside the head. Good for reality.

And, for the second, will be the return of very old fashioned host/terminal computing. One might argue that The Cloud is already a manifestation of same, but the point of the prediction is a turning away from client-centric coding (including "managing transactions in the client" mentality) to server-centric data management, relegating the client device to display/input duties.

So, come back in a year (but keep those cards and letters coming), and we'll see if intelligent life in IT has poked its spiny little head above the grass.