I See the World in Black and White

Once again, thanks to "Revolutionary" Dave for finding something interesting in the world of quant. The post-mortem is too long for the average trip to the loo, but the appendix on Value at Risk (VaR) which is the metric to be modeled, can be read. VaR is just a number of dollars assumed to be lost under varying circumstances (wiki piece). Dave's take on the Excel/R conflict is spot on.

The trader who had instructed the modeler to develop the new VaR model (and to whom the modeler reported at the time), CIO Market Risk, and the modeler himself also believed that the Basel I model was too conservative -- that is, that it was producing a higher VaR than was appropriate.

This is the heart of the continuing quant problem: unlike natural/mechanical processes, human processes (accounting and finance especially, since they only exist as games with arbitrary rules) are subject to "unexplained" perturbations. This is what Black Swans are. Black Swans exist in nature, the archetype being the Alvarez/Yucatan asteroid. The difference is that such natural Black Swans are exogenous to the process, while human Black Swans are largely endogenous; they happen because participants can and do change the rules for self-benefit. Such manipulation is generally impossible to model.

The Model Review Group performed only limited back-testing of the model, comparing the VaR under the new model computed using historical data to the daily profit-and-loss over a subset of trading days during a two-month period. The modeler informed the Model Review Group that CIO lacked the data necessary for more extensive back-testing of the model (running the comparison required position data for the 264 previous trading days, meaning that a back-test for September 2011 would require position data from September 2010). Neither the Model Review Group nor CIO Market Risk expressed concerns about the lack of more extensive historical position data.

Again, using more historical data may only have been useful for face-validity purposes. As always, the fiddlers fiddled:

There is some evidence the Model Review Group accelerated its review as a result of this pressure, and in so doing it may have been more willing to overlook the operational flaws apparent during the approval process.

While Excel is really not appropriate to any activity beyond vanilla accounting, the failure here (as with The Great Recession) was the result of Rand-ian self-interest. The real Adam Smith posited an economy in which each actor is not only autonomous, but also bereft of market power to impose collateral damage. Finance is all about collateral damage; that's how they earn it.

Here, it gets really interesting:

On January 26, the Model Review Group discovered that, for purposes of a pricing step used in the VaR calculation, CIO was using something called the "West End" analytic suite rather than Numerix, an approved vendor model that the Model Review Group had thought was being used. The Model Review Group had never reviewed or approved West End, which (like Numerix) had been developed by the modeler.

Turns out that Numerix is built on Excel (at least in part): "All Numerix functions are available via Microsoft ® Excel® as an add-in".

If you've ever inherited an analysis application executed in 1-2-3/Excel macros, you have my condolences been there done that.

In sum: while the use of Excel, even if it's also used by a major analytics vendor, is problematic; the cause, as with The Great Recession, was corruption and not only the Suits in this case. What's especially galling is that these quants are often some sort of math-y Ph.D., yet use Excel in haphazard ways. The poor will always be with us.

The Eyes Have It

Boy Howdy! Web adverts are just like TV ads:

One of the most important things that we need to do in the market is help educate people that the click is not really the most important metric for us. In fact now that we've been able to work with companies to look at in-store sales data, we find that of the people who saw a Facebook ad and then purchased the product in the store, 99% of them never clicked on an ad, so re-educating the market what the metrics are that are right for us.

That's from Sheryl Sandberg, COO of Facebook (you can get the transcript from Seeking Alpha). One might conclude that this was a bit of magician's misdirection (don't look at the hat, look at the pretty girl), or it might just be true. The efficacy of adverts has been studied pretty much forever. Ph.D. dissertations have been written.

Yahoo! has this to say:

For example, in their important paper on adfx, Abraham et. al (1990) open with the line, "Until recently, believing in the effectiveness of advertising and promotion was largely a matter of faith" -- a first sentence that might otherwise seem a bit peculiar given that before they penned it, approximately 4 trillion dollars had been spent on advertising.

If it's true that clicks really don't measure effectiveness, then what? Could the whole advert driven web go poof?

As postulated here more than once: an economy based on adverts is inherently unstable. While adverts are a useful adjunct to production, when adverts become what the economy produces? Real life becomes a version of Second Life (you may remember it?). "I'm sorry Dave. I'm afraid I can't do that."

The Light Dawns

Whilst looking for a web version of Date's puzzling "dropping ACID" muse (I don't feel like typing it all in), I came across something at least as interesting. As the subtitle says, "all things xml". That was penned before NoSql had reared its ugly head, and xml "document stores" were then the rage.

On occasion, I've railed against the NoSql saga. If data doesn't really matter, then any file datastore will do. If it does matter, then either you use an existing TPM (CICS is still around), RDBMS (TPM built in), or roll your own. Since TPMs have been around since before CICS, and there is decades of experience figuring out how to do them, a few kiddie koders with php ain't likely to get there any time soon.

But recently there's been a dawning recognition among NoSQL practitioners and those working in Big Data that the fast-iterating data they process needs to be demonstrably reliable, too. The result has been NoSQL databases adding more relational functionality to their software.

"D'oh!" quoth Homer.

It gets better:

"We think one of the reasons there are so many NoSQL databases on the market is because unless your application perfectly matches their data model, it becomes difficult to build data abstractions."

Well, of course. The NoSql approach is tied to file defs and implementation language. This is the Achille's Heel of API-centric coding. The RDBMS/SQL is client language agnostic, and so long as one sticks to reasonably standard SQL, engine agnostic as well. For those eager to port across engines, there's the likes of SwisSQL.

"A distributed data store without concurrency control is a toy and makes building things on it a lot harder."

What is left unsaid, even by Rosenthal who is, by his own admission, just a coder, is that industrial strength RDBMS (DB2, of course) have had distributed transactions for decades. Gray and Reuter devote much of chapter 16 to distributed CICS, for crying out loud. Nothing as much fun as inventing the wheel. Again. And worser (isn't that a city in Poland?).

One Step Closer to Nirvana

For years, many years, there's been the quandary: if one is updating a row, and not updating key columns, should others be able to concurrently update other non-key columns, even the one I just updated? So far as I know, Postgres is the first database to tiptoe in that direction. This isn't cell locking, though.

With this level of locking, UPDATE queries that do not involve columns of the tuple key can perform concurrently.

The argument will be "but chaos will ensue!" I don't buy it. With conventional row level locking, my updates will get overwritten by your updates, just with lowered concurrency. And, by definition, non-key attributes are fully mutable and irrelevant to identity or process; that's why they're non-key. High normal form schemas fit right in with this locking regime: tables are narrower, with fewer non-key attributes per table. The flatfile folks oft times argue that same-named attributes have to have separate ranges, depending on table; foreign keys and normalization are evil. May they fry in hell.

Data Fiddles, While Rome Burns

The Case-Shiller number is just out, at 5.5% up. More than expected, on a month-to-month basis. One might wonder how this might be happening, given that median income continues to fall. Just saying. Data tends to lag back as much as a year (Census data on median income, for example). Either the recovery is way better than reported data says, or the Banksters are fiddling mortgages again.

Rotten to the Core

(This was originally posted on Dr. Keynes, since it's mostly macro oriented. But, the mainstream punditry has seen the light, and dealing with the substance in similar terms. So, I guess it's appropriate subject matter for Dr. Codd.)

The entirety of the pundit class, both mainstream and fringe, have been weighing in on the meaning of Apple's disappointing quarterly yesterday. This is old news, from the perspective of this endeavor: I've been asserting for some time that Jobs/Apple has been pursuing a suicidal path. Where once there was a somewhat iconoclastic computer company, is now a garden variety up-scale appliance maker. The ElectroLux or Jenn-Air of the 21st century. Just because there's a cpu running the widget, doesn't make it a computer. Your Ford has as much compute power.

Up-scale anything is a low volume, low growth proposition; always has been and always will be. Ferrari sells about 6,500 units a year. And no growth. Apple's problem is that the middle class is being hollowed out. Obama's call for progressivism is also the clarion call to save Apple. Not that Jobs did or Cook does, understand this. Certainly not the proto-Jobs day trading plungers who don't have a clue. Apple has transformed itself by invading existing market sectors with novel designs, nothing more. None of the devices that have worked (remember Newton?) were original to Apple.

Rumors continue that there is an iTV in the offing. If so, and carried out as a novel design exercise, could it work? The iPhone really sped Apple away from being a computer maker, to being an appliance maker. iTV is pure appliance, but can physical design do for iTV over other set-top boxes what the iPhone did to RIM? Not likely. Physical design provides an avenue for advantage for objects which live either in the eye or in the hand; they're pretty to look at, or they're pretty to manipulate. Set-top boxes have no appreciable use case in either vector (love mixing metaphors, how about you?). How many of you bought your stereo receiver (assuming anyone still has a stereo) because of its looks? Thought so. The long time exception in that sphere is B&O, which markets its facade not its performance; a tiny niche company and the products aren't very good. That's Apple's future.

The Apple story of significant growth, through market invasion, requires that there be markets which already exist and which can be wrapped in a novel physical design. They've clearly worn out the iPhone meme. The last few versions haven't been meaningfully differentiated either from previous versions or competitive phones. The novelty is wearing thin, if not gone completely.

Apple may have slipped into Microsoft land, playing out the game as a worthy journeyman, collecting a fat salary but not playing any meaningful minutes. The days of Kobe are in the mirror. But, and here's the macro problem, could we have reached a technological plateau? Could it be that the innovators don't have a clue what more to do with cpus? With a concentrating upper class, a shrinking middle class, and an exploding lower class, what's a high-end digitizer to do? With smaller feature sizes still occurring (although that may soon end, too), thus greater compute power available, into what new device, smartly designed for the well heeled, should all this power be packaged? Consider that the basic inventions at play here all came to be in the first half (and for some quite early) of the 20th century. The smartphone makers and the carriers are cut from the same cloth as Wintel before them: in both cases the relationship is symbiosis, with each side supplying the needed demand for the other. In the case of Wintel, MS let "the hardware fix performance" while Intel needed Windoze bloat to justify the tick or tock (although tick-tock jargon happened later). Now, carriers need bloated data streams to claim higher fees, while smartphone vendors need bigger data pipes in order to support non-telephone functions.

The entire industry has to figure that out. The "emerging markets" are valuable only so long as the First World countries control the exchange rate regime. But, there's a problem with that, too. In the past, even the recent past, the regime was essentially mercantilist: the First World kept third world currencies undervalued, so that raw materials (and more recently, finished goods) could be imported cheaply. The third world, to First World capitalists, was never intended to be a consumer market. Now, having siphoned off a considerable amount of purchasing power from the First World middle class (by shipping employment to those cheaper currencies), looking to a third world middle class presents a problem: either third world wages/currencies (for First World menial tasks) have to rise enough to replace what's been lost from the First World, or the wheel stops spinning. It can't be both ways: cheap wages/currencies in the third world and middle class in the third world.

Yes, there are reports that Apple does do a decent iPhone business in China. But the value of the renminbi exists at the whim of Beijing.

Here is what largely happened in the West post WWII: unions and corporations conspired to create a blue-collar (low skill) middle class, which had sufficient incomes to absorb the burgeoning output of industrialization. This was a symbiotic relationship; capital needed consumers, but the rest of the world was either wild (most of South America and Africa) or devastated from the war, so domestic consumption of output required domestic incomes to buy. Having Bretton-Woods control of exchange rates, and thus raw materials from the third world (petroleum in particular), played a large part in that. Cheap energy makes for a higher use of energy, what is commonly called "standard of living". This was a period of leveling of income distribution, which powered economic growth. It ended when the Arabs got sick and tired of Israel, and took control of petroleum in retaliation. We still don't know how that will end, although there have been reports that horizontal drilling will yield mammoth amounts of, to date, unrecoverable deposits in North America. We'll see.

Another Ton of Bricks

About a year ago, "A Ton of Bricks" told the story of Amazon's conversion, as yet still not discussed in the channels I follow, from e-tailer to brick-and-mortar retailer. Well, the conversion continues apace. Amazon's P/E ratio, as I type, is 3,182. They make a few pennies on the share. That Amazon is able to convince so many that it has found the New Way is astounding. The cost of jet transport, per pound, is so much higher than rail, it's clear now that Amazon is finally admitting what OR folks have known for decades: JIT production is fine when one is dealing with very high value parts which are small in volume, light in weight, and a small part of the BOM. In such an instance, stockpiling can add up to more in carrying cost than transport. For garden variety retail, warehousing wins. Always has and always will; so long as fuel isn't free.

Amazon isn't such a manufacturing business. It's a distributor, and its main cost is transport. The main cost it can control, at least. Unless Jeff decides to go all Walton on its suppliers. But, since Amazon has set out to be the purveyor of everything, that ain't gonna happen.