Treat, no Trick

Talk about a Treat!!! There is a "preview" build of R Studio (note the switch to .com) and now supports vi(m)!!! Adults use either emacs (the demented side of the family) or vi(m) (the sane and smart side). Woo Hoo!!

I Vant to Biite Your Nheck

R version 2.15.2 is out, aptly name "Trick or Treat". Just in time for Sandy. Note that you need to build from source, and run configure with the --enable-R-shlib switch in order to use R with Postgres through PL/R. Works like a charm.

Merry Quant

The return of the mini (and micro-mini) skirt does my heart good. Once again, we wait for the heaven sent Sharon Stone moment. Ah, not so often. I do wonder: do our Merry Quants know they're the descendants of such a creative mind? Or, are they just in it for the money? Just the money I guess. But that begs the question: why did financialization overwhelm our economy, and other countries'? In the 1960s, Uruguay started down that road, and ended up with guerrilla warfare and military coup. While I've long since lost my thesis, this paper provides a more recent appraisal. It's not any prettier.

So, why did/do the quants hold sway, and play only in fiduciary instruments? Why financialize (conversely, de-industrialize) an economy? Is there a Maslow's pyramid of economic development? No, I think not. What's happened since Reagan is a shift of power, aided and abetted by Washington's Right Wing tilt, to the easy, fast buck. Dr. Smith used the phrase of "the quick buck versus the slow deuce". While something of a Right Wingnut (he considered himself an entrepreneur stuck in a small New England college; his signature idea was a concrete Sunfish. I'm not joking.), he had no use for the quick buck artists. Yet, we've allowed them to take over the economy. Why?

The answer, by my lights, is obvious: fiduciary capital is easier; there's no messy physics (in all of its manifestations) to muck up the plan. That which exists only as bytes in computers can be controlled (or so the thinking goes) by programs in computers. The Merry Quants, and their handlers, ignore the truth; at some point a fiduciary attaches to a real object. In the case of The Great Recession, that object was a McMansion in Boca. It was attached to a subprime, fixed reset, ARM sold to a minimum wage janitor. And so the dominoes fell.

Last Sunday's NY Times "Sunday Business" has a long piece describing the impetus for the flight to fiduciaries. When one makes direct investment in physical capital, one is tied to the market for the widgets involved. To earn that return, it's not merely a matter of scarfing some of the money flow between saver and borrower, as the vampire squid do their voodoo so well. When you build it, you're betting they will come. And buy your widgets. Which widgets you've made for cheaper due to your investment. And, if you've been really clever, they buy more widgets, to boot.

This endeavor has oft quoted Your Good Mother: "what would the world be like if everybody behaved like you?", in the context of opprobrium for bad behaviour. Turns out, if everybody tries to make and sell the same widget at the same time, nobody makes any return on the invested capital. Thus Warren Buffett's observation about moats.

In the shale gas case, a gold rush meme doesn't work out. Gold is craved for no good reason, so having it, when and where others not so much, is always profitable (aside: it was the merchants who served the miners that made out selling eggs and the like for obscene prices), while natural gas is only valuable in use. Explode the supply, and well, you get it.

But while the gas rush has benefited most Americans, it's been a money loser so far for many of the gas exploration companies and their tens of thousands of investors.

Not that the Banksters didn't try for a nice slice of the pie.

Like the recent credit bubble, the boom and bust in gas were driven in large part by tens of billions of dollars in creative financing engineered by investment banks like Goldman Sachs, Barclays and Jefferies & Company.

After the financial crisis, the natural gas rush was one of the few major profit centers for Wall Street deal makers, who found willing takers among energy companies and foreign financial investors.

Not all were convinced.

"He is like the bartender serving drinks for people who can't handle it," said Fadel Gheit, a managing director at Oppenheimer & Company, about Mr. Eads [one of the money men profiled]. "And the whole gas industry has gotten a rude awakening, a hangover, with gas prices plummeting. The investment bankers were happy to help with a smile and get their cut."

To the predictable conclusion.

In hindsight, it should have been clear to everyone that a bust was likely to occur, with so many new wells being drilled and so much money financing them.

But, there is a silver lining; well for those of us with an eye-for-an-eye streak.

Some local landowners, having spent their initial lease bonuses, are now deeply in debt. Local restaurants and other businesses are suffering steep losses now that so many drillers have left town.

From the point of view of our Merry Quants, they flew just a bit too close to the sun. Best to stick with fiduciaries. How about re-insurance in Bermuda?

The Next Time Bomb

Our Regular Reader knows my fascination with Bermuda; there've been more than a few missives about the country. Most reflect my belief that Bermuda represents a real life, real time social petri dish. As Bermuda goes, so will the USofA. The reason is straightforward: Bermuda abandoned its natural strengths in tourism, to becoming a tax evasion haven (Bermudians dislike that characterization, but so what?). The government rewrote its laws (much as North Dakota did for the credit card industry) to support tax evasion. The result of this was to import a significant number of foreigners, at high salaries. The result of this increased skewing of income distribution has led to two inflation forces: demand pull, due to the high salaried imports bidding up housing, in particular; and cost push, due to the increased demand for imported goods caused by those same high salaried minority. The predictable reaction: native Bermudians demand wage increases to "keep up with prices", as such folks always do. They'd done nothing to produce the higher prices, and didn't buy the argument that they deserve to have their incomes de facto reduced. Income is always relative, never absolute.

Last month, the NY Times ran a piece on Bermuda's next foray into manipulating the financial services sector of the global economy. Further revealing the hollowness of Bermuda protestations of innocence.

We have another mine. We need a new canary. Herewith, some eye opening quotes.

The hedge fund industry has been rushing headlong to open Bermuda-based reinsurers. ... And while the hedge funds are likely to profit, the question is: Who's watching to make sure this doesn't lead to another financial calamity?

And, of course, few if any strings attached:

Yet while they are partly hedge funds, these new companies are regulated as reinsurers. And Bermuda requires only minimal capital requirements and disclosure of financial positions, and it does not strictly regulate how these companies invest their money.

As the incentives which led to The Great Recession, hedgies are looking for better than risk-free return (Treasuries, for example), damn the torpedoes.

Let's close with one more quote (you really should read the whole piece; especially if you're the Stephen King sort of weeny):

In other words, the reinsurance market is starting to look like many of the markets before the financial crisis -- lightly regulated and interconnected in ways that policy makers can't see, with banks potentially left with the wreckage. The industry may be right that reinsurance is different and has its own checks and balances, but we've also heard that before. It behooves United States regulators to make sure.

Given that Wall Street runs DC, not the other way around, do you think that likely? Will the Merry Quants blow up the world again? How many of you, Dear Readers, have even heard of re-insurance? How many would know to even go looking? QED

A Horse's Ass?

It would appear the Gallup is trotting down the same bridle path as The Literary Digest. While likely not as well known as the Chicago Tribune "Dewey Defeats Truman" headline, it is actually more significant today.

In retrospect, the polling techniques employed by the magazine were to blame. Although it had polled 10 million individuals (only about 2.4 million of these individuals responded, an astronomical sum for any survey), it had surveyed firstly its own readers, a group with disposable incomes well above the national average of the time (shown in part by their ability still to afford a magazine subscription during the depths of the Great Depression). The magazine also used two other readily available lists: that of registered automobile owners and that of telephone users. While such lists might come close to providing a statistically accurate cross-section of Americans today, this assumption was manifestly untrue in the 1930s. Both groups had incomes well above the national average of the day, which resulted in lists of voters far more likely to support Republicans than a truly typical voter of the time.

Today, Gallup is doing something similar, by relying on landlines. They say they've decreased the sampling using such, but when one poll is the outlier among many, it's wrong.

Come Into My Parlor, Said The Spider

A minor, but recurring, theme of this endeavor is the fragility of advert driven business, web division. As most know, virtually everything on the web is either advert based, or shopping (well, by most accounts porn is the majority by far). Deriving one's revenue from folks who may or may not want to buy something else has been debated for decades. I first got a glimpse when I read "The Hidden Persuaders" by Vance Packard, either high school or junior high (now often called Middle School). It was written in the late 50s, and I read it sometime later.

Over my Panera coffee this morning, there was this article about click based adverts in my dead trees version. Made some notes, thought about posting about it, but... nah. Then, kaboom, Google pooped the bed!

What to make of all this? First and foremost: the advert based interTubes was always fragile. The 2000 dotbomb wasn't the end of things, although I do miss the Pets.com sock puppet. (Aside: I read some wag who opined that CEOs are so overpaid, because these MegaCorps could be run by a sock puppet. That was before 2008, of course.) As Mr. Telford found out, there always other, and oft times cheaper, ways to get people to buy your widgets. As I've written long before, Google is always one "innovation" away from irrelevance: a more effective (at least to the advert buyers) advert platform will do to Google what Google did to newspapers. Of note, Mr. Udell (another small businessman):

"The cost to get someone just to visit your Web site has, in some cases, become prohibitive. Something that cost $3 might be a no-brainer, but at $20 it becomes absurd. It's basic math, and if it doesn't add up, we won't do it." He said he planned to redirect some of his advertising dollars to print, television and radio. [my emphasis]

Google is trying to do, successfully, what Sun didn't do with java: leverage one product to sell something else. Sun expected java to increase its shift of servers. Google is trying much the same with Android; which is not java. We'll see. Google may have overpriced its advert platform. On the other hand, Yelp is reported to be tracking fake positive reviews, and other business espionage.

One wonders: will Google tweak its "algorithm" yet again to degrade organic matches? Do they really have any choice? For myself, I always use AdBusters, which I found when I was on dialup as a way to reduce bandwidth hogging. AdBusters causes some controversy, but I don't see the issue. If one has no intention of following ads, as I don't, why object to AdBusters? It isn't as though I'll start madly clicking away, while page loading slows to a crawl!

Make a widget and sell it.

Sherlock Holmes Plays One

Yet another dive into the deep end of the pool. Most folks interested in enterprise level SSD know about Violin, but because it's been private (unlike Texas Memory, it's newly minted, relatively speaking) the moolah folks and general public haven't heard of them much.

Get out your checkbooks. Hopefully, not another OCZ fiasco! No, I don't think so.

Lefty Loosy, Righty Tighty [picture fixed]

It's not often that I send regular reader off to some other post, but this is the exception. I'm going to explore the data referenced, just as I hope you will. There's more to quant than predicting the next 10 bagger.

For what it's worth is.R() is new to R-bloggers and already my preferred reference. You could do worse than spend time there. But don't leave Here, of course.

Just a note for Ubuntu users: you'll need to symlink gfortran-X.X to gfortran, if Hmisc isn't already installed (it requires fortran, of course and Hmisc will error out complaining it can't find the compiler). The build scripts I've run across don't look for these Ubuntu's (Debians?) versioned compilers. Also, and I haven't worked out the fix, on my 10.04/2.15.0 machine, the party legend doesn't fill, and the two legends are flipped. We'll see. Since the version in the post doesn't expand, it's really hard to read.

We'll Need a Bigger Boat

Why did/do flunked out math and physics PhD candidates go into finance/business admin/economics? The quants. Follow the money. They started in earnest in the 1970s, and became the Mongol hoards (or a school of squid?) which cratered the global economy a couple of years ago (we are past that, right?). Now that the pillaging is done with, Wall Street has turned off the money spigot, yes? No. See here. Not all that newsy, in that the previous year wasn't much different.

Between 2009 and 2011, compensation in the securities industry grew at an average annual rate of 8.7 percent, outpacing 5.3 percent for the rest of the private sector.

Note that the main data refers to 2011; we don't yet know what the numbers will be for 2012, but...

Some 48 percent of 911 Wall Street employees surveyed by eFinancialCareers.com said they felt their bonuses this year would higher than in 2011.

OK, so they've done so much better by the world's economy, they need yet more moolah. But here's the real problem:

Financial jobs accounted for nearly a quarter of all private sector wages paid in [New York City] last year, even though they accounted for just a fraction, 5.3 percent, of the city's private sector jobs.

One of the real conundra of the financial services sector is the assumption that it's driven by computers and quants and superior smarts. Put another way, in the industrial sector, wages have been falling in deference to capital. Earlier posts have discussed the fall in labor as input to production. In the finance world, not so much. With all the talk of double secret probationed HFT computers, it's the Boys in the Boiler Room who get most of the moolah. Moolah that, could, be used to buy plant and machinery for physical production. All that money just to partner savers with borrowers? My, my.

Nearly half of all revenue on Wall Street is earmarked for compensation; in 2009, Morgan Stanley, which was hit harder during the crisis than most of its rivals, found itself paying out a record 62 percent of its net revenue in compensation and benefits. That number has since come down.

I'm Melting!

Well, it really did happen, OCZ has crashed and burned. Time to muse on the effect, if any, on the thesis of this endeavor, that SSD on multi-processor/core/thread machines will take over the world. And that high NF databases will be the data store of choice.

Hmm. OCZ was never, despite its assertions, a player in the Enterprise Space, so there shouldn't be any direct effect. STEC slumped this morning pre-market when OCZ did its face plant, but has recovered. STEC does have its own CEO problem, of course, but does have a track record of producing Enterprise Space SSD that OEMs and end buyers really want. Fusion-io keeps on truckin'. The niche/private Enterprise Space continues on. The Enterprise Flash Appliance Space looks to be going well, at least according to Forbes reportage.

The loss, if it comes to that, of OCZ will most likely be felt in the Postgres/MySql world of SMB/web startups. These entities are largely populated by pimply faced boys who've never heard of Dr. Codd, or those that have, dismiss him.

In the months leading up to Peterson's "leaving" (and it's now reported that he's on the dole of OCZ for rather a while), OCZ touted its imminent release of "the first TLC" SSD. Didn't happen. Samsung's 840 (not the 840 Pro) is about to deliver, and has been tested, a bit, by AnandTech. The consumer SSD market is, it has always seemed to me, coelescing with the Big Boys, those that can survive and thrive in a race to the bottom. But then, I did predict that a while back. The boutique houses, that only assemble Other People's Parts, will either have to provide unique capabilities (STEC has it's own controller, so it doesn't count, either) or die. Darwin was right.

[Yes, if you read this earlier, the title is changed. I'd allocated that to a piece not yet completed. Too busy.]

Cheap at Twice The Price

High normal form relational databases provide the minimum byte load, and guaranteed integrity. Why have these qualities been so clearly ignored over the years? One excuse has been that normal form is too slow on conventional hard drives and uni-processors. Now that we have high quality SSD and multi-processor/core/thread machines, the technical complaints should disappear.

But my experience says that there is still an uphill slog in view. Why? Coders will remain obstructionist, since they're in self-preservation mode. Another piece today provides yet more examination of how this happens. Were humans rational, this sort of behavior wouldn't be tolerated.

It's an unfortunate reality that efficiency often goes unrewarded in the workplace. I had that feeling a lot when I was a partner in a Washington law firm. Because of my expertise, I could often answer a client's questions quickly, saving both of us time. But because my firm billed by the hour, as most law firms do, my efficiency worked against me.

Sound familiar? An unexceptional relationalist can build rings around a hoard of coders, but doing so threatens not only the hoard, but the (bureaucratic) bosses of the hoard.

...a measurement system based on hours makes no sense for knowledge workers. Their contribution should be measured by the value they create through applying their ideas and skills.

But, still, the coders circle the wagons, and insist that the olde ways are better. The bureaucrats who, ostensibly, manage them are happier with the bloat: the bigger the budget the greater the importance. Progress is wonderful.

The solution, it seems to me, has always been clear: engage in fixed price contracts, whether internal or external.

When Good News Happens to Bad People

Back in February, there was "Lies, Damn Lies, and the BLS", which you can find in any of the various incarnations of this endeavor. At the time, the numbers came out better than the pundits predicted. It happened again this week (for September), but since it is the second but last run of the numbers before the election (Election Day is Tuesday the 6th, while the numbers day is Friday the 2nd) with enough time to get all piqued up.

And did the Right Wingnuts get all piqued!!! To repeat: the data come from sample surveys, two not a single integrated survey. And if you read the fine print, as I suggested back in February, then you'll see that the estimates have plenty of room to waffle. I don't recall any articles in February that went into the details. Well, this time there is. Worth reading up. The on-line version is graphier, and therefore more useful.

As to the GOP stroking out: the BLS folks who make the decisions are Supergrades (here for explanation), who, in all likelihood, got to these positions of authority under BushII. If there's any conspiracy, it's among geeks who live in Fairfax County with a stay at home wifey and three kids in Christian School. I spent a decade working with those sorts (not at BLS, for the record).

Let's Have a Party!!

It qualifies as something of an open secret that lots of financial service (even actuarial) analysis is done in Excel. The revelation appears as snide comments in texts with some regularity.

ggplot2 has become (to my mind, anyway) the de facto graphing package in R. Recently, Hadley released an updated version, with themes getting the most attention. Today, David Smith (of Revolution Computing, and an Aussie, if memory serves) posted some comments on themes (referencing theme work done by Jeffrey Arnold), including this beaut:

or if you're an R user on the down-low, with that "classic ugly look and feel" of Microsoft Excel. (A better idea would be to just come out!)

More often than not, Americans use the phrase (in TV dramas quite noticeably) way, way out of context. One might wonder what they call Excel in Gay Paree?

Many Happy Returns

Today brings not one, but two, cautionary tales. The reason for doing quants, either for anonymous blogging or mucho dinero, is to deal with issues which can only be answered with data, not policy. I'm on record that policy trumps data every time, but with the proviso that the policy can be enforced *despite* the eventual collapse. Greenspan's crashing of interest rates was a policy which motivated The Great Recession; he almost fully admitted it after the fact. There was clear data that the collapse was in the making, but ignored by both policy makers (who wants to admit error?) and participants (who wants to be first to miss the boat?). Capital's need for real return, and the consequent need for monopoly, appears in disparate stories today.

To recap. The justification for real physical investment is to make, and sell, either more product at constant price or current/less product at lower cost. This productivity delta is the real return. The Great Recession(s) come about when fiduciary capital is placed in other fiduciary instruments, rather than physical investment. Returns on real estate, whether residential or commercial, can only come out of the income streams of the underlying entities, households or businesses. Residential housing provides no financial returns, in use; paying the vig either comes out of rising incomes (there weren't any during Bush II) or consumption shifts (the volume of moolah needed couldn't be supported by that, although some apologists asserted so). In almost all commercial cases, the same is true. One might argue that a Park Avenue address will attract more business than one in Alphabet City (do they still call it that? and is it still a slum?), in greater proportion to the rent differential; but I'll consign that to outlier status.

So, real return to real capital requires making more and better stuff. Banksters tend to ignore that. AnandTech has another Haswell piece posted today.

If all mainstream client computing moves to smartphones, and Intel doesn't take a dominant portion of the smartphone market, it will be left in the difficult position of having to support fabs that no longer run at the same capacity levels they once did. Without the volume it would become difficult to continue to support the fab business. And without the mainstream volume driving the fabs it would be difficult to continue to support the enterprise business.

There's more background in the text, but it amounts to this: Intel needs to keep its fabs running full blast to get the return on the cost of the fabs. In order to do that, it needs to produce chips which move like hotcakes. You sorta have to get it right.

On the other side of the world, we get the Chinese solar problem. The title: "Strategy of Solar Dominance Now Poses a Threat to China" in my dead trees copy, the title on-line is different.

But now China's strategy is in disarray. Though worldwide demand for solar panels and wind turbines has grown rapidly over the last five years, China's manufacturing capacity has soared even faster, creating enormous oversupply and a ferocious price war.

Trying to generate that real return? You betcha. Does it work, by default? Not hardly.

In the solar panel sector, "If one-third of them survive, that's good, and two-thirds of them die, but we don't know how that happens," said Li Junfeng, a longtime director general for energy and climate policy at the National Development and Reform Commission, the country's top economic planning agency.

We have to do something about that Ruinous Competition!!! Wind turbines? Same thing.

The Chinese government also wants to see the country's more than 20 wind turbine manufacturers, many of which are losing money, consolidate to five or six. "Wind does not need so many manufacturers," said Mr. Li, who in addition to drafting renewable energy policies is the president of the Chinese Renewable Energy Industries Association.

Capitalists continually assume that they deserve outsize returns, but every time they try it, chaos results. Will they never learn?

The modest cutbacks in production barely put a dent in China's overcapacity problem. GTM Research, a renewable energy consulting firm in Boston, estimates that Chinese companies have the ability to manufacture 50 gigawatts of solar panels this year, while the Chinese domestic market is on track to absorb only 4 to 5 gigawatts. Exports will take another 18 or 19 gigawatts.

The enormously expensive equipment in solar panel factories needs to be run around the clock, seven days a week, to cover costs.

Both Intel and the Chinese alt energy sectors are the poster children for fiduciary "investing": while fiduciary "investing" is faux, the return is largely controlled, in the short run (which is all they care about), by policy. The residential home builders made out like bandits, literally, while mortgage companies, banksters, and MacMansion buyers got the shaft. As always, one needs to follow the money. The Chinese alt energy companies (and the government) can't, or won't, find buyers for its shiny new toys. One might argue, and the government surely did, that investing in some fiduciary capital in product producing entities is better than investing only in infrastructure. Infrastructure, as MacMansions, is difficult in the same way: how does one impute (much less collect) real returns? For infrastructure, the return is explicitly societal. PhD candidates have been writing dissertations on the problem for decades. Eisenhower's "National Defense Highway System" was the earliest in my lifetime. The official name became "Dwight D. Eisenhower National System of Interstate and Defense Highways" under Bush I. What's it worth? Well, Ike wanted it because he saw the difficulty (to the Allies) caused by Germany's Autobahn; it was intended to be a network to move men and materiel during the coming wars. Just as DARPAnet was all about the military and turned into a commercial enterprise we call The Web. Who gets the return?

The Best Laid Plans

For those interested in how real world databases and operating systems get along (or not) in the post single-cpu, sorta parallel regime of today, there is a piece on lwn.net which is instructive. I had originally inserted the link, but it's not that simple. The link I used was supplied by a post on the Postgres/Performance email group. For better or worse, lwn.net is by subscription, but it appears that subscribers are free to link to content. So, go subscribe to the Performance email group (you should anyway, here), and you'll get the link in today's posts. It's got "20% performance drop on PostgreSQL 9.2 from kernel 3.5.3 to 3.6-rc5 on AMD chipsets" in/as title.

The discussion in the post revolves (that's a pun) around the inherent conflict in scheduling between an (multi-client) operating system and a (multi-client) database engine. While flagged by the PG folks, any engine which attempts its own scheduling would have gotten caught, to some degree, by the patch. Who said parallel was easy?

Here's a quote which describes the issue:

So what is different about PostgreSQL that caused it to slow down in response to this change? It seems to come down to the design of the PostgreSQL server and the fact that it does a certain amount of its own scheduling with user-space spinlocks. Carrying its own spinlock implementation does evidently yield performance benefits for the PostgreSQL project, but it also makes the system more susceptible to problems resulting from scheduler changes in the underlying system. In this case, restricting the set of CPUs on which a newly-woken process can run increases the chance that it will end up preempting another PostgreSQL process. If the new process needs a lock held by the preempted process, it will end up waiting until the preempted processes manages to run again, slowing things down. Possibly even worse is that preempting the PostgreSQL dispatcher process -- also more likely with Mike's patch -- can slow the flow of tasks to all PostgreSQL worker processes; that, too, will hurt performance.

Spinlocks have been around for at least two dogs' ages. And the description reminds me of IBM's AS/400 (now called something else) solution, and M$'s abortive (so far) effort to replicate it, winFS; in which the database engine and the operating system are merged. Those with even longer memories could be reminded of PICK.