Reverse Thrusters, Mr. Scott!

A comment on one of these missives, posted in an alternative universe, led me to comment serially. This is the part of the comment which motivated this missive:

The only way to increase the demand for stick built single family housing is to drop the price a lot. Not to mention that boomers will be dieing off soon, and all those reverse mortgages will stuff yet more inventory into the mix.

Which led me to go a lookin' for data on reverse mortgages by year and value. Haven't found same as yet, but I did find a bit scary reporting from last week.

While forward mortgages show a surplus in FHA's reestimate of reserves of $4.3 billion, the reverse mortgage portion shows a loss of $5.2 billion, resulting in the $943 million shortfall, according to FHA.

An actuarial review of the MMI fund in late 2012 indicated the reverse mortgage fund had a negative $2.8 billion economic value; part of more than $16 billion in negative economic value of the fund overall.

There have been a stream of stories about surviving spouses being tossed out of homes when the mortgage holder spouse kicks the bucket; the rules about who can be on the title/deed get convoluted. And, according to some reports, such as this, the Banksters aren't above fibbing a bit.

I wonder: is there yet another Viagra at the home we need to worry about?

Structural Failure

Remember the Tacoma Narrows bridge? It was a victim of unmindful structural deficiencies. Imagine my surprise (not really) to see this story describing the addition of the C struct to java. Just as coders have assassinated Dr. Codd's relational model, with their smart code, stupid data approach, now the coders want to kill off even the appearance of OO paradigm in java.

I suppose it was inevitable. At least in the web app world, java objectification has been executed as "data objects" (just a struct with a bit more typing) and "action objects" (just a function with a bit more typing) for a couple of decades. This is why I've been in the habit of referring to java as just COBOL without the SCREAMING CAPITALS all this time. Turning java into C de jure will just complete the de facto regression.

Charlie Chaplin

Channel surfing got me to "Chaplin", the biographical movie of Charlie, with Robert Downey, Jr. I didn't stay very long, but seeing some bits led me to WikiPedia. What was his life, from an historical perspective?

A Dog's Life, released April 1918, was the first film under the new contract. Chaplin paid yet more concern to story construction, and began treating the Tramp as "a sort of Pierrot [sad clown]."

Today, I am a pierrot. AnandTech has a review up of the Crucial/Micron M500. Nearly 1T for $599! But how they got there, which is detailed in the review, dashes any hope for SSD being a performance replacement for HDD. In the comments, a question was asked, and I had the temerity to answer. Here it is.

-- My point is, what's at stake here is who's the next Seagate? The next Western Digital? Of SSDs.

Getting harder to say. The three well known public companies doing SSD (mostly) as such, STEC, OCZ, Fusion-io, have been missing all targets for a least a couple of quarters. Violin may or may not IPO in the next few months.

The reasonable answer is that there won't be a Seagate or WDC for SSD. It's well understood how to take commodity HDD to Enterprise Drive, using tighter QA and some incrementally better parts at modest cost. With SSD, as this review shows, "progress" in feature shrink isn't improving any of the factors at lower cost. It is quite perverse. The NAND suppliers will come to dominate consumer SSD, with performance asymptotically approaching a bit better than current HDD, with a price premium. Look for TLC, with huge erase blocks, long latencies, slowing controllers (having to do all that much more work to get around the NAND).

Enterprise SSD will likely fade away, to be replaced by NAND arrays, along the line of the Sun/Oracle device, which has been around for a few years.

The SSD is dead, long live the flash array.

Finding Nemo

It must be vexing to be a quant in the financial services industries. On the one hand, you're in the Next Sexy Career: data science. On the other hand, Greenspan and Bernanke and yourselves have shoved a 20 inch long 4 inch wide dildo up your yahoo. That's got to hurt.

What happened? Well, the base cause is wealth concentration. When the few have the most, there's little left to drive real demand for goods and services. Absent real demand, there's no cause to invest in real capital. With a slackening of real capital, the trickle down to fiduciary capital is inevitable. This effect is more obvious in industrial economies, over the last century or thereabouts, than earlier. But, "let them eat cake" has been the crux of the matter for a lot longer.

Unearned income is both a social and economic porcupine. To the extent that the .1%ers demand 10% or so on their amassed moolah, the quants (who are, after all, the braceros in that sector) have to find another way when Treasuries dry up. Before Greenspan cratered interest rates, it wasn't impossible to pull off. Not so much of the moolah had to find a risker home (presaging?). Treasuries were pulling in the mid to high single digits, so only a small part of that accumulated moolah had to be allocated to "risky" assets. Preferably fiduciary assets, since physical assets is something neither the .1%ers nor quants understand all that well. Physics and engineering and real demand for goods are, taken together, mind numbingly difficult to conjure.

With the Fed cratering "risk free" unearned income, the quants looked to the, historically, nearly risk free instrument: the home mortgage. Whether there was (certain Right Wingnuts gainsay) a giant pool of money out there, or not, forcing the interest rate of record to very low values forced all those who depend on unearned income to look elsewhere. So, they did. (The giant pool argument being that the rate would have cratered no matter what Greenspan and Bernanke did, since the giant pool would, in aggregate, have been chasing a fixed field of investments. Supply exceeding demand, drives down the price, which is the interest rate.)

But there is a trickle down unintended (one hopes, at least) consequence: all other forms of reliance on unearned income were just as affected. Your life insurance and health insurance and defined benefit retirement plans (the few that still exist) all began looking for ways to opt out of cratered US fiduciary assets.

What had been the earnings? Here is a compilation.

There are a slew of tables, so I'll just highlight the numbers that speak to me (from table 2.11 $millions):
1970: mortgages - 74,375 stocks - 15,420
2010: mortgages - 366,988 stocks - 1,570,225

As is obvious, a tectonic shift. Put simply: your life insurance policy now depends, more than it ever did, on Prudential's rock solid quants picking the right stocks. And more to the point: your policy depends not on compound interest, but on fiduciary capital gains. You should feel your sphincter tighten rather a lot.

Table 12.2 is instructive. There's a vicious Prudential commercial (actually, a number of versions) on the TeeVee where some "professor" has folks put a dot on a wall, representing the "oldest" person known. These are actors, of course, and it's all made up. This table has life expectancy (from the industry Bible of data; you thought each quant figured this out on his/her own? not really!) at various ages starting in 1900. At birth, it goes from 49 to 78! We're living nearly twice as long! The sky is falling! We must kill off a hoard of old people! Not really.

Social Security was established, circa 1935 (first paid 1937). So the number that really matters is life expectancy, at 65 or 75, then and now (the closest year in the table is 1939).
1939: 65 - 12.8 75 - 7.6
2009: 65 - 18.8 75 - 11.7

So, yes we are living longer, but the sky isn't falling. More to the point, since SS is a *current account* system, having folks living until 65, which is where almost all of those 30 extra years are, means more folks pay in longer. Yes, the Boomers will be something of an extra burden for a decade or so, but after that, it's all gravy.

In sum, the issue comes back to macroeconomics, despite the fact that quants operate in a microeconomic venue (what's the slickest way to make my employer richer?). Both life and health insurance are inter-generational moolah flows; that fact can't be denied. The question of equity is both micro and macro. From a micro point of view: it's a dog eat dog world, and every man for himself. In such a world, insurance is funded solely by individuals' earnings. This is a divide-and-conquer method whose main beneficiary (pardon the pun) is the financial services industry; there are a whole lot more easily divided buyers than sellers. The principle difficulty, aside from outright equity, is that individuals are cornered by the market. If one retired, or wished to do so, around 2008, with only individual assets to support said retirement, well, good luck with that.

It's simply more equitable to remove the luck (good or bad) of the draw from these decisions. Those who are eligible for funding at some point in time ought not to be penalized for macro market conditions that are cratered. Similarly, for those who are elibible at the top of the rollercoaster ride.

Any increase in retirement funding has to come from real investment. As The Great Recession proved to anyone paying attention, fiduciary instruments are merely pass-through monies. Without underlying increases in real incomes (and they can only come from increases in real productivity), the instruments fail. And so they did.

Taking a macro point of view, one comes to a somewhat different conclusion about how to fund insurance programs. Firstly, the income streams are unearned from the point of view the funds; these are fiduciary instruments which have more or less tenuous connection to real world investment. Since both retirement and health exist at the societal level, then one needn't incur the overhead costs of administering a faux investment fund. To be specific: there have been, over the years, assertions that SS is/should be an investment system. It isn't and shouldn't. It isn't because FDR's wonks weren't stupid. And it shouldn't be because it makes no sense. If SS held private stocks and bonds, how would it regulate/police the stock markets? Recall that SS came into existence in the wake of The Great Depression, which was caused by unregulated markets.

If SS held private stocks and bonds, it would be obliged, as a shareholder, to defend Enron and Country Wide and all the other malefactors of the private markets. The current system of "investing" in Treasuries (broadly defined) is a sham, and that's OK. It seems to allay some of the idiots who insist that SS holds their "personal investment". It doesn't, and never did. Also, the notion that Average American in 2010 may/will receive more than s/he paid in, is nonsense. Average American pays for the SS of his/her parents. Would you stiff Mom and Dad, just so you could have an iPhone 7? (You would?) Yes, the Boomers are the pig in the python. And yes, Boomers didn't breed as eagerly as previous generations (well, those who aren't religious whackos, anyway). But we've seen in the last 40 years that productivity has widely outpaced employment. In other words, we don't need no stinkin' babies.

That last is the political meatball. Japan put itself in the mess back in the 1990s, and China, being so much larger, is seeing the effect faster and more pronounced. Society wide insurance demands a society wide (and inter-generational) funding mechanism. Otherwise, it's let them eat cake.

About That Gypsy...

Finally got some time to look at the Postgres version of Northwind, mentioned earlier. Disappointment, but not fatal. Turns out, of the conversions on offer, the PG one is least complete. The file contains only the schema, data, and PK defs. Hmm.

Not only that, but PG has a schizoid relation with letter casing. In the final analysis, unless all your identifiers are lower-case, you'll end having to quote (or escape quote) every bit of SQL you run. Ack. The parser lower-cases anything it sees that doesn't have quotes. So, the DDL can have not only mixed case, but spaces where quotes are used. The file is loaded with all of this. Ack. Now, if you run DML against the table, and get the mixed case correct, but don't quote, you get an object not found error, since the parser insists on lower-casing your text. Irritating. This is, undoubtedly, an artefact of the C/Unix/coders-view-of-the-world heritage of PG. Irritating.

So, out comes vim (SlickEdit version) to the rescue. That gets us the database, schema, data, and PKs.

Fortunately, the FK and view syntax (SQL Server, PG and the MySql versions) are similar enough that grabbing stanzas and editing isn't too irritating. Which leaves the procs. PG doesn't have functions, triggers, and procs as discrete entities, just functions used/labeled also as triggers and procs. Worse, ADS doesn't want to parse proc defs that psql has trouble with. Since procs aren't important, yet, to Northwind's use with R, I'm going to shelve that part of the exercise for now.

Finally, Blogger doesn't allow files for download, and I've not setup a github account; so anyone's who's interested, drop me an e-mail. I'll send along a zip of the three files: base, FKs, and views. The address is hidden in the profile info to the left. Deeply, deeply hidden. You may never find it.

They Died With Their Boots On

The title of this missive is also the title of an Errol Flynn western, 1941. That much I recalled, and the reason I wanted it for a title. What I didn't recall, but WikiPedia reminds me, is that the movie is a fictionalized account of George Armstrong Custer. Yes, that one. So, as a Dr. Codd title, it's a bit inapt. Or not, depending on your place in the RM/SQL/NoSQL spectrum.

When I was in school, I recall being told that the main difference between a job and a career was one's attachment to the work itself, rather than to the ability to pay some bills. A career lasts a lifetime, while a job is a necessity, much like sitting on the toilet in the morning. Make a bunch of money, so that one can spend as much time as possible doing something else.

The father of a high school friend was a general surgeon (the father, not the friend), who owing to the disruption of WWII and certain digressions in his youth, had been practicing for only about 20 years, even though when I met him he was a bit more than 60. This was before the attack of the vampire squid HMOs, so he was in private practice. As such, he depended, as did specialists then, on referral from general practioners to continue "cutting stomachs", as he oft times described his work. He said that a surgeon was retired by his colleagues: as they gave up their practices, the surgeon loses referrals; younger GPs referred to their age-appropriate peers, by and large. A surgeon didn't stop working because he wanted to, but because he was forced out by circumstance.

Some time later, whilst working in DC, I had the opportunity to take a couple of seminars with W. Edwards Deming. He was still well known within the OR/QA/stats world. Not so much now. He was in his 80s then. According to WikiPedia:

In 1993, Deming published his final book, The New Economics for Industry, Government, Education, which included the System of Profound Knowledge and the 14 Points for Management. It also contained educational concepts involving group-based teaching without grades, as well as management without individual merit or performance reviews.

He died in 1993. With his boots on.

Three examples of the principal of advancement: we accrue knowledge as a people, and don't purposely turn back the clock. The Dark Ages happened because the baser cultures successfully attacked (think about that in the Sandy Hook context of gun control). As attributed to Newton: "If I have seen further it is by standing on the shoulders of giants". I wonder where the IT world stands today?

Which brings us to the question: how is it that IT generally, and database applications specifically, have shown such retrograde/reactionary tendancies over the last couple of decades? That is to say, the embrace of data technologies from the 1960s (and even, one might point out, 1950s)? Why is iteration/looping over sequential data the sine qua non of coding? Is it pure ignorance? "Those that ignore the past are doomed to repeat it"? By way of contrast to other professions, (I was tempted to say that physicians don't revert to using leeches, but they do, a bit) such as medicine accrue learning and move forward. So, I'll point out that they no longer consider blood soaked clothing as a mark of expertise.

Although even some Greek surgeons had advised washing the hands before dealing with patients, this aspect was overlooked and the doctors strode around in blood-stained coats. The bloodier the coat, the higher the reputation of the surgeon.

But the younger set are willfully embracing siloed, non-transactional, client-side driven, flatfile data applications in java and such that, save for the syntax and CAPITALIZATION SCREAMING of COBOL, are semantically the same as those applications tapped out on 059 keypunches. Why? Part of the explanation lies with the residue of early web technology. By the mid 1990s, commercial computing was divided between mainframes running COBOL with mostly DB2 and a bit of Oracle (notoriously ill-suited to the 370 architecture) over SNA to 3270 character terminals, and AS/400 or *nix mini-computers running some RDBMS (or 4GL/database) over RS-232 to VT-X00 character terminals. This was the era of client/server in a box. All the data be ours sayeth the box. Such machines had an order of magnitude (or more) slower processors, smaller memories, and disk capacities than today. But they could populate their screens in real-time. While the 3270 did/does have some local coding capability, not unlike a hobbled javascript, the point was to manage the data in-situ and leave just screen painting to the client. Real time data population and editing against the datastore was the norm. One had to be careful with transaction scope, but one has to anyway.

With the young web, where bandwidth to the browser was effectively far lower than a VT-X00 would see over RS-232, javascript increasingly available, and young studs eager to make money (rather than a career)... Well, we have the "software problem". Not to mention that the math ability of US kids declined.

Unfortunately, the percentage of students in the U.S. Class of 2009 who were highly accomplished in math is well below that of most countries with which the U.S. generally compares itself. No less than 30 of the 56 other countries that participated in the Program for International Student Assessment (PISA) math test had a larger percentage of students who scored at the international equivalent of the advanced level on our National Assessment of Educational Progress (NAEP) tests.

The relational model, which doesn't demand anything more than elementary set theory, doesn't fit the bill.

Making a quick buck on Wall Street, selling bogus securities and such, doesn't require knowing algebra. So, they don't. Making a quick buck with some social app doesn't, either. So, they don't.

I recall the process of class selection as an undergraduate. When it came to electives, even for those with a math-y major, the absolute preference was for classes from sociology, poli sci, and maybe psych. Why? Because there were seldom "right" answers, and if one was an accomplished bullshit artist (and even engineers could manage that), then a high grade was assured. The lower the rigour, the higher the GPA. And so it is with the applications they gravitate towards. Unstructured data. Fuzzy logic. And so forth.

Will it be possible for Chris Date, to cite an example, to be well regarded in his 80th decade (assuming he can get that far), in the way Deming was? I hope so. On the other hand, Deming was gone before the vampire squid assault of the Bayesians, so if he'd been born in 1930 rather than 1900, he could have been ignored once he reached 60.

The infrastructure to support client/server in a box on the web only gets deeper and stronger. At some point it will win out. And we can all die with our boots on.

The Whistling Gypsy

In the first years of the 1960's, there was a thriving (by those days' standards) folk music scene. It was centered on adults, 30-somethings and older, in nightclubs. By the mid-1960's, the Beatles and Bob had put an end to that. Dylan and friends developed an acoustic music labeled folk, but based on performer penned songs, rather than traditional ones which had aged with the people. So to speak. The only exemplar of that era that's still remember by more than a handful of diehards is The Kingston Trio. For myself, while decades younger than the nightclub goers, I was drawn to the best of the best, The Limeliters. They did some sung-through commercials, identifiably sung by them, which are rare today but common then, so that's how I discovered them, Coke and Lucky Strike, if I remember correctly. Over a couple of years, their recording fame was due to a few live recordings, done quickly and frequently; they were done for by 1963.

"The Dance" by Fleetwood Mac is an example of audience interaction but The Limeliters, were more so. Audience interaction was nearly as much of the act as the singing. Their studio recordings were flat by comparison. The banter came mostly from Lou Gottlieb, a Berkeley Ph.D. (in musicology, not math stats; the latter would make the tale too cloying for words). When the group disbanded, he had his own fifteen minutes of fame when he attempted to will a farm/commune he'd established to God (the piece misspells the group's name, which happened a lot after they'd been consigned to history). The California courts weren't amused.

He opens one of their better known songs with some of his better known banter: "... the title of the song: 'The Whistling Gypsy', or where the hot wind blows." Well, there is a hot wind in the database world, and it's Northwind. Along with Date's supplier/parts database, the most well known and universally available demonstration database in the whole entire world. If your world is Windows and SQL Server.

I've been searching, in a desultory way, for a flat file dump that I can load into my various linux database engines, notably Postgres (due to the PL/R support which provides my R fix in database connectivity) to no avail. Until now. Here's the stuff. DB2/LUW is missing. May be I'll do that one too. For now, Postgres will do nicely.