Grist For The New Mill

My kindred soul amongst the highly paid curmugeons remains Nick Carr (not Cage, as my independent fingers sometimes say). Had I started these endeavors long before I did, I might be he. Sort of, of course, in a metaphorical sense. Today brings further evidence.

Susan Pinker reports the obvious: that, given the opportunity, even kiddies will while away time being frivolous.

"Students who gain access to a home computer between the 5th and 8th grades tend to witness a persistent decline in reading and math scores," the economists wrote, adding that license to surf the Internet was also linked to lower grades in younger children.

Why should we be surprised? Their parents are busy doing Angry Birds on their mobiles and slurping up porn on their PCs.

The problem is the differential impact on children from poor families. Babies born to low-income parents spend at least 40 percent of their waking hours in front of a screen -- more than twice the time spent by middle-class babies. They also get far less cuddling and bantering over family meals than do more privileged children. The give-and-take of these interactions is what predicts robust vocabularies and school success. Apps and videos don't.

The Wife spent a year or two as a middle school teacher (it was Just What I Want To Do, until it wasn't) and learned the most important lesson that early childhood professionals have known for, at least, decades: if little Billy is going to be a thug, little Billy will have arrived by five or six years of age. Raise disassociated kids and we're Shocked, Shocked I Say to find them growing up anti-social cretins. My, my.

If anything, reliance on The Network (in all its omniscience) leads to a shallow brain. Not to suggest we should just go back to Readin, Ritin, and Rithmetic of our grandparents. But the skull of a juvenile will fill up, nature abhoring a vacuum and all that, and we may as well fill it up with reasoning and base knowledge to support reasoning. Knowing how to code PHP or javascript doesn't aid that.

Question of The Day, 2015--01-29

Is it just me, or does the proliferation of native platforms as well as browsers, lead to the inevitable conclusion that it makes much more sense to treat the client platform as a generic display screen (aka, everything's a VT-100), and keep the intelligence in the datastore?

About That Other Shoe, Part the Second

I guess I'm not alone. Now, R commercialists are making the argument that M$ should put R in the database. Mind, Joe Conway did it first, so far as I know, with PL/R in PG. He could do that, and all by his lonesome, since Postgres supports user-defined functions in C. And, since R has hooks in C, the "R" language is just nomenclature for yet another C hook. Not all databases, oddly, support C. I suppose it's a security thing, since the bulk of databases these days are C, with I suspect some remnants of assembler (likely in-line). A cwaffty wabbit might be able to subvert the engine through the interface. You'd have to ask the engine writers how big a deal that might be.

What's even more interesting is that the poster left out:
PL/R itself
Netezza (and DB2, sorta) with Revo R
HANA with R (apparently, more than one way)
DB2 with some of SAS (boy howdy that's gotta cost!)

As Sony & Cher (mostly, Cher) put it, "and the beat goes on".

And, just for yucks, have a look at the PG databases that exist by different names. And some tweaks, I'll admit.

About That Other Shoe

Well, it's just fallen. M$ has eaten Revolution Analytics!!! This was predicted here first. Ok, so I didn't say M$ would eat Revolution, but I did make it clear that M$ needed to get SS some real stat support.

This acquisition will help customers use advanced analytics within Microsoft data platforms...

Does that mean M$ will build a PL/R work-alike? One hopes they realize that's the true value add.

The End of the Road, Part the Second

It was some time ago, in fact my SlickEdit told me I had already a file with the title (I had forgotten, and fortunately tried the same title. phew.), that "The End of the Road" appeared in these endeavors. And the last offering made reference to the subject with the words, "we humans are on the ever flattening asymptote of knowledge of the real world". Both pieces sprang from some part of the lower brain stem of memory. I always suspected that some famous writer had figured it out. But, try as I might, I never found patient zero, so to speak.

Well, I tried again a few minutes ago, and there it was, right at the top of the search list. Damn. That's patient zero from the point of view of nowadays innterTubes. The piece is a ten year retrospective of his book, "The End of Science". I recall reading neither before typing the first End of the Road piece. The book was the better part of two decades ago, so it is possible that I've read it. If so, lost to the bowels of that lower brain stem.

The arguments in his 10 year article are completely in line with my conclusion. For the quants out there, it matters for this reason. Economic evolution, particularly in the USofA in the 19^th century through WWII, was built on resource discovery, exploitation of said resources, and scientific expansion of knowledge to make further use of said resources. Without them stuff in the ground, we'd still be hunter-gatherers on the plains. You may notice that there are places on the planet without them stuff in the ground, and the folks live pretty much as hunter-gatherers. It's not that they're backward. It's that Mother Earth gives them so little.

Without the science and engineering of organic chemistry, including figuring out the periodic table and such, that black goo at Titusville would be no more than nasty dirt on Momma's carpet. But we know all that now. Back then, scientists knew that they didn't know what physical matter consisted of. It was this ignorance that propelled us to the Bohr model. That was 1913; a rather long time ago.

While Horgan never uses these words, the point of new discovery in science or engineering is whether there will be commercial use of same. In the 19^th century, sure. Today, not so much.

Here's a history of element discovery. Note how dominant the 19^th century is. One could argue that plutonium (1940) was the last element of consequence. Boom?

For the macros, what all this means: that rising tide isn't out there anymore. Tracing economic history up to 1950, mid-century, or perhaps to 1970, one could argue that progress in science and engineering led to old materials, methods, and product being displaced by newer. And thus, expanding economies and employment. That's not true now. Ok, some might waffle and say, "not so much". The point is: new science and engineering means new industries. Name one since WWII? You can't. All you can do is name industries that have been miniaturized by silicon and software. The growing industry is finance, and that's a zero-sum game (or less, if CDSs dominate); skimming its revenue and profit off the real economy.

Under My Thumb

Since I live in South Butt Plug, CT and grew up in western MA, I'm supposed to cleave to all things sports of Boston. Problem is, I can't stand the B boys of Foxborough. But DeflateGate does offer the opportunity to chime in a bit. So, three points.

1) Both B boys are known control freaks. That Bill never, ever considered ball hardness is laughable, considering that he pontificated at great length on the manners of degrading practice balls. Change pressure to make life difficult, along with the other silliness? He never thought of it? Yeah, right.

2) Tommy boy can tell the difference between 13.5 (which he doesn't like) and 12.5 (which he loves). And never noticed a 2.5 pound drop? Yeah, right.

3) And the officials never caught on? Well, that's entirely likely. If you've watched the refs, you'll note that they don't grab the ball and throw it a la Tommy boy: grip it palm down, over the ball, and throw overhand. They don't. They hold, really cradle, the ball palm up and spin-toss no harder than slow-pitch softball. So, no, they'd likely not notice.

Ain't No Science Like Old Science

What has become a recurring theme, or warning, in these endeavors: it is foolish to view the future as pure extension of the past. This is particularly true of anything science or engineering related. As I have described more than once (and referenced the writings of those more famous than I), we humans are on the ever flattening asymptote of knowledge of the real world. We already know, pretty much, exactly how the (macro) world works. We even know, pretty much, how the molecular (atomic) world works. We may have some things to learn about the cosmologic and sub-atomic worlds, but I'm not sure even there that anything we do learn will be economically significant. The upshot of that: there are fewer and fewer groundbreaking discoveries to be discovered. In other words: those quants/micros out there thinking they can extend their financial models based on the last X years in Y industry are morons. It ain't 1850 with a vast land of resources and new science to be uncovered.

There is no Mr. Fusion sitting the back of a DeLorean in our collective future.

One the sectors of the economy I find fascinating, which perplexes me still since I abhored biology in high school and never took such a class in college, is biopharma (or, whatever they call it these days). There are two aspects of the sector which have emerged in the last few years. One is the exploitation of the orphan drug act, where companies spend money to garner approval for drugs which may, or may not, make much difference to patients with "rare" diseases (the legal definition makes rare not so rare; that's part of the problem) at exorbitant prices. The HepC arena is gaga over the issue as we speak. And, how much of that accounting number assigned to a drug's existence was actually spent in the lab? Hmmm?

The other, more pernicious perhaps, is the active destruction of what R&D is left in companies as motivation for M&A. Derek Lowe (who merely has some of his blogging copied) has a piece on SA today, bemoaning recent events. One of the excuses given, no cite off the top of my head on offer, for the rush to China for manufacturing is that the USofA no longer has the feeder companies necessary to support large scale assembly/manufacturing, and China does. Of course, which is the chicken and which the egg? Here in South Butt Plug CT, the small metal working companies died out when the large companies to which they had been selling decamped for The Red States and The Red Country. It wasn't the other way around.

Turning all American corporations into financial firms will, sooner or later (and, be prepared for sooner) fail. Financial services is merely a matter of moving moolah from one pocket to another, or robbing Peter to pay Paul. It is non-productive. All value derives from some underlying activity. All finance profit derives from skimming off some part of that activity's value. We saw with the Great Recession that moving massive amounts of capital to housing failed because the underlying "asset" produces no saleable product, so the vig had to be paid out of the real earnings of mortgage holders. Since said earnings have been, at best, stagnant for at least a couple of decades, the whole edifice had to fall. The banksters got to keep most of what they'd taken, of course. The large builders made out like bandits, since they got the moolah up front. Once sold, the house and mortgage were somebody else's problem.

For the STEM folks? Well, some say we should be making more of them in school. But, why would a kid sign up for the brain warping hell of electrical engineering, in the face of such jobs being shipped to India (or some such)? The alternative, of course, is to do Business Administration and learn how to design the next liar loan!! Not so much strain on the brain cells, and lots more moolah for the effort. Kids may be high on dope all day, but that doesn't mean they have lost all touch with reality. They, by and large, aren't dumb enough to take out loans (which all but the top .001% of students have to) to learn an occupation which will never employ them. And, we know what happens when a country generates more STEM graduates than it has jobs for: Eastern Europe is the center of cyber crime for a reason.

Obambi said that all kids should learn to code? Yeah, right. Just what we need; a bunch of kids dreaming of grabbing the brass ring (look it up) of WhatsApp (they'll end up finding their wage driven down by IIT kids in Mumbai). Could that be worse than dreaming up liar loans? Or will we just build a domestic cyber mafia? Stay tuned.

I've Been Modelled

OK, so that's a lame pun on an age-old de-acronyming of IBM: I've Been Moved. That was in the time when account executives were essentially itinerant peddlers, since promotion meant relo. Not so much now.

But, hot off the presses, and it will take a bit of time to waddle through all the pubs, it appears that Big Blue has done the stat engine in database thingee with SPSS (and, may be, Watson) in the just announced z13. I'd rather R on Power, but ya can't have everything. But CICS.... IT LIVES!!!! BWAHAHAHAHAH.

A Yentle for the 21st Century

Quant maker, quant maker, make me a quant!

At least since the implosion of LTC (Long Term Capital Management, and follow the references for the gory details if you're interested) in 1998, or a bit earlier if one notes the proximate event, the ability of quants to get it right has been in question. Yet, they (or, at least, some) are paid wages in the seven figures.

The theories of Merton and Scholes took a public beating. In its annual reports, Merrill Lynch observed that mathematical risk models "may provide a greater sense of security than warranted; therefore, reliance on these models should be limited."

One might even go back to the savings and loan scandal(s) of the mid-1980s. Again, the references, if you're interested. Again, the 1% were looking for ways to earn greater than the market rate of interest at lower risk. They Ain't No Such Thing As A Free Lunch.

L. William Seidman, former chairman of both the Federal Deposit Insurance Corporation (FDIC) and the Resolution Trust Corporation, stated, "The banking problems of the '80s and '90s came primarily, but not exclusively, from unsound real estate lending".

In other words, the S&L scandal, or crisis as apologists prefer to call it, was directly caused by quantitative incompetence at its most elementary. This wasn't off-the-wall derivatives, but basic present value calculation. The same can be said of The Great Recession. "Money for nuthin' and the chicks for free."

But, these two banking (in the extended sense) screw ups taught both the regulators and the participants nothing. Moral hazard was ignored, and the quants went on a bacchanal with the election of W. Liar loans were created, securities made from them were created, and derivatives of said securities were created, with the credit default swap being the most evil. Imagine if you and your best friends could bet on whether an abandoned tenement crack house (which none of you own any part of) in the worst part of town might burn down in the next year or two? That's a CDS. Regulators looked the other way. Moral hazard? Never heard of it.

Which brings us to the London Whale. Bruno Iksil is reported to have graduated from a top tier French engineering school (École Centrale Paris), although I've not yet seen which degree or subject he earned. Likely not a Ph.D., since those who've got one generally boast. Even so, it appears that Excel was beyond his ken.

From the JPM internal review as reported:

"After subtracting the old rate from the new rate, the spreadsheet divided by their sum instead of their average, as the modeler had intended. This error likely had the effect of muting volatility by a factor of two and of lowering the VaR . . ."

That's some interesting history. The last few days brings the whining of petro-bankers. They never saw the reserves coming? Reminds me of this quote from Upton Sinclair:

It is difficult to get a man to understand something, when his salary depends on his not understanding it.

Here's their problem:

Still, if oil prices remain near $50 a barrel for long, economists and industry analysts expect a sharp deceleration in production this year, idling energy bankers and cutting into their lucrative fees.

Quant maker, quant maker, make me a quant?

It wasn't as if these petro-bankers didn't know how much drilling and lifting was being done with their money. Nor could they have been ignorant of both the anemic recovery in the USofA, and the stalled economies of Europe and Asia. Or that the Saudis can lift oil at a profit at a price much lower than shale or tar sands; they'll continue to lift in order to get the moolah needed to keep a lid on unrest. Econ 101: supply is increasing and demand is, at best, stagnant. Of course the price per barrel will fall. D'oh!!! These are the same yahoos who asserted, by action at least, that house prices could rise 10%/annum for-freaking-ever.

When oil prices crashed in the 1980s, many Texas banks failed not because of loans to oil producers, but because of loans to local real estate developers who had been caught in the energy bust.

How'd you like to be a S&L president in North Dakota, today?? Eh?

Let's assume that quants built the right way, and installed in organizations which deal with data, might aid in avoiding such nonsense.

So, how to build a quant? Let's assume that the goal is short term conversion of a subject expert into a subject expert who speaks data good.

Quant maker, quant maker, make me a quant?

Most who do "quant", especially in the financial sector, aren't math stats or ORs. They're just folks looking for cracks in existing regulation that can be exploited to make ever more money for themselves, and aren't the sort of folks I'm talking about. Those who do so, and call themselves quants, are the apotheosis of micros. The CDS was created by a banker, outside the realm of what then current regulation understood. Whether anyone, at the time or since, understood that the instrument amounted to allowing the entire moneyed community to bet on some crackhouse to burn down, is unknown. And, likely, unknowable, since such derivatives were and remain largely unregulated (mega lobbying against Dodd-Frank provisions to bring them to heel). Even knowing the global value of derivatives is not known with certainty, since each is viewed as a private contract. Wonderful.

Many quants, in the financial sector according to legend, come not from the social sciences, but math and physical sciences. The latter understand a rule game decreed by God (or Nature, as you prefer), which leads them, if not reminded constantly, to model based on whatever set rules they know about. That the rules of finance are created and changed by other humans (or, in the case of the CDS, themselves) seems to be forgotten. Read most texts on financial engineering, and you get the gist: we model based on fixed laws of actors' behavior. Fact is, most financial engineering is really about finding lucrative cracks in regulation.

Quant maker, quant maker, make me a quant?

What's a quant? The first question to answer. I'll exclude the flunked out hard science Ph.Ds who write C/C++ for high frequency trading firms. They're not really doing quantitative analysis, rather money flow arbitrage in microscopic time frames. Yes, in large enough scopes, such inquiry may look a bit like Brownian motion and other physical phenomena. Also, the folks who create new products based on perceived holes in law/regulation. Again, these folks are exploiting weakness in the boundaries that exist to keep Darwinian anarchy from breaking out all over. Ayn would be proud of them, but not I. Much of financial engineering is based on time series analysis of asset pricing. That's quant, but is based on a stability assumption that's often incorrect. The results don't always (may be, nearly never) live up to the promise, but the techniques are quant. Thus, ride the trend up passively and depart to cash (or market shorting ETFs) passively at inflection. When is that? The hard sciences, bio related in particular, make use of regressions and ANOVAs as a matter of course. That's quant. The social sciences, again, lots of ANOVA (the psycho- types invented it).

Second, who? It could be "me" who wants to be a quant, or it could be a Corner Office Suit who decides another quant is needed. In both cases, we'll assume that the candidate isn't already a math stat or operations researcher. Like early programming in the '50s and '60s, quant (and Data Scientist!) is largely a self-created occupation. Just look at all the flunked out maths (not math stats), physics, engineering, chemistry, etc. folks that have been in the Wall Street invasion. So: the process has to be self-administered in the former case, and divined from CVs in the latter. We want a short-course to turn either "me" or some subject expert into a real quant. Well, real enough for the organization's purpose.

The latter case is easier to deal with. The COS should be looking for a candidate who's mostly subject specific. Math stats tend toward new proofs, and we've seen from The Great Recession that applying a God's Rules world view to dig in the minute cracks of regulation leads to disaster. Student and Fisher were grounded in industrial (physical) processes, but not so much these days. If your goal is to just suss out ways to bend the current rules, or lobby for beneficial (to you) rules changes, you don't much need a quant. Beyond the exercise of adding up your anticipated windfall. You just need to find a welcoming ear to lobby for the policy you want.

Quant maker, quant maker, make me a quant?

Early in my career, such as it has been, I taught quant type courses for the CSC/OPM at Dupont Circle in Washington. Nice coffee shops in the neighborhood. They were anywhere from a couple of days to a couple of weeks, with basic stats, stat packages (pstat, BMDP, and a bit of SPSS), and elementary programming (BASIC and FORTRAN); later on 1-2-3 was very popular. One can create a quant in a few weeks, if the goal is narrow, and the content taught is focused. Hell, the London Whale was an Excel Master of the World; or so he thought.

So, the tool? R, of course. I don't take R with unbridled enthusiasm. Too much of the R community is coding-centric, likely because S was really a DSL by and for math stats. The goal here, make me a quant, is to get the subject expert up to speed with R and ancillaries sufficiently to use R function and packages to analyze some data. A user of R, not a maker of R. May be, for data acquisition, a few words about Excel. But not for analysis. The candidates are, nearly certainly, convinced that they're already Excel Guru level, and equally likely to be averse to having to go through some other training exercise. "I know Excel. What more do I need?" That sort of thing.

While in the process of composing this missive, I found this post from a prof building a bespoke text for a one semester undergraduate econometrics course. (Ironic asides: while not the London Whale's ecole, it is a French one. If you know the name Bourbaki, you see the irony. Second irony: Kind of a NetPaper course book thingee. If only NetPaper hadn't crapped out.) So, we'll say 8 or so weeks of material to make an econometrician. I'll say an 'applied' econometrician, in that economists continue to shave off new bits of stuff; some of it actually useful. Chapter 3 is the meat of stats meets economics.

First, RStudio. It makes no sense to force such folks, who will be menu clickers, to suss out command line R. Ick.

Second, what should be the reference? One could construct a bespoke "course pack" text as above, but that's more in service to a prof's publishing CV than to the students. It makes more sense to work from a comprehensive text, pick and chose the topics/chapters that pertain, and provide precursor topics in digest form. They students leave knowing the focused material, and a reference that will serve in their work. Crawley's The R Book retains first place, for me. Graphics are still from base, so Hadley's ggplot2 book, too. He's recently written that a new edition is in the works. Chapters referenced are from Crawley.

Quant maker, quant maker, make me a quant?

Here are the topics I'd want in a short-course quant making:

- basic R and Rstudio

Chapters 1 and 2. RStudio functionality introduced as needed along the way.

- elementary probability

While not likely to be used directly, and I've always been leery of its importance, the notions of probability are important. The course completer isn't likely to discover new probability distributions.

Crawley doesn't have a probability chapter, somewhat oddly. I'd substitute Wikipedia page, and referenced topics, for discussion purposes.

- the data

My preference, no surprise, is to stress getting fully munged data from the RDBMS using SQL. A bit of talk about Postgres and PL/R capability is in order, given that most other industrial strength databases offer similar now.

Parts chapters 3 and 4.

- descriptive statistics

Very important, particularly in these days of Big Data, which devolves from inferential stats into just descriptive. I say "just" because inferential machinery no longer applies. Now, for some areas, such as consumer manipulation, correlation is important.

Most of chapter 7 deals with distributions and their parameters. That will do.

- graphics

Why in third place? Well, for those in the Big Data and descriptive world, and for those who've got to make pretty pictures for the COS, that's 99.44% of the job. So, basic R graphics and ggplot2.

Chapters 5 and 29 (no, no one has figured out why he split that way or why no ggplot2) deals with base graphics, which are sufficient for most purposes. From there, ggplot2.

Now, things get goal specific.

- finance and bio and other physical realms

The first bit of chapter 9 on modeling, then chapter 10 on regression and 13 on generalized linear models. Time series in chapter 24.

- social sciences and such

Chapter 11 on ANOVA and 12 on ANCOVA.

The obligatory "advanced topics"

Chapter 22 on Bayes, 25 on multivariate.

By the time you're done, you'll know enough to get around R, and apply existing stat processes to data. You won't be an R coder; you'll be an SPSS/SAS user with a different syntax. Coding, in the sense of writing new stat procs, will come much later if at all. Don't worry about it. One aspect of such short courses, as distinct from the more contemplative semester in college approach, is that we've jammed a lot of material into the skull quickly; a week or two or three. (Recall that a 30 hour semester course is 3 hours a week for 10 or so weeks. We'll be chewing up 40 hours in a calendar week. Some brains will explode in the classroom. Pay them no mind.) It can spill out just as quickly if it isn't used; the difference between short-term and long-term memory isn't a myth. You want your base quant in short order. You'd better put that new knowledge to work in equally short order. Over time, months and years, the knowledge will become internalized, much like a shortstop learns to throw both side arm and overhand, and when, in an afternoon. Takes a while until he does it right without thinking about it (well, unless you're A-Rod, in which case there's never much thinking ever).

Quant maker, quant maker, make me a quant?

Inappropriate Behaviour

Not that I approve, mind you, but SQL (if not the RM) is invading the Data Science space. R has had join semantics for some time, and very slow so if your data is already in a RDBMS do it there, but Hadley has gilded the lily. I guess I should be thankful, sort of. However, the Data Science brigade will immediately shift to doing RDBMS work in R, where it really isn't appropriate. Another case of turning the RDBMS into a file server. I'm irritated already, just in the time it took to type this out.

In Praise of Free Enterprise

As reported today:

Meanwhile, Transportation Minister Ignasius Jonan cracked down on five airlines Friday, temporarily suspending 61 flights because they were flying routes on days without permits. Earlier, all AirAsia flights from Surabaya to Singapore were stopped after it was discovered that the low-cost carrier was not authorized to fly on Sundays.

Jonan also sanctioned nine more officials for allowing the AirAsia plane to fly without permits, bringing the total to 16.

Governments only interfere with profit making. Fie on them.

Early Adapters

No, that's not a typo. It's a pun. Here's the punch line.

The plummeting of oil shares, with the attendant fall in the price of crude, distillates, and such should give the quants, micros, and macros pause. Yet again.

The notion of an unfettered Rand-ian society rests on a basic assumption: that humans can adapt to any change before it's too late to avoid catastrophe. Yet, each time there's been a popping oil bubble, due to a miraculous (that's sarcasm, just so ya know) increase in supply, Mr. Market and his attendants go berserk within weeks, if not merely days. "We can't live with $X dollar oil!!! We need $Y dollar oil!!! We'll lose so much profit!!!" And so on. X < Y, of course. The same thing, in the other direction, happens whenever the Peak Oil Pundits appear to be right. Wealth share flows to the Oil Patches, just as now it is flowing to everybody else. The best thing that could happen for Apple, and such, all that more discretionary moolah in what's left of the middle class. All those Red States that have been living high on the hog for the last decade or so, not so much. Does it really make sense to pay some knucklehead in North Dakota tons of money to do brain dead manual labor? Of course, all those knuckleheads getting paid tons of money drives up the price of whatever it is they want to own. Localized inflation is a real phenomenon. Ask anyone's who's lived in the DC area. May be not Congresspeople and lobbyists, of course; they get to not only live high on the hog, but own most of them too.