A Hop, A skip, and A Jump

Free at last, Free at last, Thank God almighty we are free at last.
-- Martin Luther King

OK. So what does such a quote have to do with a site, titularly at least, devoted to the RM, stats, and data generally? Well, this new report on Intel's Optane DIMM.

[On a side note, the earlier piece on Optane SSD has a comment from the author to the point that the byte addressability of Optane is of no value for files. That's true for fixed sector non-mainframe HDD, and later SSD, but not true for CKD DASD:

IBM CKD subsystems initially operated synchronously with the system channel and can process information in the gaps between the various fields, thereby achieving higher performance by avoiding the redundant transfer of information to the host. Both synchronous and asynchronous operations are supported on later subsystems.

]

Current SQL engines do their work with a hop, skip, and a jump. Optane, with proper motherboard, chipsets, and (heavily?) re-written applications can go from cpu to datastore. We could, then, do what Dr. Codd said: "all at once". Yum.

The current paradigm is from cpu registers to on-chip cache to memory to SSD/HDD; more or less a hop, a skip, and a jump. Now, for SQL databases (or any serious datastore), what generally happens is that the engine writes to the trans log (more or less synchronously), which is at some later time flushed to the "permanent" datastore on SSD/HDD. A system/application crash is supposed to only lose open transactions; all commited trans exist in durable storage either in the trans log or named tables. Wouldn't it be cool if transactions commit to durable storage immediately?

As the piece says:

Intel has been laying the groundwork for application-level persistent memory support for years through their open-source Persistent Memory Development Kit (PMDK) project, known until recently as NVM Library. This project implements the NIA NVM Programming Model, an industry standard for the abstract interface between applications and operating systems that provide access to persistent memory. The PMDK project currently includes libraries to support several usage models, such as a transactional object store or log storage. These libraries build on top of existing DAX capabilities in Windows and Linux for direct memory-mapped access to files residing on persistent memory devices.

Application re-write may be non-trivial:

Intel will be offering remote access to systems equipped with Optane DC Persistent Memory so that developers can prepare their software to make full use of the new memory. Intel is currently taking applications for access to this program. The preview systems will feature 192GB of DRAM and 1TB of Optane Persistent Memory, plus SATA and NVMe SSDs. The preview program will run from June through August. Participants will be required to keep their findings secret until Intel gives permission for publication.

But for the RM world, Optane offers the ability to do what Dr. Codd said, "all at once".

One might ask, in a semantic sense, how a SQL engine on such a machine might do its work. Let's chew on that.

First, what about the divide between trans log and named tables? Do we still need a trans log? The blockchain example, which is a database-less trans log, suggests we do. For audit purposes, it is needed, too. Does the trans log still need be the bottleneck to the named tables? Not with such Optane storage. The engine would just do immediate writes to the log and the table(s); that's the duration of the transaction.

Second, do we still need buffers? May be, but may be not. The purpose of buffers is to mediate between fast memory and slow disk, now SSD, but still slower. According to the report, 512GB DIMMs will be available. Current boards go to eight slots, which works out to 4TB. How many applications need more than that? Google and Amazon and Facebook. Commercial applications? Not so much. Taking into account the data reduction side-effect of 3 or 4 or 5 normal form, may be most would be happy with that much storage.

Third, typical applications don't keep all data in hot storage even now, so 4TB looks to cover many, if not most, use cases.

Fourth, some DIMM would be normal DRAM to hold OS and application code and code's invariant data; this appears to be supported according to the report and comment stream. It would certainly make sense to do this.

So, how would a SQL engine work with such a machine? The engine and its data would reside in DRAM, while the tables and (active) log in Optane. By active log, we mean that the engine would flush completed transactions to SSD as needed; in reality, only long running transactions (you don't write such monsters, do you?) would be in the active log. There would be no need for memory buffers, but the notion of locks needs to be considered. Current practice whether locking or MVCC relies on memory buffers to "improve" performance. Simple, single row, update would be written directly to table and log. Normal memory locking would suffice. Multi-row/table update? Umm. In the MVCC case, it depends on multiple images of rows to segregate transactions, but would there still be any point to MVCC semantics with Optane storage? For now, I'd say not. Since update (could/should) happens "all at once", collision doesn't occur; each client sees the current state of each row all the time, modulo duration of actual write. Isolation amounts to memory locks. Cool.

Years ago, diligent reader will remember, I had occasion to work with TI-990 machines. One ran a Ryan McFarland COBOL application on a TI machine while the other ran the chip version on a custom board/OS/language (ah, those were the days). What was (nearly?) unique about the 990 architecture was that it was registerless, sort of. All code and data resided in memory, and there were three cpu registers, one of which was a pointer to the current application's base address in memory. A context switch required only resetting these registers. Each application retained its context in situ. I wonder whether TI will resurrect it?

The TI-990 had a unique concept that registers are stored in memory and are referred to through a hard register called the Workspace Pointer. The concept behind the workspace is that main memory was based on the new semiconductor RAM chips that TI had developed and ran at the same speed as the CPU. This meant that it didn't matter if the "registers" were real registers in the CPU or represented in memory. When the Workspace Pointer is loaded with a memory address, that address is the origin of the "registers".

Whether Optane has much value-add for most applications, I don't know. But for heavy multi-user/single datastore applications such as SQL engines, yeah man!!

This, of course, is merely first-pass thought experiment. More to come.

The Object of My Affection

Return with us now to the thrilling days of yesteryear. If you're old enough, you remember that as the intro to "Lone Ranger" episodes. One might also see that as the devolution of application development. So, here goes.

Recall the Holub Mantra (my description), this "Bank of Allen" tale. The signature observation:

The maintenance manager is happily sleeping in the back office instead of running around changing ROMs.

Holub is talking about regular application development, independent of datastore. But the principle applies in spades to databased applications.

The development of the relational model was driven by maths, specifically predicate logic. It was not, as many infer, data reduction; which does come as part of the package. But the whole point is data integrity. The primary way to ensure data integrity is to limit the number of hands who can mess with it. As Holub points out, if one models an ATM network based on object capabilities, the datastore is the controller of the data. That is how it must be. In time, Holub gave in to the innterTubes meme, alas. That might have been defensible back before highspeed connectivity, but not now.

If there is just one point of control, then there's just that one point of modification and maintenance. That's why the manager doesn't have to run around changing ROMs. But the driving point of object oriented design and programming is that an object encompasses both data and function, aka methods. What has happened, especially in the java world, was the re-invention of the language, called either javBOL or COBava. Same old procedure/data protocol from the 50s. Swell.

Every now and again, one sees a crack in the wall of reactionaries. Here's a recent example. The author doesn't reach the logical end of the argument, but does show more insight than usual.

Too often, though, business logic is built and added late in the process, forcing it into whatever nooks and crannies are available. While this duct-tape approach sometimes works, it makes the resulting system difficult to maintain when the business logic is spread across many moving parts in the data architecture.

The standard response from the COBava folks is, "it's impossible to design an application in the first place, so let's just build it like a Frankenstein monster." Or, parts is parts. Remember, all code logic comes down to data compares. Period.

Your Tax Dollars at Work

Remember all that braying about how corporations would use offshore profit to add more workers if/when they brought it back? You do remember? Did you believe it? You shouldn't.

16:17 CW Curtiss-Wright to utilize $50 million in repatriated foreign cash to expand 2018 share repurchase program (133.92 +2.03)

Board of Directors has authorized an additional $50 million for share repurchases in 2018, to begin immediately via a 10b5-1 program, by using cash that the Company has begun repatriating from its foreign subsidiaries.

Earlier this month, the Company repurchased more than $12 million in shares opportunistically through a supplemental share repurchase program and has ~$36 million remaining under the current share repurchase authorization. Since early 2014, the Company has repurchased more than 7 million shares for an aggregate purchase price of nearly $550 million

Mask of The Red Death

Have you noticed that the "unmasking" meme has fallen away from the Orange Julius Caesar cabal's whining? Why might that be? Reporting that the FBI had an informant talking to Page, Papadopoulos, and Clovis very early in the investigation has created the spy meme. Hmmm. May be they were warned that such talk will only speed up the inevitable end.

Last night's MSNBC nightly feed included Maddow's refreshing of the Nixon Enemies List, in the context of Orange Julius Caesar's secret attempt to coerce the PO into doubling(!) Amazon's delivery charge. That shouldn't be surprising. What I noted was Maddow's use of video and audio from the Watergate investigation. The tapes proved that Nixon, et al, really did subvert the law. The outing of the Amazon/PO corruption was offered as today's version. But I disagree.

From the moment that Comey told us that the FBI had an open investigation into Trump/Russia, I inferred the jig was up. Maddow came this close to closing the circle. Have I mentioned in the course of these missives that I spent some time working for Jack Anderson? Well, yes. And you can find the articles I worked on if you look hard enough. This was after Anderson was co-opted by Reagan, but he had long since quit doing much reporting or writing, leaving that to the staff. I got to meet some spooks. That was fun.

My time in DC convinces me that the intelligence community, the NSA in particular, has the Trump Tapes. It's perfectly legal for NSA/CIA/FBI to sweep up Americans who engage with foreigners in treasonous activities. Doesn't matter where their feet are at the time. The cabal knows what it did, and was counting on not winning to keep their activities from public view. Thus the FISA nonsense. They just wanted the money. Just as Jared extorted from Qatar. In due time, the community will release the tapes. Whether Orange Julius Caesar spends time in prison is the only question.

The Book I Didn't Get Around to Writing

On more than one occasion, I've bemoaned the silliness of Big Data. So have some others, every now and again. This morning's email brought a spiel from Amazon, with some books I might like. There is one: "Big Data, Big Dope".

While others have written about the dangers of Big Data, Stephen Few reveals the deceit that belies its illusory nature. If "data is the new oil," Big Data is the new snake oil. It isn't real. It's a marketing campaign that has distracted us for years from the real and important work of deriving value from data.

Exactly. OTOH, I worry that BLS/Census survey efforts have been corrupted by Orange Julius Caesar's minions in the agencies. Remember, it was the political appointees who made the case for WMD, not the grunts doing the work. When something is too good to be true, it likely is. One doesn't need all the population to make appropriate data-driven decisions. But it is easier to put a thumb on the scales when sampling is involved. It was bad stratification that led Crooked Hillary to believe she had it in the bag.

You Must Read This

Finally, a more famous than I data nerd tells the truth.

Instead of learning, I was spending 2-3 hours a day mindlessly scrolling through Facebook looking at cat videos and political hate speech. I was also posting family pictures of our adventures and eagerly awaiting the dopamine reward of getting likes from friends and family. My attention span was getting shorter and shorter by the day.

Now, lo those many decades ago, one of my graduate profs noted that you'll know you're working a profession when you spend 80% of your time thinking. And the greatest asset you acquire while a grad student is your library. Not so much these days.