Dr. Codd Was Right

R We to Delphi Yet?

2012-02-09T21:19:00.003-05:00

Well, thanks to R-bloggers, I found this Oracle page/blog.

The money quote:
"Oracle R Enterprise lifts this memory and computational constraint found in R today by executing requested R calculations on data in the database, using the database itself as the computational engine."

Tres interessant. Now, that sounds like they're saying this will be PL/R for Oracle. Not that I'm in the market for Oracle Enterprise, of course. If they make it available for the free as in beer version of Oracle; that I'd be interested in. (Miss Morris would be so bent by my ending a sentence a preposition with.)

Camel Passes Through Eye of Needle!

2012-02-08T15:51:00.000-05:00

The Brave New World I've been talking about makes high normal form database centric applications all the more appropriate. I know of at least two extant applications which implement this way. That said, the issue remains: what will the interTubes infrastructure look like in two years and further, and what sort of RDBMS architecture best exploits it? It's worth noting that there is precedent; Bill Gates got very rich by targeting software to not yet available hardware. That said, some comments on some comments.

[Chris]: [Robert asserts] threads are more efficient, processor-power-wise to spin up than processes, and therefore if you insist on a single-threaded multi-process model, you can't have sufficient performance to make your database system relevant.

No, that's not exactly what I said. The issue is lost cycles as one moves down to a single thread. It's more about the number of clients that can be supported "simultaneously". At one time I was also enamoured of thread count to execute queries in parallel; not so much now. For reporting generally, and BI/DW specifically, sure. But for OLTP, where my interest lies, not so much. There, client count is what matters. Clients run relatively discrete transactions on few rows; parallelism isn't much help for that sort of client. Yes, to some extent I've changed my mind.

Think of the issue this way: if it were true that scaling down threads to one per processor/core yielded a scale up in performance equivalently (only one of two threads yields twice the performance on that last thread), then the question gets more difficult. But threads and cores don't scale that way. Such reverse scaling could only happen if the core's frequency increased as the inverse of threads active. That just doesn't happen, you get a bin or two; that nasty frequency brick wall is the reason vendors have turned to multi-core and threads in the first place.

It's a matter of relative performance, not absolute. If the PG developers decide to not implement a threaded engine (some or all sub-systems), PG will inevitably be viewed as not worth the cost. I never said that this would happen tomorrow. Threading isn't a fad, but a hardware evolution. As the quote from Intel states, that's where they intend to go. Countering that is the massively parallel single thread/core machine, such as the Transputer or Thinking Machine or possibly an armada of ARM (which has been reported to be adding threaded cpu to its menu) cores on a chip. Whether a *nix could better manage such a machine (as opposed to a threaded one) is up in the air; experience from the 1980's when parallel machines were all the rage suggests not (Amdahl's Law put an end to them, and is still in effect). Some feel that the ARM armada will take down Intel. Way too early to bet on that. Couple that with the rapidly evolving web infrastructure, and we have a Brave New World where the interTubes is virtual RS-232 passed on virtual Cat-5; what's on the client end of the wire is just a terminal, prettily pixelated nevertheless only a terminal. That evolution matters to how a RDBMS (and its applications) is optimally architected.

Intel needed MicroSoft to bloat up its software in order to create demand for ever higher cycle count chips; it's the Wintel Monopoly for a reason. Now that frequency is near a brick wall, clients are devolving from PCs to wrist computers and such, the likes of Intel can't count on Office to justify new chips (unless Steve's boys and girls can parallelize Office; I don't see goods odds on that). Servers are pathologically multi-user, and therefore the logical vein to mine. That's what's happening and will continue.

[Chris}: Instead things like network connection/teardown, command parsing, and, well, actual work dwarf process/thread startup cycles by several orders of magnitude.

As I've said before, the main issue isn't process startup cycles versus thread, but all those cycles lost by running single threaded, which could be used for actual work. All that network stuff gets kicked to the curb in the Brave New World anyway; that's a large measure of the point. And, as above, the issue isn't just "startup cycles", but total processing cycles doing useful work. Throw away all but one thread, and you throw away nearly (threads - 1 + (bin bump))/threads of useful work. See this testing. That's the issue. They No Such Thing As A Free Lunch, and the cpu is about the most significant example.

Which is precisely why client multi-plexing is superior; with a threaded engine, you get twice (or more) active clients without the falderal. The comment on connection pooling reveals the same issue: auditability and transaction integrity if nothing else. You end up having to do all that manually, if at all. While connection pools were initially at the database end, they are now most commonly in the web-server, and motivated by the (un)connectedness of HTTP. Web server writers face the same issue as database server writers. When I was building servlet/applet thingees in the late 1990's and early 2000's, we had to build quick and dirty poolers ourselves. By about 2002, databases and web servers had integrated the support. Had to happen. Here's an example for Tomcat. Folks can, and do, argue about which piece of software should own the connection pool; I'll avoid that for now.

What's missing from the comments is discussion of my main point: with a threaded engine working on threaded (and likely with higher thread counts in the future) cpu's, high bandwidth connections, and client multi-plexing; one ends up with a RS-232 like environment. While I haven't explored it, there's an environment named Opa^[1] (Wikipedia has a short article), which is said to include web server and database server all in one. That's just begging for a fully connected infrastructure. Ideally, and I haven't explored enough to know as I type, there would be a single "connection", from the client's perspective; Opa takes care of managing the database/web server interaction in a way which preserves transaction semantics and client identity.

Unless you've been building database applications from before 1995 (or thereabouts), which is to say only webbie thingees, the efficiency and user friendliness of *nix/RDBMS/RS-232/terminal is a foreign concept; just as it still is for mainframe COBOL coders writing to 3270 terminals, same semantics (I'll grant that javascript is more capable than 3270 edit language, and Wikipedia does it again, with this write up). In the web age, AJAX was the first widely known (and possibly absolute first) attempt to get back to that future, then there is Comet. Now, we have WebSocket, discussed in earlier posts.

The Achilles heel of those *nix/RDBMS/RS-232/VT-X00 applications was the limit on the number of clients that could be accommodated since the client patch remained while the user kept a session. The emerging hardware/infrastructure that I'm discussing may/should remove that restriction. This also means that developers must heed the mantra, "the user doesn't determine transaction scope". With HTTP (un)connectedness, that isn't much of an issue.

With ever lighter clients, database centric (as opposed to coder's wet dream of client centric) applications become far more appropriate. Databases which can support higher client count on a given hardware platform (through threading, that is) will eventually clobber those that can't. That MySql, kind of OS, has threaded engines will make it difficult for other OS databases that don't. The question will not be PG versus Oracle/DB2/SS but PG versus MySql. While that's true today, the performance and configuration differences will only get greater over time.

While I surely agree that writing a threaded server, of any kind, is far more difficult than letting the O/S take care of the details (by writing a process based server), it's clear that the cpu makers are going to threads for performance. Think of this as a game of musical chairs.

[1] There is this internship listed on their website (typos their's):

Adding support for relational databases in Opa. The Web developping platform Opa has so far concentrated on its own database engine as well as the connection to existing NoSQL databases. The goal of this ambitious internship is to create the fundations of a framework for connecting relational databases to Opa (starting with MySQL). Different levels of integration will be experimented and successively made available (roughly: an untyped low-level API, a dynamically typed API, and a statically typed API generated at compile time). This internship requires excellent skills in functional programming (Ocaml, F#, Haskell, Scala) and a very good knowledge of the SQL language and its concepts.

Mr. Sandman, Bring Me a Dream

2012-02-06T16:53:00.000-05:00

AnandTech has just published its review of the Intel 520, driven by a customized (their firmware) Sandforce 2281. Lots of useful information. See, in particular, the very end of the review. The main point is whether this means the end of Intel SSD controllers. Busy day in SSD land.

I Do Thee Wed

2012-02-06T15:04:00.000-05:00

In unity, there is strength; or something like that (some say it's from Aesop, others from The Bible). Well, Rambus just scarfed up Unity Semiconductor. This is an interesting development, viewed from at least two angles: 1) Rambus has a history of attempting to proprietarize memory technology, and 2) if CMOx works, NAND vendors are in trouble.

According to the various write-ups, CMOx has been in development by Unity for nine years. I found it about five years ago, and the technology was just ready to go. Guess that part didn't work out. Yes, this endeavor has mentioned Unity a few times over the years. With an aggressive new owner, we could see a major course correction in storage. An interesting point: the Unity website still has a page devoted to a JV with Micron to manufacture PoC (my inference) devices on Micron fab. Wonder what's going to happen to that?

So far as point 1) goes, Rambus has been losing consistently over the last year or so in its attempts to enforce patents. Taking on the NAND vendors won't be a walk in the park. Micron, by the way, didn't settle and turn evidence against the other DRAM vendors; Samsung and others did. Since much of the patents in contention has just been ruled invalid, I wonder how they feel?

I Hate You, Diana Ross!

2012-02-05T13:33:00.002-05:00

Newt and Mitty have proved that Triage would be a waste of time and effort. I really wanted to build that thing, but SuperPAC money makes such an application irrelevant. SuperPACs are "prohibited" from co-ordinating with campaigns, and vice versa, thus rendering control impossible. Thanks a lot, Supremes.

Damn You, Damocles!

2012-02-03T16:24:00.000-05:00

The earlier thread related post engendered some comments that pooh-poohed the importance of threads going forward. While the points were rational, in that not all of the engines mentioned are threaded in all subsystems on all platforms (and, I never said they were), the fact remains that threading, in addition to coring (if that's a real word), is the architecture we'll live with so long as cpu's are silicon based. Lithography can only go so small, and power can only go so high, especially as feature size diminishes. I really don't think that notion is up for debate.

What can be debated is whether a single threaded engine, all subsystems that is, can keep up with threaded engines, some or all subsystems. The answer, in the limit, is no. Eventually, the fork in the road will split the performance paths just too widely to make even a "free" engine worth the money. Will that happen next year, or within five? Five seems more likely, except that we're working, whether it's recognized or not, in a different Moore world.

Moore's actual observation was that processors would double in productivity (measured in $$$) about every 24 months. He didn't predict anything about feature size, per se. He did take into account that the Law derived from feature shrinkage, but from a financial point of view. Intel is leading the pack in implementing hardware threading across the board, no question; although not in thread count. It is equally unquestionable that clients, more and more phone-ish devices, are inherently limited in power, given the physical size limits. Given Amdahl's Law and inherently limited power, the client isn't a solution to the database power situation. In other words, success going forward will rest with those who *do* go Back to the Future, but not by ignoring threading.

With bidirectional "small" clients (AJAX, WebSockets, et al) talking to servers a la *nix databases to VT-X00 terminals, it behooves us to look back at how those servers functioned, since they are the paradigm which leverages the current, and evolving, hardware. As interTubes communication behaves more and more like RS-232, there is a maximally efficient way to use this. Jamming lots o data on the line (and leaving logic hanging out there, too) isn't it. The network of tomorrow is much like the client/server in-a-box environment of the first instance. One might argue, tee hee, that The Cloud is uber-centralized data and that clients are/will be relatively passive devices; very much like a VT-220 I fondly remember. More to the point, with a client/server in-a-box paradigm (which can now be achieved with WebSockets and such) we have a patch of memory for the server, and another patch for the client, with the screen/terminal/smartphone/whatever on the other end of a wire. The screen only has responsibility for display and input. With proper NF and DRI, there's very little active code in this server resident client. Schweeeet. But, as the previous experience demonstrates (if you've actually been there), all edits are done, input box by input box, in real-time against the live datastore; the VT-X00 are referred to as "character mode" devices, as opposed to the "block mode" 3270 mainframe terminals and (regular) HTTP clients of today which re-connect to send a screen's worth of input. Schweeeeet. All the client does is paint the screen and ship the input back. Schweeeeeeeet. Coders?? We don't need no stinkin' (and do they ever) coders!!

Each of those client patches can run on a thread. Give up on threads, and you give up on half your capacity (or more); it's not just about how much faster a thread switch is than a context switch. Here is a schematic (about half way down the page) of the i7; a core (cpu) is about 10% of the real estate, or the transistor budget. In other words, while feature size has diminished over the years thus pushing up the transistor budget, not much if any of that largess has gone to instruction set implementation. Current Intel chips don't even implement X86 instructions in the hardware, at all (appears that X86 native execution began to disappear with the P4; thanks to Andrew Binstock). So, we get more RISC cores/threads. Here's an Intel thought: "[T]he multithreading capability is more efficient than adding more processing cores to a microprocessor."

When asked in the 1970's why his machines were so much faster, Cray said, "shorter wires". That remains true; certainly cpu designers are quite aggressive in the pursuit as they jam evermore features on nanometer long wires. Why software folks continue to think that longer wires are smarter is puzzling. Even if McNealy had been right that the network is the computer (for a local engineering net, not so much the interTubes), there's still the problem of reconciling all those nodes.

While Intel cpu's mostly are two threads/core, others have up to eight.

So, what margin of advantage is there to "turbo" mode at the thread level? Turns out, there is a bit of it, with the current Intel chips:
"When there are only two active threads, the Intel Core i7 will automatically ramp up two of its processing cores by one speed grade or bin (133 MHz). When only one thread is active, the Core i7 will ramp up the sole active processing core, not by one bin, but by two bins (266 MHz). However, if there are three or more active threads though, the Core i7 processor will not overclock any of its processing cores." Not a lot, compared to turbo at the core level. So, I still predict that ignoring thread support in the cpu is a losing proposition.

Here's a new posting testing threading in SQL Server. While not a runaway, in virtually all test cases, threading improved performance. Now, SQL Server in default mode, is a locker based engine, while Postgres is MVCC, so the engine mechanics might have an effect. Given that MVCC databases promise no conflict between readers and writers, one should expect that those sorts of databases should gain more from a threaded engine than a locker; threads stay active.

While it doesn't explicitly compare Oracle on a process OS versus a thread OS, this paper does discuss the advantages, and methods of use, of parallelism and threads. Oracle and Postgres are both MVCC, although the implementations are different. And, this is a Stanford paper discussing the Oracle Windows thread model. Making comparisons between Windows and linux (or any specific *nix) is difficult, in that what we want to know is whether a "good" thread implementation is better than a "good" process implementation, for an otherwise equivalent OS.

Cut to the chase. Since DB2 switched to a threaded engine from a process engine, there ought to be some evidence on the interTubes as to whether this was a Good Thing. This is a presentation by Serge (guru to DB2 weenies), see slides 6, 15, 16. And, this is the IBM justification. Another example: this is a paper by SAP for DB2/LUW on HP-UX (not the most popular *nix), note point 1. under "Operating System". IBM didn't move to a threaded model for yucks; they've been treating LUW as red-headed stepchild for so long that any significant expense would require significant bang for the buck. Finally, at long last, a recent presentation on the new threaded DB2. Note slide 17.

We need two additional pieces to get the most out of the system: multiplexed clients and fast swap to enable the clients to not stall. Here's IBM's take on multiplexing. Others can do likewise. The way to keep things running smoothly is Sacrificial Swap^©, by that one means using SSD as swap on the database and/or web server machine. Since SSD have determinate lifetimes, more cliff like End Times than HDD's decay slope, simply replace the swap drive periodically. Swap on SSD provides vastly better performance.

On a related note, this is an IBM (mostly) paper on bufferpools (what DB2 calls them) and SSD. Of note, this isn't using SSD as primary datastore, but only in support of bufferpools.

My conclusion: threaded models have the lead, and they're not likely to lose it.

Lies, Damn Lies, and the BLS [updated]

2012-02-03T15:18:00.004-05:00

That's Bureau of Labor Statistics. The headline over the column in my dead trees version of the NY Times reads: "Stagnant Job Growth is Expected in Report". Yet, the number this morning was glorious. How does that happen? Why did the numbers come in so much better than expected? Were the numbers figured? Let's see whether we can find out.

First: these numbers are estimates from sample surveys. The only population number related to employment is the weekly UI filing number. Everything else is an estimate from some kind of survey.

Second: as every fourth grader knows, a percent is just a decimal fraction with the point shoved over two places to the right. And every fourth grader knows that a percent goes up whenever the numerator goes up or the denominator goes down. Or both.

This is real time, in the sense that I haven't looked at the numbers. I'm going to go out on a limb and say that, at least, the denominator went down. That number is the estimate of the total labor force. That number has been declining through 2011 - 2012 (here is the table and this is the almost raw data), the difference being seasonal adjustment. Seasonal adjustment of economic data is still controversial.

Now, have a look at the next table. What we see is that the number not in the labor force is up, as is the number working multiple jobs.

Table A-15 is where the doubters gather. U-6 is the number cited by both Left Wing and Right Wing sympathizers as justification for either blaming all those lazy poor people who can't seem to stay put in Mitt's safety net, or blaming Mitt and Friends for slicing away at the safety net. The not seasonally adjusted is the number which matters. And it's up a tad from December to January.

I rest my case.

[update]
Today's Times has a longer piece, complete with interviews, which I don't get to do from my drafty Frost Belt garret. As well, I see that I wasn't sufficiently explicit about seasonal adjustment. What I expect happened is that the adjusted numbers (the ones nearly always quoted) overstate the level of employment. One of the reasons to adjust the numbers is truly seasonal: weather is lousy here in the Frost Belt in January, and the purpose of adjustment is to make a level playing field, month to month. Since the Frost Belt last month was more like October (the January "season" was largely absent), the quoted number got a double dose. I don't expect to see it continue, alas.

Don't Thread On Me

2012-01-25T15:57:00.002-05:00

This thread was mentioned in a performance sub-group posting. Give it a read.

Back? It means, so far as I can see, that PG is toast. It will fall down to being the cheap and dirty alternative to MySql, which even has, at least two, multi-threaded engines. DB2 switched it's *nix engine to threads from processes with release 9.5. Oracle claims it for releases going back to 7 (I haven't tried to determine which parts or applications; Larry has bought so many tchochtkes over the years...). SQL Server is threaded.

Given that cpu's are breeding threads faster than cores, PG will fall into irrelevance. Too bad, it's kind of a nice database.

Can't Find My Way Home

2012-01-24T19:44:00.000-05:00

I have a second fifteen seconds of fame over on Simple Talk, here. In the course of the commentary, I was asked why Google is a toy application, along with Facebook and Twitter. What follows is a much longer version of what I posted.

Look closely at Google, and all you've got left is a massive advert agency. Adverts are important to life? Advert engines advance civilization? Of course not. We Yanks are fascinated with "Mad Men" in a tee-hee sort of way, yet base more of our system building, and one might assert our economy, on selling adverts. Rather than making things for ourselves, we make insubstantial playthings. Psychologists call this behaviour cognitive dissonance.

Decisions, and the actions which follow from them, are manifestations of value judgments. To the extent that our systems building efforts concentrate on toys rather than meaningful (for some definition of that) systems, we've revealed what it is we value. Pixelated toys. I'd love to hear an argument that Facebook and Twitter etc. represent advancement of civilization. And, no, they're not improvements in communication.

The share price just got spanked because they've not kept up the advert machinery, consuming all advertising on the planet. If they manage that, then what?

Facebook just arbitrarily skunked its users.

These are toy applications since their only point is to support play. They encourage adolescent tribalism, which the planet doesn't need in excess. The vehicle is fancier than passing notes in class back when your grandfather was a kid, but it's the same behaviour only prettier to do.

For a more formal, and literate, take on the issue, read up Nick Carr. He also has an infrequently posted blog.

From a purely systems point of view, Google is a re-hash of code/file systems from the 1960's. Size isn't distinguishing. Read the Wikipedia write up on Map/Reduce.

Wikipedia is a more important piece of software.

What Language Do You Spook?

2012-01-16T11:23:00.003-05:00

Drew Conway is my kind of guy: a spook. Now, before any of the uber-PC folk get their backs up, an alternate definition of the word: colloquial for intelligence operative, origination unknown. I don't always put it on my resume, but I spent some time in the 1980's in Jack Anderson's shop. Officially, an intern; I did get paid some money. That's likely because I brought in a two column story (a story getting more than one was highly unusual) on the CIA running guns from Argentina (if memory serves) to Ghana. Corky Johnson worked on it with me. This was the time of Iran-Contra and the like. The story falls into the "like" category. We could never figure out why the Post didn't follow-up. I still have the galleys.

So, I was surfing through R-bloggers looking for background using R with Powerpoint; I know, yucky mucky. But I'm supposed to chat up some folks about that today, if they bother to call. In the search results is a long piece by Mr. Conway on proper data visualization. Comes out against pie charts. Good for him. I highly recommend it; he manages a bit of humor, when offering that a graph should have a 140 character footnote. Such a twit!!

Don't Be a Dupe

2012-01-13T11:02:00.000-05:00

The post on the NoSql dust up from Postgres led to me adding a comment on one of the posts, and now Chris Travers has commented on my post (ring around a rosy?). You can find his comment here(scroll to the bottom).

Rather than waste the opportunity in a comment, and because I truly believe it's worthwhile on it's own, herewith.

I do not believe that there is any use case for NoSql, on technical grounds. That is to say, there are no use cases where NoSql provides a superior solution to the RM/SQL (henceforth, just SQL). I do believe that NoSql is chosen 99.44% of the time just because it implements Programmers' Perpetual Employment Paradise. No database, all COBOL forever; well, java or C++/#.

The reason SQL is always superior is because:
- it implements the smallest real data footprint to logical data footprint
- it enforces logical data integrity

One of the hot new technologies, over the last few years, has been data deduplication. Dedupe is just an ad hoc approach to 5NF, with none of the benefits of SQL; get rid of the redundancies. But there's no structure or control. One of the objections to dedupe tech is the fragility of it. How much of a glitch will cause data corruption, and so on.

Now, for non-transactional datastores with no logical structure, then SQL databases probably don't matter much. If one considers that a use case, then OK by me. I wouldn't spend my time with such an application, that's for sure.

Color Me Happy

2012-01-07T17:43:00.000-05:00

I posted the following comment on Offensive Politics' post on its graphic presentation of Iowa results:

This post got referenced on Revolutions Analytics, and garnered a comment that the (default) HCL scale used by ggplot is faulty. I commented that the closeness of the vote is perfectly reflected in the color scale; the confusion is correct.

So, off I went to Wickham's ggplot2 book, chapter 6 on scales. scale_full_gradient2() and library(vcd) provide ways to impose non-linear color scales. At first, a good thing. But I wonder. There's been a good deal of discussion, here and there, with regard to spinning data. It would seem that using a non-linear color scale is just as bad as fiddling with axis scales.

What do you think?

I mentioned this post earlier, which was then followed by its linking from Revolution Analytics. One of the prime directives of professional statistics, as distinct from polemic number spewing, is that form shouldn't affect perception. In other words, don't fiddle the picture to plant an idea not supported by the data. In this case, the original commenter complained that the color scale made it difficult to distinguish among the counties' vote winners. My view, so far at least, is that the defaults used by R/ggplot for this implementation rightly reflect the similarity between Romney and Santorum.

The default color scale is, essentially, linear. The alternatives mentioned are decidedly non-linear. Now, the notion of linearity with respect to color scales may be problematic, but the ggplot defaults do implement a notion of linearity, in that two opposite colors are blended along a linear mixing.

Here's examples from Wickham's ggplot2 text. I suppose it was uploaded legally. I hope.

This is the various color scalings:

As can be seen, the color gradient in the right two alternatives is far more extreme. I'm still not convinced that one should do such fiddling.

You Rook Mahvelous

2012-01-06T09:24:00.002-05:00

A couple of other pieces are in the works, but this just announced from OCZ and Marvell. I didn't see any mention of exclusivity in the PRs, but AnandTech says so and mentions what caught my eye when reading the PR: it's PCIe on x1 channel. What's up with that? There's that line from "The King and I" (film version, at least), that goes "A bee flits from flower to flower...". OCZ is sure doing that with regard to controller.

Last I knew, PCIe SSD were controller-less (in the silicon sense), off-loading control to code run on the cpu. Depending on who one follows, this is either a good thing or a bad thing. Said code is written by the SSD vendor (or bought in).

Be Kind to Idiots [updated]

2012-01-03T15:03:00.002-05:00

In the olden days, when I was a child, good parents instilled good values in their children. One was to not make fun of idiots; being an idiot was not the idiot's fault. Or, so the thought was. We know different now. Many idiots are so because they've chosen the path of ignorance, devolving further into idiocy. We see this in politics and IT, specifically the nexus with RDBMS.

I've been following the LedgerSMB postings (those that appear on the Postgres site, anyway), because the structure is similar to that taken by xTuple. Both, to greater or lesser extent, espouse integrity in the database, as opposed to client application code. Neither prevents editing in client screens, but both assert that the constraints, of record, are in the database; and can be duplicated in the client.

One of the fascinating aspects of this approach will be, soon, the picture when the likes of WebSocket is routinely available. Data centralization is always the future. When the IBM PC was first released, it was as a low powered programming work station; the "user" would use it to write "non-professional" code to suit his (not many hers at the time) needs. Scientists and engineers were the market, and IBM estimated 2,500 per year would sell. Then, along came 1-2-3, and the computer appliance was born; only now have I finally been vindicated in making that description. Soon, Netware showed up, to connect all those independent PCs to printers and file servers. The kodder kiddies really are convinced that The Cloud is something new. Some of them assert that it's really different this time; that this time it's about the cheapest hardware possible appearing like manna from heaven on demand. Foolishness. They'll find out, and another few hundred billion dollars will be wasted by the Fortune X00 chasing this as was when they chased J2EE applications.

From the very beginning, Dr. Codd had to cram the RM down the throats of heel dragging coders, who willfully ignore the fact that logic in code is just data compares made complicated. Do the logic with data where it lives. But, NO, that threatens the rice bowl of all them coders.

So, Chris Travers (of, and perhaps all of, LedgerSMB) has been posting about the application structure.

He published this post recently.
This led to the following post.
And that led to a supportive post from Joe Abbate.

Chris was too kind. Just as Ron Paul shouldn't be taken seriously, neither should Tony Marston. Neither has a clue, only zealotry. Joe, to his credit, took up this quote: "The database has always been a dumb data store, with all the business logic held separately within the application." His response was also too kind.

Marston has clearly never read, or if he did hasn't a clue what he read, Codd or Date. His statement is only true of coders back to AutoCoder and COBOL, not databases. As with the COBOL/VSAM/IMS folks that gave Dr. Codd the finger, Marston is just another guy who wants to keep writing lots of code, no matter how dumb that is. Prior to Dr. Codd's RM/RDBMS, there was IDMS and IMS; network and hierarchical database respectively. Both implemented logic in the database structure. Moreover, in 1968/9, IBM released CICS, likely the longest lived TPM (transaction processing monitor). A TPM does the read/write control for external code, largely COBOL with the IBM mainframes of the time. So, Marston is wrong in toto. Marston embodies the latter day Goths, who wish to take us back to the Dark Ages of their crude ancestors. A lot like Ron Paul, come to think of it.

The entire purpose of TPMs, pre-relational databases (IDMS, IMS, PICK), and RBDMS is to put data control with the data where it belongs. That is a threat to the horde of coders. Too bad.

[update]
For the record, I went back to see how the thread had gone, and couldn't keep my mouth shut. You have been warned.

Back to the Future

2012-01-01T15:18:00.002-05:00

As those who've been following this adventure since the beginning must know, one part of the journey down The Yellow Brick Road involves the nature of application execution. I've been saying for many years, since long before I ever put fingers to blog, that the inevitable result of "progress" in application design is to replicate, albeit with pretty pixels, the world of *nix databases and the VT-100. The reason is simple: this paradigm provides the greatest control with the greatest ease of use.

The continuing problem with actually getting there is the disconnected nature of http. "Server push", as it's often called, hasn't been functional. So, imagine my surprise to see this Eckel post on Artima. Here's the Wikipedia entry, too. The upshot of WebSocket is a terminal on a long wire to the database. The winners in this world will be those who recognize that the high normal form database, with small bits of data per UI screen, will be the winners. This paradigm supports centrally edited data, to any screen. In other words, client agnostic.

In due time, perhaps with my consulting advice, organizations will realize what WebSocket means. There's that old joke: "Doctor, it hurts when I do this." "Don't do that." The easiest approach to dealing with Big Data is to not build stupid flatfile datastores. *nix databases, of high normal form, on multi-core SSD machines allow data to be sent to clients in whatever form needed. Smaller is better, of course, but massive joined rows can be sent. Not that one would want to, of course.

Head In The Cloud (or Somewhere)

2011-12-24T14:18:00.000-05:00

Came across this posting whilst surfing, with the following:

"For example, one large package deliverer sees a 400% increase in its network traffic, database needs, and computing power over a single 45 day window. Utilizing a service such as Azure will allow them to pay for that excess capacity only in the 45 days it is needed, not the other 320 days a year when it is not utilized, but must be maintained, upgraded, and licensed."

Why folks persist in believing that there is a Free Lunch up in The Cloud??? The notion that seasonal/repetitive demand spikes for IT resources are uniformly distributed over a time period (day/week/year) is just silly. Yes, as a loss leader, a Cloud Vendor may choose to not add a Load Factor Penalty. In the beginning. But, that's not sustainable, because demand spikes aren't uniformly distributed. Cloud vendors will make the clients pay for all that idle storage and cpu (and a nice profit on the idle riches), you betcha. That package deliverer experienced the same spike as all of the retail chain. And all of the energy vendors. And so on. It's Econ. 101, folks.

Icarus

2011-12-21T13:07:00.000-05:00

The sky is falling. Oracle reported down, a bit, but, in particular, didn't report above expectations. Larry has been sly for a long time, in setting guidance low enough that bettering it is a piece of cake. Not this time. As I type, it's down 15%, and news is that much of the tech sector is getting the flu.

The knee jerk reaction: mortgage the farm and buy Oracle stock. IIIIIIIIIIII'm not so sure this time. Here's why.

Oracle didn't get quite the *new* software sales and *new* hardware sales. The latter is, by all accounts, due to customers waiting on the new machines during the quarter. The former is more speculative. The reports are vague. My take: given the aggressive pricing of Oracle RDMBS, ditto for MySql (yes, it's GPL, but Oracle blasted its support prices into the sky in the past year), and putting the screws to java adopters; folks are looking for a safer port.

On the RDBMS side, Postgres gets ever closer to Oracle, if you're not a Fortune X00 company (and even if you are, and building apps off the mission critical axis). Mainstream pundits are crying that "Da Cloud, boss, Da Cloud" is putting Oracle in an untenable position. It is said that cloud providers use dirt cheap components, soft and hard, and Oracle's RDBMS and Sun-ish machines are just too expensive Up There. As if being cheap were the best way to make money!? "Cheap goods sold dear" is an aphorism that's been around forever. The Cloud is shaping up that way, and if so, I'd avoid Fortune X00 companies that chose to put *my data* Up There. In this nascent era of Cloud, too many stories of wandering data to suit me.

What's really stupid about Larry's ploy: the Oracle RDBMS is built on an engine (the piece that actually does all the inserting and updating) which uses Multiversion Concurrency Control (MVCC, as it is known) which is better suited to the asynchronous nature of the Web/Cloud than the locker paradigm that most other (notably, not Postgres) RDBMS have been using for decades. They've been backing in, so to speak, MVCC support recently, but none is a true MVCC database. In other words, Larry has the proper mousetrap for the setting, but has managed to offend his customers. But, that's Larry's way.

Reports say that Oracle claims the shortfall is due to last minute non-signings. If so, then this is an aberrant glitch. Given that Fortune X00 companies are sitting on, by some accounts, more than $3 trillion, there's no macro reason to not buy new IT. Unless you're a Fat Man yearning for Famine.

A Warren-ted Search

2011-12-20T15:07:00.002-05:00

One of the points "for further research" as I used to say when I was an academic, in the Triage exercise was using social media to measure outcomes. R has a library, twitteR, (yes, R folks tend to capitalize the letter at every opportunity), which retrieves some data. I was at first disinterested, since I don't have a twitter account. Thankfully, twits can be gotten without being a twitterer. Since Elizabeth Warren's campaign is just over the border, and sort of important in the grand scheme of things, I've been exploring.

Here's the entirety of the R code (as seen in an Rstudio session) needed to return the twits (1,500 is the max, which will prove troublesome when the battle is fully engaged):

> library(twitteR)
> warrenTweets <- searchTwitter('@elizabethwarren', n = 1500)
> length(warrenTweets)
[1] 9
> warren.Text <- laply(warrenTweets, function(t) t$getText())
> head(warren.Text, 10)
[1] "@elizabethwarren i hope you win agianst sen scott brown. the 99% r with u"
[2] "@elizabethwarren More $$$ coming your way!"
[3] "#HR3505 PAGING: @ElizabethWarren Help us!!!!"
[4] "@elizabethwarren - not to worry, the only job Karl Rove ever got somebody was George W. Bush. and look how that turned out."
[5] "RT @SenatorBuono: What an amazing turnout 4 a superstar. @elizabethwarren"
[6] "HELLO @ElizabethWarren ! PLEASE RUN as a 3rd party or Ind. FOR POTUS2012. Dems just threwSENIORS underthebus for the working tax cut! EXdem"
[7] "@chucktodd We hope 2011 will be remembered for something a LOT closer to home. #ows #OccupyWallStreet @ElizabethWarren #WARREN/PELOSI-2016"
[8] "RT @SenatorBuono: What an amazing turnout 4 a superstar. @elizabethwarren"
[9] "What an amazing turnout 4 a superstar. @elizabethwarren"

The lines starting with > is the R code. The lines starting with [x] are the output. Here we have 9 twits.

Now, what do we do with the text? For that, I'll send you off to this presentation which came up in my R/twitter search (and is the source of what you've seen here), conducted in Boston. Missed it, dang. With slide 11, is the explanation of how one might parse the twits looking for positive/negative response. By the way, even if you're not the least bit interested in such nonsense, visit slide 29.

As I mentioned in Triage and follow-ups, getting the outcomes data is the largest piece of the work. Simply being able to "guarantee" the accuracy of twitter (or any other uncontrolled source) data, given the restriction on returned twits and such, will require some level of data sophistication; which your average Apparatchik likely doesn't care about. The goal, I'll mention again, isn't to emulate Chris Farley's Matt Foley and pump up a candidate no matter what the data say, but to find the candidate out of many most likely to win given some help. Whether Triage would be useful to a single candidate; well, that depends on the inner strength of the candidate.

Ya Can't Get Theah From Heah

2011-12-19T16:38:00.000-05:00

I'm working my way through Wilkinson's graphics book, and right there, on page 404 (could there possibly be a better page for this?) is this:
"The price paid for this efficiency is the loss of flexibility. We cannot examine relations not represented in the hierarchies. Navigation through OLAPs is quite difficult, which is why so many graphical models for representing them have been proposed."

Now, this is a graphics coding guy; he created SYSTAT. He's not, so far as I know, an RM guy. But he gets it.

Lies, Damn Lies, and Statistics

2011-12-16T14:55:00.000-05:00

The Other-R folks have posted a recent entry which references an EMC paper (here if you follow the breadcrumbs) on the state of Data Analysis and Business Intelligence, from the point of view of practitioners. The blog post makes some useful points, but misses some.

I'm referring to the graph in the original post, which is on page 3 (in my screen) of the EMC paper.

What this graph tells me, mostly, is that BI folks are still tied to MS, Excel in particular. Data analysts, not so much; although they'll be tied to corporate policy in such venues.

A few words about each.

Data Storage: SQL Server is tops, which means that most folks, in both camps, are tied to corporate group level machines, not the Big Iron. It's been that way for decades; the analysts have to extract from the Big Iron, and crunch on their own PCs. The categories Other SQL, Netzza, and Greenplum leave room for the Triage with PL/R approach, since the latter are explicitly Postgres and Other SQL is likely as much Postgres as MySql (yuck!). The category is, possibly, misleading if one jumps to the conclusion that companies are MS centric with their data.

Data Management: No real surprise here. Excel is the tool of choice. Way back when I was teaching PC software courses, 1-2-3 was the spreadsheet of choice and all data went through it, and Excel inherited the mindset that a spreadsheet was sophisticated analysis. It is a bit unnerving to realize that so much of what corporations decide is supported by such drek. Note: the BI folks, in the past executive assistants and "secretaries", still use spreadsheets a lot. The Data folks, the other way round. There is small comfort in that. The presence of BASH (or Korn or ...) and AWK (Python and Perl too, but not quite so much; each has bespoke language I/O in the mix) is interesting, in that it means that a fair amount of data is clear text ASCII files. Think about that for a second.

Data Analysis: Clearly, the Data folks use stat packs while the BI folks mostly don't. SAS and SPSS and Stata leading says that the EMC client base is largely large corporate, which isn't a surprise. What is a surprise is the absence of Excel. On the other hand, in the original paper is this (next by each the graph): "While most BI professionals do their *analysis* and data processing in Excel, data science professionals are using SQL, advanced statistical packages...", which corresponds to my experience (emphasis mine).

Data Visualization: The absence of R is suspect, as any R user would understand.

And, finally, this has nothing to do with Big Data, in any case. BD is just another attempt to money-spin by those with an agenda. Janert, in his book "Data Analysis...", makes clear that BD isn't worth the trouble (my inference). The point being that population data, which is what BD offers, is just descriptive stats, and smart data folks aren't interested in descriptive stats. Sports fans, well yeah.

Jumpin Jack Flash [UPDATE]

2011-12-13T10:45:00.001-05:00

In today's posting of the Momentus XT review, Anand (nearly) ends it with this:
"Longer term it's unclear to me whether hybrid drives like the Momentus XT will fill the gap left by SSDs or if software based caching technologies combined with NAND on motherboards will be the preferred route."

Oddly, I wondered much the same a few years back when Sun (pre-Oracle) announced its flash "appliance". That, of course, was for near-mainframe level servers, but was an early assault on the flash-as-HDD approach to using flash. There've been discussions about, particularly in the context of PCIe (Fusion-io, of course) "drives", whether there need be this SSD-as-HDD approach. More folks are talking about storage now as just flash, pretty much, directly wired to the cpu and/or memory manager. As Linus said all those years ago, it will change how file systems are built; if we bother with them at all.

[UPDATE]
Not to mention, this tickled a lower brain stem memory: winFS. Go spend some time with the WikiPedia article. Mayhaps SSD will be the crutch needed?

Parallax View

2011-12-11T14:51:00.000-05:00

"The Parallax View" was a fun movie, especially for those with a conspiracy slant on the world. One might even see it as harbinger of "Robocop". Well, it turns out that there is at least one other take on scatterplot matrix. In the context of the authors' data, it makes some sense.

I tried it with the Triage database table, but am not so convinced it helps for the discrete data proposed. Have a look, and see what you think:

CREATE OR REPLACE FUNCTION "public"."pairs_graph" () RETURNS text AS
$BODY$
library(ggplot2)
library(GGally)
X11(display=':5');
events <- pg.spi.exec ("select period, event, amount, choice, guncontrol, outcome from public.events where candidate = 'Doe' ");
png('ggpairs_graph.png');
events$gcfactor <- as.factor(events$guncontrol)
p <- ggpairs(events, columns=c("period", "amount", "choice", "gcfactor"),
    diag=list(continuous="density",   discrete="bar"), axisLabels="show")
print(p)
dev.off();
print('done');
$BODY$
LANGUAGE 'plr'

And here's the graph:

Smaller, But Not Worse

2011-12-06T09:59:00.002-05:00

Just when the future looks most bleak, comes this news, today. IMFT announces that 20nm flash is in production, and that (look at the pretty picture) 128GB will soon be at your finger tip. The industry may just turn around and do the faster tape waltz. On the other hand, with erase cycles *not* deteriorating, 5NF for large systems is in reach. Perhaps, just may be, The Yellow Brick Road isn't the Road to Perdition (Jude was creepy).

What's The Difference

2011-11-29T19:03:00.002-05:00

To continue with the Triage project, I've spent a day or two with more graphics texts (about which I'll be musing anon), and getting more familiar with the mapping scenarios.

Separate from the scatterplot matrix data shown in Triage, which would be used to measure the micro components of a campaign, is the question of displaying national trend, twixt Us-uns and Them-uns. For that one turns to map graphics, which is a whole other world. Still in R, mind, but not statistical in nature.

What I have recently found is this site, which replicates a US map with 2004 election results. Now, our Apparatchiks won't be downloading zip files from outside sources, of course. On the other hand, the files make for a perfect dive board for the PoC. Load them into PG, swapping Republican for Bush and Democrat for Kerry and Other for Nader (that's not much of a stretch!). Just for completeness, I'd found much earlier (but can't find that I'd cited), this map exercise, but as of now, the author has been too embarrassed to post the R that does it. While only some form of income data (not specified), it is a follow-on (linked to) to an election stream map set, also not supplied with the R that made it. Nevertheless, one can conclude that with enough time, this is a task suited to R. As mentioned in an earlier post, the animation bits are likely via googleVis.

I'll be using his data, since it provides a basis and I don't have to concoct some, though not the R he used (still using the stock R from Wickham). It's not clear how the numbers were derived.

What is really useful about the 2004 map posting is the data source: a county level count. Get these into a PG table, and we have a surrogate for data which our Apparatchiks would have, and which we can further expand with relatively simple SQL; just to see how a map would change. The notion for this part of the Triage effort is to measure the effect of national campaign spending, post some event/ad/debate/foo, at the POTUS/party level; a RNC/DNC (or 501/527/foo group) view of the country.

Here's the new PG table where we load:

CREATE TABLE public.election (
state varchar(25) NULL,
county varchar(25) NULL,
tot_precincts int4 NULL,
precincts_reporting int4 NULL,
republican int4 NULL,
democrat int4 NULL,
other int4 NULL,
constraint pk_election unique(state, county)
)
WITHOUT OIDS
TABLESPACE pg_default

And we get it loaded thus (concated from the state/county files in the zip):

copy public.election from '/databases/rawdata/2004election/output.txt' using delimiters ';' csv header

Note that column names are underscored, rather than camelCase, since PG forces quoting to use anything in the database if there are Caps in names. Yuck.

And here's the PG + PL/R (I've left it as is; comment/uncomment to generate each of the maps, this is the difference map, shown last. The first set are for the two event maps, while the other is for the diff map):

CREATE OR REPLACE FUNCTION "public"."us_graph" () RETURNS text AS
$BODY$
X11(display=':5');
pdf('US_graph_diff.pdf');
library(maps)
library(plyr)
library(proto)
library(reshape)
library(grid)
library(ggplot2)
library(mapproj)
states <- map_data("state")
#elections <- pg.spi.exec ('select state, sum(republican) as "Republican", sum(democrat) as "Democrat" from election where event_number = 2 group by state order by state');
elections <- pg.spi.exec ('SELECT a.state, sum(a.republican - (SELECT b.republican FROM election b WHERE b.event_number = a.event_number - 1 and a.state = b.state and a.county = b.county)) as Republican FROM election a where a.event_number = 2 group by a.state ORDER BY a.state ')
elections$state <- tolower(elections$state)
elections$republican <- elections$republican/10000
choro <- merge(states, elections, sort = FALSE, by.x = "region", by.y = "state")
choro <- choro[order(choro$order), ]
#p <- qplot(long, lat, data = choro, group = group, fill = Republican / Democrat, geom="polygon", asp=.6)
p <- qplot(long, lat, data = choro, group = group, fill = republican, geom="polygon", asp=.6, main = "Poll Shift", xlab = "", ylab = "")
p + labs(y = "", x = "")
p + opts(panel.grid.major=theme_blank(), panel.grid.minor=theme_blank(), panel.background=theme_blank(), axis.ticks=theme_blank())
p + scale_x_continuous("")
p + scale_y_continuous("") + coord_map()
p + opts(axis.text.x = theme_blank(),axis.text.y = theme_blank(), axis.title.x = theme_blank(), axis.title.y = theme_blank(), axis.tick.length = unit(0, "cm"), axis.ticks.margin = unit(0, "cm"))
p + scale_fill_gradient(limits = c(0, 90))
print(p)
dev.off();
print('done');
$BODY$
LANGUAGE 'plr'

All that spinach for the library calls got eliminated by making an .Rprofile in postgres user's home with the following line:

.libPaths("/home/postgres/R/x86_64-unknown-linux-gnu-library/2.14/")

You could also call out the libraries explicitly; both ways work. The additional spinach is various directions to eliminate the lat/long grid on the maps. None work!

Here's the Event 1 map:

Now, let's update the table to include an event_number (easier than using a date, anyway) and an event_type. That way, we can generate maps in sequence, but also note what sort of event just/last happened. We could also generate maps sequences for only certain sorts of events (they'd be in a check constraint).

So, let's make some new data:

insert into election (select state, county, tot_precincts, precincts_reporting, republican * .8, democrat * 1.2, other, 2, 'foo' from election where event_number = 1);

We wouldn't get such dramatic shifts (modulo Swift Boats) in the real world, but this is PoC territory.

This yields a new Event 2 map:

I'm still grappling with my main wish list item: showing the changes in the colors. As it stands, each map takes the full gamut, leaving the legend to display the shifts; doesn't do that all that well. Viewed another way, why not show the delta of polling strength (vote displays are a bit late, after all)? We can do that with a single map. How to get the data out of the election table? For that a correlated subquery is sufficient. It's that big SQL statement.

Here's what the delta map looks like:

What we see is the shift, in absolute, not relative, numbers. So Texas looks to be more Democrat from Event 1 to Event 2 just because it started with more votes; same with California.

Getting rid of the lat/long grid is still a problem, but then, this is a free PoC. Cheap at half the price.

The Red and the Blue

2011-11-15T15:43:00.001-05:00

I was going to build a US map (using R facilities) showing net federal funds at the state level, but found there are a colossal number of these already. No need to demonstrate that yet again. The point would be to demonstrate doing so within the RDBMS, following in the Triage piece's footsteps. I'll just show the code to generate a US map, as shown by Wickham's book.

CREATE OR REPLACE FUNCTION "public"."test_graph" () RETURNS text AS
$BODY$
X11(display=':5');
pdf('test_graph.pdf');
library(maps, lib.loc="/home/postgres/R/x86_64-unknown-linux-gnu-library/2.14/")
library(plyr, lib.loc="/home/postgres/R/x86_64-unknown-linux-gnu-library/2.14/")
library(proto, lib.loc="/home/postgres/R/x86_64-unknown-linux-gnu-library/2.14/")
library(reshape, lib.loc="/home/postgres/R/x86_64-unknown-linux-gnu-library/2.14/")
library(grid, lib.loc="/home/postgres/R/x86_64-unknown-linux-gnu-library/2.13/")
library(ggplot2, lib.loc="/home/postgres/R/x86_64-unknown-linux-gnu-library/2.14/")
states <- map_data("state") arrests <- USArrests

names(arrests) <- tolower(names(arrests))

arrests$region <- tolower(rownames(USArrests))

choro <- merge(states, arrests, by = "region")

choro <- choro[order(choro$order), ]

print(qplot(long, lat, data = choro, group = group, fill = assault, geom = "polygon", asp = .6) + borders("state", size = .5))

dev.off();

print('done');

$BODY$

LANGUAGE 'plr'

Rather more text than the Triage demonstration. R is built on a multi-user model, but is normally used as a standalone application on a PC. And then, there's the *nix issue. The upshot is that nothing need be done to use "base" modules, and those include the scatterplot matrix in the Triage piece. As mentioned in the piece, R supports (at least) two other graphics engines: lattice and ggplot2. Lattice is an extension of base graphics, while ggplot2 is an implementation of a grammar based graphics engine. This Grammar of Graphics is documented (but not a code base) in Wilkinson's book, at 712 pages and no code, we'll see (just Amazoned it)!

This map was created with ggplot2 functions, although no database data is used. It is necessary to call out each package/library explicitly, as well; PL/R doesn't know to load dependent packages, alas. In the context of the Triage piece, the application would show the net position of the party's candidates by state, along a Blue/Red vector. Just so happens that the R installation includes some state level data, which Wickham uses illustratively. One might extrapolate that Red States are more violent than Blue States, on the whole. Not that I'm making such an extrapolation, of course.

Loading of non-base libraries can be done in one of two ways: if the R engine library directory has global write permission (not normally so under *nix) then any package (which is then called a library in use, yeah, I know) loaded by any user goes to the directory and can be referred to directly by PL/R; on the other hand in usual installs, each user has packages installed to a local directory. Since postgres (the engine) runs as postgres the user, the packages need to be installed by postgres (the user) from an R session. In a corporate (and political campaigns are very much so) environment, standards and conventions would need to be established.

Ideally, what I'd want, following on the thesis of the Triage piece, is a clickable map (states), but that gets into non-rectangular html buttons (Google maps, I'd wager); not a topic I'm conversant with, yet. Whether it would make sense to generate the map in R with the cruft needed to implement the button logic is another puzzle. I think not, but not sure; R doesn't impress me as a strong string manipulation language. Ideally, then, the map would not only be generated by R, but each state would be a button, which would call a second R function in postgres to show the county/municipal/zip map. Could be a bit of work, but your candidates are worth it.

Here's the picture (this is a png, since Blogger won't chew pdf):

Honesty in Government

2011-11-09T10:27:00.001-05:00

[UPDATE] -- copied Sales the first time, same issue.

As I transition into data scientist, which means re-adding my stats mojo to my RDBMS mojo (not replacing the latter with the former, by the way), I've come across more than a few postings and writings in the statosphere about truth in data. The writing is always by data professionals (not lobbyists and the like, near as I can tell), and the point is always that the data is truth. By truth one means the most accurate picture of the real world, unadorned by propaganda.

Today's Federal data dump includes September wholesale inventories. They were down .1%. Here's the quote: "..were $462.0 billion at the end of September, down 0.1 percent (+/-0.2%)* from the revised August level." What's the starry thingee, one might ask? Well, it's the link to a footnote.

Here's the footnote:
"* The 90 percent confidence interval includes zero. The Census Bureau does not have sufficient statistical evidence to conclude that the actual change is different from zero."

Two points to note about the footnote: 1) the CI is 90% level, which is very generous and 2) it spans 0, which means what the note says. I wonder how many of the reports about the report will bother to tell us about that.

Here's the link to the original; click the link for Excel or PDF.

Write Me a Song So Lovely

2011-11-07T12:52:00.000-05:00

It's mostly considered declasse' to simply refer to somebody else's post on some subject. And I don't do so very often, but this post on writeable CTEs is too much to pass up, not least because I've yet to move to 9.1, and won't likely until it hits 9.1.3 (currently at 9.1.1). What makes the post of interest: Brown states that PG is alone with wCTE, and some quality time with searching supports that. My interest rests on the motivation for this endeavor: put, and keep, the logic in the datastore and let the client pretty-pretty the screens as it wishes. Writeable CTEs take this notion yet another step forward.

Not that I'd choose a joined table with a calculated, by code, column. But that's another story. I'd be more inclined to implement an inventory/order line update, since that fits with the canonical example of join. For example, if your customers are mixed with regard to price changes, some will get updated prices on unfilled order lines, the rest not. In that circumstance, you'd want to update some order lines when the inventory (or price break) table is updated. That sort of thing. 9.1.3 shouldn't be too far away.

Good on 'ya, as Texans say.

From Sea to Shining Sea

2011-10-25T12:03:00.001-04:00

As follow up, or update, to the Triage piece, I offer up this post from an R-blogger. As it stands, there's no code (the author claims ugliness), but does applaud ggplot2. The latter I expected, in that Wickham's book has a section (5.7) on using maps, but not much detail.

Of more interest, is the data source, shown as CCES on the plots. Turns out that this is CCES. While not real time data, as Sparks demonstrates, R and ggplot2 can show both categorical and discrete variable impact over a map. For the Triage project, one would need internal real-time (or close) for the effort to be worthwhile, but I'd wager that it is.

A Model Citizen

2011-10-11T14:19:00.000-04:00

While it is gratifying to be published by Simple Talk, so many more eyes that way, it isn't a platform where I can continue to prattle on at will. Each piece they publish, most of the time, is a stand alone effort. Since the piece was already rather long, there was one tangent I elected not to include, since it is a separate issue from the task being discussed.

"That subject: cleavages." Well, I only wish (and if you know from whence that quote came, bravo). No, alas, the topic is what to do with regard to fully understanding "bang for the buck". I elided that in the piece, since the point was to show that a useful stat graphic could be generated from the database. But how to discover the "true" independent variables of electoral primacy, and their magnitude? Could it be that with all the data we might have, both for free on the intertubes and costly which we generate, our best model is only 30% predictive? To reiterate, the exercise isn't to predict who'll win (FiveThirtyEight has been spectacular), but rather which knobs and switches a given organizations can manipulate to *change* a losing situation.

If you'll recall, most of the explanatory variables weren't of a continuous nature, that is, real numbers. The fitted lines in the scatterplots used a variation on simple linear regression to fit. The variation dealt with the differing best slopes over ranges. The technique doesn't account for the fact that most of the explanatory variables are either categorical (yes/no) or discrete (strongly disagree to strongly agree).

For this kind of mixed data regression, one typically uses analysis of covariance (aka, ancova). R, as one would expect, provides this. The Crawley book devotes a full chapter to ancova. I'll direct you there. Some say that discrete independent variables can be used directly in simple linear regression. Others would run to ANOVA immediately. Some distinguish categorical variables (gender) from discrete scaled variables (the 5 point agree scale on gun control). It is, suffice to say, not a slam dunk any way you go.

Exploratory data analysis, what R is particularly good at, is where the apparatchiks should be spending much of their effort (not worrying about the entrails of Rails!). Assuming that money is the driver of winning is an assumption, frequently wrong in the real world. Since their organization is large, national in scope, and full of dollars to spend; spelunking through all available data is the directive. That assumes, of course, that winning elections, without regard to policy positions, is the goal. Think of selling nappies.

While the goal of the piece was to display something simple to the Suits, determining a more accurate predictive model, which will be implemented with traditional text output, is the real goal. Same is true of selling nappies. The analogy is not so far fetched, as this book demonstrates; there have been similar treatises in the years since.

By The Numbers

2011-10-10T11:02:00.001-04:00

There's that famous quote from The Bard, "The fault, dear Brutus, lies not in our stars, but in ourselves if we are underlings." As my fork in the Yellow Brick Road tracks more towards (what's now called) Data Science, various notions bubble to the surface. One lies in an age old (within my age, anyway) dispute between traditional (often called frequentist) math stats and those who follow the Bayesian path. From my point of view, not necessarily agreed to exist by those on the other side, Bayesian methods are merely a way to inject bias into the results. Bayesians refer to this "data" as prior knowledge, but, of course, the arithmetic can't distinguish between objective prior knowledge and fudging the numbers.

So, I set out this morning, being Columbus Day (a day honoring Discovery for some, invasion for others), to see whether there're any papers floating about the intertubes discussing the proposition that our Wall Street Quants (those who fudged the numbers) bent Bayesian methods in their work. As I began my spelunking, I had no prior knowledge about the degree to which Bayesian had taken over the quants, or not. Quants could still be frequentists. On the other hand, it is quite clear that Bayesian is far more mainstream than when I was in grad school. Could Bayes have taken significant mindshare? Could the quants (and their overseer suits) abused the Bayesian method to, at least, exacerbated, at most, driven The Great Recession. It seemed to me likely, any crook uses any available tool, but I had no proof.

Right off the bat, search gave me this paper which references one (at a pay site) from the Sloan Management Review. The paper puts the blame on risk management that wasn't Bayesian. You should read this; while the post does discuss the SMR paper on its merits (which I couldn't read, of course), it also discusses the flaw in Bayes (bias by the name of judgment) as it applies to risk management.

Continuing. While I was a grad student, the field of academic economics was in the throes of change. The verbal/evidence/ideas approach to scholarship was being replaced by a math-y sort of study. I say math-y because many of the young Ph.D.s were those who flunked out of doctoral programs in math-y subjects. Forward thinking departments recruited them to take Samuelson many steps further. These guys (almost all, then) knew little if anything about economic principles, but department heads didn't care. These guys could sling derivatives (initially the math kind, but eventually the Wall Street kind) on the whiteboard like Einstein. I noted the problem then, the 1970's. This paper touches on this issue (linked from here). "These lapsed physicists and mathematical virtuosos were the ones who both invented these oblique securities and created software models that supposedly measured the risk a firm would incur by holding them in its portfolio." Nice to know it only took 40 years for the mainstream pundits to catch up.

And, while not specifically about Bayesian culpability, this paper makes my thesis, which I realized about 2003 and have written about earlier: "Among the most damning examples of the blind spot this created, Winter says, was the failure by many economists and business people to acknowledge the common-sense fact that home prices could not continue rising faster than household incomes." One of those, D'oh! moments. McElhone, the Texas math stat, introduced me to the term 'blit', which is 5 pounds of shit in a 4 pound sack. By 2003, and certainly following, the US housing market had become rather blit-y. The article is well worth the reading. There are links to many other papers, and it does raise the question of the models used by the rating agencies. Were these models Bayesian? Were the rating agencies injecting optimism?

Which leads to this paper, which I'll end with, as it holds (so far as I am concerned) the smoking gun (which I found to be blindingly obvious back in 2003): "Even in the existing data fields that the agency has used since 2002 as 'primary' inputs into their models they do not include important loan information such as a borrower's debt-to-income (DTI)..."

This few minutes trek through the intertubes hasn't found a direct link between Bayes and the Great Recession. I know it's out there. I need only posit such as initial condition to my MCMC (look it up).

Book 'Em, Danno

2011-10-07T17:51:00.000-04:00

For those of us of a certain age, the notion of physical books is important. I recommend any and all of Nick Carr's books, which deal, in significant manner, with ... books.

After finally figuring out where the house is, UPS dropped off my copy of "Visualizing Data" by Cleveland a day late (the widely regarded as incompetent Post Office and FedEx and the Pizza Guys all manage to find it). It's published by Bell Labs/AT&T (back when it still sort of was, 1993) and Hobart Press which is kind of down the street from Bell Labs. Their only listed books are Cleveland's.

What makes me giddy is what's printed as the end of the Colophon (few books even have such any longer). This is it:
Edwards Brothers, Inc. of Ann Arbor, Michigan, U.S.A., printed the book. The paper is 70 pound Sterling Satin, the pages are Smythe sewn, and the book is covered with Arrestox linen.

This is a real book. See you in a bit. Time to do some reading.

King Kong, Enter Stage Right

2011-10-04T10:26:00.000-04:00

Well, the Gorilla just sat on the couch. Oracle OpenWorld has this announcement.

Buried kind of deep is this:
Oracle R Enterprise: Oracle R Enterprise integrates the open-source statistical environment R with Oracle Database 11g. Analysts and statisticians can run existing R applications and use the R client directly against data stored in Oracle Database 11g, vastly increasing scalability, performance and security. The combination of Oracle Database 11g and R delivers an enterprise-ready deeply-integrated environment for advanced analytics.

OK, so now the King Kong has adopted R. Do you see a trend?

Don't Pay the Piper

2011-10-03T15:15:00.000-04:00

Big news day, today. And yet more of interest. We don't need no education.

This is specific to Britain, of course. Note that tuition is £9,000 (at today's rate, that's about $15,000), which is a piddling amount here in the USofA. Community college might be cheaper, in state and all that. More reactionary, back to the dark ages, assertions. Education isn't just vocational. That's why the business has VocEd and real college. And sure, if you want to be an Excel whiz, then learning all that mathy and logicy stuff is boring and a waste of time. I mean, how much do ya need to know to slap together a PHP web site?

Deja Vu, Yet Again

2011-10-03T14:45:00.000-04:00

My (single?) long time reader may recall that I concluded that the Oracle buy of Sun wasn't about java or Solaris or any software. It was about stealing the one segment of computing Larry didn't own: the mainframe, the IBM mainframe. I was initially alone, so far as I could see, although in the months following, I would read an occasional story tending toward the hardware motivation. If memory serves, some Mainstream Pundits explicitly stated that hardware was dead in the new Oracle.

Time to feast on some baked bird, crow specifically. Here's the latest from Oracle.

"'We want to take IBM on in their strongest suit, which is the microprocessor,' said Ellison."

Oracle may, or may not, be able to pull it off. Given that IBM's DB2, off mainframe at least, is adopting (well, it depends on who's defining that word) MVCC semantics, one could conclude that Oracle has gotten the mindshare part of the problem solved.

Political Science

2011-10-03T08:04:00.002-04:00

A while back, simple talk offered me an article, suggesting that something controversial would be appropriate. I pondered for a bit, and decided not to throw Molotov cocktails as I usually do here. Instead, based on some abortive conversations with apparatchiks in Washington, I set out to demonstrate how one can generate dashboard style graphs using stat output from R all within the database. In this case, the database is Postgres. Here's the piece. Enjoy.

Are We There Yet?

2011-10-01T16:46:00.000-04:00

An update on the world of (semi) serious SSD is in order. The Intel 710 is the successor, sort of, to the X25-E. AnandTech has a review and status update. Worth the read for the industry background alone.

The clearest description, and the one that was most logical: "Fundamentally, Intel's MLC-HET is just binned MLC NAND."

I'll mention in passing that AnandTech is dipping a toe into "Enterprise SSD" review with a piece on OCZ. Not that OCZ is really serious, of course; the Sandforce controllers depend on clear text data streams, which are getting yet more scarce in the Enterprise.

Pretty as a Picture

2011-09-29T10:38:00.000-04:00

Along with an interest in stats and graphs comes a level of responsibility. Kind of, guns don't kill people, people kill people. The canonical text is "How to Lie With Statistics", which was first published in 1954. Legend has it, it's never been out of print. Likely so.

It so happens that I've found a couple of blogs/sites which both deal with graphing stat data in non-disinterested ways. I'll note once again that a stat/quant/analyst/foobar is supposed to be disinterested. S/he's just an impartial judge of the data, trying to scope out the real relationships in the data; if there are any, there may not be. Data associated with politics is particularly susceptible to bias. But others face the same pressure. Worker stat bees (having been there) are often encouraged to slant the presentation in a way to make the nappie marketing Suits look like geniuses. It's a problem everywhere; all worker bees are expected to behave as attorneys; staunch defenders of whatever the Suits have done.

Watching the response to drug clinical trials is particularly amusing. Rather often, the sponsor will be shocked (shocked, I say) that its new FooBar Resolver didn't blast the .05 requirement out of the water. There'll be "unexpected placebo levels" or "unbalanced randomization" or "the FooBar Resolver patients were sicker than placebo". And so on.

Be that as it may, here are a couple of sites worth grazing:
The R Graph Gallery, from Romain François
The Gallery of Data Visualization, from Michael Friendly

Figures Don't Lie, But Liars Figure

2011-09-27T15:43:00.001-04:00

I just found this link, which says it all (well, most all) about lying, stats, and graphs. It's only a bit beyond 5 minutes. Time well spent.

It Ain't The Meat

2011-09-25T20:38:00.000-04:00

Back in the 70's a married lady (but not to me) of my acquaintance had a preternatural affinity for the Maria Muldaur song, "It Ain't The Meat It's The Motion". Nothing to do with me, I'll warrant. It was the 70's, of course, and it means what you think it does. Still true today, but the context relevant to this endeavor is a bit different. Welllll, may be a whole lot different.

One of the neat aspects of R is the ability to talk to most any other application, and vice-versa. R is, justly, known for the support for graphical display of statistical data. googleVis is an R package which links R to the Google Visualization API, empowering "moving" data in an html page. I've not played with it yet, but here's a sample from a blogger who has. Yet another case where the R community builds spectacularly useful widgets for the rest of us to exploit. Who said open source is anti-American communism? For the record, at least little Darl.

A few years back, I was involved with Business Objects, building dashboards. But using BO requires building a shadow schema of the RDBMS it talks to, and runs as its own application; generally a pain in the butt. With R, and PL/R with Postgres, one can drive the data and statistical analysis applications through the database. With googleVis, one can create animated graphs into the browser. Very cool. And his talk was on my birthday. Damn, I missed it.

The advantage of moving graphics is that this is a way to display higher dimension data; using bubble charts and animation, we get four dimensions, the bubble size and the motion axis (classically, time).

There are other plotting packages, beyond the base plot() functions, but I'd be willing to say that googleVis is the least difficult of the bunch. It does mean that it's for browser applications.

Newest Meme: NoClient Database

2011-09-19T15:26:00.002-04:00

What's that line from "Network"? "I'm as mad as hell, and I'm not going to take this anymore." Such is my view of NoSql nonsense. I'm not quite as mad at client coders who want to rule over the database, but close.

So, it was heartening to read Dunstan's latest post, in which he describes the end result of banning the client from the system. Save for rendering pretty, pretty screens I gather.

Good on him.

Don't Mess With Texas

2011-08-27T16:57:00.000-04:00

When I was somewhat younger, I worked for a math stat who was born in Rhode Island, grew up in Las Vegas, and did his graduate work in Austin. This was when I first heard the phrase "Don't mess with Texas". Context is everything, and today the context is data storage. Here's one version of the news. Which wouldn't be all that interesting, given that Texas Memory has been doing SSD for decades.

No, what makes this of interest is the following quote:
"TMS is targeting relational databases with its new storage device, just as Fibre Channel drives would be used as the primary storage."

Rad. BCNF support in the flesh. YeeHa.

Epiphany

2011-08-24T14:33:00.003-04:00

Whilst bloviating on the OCZ message board, I had an epiphany. It follows, including the snippet upon which I was commenting.

-- Maybe the strategy is to sell more consumer products at a low GM so they can increase the brand awareness of OCZ which will help sell more enterprise products that have huge GM's.

Not likely. The Enterprise SSD vendors, modulo Fusion (may be), build parts which are lots more expensive, and generally have bespoke controllers and SLC NAND (eMLC, whatever that might really mean, too).

To the extent that Enterprise SSD goes the route of Enterprise HDD (buy 'em cheap and swap 'em when they crap out), OCZ could best the likes of STEC, Violin and Texas Memory. We're not there yet; whoever figures out how to make a cheap SSD which dies gracefully wins. That may not be possible, given the physics, of course.

Viagra At Home

2011-08-19T19:49:00.003-04:00

A bit of R. I've mentioned a few times that I "knew" we were headed into the ditch around 2003. I don't recall that I'd read Shiller at that point (or even that I was aware of him), it was just obvious that house prices were outstripping median income. The raw data is available (at Shiller's blog, http://www.econ.yale.edu/~shiller/data.htm no idea how long it has been), so here's a picture worth a few words. The data run from 1890 to 2009.

Where's my little blue pill???

Old Frankenstein

2011-08-18T21:38:00.002-04:00

In "Young Frankenstein", The Doctor asks Eye-gor (Igor) whose brain he *really* retrieved. Igor replies, "Abby Normal?" I've spent the last hour or so wandering amongst some web sites, blogs, and whitepapers which seek to explain Normal Forms to normal folks; no math, just words.

This one says this: "'Normalization' just means making something more normal, which usually means bringing it closer to conformity with a given standard." Alas, not even close.

Which, since I've been re-reading my probability, stats, and stat pack books and docs, this flipped a switch. Which switch leads to a clearer, albeit slightly mathematical, definition.

I've done a quick search, and can't confirm that he explicitly said so, but given that Dr. Codd was trained as a mathematician, I'll surmise that he used the word in the following sense. In math, two terms are used as synonyms, orthogonal and normal. Remember from geometry class that a 90 degree line is the normal line? It's also orthogonal. Orthogonal as a concept means independence of influence (just as the X axis is independent of the Y axis; there some math), and Codd uses that term liberally in his paper.

So, the normal forms have nothing to do with not insane or seeking standards, but with data independence. Which is normal.

How To Mistreat Life

2011-08-16T16:39:00.000-04:00

It is amazing, but so far as I can remember, web apps have gotten more than half-way through 2011 before an article which takes client side code to task for being silly. Hard truth #1 is the worst. And the only way to avoid it: database enforced integrity. There, I said it again. NO DATA GETS WRITTEN WITHOUT THE ENGINE'S SAY SO.

The Know Nothing Party

2011-08-08T15:04:00.000-04:00

I came upon this rant/essay via R-bloggers. Beyond the fact that Zed (love that name) has a background fairly close to mine, is an R afficionado, and is willing to call the Emperor naked, one could substitute "RDBMS" for "statistics" in his piece. It would then read like a few of those which have appeared in this endeavor.

I really should send the link along to some of those folks in Washington I've chatted with over the last few weeks. Nah. They wouldn't get the joke.

Of particular relevance:
"It's pretty simple: If you want to measure something, then don't measure other shit. Wow, what a revelation."

And The Survey Says...

2011-08-04T13:43:00.003-04:00

As my dive into stats, and possible departure from RDBMS as the site at the end of the Yellow Brick Road, continues, I came across a ruby library called fechell. My inital thought: "shouldn't that be fechall, as in Fetch All, Fetch Ell. What does that mean? Well, D'oh! The normal name for the code is FECHell. Ah, much more to the point.

I found two posts, by way of R-bloggers by the person who developed the library. Here's the post where he develops the use of the data and the library. He references a Part 1 post with the background.

This intrigues me not a little bit. Suppose, just for grins, that you're the campaign manager for a state wide (or larger) candidate. That is, one where monies are allocated to distinct locations. Further, suppose that you have this data in close to real-time, and you also have data measuring "outcome" for the use of these monies, say polling data. And let's say that the two maps, monies and outcomes, are congruent.

Could one make predictive decisions about monies allocations? Well, it depends. The naive' answer is: abso-freakin-lutely!!!! The real answer: not so much. The naive' notion is that money well spent is indicated by winning the election (which is kind of too late for allocation decisions) or some upward movement in polling data. Ah. Let's spend where the spending works. Superficially, makes a lot of sense.

The only problem: stat studies invariably show little correlation between money and winning. I know, Liberals in particular are worried about the Citizens United effect, where corporations have gobs more loot than anybody else. They'll just buy the elections. And they well might. This would not make me smile. But, the studies of the data show that the effectiveness of campaign ads is less grounded in their expense, rather their content. Sometimes, may be often, attack ads work.

Here's an academic attempt to find out.

And yet another.

A quote from the second story (not, that I know yet, cited from the study):
"While we see an influence of the campaign ad in the short-run, in the long run the ad loses its effectiveness. This finding begs the question: how cost effective is it for politicians to spend millions of dollars on campaign ads which have little long-term effect on voter opinion?"

StatMan to the rescue!!! The problem is that it's now August, 2011, and any application being written as I write (assuming that folks have started) need to be up and running by January. In order to be worth the time and money expended, the application has to have *predictive* value. FECHell data passed through some software is only retrospective. Political ops should know enough about their candidates and opponents to design ads that work. Making a simplistic leap from $$$ to polling/winning is a waste of that time and money. The retrospective data needs to be run through some multi-variate hoops (either multiple regression or ANOVA, most likely; PCA and MDS are less applicable here) to identify the attributes, besides money, which move the bar toward higher polling or winning.

The problem with the simplistic model is that the knee jerk reaction to positive feedback in some campaign is to toss yet more money to that campaign. But that's likely a waste of money. The goal is to use the data to identify those trailing candidates today who'll win tomorrow if they get more $$$ and *spend it on what works*. Pouring money into a winner is a loser. Pouring money down a rat hole is, too. The latter case is more obvious, but the former is just as wasteful.

Economists refer to "opportunity costs"; I can spend $1 on toothpaste or candy. I can't have both. In the short run, candy is dandy. In the long run, toothpaste wins. Campaigns don't, generally, last as long as the toothpaste's long run, but you get the point. Money is finite, and should be spent on those activities/goods/services which gain advantage to the goal. In the case of FECHell data, the goal is winning elections. Looking retrospectively only at $$$ and winners is just the wrong goal.

STEC Crashes

2011-07-29T10:08:00.000-04:00

What's it all mean? Beyond losing a bunch o' cash for those holding (not I)?

It might be a bad thing for BCNF on SSD, but may be not. It kind of depends. According to reports from the conference call, STEC parts are being replaced by its clients (who are mostly storage vendors, not the user enterprises) with cheaper SATA drives and protocol morphing dongles. If true, then STEC's fall, while not good for them, is not relevant to the SSD Revolution.

On the other hand, if this means that SSD is being shifted aside from primary datastore to cache/Tier0/foo, then it bodes ill for my version of the Revolution. In Enterprise, at least. I could live with that. Enterprise has an absolute reactionary tilt; they keep 40 year old COBOL systems alive. Why isn't there a Do Not Resuscitate for dying code?

New systems, from smaller builders, are where the "innovation" will come from. I can live with that. If I never see the inside of a Fortune 500 (as an employee, that is) building, that is perfectly OK.

Mongo Loves Candy

2011-07-28T08:11:00.000-04:00

I recently chatted with some folks about real databases, in RoR, to solve real problems. Not so sure they're interested in real databases, but they're interested in Rails. Along the way, they mentioned that they'd been using Fusion-io SSDs. Be still my heart! Turns out that they've a separate datastore, in MongoDB, which had become as slow as molasses uphill in winter. So they bought a 1T Fusion-io card, in hopes of speeding things up. Didn't work out.

What's not widely understood about PCIe SSDs is that they're, more or less, heavily dependent on the cpu to get the work done. Or, as Zsolt puts it (on today's front page): "how much of the host CPU power is needed to make the SSDs work? - this is important if you're trying to fix an already overloaded production server - because you can't afford to lose performance while you tune the hot spots (even if the theoretical end point of the tuning process is faster)". I suspect they might decide MongoDB is the problem (document datastores make my teeth hurt). SSD with BCNF databases will generate real performance improvements. PCIe cards are not indicated if the problem is cpu bound.

One can find out, well enough anyway, whether the process is cpu or I/O bound with iostat and vmstat on *nix systems. That's the place to start.

Ohm's Law

2011-07-27T10:05:00.000-04:00

I've gotten to enjoy Christophe Pettus' postings linked from the PostgreSQL site. He does a neat presentation. This is his latest. Note especially pages 50 and following. While he's a Python/Postgres kind of person, and I'm currently exploring RoR again (long story), he does say things the way I do. Not quite as famous as he is, of course. In the database is truth. Note in particular his observations with regard to "cloud" I/O; it's what I've always suspected. It's your data, don't treat it like a red haired step-child. The SSD is the future of normalized, i.e. fast, data. The "data explosion" is largely the result of bad (non-existent?) data modeling. Cloud is all about minimalist/commodity parts which are easily re-assignable. If anything kills off the RM, it will be public clouds. Coders get infinite employment, and the profession relives the 1960s. Sniff.

So far as that goes, what he's saying about coders abusing the database from Django is about what I've seen with coders abusing the database from RoR; may be more so, given David's attitude toward data. The problem with ORMs is that they seek to solve a problem created by OO coders, but which doesn't exist in the Real World. Such coders refer to the problem as Impedance Mismatch, which is merely an assumption that objects can't be populated with data from the RM. But it's just an assumption. What they steadfastly (shades of Tea Baggers, what?) refuse to acknowledge is that BCNF databases allow for construction of arbitrarily complex data structures, unlike the hierarchic/IMS/xml approach, which is locked in to a parent/child structure. Change that, and all the application code which manages it has to change. Well, unless you've written a bare bones RM engine into your application. Don't laugh; I've lived through folks doing just that.

The world isn't hierarchic, no matter what OO/xml folks want to assert. I've worked lots of places, small to huge, and the archetype for the hierarchic structure doesn't actually exist. That structure is the Org Chart. In the hypothetical world, each worker bee has one, and only one, supervisor. The real world is run on Matrix Management, one has supervisor du jour, never the same one each day, varies by project/location/assignment/foobar. The real world is relational, connections come and go, in vivid multiplicity. The relational model stores such natively. From this structure can be built any set of connections which arise. By *not predefining* the connections, only the absolute identities of each type/rule, one can create new relationships simply by naming new foreign keys (cross-reference tables, by various names, for many-to-many relations).

One can also add new data without (if one has been moderately smart with the DDL/SQL) clobbering any existing SQL (or, heaven help us all) application code which directly queries the DB. Existing queries can ignore, if desired, new columns and new tables; so long as one avoids 'Select * from ...', of course. You would never do that, right?

M'mmm, Kool Aid

2011-07-13T13:59:00.002-04:00

I'll include some of the text, since the way Zsolt's site is structured, entries tend to disappear down a rabbit hole. Today, this is still front page. Go there to finish it up: he's got quite a lot to chew on, and he does the site for a living.

Editor:- July 11, 2011 - I recently had a conversation with a very knowledgeable strategist at a leading enterprise storage software company. I won't say who the company is - but if I did - most of you would know the name.

The interesting thing for me was that he'd recognized that if the hardware architecture of the datacenter is going to change due to the widespread adoption of solid state storage - that will create new markets for traditional software companies too.

And I'm not talking here about new software which simply helps SSDs to work or interoperate with hard drives - but software which does useful things with your data - and which can take advantage of different assumptions about how quickly it can get to that data - and how much intensive manipulation it can do with it.

While he doesn't say BCNF-RDBMS in his text, he's saying it. I've been bugging him for some time to drink the Kool-Aid. Sounds quite like both he and the unnamed "strategist" (no, not I, alas) have quaffed deeply. Face it, if all you do is keep appending flatfile "fields and records" to some file, not only do you never get ahead of the bull, but you get gored sooner or later. BCNF is the *only* hope. Yes, this requires designers/developers to actually *think* about the data. But, isn't that why we get paid the *big bucks*?

(OK, I went a bit asterisk nuts with this one. Finding validation does do that.)

What's Up Doc??

2011-07-12T10:51:00.002-04:00

Another in the occasional post from elsewhere. This time, simple-talk (no surprise there) with thread on bugs. Herewith my contribution, because the issue of buggy software can't be divorced from the application architecture and data language.

Ultimately there are two categories of bugs:
A) those caused by stupidity, inattention, carelessness, etc.
B) those that are the result of extending the developer's/team's experience

The entire ecosystem around each is necessarily different, and there are multiple approachess.

The A variety will be dealt with as the ethos of the organization dictates; anywhere from fired on first mistake to employed forever out of harm's way. Detecting such bugs should be possible with known testing harnesses/practices.

The B variety is more interesting.

For those in the BCNF realm, much of what passes for "new technology" in data stores and processing is VSAM redux, which brings with it the COBOL RBAR mentality, irregardless of the source language. This POV is wrapped in whatever jargon is native: NoSql, Hadoop, Map/Reduce/BigData/foobar. But the fact remains that coders are implementing ACID (if they care at all about their data) in some (high-level) language outside the storage engine.

Whether the organization realizes its mistake, and implements engine side processing, a la Phil's current article, or undertakes to use the FOTM client side framework, the coders are left in unexplored territory.

Left unexplored, generally, is an analysis of what architecture (engine side vs. client side vs. application language vs. database engine [not all do all things well]) is the least prone to both type A and type B errors for the application in hand.

Declarative languages (SQL, Prolog) just tend toward fewer errors. SQL is dependent on schema quality, and coders tend to view schema specification as a low value, unimportant task. Certainly not one for which specific expertise and experience is required; any coder can do it.

The bugs that matter, which mess up the datastore, are just less likely if processing stays in the engine. Bugs which consist of ugly fonts, not so much.

As to IT managers, again, two categories: those that were and are technically superior, and those who never were. The former, albeit rarely do they exist, generally get more done. The latter do awesome Power Points.

Workin' on the Chain Gang

2011-07-09T15:56:00.003-04:00

LinkedIn, LinkedIn whatever are we to do with you? I've not had anything to say, given how silly the whole mess is, but today's NY Times has an almost true article. I don't have meaningful disagreement with the problems raised in the article, but it avoids the underlying issue. (That it makes my thesis that advertising based business is inherently unstable, is another atta boy for me.)

No, the problem with LinkedIn is that the business model is foolish. The business model is based on the assertion that people without employment and income will rush out to buy stuff. How stupid is that? One can slather on all sorts of finery, but that's the business model. At least Google attaches ads to activities utilized by everybody.

There's a reason that employment agencies charge money for their services; they actually do some work. Most of it is negative, removing for essentially arbitrary reasons otherwise qualified folks. LinkedIn presumes that if an unemployed is known to the employed, that this will embolden hiring agents to consider an unemployed for a position. Factually false. Been there, done that. Employers, though it be illegal, are more than willing to admit not interviewing an unemployed.

What, then, about those on LinkedIn who are currently employed? Will they be buying stuff? May be. May be not. The folks from my last employer that LinkedIn offers up each week or so, for instance. Are they looking? I don't know. I do know that 99.44% of them have never worked anywhere else (both young and old) or on any other software. In many cases only the decades old COBOL that constitutes the application. On a mainframe. Will such folks be buying stuff? Probably not.

Near as I can tell, LinkedIn, whether its progenitors say so or not, is attempting to implement what the high end (or low end, depending on your point of view) agencies promote: access to the hidden job market. Whether such actually exists has been a matter of controversy at least since the 1970's, lawsuits and all. For companies large enough to have an HR department, ain't nobody gettin' through without they go through them. It's job preservation, after all. For the SMB crowd, it might work. For startups (where the really interesting, and vastly stupid, activity is), even less so.

LinkedIn is a bottle rocket, soon enough to come crashing down. Google needn't worry that it is the advert server to fear. There will be such an advert server, as I have written. LinkedIn isn't it.

I'm Just Going Through a Bad Phase

2011-06-30T21:09:00.002-04:00

"Phasers on stun", said Captain Kirk. Well he said that just about every episode. Today ComputerWorld reported on IBM's PCM flash replacement. For background on PCM, see WikiPedia, and the article has a link to a much earlier one on PCM.

From the WikiPedia piece:
"PRAM devices also degrade with use, for different reasons than Flash, but degrade much more slowly. A PRAM device may endure around 100 million write cycles."

In the past, I've written about Unity Semiconductor, which has had a flash replacement in development for some years; I found them when I first began looking into SSDs. One way or another, we'll soon have a solid state datastore that is effectively infinite in write capability, just like HDD.

Once again, Oz looks larger and brighter. Be still my heart.

A Bump in the Yellow Brick Road

2011-06-28T15:33:00.000-04:00

No, I'm not renouncing. Events of the last year or so caused me to ruminate on this journey down the Yellow Brick Road. Some of the events:

Consumer/prosumer SSDs persist in not being built with data caps. The industry is, perhaps, more divided now than at any earlier time. Consumer devices use barely tractable MLC flash (~3,000 cycles), and SandForce continues to gain traction in the consumer side. Given the finite nature of flash, an SSD will die in the near future. An HDD, on the other hand, might well continue to function for the better part of a decade. In any event, the HDD doesn't have defined drop dead time.

Capacity remains under a TByte for the vast majority of parts. This is important because:
most folk continue to view SSD as just a faster HDD; which isn't important outside of the RDBMS arena, but critical to getting the most bang for the buck there. For RDBMS installs, where (re)normalizing is ignored, the cost of moving from HDD to SSD is expensive, so is often attempted with consumer level drives. In the HDD world, that's not unusual; most drives are both over there.

Small scale (web and SMB verticals, for instance) databases, often on MySql or Postgres, just won't be safe enough on consumer drives. The various threads on postgresql-performance make the case, much as I'd wish the truth be otherwise. What's particularly odd is that both vendors and most consumers appear to be OK with catastrophic loss of data in normal life. Very odd.

Given the physics of writing, SSD vs. HDD that is, is just way cool different. SSD controllers spew the bits all over the flash, and the erase process can hardly be considered atomic. The majority of SSD controllers use RAM caching to reduce write amplification, and this is an additional fault point. HDD based engines, industrial strength ones like DB2, can guarantee that only the open transaction(s) will be hosed on a failure. SSD based storage just can't if there isn't persistent power available.

The failure of developers, at least those who publish, to lobby for (re)normalization as part and parcel of transition from HDD to SSD is regrettable.

Is there still a Yellow Brick Road leading to Oz? I still believe so, but Oz looks to be more a Potemkin village than a New World. Only shops with the fortitude to make a full transition using enterprise quality SSDs will actually get there. One can eliminate 99.44% of web sites and SMB verticals; they're just content to be penny wise and pound foolish. Oh well.

Sprechen sie Deutsch? Habla Espanol?

2011-06-14T10:47:00.000-04:00

Artima has been quiet of late, not many new articles or comments. Could be that coders are satiated? Then I went over today, and Bruce Eckel has praise for Scala. Last I read, he and Bruce Tate (not related, so far as I know) had gotten Python fever. Well, Eckel has been infected for some time. Which is why I never expected to get a Scala piece from him.

Smitten, I ordered "Programming in Scala"; finally bit the bullet. It's one of the few languages, not on a database engine, that still intrigues me. Yes, I've done Prolog, Erlang, and Haskell, to name some. And, yes, Prolog is the closest to a RM language out there. But it's just not used enough. That, and the syntax is just plain wacky (not as extreme as Lisp, but that's to be expected). 2.8 is the current version, and is covered in this second edition.

But, back to Eckel's article. What struck me, yet again, is the emphasis on iteration in his discussion. Yet another language creating yet another syntax to loop. Why are we still doing this in high level languages, not-assemblers? For some code, that which doesn't deal with real data in databases, it could be argued that application code needs to loop. But, even then, I don't quite buy it. I spent/wasted some years with Progress/4GL, a database engine + application language. It had flippant support of SQL in the engine, but 99.44% of coders used the bundled 4GL. And how did this language deal with table data? You guessed it, --For Each-- . Now, this was promoted as a *4GL*, not COBOL. Fact was, it was effectively COBOL. We've been saying to each other for at least three decades that the future is now, and the future is declarative coding, yet we keep focusing on application level iteration. The datastore should do that. It's written in the lowest level language, typically naked C (with, I suspect, performance bottlenecks in each support OS assembler).

Let the datastore be the datastore!! (Yes, that does remind you of some political hackery from the 1980's.)

Declarative development is exemplified by the RM and RDBMS. Why the refusal?

We continue to see web/client app code which, even on database support/discussion sites, we're concerned about sending whole result sets to the client. Why? If you're intent is to make changes to multiple rows in a specific fashion, that's a stored procedure. Do it all on the engine. That's what it's good at. Don't ship data off the server, just so you can iterate (using the very special syntax of your fave language) over thousands or millions of rows. This is folly. Such discussions always then devolve into arguments about transactions locking rows; and so forth. Yikes!

While it isn't quite what one might expect, given its title, everybody should read Celko's "Thinking in Sets". It's not a treatise on set theory in the datastore, but still useful in providing examples where table data makes more sense (even in a performance metric) over code.

My Security Blanket

2011-06-09T12:03:00.000-04:00

Regular readers may note that Linus Torvalds has pride of place in the quotation section of this endeavor. He's recently been interviewed by some e-zine I've not heard of.

Here's a quote:
I'm also a huge fan of SSDs, and the huge reduction in latency of storage technologies has some big impacts on OS performance. A lot of people end up spending a lot of time waiting for that traditional rotational media.

Pretty much what he said back in 2007. Let's get a move on.

Fiddlers Three

2011-06-07T09:33:00.002-04:00

In the event that you haven't been following the news, Fusion-io is in the midst of an IPO. It turns out, that Violin Memory is closing in on doing so, too. Since I began digging into the SSD story some years ago, Violin has been kind of (but not quite) stealthy. In part, because it is still private, so doesn't end up in various "investor" discussion forums, and in part because it focuses on the True Enterprise SSD; Fusion-io and OCZ in particular are public and consumer focused.

The article mentions Oracle, which fits with my vision of where SSD databases are going. Eventually even Larry will figure out that the bang for the buck (his, not necessarily his customers) lies in Being Normal. For Larry, that's saying a bit.

You're Solid, Man

2011-05-31T07:46:00.000-04:00

If you're of a certain age, "solid, man", has a certain meaning (if you go here, and open up the list and then go to the bottom. SolidFire is a company which claims to be building SSD in the cloud; somehow or other I got on the early announcement list.

According to their press section, they've been in business since January of this year. It'll be interesting to see whether they make the connection between the value of SSD and the value of the RM/SQL/RDBMS. Minimize footprint, wear SSD galoshes.

Do the Limbo Rock

2011-05-24T09:27:00.002-04:00

There's been some whoopla recently about OCZ's latest drives, and the SandForce controller used. Today, AnandTech looks at the latest of the latest, the Agility 3. Of most interest to this endeavor is the background on NAND architecture.

What's it all mean, Mr. Natural? Well, with respect to Enterprise (or even SMB) systems, not much I expect. OCZ and SandForce remain, so far as I can determine, firmly in the consumer/prosumer territory. Until SF drives start showing up from a major vendor, of course (there's not been an announcement from SF, yet). I don't see that happening, what with the implementation of encrypted/compressed data in industrial strength databases. Workgroup/internal document storage, may be.

Mr. Wonka's Factory

2011-05-05T14:53:00.002-04:00

Here's a post on simple-talk which was dormant for most of a month (it's down to the last place on the front page today), then got really popular the last few days.

The subject: cleavages; well no, but I've liked that line since Lou Gottlieb used it to introduce a song lo those many years ago. Well, may be, actually, in a manner of speaking.

The subject is database refactoring. The post and discussion started out blandly, but then opened up (cleaved, see?) that long festering wound of front-end vs. back-end design/development. Simple-talk subjects and discussions tend toward a coder perspective, I suppose because T/SQL is often germane to topics, so a certain amount of self-selection exists both for authors and readers.

The idea of refactoring a database, especially given the tripe espoused by the likes of Ambler (some of his stuff), has been hijacked by the kiddie koders. Refactoring, for a RDBMS means getting more normal. It doesn't mean twisting the schema to fit some single access path, a la IMS or xml (hierarchies). That's not why Dr. Codd made the effort. He made the effort because he saw the mess that IMS was making; he'd been there when IMS was released and devised the RM a couple years later. In other words, it didn't take a math guy very long to identify both the problem and the solution.

Now that we have high-core machines with SSD as primary datastore, we can implement BCNF (at least) schemas/catalogs with no worries about "let's denormalize for speed". Those days are truly history.

Get Offa That Cloud

2011-05-03T18:26:00.000-04:00

On more than one occasion, I've criticized all things cloud, since I see cloud as a lowest common denominator storage approach. But then, this is based only on my experience with cloud-like provisioners over the last couple of decades. Well, turns out I'm not the only one, and some of these folks have as much fun with the silliness as I have. The thread is new, so keep track for further hilarity.

Ruler of All I Survey

2011-05-01T23:22:00.002-04:00

There's that old phrase, "master of my own domain", and it is particularly useful in relational databases. While working in the COBOL oriented DB2 world of Fortune X00, I was continually rebuffed whenever I suggested the use of check constraints. I had to sneak them into some of my lesser databases, but that was cool.

In addition to regular check constraints, there is the notion of domains. They're sometimes referred to as user defined types. Here's a Postgres based treatment. Interestingly, Postgres has better support than DB2; PG allows check constraints while DB2 doesn't. Once again, the author's make my point (admittedly, the point made by anyone who takes RDBMS seriously) that control of the data in the database, rather than relying on each application code. Bulk loads, with most engines, enforce constraints so batch data transfers are a breeze. Not to mention that client code can be in any convenient language. And that a smart generator can use domain/UDT definitions whilst doing its thing. I've said that before, I believe.

He Ain't Superman

2011-04-25T11:05:00.003-04:00

Back in the early days of java and the time of the dotcom bust, George Reese was a minor pundit/author (O'Reilly division) in the database part of the world. You can look up his stuff at O'Reilly or Amazon. Not much heard from since then. He would occasionally show up on the O'Reilly site. And he has again.

Here's his take on the Amazon fiasco. What's so intriguing about this missive is the Up is Down meme involved here. The Cloud meme has been promoted as a less: expensive, time, resource, attention answer to the Data Processing Problem, particularly for web sites. Reese spends his few thousand words telling us that the SLA is *our* responsibility, not Amazon's. Ditto for infrastructure design. Ditto for physical design. Ditto for just about everything. Well, you're not allowed RDBMS, but you didn't want that anyway, did you?

He spends all that ink telling us that we have to work around the fragility of Cloud provision, in order to utilize Cloud. And we can't do databases, because, well, Cloud just isn't quite up to that. Where, exactly, is the win for clients who care about their data? He admits that such services won't provision such that there is sufficient excess capacity to support the loss of significant resources. Kind of like what might happen if you ran your own datacenter, only worse.

What's most annoying about his missive is that he makes, nearly explicitly, the assertion that the storage method, Cloud, determines the nature of the datastore. Not only can you not have SSDs as primary store for a BCNF databases, you can't even have *any* sort of RDBMS if you use Cloud. Last time I checked, that's the tail wagging the dog. But knuckleheads who admire the Emperor's New Clothes typically ignore such conflicts.

Oh, and you really should go read the piece. The commenters have a field day with his silliness. Yum.

[UPDATE]
In my haste to get this in words, I neglected to explicitly state the objection to Cloud I (and others) hold: the value proposition for Cloud is that it enables organizations (even just the IT group on its own) to out source an unwanted responsibility at lower cost; don't do it yourself and save money. Reese's treatise, and the failure, deny the existence of that proposition. This, while I gather he doesn't get it, is kind of a Big Deal.

Was A Cloudy Day

2011-04-24T22:45:00.002-04:00

As regular readers know, I've not been a fanboy of anything Cloud. My reasons are less to do with security, reliability, and other mundane considerations; rather that Cloud represents lowest common denominator (race to the bottom) disk storage. As a means for storing family vacation photos, well, OK. I'd prefer to keep those on my own storage, but each to his own. But Cloud for serious storage, I've never been a fan. My Yellow Brick Road is paved with SSD running BCNF databases. Unless, and until, Cloud provisioners recognize that it ain't "about just bytes", I'll pass. Some times, if it's too good to be true, it ain't true.

Then Amazon jumped the shark.

Here's another take on the situation. This is from an experienced Cloud provider, of a sort. As he says, the Fortune X0 have been trying to provide a central storage solution for rather a long time, with little obvious superiority. It's worth noting that the IBM Service Bureau service goes back to, at least, the early 1960's. Cloud is neither new nor a walk in the park. I guess Amazon and its clients now know that.

The Service Bureau (and similar) were able to provide some semblance of service over leased lines. The notion that TCP/IP, with HTTP tossed in, over normal phone lines is sufficient is, well, immature. What was that you said? The Emperor has one fine set of threads. Yes, yes he does.

Not A Cloud Was in the Sky

2011-04-20T19:05:00.003-04:00

On more than one occasion, I've made the point that "cloud computing" is really very old hat. Or, old wine in new bottles. I came across this interview with IBM honch Steve Mills.

Here is a clip:

Mills: I think that's the number one reason why this is appealing to CIOs. The interesting part about "cloud speak" is that many people want to isolate their discussion of cloud to only certain classes of companies. And my view is that's too narrow. Service bureaus emerged in the 1960s. I mean, ADP is one of the industry's biggest and most successful cloud companies.

Knorr: And earliest, right?

Mills: Yeah. If I'm in the accounting department and I use ADP, ADP is my cloud company.

The "service bureau" is actually older than ADP, and was an IBM invention. See this search result.

Your Diagnosis, Herr Freud?

2011-04-06T23:09:00.000-04:00

Just a short note about the new Intel 510/320. In a nutshell, Intel claims the Marvell powered, sequential biased, 510 is the Prosumer part while the Intel powered, random biased, cap protected 320 is the Consumer part.

This makes no sense at all. The early reviews, both systematic and informal, say that the 320 is a righteous part. I'm going to order the 160 gig in a few weeks (or so), depending on whether there're reports of funny business. Amazon shows two part suffixes: B5 and K5; the AnandTech review is a K5 part, so that'll be the one. Now I can, finally, get around to running some data (leave the other stuff to the side for a week...) through HDD, G2, and G3 (as the 320 is being called). Fingers dirty. Yum.

What's Your Preferred Position?

2011-03-29T15:39:00.000-04:00

Regular readers know that I've been talking up the synergy between iPad type tablets and the normalized relational database. And that such synergy must recognize the nature of input on a tablet. Little bits of pickable data. Hors d'oevres, so to speak; not a four course meal.

A few minutes ago Anand posted a quiz. I guess the question has reached the mainstream pundit class, although it's not quite the right question. The right question is: what kinds of data can a keyboard-less device support, and therefore what kinds of applications are best suited to such devices?

Mr. Natural Answers Your Questions

2011-03-28T23:10:00.000-04:00

A posting on the PostgreSQL/Performance group (having to do with the Intel 320 announcement, which I'll save for a different posting after the dust settles a bit) got me to looking again for published tests of SSD vs. HDD and databases. As you can see, Dennis Forbes is listed in the links block. I don't recall whether I've mentioned this post of his before; may haps I have.

But, in some sense related to the Intel 510/320 situation, he makes the salient point (my point since I discovered SSDs years ago) thusly:

Of course NoSQL yields the same massive seek gain of SSDs, but that's where you encounter the competing optimizations: By massively exploding data to optimize seek patterns, SSD solutions become that much more expensive. Digg mentioned that they turned their friend data, which I would estimate to be about 30GB of data (or a single X25-E 64GB with room to spare per "shard") with the denormalizing they did, into 1.5TB, which in the same case blows up to 24 X25-Es per shard.

Of course, his insight is rare, even among those I've read who've been positive about RDBMS/SSD synergy. It's always been obvious: normalized datastores are orders of magnitude smaller than their flatfile antecedents. This smaller footprint comes with all the integrity benefits that Dr. Codd (and Chris Date, et al since) defined. There are a whole lot of ostriches out in the wild, insisting that massive datastores are needed by their code. What does it all mean Mr. Natural? Don't mean shit.

Shape Shifting

2011-03-25T09:55:00.002-04:00

One thing that I really like about O'Reilly books is the Rep-Kover binding; the original better than the current, however. I find that most computer texts are near interchangeable with respect to content. It's nearly always marginal, so what matters is ease of use. For that, Rep-Kover is better than current "hardcover" bindings. What I tend to dislike about O'Reilly is their (his?) incessant need to create "new" memes in the computing world. Web 2.0 is, I think, the first; certainly the most infamous so far.

The last few months have seen the aborning of another: Data Science. This one is even worse, in that it seeks to dumb down a perfectly legitimate pair of professions; statistician and operations researcher. Long ago, I got involved in ISO-9000 certification, which was another early attempt to dumb down those professions (these days it's Six Sigma, which I had the pleasure to mentor at CSC). It irritated me then, too. It's of a piece with DIY neurosurgery, although not as directly deadly.

Yesterday's Forbes on-line version published a story about this newfangled profession, in the context of EMC. Regular readers may remember that STEC, gorilla of the Enterprise SSD jungle, first touted, then crashed, on its relationship to EMC. The article whispers that STEC, or whoever is currently supplying, is and will do well.

What's most bothersome about this meme is, as many others have remarked, both math stats and ORs do inferential stats, and inferential stats is based on the math of sampling and inference. The fact is, one needn't have much training to calculate the parameters of populations. Fact is, math stats and ORs don't even refer to these numbers as statistics, because they aren't. It is exactly the same as baseball stats; they aren't stats, just numbers. But, of course, the meme-sters once again wish to wrap themselves in the blanky of higher math.

On the other hand, stats as a profession and work product is more interesting than computers. Even databases, by golly. May be I'll try to parlay both; the article says that such folks (humble self qualifies) are in demand.

You Rook Mahvelous

2011-03-11T16:12:00.002-05:00

Just when you've figured it out, the world has a habit of slapping you upside the head. By now, you've likely heard that Intel has announced its next SSD, the 510. It's not, explicitly, the X-25/G3. From the various sources I read, a G3 will be coming along in due time, but is intended to be the "consumer" version, while the newly announced 510 is the "pro" part.

Here's what's puzzling: as you can see from this AnandTech article, the 510 is biased toward *sequential* processing! Boy howdy, I never saw that coming. That, and the fact that the controller isn't home grown, but bought in from Marvell. The G3 is said to be driven by Intel's controller, but not yet confirmed.

The world has been turned upside down. Either that, or Intel has completely misread both the technical and buyer worlds. A sequentially biased SSD makes sense for consumers: gamers, video processing, and such. I'm truly puzzled. The parts aren't big enough in capacity to store anything like the massive files that a file based coder would use. For the prosumer world that the X25 parts targeted, the 510 just won't be useful, it's barely on par with the X25-G2.

We'll see. The 510 still has an advantage over the SandForce drives for compressed/encrypted data, but that's usually things like my beloved relational databases and random processing, not the 510's strength. Weird.

32 Heads Are Better Than One

2011-03-04T14:18:00.000-05:00

Simple-talk is one of my favorite sites, and now they have what will be a series on parallelism in SQL Server. This first installment is light and airy.

A few months ago, I got into a bit of a tiff on a Postgres email group when I had the temerity to suggest that query level parallelism is not only a Good Thing, but the only way to maintain performance as we segue from ever faster clocks on single thread/core cpu's to multi-thread/core/processor machines. That group assembled (I don't recall anyone joining in my defense) asserted that engine level parallelism (doling out queries to threads) was enough.

I've been arguing for years that RDBMSs (not just SQL Server) will be better applications if they're designed to the multi-core/processor/SSD machine. After all, at least the multi-core part is now fait accompli, so why not? The beneficial side-effect is that BCNF schemas, with SSD as *primary* storage, are fully feasible. They are the minimal data (bytes, that is) needed to fulfill demand, and since they are "fully" normalized, DRI implements most if not all of the constraints on the data. That's been written about here, a bit.

For all the heat that MicroSoft gets, even from me on occasion, they do get databases. Good on them.

For completeness, here's DB2 docs, rather dry, but then...

And here's Oracle.

Neither is as seamless, at first blush, as SQL Server. There, I said something nice about a MicroSoft product.

Something Blue

2011-03-03T10:40:00.002-05:00

There's that old saying, "Like father, like son". The iPad 2 (or is it iPad2??) is out, and AndandTech has a preliminary review up. In the Conclusion section, there is this:

However, the new iPad does attempt to further blur the line between full computers and tablets, a line that is only going to get blurrier as more Honeycomb tablets invade the market. The iPad still lacks a dedicated keyboard, which will probably always hamper its utility as a content creation device for me, but iMovie and GarageBand join the already existing iWork apps as decent tablet versions of desktop programs.

What is, I guess, slowly dawning on the Pundit Class is that pickable interfaces are necessary for tablet devices. Such devices can be used as input devices, IFF the data can be displayed in an easily pickable way. I've always asserted that this simple fact means that a new (set of) widget(s) is needed, along with the recognition that data (as opposed to content) should be normalized at the datastore, i.e. BCNF. I know that most of the Pundit Class hasn't figured it out yet, but you can't fool Mother Nature.

Shorter Wires

2011-02-28T08:34:00.003-05:00

One of the themes/memes of this endeavor is that we are inevitably headed back to the 1970's, in more ways than one. An oil embargo looks to be an added carrot to the stew, but I digress. The 1970's from a macro-architectural point of view: centralized data and dumb-ish screens connected to same. I was the first of the pundit class, that I can find, to tell this story.

Others are beginning to see it, too. I keep track of Mr. Market through, among others, the Seeking Alpha site (a stock pumping outfit, that also sometimes publishes technically "interesting" stuff). In today's episode is a piece on Intel's Thunderbolt tech; worth reading. What caught my eye, and inspired this post however, was a comment from one who signs as ryanclarke:

By having an office full of 'data screens' with Gigabit or Thunderbolt fiber connectors ... all could be using the same processor sitting in a storage rack in the coffee room. Back to the 'thin client' and mainframe days of the 1960s-1970s. That's the technological push underway right now.

When I was a Fed, back in the days of yore, we four (not horsemen of the Apocalypse) were OR/stat guys in the midst of HR managers. There was a bit of friction. One of my colleagues, Frank the Elder (as opposed to Frank the Younger; last names omitted to avoid legal action, one never quite knows about these things), was a psychometrician who was fond of the term "subvision". This is not to be confused with Subversion, the recent bit of VCS. No, subvision was the counter to supervision, whereby the worker bees led the queen bees (most of whom, obviously, were male; I've always wondered how staff got tagged as worker bees, given that the apian world is female dominant yet the human male) forward. This evident shift, and a tectonic one it is too, looks more and more to be understood by the troops on the ground, but ignored by general staff. (Shift in metaphor.)

The main issue is whether we also take advantage of RDBMS/SSD datastores, or go all the way back to Neanderthal times to COBOL/VSAM. I pray not.

Hit the Ground Running

2011-02-12T14:32:00.002-05:00

One of my pet peeves with the current generation is their apparent lack of understanding of physics. The iStuff folks, both producers and consumers, act as if there is infinite bandwidth for each unit in the wild. And, of course, there isn't. Turns out, not everybody in the industry is quite so lunkheaded. It turns out, if you increase the density of receivers, lots of good things happen.

Implicit in the story: wireless is rapidly going wired; just as it has to and should have been from the beginning. All these mini-cells are like the spines of a jelly fish, tethered back to the mothership. Far more signal can be carried on optic, even copper, than on the ether.

Now, if only we can get Mr. Fusion up and running.

Talkin' 'Bout My Generation

2011-02-10T08:36:00.000-05:00

Another candidate for CUD (doesn't attempt to do Read) from our friends at SQL Server Central. Nice to know that some people are connecting the dots.

Along similar lines, Cringely has a recent post dealing with the Ken Olsen's, alleged, misreading of the PC movement. The point being, of course, that Olsen was right (by now; not so much at the time). We've long since past the point that owners/users of PC's actually write programs for the blighters. And, no, I don't count Excel or Word macros as programs. With the rise of iStuff (and competitors), we'll just be using pixelated terminals connected to the Big Host in the Sky. For those that might be interested, that notion was made manifest in the 1960's. A more complete history is here.

This really is back to the future. If only these people knew, they'd save us all a bunch of ulcers.

(Not So Simple) Simon Met a Pieman

2011-01-31T23:53:00.000-05:00

Oh my. Another emperor's new clothes apostate. Another O'Reilly cite; they're coming in waves lately. Who knew? Anyway, they've just published an interview with Simon Riggs, who heads up 2ndQuadrant, a PostgreSQL consultancy. I run hot and cold on PG, mostly because I still believe in the locker paradigm, on balance, and that means DB2.

Here's the quote:

How come CouchDB, FluidDB, MongoDB etc. are so much in the news at the moment?

New things are sexy. Nobody wants to hear that the way your Dad used to manage data is still the right way, in most cases. Especially when it turns out that what you just suggested isn't new at all, and that your Granddad used to manage data that way too before he gave it up.

Ouch!! I swear I haven't done a Vulcan mind meld with the guy; never met him. But he's got that sharp tongue I do appreciate.

Game, Set, Match (not the other thread)

2011-01-31T21:19:00.000-05:00

Big Data is among the juvenile memes (both in terms of the meme's age, and the maturity of its proponents) out there. I've been reading Janert's book off and on for the last couple of months, motivated by a scan in my local B&N. That he has a, shall we say guarded, view of what good Big Data is, led me on. My interest in math stats goes back to before my dive into RDBMS, and, given the absolute choice to do it all over again, I'd have stayed with stats; if for no other reason than the fact that experience is far more respected there than in databases (hissssss).

Imagine my surprise then to find an article which promotes set-oriented semantics at the O'Reilly site. There was a time when I worshiped O'Reilly books. Not so much since he/they went on the meme pimping road; web 2.0 being the most obvious. Perhaps because web 2.0 has faded, they/he have invented another, data science. And, as usual, it's code/language oriented for the most part; the irony of this approach appears utterly lost on the group assembled. Except for this article. There are two, the one on the site page references the one in the link here, which goes the full monty. The site page article is a tad more than an executive summary.

I'm not going to sprinkle quotes, since I'd end up reprinting pretty much the whole thing. Suffice to say, just go read it.

Passion Fruit

2011-01-24T11:40:00.000-05:00

As I've mentioned a number of times, while I've spent the last decade mostly in DB2/LUW, I've done some work with SQL Server, and hang out at the Simple-Talk site. They have an editorial every now and again, which generates comments on RDBMS generally (usually) rather than SQL Server specific bells and whistles. The current one is "Passion" (yeah, that's right), and I was moved to comment. Shown here for your amusement.

I've been hanging around here for a few years, and I sure do view databases (even SQL Server) as a "calling" which is sorta, kinda like a "passion". All of the places I've managed to find employment have been in spite of this calling, alas. Perhaps it's different on the other side of the Pond, or with SQL Server centric venues (my main database has been DB2/LUW for some while), but "passion" isn't what's rewarded.

Certainly, the advent of multi-core/processor SSD machines will have a profound impact on how RDBMS applications are built. The transition just isn't happening fast enough to suit me.

The corporate bent of DB2 does seem to make it more hidebound than other databases; which is such a shame, in that the LUW version is so much better than any of the other databases that run on those platforms. IBM is truly squandering a major opportunity.

Trains, and Planes, and Automobiles

2011-01-14T09:10:00.000-05:00

Well, Bob and Oracle and IBM. Cringely is at it again, and he provoked me into a reply to his drivel. It was interesting enough, to me at least, to warrant repetition here. You're welcome. (I've cleaned up a couple of typos here; rather early for my eyes to focus.)

@Bob:
IBM's competitive advantage over HP and Sun was IBM had a services business.

Well, no, not exactly. As I have predicted since Oracle bought Sun (and Bob decided not to include in his 2011 list), the advantage IBM has are all those z/Series (or whatever it's called today) Fortune X00 clients. Lose the z/Series box, and services disappear.

Larry has lusted for those clients for a long time. There's a reason Relational Software (not even its first name) had the first "commercial" RDBMS, stealing a march on Armonk. Armonk hated Codd, since he stuck a fork in IMS shortly after its birth. Larry wanted an alternative to the mainframe app world. He had only software, but did OK.

But the only way to get all those big, juicy Fortune X00 z/Series clients away from IBM is to have a full stack. He has to find a way to convert 20, and 30, and 40 year old COBOL code to something modern. With the Sun hardware and Oracle he now has a shot at it. It won't happen tomorrow; I'm wagering a prediction that there'll be concrete progress this year.

As to overplaying his hand with the *nix client base? It wouldn't be the first time he's done that, too. I don't see one strategy precluding the other tactic.

Cringely Hates Me, He Really Hates Me

2011-01-07T08:53:00.002-05:00

Bob didn't include either of my predictions in his list. I feel so violated. But, if you look at the 10 in toto (here they be, if you don't tarry), the pattern I've been predicting emerges. A world of centralized data, and pretty pixelated clients. And, may be, not browser driven.

Could it be Bob that doesn't really hate me? I suspect his mission was to wrap it all up in Apple; he's been fixated on Steve for some time now.

Pundit For a Second Day

2011-01-06T12:19:00.000-05:00

Regular readers know that I submitted to the Real Cringely a prediction. He's only up to 6 so far, thus, I don't know whether he's deigned to accept it. When he made the announcement, he only said that any outside predictions would be credited, not that the author(s) would get advance notification of "winning".

While we wait, and given the recent spate of SSD and processor news, a new idea has been worming around my cerebrum. This post is just to establish originality, not the full blown patent. I could well abandon the idea. It's just one of those thought experiments.

My submitted prediction was that Oracle would leap frog the other RDBMS vendors by pushing the BCNF approach in order to win over the IBM/COBOL mainframe application crowd to the Oracle/SSD machines. The magic bullet approach.

But what if a vendor, may be Oracle may be not, took the vision a step further? What direction would such a step take? How about writing the *engine itself* to multi-core/processor + SSD machines? How would such an engine differ from today's versions? Well, at minimum it would have an optimizer and execution unit which are parallelized. Some (most?) engines are parallel only to the extent of being able to execute multiple queries at once, one per thread (generally). What is not common is parallel execution of a query into the datastore.

With rising processor/core/thread counts, blasting a query through many threads begins to make sense, if the datastore can respond fast enough. The existing model of concurrency, whether the locker model or MVCC, seeks to minimize the time that a given row is locked. Row level locking developed to make this possible, and MVCC developed to make it "irrelevant". Both approaches are based on the concept of a conflict serializable schedule (Weikum & Vossen, pg. 92 et seq). In practice, the engine does what the COBOL/VSAM coder used to do: iterate over a bunch of "records" doing stuff one at a time. The RDBMS presents to the client an "all at once" facade, as Dr. Codd demanded, but there's just a really fast squirrel spinning the cage.

But matrix operations are what Dr. Codd really meant; one can view the relational algebra as essentially linear algebra for a limited domain. What if a RDMBS vendor looked at current, and near future, machines rather than past machines? Current databases are still based on uniprocessor slow disk technology. The emerging machines have been obvious for five years, anyway. How far along could, Oracle for example, be toward an engine that *demands* X number of cores and Y amount of SSD primary storage? Could you build a database engine from scratch in five years? Abso-freaking-lutely. OS/360 was written in less time, and in assembler. So, yes, this could be the year. My guess: Microsoft.

Losing My Religion

2011-01-06T10:42:00.000-05:00

I don't know what OCZ is supposed to be an acronym for (don't end sentences preposition with), but 'Oly Crap Zoroaster will do. They're at CES, and here's some info from our friends at AnandTech. Note that the thing screams even with compressed data. Hmmm.

Later in the article is the toss-away:
"On the other end of the spectrum, OCZ presented an even bigger (physically) drive: the IBIS XL. Now this isn't going to be productized, but it's simply something to test the waters with. The IBIS XL fits into a standard 5.25" drive by and starts at 4TB."

As I've mentioned a few times, Zsolt at storagesearch makes the case that sooner, not later, SSD will even take over the petabyte world of storage. I've always thought that a bit daft. Now, I'm not so sure it's daft.

Gimme a B! Gimme a C! Gimme an N! Gimme an F! What's it spell??? The future. (This is where I toss the cute blonde cheerleader over my shoulder.)

Is That Oz Up Ahead?

2011-01-05T21:40:00.000-05:00

Is that an Emerald City over the horizon? Are we almost to Oz? Yes, yes we are.

The last week has brought a host of news, which taken together, indicate that the near future is nearly here. If I keep stepping half-way to the wall, do I ever get to the wall? Yes, yes you do, to within any delta you wish to name.

First, there was the Windows on ARM announcement.

Next, we have two reports from CES, courtesy AnandTech: the next Tegra, and its successor.

The future is clearer: it will be a pixelated VT-220 connected to a wireless network and thence to a relational database stored in BCNF on SSD. Such applications (not ported, in the common way, from COBOL) will run rings around all that legacy file based stuff. With sufficient bandwidth, and a persistent connection (your phone is, right?), it's back to the future. No, I don't believe that the phone/web (or web/phone if you prefer) paradigm is the winner. The flexibility and accuracy of the persistent connection paradigm will win.

For those who think that the web is really great progress over what came before, you need to know the simple history. I'll start with the consolidated mainframe world of the late-60's. There were mainframes and terminals (3270 is the archetype), over a disconnected connection. There were local edit programs written into the 3270. The transfer was block mode, meaning that all keyboard activity was between the user's fingers and the local edit program. Hitting Send (what we now call Carriage Return/Enter) sent off the edited screen to the mainframe app code.

I've just described a browser/html application.

Then, along came Unix and the VT-100 (there were later, more capable VT's, the VT-220 in particular). While connected by a wire, just as a 3270, this wire is always on. Ah. As a result, database engines and application code residing on the server see each keystroke on the VT-220. In fact, it was common to have 4GL's resident with the RDBMS in what was often referred to as "client/server in a box": the database engine being the server in its patch of memory, and a patch of memory being the application code for each client connection. Blindingly efficient, and allowed the database and the application code to edit/check *character by character* as the user progressed; if that sounds a bit like AJAX, well, yes it is. Not quite as cheap in memory, but this was the early 90's and later, when memory began to get cheap. Supporting 1,000's of terminals (or PC's running a terminal emulator) was not uncommon.

Then, along came the www, and young-uns thought this was new and fabulous. Negatory.

With the increasing density of chips we see an artefact of Moore's Law not often (except por moi) remarked: look at block a diagram of recent multicore chips. Most of the real estate is dedicated to various caches, oddly it seems, rather than using all those transistors to execute native instructions in hardware, the trend has been to emulating, for example, X86 instructions in a "hidden" RISC machine. Not that this is new; the 360/30 was widely believed to use a PDP-11 to run the instruction set. IBM did acknowledge that most of the 360 series emulated the instruction set; only the top end machines executed in hardware. The upshot is simple: few, in any, normal client machines have need for the compute power on tap.

So, we see the rise of ARM and MIPS (you did buy ARMH when I told you to, yes?) running minimalist machines at low power and low cost. With a persistent connection to a persistent datastore, let's dance like its 1995. And we will. You can't fool Mother Nature, and She's saying: keep the data and its management in one secure place, and let the kiddies make pretty, pretty pictures on their phones and pads. Just don't let them mess with the data.

Just stay away from the poppies. You heard me. Don't go into the field.

Scotty's Wisdom

2010-12-28T10:05:00.000-05:00

Message boards can actually be useful to the exercise of figuring out where an industry is going. STEC is the principal publicly traded Enterprise SSD vendor, so it is the public bellwether with respect to "Enterprise SSD". They've been segueing from SLC dominant to MLC dominant product mix, which ends up being a topic of discussion, especially recently. A thread is running now about the qualification of an MLC version of STEC gold standard "Enterprise SSD" (ZeusIOPS). I was moved to contribute the following:

"I canna change the laws of physics"

That will be true in the 23rd century and is true now. The number of erase cycles of MLC is fixed by the NAND tech, controller IP can only work around it, usually by over-provisioning (no matter what a controller vendor says). Whether STEC's controller IP is smarter (enough, aka, at the right price) is not a given. As controllers get more convoluted, to handle the decreasing erase cycles (what? you didn't know that the cycle count is going down? well, it is, as the result of feature size reduction), SLC will end up being viable. Cheaper controllers, amortized SLC fabs.

If STEC (or any vendor) can guarantee X years before failure, then the OEMs will just make that the replacement cycle. It would be interesting to see (I've not) the failure distribution functions of HDD and SSD (both SLC/MLC and STEC/Others). Failure isn't the issue, all devices fail. What matters is the predictability of failure. The best to have is a step function: you know that you have until X (hours, writes, bytes, whatever), so you replace at X - delta, and factor that into the TCO equation.

I think the failure function (in particular, whether and to what extent it differs from HDD) of SSD does matter, a lot. Consumer/prosumer HDD still show an infant mortality spike. Since they're cheap, and commonly RAIDed, shredding a dead one and slotting in a replacement isn't a big deal. Not so much for SSD, given the cost.

I found this paper, but I'm not a member. If any reader is, let us know. The precis' does have the magic words, though: Gamma and Weibull, so I gather the authors at least know the fundamentals of math stat analysis. If only there were an equivalent for SSD. It's generally assumed that SSDs are less failure prone, since they aren't mechanical; but they are, at the micro level. Unlike a HDD, which writes by flipping the flux capacitor (!!), the SSD write process involves physical changes in the NAND structure; which is why they "wear out". Duh. So, knowing the failure function of SSD (and knowing the FF for NAND is likely sufficient, to an approximation) will make the decision between HDD and SSD more rational. If it turns out that the FF for SSD moves the TCO below equivalent HDD storage (taking into account short stroking and the like to reach equivalent throughput), SSD as primary store becomes a value proposition with legs. Why the SSD and storage vendors aren't pumping out White Papers is a puzzlement? May be their claims are a tad grandiose?

The ultimate win will happen when MRAM (or similar) reaches mainstream. Very Cool.

Mr. Fusion Powered Database

2010-12-23T15:06:00.000-05:00

I keep track of my various database interests with a gaggle of sites and blogs. For PostgreSQL, I've signed up for the Performance sublist. It's mostly about fixing things in parts of the engine in response to questions like: "my query is slower in PG than it is in SQL Server; how come?", and such.

A thread started today that's of interest to this endeavor. It started out with one person wondering why a Fusion-io drive runs so fast, but PG doesn't run any faster. Then another chimed in to say he was setting up PG with Fusion-io drives. Looks to be an interesting discussion. Now, if only my employer would let me buy some of those Fusion-io drives! BCNF to the rescue.

Here's the list.

The thread is titled: concurrent IO in Postgres?

Django Played Jazz

2010-12-22T09:24:00.000-05:00

The PostgreSQL site has been linking to this blog a bit recently, and he's refreshing. I'm going to spend some time looking into it. It could be there's some intelligent life out there after all.

Here's the start of today's entry, if you didn't slide off immediately:

Don't retrieve a whole row just to get the primary key you had anyway. Don't iterate in the app; let the database server do the iteration for you.

And he signs off with this (my heart went pit-a-pat):

but far better is to make the database do all the work

It is shocking how many coders still insist on their for loops in code. I mean, Dr. Codd made that obsolete, in the sense of providing an abstract declarative data model, in 1969/70 (the year depends on whether you were inside or outside IBM then). In a few years, Ingress and Oracle were live. I've concluded that MySql, PHP, java, and the web generally is what motivated the regression to COBOL/VSAM paradigms (that is, data is just a lump of bytes which can only be accessed through bespoke source code). One didn't need to know much, and frankly most webbies didn't and don't, about data to build some snappy web site that just saves and moves gossip. I suppose that's OK for most of the juvenilia that passes for web stuff, but not for the grown ups.

An Once of Prevention

2010-12-16T22:05:00.000-05:00

Andy Lester is a Perl coder, and I loath Perl, so there has to be a good reason for me to mention him. And that reason is a posting of his, linked from Artima, which contains the following:

This person was one of those programmers who tried for the premature optimization of saving some typing. He forgot that typing is the least of our concerns when programming. He forgot that programmer thinking time costs many orders of magnitude more than programmer typing time, and that the time spent debugging can dwarf the amount of time spent creating code.

Now, when I was young and impressionable, the notion that a developer is paid to think, and not to type, was widely accepted. But I've certainly noted that in recent years, java perhaps the culprit, lots o typing is now the metric. LOC rules, even if most are worthless. Moreover, development by debugging is also normative. Ick.

What might this have to do with the point of this endeavor, you may be asking? Simply that declarative data is so much lazier than typing, and that a BCNF schema is easy to modify (since there aren't covariances to worry about). It does require some forethought, what Spolsky calls BDUF (you should look it up, if it's foreign), but that forethought isn't carved in stone, rather a strategic battle plan which permits many tactics. The "Agile" meme appears to have eaten its children, in that its zealots really, really do believe that all projects can be built from daily hacks by masses of coders. Ick; double Ick.

Pundit for a Day

2010-12-15T21:59:00.001-05:00

I've been reading Cringely for decades, and especially, along with most who read him it turns out, his annual predictions. Since leaving his PBS gig, he hasn't been doing them. Sniff. But today he announced that he would do another, and invited his readers to contribute same. Well. Not one to turn my nose up at the possibility of 15 seconds of fame (he allowed that any reader predictions would be printed with attribution, which he sort of has to do) I offered up what follows.

Just one, sort of.

I've been banging a drum for SSD for a number of years, at least since Intel released their flash version (in true Enterprise, Texas Memory has been shipping DRAM parts for decades, but that's another story).

When STEC, Violin, et al started to build "Enterprise" flash SSD those few years ago, the notion they promoted was that SSD would replace HDD, byte for byte. That didn't happen, largely IMO because the storage vendors (SSD makers and storage OEMs) couldn't develop a value story.

There always was a story: the Truly Relational RDBMS (not the flatfile dumps common in Fortune X00 companies which moved their COBOL/VSAM apps to some database) is (so far) the only thing which exercises the real strength of the SSD: random IOPS. But to get that benefit, you have to have a BCNF (or better) database, and join the shit out of it. The COBOL/VSAM and java apps devs don't think that way; they love their bespoke written loops.

So, what we've got now is SSD as front end cache to HDD arrays. And SSD as game machine and laptop speed up. Enterprise hasn't yet bought SSD as primary storage. Hmmm.

In 2011, we will see that. My guess is Oracle will be the lead. It works this way. Larry wanted Sun, not for java or MySql, but the hardware stack. What Larry needs is that last group of holdouts: IBM mainframe apps. To do that, he needs a credible alternative to the z machine ecosystem.

He has that now, but it ain't COBOL. He needs a value story to get those COBOL/VSAM apps. Whether you buy that Oracle is the best RDBMS or not, Larry can make the case, particularly since his competitors (save IBM) have adopted the MVCC semantic of Oracle.

Pushing highly normalized databases, with most/all of the business logic in the database (DRI and triggers and SP) running on SSD makes for a compelling story. But you've got to spend some quality time building the story and a POC to go with it. Larry's going to do it; he hasn't any choice. And it makes sense, anyway.

Remember, a RDBMS running on SSD is just an RCH from running an in-memory database. You don't need, or want, lots of intermediate caching between the user screen and the persistent store. Larry's got the gonads to do it.

Regards,
Robert Young

Hand Over Your Cash, or I'll Shoot

2010-12-09T22:15:00.000-05:00

The recent events led me to consider, yet again, the SSD landscape. The point of this endeavor is to promote the use of SSD as sole repository/persistence for BCNF databases. The reasons have been written to a great extent.

Since beginning this endeavor, there has been a clear shift in storage vendors' marketing of SSD, whether this shift was proactive or reactive, I do not know. These days, there is much talk of tiering and SSD as cache, less talk of SSD as whole replacement of HDD. Zolt, over at storage search still promotes the wholesale replacement angle, but he seems to be in the minority. I stopped over to copy the URL, and there's an interesting piece from 7 December worth reading, the column headed "MLC inside financial servers new interview with Fusion-io's CEO" (the way the site works, the piece will likely be hard to find in a couple of weeks, so don't tarry).

So, I reveried into the middle of a thought experiment: what difference, if any, does it make whether an SSD is used as sole/primary store or as cache? Well, I concluded that as cache, dirt cheap SSDs are just as good as Roll Royce SSDs (i.e., STEC, Violin, Fusion-io, and the like) from one angle, for the simple reason that the data on the SSD is really short-term. From another angle, those cache SSDs had better by really high quality, for the simple reason that the data is churned like a penny stock boiler room, and SSDs need robust design to survive such a pounding.

The OCZ contract announcement leans toward the first answer; $500 doesn't buy much (if any) STEC SSD. With error detection and hot swapping in the array, just pull them as they die and toss 'em in the shredder. I'd unload any STEC shares real soon now. There'll still be full-blown SSD storage for my kind of databases, but the American Expresses are more likely to go the front-end caching route (they've no stomach for refactoring that 1970's data), and for that implementation, commodity (this soon???) SSD is sufficient.

Like a Rolling Stone

2010-12-09T09:11:00.000-05:00

One Hit (To The Body), STEC's and Compellent's that is, appears to have happened today. And, I'll 'fess up, I never saw it coming. OCZ, which had looked like a mix of prosumer/consumer SSD builder, is now shipping Enterprise parts. Who knew??? And the stated price is $300-$500. Either some OEM is willing to take a really big chance, or the cost of Enterprise SSD just went over a cliff.

Here's hoping for the latter. Do you get it? SSD for only 2 to 3 times the cost up front!! Not the 10 times (or more) it has been. And if the buyer is EMC???? STEC's corporate sphincter just got puckered.

The devices are the Deneva line, which they only announced back in October? They run SandForce's SF-2000 controllers, and will be shipping by February!! "Watson, the game's afoot."

As I was composing this missive, came word of the Compellent smash, and it's not a technical problem. Compellent is an early adopter of STEC SSD. Months ago, you may remember, a rival, 3PAR, was the hockey puck in a takeover game. Compellent's share rose, a lot, in sympathy, and kept going. Today's story has it that the company is going to Dell, but for substantially less than the share bid up by all those plungers. Irrational exuberance strikes again.

Simple Simon Met a Pie Man

2010-12-08T16:06:00.000-05:00

Another case of an interesting thread and an interesting post (mine, of course). And, once again, it's from Simple Talk; on this thread.

Since you mentioned it, I'll beat the drum, yet again, for the necessary paradigm shift (in many places, anyway) which small keyboardless (in practice, even when one exists) devices.

- Mostly, it was seeing that the best existing tablet and smartphone apps do simple, intuitive things, using simple intuitive interfaces to solve single problems.

As I've said here and elsewhere for some time, by (re-)factoring databases to high normal form (narrow tables, specifically), one gains a number of advantages.

1) such schemas are inherently candidates for UI generation, due to DRI
2) they're inherently robust, due to DRI
3) they're likely most of the way to being "pickable", which is what tablets do
4) given the ability to host high normal form databases on SSD, then building them to such a UI is feasible

Tablets have a long history, in fact; the iPad is nothing new, except for its venue. Those doing VAR systems that work in warehouses have been writing to RF tablets for a couple of decades, and designing to high normal form (or, as often, working around its absence).

Pink Floyd

2010-12-07T08:56:00.000-05:00

For those following that other story, what's Larry up to with Sun, I've been in the it's-about-taking-down-the-mainframe camp from the first nanosecond. It's been kind of a small camp, amongst the Usual Pundit Suspects. But today comes a bit of news along that line of thinking.

It's becoming clearer that Larry wants a database machine that can slurp up all those renegade IBM mainframe folks. He knows he's got to get them off COBOL, somehow too, but first he needs a credible stack. Another brick in the wall.

Tin Man

2010-12-04T22:10:00.000-05:00

I've met the Tin Man. Whilst looking for some MVCC/Locker debate I happened onto a Sybase Evangelist blog. Kind of like what I do, but he gets paid for it. Sigh. May be soon.

Anyway, this post is his paen to SSD, and this:
How big is your database?? **light bulb** Those same 10 SSD's get you a whopping 300-600GB of storage. You could just put the whole shooting match on SSD and forget the IO problems. Rep Server stable queue speed issues - vaporized.

Be warned, he makes the leap from Amazon sourced SSD to enterprise database storage (he doesn't mention STEC or Violin, for instance, but appears to be aware of earlier Texas Memory DRAM units); not really going to happen, so his arithmetic is off by a decimal point. But otherwise, he and I are on the same page, especially with skipping the "cache with SSD" silliness, and just storing to SSD. Schweeet. And he knows from schmere.

Now, hold Dorothy's hand.

Thanks for the Memory

2010-12-02T09:14:00.000-05:00

AnandTech has an article about 25nm flash today. Well worth the read. I'm not sure how this affects this endeavor. On the one hand, the physics of smaller feature makes flash less worthy of enterprise storage. On the other, the increased density supports greater over-provisioning to solve, maybe, the problems. A classic engineering problem.

I stopped by Unity Semiconductor to see if there's any news on shipping of their "new" device. They include this article. If it works, SSD has some calm seas and wind at its back.

You've Earned a Good Thrashing

2010-11-24T11:18:00.000-05:00

One of the aspects of RDBMS on SSD that has been worming around in my lower brain stem for a bit is, what difference does it make whether the engine is locker or MVCC? Now, for those just joining the partay, my database of preference has been DB2, with SQL Server and PostgreSQL and MySql as adjunctants, for the last decade or so. DB2 is the last major database sticking to locker semantics; with good reason, so far as I am concerned. The engine implements a deep locking scheme, and fail fast by the database is smarter than fail late by the user.

The question which as been nagging me is, for MVCC semantics, that data is spread out among both the tables and supporting storage. In Oracle's case, these are rollback segments. MVCC, to use my term, is Read Last Committed semantics; and to do that, the engine has to keep track of changes on the fly such that any query can get any rows from any table *as of* some time/commit/transaction (take your pick of term).

My worry is that garden variety SSD may not be up to storing rollback segments, due to the heavy writing. On a HDD, it's no big deal. But this method of storage necessarily slows down the engine.

In looking for answers, I came across this paper from 2008. You can skip down to slide 22 for the specific discussion. The paper doesn't present evidence, one way or another, about this concern, but does show that putting version data on SSD is a huge performance win. Here's an update to December, 2009 from the main author.

So, in all, I haven't found any clear answers from the literature. What seems clear is that garden variety consumer SSD wouldn't survive (not that I'd ever recommend such parts anyway), and I'm not so sure about prosumer parts. The STEC's out there, not concerned.

Food Fight

2010-11-18T21:15:00.002-05:00

I do so love it when the Kiddies finally figure out databases; well a couple and a little bit. This PHP post came across my bow, and it's just too much fun not to pass on. Thing is, the blogger looks to be a bit of a young-un. And he takes a good deal of heat from the ridgebrows, but doesn't back down. Good for him. There is hope for us all.

An Open and Shut Case

2010-11-16T15:35:00.000-05:00

There was an OpenSQL camp up in Cambridge last month, and I considered going, but the agenda listed far too many NoSQL projects. I decided that I'd just spend a long weekend being irritated. Today I read Josh Berkus's write up on lwn.net, and this was the one nugget:

Some of the SQL geeks at the conference discussed how to make developers more comfortable with SQL. Currently many application developers not only don't understand SQL, but actively hate and fear it. The round-table discussed why this is and some ideas for improvement, including: teaching university classes, contributing to object-relational mappers (ORMs), explaining SQL in relation to functional languages, doing fun "SQL tricks" demos, and working on improving DBA attitudes towards developers.

There are times (quite often, truth be told) that I wish the relationalists had the gonads of the Right Wingnut Zealots or Tea Baggers or what-have-you. The RM isn't just another data store. By attempting to make nice with folks who refuse to listen, you'll just tick off the folks who do get it, but won't convince those who've no intention of being swayed. Use relational databases to make better systems than the knuckleheads who belittle them. Don't get mad (well, a bit some times), get even.

If you go and read the piece, you'll find the attendees worrying about problems that the commercial vendors (with lots more folks, of course) dealt with years, if not decades, ago. Some of the problems are fully discussed in textbooks; Weikum & Vossen in particular. There Ain't No Such Thing As A Free Lunch. If you want minimal byte footprint, maximum structural integrity, minimum modification hassle, then the RM as embodied in current industrial strength RDBMS's is the way to go. Open Source databases can do the same, so long as they concentrate on implementing the fundamentals, and stop worrying about pandering to the FOTM programming language. Languages will come and go (COBOL and java relegated to Big Business), but the data is forever. Best put it some place safe.

Convicts and Cane

2010-11-09T14:45:00.002-05:00

If you're of a certain age, or were precocious at a young age, you may be familiar with the following lyric: "In the early part of this century, convict labor worked the cane fields on the bottoms of the Brazos river... Go down old Hannah don't you rise no more, if you rise in the morning, bring the judgment day".

I've no idea whether the Thought Leaders at AMD are familiar with old folk songs, but they've labeled its latest Intel beater Brazos. The good folks at AnandTech have some details. While this is a notebook implementation, and not especially pertinent to this endeavor, the graph on page one surely is. I gather it represents AMD's view of machine development over the next years, and that AMD will have victory on judgment day.

That last curve, for what AMD calls Heterogeneous-Core Era, is Brain Viagra. "Abundant data parallelism" is what the SSD/BCNF database is all about. Another step along the Yellow Brick Road. It will be a fun journey.

Ahhhhhhhhh. The Stay Puft Man

2010-11-09T10:30:00.001-05:00

I once knew a man, father of a friend of mine, who remarked that one of his sons "costs me a lot of money"; private school and all that. Applications designed to be difficult to maintain are a lot like prodigal sons, the money just seems to fly out the window. Coders remain adamant that their API's are what make software easy to maintain. Baloney. The proliferation of code, much as it was in the 1960's (when COBOL was going to make application development so easy, a manager could do it), is justified on the grounds that the latest New Thing in coding will make all the angst go away. Hasn't happened, now has it? Perhaps we should stop looking to bloatcode for the answer.

SQL Server Central has an article on maintenance. I was moved to post a reply, herewith entered for your approval. Another opportunity to ring the bell for SSD/BCNF systems.

In the world of commercial/business software, aka database systems, the answer to maintenance costs is to embrace SSD/BCNF. Why, you may ask? Let me count the ways.

1) by putting the data and its integrity logic in one place, the server, letting the client code be responsible only for screen painting and data input; one small group of smart database geeks keeps the data under control. compare to the human wave approach of client-side coding. in fact, embracing SSD/BCNF means the data is utterly agnostic to the client code. could not possibly care less whether it's a java screen, or VB screen, or csv file. makes no difference. I offer xTuple as an example; not yet a SS application, not that this matters much.

2) by embracing SSD/BCNF, maintenance amounts to adding columns/tables/rows (a row is a business rule, remember) to the schema. the data hangs together on the RM.

3) by embracing SSD/BCNF, client side code can be generated from the schema. not saying it has to be, or should be (well, yeah, it should), but it can be. at the least, clients should interact through SP.

4) with significantly (and soon to be, massively) parallel servers available for small bucks, what's the most adept application for such machines? well, the relational database engine, of course. client code, not so much; as client coders are discovering.

5) by embracing SSD/BCNF, the byte footprint is an order of magnitude less than it is with the flat-file storage so beloved by C#/java/VB/COBOL coders.

6) as Celko (at least) has written (in "Thinking in Sets"), using auxiliary tables to implement constraints makes maintenance still simpler: just add (or delete) rows to update constraints. for that matter, authorized users can update check constraints and the like from screens; such constraints are just text stored in the catalog.

That should do it for now. Remember, the cost of maintenance is *directly* a function of the code/data structure. The more obscure that structure, the higher the cost. Historically, for those that have been paying attention, coders view life (largely because those that employ them are dumb enough measure them so) as a LOC exercise. Anything which increases the LOC future is good; likewise, anything which decreases LOC future is bad. Those that employ them often take the same view, though few will admit it. The reason is that such organizations are inherently bureaucratic, and in that environment the one with the deeper org chart gets more money ("I manage 5 managers and 100 staff, you've only got 3 and 50"). Efficiency and productivity really aren't the goal. The hardware and software to solve the issue, in the commercial world, has existed for decades, yet the COBOL/VSAM paradigm persists; only the syntax has changed. That's not an accident. The RM and RDBMS are actively opposed in many shops just because fewer coders would be needed, to do all that maintenance that CIO's complain about; which fact keeps the CIO's org chart growing. Hmmm. Curious.