28 October 2013

Obligatory Post Mortem: healthcare.gov

Phil Factor has a piece on Knight Capital, to which I was obliged to comment. The comment closed with a bit of snark (you're surprised?):
I haven't yet found more than vague descriptions of the architecture of healthcare.gov (herein, hc.gov), but I'm willing to bet a quid that it's a labyrinth of xml data streams exploding all across the InnterTubes. If ever there were a case for federated/distributed data, that would be it. Just like kumite', it's always easier to cut corners around the right way to do things. At least in the short term.

At first, and second, blush, about right. It's built as a client heavy, javascript heavy, static application, with JSON and Ajax. JSON, aka almost-xml, and ajax to make an applet-like system. Hmm. Moreover, Ajax (XMLHttpRequest) is still considered dangerous, and I'd be willing to bet that many users' browsers either don't support it (all those IE7/8 XP machines) or have it disabled.

Some pointed to Ruby as the culprit, since jekyll is a Ruby gem, but, ah the detail. jekyll generates the static pages (and a boat load of 'em, it seems), but Ruby on Rails doesn't seem to be in the mix. Thank Codd for small favours.

There are references to CORS, in addition to regular ajax. Add in that SSL/HTTPS must be in use during the personal information transfers... Moreover, some published response testing showed that midwest users had worse response than the northeast and west coast. Differential bandwidth? I vote yes. Moreover, the Wiki has this to say (quoting a 2009 document):
However, since SSL is designed to highlight hidden intermediaries, Akamai has struggled to make secure web pages work with their service, and an attempt to connect to a popular website over HTTPS will often reveal Akamai.
Still an issue?

In pre-launch interviews, the front-end folks bragged about needing just two servers (one active, one backup) rather than 32 (or some other number). Talk about penny wise and pound foolish? A dual Xeon, 384 gig, SSD balls out machine can be had for about $17K at System76. They've dropped nearly $100 million on creating the system? Get real.

The front-end 'open source' code has been closed at GitHub, but there're dozens of postings/articles/rants about the structure. Moreover, the real meat of the system, the backend, is closed contracted code from CGI Federal. That part's interesting: a Canadian beltway bandit that got tossed out of at least one contract in its native country. It appears that said backend resides in some sort of Oracle database.

Which leads to another source of sluggishness: commits. It could be that the hc.gov database has to sync with other agency/state databases. I'd wager that's a lead pipe cinch. So, either two-phase commit twixt the hc.gov database and the foreign databases, or some convoluted COBOL/java transaction management. According to hc.gov runs on Apache server. That's fine for simplistic web sites, but no where near right for, what amounts to being, an OLTP application. WebSphere, IBM or not, or another industrial strength webserver. Since it appears that the backend RDBMS is Oracle, then WebLogic. Kiddie koders need to grow up. Not everything on the web is simple search or "eventual consistency". Yikes.

Ultimately an OLTP application, from a front-end culture that is CMS based, and a database which is MVCC. The former knows from nothing about transactions, and the latter thinks they don't matter (at least, the inexperienced). As a pig singing, it's not so much that the singing is done badly, but that any is done at all.

No comments: