The US government is a pretty big organisation, doing a lot of important stuff. Its spending on Information Technology is north of $75 Billion says Vivek Kundra the federal CIO. Like IT projects everywhere, big projects are subject to big failures. As Mr. Kundra says;

Over the past several years, we have witnessed numerous public failures of major information technology systems and just last year saw roughly one third of all investments reported as poorly planned or poorly performing. Many of these investments may never deliver on their original promises.

It is not news to my readers that implementing information technology, and data management is not trivial. The United States government has recently launched an IT specific cost site, it.usaspending.gov. I think the new approach by the US adminstration to open up the numbers, and make the data available is fantastic, and I wish my government here in progressive Canada would start sprinkling some of this kind of data access goodness around. Of course its very high level data- but its data, and it outlines where the $75 billion or so are going.

The best feature of this new site (in my view) is that there is the capability to download the data, either in a simple flat file, or to set up an XML feed via RSS.

The site also includes some visualisation capability, although I found it to have a few rough edges. The trendy visualization controls are there with a time animated bubble chart, and a tree map, but I found the controls to be a bit rough in a few places (pick lists with lots of items sized to see only two at a time and tool tips that weren’t rendered in my Chrome browser) But it’s more than just a token effort, and hopefully it will continue to evolve.

One glaring transgression in the eyes of the visually inclined was the use of pie charts and horizontal stacked bars on the main page; someone needs to read a bit of Stephen Few – this one looks like the “common mistakes” examples in one of his books.

Oh, NO! Not a pie chart, I gotta talk to these guys.

And of course, since I’m hardly ever satisfied with the raw data, I had to do a bit of desktop data transformation in the Datamartist tool to see just what the feds are spending things on.

Since the data had only limited categorization, I used Datamartists segmentation block and its text parsing functionality to create a new segmentation column based on the content of the project description field. With this, I was able to take at least a high level look at how much data warehousing or business intelligence was going on. Looks like it’s largely integrated into various projects, “only” about $100 million a year in a handful of projects that specifically contained terms like data warehouse or data mart etc.

What is interesting, is that almost 40% of the spending is in projects that use the terms “consolidate” or “integrate” in the descriptions- obviously the federal government is no more immune to data and system silos than the rest of us.