A Data Age proclamation

mac-glassesTwice in two days now, I’ve gotten in discussions with people about the state of data usage…the idea that we have absolutely massive amounts of data at our disposal – especially us, as if you’re reading this, you’re likely a DBA or at least an IT professional – but that people are not using data to its fullest extent. To use an overused word, companies, groups, and individuals are not leveraging data enough.

The Data Age

This most recent talk on the subject was spawned by Jorge Segarra (blog, Twitter), who posted an image from a book (“Signal”, by Stephen Few). The text said, in part:

“IT tools can at best assist us in performing work that we already understand. … The information age continues to exceed our grasp. At best, we live in the data age. We have made great strides in accumulating data. By comparison, we have made little progress in using data to inform better decisions.

(Bolding is mine, italics are the author’s.)

That hits the nail precisely on the head. One of the most egregious examples of this is the non-use of data in healthcare. Any data professional who spends five minutes thinking about the potential power of data in healthcare will get really excited, really fast. Universal doctor access to medical history! Alerts on drug interactions before they’re administered! Data mining for unusual trends in symptoms and diseases! Connect the medical dots, man, this is awesome! My fellow MCM and spouse Sean has a whole lot to say about why this sort of thing generally doesn’t happen in healthcare, so I won’t repeat it here. Suffice to say, the healthcare industry is not leveraging data.

This sort of thing is repeated in most industries across the board. Some financial firms might actually be ahead of the curve when it comes to getting intelligence out of data, I don’t know…but it would make sense. There’s money to be made immediately and directly by determining trends and patterns in the world of, well, money.  But in most other industries, what we’ve got is a long, slow march toward actual data intelligence.

(As a side note, this is why I really am pleased to see Business Intelligence exploding on the scene in the last few years. Companies should be excited about the possibilities. Go with godspeed, all you BI folks, to your new positions.)

Changing the world

So why am I talking about this today? Mostly because it’s on my mind.  And, because I’ve been thinking a lot about the problems associated with organizations with data management solutions (like SQL Server and all the attendant BI integration). The issues range quite a lot:

  • politics, which is always big fun
  • funding, which in truth is a matter of perception; a company will fund those things it views as important
  • a deep misunderstanding of the potential of data
  • a lack of understanding of the inherent roadblocks to data intelligence

We – meaning Sean and myself – aren’t BI guys. We are company owners, but we’re already leveraging the hell out of the data that we have. We’re not in a position to change the world in a single stroke, but we can do what we can in our small corner. And in our small corner, what we can affect most is that last bullet point: the roadblocks.

We’re doing two things: educating, and tool building.

Education

First, we are educating.  We give sessions at technology conferences that include elements of data philosophy (wouldn’t that be a good 201 class? Someone sign me up…) that aren’t common in the business world. We record those sessions and put them online for free, and point more people back to them. The sessions are about technology, but we also get an approach across.

I’ll give you one specific example: administration is a huge roadblock to data intelligence. If the data isn’t quickly accessible (indexing, maintenance, configuration), or if it isn’t there (backups, HA/DR, security), then you’re not going to have much luck mining the stuff and tying it into other data.

Part of the problem with data admin culture (in both companies and in tools vendors) is the round-robining of servers. Data philosophy point 1, therefore, is the holistic enterprise. Any task that requires a DBA to hit 20, or 100, or 10,000 instances one by one (or by some other inelegant means) is a task that’s not going to get done.

A more thorough discussion would take more room than I want to use here, but anyone who has click-click-clicked through a vendor tool – SQL instance by SQL instance – has an idea of what I mean.

Tools

Second, we are building tools. Yeah, I just said that we’re building tools to change the data world, and I mean it quite literally.

(A side note on this: it’s old even to my ears to hear “our tool will change the game!” from a vendor. I know that some vendors really do believe their tool is a game changer, but I also know that too many others use the same words just because. But there are only so many words available to describe it when you’ve Gotten Excited and Made Something.)

A database administrator* has several essential job functions, and both ideology and the RDBMS itself get in the way of those jobs. Common security tasks and auditing is maddeningly complex. Maintenance is weak and default configurations are actually detrimental to an environment (who on this planet actually wants 10% file growth, I ask you??).

And Microsoft (and other vendors) have been trying, but they haven’t addressed the problems well. Each new solution forces a DBA to do that round-robin across all their servers, presents a pretty GUI that you have to stare at to get any use of, and most of all treats the DBA like an everyday end user.

You know why DBAs aren’t end users? Because they can do amazing things with collected data. But again, any tool that gathers data either hides it, or sticks it in XML and/or flat files. How infuriating is that?

We Proclaim …

I won’t give you the hard sell** on Minion Enterprise (or, “ME”). That’s not really the point of this particular tome. I want to start on a list of proclamations that, as it turns out, we have implemented into ME because they should be founding principles of any third-party data tool.

Here’s a start on these principles:

  • A data systems tool should make work easier across many instances, not just one instance.
  • Data should be in a usable format. Flat files aren’t usable to data professionals.
  • Data should be in a usable format, part 2: Administrative data – including security, hardware stats, trace information, etc – must be accessible to the DBA. In a usable format.
  • Any system that generates data that will not be needed in the long term, must clean up after its own damn self.
  • A solution meant to help the DBA in the execution of their job must not be a burden on SQL Server instances. (I’m looking at you, monitoring-software-that-has-a-huge-overhead-on-monitored-instances.)

This is just a start, after several weeks’ introspection on the Minion products and our own fledgling software company. There are more principles out there, already hiding in plain sight in our code.

Happy days,
Jen

*For those of you who don’t believe that the database administrator position will be around in N years: horsefeathers. Data systems aren’t, and will very likely never-ever-ever be, smart enough to take care of themselves entirely. To add to the weight of my “horsefeathers” argument, I say that at the 2015 PASS Summit I paneled with Allan Hirt, Denny Cherry, and Karen Lopez (emcee) on this exact topic, and there was not one iota of disagreement among the four of us. Though the job will continue to morph and grow, DBAs are here to stay.

**More information on Minion Enterprise: we’ve got introductory and topic videos up on YouTube, and of course product documentation and download at MinionWare.net. You can also join the MidnightDBA newsletter here.