Fellow DBA: There are so many alerts! It takes me a long time to get through them all. How do you filter through them?
Me: That’s called “alert storming”, and it’s bad for that exact reason.
Alerts are like money: People always seem to have way too little, or way too much*.
If you have too few alerts, of course things will go terribly wrong and no one will notice it until customers come calling**. If you have too many, though, that’s about as bad. The really important alerts get buried in an avalanche of meaningless emails, and IT folk are very quickly trained to pay zero attention.
Dealing with an existing, ongoing alert storm is a process:
First, fix the problem
Go through groups of alerts. See if you are able to address them. In other words, fix the problem.
- Maybe the problem is that the alert threshold is set too low. So, adjust it.
- Or, are you consistently getting “low on space” alerts for the D drive? Then add space or move files as appropriate!
- The “NewbApp” login keeps failing? Find out where it’s coming from, and deal with it!
Those alerts are there for a reason. Unless, of course, they aren’t.
Next, clean house
If the reason for the alerts can’t be fixed, isn’t really a problem, or the alert is otherwise useless, find out if you can modify those alerts so they don’t go off as much, or at all. This will require a conversation with whoever makes those decisions – a project lead, a senior DBA, or a manager, depending on your shop. You will find, as I have, that businesses get very tetchy if you just go around changing their alerts system willy nilly.
But back to useless alerts. Why oh WHY are you getting 534 alerts per day about forty separate jobs, each of which is running successfully? Alerts aren’t meant to be reassuring; they are not a pat on the back. They are ALERTS. Have you ever noticed that there is no such thing as a “nothing is on fire” siren in your city, right? How ludicrous would that be?
“WHEEEEEEOOOOOOOWHEEEEEOOOOOOO!!”
“Oh great! Nothing is on fire again!”
Positive alerts are just noise. Get rid of them.
As a last resort, redirect
If, for whatever reason, you’re stuck with alerts that don’t require action, can’t be adjusted, and aren’t allowed to be altered/reduced/redirected, then set up Outlook filters to “file” them to your trash folder. Make sure these filters are VERY specific, so you don’t miss something important.
Years ago, there was a VP who was in love with alerts. After an extended discussion that would have been the very echo of this blog post (had I been blogging at the time), I finally asked him WHY he wouldn’t get rid of the success alerts, at the very least.
“I just feel better if I know things are running.”
At long last, I gave in, and set up an extended filtering system on my email. After that, I was the only one at the place who ever know if a job failed, a database was corrupted, or indeed if anything important had happened at all.
So, of course, I was the superstar.
After, recover
If you’ve been subject to alert storming for any amount of time, you’re completely jaded. Even an email titled “ALERT URGENT HELP OMG THE SERVER IS DOWN AND HUNS ARE STORMING THE BUILDING” would get a quick click to the X, right?
So the period of time after an alert storm is fixed – or being fixed – is your time to live the examined life. Consider each and every incoming alert: does it need a fix, an adjustment, or does it need to be filtered?
Your new alert mantra is: Fix, Adjust, or Filter
A few weeks of this, and alerts will start to look like things of importance again, because they will be. Continue to remain vigilant against the slippery slope of alert storm, and enjoy your (relatively) clear inbox.
Happy days,
Jen McCown
http://www.MidnightDBA.com/Jen
*Admittedly, “way too much money” is a not only rare, but desirable state. Okay fine, alerts really aren’t like money at all. They’re like…um….they’re like pets. No, wait, they’re more like…Tweets. Yes, alerts are like Tweets. I’m sticking with that. Oh! Or like calls from your mother.
**Calling, screaming, crying, threatening….you get the idea.