Tag Archives: blunders

Admin, Time-savers

An Upgrade Disaster

1 Comment

I got an email from a user at another company today and he told me about the SQL 2012 upgrade they just finished. And apparently it was a disaster. Here’s the general gist of what happened.

They have a huge app that runs their entire business and the vendor talked them into upgrading to 2012. Originally they were slated to do tons of testing and upgrade probably sometime in november. But they decided to not listen to their DBA and instead allowed themselves to be lead by the vendor who told him that SQL upgrade was easy and nothing to worry about. So they did some perfunctory testing and pushed the upgrade to this past week. I know, smart right?

So this vendor did their upgrade for them and it completed ok from what I know about it. The problems came after the upgrade. Now, I don’t have any real specifics, but I do know that it caused a 10hr downtime. One of the directors asked about the backout plan and he was politely told to mind his own business. Everyone is calling the upgrade a disaster. They didn’t have any way to restore in case the upgrade failed in a really bad way… and that means no final backup, no scripted objects, and no mirrored system. This was an in-place all or nothing upgrade.

Just so we’re clear on this, that’s not the way you run an upgrade. Upgrades take plenty of testing from the DB side, and the app side. You should never upgrade anything without extensive testing. And you should always have a backout plan. Now, what does a backout plan really mean? Because I find that often times the backout plan gets overlooked and I think it’s mainly because they have a hard time defining it.

To me a backout plan means a few different things depending on what you’re after. Let’s take this upgrade as an example. No matter how good you think SQL upgrade is, there’s always something that can go wrong. So at the very least, you need to take a final backup of ALL the system and user DBs right before the upgrade. Make sure you kick everyone out of the DB first because it’s not a final backup if there are still going to be modifications afterwards. That’s a good start for sure, and what I’d consider to be a minimum effort. Here’s a quick list of the steps I like to take for an important upgrade such as this:

1. Copy all system DBs to another location. This just makes restore much easier because with DBs as small as most system DBs, you can just drop them back in their original location and you’re good to go.

2. Script all logins with SIDs.

3. Script all jobs.

4. Make sure I have all SSIS pkg projects at the ready so I can redeploy all pkgs if I need to.

5. Do a test restore of the final backup before starting the upgrade.

6. Script any system-level settings like sp_configure.

7. Script any repl, log shipping, mirroring scenarios.

8. Make sure I have pwords to any linked servers. While I try to keep everyone off of linked servers I have to admit they’re a part of life sometimes. And you don’t want your app to break because you don’t know the pword to the linked server. It’s not the end of the world if this doesn’t happen, but it’ll make life easier.

So basically, the more important the DB, the more of these steps you’ll follow. You need to prepare for a total meltdown and make sure you can recover in as timely manner as possible. As I sit here and write this I feel stupid because it seems so basic, but there are clearly those out there who still need this kind of advice, so here it ia.

And if you have a good test box handy, make sure you test as many of these procedures as possible. Script out your logins, etc and restore them to the test box and see if things work as they should. Deploy your SSIS pkgs to your test box and make sure they run, etc. Don’t just rely on what you think *should* work. Actually make sure it works. This is why some upgrade projects take months to complete. It’s not the upgrade itself, it’s all the planning around it. And while this isn’t a full list of everything you could do to protect yourself, it’s a damn good start.
Happy upgrading.

Admin, Development

What an idiot!

2 Comments

As DBAs we quite often run into others who aren’t as smart as us.  The dev is an idiot.  The .net guy is an idiot.  The users are idiots.  The manager is an idiot.  The VP, well don’t even get me started.  And other DBAs are really idiots.  At least that’s how it is in our heads anyway.  We fall into this cycle of calling everyone idiots for every little thing they do wrong.  The dev uses a wrong data type and it makes a few queries a lot slower, what an idiot, he should’ve known better.  A .net guy uses EF instead of putting it in an SP and it causes tons of blocking, what an idiot.  Another DBA tries to fix a DB that’s down and he does something that ends up making it worse… what an idiot.

It’s pretty easy to say everyone’s an idiot when we have the luxury of hindsight isn’t it?  Sure, I could have told you that every single one of those decisions was wrong and why.  But could I have told you before you did it and it went south?  Maybe, maybe not.  I’ve made plenty of mistakes in my career (and am dedicated to continuing) that in hindsight weren’t the best option, but was I an actual idiot for doing it?  Again maybe, maybe not.

I just think we jump on the idiot bandwagon too early and too often.  And I know I’m a big offender.  It doesn’t take much for me to start branding people left and right, but I also try to temper it with some reason.  Just because someone doesn’t have the same experiences I do doesn’t make them an actual idiot.  A dev chooses the wrong data type for a column.  Is he an idiot, or does he just not have the same experience with data types that I do?  I’d have to say it depends on what the mistake was.  Did he choose a varchar(25) for Address, or did he choose datetime?  Because one makes him less experienced with addresses and the other one makes him pretty close to an idiot. Well, what if he chose the bit data type for a SalaryAmount column? Well, I can only hope that he’s writing the table his own salary will be stored in.

I’ve seen plenty of things that seemed to be basic that I didn’t know. And that’s becuase there’s just so much to know it’s hard to quantify. That’s why I make sure I interview everyone I see for at least an hour before making a decision. I honestly believe you can’t judge the sum of someone’s experience in just a handful of questions. In fact, I’ve found plenty of guys who got the first 20 questions wrong and then we suddenly got to their area of expertise and they started blowing the questions out of the water.

So anyway, just give some of these guys a break and realize that they may not be complete idiots just because they don’t know something you don’t. That’s not to say there aren’t any real idiots out there. You guys know that I’ve definitely run into my fair share of them. But I’m trying harder to lighten up on them.

Part of the problem is the learning process we go through, which is next to none. Computers are hard. SQL is hard. .NET is hard. They’re all hard. And yet training is so poor. I’ve seen so much IT training I can’t even count, but the number of courses I’ve been in that actually taught the topic is very few. sure, the high level stuff gets taught, but the hows and whys of doing things is rarely covered. There are some guys out there who really take the time to break it down for you, but try to find one of them. One of the biggest reasons I never got into BI is because all the BI guys teach beginning BI like you’re already a BI expert. They explain BI terms with other BI terms and everyone just nods and smiles. But I guarantee you that most of them walk away without a good understanding of what was just said. .Net guys are big offenders in that area too. They explain .Net to you like you’ve been a coder for years and you’re just supposed to know what all this stuff is. So it’s no wonder that so few people really know their jobs well. They’re never taught what they need to know. So are they really idiots for not knowing something they weren’t taught? There are so many things that can go wrong with a system at any given time how can they be sure that the issue is being caused by a bad data type, or by one particular piece of code? There are of course ways to find out, but so many companies are in such a hurry to move on to the next project they never get a chance to dig into these issues. And again, were they really taught how?

So here we are in the middle of the learning revolution and there’s so little quality training to be had. You can go almost anywhere and learn how to perform the steps for a task, but where do you go to learn what you actually need to know? How do you learn that one thing is stupid over another thing, and that other thing exists for a reason, so when is it supposed to be used? I was talking to someone about this very topic just this morning.

So this whole thing was prompted by a training session I had with someone not long ago. Someone did something they shouldn’t have and when I corrected them they asked why. And when I gave my reason he said oh y, I never thought of that. And I could clearly see that he wasn’t an idiot, he just didn’t have the experience he needed. And since then he’s done it right and even did it the other way a couple times because the situation was different. See, I gave him the reasoning so now he can reason out for himself when to use one method over another. And that’s training that’s worthwhile.

To me, a true idiot is someone who gets shown the way to do things right and still refuses to employ them. He is also someone who has been in his current career for many years and doesn’t even know the basics. I have very little patience for say a SQL dev who’s been doing it for 10yrs and doesn’t even know the basics of the data types. Because you can’t tell me that it’s never come up. I also don’t like DBAs with 10yrs behind them who can’t write a simple backup statement. Again, that’s a basic that you should know cold.

Uncategorized

Healthcare isn’t ready

8 Comments

I just left the healthcare industry for the 2nd time and it’s sad the level of ignornace and superstition that exists around computers… and SQL especially.  The entire industry treats computers like big electronic pieces of paper.  They print things they can easily email, they manually enter in things they could easily write a form for, and they perform repetative manual tasks they could easily script.  It’s pathetic how far behind the industry as a whole is and the people who work in it are so close-minded I don’t see how they ever get anything done.

Part of the problem is the doctors.  Doctors think that because they’re doctors that they know everything.  Several times I’ve had one doctor or another tell me specifically how they wanted me to do something in SQL.  They didn’t know the first thing about it, but they heard a few terms here and there so they decided to run the show.  And here they are in meetings insisting that I follow their HA architecture that was just ridiculous.  I got a reputation in my company for being difficult to work with because I always called them on it and told them to let me do my job.  Then they would complain and my boss would be at my desk the next day.  It’s just incredible ego to think that you’re a expert in all fields because you’re an expert in your own.

However, doctors aren’t the only problem.  Vendors are also a huge problem because they’re very slow to adapt to new technologies.  And by slow, I mean 15-20yrs too slow.  We’ve had so many vendors who only code against SQL2K.  Their support personnel is pathetic to say the least as well.  These vendors know nothing.  And they’re guiding hospitals in their implementations.  And of course now you’ve got the blind leading the blind because while there’s nobody at the vendor who knows what he’s talking about, there certainly isn’t anyone at the hospitals to call them on it.  And when they do get someone in there who knows what they’re talking about they can’t keep them because what really good IT person wants to work with an entire floor of people who don’t know the first thing about IT?

The biggest issue we had with staffing was that everyone who does the hiring thinks that you have to have hospital experience to be able to work in IT at a hospital.  So they end up hiring ex nurses, or other clinical people and give them jobs as programmers, system admins, etc.  These people don’t know the first thing about being in IT or about C# yet they’re given positions based off of their hospital tenure.  So someone who wanted a career change could come in as a Sr. Programmer yet they’ve never even had a simple online coding course.  So now they’re in there trying to figure this stuff out.  They’re architecting solutions that they could barely qualify as end users for.  And anyone in IT who knows what they’re doing has to put up with this idiocy.  And make no mistake… it is idiocy.

The industry itself has too many older managers in it and they need to bring in some fresh blood that actually knows something about IT and how to actually get things done.  As it stands they’re just too scared of the change, too scared of the data, too scared of being sued, too scared of pissing off the doctors, and too scared of technology in general.  Oh sure, they’ll bring in iPads for the doctors to carry around, but big deal.  They’re not doing anything cool with them, and everything they put out there costs tons of money in support because they weren’t put together correctly.  Want a perfect example of how far behind they are?  Whenever you go to a new doctor you still have to fill out all that damn paperwork by hand don’t you?  You have to put your name, address, SSN, DOB, etc on like 9 forms.  Doesn’t that sound like something they should be able to get past by now?  And there’s more to that specific story than just being afraid of computers.  That particular one is caused by the system itself.  I won’t go into specifics though.  I’ve also seen plenty of people print online forms, fill them out, and then scan them back in and store that into the DB in a text column.  Seriously dudes?

So what can they do to change?  How can healthcare move into the 80’s?  For starters they can hire some younger more hip managers who understand how IT works and the benefits it brings, and give them the power to do what they need to do.  Next they can stop hiring from hospitals.  C# coders, or SQL guys don’t have to know crap about your business.  They have to know their business, which is IT.  And they’ll have to pony-up the money for some real IT folks.  IT folks aren’t going to work for peanuts… not when they can go somewhere else and get 20-30K more.  Oh yeah, and you’re also going to have to start treating them like they’re professionals.  IT guys don’t want to hear how much the doctors know about IT.  They want you to let them do their jobs.  So seriously, stop treating them like they’re nothing compared to the doctors.  Doctors are essential to hospitals, but your IT staff is too.  It’s getting so that hospitals are crippled without IT.  So why do you still insist that all IT guys are the same?  Hell, even all janitors aren’t the same.  I can easily tell the difference between one who cares about what he does and one who doesn’t.

Here’s a scoop for you.  Healthcare is going to need to get their act together or else.  The government is mandating that everyone have their health records in a meaningful use format by 2015 so the time of getting by on the idiots you’ve got is over.  You’re going to have to get some real talent and do what it takes to keep them.  If that means paying them a good salary, and listening to them, then all I can say is ‘you poor baby’.  Hospitals jump through hoops all the time to attract some new doctor because of what he brings to the network.  If anyone in healthcare is reading this then you’d better start planning now.  Start gathering some talented IT guys and let them do their jobs.  And NO, before you ask, you don’t know what IT talent looks like.  Get someone to help you find that talent.  And I’m not talking about recruiters either.  Go to the Microsoft MVP site and google someone in the field you’re looking for and start emailing them.  Ask them to help you interview a few guys.  I’m sure they’ll charge you a little, but it’ll be more than worth it.  Then once you get these guys on staff don’t treat them like 2nd-class citizens to the doctors.  You’ve got no choice anymore.  You have to do something.  You can’t keep this up.

My guess is that it’ll probably take about another decade before this starts really turning around though.

Uncategorized

SQL Server 2012 Launch is a joke (#SQL2012)

3 Comments

Ordinarily I don’t jump on these bandwagons and slam something publically like this, but I have no choice.  I’m amped up ready for a launch event and I’m in the office so I don’t have access to porn, so I’ll blog-off this energy. 

Jen’s already written an excellent blog on the issues with the event so far, but I’ll add another perspective.

I signed up to be a moderator in a couple of the expert pods and not only have I had an hour worth of trouble getting logged in, but now the chat window in the pod won’t come up.  Apparently, the site doesn’t work in IE.  REALLY!?!?!?!?!?!? I mean REALLY?!?!?!?!?  Now, I’ve had a couple people tell me to just install chrome and it’ll work just fine, but I’m not going to install a new browser just to watch an MS webcast.  If anyone should be IE friendly it should be the SQL Server Launch Event.  This is just ridiculous.

However, I will say that this is par for the course with vendors.  This is the type of lazy, poorly-executed vendor crap that’s pushed on us all the time in the DBA world, so we’re actually used to it.  I don’t know who this vendor is, but if I did I would do my part to run them out of the business.  So aside from all the issues Jen outlined in her blog, which I’m fully behind, basic compatibility for the customer’s main web platform isn’t even there.

And on top of everything else, I’m hearing from those who were able to get in that they’re not classified as the MVP expert in the chat session, so nobody knows who they even are.  This vendor is a joke and they’ve ruined the launch event.

Admin

2 Addictions worth fighting

5 Comments

A recent email chain with a friend has spawned me to write this post.  What I’m trying to do here is help the 2 extremes of people you’ll find in your company:  the idiot, and the DBA.  Ok, I know all DBAs aren’t super smart, but I had to call the non-idiot something.  So this email chain I had was with a DBA who was stressing-out about something stupid his company is doing even though he warned them.  So what you’ve got to understand is that both sides of this equation are an addiction.

The Addiction of Stupidity:

It’s easy to get caught-up in the fact that our users are morons, or that the other IT guy is an idiot, but you’ve got to have a little compassion.  You have to understand that stupidity is an addiction.  It may have started through a combination of them not knowing anything at all about SQL, and them not having any qualified DBAs to help them learn.  How it started though doesn’t really matter in the end.  What does matter is that now you’ve got to help them through it.  You need to advise them on the proper way to do things.  At first they won’t listen.  Keep it up though and you’ll start to prove yourself a little at a time.  Then they’ll start to trust you.  Even when they trust you though, they’ll still fall back into being stupid all the time.  You have to urge them to fight it.  It’s hard though, right?  They really want to fall back into their old habits but you can’t let them.  It’s all they know and a little knowledge is empowering and dangerous.  So you have to guide them and not let them slip back into their bad habits of making stupid decisions.  Remind them that trusting you in the past has paid off and it will again. 

 

The Addiction of Caring:

The other side of this is the DBA himself.  See, as DBAs we tend to own the DBs under our control.  We want everything to work together like a well-rehearsed ballet.  And when it goes that way it can be a beautiful thing.  But it almost never goes that well, does it?  The company doesn’t see things the way we do.  The addicts in the above section quite often have the power to block our intelligent moves and even put things into play without us even knowing.  People in power are quite often worse than anyone else.  So how do you deal with this?  How do you get them to do the right thing?  Well, you can discuss it with them and get them to see reason, but if they don’t then you have to let it go. 

You’re not being paid to do a good job.  You’re being paid to make your boss happy.  I’ve said this many times before, and it’s still true.  Turn yourself into an internal fulltime consultant.  Advise them of the right thing to do, but if they make the wrong decision then it’s their business and their decision.  You can’t make them run their business the right way.  You can only advise them.  Remember that.  It’s not your business.  So stop insisting that things be done a certain way.  Tell them what should be done and let them make the decision.  And if they make the wrong decision just make sure you’re covered with an email.  You want it on record that you advised them correctly and they made the wrong decision.

But caring to the point where you’re pulling your hair out is just as much of an addiction as being stupid.  You have to constant remind yourself that you’re just a consultant and if they want to pay you a bloated salary to ignore you and have you spend your days cleaning their messes up instead of providing real value then that’s their business.  Again, you’re being paid to make them happy, not to do a good job.  Plenty of people disagree with me on this, but they’re wrong.  It just so happens that some bosses want you to do a good job and want you to guide the company, but that’s just a case where their priorities align with yours.  If their priorities were to change then I bet you’d find that you’re all of a sudden butting heads.  I’ve been in several companies where everyone in the world has sa and they’ve even got lots of apps connecting with sa and everyone knows the password.  I’ve advised them how unsafe this is, but they insisted that it’s too engrained and can’t be changed.  Fine, I put my advice in an email and went on about my business.  Things kept happening to the servers and they wanted it to stop, but I said I can’t stop it because everyone has sa.  They said it’s impossible.  So ok then.  Then they started failing audits.  They lost clients because they couldn’t produce  the report of the passed audit.  Now they came to me again and said how can we pass this audit.  I said you have to get everyone out of sa and do about 10 other things.  They said but the effort would be too great.  I said ok.  Then they lost another client.  Then they asked me what it would take to pass the audit.  I repeated the same thing.  They said it would be too hard.  I said ok.  They said can you give me another way?  I said no, but have you looked into what it would take?  What’s the actual effort?  They said no, we haven’t looked into it, but I know it would be really hard.  I said, then why not ask the question to the right people?  As it turns out it was a single VP who was blocking it because she had NO IT knowledge and just thought this kinda thing was really hard.  All they had to do was change a connection string in a .ini file and all was well.  Then they just told everyone to get over it and they weren’t getting sa anymore.  They would have to stop doing things that a DBA needs to do.  All of a sudden the environment became much more stable, they passed their next audit, and they started getting those contracts they were losing before.  That’s a true story BTW.

So find a way to divorce yourself from the outcome of these things.  I like to look at it like this… if you see a guy on the street getting into his car and he looks like he’s about to run the red light, and you call into his window and warn him there’s a cop sitting there, and he does it anyway you just look at him and say dumbass.  Chances are you probably won’t get upset and give yourself an ulcer over it because you’re just not that invested in the outcome of your advice.  You warned the guy but he didn’t listen… so what!  And that’s the attitude you have to take with your own company.  You can’t invest yourself in the outcome of your advice.  It’s not your company.

So anyway, I’m just trying to help you keep a little perspective.  If you’re truly in a job where they hang on your every word then consider yourself lucky.  The vast majority of us don’t have that luxury so we have to find ways to cope with the fact that everyone doesn’t listen to everything we say. 

And remember, both of these are addictions.  As much as we have to help our users fight against being stupid, we have to constantly fight against becoming too invested in the outcome of our advice.  And I struggle with this with different degrees of success.

 

Admin, Time-savers, Uncategorized

Good practice saves the day

1 Comment

Ok, a funny thing happened at work today. I had to write a script to move a DB’s files to another drive. Sounds easy enough, huh? Well in theory it was easy, but leave it to me to mess it up. Now, this was just a dry-run so I don’t feel too bad about it, and it gives me a chance to test my process. So, here’s what happened.

My process was this:

1. Run move stmts for all files.
2. Take DB offline.
3. Run PS script that moves the files.

The PS script selects from sys.master_files to get a list of the files to move. Here’s the problem… I’ve already run the move statements so SQL thinks they’re already in the new location. Therefore, the PS script can’t move them from their current location. However, good practice saves the day.

As a habit I always save a copy of sys.master_files to another table in case something happens. Now it’s a simple matter of changing my PS script from selecting from sys.master_files to selecting from sys.master_filesSEAN. Now I’m back in business. That’s the thing I find about good practices; they help you in ways you can’t even predict most of the time.

Here are some other general steps I take to protect against one thing or another:
1. I always try to group like objects in the same schema. It’s just a good idea.
2. I always copy the system DBs when upgrading or patching SQL. Sure I can restore from backup, and I have those too, but if something big happens and I have to completely re-install from scratch it’s much easier to just drop the old system DB files out there than it is to restore from backup.
3. Whenever I’m done working on an important prod system and it’s time to move on to something else, I always close all prod windows and even disconnect from it in object explorer. I don’t want any chance there’s a way to open a new connection to it and do something dumb.

I’d love to give you guys a huge list, and I know there are more, I just can’t think of any right this second.

Admin, Time-savers

Over-tuning Backups

This is going to be a short post, but an important one to demonstrate the importance of thinking about what you’re doing. I recently taught a backup tuning session at SQLSaturday 90 in OKC. You can see the session recording here if you like. And I’ve been teaching this session off and on recently so I’m not really sure where this user saw my session, but I had a user write me with a tuning issue and I thought I’d share it with you.

In his email he outlined his restore scenario that he was unable to tune. No matter what he did, the restore didn’t happen any faster, and he would expect to see some improvement. And in fact, his attempts to tune it have only made it take longer. Here’s the code he sent me, and I’ve changed things to protect his environment.

RESTORE DATABASE AdventureWorks
PAGE = '1:200'
FROM DISK = 'c:\AW.bak'
WITH init, FORMAT,
buffercount = 500,
maxtransfersize = 2097152,
norecovery

So there’s nothing wrong with this syntax, and the restore should work just fine. But he’s right, he won’t see any performance gain at all and here’s why. How fast do you think you can restore a single page?
There’s just no way to copy 8KB of data any faster than it already will. And I’m not surprised at all that his restore takes longer. SQL is still gonna spin up all those resources because you’re telling it to. But since you’re only copying a single page, it’s not going to use them. So spinning up the resources won’t do you any good on this small operation and will only serve to cost you extra time. The resource spin-up will only take a few seconds, but on a page restore like this, it can be in the neighborhood of double the time. So it could take the restore from say 6-10secs to like 10-20secs or something like that. So it’s really not a big deal time-wise.

If you came in late to the backup tuning debate, the flags we’re arguing over are buffercount and maxtransfersize.
I’ll let you watch the session above for a full explanation of them, but I’ll go ahead and say here that 500 buffers just doesn’t even make sense. And transferring 8KB worth of data in 2MB chunks doesn’t make sense either.

The point here is this is a definite case where there’s really nothing to even tune. You can only move 8KB so fast. And while I applaud his efforts to apply something he learned in his shop, he needs to put a little more thought into where he applies it. However, he did the right thing. He tested, he saw an issue, and then he emailed me. We’ve discussed it fully and he now understands why he saw what he did. This is how you learn. And even more importantly, he’s out there trying to apply something he learned from a session. He’s not content with leaving things the way they are in his shop, and he’s not content with his current level of skill. You go girlfriend.

And he’s given me permission to blog about this issue so I don’t want any emails about how I screwed over a viewer who came to me for help.

Admin

A fun conversation about backups

4 Comments

While I was talking to one of my Jrs today about backups, a .Net guy poked his head around the corner to offer his opinion on the matter.  The subject was basically whether everything will be copied over if you do a full backup and restore it to another system.

Here’s basically how the talk went:

.NET – Well, it really depends on whether you have different filegroups as to whether everything will be restored.

DBA:  No, if it’s a full backup and it restores, everything is there.

.NET:  Well, yeah, but I’m just saying that DBs can have a lot of filegroups sometimes and if it does, then you might not get all o fthem.

DBA:  No, if it’s a full backup and it restores, everything is there.

.NET:  But…

DBA:  There are no buts… if it’s a full backup and it restores, everything is there.

.NET:  I’m just saying that…

DBA:  No, if it’s a full backup and it restores, everything is there.

.NET:  You can’t deny that there are several filegroups, right?

DBA:  I would never try to deny that.

.NET:  And if you backup those different filegroups, then you can only restore some of them, therefore you can have a DB with some filegroups unrestored.

DBA:  This is correct.

.NET:  So back to my original…

DBA:  No, if it’s a full backup and it restores, everything is there.

.NET:  No, because you just said that you can backup different filegroups and restore only part of them.

DBA:  Yes I did.

.NET:  So aren’t we saying the same thing?

DBA:  No, not even close.

.NET:  Why not?

DBA:  Because we were talking about full backups, not filegroup backups.  Full backups backup everything… it’s in the name.  Filegroup backups only backup filegroups… that’s also in the name.  But a full backup can only be restored fully.  There is no partial restore of a full backup.

.NET:  So you’re telling me that you backup the full DB and you can backup filegroups, but it’s a different kind of backup?

DBA:  Yes.

.NET:  But still, if you have…

DBA:  No, if it’s a full backup and it restores, everything is there.

.NET:  You’re not letting me finish.

DBA:  Yeah, because I know what you’re trying to say and there’s no wiggle-room.

.NET:  But if you have multiple filegroups, then…

DBA:  No, if it’s a full backup and it restores, everything is there.

.NET:  So are you telling me that there’s absolutely no way to restore only certain filegroups from a full backup?

DBA:  That’s exaxtly what I’m telling you.  Again, it’s in the name ‘Full Backup’.

.NET:  So is there a way to change the location of a filegroup when you restore the full backup?

DBA:  Only if it’s tied to a specific file and you restore that file to another location, then you’re really moving the file itself and the filegroup is coming along for the ride.

.NET:  But I keep thinking there’s got to be a way to…

DBA:  No, if it’s a full backup and it restores, everything is there.

.NET:  But if there’s not enough disk space and it only restores part of the DB then that would leave you with only part of the DB.

DBA:  No, it wouldn’t complete the restore and you’d have nothing.

.NET:  So if you take a filegroup backup then you can restore different filegroups.

DBA:  Yes.

.NET:  Then I’m still right.

DBA:  No, not even close.

.NET:  Yeah, because you can still do what I said you could do.

DBA:  No, because you’ll recall that there are 2 very important aspects of what I’m saying and they both have to be there… which is part of the original topic:  “If it’s a full backup” and “if it restores”.  If both of those exist, then you’ve got everything in the DB.

.NET:  But I still think you should be able to…

DBA:  No, if it’s a full backup and it restores, everything is there.

And then it just kind of tapered off into other backup discussions from there, but that was just fun.

Thought you guys might like a funny story on a mon.

Admin

Troubleshooting a replication scenario

I just had a nice replication troubleshooting experience I thought I’d share with you guys.  I got a call from our app team saying that replication was failing.  I looked and the error was a bit unexplainable.

“Column names in each table must be unique.”

OK, so why the hell is it telling me that?  This publication has been running for months now with no problems.  I checked the table in question on both sides, and as it turns out the column it was complaining about had and int data type on the publication side and a uniqueidentifier on the subscription side.  So how did these columns get different data types?  Did someone change the data type on the subscriber to an incompatible data type?  No, that’s probably not it.  And that’s not the error message you would get for something like that anyway.  But someone had to change the data type right?  Well, only kinda.  Here’s what actually happened. 

One of the devs decided he needed this column in the table, and instead of going to the DBAs, he added it to the subscriber himself.  He then added it to the publisher (with a different data type).  So when replication went to replicate the DDL change, it saw that the subscription side already had the column and spit out that excellent error message.

Let this be a lesson to those of you who have DBAs and you constantly look for ways to circumvent them.  Why don’t you stop every now and then and implement things properly.  And by properly I mean slow down and take the time to think about what you’re doing.  The days are over when you can just do whatever you want because it crosses your mind… esp when you have replication running.  You have to plan schema changes or you’ll spend tons of time troubleshooting and fixing your scenario.  Not to mention what it does to whatever processes rely on this replication process.  Anyway, that’s all I’ve got for now.  Let’s all be safe in production.

MidnightDBA News

Why GoDaddy Sucks

12 Comments

Well, the site if finally back online.  I’m still moving a lot of the content over, but most of the vids are online again.

This whole debacle was caused by GoDaddy.  MidnightDBA.com has been up and running for 2yrs and for no reason whatsoever, it just stopped working last week.  None of my .net controls would work, so the menu and datagrids were all returning errors.  I tried to see if something went wrong on my end, and even pulled in the best .net guy I know from work, and nothing fixed it.  My work guy said that it seemed like GoDaddy was blocking .axd files and that’s why the content wasn’t working.  I finally got on with GoDaddy support, something I always dread because every time I’ve ever called them they’ve been absolutely useless.  Oh they’re polite, but every tech I’ve talked to with GoDaddy has been completely worthless and they do nothing but try to find ways to boot the call the second they start talking to you.

So anyway, I called support and no matter what I did I heard the same song again and again… “We don’t support custom coding.”

I tried and tried explaining to them that it’s not custom coding and that the site had been working for 2yrs.  And they insisted that nothing was wrong with the server and that my code had to be bad.  And to prove it they pushed a stupid little text file up and since they could read it, they used that as justification to try to close the case.  So I had the ticket escalated and talked to a couple managers.  The problem is with me at this point… for some reason I actually expected a different result.  I didn’t get one though.  The advanced escalation team fed me the same line of BS.  I even went around and around with them showing them a simple test page, and how the site worked on 3 other web servers, but they refused to listen to the simplest logic.  None of their support people know even the most basic principles of .net so they only gave me assinine ideas.  And that’s if the idea were even an idea at all.  The first guy I talked to went to his 2nd-level support guy and came back and said, “He said you have to push a ticket to your web.config file and do something I didn’t understand.”  Now, I have no idea what ‘push a ticket to your web.config’ means, but even if it meant something just the fact that he couldn’t finish the sentence means something doesn’t it?  He should have at least gone back to the guy to ask for more details, but he wouldn’t.  He was more concerned with booting the call because he doesn’t support custom coding.  And of course, I’m not allowed to talk to the 2nd level techs because I’m on a shared host and we’re not important enough to talk to the higher level morons.  So I was refused and was only allowed to talk to 1st-level guys.  And every time my tech would go away to ask someone something he’d be gone for around 30mins.

I did talk to one manager who actually got what I was saying though and she really went to bat for me.  Unfortunately, she works in a sea of morons and there’s just no way she’ll ever make a difference.  So after a week of being down, I signed up with another host, and amazingly the site came online right away.  I guess my code just decided to work again and it really didn’t have anything to do with GoDaddy’s servers, huh?  I mean, it must have been my mistake for thinking that code would run on their servers through whatever they do to the box.  It certainly wasn’t because they were blocking .axd files all of a sudden.  Because if they had for some reason decided to start blocking those file types, I know they would be responsible enough to tell their customers they were making a major change like that.  So I know now that it HAD to be me.  I didn’t do something right in my code.  So the code that I thought was solid and had been working for 2yrs just stopped on its own.  How could I have been so stupid? 

This is a warning to anyone who’s considering putting their site on GoDaddy.  If you do you’ll be sorry because they make changes on their servers without telling you about it and then blame you when something goes wrong.  If I hadn’t gone to another host, my site would have to be completely re-coded to use nothing but static HTML or classic ASP.  It’s ridiculous, unprofessional, and shows a real apathy for their customers.  GoDaddy has proven to me again and again that they don’t give a crap about their customers.  We’re just a paycheck to them.  But you know what?  If they keep this up we won’t be a paycheck much longer.  CIHost pulled a bunch of the same crap and look what happened to them.

SO DON’T GO TO GODADDY FOR ANYTHING!!!