All posts by Jen

Coming soon: Minion Backup, featuring table based scheduling!

minion backup
The MidnightDBA team is announcing the release of a new, free backup solution for SQL Server: Minion Backup arrives on June 1!

Minion Backup by MidnightDBA is a stand-alone database backup module.  Once installed, Minion Backup automatically backs up all online databases on the SQL Server instance, and will incorporate databases as they are added or removed.

We created Minion Backup (or MB, for short) to be the most flexible, feature-rich backup solution possible. Our goal for this initial release was to include functionality for as many possible backup scenarios as possible. We’ve included certificate backups, HA and DR awareness, restore script generation, “what if” functionality for deletes, the ability to run a batch for “missing” backups, built in manual runs, rollup and detail data in the backup logs, the ability to deactivate most settings, copy / move / stripe / mirror backup files, etc.

Table based scheduling

While there are about fifty features I’d like to talk about, I’m going to restrain myself (today) and talk about the one feature I’m most excited about (today): table based scheduling.

When Minion Backup is installed, it creates a single backup job that runs the master backup stored procedure every 30 minutes.  That master procedure checks the Minion.BackupSettingsServer table to determine what backups should be run for the current day and time.

By default, Minion Backup comes installed with the following scenario:

  • Full system backups are scheduled daily at 10:00pm.
  • Full user backups are scheduled on Saturdays at 11:00pm.
  • Differential backups for user databases are scheduled daily except Saturdays (weekdays and on Sunday) at 11:00pm.
  • Log backups for user databases run daily as often as the backup runs (every 30 minutes).

Let’s look at just a few of the columns of this default scenario in Minion.BackupSettingsServer:

ID DBType BackupType Day BeginTime EndTime MaxForTimeframe Include Exclude
1 System Full Daily 22:00:00 22:30:00 1 NULL NULL
2 User Full Saturday 23:00:00 23:30:00 1 NULL NULL
3 User Diff Weekday 23:00:00 23:30:00 1 NULL NULL
4 User Diff Sunday 23:00:00 23:30:00 1 NULL NULL
5 User Log Daily 00:00:00 23:59:00 48 NULL NULL

I’m not going to fully document this table here – I’ll be happy to send you a draft of the product documentation if you can’t wait for the release date – but you get an initial impression of how flexible this scenario can be, especially in conjunction with other settings tables. I will note that “Include” and “Exclude” allow comma delimited lists of databases (and/or LIKE operators) to include in, or exclude from, the particular backup scenario; a value of NULL means that all databases are included.

This is how MB operates by default, to allow for the most flexible backup scheduling with as few jobs as possible.

Table based scheduling presents multiple advantages:

  • A single backup job – Multiple backup jobs are, to put it simply, a pain. They’re a pain to update and slow to manage, as compared with using update and insert statements on a table.
  • Fast, repeatable configuration – Keeping your backup schedules in a table saves loads of time, because you can enable and disable schedules, change frequency and time range, etc. all with an update statements. This also makes standardization easier: write one script to alter your backup schedules, and run it across all Minion Backup instances (instead of changing dozens or hundreds of jobs).
  • Mass updates across instances – With a simple Powershell script, you can take that same script and run it across hundreds of SQL Server instances at once, standardizing your entire enterprise with ease.
  • Transparent scheduling – Multiple backup jobs tend to obscure the backup scenario, because each piece of the configuration is displayed in separate windows. Table based scheduling allows you to see all aspects of the backup schedule in one place, easily and clearly.
  • Boundless flexibility – Table based scheduling provides a stunning degree of flexibility that would be very troublesome to implement with multiple jobs. With a single backup job, you can schedule all of the following:
    • System full backups three days a week.
    • User full backups on weekend days and Wednesday.
    • DB1 log backups between 7am and 5pm on weekdays.
    • All other user log backups between 1am and 11pm on all days.
    • Differential backups for DB2 at 2am and 2pm.
    • Read only backups on the first of every month.

…and each of these can also use dynamic backup tuning, which can also be slated for different file sizes, applicable at different times and days of the week and year.

and each of these can also stripe across multiple files, to multiple locations, and/or copy to secondary locations, and/or mirror to a secondary location.

Like I said, there are a zillion and a half more things I’d like to talk about, but we’ll keep it right here for now. Reply below, email, or ping @MidnightDBA on Twitter with questions or comments. And keep an eye out on June 1!

 

Check out Minion Enterprise, our new enterprise management solution for centralized SQL Server management and alerting! 

P.S.  Anticipating a few FAQs (and I’ll add to this as things come up):

  • Yes, you can change how often the backup job runs. If, for example, you only want log backups to run hourly, set your job to run hourly.
  • Yes, absolutely, you still have the option to use the more traditional “multi job” backup scheduling. You’d just disable the single job mentioned above, and configure the new jobs with individual schedules and a parameterized master query. Easy.
  • The Include and Exclude fields aren’t the only way to include and exclude databases, but we’re not going to get into that just now.

Precon: The Enterprise Scripting Workshop #SQLSaturdayBR

midnightsqlYou are cordially invited to a fantastic precon event before SQL Saturday Baton Rouge! The Enterprise Scripting Workshop is on Friday, July 31, and if you register before June 30 you’ll get the Early Bird rate of $99…not bad at all for a full day’s training.

Oh, and don’t forget to register for SQL Saturday #423 Baton Rouge, too!

Abstract

The DBA life is frustrating: you rarely have time to innovate, because the same tasks fill up your time day after day. Your users are unhappy about how long it takes to resolve “simple” tickets. You have to put big items on hold because special requests come in. As careful you are, mistakes creep in as you get busier.

We’ve been there.

In this workshop, the MidnightDBAs show you how to develop enterprise scripts with a huge range of uses. A good set of reusable scripts will reduce task time from hours – or even days – to just a few minutes, and eliminate mistakes from your environment.

You’ll learn:

  • Enterprise philosophy – tackle simple tasks with the whole environment in mind
  • Single data store – define the benefits and uses of a single central database for common-use data and metadata
  • Choice of tools – choose the best tool (Powershell, T-SQL, SSIS, or other) for the job
  • Environment ground work – prepare your environment for enterprise scripting
  • Real world scripts – work through dozens of enterprise scripting issues (like alerting, error handling, multiple SQL versions, and more) as we develop a real enterprise script in class

This session is for database administrators, with a basic understanding of Powershell. It’s for anyone who touches backups or security, maintains databases, troubleshoots performance, monitors disk space, or any of a hundred other DBA tasks. Enterprise scripting is for anyone who has more tasks than time.

http://www.MidnightDBA.com/Jen

Check out Minion Enterprise, our new enterprise management solution for centralized SQL Server management and alerting! 

Minion Reindex 1.1 Release (and re-release)

We released Minion Reindex version 1.1 last week. The changes are fairly small:

  • Made sure Minion Reindex handles all nonstandard naming (e.g., object names with spaces or special characters).
  • Added support for Availability Group replicas. (Basic AG support has been added by only permitting Minion Reindex to run on an AG Primary DB.)
  • Fixed formatting in Minion.Help stored procedure.
  • If you run the installation script in a database other than master, this will now automatically be reflected in the Minion Reindex jobs. (The documents still say that you have to change this manually…I need to update that…)

But, and this is important, if you downloaded Minion Reindex 1.1 before this blog post hit, you need to re-download it. Yep, Jen messed up. She posted an earlier, incomplete version of the 1.1 package, and so introduced a syntax error.  As of now, this is fixed. (Her delicious dish of crow is in the oven, almost ready to eat.)

We timed the release to coincide with Grant Fritchey’s review of Minion Reindex, which has sparked a fantastic discussion in the comments.

So, once you download the fresh, non-syntaxy version of Minion Reindex 1.1, you can chime in and let us know how you feel!

 

Video: A Better Way to Reindex

minion reindex-01Edit: Corrected the recording link.

Yestreday I taught “A Better Way to Reindex” for the PASS Performance Virtual Chapter (event link, with recording).

The recording is up, if you missed the live event, or if you want to re-watch it:  https://attendee.gotowebinar.com/recording/8483181173057914370

And by the way, the demo that fails (there’s ALWAYS one demo that fails) is due to the fragmentation routine we use, fragmenting the data way more than usual. All we had to do was to increase the reorg threshold, and it would’ve been fine. So for the record, it’s the fault of the test harness, not of Minion Reindex.

Here’s the abstract:

A BETTER WAY TO REINDEX

Let’s play a guessing game: I guess that you don’t want to spend time on index maintenance, but you know your servers need it. You want something that’s straight plug and play, dependable, and preferably free. I also guess that you don’t want to manage multiple jobs for this one maintenance task, even when you have exceptions and special settings. Am I close on this? Come and take a look at Minion Reindex. It does all those things that you want for index maintenance, plus some really innovative stuff. Watch your reindex progress live. Configure settings and exceptions for individual databases or tables. Gather fragmentation stats separately, outside your maintenance window. There’s quite a lot more so come see how you’ll make your index maintenance a lot easier.

Of course, you should go and download Minion Reindex yourself, and see the documentation, and follow along with all the fun stuff I do in the session.

Minion Reindex arrives this Friday!

The MidnightDBA team is announcing the release of a new, free index maintenance solution for SQL Server: Minion Reindex by MidnightDBA.  We designed Minion Reindex to solve all of the reindexing headaches we’ve experienced at dozens and dozens of client sites.

The Bottom Line

Your databases need index maintenance for optimal performance, and most shops don’t have a decent solution in place.  Minion Reindex is extremely simple to implement, and deeply configurable. Junior DBAs will love the ease of use, and expert data professionals will love the extensive settings and features available.

Read more on the Minion Reindex page:

  • Check out the full features list.
  • Get the download link there starting on Friday, October 24, 2014.
  • Join us for the Minion Reindex webinar Monday, October 27.
  • See documentation and tutorials as they come in!

You can ask questions and follow along with the discussion on Twitter using the #MinionReindex hashtag.

Stay tuned for the best thing index maintenance has ever seen…

 

SQL Saturday Oklahoma City, now with a Sean and Jen precon!

We’ll be at OKC next month for their SQL Saturday on August 23, and we’re presenting our precon “Becoming an Enterprise DBA” August 22!.  Get your early bird ticket by July 15, and save over 15%!

Precon: Becoming an Enterprise DBA

August 22, 2014, Norman Technology Center, Moore OK

There’s a real and quantifiable difference between a regular DBA and the leader of an enterprise.  Guiding the database effort of a huge enterprise takes business acumen, experience, problem solving, street-smarts, and above all… vision.  And quite often regular DBAs are thrown into big enterprises without knowing what it takes to get the job done.  In fact, most DBAs get thrown into this situation without even a hint that there’s a difference.  And while they do the best they can, they tend to do more harm than good which is why many companies have such a hard time managing their data.  What you need is to change your tactic when dealing with large numbers of servers.

The biggest piece of missing training in databases is enterprise training.  DBAs are often taught syntax and usage of features, but they’re never taught how to actually use them to plan an enterprise, run an enterprise, or cope with changing requirements, manage difficult users or groups, manage growth, manage vendors, etc.  There are so many more requirements to running an enterprise than just knowing how the individual features work.  There are disk and backup requirements, auditing requests, end-user reporting, paranoid VPs, and much more.  And there’s no way to gain this type of business sense on your own.  You have to learn from seasoned pros who have been there many times over.

I’ve built many successful DBA teams in a few companies that were resistant to the change.  And I can show you how to do it too.  Take your daily tasks down to almost nothing while you increase your visibility in your company (in a positive way).

What you will learn:

  1. How to properly manage a large environment.
  2. Managing disk space and growth requirements across hundreds of servers.
  3. Manage end user expectations.
  4. Build a self-service reporting model to keep DBAs from having to field every day requests.
  5. Reduce security management issues by up to 90%.
  6. How to assess your biggest DBA resource hogs to make the most impact in your environment.
  7. Strategies for dealing with vendors and vendor security issues.
  8. How to hire a competent DBA.
  9. How to staff your company appropriately.  Do you know how many DBAs you need?
  10. How to get a bigger raise and make your company pay for training.

And much much more.  If you manage an enterprise or would like to then you can’t afford to miss this seminar.  No other training exists to take you from being a regular DBA to running and leading an enterprise database effort.  And don’t fool yourself; There is a real difference.  Get ready to take your skills and your career to the next level.

 

SQL Saturday Oklahoma City precon: Becoming an Enterprise DBA

Spork instructional module not included. Probably.

We’ll be at OKC this summer for their SQL Saturday on August 23, and we’re presenting our precon “Becoming an Enterprise DBA” August 22!. Get your early bird ticket by July 15, and save over 15%!

Precon: Becoming an Enterprise DBA

August 22, 2014, Norman Technology Center, Moore OK

There’s a real and quantifiable difference between a regular DBA and the leader of an enterprise. Guiding the database effort of a huge enterprise takes business acumen, experience, problem solving, street-smarts, and above all… vision. And quite often regular DBAs are thrown into big enterprises without knowing what it takes to get the job done. In fact, most DBAs get thrown into this situation without even a hint that there’s a difference. And while they do the best they can, they tend to do more harm than good which is why many companies have such a hard time managing their data. What you need is to change your tactic when dealing with large numbers of servers.

The biggest piece of missing training in databases is enterprise training. DBAs are often taught syntax and usage of features, but they’re never taught how to actually use them to plan an enterprise, run an enterprise, or cope with changing requirements, manage difficult users or groups, manage growth, manage vendors, etc. There are so many more requirements to running an enterprise than just knowing how the individual features work. There are disk and backup requirements, auditing requests, end-user reporting, paranoid VPs, and much more. And there’s no way to gain this type of business sense on your own. You have to learn from seasoned pros who have been there many times over.

I’ve built many successful DBA teams in a few companies that were resistant to the change. And I can show you how to do it too. Take your daily tasks down to almost nothing while you increase your visibility in your company (in a positive way).

What you will learn:

  1. How to properly manage a large environment.
  2. Managing disk space and growth requirements across hundreds of servers.
  3. Manage end user expectations.
  4. Build a self-service reporting model to keep DBAs from having to field every day requests.
  5. Reduce security management issues by up to 90%.
  6. How to assess your biggest DBA resource hogs to make the most impact in your environment.
  7. Strategies for dealing with vendors and vendor security issues.
  8. How to hire a competent DBA.
  9. How to staff your company appropriately. Do you know how many DBAs you need?
  10. How to get a bigger raise and make your company pay for training.

And much much more. If you manage an enterprise or would like to then you can’t afford to miss this seminar. No other training exists to take you from being a regular DBA to running and leading an enterprise database effort. And don’t fool yourself; There is a real difference. Get ready to take your skills and your career to the next level.

 

InfoWorld Review: SQL Server 2014, by Sean McCown

InfoWorld Magazine (http://www.InfoWorld.com) has published a new article by MidnightSQL owner Sean McCown, “Review: SQL Server 2014 pushes the pedal to the metal”. The article announces the upcoming release of Microsoft SQL Server 2014 – due out on April 1, 2014 – and reviews some of the top new features.

InfoWorld Magazine is “the leading source of information on emerging enterprise technologies, and the only brand that explains to senior technology decision makers how these technologies work, and how they can use them to drive their business.” Sean’s articles are targeted to this group, detailing what executives need to know about deeply technical topics and products.

Sean has been a contributing editor at InfoWorld Magazine since 2003. His writing is focuses on the database sphere, including database systems, product reviews, and head-to-head comparisons.

 

[InfoWorld] Review: SQL Server 2014 pushes the pedal to the metal
http://www.infoworld.com/d/data-management/review-sql-server-2014-pushes-the-pedal-the-metal-238739

Author bio:
http://www.infoworld.com/author-bios/sean-mccown

What’s the CPU usage on the server? Use #Powershell!!

We have here an easy, handy-dandy way to find out how much CPU is in use on your server. Of course you can find out by opening up task manager, or by running perfmon, but that’s not really the point. We want something quick and easy. We want something we could potentially use for monitoring, alerting, or tracking over time in SQL Server.

Well of course the answer is Powershell!

Open up Powershell right now, and type “Get-WmiObject win32_processor | select LoadPercentage“. That’s how much CPU is in use right now.

Now let’s say that you have a recurring intermittent CPU spike on your SQL Server. How could we use this? Let’s take a look at our script:


CREATE TABLE #PS
(
PSoutput NVARCHAR(2000)
);

DECLARE @posh NVARCHAR(2000);
SET @posh = ‘powershell “Get-WmiObject win32_processor | select LoadPercentage”‘;

INSERT  INTO #PS
( PSoutput )
EXEC xp_cmdshell @posh;

SELECT  CAST(LTRIM(RTRIM(PSoutput)) AS INT) AS LoadPercentage
FROM    #PS
WHERE   PSoutput LIKE ‘%[0-9]%’;

DROP TABLE #PS;

Here’s what we’re doing:

  1. Create a temp table #PS to hold our output.
  2. Save the Powershell command as a variable.
  3. Run the Powershell command using xp_cmdshell (this will have to be enabled using sp_configure), and load the results into #PS.
  4. Select the only numeric value in #PS, trim the spaces off it, and cast it as an integer.

From here, we could do several things.

  • If we wanted to track CPU over time, we could stick this in a job to save that value to a table, and have it run every few minutes.
  • Or, we could set up an alert if the CPU goes over a percentage.
  • Better yet, a hybrid solution would be to track just the instances of CPU over a threshold (say, 90%), and alert if it happens several times.

We have a solution in place (on a server that spikes CPU randomly) that runs sp_WhoIsActive to a table if CPU is above threshold, and alerts if the CPU has been high for 3 samples out of the last 5. It keeps us from running pell-mell to the computer every time an alert happens, because we set it up to gather information automatically.

Of course it would also be useful to see how much CPU that SQL Server itself is using, to compare against the server total. But that’s a blog for another day.

 

Sean McCown
www.MidnightDBA.com/DBARant

Jen McCown
www.MidnightDBA.com/Jen

 

This is a MidnightDBA collaborative blog.

#SQLHighTea: The classiest party in Charlotte!

The High Tea Seal says "get your tix, wot-wot!"
The High Tea Seal says “get your tix, wot-wot!”

There’s a place the people go to present the unprofessional. To perform the profane. To outline the offensive. To have a bit of fun, in fact.

A few years back, some of us got to talking about all the things we wish we could present at a proper conference, but that just wouldn’t fly. You know, rants, anything with swears, the funny stuff, things that are rather risqué. And then we thought, let’s do that. And let’s do it for charity!

Our SQL High Tea party – coming up in Charlotte late on Wednesday, October 16 – is that place, that thing, and that charity. This year we’ve got a great sponsor (thanks Pluralsight!!!), great venue, fantastic speakers, and you’ll never hear a thing about it after we’re done.  People get up and say what they want, be it funny, or crude, or ridiculous…and it’s all done under NDA.

Get your tickets now for SQL High Tea – the classiest event in Charlotte this October – and bring along your top hat and monocle.

 

Thanks to our sponsor!

Pluralsight is a global leader in high-quality online training for hardcore developers and IT professionals. Pluralsight provides flexible and cost-effective subscription plans for individuals and businesses starting from as little as $29 a month. See what you can learn at pluralsight.com.