Tricky TSQL: NOT IN (NULL)

We’ve done this before, but we can go one better this time.

Let’s take this step by step.

NULL means “I don’t know”. It stand for an unknown value.

Nothing can be equal to NULL. We simply can’t say that 1 = NULL, or ‘ABBA’ = NULL,  because we don’t know what value NULL might possibly be in the real world.

Nothing can be NOT equal to NULL. We can’t say that 1 <> NULL, or ‘ABBA’ <> NULL. NULL may well turn out to be 1, or ABBA, or whatever value we’re trying to check. We simply don’t know.

A simple example

So let’s take a very simple example, to demonstrate how this works. This particular query will return the value ‘True’, because 1 is demonstrably equal to 1:

IF 1 = 1
   SELECT   'True!';

Yep, that makes sense. I can tell: 1 = 1.

This query will not return ‘True’; it will return nothing, because 1 cannot be proven to be equal to NULL:

IF 1 = NULL
   SELECT   'True!';

We have an unknown, and nothing can possibly be equal to the unknown.

Another way to put this, is to use the keyword IN:

IF 1 IN ( SELECT    NULL AS a )
   SELECT   'True';

That parenthetical SELECT NULL simply returns NULL, so the query ends up asking: “is the value 1 in the set of values (NULL)?” We have no idea, so the query does not return ‘True’.

Almost the same thing, but trickier

Here’s yet another query that will not return ‘True’; 1 cannot be proven to be NOT equal to NULL:

IF 1 <> NULL
   SELECT   'True!';

NULL is unknown, and nothing can possibly be NOT equal to the unknown.

NULL will mess up your  NOT IN queries

And the absolute trickiest bit – the one that will trip you up, if you’re not thinking? NOT IN.

This query also returns nothing!

IF 1 NOT IN ( SELECT    2 AS a
              UNION
              SELECT    NULL AS a )
   SELECT   'True';

The IF statement asks: “is 1 in the collection of values (2, NULL)?” And we must say, “ah, no idea. That can’t be proven. Therefore we can’t return true.”

The value 1 cannot be determined to not be in the set of (2, NULL).

Play around with these, let it sink in. And watch out for NULLs in your NOT IN result sets!

One Fix

So, always use IS NULL or IS NOT NULL for your NULL comparison needs. And if you have a [value] NOT IN situation, read on.

Here’s what I did* to fix this exact situation in a recent query:

 SELECT ID
 FROM   Table1
 WHERE  ID NOT IN ( SELECT  ID
                    FROM    Table2
                    WHERE   ID > 0 );

While we could have used ID IS NOT NULL instead of ID > 0, it would make any relevant index on Table2 useless to us. In this particular case, ID is restricted to only positive above 0, and NULL.

To read up on even more solutions, see Mladen Pradjic’s article JOIN vs IN EXISTS.

Happy days,
Jen
www.MidnightDBA.com/Jen