Skip to main content

SQL Server performance for NOT EXISTS vs NOT IN

Though "NOT EXISTS" and "NOT IN" sounds similar there is quite a lot of difference between them.  To start with check out the blog post by Mladen.

In continuation to what Mladen has already written I thought I would show the differences it makes on the Execution Plan and the IO / Time when we use NOT EXISTS or NOT IN in our queries. Let's see few of the differences between them.

Case 1: Lets use them in columns which are declared as NOT NULL

SET NOCOUNT ON
GO


CREATE TABLE PackageInformation
(
Sno INT IDENTITY(1,1) PRIMARY KEY,
PackageID INT NOT NULL,
PackageName VARCHAR(20)
)
GO

-- I am using the random records generator which I wrote few days back to populate data into this table.
-- generating 10000 records
INSERT INTO PackageInformation  (PackageID, PackageName)
SELECT CAST(RAND(CHECKSUM(NEWID())) * 10000 AS INT),
dbo.udf_StringGenerator('A', 20)
GO 10000


CREATE TABLE ChildTable
(
RID INT IDENTITY(1,1) PRIMARY KEY,
PackageID INT NOT NULL
)
GO

-- Lets take some 40% (approx 4000 records) from the PackageInformation table to populate this table
INSERT INTO ChildTable (PackageID)
  SELECT PackageID
  FROM dbo.PackageInformation
  TABLESAMPLE (40 PERCENT); -- This would work only on SQL Server version 2005 or above
GO

Let's write a query to list Package details from PackageInformation table which is not present in ChildTable.

--Clear out the cache (DONT TRY THIS IN PRODUCTION ENVIRONMENT)
DBCC FREEPROCCACHE
GO

SET STATISTICS IO ON
SET STATISTICS TIME  ON
GO

--Press Control + M to display the Actual Execution Plan of the queries
--Query1: Using NOT IN
SELECT PackageID, PackageName 
FROM dbo.PackageInformation 
WHERE PackageID NOT IN (SELECT PackageID FROM ChildTable)
GO

--Query2: Using NOT EXISTS
SELECT PackageID, PackageName 
FROM dbo.PackageInformation
WHERE NOT EXISTS
 (
   SELECT PackageID FROM ChildTable 
   WHERE ChildTable.PackageID = PackageInformation.PackageID
)
GO

SET STATISTICS IO OFF
SET STATISTICS TIME  OFF
GO

Result:

i) Query using NOT IN : Returned 4690 records
ii) Query using NOT EXISTS : Returned 4690 records

iii) Let's see the Actual execution plan for both the queries. 

iv) Let's also check on the logical reads and CPU time taken for these queries.

So when the column is declared as NOT NULL then both NOT IN and NOT EXISTS seems to perform the same way.

Case 2: Let's change the PackageID column as NULL

ALTER TABLE dbo.PackageInformation
ALTER COLUMN PackageID INT NULL
GO

ALTER TABLE dbo.ChildTable
ALTER COLUMN PackageID INT NULL
GO

--Lets insert some 100 null values into the PackageInformation Table
INSERT INTO PackageInformation 
SELECT NULL,
dbo.udf_StringGenerator('A', 5)
GO 100


Let's run the same query which we used in Case 1 to list Package details from PackageInformation table which is not present in ChildTable.

--Clear out the cache (DONT TRY THIS IN PRODUCTION ENVIRONMENT)
DBCC FREEPROCCACHE
GO

SET STATISTICS IO ON
SET STATISTICS TIME  ON
GO

--Query1: Using NOT IN
SELECT PackageID, PackageName 
FROM dbo.PackageInformation 
WHERE PackageID NOT IN (SELECT PackageID FROM ChildTable)
GO

--Query2: Using NOT EXISTS
SELECT PackageID, PackageName 
FROM dbo.PackageInformation
WHERE NOT EXISTS
 (
   SELECT PackageID FROM ChildTable 
   WHERE ChildTable.PackageID = PackageInformation.PackageID
)
GO

SET STATISTICS IO OFF
SET STATISTICS TIME  OFF
GO


Result:

i) Query using NOT IN : Returned 4690 records (It hasn't considered those 100 new NULL records which we added!!)
ii) Query using NOT EXISTS : Returned 4790 records

iii) Let's see the Actual execution plan for both the queries. 

iv) Let's also check on the logical reads and CPU time taken for these queries.

So when column is declared as NULL then NOT IN seems to generate a pretty complicated execution plan and does NUMEROUS number of logical reads more than NOT EXISTS. So the winner here is NOT EXISTS.

Case 3: Adding NULL values into ChildTable

INSERT INTO ChildTable (PackageID)
SELECT NULL

Result:

i) Query using NOT IN : Returned 0 records!
ii) Query using NOT EXISTS : Returned 4790 records

So if the Subquery returns even one NULL then NOT IN operator would not return any result which isn't right. So again the winner is NOT EXISTS.

I think it would be safe to say that we should use NOT EXISTS instead of NOT IN as it seems to work as expected by us in all the scenarios which we saw in this post.

--Cleanup
DROP TABLE ChildTable
GO
DROP TABLE PACKAGEINFORMATION
GO

Comments

Popular posts from this blog

Registry manipulation from SQL

Registry Manupulation from SQL Server is pretty easy. There are 4 extended stored procedure in SQL Server 2000 for the purpose of manupulating the server registry. They are: 1) xp_regwrite 2) xp_regread 3) xp_regdeletekey 4) xp_regdeletevalue Let us see each one of them in detail! About xp_regwrite This extended stored procedure helps us to create data item in the (server’s) registry and we could also create a new key. Usage: We must specify the root key with the @rootkey parameter and an individual key with the @key parameter. Please note that if the key doesn’t exist (without any warnnig) it would be created in the registry. The @value_name parameter designates the data item and the @type the type of the data item. Valid data item types include REG_SZ and REG_DWORD . The last parameter is the @value parameter, which assigns a value to the data item. Let us now see an example which would add a new key called " TestKey ", and a new data item under it called TestKeyValue :

Screen scraping using XmlHttp and Vbscript ...

I wrote a small program for screen scraping any sites using XmlHttp object and VBScript. I know I haven't done any rocket science :) still I thought of sharing the code with you all. XmlHttp -- E x tensible M arkup L anguage H ypertext T ransfer P rotocol An advantage is that - the XmlHttp object queries the server and retrieve the latest information without reloading the page. Source code: < html > < head > < script language ="vbscript"> Dim objXmlHttp Set objXmlHttp = CreateObject("Msxml2.XMLHttp") Function ScreenScrapping() URL == "UR site URL comes here" objXmlHttp.Open "POST", url, False objXmlHttp.onreadystatechange = getref("HandleStateChange") objXmlHttp.Send End Function Function HandleStateChange() If (ObjXmlHttp.readyState = 4) Then msgbox "Screenscrapping completed .." divShowContent.innerHtml = objXmlHttp.responseText End If End Function </ script > < head > < body > &l

Script table as - ALTER TO is greyed out - SQL SERVER

One of my office colleague recently asked me why we are not able to generate ALTER Table script from SSMS. If we right click on the table and choose "Script Table As"  ALTER To option would be disabled or Greyed out. Is it a bug? No it isn't a bug. ALTER To is there to be used for generating modified script of Stored Procedure, Functions, Views, Triggers etc., and NOT for Tables. For generating ALTER Table script there is an work around. Right click on the table, choose "Modify" and enter into the design mode. Make what ever changes you want to make and WITHOUT saving it right click anywhere on the top half of the window (above Column properties) and choose "Generate Change Script". Please be advised that SQL Server would drop actually create a new table with modifications, move the data from the old table into it and then drop the old table. Sounds simple but assume you have a very large table for which you want to do this! Then it woul