Reducing Memory Consumption of .NET Dictionary Class

.NET programmers generally use parameterless constructor of .NET Dictionary class. Actually, the class has various constructors. Here I am focused on the one, which takes the initial capacity as parameter.

Using public Dictionary(int capacity)constructor, reduces memory consumption of instance dramatically.

To measure the difference clearly, we create here 2 dictionaries of type Dictionary<long, string>;

n  One with unknown capacity,

n  One with known capacity

We add 1 million items to each of them. Here are the results:

Code:

using System;
using System.Collections.Generic;

namespace caDictionaryMemory
{
    class Program
    {
        const int length = 1000000;
        static void Main(string[] args)
        {
            Dictionary<long, string> dictWithCapacity = new Dictionary<long, string>(length);
            long lengthWithCapacity = AddItemsAndFindMemoryUsage(dictWithCapacity, length);

            Dictionary<long, string> dictWithoutCapacity = new Dictionary<long, string>();
            long lengthWithoutCapacity = AddItemsAndFindMemoryUsage(dictWithoutCapacity, length);
            Console.WriteLine("Memory Usage When Capacity Known: {0:0,0} KB", lengthWithCapacity);
            Console.WriteLine("Memory Usage When Capacity Unknown: {0:0,0} KB", lengthWithoutCapacity);
            Console.WriteLine();
            // Dummy usage of dictionaries to prevent Garbage Collector to collect them.
            Console.WriteLine("Item Count When Capacity Known: {0:0,0}", dictWithCapacity.Count);
            Console.WriteLine("Item Count When Capacity Unknown: {0:0,0}", dictWithoutCapacity.Count);
        }

        static long AddItemsAndFindMemoryUsage(Dictionary<long, string> dict, int count)
        {
            long beforeBytes = GC.GetTotalMemory(true);
            for (int i = 0; i < count; i++)
            {
                dict.Add(i, i.ToString());
            }
            long afterBytes = GC.GetTotalMemory(true);
            return (afterBytes - beforeBytes) / 1024;
        }
    }
}

The dictionary with unknown capacity uses 221% memory compared to known one.

Programmatic Analysis of SQL Server Execution Plan

Developers generally display the execution plan on SQL Server Management Studio visually. (Most known method: Select name of the procedure and press CTRL + L). SQL Server Database Engine serves execution plans in XML format. The images you see for plan in Management Studio is the interpretation of this XML string.

This XML formatted execution plan can be examined programmatically in order to check for certain operations.

Scenario: For a given stored procedure, I want to determine if the proc performs table scan or index scan operations on large tables.

In SQL Server connection, queries which are executed after "SET SHOWPLAN_XML ON" are not executed, but execution plan is created only.

Here is the class that retrieves Execution Plan XML of a stored proc.

Code:

static class ExecutionPlanRetriever
    {
        public static string Retrieve(string spName, string connectionString)
        {
            using (SqlConnection conn = new SqlConnection(connectionString))
            {
                try
                {
                    using (SqlCommand cmd = GetCommandWithParams(spName, connectionString))
                    {
                        conn.Open();
                        cmd.Connection = conn;
                        SetShowPlanOn(conn);
                        SetCommandParamsByDefault(cmd);
                        cmd.CommandTimeout = 0;
                        string planStr = cmd.ExecuteScalar().ToString();
                        return planStr;
                    }
                }
                catch (Exception e)
                {
                }
            }
            return null;
        }

        static SqlCommand GetCommandWithParams(string spName, string connectionString)
        {
            using (SqlConnection conn = new SqlConnection(connectionString))
            {
                conn.Open();
                SqlCommand cmd = new SqlCommand(spName, conn);
                cmd.CommandType = System.Data.CommandType.StoredProcedure;
                SqlCommandBuilder.DeriveParameters(cmd);
                return cmd;
            }
        }

        static void SetShowPlanOn(SqlConnection conn)
        {
            string cmdText = "SET SHOWPLAN_XML ON";
            using (SqlCommand cmd = new SqlCommand(cmdText, conn))
            {
                cmd.ExecuteNonQuery();
            }
        }

        static void SetCommandParamsByDefault(SqlCommand cmd)
        {
            foreach (SqlParameter prm in cmd.Parameters)
            {
                object val = GetDefaultSqlTypeValue(prm.SqlDbType);
                if (val is DataTable)
                {
                    prm.TypeName = ((DataTable)val).TableName;
                }
                prm.Value = val;
            }
        }

        static object GetDefaultSqlTypeValue(SqlDbType dbType)
        {
            object val = null;
            switch (dbType)
            {
                case System.Data.SqlDbType.BigInt: val = 0;
                    break;
                case System.Data.SqlDbType.Binary: val = val = new byte[1] { 0 };
                    break;
                case System.Data.SqlDbType.Bit: val = 0;
                    break;
                case System.Data.SqlDbType.Char: val = 'A';
                    break;
                case System.Data.SqlDbType.Date: val = new DateTime(2010, 01, 01);
                    break;
                case System.Data.SqlDbType.DateTime: val = new DateTime(2010, 01, 01);
                    break;
                case System.Data.SqlDbType.DateTime2: val = new DateTime(2010, 01, 01);
                    break;
                case System.Data.SqlDbType.DateTimeOffset: val = null;
                    break;
                case System.Data.SqlDbType.Decimal: val = 0;
                    break;
                case System.Data.SqlDbType.Float: val = 0;
                    break;
                case System.Data.SqlDbType.Image: val = null;
                    break;
                case System.Data.SqlDbType.Int: val = 0;
                    break;
                case System.Data.SqlDbType.Money: val = 0;
                    break;
                case System.Data.SqlDbType.NChar: val = string.Empty;
                    break;
                case System.Data.SqlDbType.NText: val = string.Empty;
                    break;
                case System.Data.SqlDbType.NVarChar: val = string.Empty;
                    break;
                case System.Data.SqlDbType.Real: val = 0;
                    break;
                case System.Data.SqlDbType.SmallDateTime: val = new DateTime(2010, 01, 01);
                    break;
                case System.Data.SqlDbType.SmallInt: val = 0;
                    break;
                case System.Data.SqlDbType.SmallMoney: val = 0;
                    break;
                case System.Data.SqlDbType.Text: val = string.Empty;
                    break;
                case System.Data.SqlDbType.Time: val = new DateTime(2010, 01, 01);
                    break;
                case System.Data.SqlDbType.Timestamp: val = new DateTime(2010, 01, 01);
                    break;
                case System.Data.SqlDbType.TinyInt: val = 0;
                    break;
                case System.Data.SqlDbType.UniqueIdentifier: val = string.Empty;
                    break;
                case System.Data.SqlDbType.VarBinary: val = new byte[1] { 0 };
                    break;
                case System.Data.SqlDbType.VarChar: val = string.Empty;
                    break;
                case System.Data.SqlDbType.Variant: val = null;
                    break;
                case System.Data.SqlDbType.Xml: val = null;
                    break;
                default:
                    break;
            }
            return val;
        }
    }

Now, once the XML string is ready, I can search for certain operations by using XPath.
Assuming we have the list of large tables, we can determine risky scan operations as:

Code:

    class DbObjectName
    {
        public string Schema { get; set; }
        public string Name { get; set; }
    }

    static class ExecutionPlanChecker
    {
        static DbObjectName[] TableIndexScanRiskTables = new DbObjectName[] {
            new DbObjectName { Schema = "SAMPLE_SCHEMA", Name = "SAMPLE_TABLE" }, 
        };

        // Resource for scan types:
        //http://technet.microsoft.com/en-us/library/ms186954(v=sql.105).aspx
        static readonly string[] riskyOperationNames = new string[] { "Table Scan", "Index Scan", "Clustered Index Scan" };

        public static bool PlanHasRiskyScanOperation(string planXml)
        {
            XmlDocument doc = new XmlDocument();
            doc.LoadXml(planXml);
            doc = StripNamespace(doc);
            foreach (var opName in riskyOperationNames)
            {
                if (HasRiskyScanOperation(doc, opName))
                {
                    return true;
                }
            }
            return false;
        }

        static XmlDocument StripNamespace(XmlDocument doc)
        {
            if (doc.DocumentElement.NamespaceURI.Length > 0)
            {
                doc.DocumentElement.SetAttribute("xmlns", string.Empty);
                XmlDocument newDoc = new XmlDocument();
                newDoc.LoadXml(doc.OuterXml);
                return newDoc;
            }
            else
            {
                return doc;
            }
        }

        static bool HasRiskyScanOperation(XmlDocument doc, string scanOperationName)
        {
            string pattern = string.Format("//RelOp[@LogicalOp='{0}']", scanOperationName);
            XmlNodeList nodeList = doc.SelectNodes(pattern);
            foreach (XmlNode node in nodeList)
            {
                if (NodeTargetsRiskyTable(node))
                {
                    return true;
                }
            }
            return false;
        }
        
        static bool NodeTargetsRiskyTable(XmlNode node)
        {
            foreach (var item in TableIndexScanRiskTables)
            {
                string pattern = string.Format(@"OutputList/ColumnReference[@Schema='[{0}]' and @Table='[{1}]']", item.Schema, item.Name);
                if (node.SelectNodes(pattern).Count > 0)
                {
                    return true;
                }
            }
            return false;
        }
    }

Here, the client code calls "PlanHasRiskyScanOperation" function by passing the execution plan.

An Algorithm for Restricting CPU Usage of CPU Bound Programs

For a CPU bound program which requires multi thread execution, we sometimes need certain number of processor cores to be used by our program and other cores to be left unassigned so that, other processes may utilize them.

Let's make the idea more clear:

I develop an exe application which performs long running calculations, such as running heuristic algorithms. During the calculations, I want to reserve some CPU resource for other processes so that my program will not "eat up" all the CPU resource.

Processor assignment can be achieved by setting processor affinity to threads:

C# Code:

        [DllImport("kernel32")]
        static extern int GetCurrentThreadId();

        static int unusedProcessorCount = 1;

        static void SetProcessorAffinity(int threadOrder)
        {
            int curThreadId = GetCurrentThreadId();
            int processorOrder = threadOrder % (Environment.ProcessorCount - unusedProcessorCount);
            int processorAddress = 1 << processorOrder;
            foreach (ProcessThread th in Process.GetCurrentProcess().Threads)
            {
                if (th.Id == curThreadId)
                {
                    th.ProcessorAffinity = (IntPtr)processorAddress;
                }
            }
        }

We can specify the number of cores that we want to be unassigned by setting the variable "unusedProcessorCount".

In the code above, the function that fetches current thread id needs to be imported from kernel32.dll. 

As soon as the function that thread executes is called, this function needs to call SetProcessorAffinity by passing its thread order so that the thread is assigned to the processor which comes up next.

Retrieving All System Details by Windows PowerShell

Have you ever heard about Windows PowerShell? PowerShell is a great scripting tool for administrative tasks on computer.

So why is it "power"? Because it extends standard shell scripting with the power of .NET Framework. While accessing standard "command let"s of PowerShell, you can use any .NET Framework class. In addition to this, standard cmdlet library of PowerShell is quite enough for basic tasks that a system administrator may need.

Common Scenario: I want to display, all essential details of any computer on network. I want something quite more "advanced" than typing systeminfo on Windows command prompt.

The details I need are:             

n  Computer Name

n  Operating System,

n  Model

n  Manufacturer

n  Last Bootup Time

n  IP Address

n  Physical Memory Details: Capacity of each RAM on slots

n  Processor Details: For each  physical processor; number of cores, number of logical processors

n  Logical Disk Details: For each disk, free and total size

n  Process Details: Process name, Id, Memory Usage, Percent Load

We can achieve all, for any computer on network, by specifying computer name only.

Here is the script:
(You can run this script on Windows PowerShell ISE)

$computername = Read-Host 'Please Enter Computer Name'

$Name = Get-WmiObject win32_OperatingSystem -computer $computername | Select csName
$OperatingSystem = Get-WmiObject win32_OperatingSystem -computer $computername | Select Caption
$Model = get-wmiObject Win32_ComputerSystem  -computer $computername | select Model
$Manufacturer = get-wmiObject Win32_ComputerSystem  -computer $computername | select Manufacturer
$LastBootTime = Get-WmiObject Win32_OperatingSystem -computer $computername | select csname, @{LABEL='LastBootUpTime'; EXPRESSION={$_.ConverttoDateTime($_.LastBootupTime)}}
$PhysicalMemoryList = Get-WmiObject Win32_PhysicalMemory -computer $computername | select DeviceLocator, @{Name='Capacity_MB'; Expression={$_.Capacity / 1024 / 1024}}
$FreeMemoryMB = get-WmiObject Win32_PerfRawData_PerfOS_Memory -computer $computername | select AvailableMBytes 
$DiskList = Get-WmiObject Win32_LogicalDisk -computer $computername | select Caption, Description, @{Name="Total_Size_GB"; Expression={$_.Size/1024/1024/1024}}, @{Name="Free_Size_GB"; Expression={$_.FreeSpace/1024/1024/1024}}
$ProcessorList = Get-WmiObject Win32_Processor -computer $computername  | select Name, NumberOfCores, LoadPercentage, NumberOfLogicalProcessors
$allIpList = [System.Net.Dns]::GetHostEntry($computername) | select AddressList
$IP = ''
foreach ($item in $allIpList.AddressList)
{
 if ($item -notlike '*:*')
 {
  $IP = $item
 }
}

$ResObject = @{
ComputerName = $Name.csName
OperatingSystem = $OperatingSystem.Caption
Model = $Model.Model
Manufacturer = $Manufacturer.Manufacturer
LastBootUpTime = $LastBootTime.LastBootUpTime
FreeMemoryMB = $FreeMemoryMB.AvailableMBytes
IPAddress = $IP.IpAddressToString
#PhysicalMemoryList = $PhysicalMemoryList | Format-Table
#DiskList = $DiskList | Format-Table
#ProcessorList = $ProcessorList | Format-Table
}

$ResObject | Format-List

Write-Host 'Physical Memory Details: '
$PhysicalMemoryList | Format-Table

Write-Host 'Disk Details: '
$DiskList | Format-Table

Write-Host 'Processor Details: '
$ProcessorList | Format-Table

$answer = Read-Host "Do you want to get process details? Yes[Y]; No[N]"
if ($answer.ToUpper() -ne 'Y')
{
 Read-Host 'Script Ended'
 Exit
}

$procListDef = Get-WmiObject win32_process -computer $computername | select Name, ProcessId, @{Name='Owner'; Expression={$_.getowner() | select Domain, User}  }
$procListResource = get-WmiObject Win32_PerfFormattedData_PerfProc_Process -computer $computername | select Name, IDProcess, PercentProcessorTime, Timestamp_Sys100NS, WorkingSet

foreach ($proc in $procListDef )
{
 $ownerFullName = $proc.Owner.Domain + '\' + $proc.Owner.User;
 $res = $procListResource | Where-Object {$_.IDProcess -eq $proc.ProcessId}
 $percent = $res.PercentProcessorTime / [Environment]::ProcessorCount;
 $ws = $res.WorkingSet / 1024 / 1024
 
 $proc | Add-Member -type NoteProperty -name OwnerName -value $ownerFullName 
 $proc | Add-Member -type NoteProperty -name MemoryInMb -value $ws
 $proc | Add-Member -type NoteProperty -name PercentProcessorTime -value $percent
}

$procListDef | select Name, ProcessId, OwnerName, MemoryInMb, PercentProcessorTime | sort PercentProcessorTime -Descending | Format-Table
Read-Host 'Script Ended'

Sql Server Delayed Query Execution: Delay until databases available

In some scenarios, the databases that our query references, may disappear for a period of time and we need to wait until these target databases are available for querying.

Continuously looking at Sql Server Management Studio Object Explorer and wait the database to be online is a quite boring activity.

Common scenario: In Data Warehouse (DWH) environment, the databases we are querying refreshes in regular period from production (live) environment. Generally, this period is daily. During the refresh period, target databases disappear and they are inaccessible until  new version is restored. This process takes several hours for enterprise databases.

This kind of wait operation can be assigned to a T-SQL query, so our query runs immediately when target databases are available (in ONLINE state).

Firstly, here is the stored proc that determines whether databases are in ONLINE state:

CREATE PROC.IS_DB_READY @IS_READY BIT OUTPUT
AS
BEGIN
 SET @IS_READY = 1

 DECLARE @DATABASE_NAME_LIST TABLE (DATABASE_NAME VARCHAR(100))

 -- Insert here the names of the databases
 -- that you're waiting to be online.
 INSERT INTO @DATABASE_NAME_LIST
 SELECT 'myDatabaseName1'

 INSERT INTO @DATABASE_NAME_LIST
 SELECT 'myDatabaseName2'

 ------------------------------------------------------------------------
 IF EXISTS (
   SELECT 1
   FROM @DATABASE_NAME_LIST L
   LEFT JOIN sys.databases D WITH (NOLOCK) ON L.DATABASE_NAME = D.NAME
   WHERE ISNULL(D.state_desc, '') <> 'ONLINE'
   )
 BEGIN
  SET @IS_READY = 0
 END
END

Now we can write the proc that delays the execution until databases are online:

CREATE PROCEDURE DELAY_UNTIL_DB_READY
AS
BEGIN
 DECLARE @IS_DB_READY BIT

 WHILE 1 = 1
 BEGIN
  EXEC IS_DB_READY @IS_DB_READY OUTPUT

  IF @IS_DB_READY = 0
   WAITFOR DELAY '00:00:10'
    -- Wait for 10 seconds.
    -- Equivalent of C# Thread.Sleep(10000);
  ELSE
   BREAK;
 END
END

Now developer only needs to execute the procedure for delaying execution until all databases are in online state.
Sample usage:

EXEC DELAY_UNTIL_DB_READY
-- Your queries to run after databases become online

Compare IL Content of .dll and .exe Files (Assemblies)

In the programs we develop, sometimes we need to be sure if versions of the compiled code (.dll, .exe)  we wrote are equivalent of each other on various environments.

To concretize the scenario, let's consider this:
In enterprise solutions, generally there are many environments prior to production environment. In my case, these environmets are:

-- DEV: Development.
-- UAT: User Acceptance Test.
-- PREPROD: Pre-Production.
-- PROD: Production. (LIVE)

At the source control side, each of them has corresponding branch. (In my case, I have branches in TFS).
After developer is finished with development phase, he performs merge operations on branches in order to make his code tested and running on target environment.

Sometimes, developer cannot be sure if the running code is the same as he merged. So he needs to check, if the running assembly (.dll, .exe) version is the same as he coded, or some problem occured while merging the branches.

To smooth away these kind of suspicion, contents of assemblies can be compared to check if they are equal.
In a "headlong" way, if such a comparison is performed in bitwise manner, the result of the comparison will be incorrect. Because, compiler may put some code-irrelevant details into the assembly at the moment of compilation, which should not affect the result of comparison. For example, CSharp compiler injects date and time of build into the dll. These kind of details does NOT affect the code running in dll.

The comparison program should compare equality of running code in assembly, and only this.
So, the only way to check: Inspecting MSIL (Microsoft Intermediate Language), or IL in short, in the assembly, which is the output of compiler and input of JIT compiler.

We can retrieve IL content of an assembly through ildasm.exe, Microsoft IL disassembler tool which is shipped with Visual Studio.

After IL content is written to a text file, the program we write can compare to generated IL's line by line. If any line is different, then we can say these two assemblies are different.

Some points to consider:
-- The IL output of ildasm.exe contains comment lines. These lines need to be removed before comparison.
-- The compiler may inject GUID attributes to some types. So, the user may need to have an option of ignoring Guid attributes.

Steps of comparison
-- Call ildasm.exe to generate IL content and redirect the output text file
-- When .exe terminates, read text file newly created.
-- While reading the text file, line by line, remove comments by Regexy (//.*)
-- Merge lines and remove guid attributes with regex.
-- Compare two strings. If they are equal, then IL contents in asssemblies are equal.

Code:

        string GetIL_ContentOfAssembly(string assemblyPath, bool ignoreGuidAttribute)
        {
            string tmpFileNameWithoutExt = Guid.NewGuid().ToString();
            string tempFileNameIL = string.Format("TmpIL_{0}.il", tmpFileNameWithoutExt);
            StoreIL_InTextFile(assemblyPath, tempFileNameIL);
            StringBuilder sb = new StringBuilder();
            using (StreamReader sr = new StreamReader(tempFileNameIL))
            {
                string str = null;
                while ((str = sr.ReadLine()) != null)
                {
                    str = RemoveComments(str);
                    sb.AppendLine(str);
                }
            }
            string res = sb.ToString();
            if (ignoreGuidAttribute)
            {
                res = RemoveGuidAttributes(res);
            }
            return res;
        }


        string RemoveComments(string input)
        {
            return Regex.Replace(input,
                                 @"//.*",
                                 string.Empty);
        }

        string RemoveGuidAttributes(string input)
        {
            return Regex.Replace(input,
                                 @"(\.custom instance void \[mscorlib\]System\.Runtime\.InteropServices\.GuidAttribute::\.ctor\(string\) = \().[^.]*",
                                 string.Empty,
                                 RegexOptions.Multiline);
        }

        void StoreIL_InTextFile(string assemblyPath, string tempFileName)
        {
            ProcessStartInfo info = new ProcessStartInfo
            {
                FileName = ildasmPath,
                Arguments = string.Format("\"{0}\" /output:{1}", assemblyPath, tempFileName),
                CreateNoWindow = true,
            };
            Process proc = new Process();
            proc.StartInfo = info;
            proc.Start();
            proc.WaitForExit();
        }

The ildasmPath is read from .config file in this example, which is set to: C:\Program Files\Microsoft SDKs\Windows\v6.0A\Bin\ildasm.exe

You can dowlonad the solution here

Rule Expression User Input Validation

If you are developing a business rule engine; your application needs to have an admin page that rules are defined through.

For example, imagine we are creating campaign rules:
"If the sum of the amounts of customers transactions within last 24 hours exceeds 10.000 or count exceeds 10 then send him an SMS"

In the rule definition pane, admin user inputs an expression like: "(R1 || R2) && R3"
Rule expressions are generally written in a free text format by the user.  Because user may insert parentheses, group rules and combine them.

The input of user has to be validated before saving it. Because syntax of such an expression above may be incorrect, may have missing parentheses, etc.

The easiest and effortless way of performing this validation is, using JavaScript eval() function:

function IsValidRuleText() {
    var txt = $('#txtInput').val();
    if (txt.length == 0) {
        return false;
    }
    var replaced = txt.replace(/R\d+/gi, "true");
    try {
        if (eval(replaced) == true) {
            // Trivial Comparison. Just for test if throws error.
        }
    } catch (err) {
        return false;
    }
    return true;
}

All the numbered [R] expressions are determined with regex. Then regex matches are replaced with "true" string. Now, final string contains "true" words and punctuation characters ((, ), &&, || ). If the "eval()" function successfully evaluates the replaced expression, then the expression is valid. If it throws error, then the expression has syntax errors.