Slow Assembly loading in Intranet environment

By Eli Ofek at November 08, 2007 16:37
Filed Under: .Net, .Net, Proffesional, Proffesional

Hi,

Lately we ran into a weird problem when using Enterprise Library 3.1 in the Intranet environment,
Where Internet connection is not available.

The symptom was a slow loading of the referenced assemblies during the application startup.
We managed to reproduce the problem using a simple console application:


________________________________________________________________________________________________

using System;
using System.Diagnostics;
using System.Collections.Generic;
using System.Text;

namespace TestCrlSample
{
class Program
    {
        static void Main(string[] args)
    {
        Console.WriteLine("Started at:" + DateTime.Now.ToString());
        Debug.WriteLine("Started at:" + DateTime.Now.ToString());

        Stopwatch sw = new Stopwatch();
        sw.Start();
        DoSignedLibWork();
        sw.Stop();

        Console.WriteLine("Total Time elapsed(Milliseconds):" + sw.ElapsedMilliseconds);
        Console.WriteLine("Ended at:" + DateTime.Now.ToString());
        Debug.WriteLine("Ended at:" + DateTime.Now.ToString());
        Console.WriteLine("Press any key to exit...");

        Console.ReadLine();
}

    public static void DoSignedLibWork()
    {
        Console.WriteLine("Before Work at:" + DateTime.Now.ToString());
        Debug.WriteLine("Before Work at:" + DateTime.Now.ToString());
        Microsoft.Practices.EnterpriseLibrary.Data.ConnectionString cs = new
Microsoft.Practices.EnterpriseLibrary.Data.ConnectionString("Data Source=DBSRV;Initial
         Catalog=Repository;Integrated Security=True"
, "Admin", "Bla");
        Console.WriteLine("After Work at:" + DateTime.Now.ToString());
        Debug.WriteLine("After Work at:" + DateTime.Now.ToString());
    }

    }
}

________________________________________________________________________________________________

Here are some test results we got when diagnosing the problem:

Running the program as usual in an Intranet environment, no Internet connection at all:

___________________________
Started at:22/07/2007 18:16:47
Before Work at:22/07/2007 18:16:56
After Work at:22/07/2007 18:16:56
Total Time elapsed(Milliseconds):9234
Ended at:22/07/2007 18:16:56
Press any key to exit...
___________________________

Notice how long it took this simple program to run… almost 10 seconds !
We ran Microsoft Network Monitor during the test to check what's going on behind.

This is the Netmon output:

_____________________________________________________________________________________
111    5.019387        DNS    DNS: QueryId = 0xAEAE, QUERY (Standard query), Query for crl.microsoft.com of type Host Addr on class Internet
112    5.024270        DNS    DNS: QueryId = 0xAEAE, QUERY (Standard query), Response - Server failure
113    5.024270        DNS    DNS: QueryId = 0xAEAE, QUERY (Standard query), Query for crl.microsoft.com of type Host Addr on class Internet
123    6.024270        DNS    DNS: QueryId = 0xAEAE, QUERY (Standard query), Query for crl.microsoft.com of type Host Addr on class Internet
124    6.024270        DNS    DNS: QueryId = 0xAEAE, QUERY (Standard query), Response - Server failure
125    6.024270        DNS    DNS: QueryId = 0xAEAE, QUERY (Standard query), Query for crl.microsoft.com of type Host Addr on class Internet
126    6.024270        DNS    DNS: QueryId = 0xAEAE, QUERY (Standard query), Query for crl.microsoft.com of type Host Addr on class Internet
127    6.024270        DNS    DNS: QueryId = 0xAEAE, QUERY (Standard query), Query for crl.microsoft.com of type Host Addr on class Internet
128    6.024270        DNS    DNS: QueryId = 0xAEAE, QUERY (Standard query), Query for crl.microsoft.com of type Host Addr on class Internet
129    6.024270        DNS    DNS: QueryId = 0xAEAE, QUERY (Standard query), Response - Server failure
130    6.024270        DNS    DNS: QueryId = 0xAEAE, QUERY (Standard query), Response - Server failure
131    6.024270        DNS    DNS: QueryId = 0xAEAE, QUERY (Standard query), Response - Server failure
184    10.024270        DNS    DNS: QueryId = 0xAEAE, QUERY (Standard query), Query for crl.microsoft.com of type Host Addr on class Internet
227    14.025102        DNS    DNS: QueryId = 0xB6AF, QUERY (Standard query), Query for crl.microsoft.com.myorg.com of type Host Addr on class Internet
228    14.030961        DNS    DNS: QueryId = 0xB6AF, QUERY (Standard query), Response - Name Error
229    14.030961        DNS    DNS: QueryId = 0xBFAC, QUERY (Standard query), Query for crl.microsoft.com.myorg.com of type Host Addr on class Internet
230    14.035844        DNS    DNS: QueryId = 0xBFAC, QUERY (Standard query), Response - Name Error

_____________________________________________________________________________________

Notice how the DNS requests for "crl.microsoft.com" took almost 9 seconds !
This was very funny, so we googled for this symptom.

Internet research shows these results:

Support Certificates In Your Applications With The .NET Framework 2.0:
http://msdn.microsoft.com/msdnmag/issues/07/03/NETSecurity/default.aspx

Microsoft, VeriSign, and Certificate Revocation:
http://amug.org/~glguerin/opinion/revocation.html

How Office Performs Certificate Revocation:
http://office.microsoft.com/en-us/ork2003/HA011403081033.aspx

This one is talking about IE slowness:
http://www.wilderssecurity.com/archive/index.php/t-47121.html

Management Studio slowness:
http://weblogs.sqlteam.com/tarad/archive/2006/10/05/13676.aspx
http://blogs.msdn.com/dtjones/archive/2006/08/23/714738.aspx
talking about the similar symptoms, only when using MS SQL Management Studio.
The cause of the problem is the same…

FAQ, Why does SSMS take 45s to start up?
http://blogs.msdn.com/euanga/archive/2006/07/11/662053.aspx

Why does the .NET Runtime Optimization Service keep trying to use the internet:
http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=522726&SiteID=1

so, what have we got here?
It seems that Microsoft added a mechanism to the .Net CLR that checks every signed assembly when loading it
against an online revocation list.
When working in an intranet environment with IP segments
that are no defined as local Intranet, the mechanism is trying to locate the crl server for 9 seconds before it
gives up, delaying the assembly loading.

When checking the Enterprise Library Common assembly,
We see that is indeed signed using a Microsoft Certificate:



How do we work around it you say ?
Well, there are several options.
The easiest one is to disable the crl check. Oddly it is done from the Internet Options dialog available from Internet Explorer:



Now, let's run our test program again:

Running the program after removing the check:

__________________________
Started at:22/07/2007 18:17:47
Before Work at:22/07/2007 18:17:47
After Work at:22/07/2007 18:17:47
Total Time elapsed(Milliseconds):147
Ended at:22/07/2007 18:17:47
Press any key to exit...
__________________________

Notice that this time we are down to a total of 147 milliseconds, which is much more reasonable.
What about the netmon output ? Well, since there is no check
Netmon output is empty !

What about security you say?
If you are canceling the crl check, you are exposed to bogus certifications.
My answer to that is: If you are already disconnected from the Internet,
then you are just exposed as before, only this time you are not slowing down your applications for nothing.

A questions might be asked about situations where Internet connection is partly available through a firewall.
In this situation you might consider asking the network administrator to allow connections to the crl server,
Or if you which, make it fail the requests immediately, so you don't need to configure the Internet Options for every node.

Another optional workaround is to define the crl address in the etc/hosts file pointing it to localhost (127.0.0.1),
which will quickly fail every crl request it gets.

Let's run our test again, this time without removing the revocation check, but defining the crl address in the hosts file:

Running the program after redirecting the dns name to localhost:

___________________________
Started at:22/07/2007 18:18:37
Before Work at:22/07/2007 18:18:37
After Work at:22/07/2007 18:18:37
Total Time elapsed(Milliseconds):219
Ended at:22/07/2007 18:18:37
Press any key to exit...
___________________________

Notice that this time we are using 219 milliseconds, which is a bit more then the first workaround, but still reasonable.
What about the netmon output? Well, since there is no outside communication
Netmon output is empty in this case too!


Conclusion:

When working with signed assemblies having no Internet connection to crl.microsoft.com,
you need to consider the loading delay time for the assemblies, Or work around them as suggested above.

Good Luck,

Eli.

Add comment



  Country flag
biuquote
  • Comment
  • Preview
Loading


RecentPosts

Calendar

<<  May 2019  >>
MoTuWeThFrSaSu
293012345
6789101112
13141516171819
20212223242526
272829303112
3456789

View posts in large calendar

Page List

    Disclaimer

    I work for Microsoft Israel as s Senior Premier Field Engineer.
    The opinions expressed here are my own personal opinions and do not represent my employer's view in anyway.