Steve's Blog

Random comments on .NET, debugging, and more.

Implementing SOS with SPT - Part 3 of N - DumpMD & IP2MD

This is part 3 of N of my SOS series

It's a Saturday, so I'm going to do a nice easy one today, DumpMD, and IP2MD.  I group these together because ip2md is just a level of indirection from DumpMD.

Let's start with a quick refresher of !dumpmd:

0:036> !dumpmd 5b2351ec 
Method Name:  System.Web.HttpContext.Init(System.Web.HttpRequest, System.Web.HttpResponse)
Class:        5b220e1c
MethodTable:  5b458094
mdToken:      06002627
Module:       5b201000
IsJitted:     yes
CodeAddr:     5b3bb0f0
Transparency: Safe critical

Most things here are self explanatory.  The only thing many people may not be familiar with is the "Transparency" row.  Transparency is a core part of the .NET security model, I'm not going to get into it, but you can read about it on MSDN.

DumpMD is very simple because almost all of the displayed fields are obtained via IXCLRDataProcess3::GetMethodDescData, which takes a methodDesc address and returns a ClrMethodDescData structure.  Name we can get via IXCLRDataProcess3::GetMethodDescName.  The EEClass ("Class") can be retrieved via IXCLRDataProcess3::GetMethodTableData with the methodTable address on ClrMethodDescData.  All we're left with is Transparency, which we can get with IXCLRDataProcess3::GetMethodDescTransparencyData.

IP2MD adds one layer on top of DumpMD, it figures out the MD first, then runs the rest.  Given an arbitrary IP, IXCLRDataProcess3 has a convenient method to resolve it to a MethodDesc, unsurprisingly named GetMethodDescPtrFromIP.  Calling this will give us the MethodDesc address for that IP, which we can then pass to DumpMD.

As always, the source for this is on my github.

Implementing SOS with SPT - Part 2 of N - DumpStackObjects

This is part 2 of N of my SOS series.

For part 2, I’m going to talk about DumpStackObjects.  This one is fairly straight forward, and we already have most of the functionality required built in SPT, so all that is left is to hook it up.

The general algorithm for scraping the stack is this:

  1. Find the stack limits (stackBase, stackLimit)
    • stackBase can be pulled from the current thread’s TEB (thread environment block)
    • stackLimit is the value of [e/r]sp for that thread.
  2. Get the bounds of the GC heap segments.  This algorithm can be found in CSDbgExt::EnumHeapSegments.
  3. Start at stackLimit, moving 1 pointer at a time, until we hit stackBase
    1. Read a pointer at our current stack location
    2. Use IClrProcess::IsValidObject to check if the address is valid or not
    3. If valid, also make sure it’s within the GC heap segments found in (2)
    4. We can find the MT using IXCLRDataProcess3::GetObjectData.  For strings, we can read their value easily using IXCLRDataProcess3::GetObjectStringData.

Once we’ve done this, there’s one more thing to consider: some registers may also contain CLR objects.  We can evaluate all the registers in a similar way to #3 above.

You can find the source for this on github here.

Fun with .NET remoting - Building a RealProxy implementation

[Steve’s note: I wrote this post a year or so ago and never had posted it.  It’s still relevant today, but I don’t recommend using shady things like this in production code without understanding the implications.]

A side project I worked on recently involved building an abstraction layer around DB access, specifically calling stored procedures.  My goal was to reduce code redundancy and add type-safety to the parameters of stored proc calls.  The road I went down involved creating an interface for a group of stored procs, with the interface methods mapping to the stored procs.  I might resurrect this as an example and post the code on github, because it actually worked out really well.

What's a RealProxy?

The .NET remoting infrastructure is built around 3 key classes:

  1. MarshalByRefObject
    • This is the "opt-in" marker for by-ref object marshaling.
  2. RealProxy
    • This is the base type for method/constructor call dispatch over remoting.  The CLR has special knowledge of this class, and this is the managed entry/exit point for calls dispatched through a TransparentProxy.  Calls enter and leave managed code via RealProxy.PrivateInvoke.
  3. __TransparentProxy
    • The other piece of the proxy pair.  The CLR has special knowledge of this class in terms of type casting and method invocation.

Implementing RealProxy only requires you to implement Invoke(IMessage msg) and do whatever you want inside it.  In my above example, I took the method name and parameters and handed them off to a SqlCommand and ran ExecuteReader, ExecuteNonQuery, etc.  The implementation details for the synchronous case aren't that exciting.

Supporting BeginInvoke/EndInvoke

One big feature I wanted to implement was asynchronously executing SQL via BeginInvoke/EndInvoke on the proxy methods.  Unfortunately, Microsoft made this impossible to do without resorting to some sketchy reflection.  Inside RealProxy.PrivateInvoke, there's a block of code that looks like this:

if (!this.IsRemotingProxy() && ((msgFlags & 1) == 1))
{
    Message m = reqMsg as Message;
    AsyncResult ret = new AsyncResult(m);
    ret.SyncProcessMessage(retMsg);
    retMsg = new ReturnMessage(ret, null, 0, null, m);
}

There are a few things going on here

  • The constants for msgFlags are defined in System.Runtime.Remoting.Messaging.Message. We'll need this value later on.  The interesting values are:
    • BeginAsync (eg a call to BeginInvoke) = 1
    • EndAsync (EndInvoke) = 2
    • OneWay (eg a method with a [OneWay] attribute on it) = 8
  • The consequence of the block of code above is that the remoting infrastructure will run BeginInvoke synchronously for any RealProxy that isn't a RemotingProxy.  I'll get into details on this in a bit.
  • IsRemotingProxy checks the _flags field on RealProxy.  This is set in the RealProxy constructor if the type is RemotingProxy.  We can't inherit from RemotingProxy (it's internal), but we can use reflection in our own constructor to set this to RealProxyFlags.RemotingProxy (1).  Once we do this, a "normal" Invoke implementation won't work correctly for async invocations.

How proxy invocation works

So back to msgFlags and how asynchronous method invocation works with the remoting infrastructure.  As I mentioned above, there's two possible code paths for an async invoke on a RealProxy, the first is the "normal" code path that you get when you invoke a delegate bound to a non-RemotingProxy instance.  In this case, the code path looks like this:

  1. someDelegate.BeginInvoke(...)
  2. Native CLR
  3. RealProxy.PrivateInvoke, msgFlags = 1 (BeginAsync)
  4. PrivateInvoke calls our Invoke(...)
    • Since our Invoke isn't aware of anything async (in fact, with the public API there's no way to know), it runs synchronously.
  5. PrivateInvoke sets up an AsyncResult instance with the result from Invoke(...) and returns this.  Note, at this point, everything has run synchronously, our BeginInvoke was exactly the same as an Invoke, except the last step of returning an IAsyncResult.  At this point, all our work is done.
  6. We call someDelegate.EndInvoke(iar)
  7. RealProxy.PrivateInvoke, msgFlags = 2 (EndAsync)
  8. PrivateInvoke calls EndInvokeHelper, which takes care of returning the return value or throwing an exception if one occurred.

The pro of this code path is that you (the RealProxy implementer) don't need to worry about the async code path, in fact, you can't.  The con is that there's no way to get a truly async invocation with this.

Now, the alternate code path is "enabled" by telling the infrastructure that we're a RemotingProxy.  This code path looks roughly like this

  1. someDelegate.BeginInvoke(...)
  2. Native CLR
  3. RealProxy.PrivateInvoke, msgFlags = 1
  4. PrivateInvoke calls our Invoke, this time however, it's expecting an IAsyncResult in return.
    • We set up whatever async stuff we want to do here and start it.
    • FUN FACT: You can actually return whatever you want here!  The CLR is missing the type check to make sure your ReturnMessage actually returns an IAsyncResult.  The resulting chaos that ensues from not returning the correct type is entertaining.  I made a post about it on StackOverflow awhile ago here.
  5. PrivateInvoke returns, we have an IAsyncResult
  6. We call someDelegate.EndInvoke(iar)
  7. PrivateInvoke calls Invoke, msgFlags = 2
  8. Invoke handles getting the correct return value or throwing an exception if one occurred.  Possibly through RealProxy.EndInvokeHelper as above.

As you can see here, there's a lot more burden on us to "get things right." with the implementation.  We now need to handle synthesizing an IAsyncResult, invoking a callback (if passed in), raising an exception in the EndInvoke (if thrown), and getting the value back.  Also, since none of this is exposed publically via the API, it makes things even more difficult because we now need to rely on reflection.  I'm going to drill into step 4 of the second code path and explain how it works.

Implementing an async-aware Invoke

The first problem of implementing an async-aware Invoke is getting the msgFlags.  This is the core of Invoke, and we'll be using it to branch to synchronous/BeginInvoke/EndInvoke/OneWay code paths.  Getting the value is pretty easy, looking at System.Runtime.Remoting.Messaging.Message, you can see there's a method called GetCallType.  Unfortunately this isn't exposed in any of the public interfaces Message implements, and Message itself is internal, so we need to use reflection to call it. (int)msg.GetType().GetMethod("GetCallType").Invoke(msg, null) will do the trick.

At this point, we need to branch for a few conditions.  The 4 main cases are: Begin, End, OneWay, Synchronous.  The simplest way is to create a method for each one and call them in Invoke depending on call type.  Begin is the most interesting case because it requires the most work, so I'm going to focus on that.

BeginInvoke is responsible for 3 main tasks: creating the IAsyncResult to return, starting the async work, and handling the task completion.  It turns out the framework exposes a class System.Runtime.Remoting.Messaging.AsyncResult that wraps a lot of the more complicated async completion logic.  For example, the message we get in Invoke doesn't contain the last two "extra" parameters to BeginInvoke, the AsyncCallback delegate and state (it accomplishes this through Message.GetAsyncBeginInfo.)  AsyncResult will get these for us, and once we give the AsyncResult the IMessage result, it'll handle invoking the callback if needed.

I’ll write more about EndInvoke at a later point, but it’s a fairly simple implementation.

Implementing SOS with SPT - Part 1 of N - DumpObj

I decided to have some fun re-implementing SOS methods using SPT.  I chose DumpObj to start with because it’s one of the more complicated methods in SOS.  Hopefully this will be part 1 of a multipart series.

Let’s start by looking at some sample output from !do on a hashtable…(sorry for the weird layout, I’m going to try to change my blog theme at some point soon).

0:036> !do 11faa984 
Name:        System.Collections.Hashtable
MethodTable: 7370d288
EEClass:     7337c4a0
Size:        52(0x34) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
7370ae80  4000c01        4 ...ashtable+bucket[]  0 instance 11faa9b8 buckets
7370c770  4000c02       18         System.Int32  1 instance        2 count
7370c770  4000c03       1c         System.Int32  1 instance        1 occupancy
7370c770  4000c04       20         System.Int32  1 instance        2 loadsize
737070b4  4000c05       24        System.Single  1 instance 0.720000 loadFactor
7370c770  4000c06       28         System.Int32  1 instance        2 version
73707208  4000c07       2c       System.Boolean  1 instance        0 isWriterInProgress
733223ec  4000c08        8 ...tions.ICollection  0 instance 00000000 keys
733223ec  4000c09        c ...tions.ICollection  0 instance 00000000 values
7370222c  4000c0a       10 ...IEqualityComparer  0 instance 00000000 _keycomparer
7370b350  4000c0b       14        System.Object  0 instance 00000000 _syncRoot

We can go through this line by line and figure out what’s going on.

  1. MethodTable : This can be found on an object via IXCLRDataProcess3::GetObjectData.
  2. Name : This is the type name, it can be obtained via IXCLRDataProcess3::GetMethodTableName with the method table from above.
  3. EEClass : IXCLRDataProcess3::GetMethodTableData will give you the EEClass, as well as the Module (which we need in a minute).
  4. Size : This is on the ClrObjectData we got in #1.
  5. File : Given the Module (from #3), we can get the assembly that contains it via IXCLRDataProcess3::GetModuleDataOnce we have the assembly, IXCLRDataProcess3::GetAssemblyName will give you the full path to an assembly.
  6. Name : This one is a little interesting.  As far as I can tell, IXCLRDataProcess doesn’t directly expose a way to get a field name, however, we can use IMetaDataImport to get the values.  The method of getting an instance of IMetaDataImport is fairly simple, once you know it’s implemented by the module itself.  We can use IXCLRDataProcess3::GetModule (with the module address from #3) to get an IUnknown.  That can be QI’d for IID_IMetaDataImport.  Once we have that pointer, we can GetMemberProps to get the name.

Fields

Fields are the fun part.  The whole algorithm for iterating over a classes fields is in ClrProcess::FindFieldByNameExImplThe basic idea is:

  1. Get to the root type in the inheritance hierarchy
  2. For each type, traverse the linked list of FieldDescs (obtained via GetMethodTableFieldData.FirstField, stop traversing when you’ve seen all instance fields and static files on that type. 
  3. Once you’ve seen them all, go to the next type in the hierarchy and repeat #2.  Note, NumInstanceFields includes the fields inherited from parent classes as well.  This is why we’re starting at the root (System.Object) and moving down, otherwise we wouldn’t know when to stop traversing each list of fields.

Once we have all the fields, we can start drawing each line of output.  Most are fairly trivial to get.

  1. MethodTable : Using the ClrFieldDescData (from IXCLRDataProcess3::GetFieldDescData) saved during iterating over the fields, we can get the FieldMethodTable.
  2. Field : This is the token for the field.
  3. Offset : This is simply the Offset from the field data.  However, instance fields need to be offset by sizeof(void*) to account for the type handle (MethodTable *) at the start of every object.  Static fields do not need to be offset since they aren’t actually stored on the object instance.
  4. Type : This is the same as the typename from above, using the FieldMethodTable.
  5. VT : As far as I can tell, there’s no simple “IsValueType” flag anywhere.  However, you can use the FieldType to figure it out.  It corresponds to the CorElementType enum in corhdr.h.  Anything before and including ELEMENT_TYPE_R8 is a value type, as well as ELEMENT_TYPE_VALUETYPE.  Everything else can be counted as class.
  6. Attr : I’ve only used two values here, “shared” (for IsStatic, Is[Context,Thread]Local), or “instance” for anything else.
  7. Value : The algorithm to read the values falls into two categories, which further break down into more subcategories.  The top level categories are instance vs static.  We’re not even going to try to get values for context local and thread local fields.

    Instance Fields

    Instance fields break down into 3 more subcategories, class, primitive, and value type.

    • Class is the simplest, just read one pointer at obj+offset+sizeof(void*).  Display that value
    • ValueType is similar, but one less level of indirection.  ValueTypes are stored in-type, not as a pointer to another object on the heap.  The address shown under value is simply obj+offset+sizeof(void*).  This address could then be passed to !dumpvc to get the value.
    • Primitive.  For these, we need to compute the size of the field via ClrProcess::GetSizeForType.  This method is simply a switch statement with the built-in primitives and their respective sizes.  Once we have a size, we can read S bytes at the correct offset (as above) into the object.

    Static Fields

    Static fields break down similar, but there is already logic in ClrProcess::GetStaticFieldValue to handle it.  The complication of static fields is we need to consider all app domains.  ClrProcess::GetStaticFieldValues handles all of this for us.  We can simply call it and display the results.  One thing I’ve noticed is that SOS displays “NotInit” on some fields, I’m not sure if it’s just doing a null check here, or there’s more logic to figure out if a domain has somehow initialized a static field yet.

    Putting that all together with some pretty formatting, we’ve got the same information as SOS’s !dumpobj.

    I’ve pushed this to my github repo in case anyone wants to check it out.

    Diving into SDbgExt2, the core interface

    The SDbgCore static library is the foundation of SDbgExt2.  It contains the definition of IXCLRDataProcess3 (I think it’s the 3rd revision, I just made the name up), as well as the core classes and interfaces IClrProcess and ISDbgExt.  It also contains many helper interfaces such as IClrObject(Array), and lots of callback interfaces.  We’ll go through these one by one.

    IXCLRDataProcess3

    This is the interface that makes everything work.  In the pervious version (.NET 4.0), almost all functions were executed via a single Request(…) method on the interface.  In .NET 4.5, these requests were split into strongly typed functions. 

    I started by reverse engineering the VTable via the embedded symbols in mscordacwks.dll.  The interface is implemented by ClrDataAccess, and is the 3rd interface in the VTable (hence IXCLRDataProcess3).  Next, I began reverse engineering the parameters.  This is obviously more difficult, but you can get an idea of the parameters in two ways. 

    First, since most functions are similar to their request counterparts in .NET 4, you can get an idea of the parameters via the request object.  Second, by inspecting the call sites in SOS, we can get an idea of the parameters being passed in.  It turns out that a large chunk of functions follow a few very similar signatures, so figuring out the parameter isn’t actually that tough.  Also as a side note, the x86 calling convention (stdcall) here makes it a lot easier to reverse engineer than x64, so I did most of my reversing on the x86 version.

    For the output structures, again I leveraged .NET 4.0’s structures.  Obviously there were some changes / additions / removals, but again by looking at the call site in SOS, it’s not too hard to figure out which fields do what.  Some of these structures are obviously incomplete, but there’s enough there to support the functionality I need.

    The result of this is IXCLRDataProcess3.idl, which MIDL compiles to a C++ interface and type library.

    ClrProcess

    ClrProcess is the lowest level new interface in SDbgExt2.  It contains various utility methods, field accessors, heap enumeration, sort-of-reflection, and thread inspection.  I tried to keep this interface as low-level as possible, which providing reusable methods for what would be used by the higher level methods.

    For example, accessing a static field (see ClrProcess::GetStaticFieldValue) via IXCLRDataProcess3 is a fairly complicated process.  This is the simplified algorithm:

    1. You need to find the actually field address (the FieldDesc)
      1. This can be done by scanning the method table recursively, and using IMetaDataImport to resolve a token to a name.
    2. Given a FieldDesc and app domain, figure out the module that contains it.
    3. Figure out if the module has data in the domain-neutral store or not.
      1. If so, use that to get the domain local module data
    4. Otherwise, use the app domain’s domain local module data.
    5. Using the DLM data, use the GCStaticDataStart, or NonGCStaticDataStart (for non-reference types) to find the base offset
    6. Add the field’s static offset to the base offset to get where in memory the field is
    7. Read from the memory offset to get the field value.

    Next up: more details on implementing ClrProcess

    Introducing SPT for .NET 4.5

    This has been a post long in the making.  I’d written this in December/January as a complete re-do of SPT for .NET 4.0,  (Side Note: Interestingly enough, in the mean time MS released ClrMD, which is similar to the direction I was going, but still significantly different (I’m focusing more on a lower level interface to be plugged into WinDBG, ClrMD is focused more on a managed vector).) but never released it publically.

    I’ve decided this time around to release the entire library under the GPL.  For anyone who would like to use this in commercial software, contact me and we can probably work something out.  I’ll be pushing the full source tree to my github account shortly.  https://github.com/steveniemitz/SDbgExt2.

    The initial release supports most of the original SPT commands:

    • DumpAspNetRequests
    • DumpDictionary
    • DumpSqlConnectionPools (aka !sqlblame)
    • DumpThreadPoolQueues
    • FindHttpContext
    • GetDelegateMethod

    In the next few days, I’ll be posting more about SDbgExt2 (and the WinDBG extension, SPT), it’s inner workings, and how the new IXCLRDataProcess3 interface works.  My main design goals for SDbgExt2 were:

    • Easily support a managed wrapper interface
    • Separate the core debugging interface from the “value add” interface.  As you can see, SDbgCore is a fairly small static library, which SPT links against.
    • Provide a cleaner code base which is easier to maintain and enhance in the future, as well as which abstracts IXCLRDataProcess as much as possible.
    • Have near-100% code coverage via unit tests from the start.  This is mostly achieved with pre-canned mini-dumps which the unit tests can operate on.  (Side note: for now I’m not releasing the mini-dumps as they contain machine-specific data, I may in the future release them if I can prove they contain no personal data, but for now the unit tests are being released as more of a reference guide.)

    I’m also posting pre-compiled, ready to run SPT.dll binaries for use in WinDBG (or elsewhere).

    Download here:  x86 / x64

    As always, feel free to post here or email me if you have any questions / comments.

    COM/C++ Programming – the modern way

    For a new project I’ve been working on I was forced into working with the world of C++ and COM for a sizable portion of it.  In my past life I’d previously worked on some large-ish projects involving COM (office interop in particular), but never on a fresh codebase from the ground up.

    I’ve found that life in “modern COM” is significantly different than the “old school” COM I had worked with before.  A lot of this is probably very obvious to people working with COM/C++ for years, but for someone working in a mostly managed environment without much enterprise C++ experience, a lot of it wasn’t obvious.  Below are some observations.

    Smart pointers are ridiculously good.

    I know some people disagree with this (like Eric Lipert), but I can honestly say that my code using CComPtr<T> and equivalents has at least 95% less bugs than code using AddRef() and Release() everywhere.  Further, the code is significantly more readable, and I probably was 50% more productive.  Not having to reason about the correctness of a function (are all objects being released?) in almost all cases allows much more time for actually solving problems.

    Further, using STL containers for dynamic memory allocation is significantly better than manually allocating storage.  Need a dynamic LPWSTR?  Use a vector<WCHAR> (although technically you can use a std::wstring and pass its buffer around, its not supported, and functions like length() behave differently than expected.  Need a BYTE[]?  Use a vector.  You get the point.

    Lambda Expressions in C++11 are amazing.

    It’s amazing, coming from C#, how much time and code lambda expressions can save you.  Lambda expressions get you two major things.

    1. Anonymous, inline functions
    2. Closures

    Manually creating closure classes and functors, or constantly casting PVOID state objects in callbacks wastes time and makes code ugly.  Anonymous functions make using algorithms that take predicates or other functions much easier.  Converting code from callback + PVOID pairs to use functors and lambdas made my code cleaner and easier to read.  Being able to convert almost ANY instance method to a functor (or even normal function) with a couple lines of code is also a huge win.

    The ATL is useful, but horribly documented.

    I started my project not planning on using ATL, but after implementing IUnknown for the 10th time, decided this was a mistake.  Unfortunately, I don’t think there’s a single good guide online for making a simple COM object with ATL.  In addition, almost every tutorial I could find went something like “open up the <something> wizard and click next.”  I hate wizards when it comes to coding with a passion so I promptly ignored the tutorials.  Here’s my 3 step guide:

    1. Inherit CComObjectBase and your COM interface (or just IUnknown)
    2. Use BEGIN_COM_MAP, COM_INTERFACE_ENTRY, END_COM_MAP to implement your interface
    3. Create instances of your object with CComObject<YourClass>::CreateInstance.

    That’s it!  Why is that not on the front page of the ATL docs on MSDN?  Note, you also need to define a CComModule somewhere in your library or you’ll get null reference exceptions in AddRef() on create.  Also, don’t forget ATL starts your refcount at 0, so you should AddRef immediately after CreateInstance.

    IDL makes interface with .NET trivial

    MIDL + TLBImp can round trip COM objects (and data structs) with almost 100% fidelity.  There are some gotchas though:

    • It seems like MIDL randomly will capitalize/lowercase members of structs and parameters.  I couldn’t figure out why, or the pattern, which is annoying.
    • TLBImp (because of limitations of type libraries) can’t handle variable sized OUT arrays, and instead define them as simple out variables in the managed interface.  The only way around this is to edit the IL (with ILdasm, ILasm), and add [] marshal([+1]) to the parameter.  The +1 tells the marshaller the index of the size parameter, 0 indexed left to right.  For example, you might end up with: [out] valuetype int[] marshal([+1]) pValues

    Visual Studio 2012 C++ Unit Testing is really good

    TDD is a great paradigm, but until 2012 there were few (no?) good, integrated solutions to unit test C++ code.  Visual Studio 2012 integrates C++ unit tests seamlessly into the IDE, just like managed unit tests.

    Conclusion

    The project that I started using these new technologies in is basically a rewrite of SPT for .NET 4.5 (I’ll be blogging about that soon).  It’s codebase is significantly cleaner and less buggy than the first version of SPT.  High code coverage with unit tests and smart pointers handling most COM objects contributed substantially to that.

    Intermittent hangs in an ASP.NET application when serving asmx files via HTTP GET

    I was recently debugging an intermittent, brief hang (5-15 seconds) in an ASP.NET application that would occur once every few days.  The application was fully precompiled (non-updatable) and running .NET 4.0.

    When the hang occurred, most threads would be sitting in a stack similar to this: (method signatures slightly modified for visibility reasons)

    System.Web.Compilation.BuildManager.GetBuildResultFromCacheInternal
    System.Web.Compilation.BuildManager.GetVPathBuildResultFromCacheInternal
    System.Web.Compilation.BuildManager.GetVPathBuildResultInternal
    System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert
    System.Web.Compilation.BuildManager.GetVirtualPathObjectFactory
    System.Web.Compilation.BuildManager.CreateInstanceFromVirtualPath
    System.Web.UI.PageHandlerFactory.GetHandlerHelper
    <snip>

    With one thread in a stack like this:

    System.Threading.WaitHandle.InternalWaitOne
    System.CodeDom.Compiler.Executor.ExecWaitWithCaptureUnimpersonated
    System.CodeDom.Compiler.Executor.ExecWaitWithCapture
    Microsoft.CSharp.CSharpCodeGenerator.Compile
    Microsoft.CSharp.CSharpCodeGenerator.FromFileBatch
    Microsoft.CSharp.CSharpCodeGenerator.CompileAssemblyFromFileBatch
    System.Web.Compilation.AssemblyBuilder.Compile
    System.Web.Compilation.BuildProvidersCompiler.PerformBuild
    System.Web.Compilation.BuildManager.CompileWebFile
    System.Web.Compilation.BuildManager.GetVPathBuildResultInternal
    System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert
    System.Web.Compilation.BuildManager.GetVPathBuildResult
    System.Web.UI.PageParser.GetCompiledPageInstance
    System.Web.UI.PageParser.GetCompiledPageInstance
    System.Web.Services.Protocols.DocumentationServerProtocol.GetCompiledPageInstance
    System.Web.Services.Protocols.DocumentationServerProtocol.Initialize
    System.Web.Services.Protocols.ServerProtocolFactory.Create
    System.Web.Services.Protocols.WebServiceHandlerFactory.CoreGetHandler
    System.Web.Services.Protocols.WebServiceHandlerFactory.GetHandler
    System.Web.Script.Services.ScriptHandlerFactory.GetHandler
    <snip>

    (note, I didn't have parameters to these stack traces, had I, the diagnostic would have been much simpler in hindsight)

    Every time this happened it'd be an ASMX file triggering the compile, never any other type.  I was pretty perplexed by this because:

    1. The application is fully precompiled, so ASP.NET should never attempt to compile anything.
    2. There's not really anything to compile in our ASMX files (they're just pointers to a class in a DLL), so even if ASP.NET did try to compile it, it shouldn't take any time.

    After a few days of continuing to be perplexed, I had a breakthrough.

    In ASP.NET, if you just hit an ASMX file via a browser, you get a pretty web page that lists all the methods on the service and allows you to invoke methods through the browser.  This page is actually an ASPX file (named DefaultWsdlHelpGenerator.aspx) that lives in the Config directory of whatever framework you're running (in my case it was C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config, ymmv).  Bingo, that's what's getting compiled!

    Once I figured this out, the solution was fairly simple.  ASP.NET allows you to specify the WSDL help generator file via system.web/webServices/wsdlHelpGenerator in your web.config.  I simply copied DefaultWsdlHelpGenerator.aspx into my application's root directory and added the wsdlHelpGenerator element to my web.config.  Now, when my application needs to generate a WSDL file or page, ASP.NET uses the precompiled help generator instead of the default one and doesn't trigger a compile.  Problem solved!

    The moral of the story here: if you have a precompiled site with asmx services and care about short hangs, do what I outlined above.  If you don't precompile, this is the least of your worries.

    SPT - x86 Release

    Read more about SPT here

    I've compiled the latest build of SPT for x86.  All functionality should be working fine, but since I don't do much x86 debugging it hasn't been as thoroughly tested as the x64 build.  Let me know if you run into any issues.

    This is actually a later build than the x86 version and has some enhanced features, like !findtimers to dump all the active timer objects in the process and enhanced output for some commands.  I'll get the x64 version update posted soon as well.

    Download it here

    SPT - A WinDBG extension for debugging .NET applications

    In my spare time I've been working on a WinDBG extension containing some useful methods that automate pretty common .NET debugging tasks for me, and hopefully other people as well.  The extension right now targets .NET 4/x64 only, as that's my primary development platform.  I'll probably make a x86 build soon (especially if people request it), it shouldn't be very difficult to make it work on 32 bit platforms.  Currently a few commands support DML (!DumpSqlConnectionPools is the most noticeable), I'll be adding support for DML to other commands soon.

    A few of the commands are targeted at ASP.NET/server apps (!DumpHttpContext, !DumpASPNetRequests, !DumpSqlConnectionPools, !DumpThreadPool for example), but most are fairly general purpose.  In the next few blog posts I'll go into details about the implementation of each of the commands, but for now here's a list of them with quick summaries and examples.  Feel free to comment/email me with any comments/suggestions.

    Download the extension here (x64)

    And the x86 version here

     

    ASP.NET

    !DumpASPNetRequests

    Prints out all threads with a HTTP context.
    Example output:

    ThreadID  HttpContext       StartTime             URL + QS 
    0123      0000000002f62b38  05/29/2011 18:23:02   /hello/world.aspx  
    0124      0000000002f62c98  05/29/2011 18:23:05   /test.aspx?w00t=1
    ...
     

    !FindHttpContext

    Prints the context information for the current thread, including the managed thread object address (System.Threading.Thread), ExecutionContext, and Logical/IllogicalCallContext.
    Example output:

    OS ThreadId: e64, N: 0
    Managed Thread @ 0000000002f58580
    
    Thread ExecutionContext @ 0000000002f64af8
    IllogicalCallContext @ 0000000002f64c40
    HostContext (HttpContext) @ 0000000002f64d80

    Note: of all the commands here, this is the most likely to break from service releases.  It relies on reversed offsets to find the managed thread object from the unmanaged thread object.

    Hashtables/Dictionaries/Etc

    !DumpDictionary (!DumpHashtable, !ddct, !dht)

    Note: I used hashtable/dictionary pretty interchangeably for the rest of this post, anything that supports Hashtables also supports Dictionaries.

    Iterates over all the key/value pairs in a dictionary, printing them out, and translating keys/values to strings if possible.
    Example:

    0:000> !ddct 0000000002f621c0  
    entry at: 0000000002f622e8
    key: 0000000000000001 [(null), 1]
    value: 0000000002f62128 [System.String, "test"]
    hashCode: 0x00000001

     

    entry at: 0000000002f62300
    key: 0000000000000002 [(null), 2]
    value: 0000000002f62150 [System.String, "two"]
    hashCode: 0x00000002

    These methods have a few switches you can use.

    • The first is -short, when used prints only the key/value/hash tripplet for each entry, useful for feeding into a script.
    • The second is -kdo and -vdo.  These switches allow you to pass a tokenized command in for the key/value in each entry.  For instance, say you want to print out all the values in the dictionary, you could run

      !ddct -vdo "!do %p" 0000000002f621c0

    These switches are one of my favorite features and have saved me a ton of time already.

    Note: DumpDictionary and DumpHashtable are both actually the same function, the export table links them together.

    Note 2: This also supports the HybridDictionary class, just use any of the above functions on it.

     

    !DumpMemoryCache (!dmc)

    Basically the same as !DumpDictionary, but goes through all the sub-hashtables inside the memory cache object.  Doesn't support the same switches as !DumpDictionary yet though.

     

    Other misc stuff

    !FullStack (!fs)

    Prints out the mixed-mode callstack of the current thread.
    Example:  (parameters snipped by me so it doesn't wrap on my blog)

           IP        M  Name 
    
    000000007745153a 0 ntdll!ZwRequestWaitReplyPort+a
    0000000076d42f58 0 KERNEL32!ConsoleClientCallServer+54
    0000000076d75321 0 KERNEL32!ReadConsoleInternal+1f1
    0000000076d8a762 0 KERNEL32!ReadConsoleA+b2
    0000000076d58e04 0 KERNEL32!TlsGetValue+899e
    000007feef4f17c7 0 clr!DoNDirectCall__PatchGetThreadCall+7b
    000007feec6834a1 1 DomainNeutralILStubClass.IL_STUB_PInvoke(...)+101
    000007feece2f58a 1 System.IO.__ConsoleStream.ReadFileNative(...)+ba
    000007feece2f3f2 1 System.IO.__ConsoleStream.Read(...)+62

    (Note: I've seen some version of dbghelp have problems resolving native methods in CLR.dll to symbols.  The latest version seems to work fine though.)

     

    !FormatDT

    Given a System.DateTime or TimeSpan, formats it as a string MM/dd/yyyy hh:mi:ss.  I think psscor2 had something similar back in the day.

     

    !EvalExpression (!evalexpr)

    Given an expression consisting of field access or hashtable lookups, evaluates the expression starting at a root object.

    Fields are separated by periods just like C++/C#.  Hashtable lookups are specified by the [ ] operator (like in C#) and are done in two ways.

    • The first is if the hashtable/dictionary key is a string, you can use a quoted string inside the [ ] operator to tell the evaluator you want to match a string.  For example _myHashtable["steve"].lastName will lookup the value in _myHashtable with the key of "steve", then evaluate the lastName field. 
    • The second is by hashcode.  If the expression inside the [ ] operator is an unquoted number, it's used to do a lookup by hashcode.

    Also, if the root object itself is a hashtable, it's legal for the expression to begin with simply a [ ] operator.  For example, ["steve"].lastName.

    Expressions ending with a "$" sign are evaluated to a string and the result string printed.  If the final result isn't a string object, stuff will probably break.

    Note: Hashtable lookups are actually O(n) (you'd need a full-out IL interpreter like in VS to do it in O(1)), so they might get slow on really big dictionaries.

     

    !PrintDelegateMethod (!pdm)

    Attempts to resolve a delegate to a method name.  This should work at least for non-multicast delegates (eg delegates without an invocation list) to static or instance methods.  For instance method calls, the value in _methodPtr is simply a pointer to the managed method and resolved in a way similar to ip2md.  For static methods, the value in _methodPtrAux is an entry into the JIT stub table, and used to get a MethodDesc handle to the method, which is then resolved to a name.

     

    !DumpThreadPool

    Dumps the items currently in the .NET 4.0 ThreadPool queues, resolving work items to method names.  This currently supports work items enqueued via ThreadPool.QueueUserWorkItem and Task[<T>].  These are the only two classes in .NET 4 that currently implement IThreadPoolWorkItem.  This method only works if ThreadPoolGlobals.useNewWorkerQueue = true.

    There's a also special case: if a work item was created via BeginInvoke, DumpThreadPool will attempt to resolve it to the real method being called by looking at the MethodCall object's MethodInfo field.  For this to work, the method being called must be System.Runtime.Remoting.Proxies.AgileAsyncWorkerItem.ThreadPoolCallBack, and the callback argument must be a System.Runtime.Remoting.Proxies.AgileAsyncWorkerItem.

     

    !DumpSqlConnectionPools (!dsql)

    Dumps out all SQL connection pools in all app domains of the process.

    Example output:

    0:046>  !dumpsqlconnectionpools;
    [0] Factory @ 00000001df0614c8
    Connection String: <snip>
    PoolGroup:    000000011f0877a8
        SID:  S-1-5-21-3430578067-4192788304-1690859819-9294
        Pool: 00000001bf077d08, State: 1, Waiters: 0, Size: 12
        Connections:
                  ConnPtr           State  #Async  Created            Command/Reader    Timeout  Text
            [000] 00000001bf0788e8      1       0  6/10/2011 17:30:32
            [001] 000000015f1ebf80      1       0  6/10/2011 17:30:33
            [002] 000000013f30da08      1       0  6/10/2011 17:30:39
            [003] 000000013f3315e0      1       0  6/10/2011 17:30:54
            [004] 000000011f4852f8      1       0  6/10/2011 17:30:54
            [005] 00000001df1b5998      1       0  6/10/2011 17:30:54
            [006] 00000000ff41da90      1       0  6/10/2011 17:30:54
            [007] 00000001bf1c08f8      1       0  6/10/2011 17:30:54
            [008] 000000019f5f8080      1       0  6/10/2011 17:30:54
            [009] 000000015f2b4f90      1       0  6/10/2011 17:30:54
            [00a] 000000013f344120      1       0  6/10/2011 17:30:54
            [00b] 000000019f60ab18      1       0  6/10/2011 17:30:54

    Connection String: <snip>;
    PoolGroup:    000000017f125730
        SID:  S-1-5-21-3430578067-4192788304-1690859819-9294
        Pool: 000000017f4495b8, State: 1, Waiters: 0, Size: 4
        Connections:
                  ConnPtr           State  #Async  Created            Command/Reader    Timeout  Text
            [000] 000000017f449f50      1       0  6/10/2011 17:30:54
            [001] 000000017f471280      1       0  6/10/2011 17:30:54
            [002] 00000000ff4305b8      1       0  6/10/2011 17:30:54
            [003] 00000000ff442f50      1       1  6/10/2011 17:30:54  000000017f4372a0       50  do_something_prc

    What are we seeing here?  Connection pooling is basically a 5 level hierarchy:

    • Connection String - top level, defines the server and connection options.
      • Pool group - Second level, varies based on the identity used to connect to the DB if using SSPI
        • Pool - Actually contains the SqlConnection instances.
          • SqlConnection - The DB connection instance
            • Active readers - SqlDataReaders that are actively doing something

    Some fields that probably need some explaining:

    • Pool -> State
      • Corresponds to the System.Data.ProviderBase.DbConnectionPool+State enum, Initializing, Running, Shutting Down
    • Connection -> State
      • Corresponds to the System.Data.ConnectionState enum.
    • Created is the time the connection was first opened in UTC.
    • Command/Reader
      • If Async = 1, the pointer is to a command, otherwise it's to a SqlDataReader.  With a SqlCommand, you can get to the reader via the _cachedAsyncState field, with a SqlDataReader, you can get to the command via the _command field.

    Note: This doesn't currently dump connections in the transacted connection pools.

     

    !FindTimers

    (x86 only for now)

    Dumps out all timers registered in the process.

    Example:

    0:006> !findtimers
    Handle            TimerObj          ADID  Period[ms]  State             Context           Delegate          Method
              293e48  00000000022cbbb4     1        1000  00000000022cbb48  00000000022cbbc8  00000000022cbb6c  SomeCallback(System.Object)

    Columns are:

    • Handle
      • The timer handle, can be used to cross reference with an existing TimerBase object
    • TimerObj
      • A reference to the managed _TimerCallback object
    • ADID
      • The appdomain ID the timer is associated with
    • State
      • The state object associated with the constructor.
    • Context
      • The ExecutionContext captured by the timer.
    • Delegate
      • The delegate that will be called when the timer fires.

    !SPTHelp

    Prints out the built-in help.