Samurai Programmer.com

In my previous post, I added to my series of entries on making sense of your ASP.NET event log error messages.  Note that this is entry #4 in this series.  The previous three entries can be found here:

• Part 1:  Parsing ASP.NET event log error messages for fun and profit
• Part 2:  Don’t guess when it comes to performance…
• Part 3:  Pivoting your ASP.NET event log error messages

In that last post, I walked through the PAuthorLib.dll and showed you how to crawl through your event log error messages and create a pivot collection.  The result of that initial effort was a nice view into our events:

While this certainly got the job done and is a very powerful and compelling view into our data, we need to realize that as our data grow, the amount of entries in our linked collection is limited.  From the Developer Overview, we see that the maximum number of items we should have in a single collection is 3,000:

So, while a simple collection will get the job done for your smaller amounts of data, you will really run into some challenges with your larger datasets like our ASP.NET event log error messages.  To combat this limitation you can create what’s called a Linked Collection.  The idea is that it’s just a way for you to link together related collections in order to provide a seamless experience for your users.  In our case, a natural break for our collections will be based upon the exception type with a summary collection and then a separate collection for each exception type.  If I were to draw this out: 

Event Log Header Collection Source

The idea behind this structure is that the Exception summary would simply link to each of these exception collections.  First, we’ll create a colleciton source for our exception summary.  As in our previous collection source (in my last blog post), we inherit from the AbstractCollectionSource class and use the LoadHeaderData() method to add our facet categories to the collection.  In this case, we’ll create two categories – the number of Occurrences and the Urls where the exception occurred.  Another difference is that we are going to pass the already parsed collection of messages into the constructor.  The reason for that is so we don’t have to repeat the parsing of the event log messages multiple times.

class EventLogHeaderCollectionSource : AbstractCollectionSource
{

    private IEnumerable<EventLogMessage> m_messages = null;

    public EventLogHeaderCollectionSource(IEnumerable<EventLogMessage> messages, 
                                          string inputFile)
        : base(inputFile)
    {

        m_messages = messages;


    }

    #region Facets

    private const string OCCURRENCES = "Occurrences";
    private const string URLS = "Urls";

    #endregion

    protected override void LoadHeaderData()
    {
        this.CachedCollectionData.FacetCategories.
                        Add(new PivotFacetCategory(OCCURRENCES, PivotFacetType.Number));
        this.CachedCollectionData.FacetCategories.
                        Add(new PivotFacetCategory(URLS, PivotFacetType.String));

        this.CachedCollectionData.Name = 
                        "ASP.NET Error Messages - Summary";
        this.CachedCollectionData.Copyright = 
                        new PivotLink("Source", "http://www.samuraiprogrammer.com");

    }
}

Then, in the LoadItems() method, we provide the logic to generate the PivotItem collection.  The one key item to make note of is the use of the Href property of the PivotItem object.  This is where we specify the collection we wish to link to this item.  Since each of the PivotItems will be a summary of the number of each exception type – we’ll name the sub-collections by its exception type.  For example, NullReferenceException.cxml, SqlException.cxml, etc.

protected override IEnumerable<PivotItem> LoadItems()
{
    var results = from log in m_messages
                  group log by log.Exceptiontype into l
                  orderby l.Count() descending, l.Key
                  select new
                  {
                      ExceptionType = l.Key,
                      ExceptionCount = l.Count()
                  };


    int index = 0;
    foreach (var result in results)
    {
        PivotItem item = new PivotItem(index.ToString(), this);       
        item.Name = result.ExceptionType;
        item.Description = "# of Exceptions: " + result.ExceptionCount.ToString();
        item.AddFacetValues(OCCURRENCES, result.ExceptionCount);
        item.Href = result.ExceptionType + ".cxml";

        ...                

        index++;
        yield return item;
    }

    yield break;
}

Event Log Collection Source Redux

Previously, when we generated the pivot collections, we were outputting all of the records into a single collection.  Now that we are generating a collection for each exception type, we will need to put a filter in our exception collection and then incorporate that filter into our item generation.  Other than that, the code we wrote previously remains largely unchanged, so I left the majority of it out and only included the snippets that we care about below.

class EventLogCollectionSource : AbstractCollectionSource
{
    private IEnumerable<EventLogMessage> m_messages = null;
    private string m_exceptionType = string.Empty;

    public EventLogCollectionSource(
                    IEnumerable<EventLogMessage> messages, 
                    string exceptionType, 
                    string path)
        : base(path)
    {
        m_messages = messages;
        m_exceptionType = exceptionType;
    }

    protected override void LoadHeaderData()
    {
        ...
        this.CachedCollectionData.Name = 
                string.Format("{0} Error Messages", m_exceptionType);
        ...
    }

    protected override IEnumerable<PivotItem> LoadItems()
    {
        var results = (from message in m_messages
                       where message.Exceptiontype == m_exceptionType
                       select message);

        int index = 0;
        foreach (EventLogMessage message in results)
        {
            PivotItem item = 
                    new PivotItem(index.ToString(), this);
            item.Name = message.Exceptiontype;
            item.Description = message.Exceptionmessage;

            ...

            index++;
            yield return item;
        }
        yield break;
    }
}

Generate and test the collection

Then, the only thing we have left to do is generate and test our linked collections.  I won’t go into a lengthy explanation of how we generate the collections because I did that in the last blog entry.  I will show the broad strokes required to tie this all together, though:

// Load the raw messages into a collection
IEnumerable<EventLogMessage> messages = 
                LoadEventLogMessages(inputFile).ToList();

// Generate summary pivot collection
EventLogHeaderCollectionSource sourceSummary = 
                new EventLogHeaderCollectionSource(messages, inputFile);
...
summaryTargetFilter1.Write(sourceSummaryFilter1);

// Get the aggregate results so we know the filters
// for our exception pivot collections
var summaryResults = from log in messages
              group log by log.Exceptiontype into l
              orderby l.Count() descending, l.Key
              select new
              {
                  ExceptionType = l.Key,
                  ExceptionCount = l.Count()
              };

foreach (var resultItem in summaryResults)
{
    // Generate pivots for each exception type
    EventLogCollectionSource source = 
            new EventLogCollectionSource(messages, 
                                         resultItem.ExceptionType, 
                                        inputFile);
    ...
    targetFilter1.Write(sourceFilter1);
}

Once we we have this code and everything has been generated, if we open the output folder, we’ll see the following structure:

We see our ExceptionSummary pivot collection and all of the deep zoom folders.  So, when we open the Pivot tool, we’ll see a nice parent collection:

This gives us a nice breakdown of the number of occurrences for each exception in our source data.  Immediately we see an outlier (more on that later) between the 6,000 and 7,000 item mark and when we select that tile, we see the following:

We also see a green “Open” box (surrounded in a red rectangle, my emphasis) which links to our NullReferenceException.cxml.  When we click that box, the tool will immediately open that collection in the UI for our perusal – providing a very similar look to what we saw in the last entry:

Closing Thoughts

Now, you may have noticed a contradiction above.  I said that a collection should have no more than 3,000 items and yet, with the NullReferenceException collection, we saw in the summary that it had over 6,000 items.  That is a very good point and will be a subject of a future blog post.  I wanted to illustrate the simple collections and the linked collections before we got into that third type of collection from above – the Dynamic Collection.  Stay tuned!