Integrating with third party systems using CQRS + Event Sourcing – Part 1

I’m developing a mobile application for tracking bugs called PhoneBugz. Instead of re-inventing the wheel I decided I would integrate with as many third party bug tracking systems as I could. This of course is a massive undertaking but I’ve decided to pick a few fairly popular ones and then expand from there in future versions. A major design goal of my application is a fast reliable way of logging bugs and doing bug “triage” even while disconnected from the server. In order to achieve this goal I need to make sure I can defer the work of synchronizing the server to a single operation that will happen in the background while the user continues to work. This enables scenarios where a user is offsite logging bugs they found at a customer site, and then can connect to their corporate Wi-Fi to sync the bugs once they get back to the office and synchronize with the internal bug tracking system.

This poses some major design challenges:

  • How do I ensure that changes merge seamlessly in a disconnected scenario?
  • How do I make the code flexible enough that it will work with almost any bug tracking system, especially when they have different data models?

This is where CQRS + Event Sourcing can help, at least a little. If you haven’t read up about CQRS + Event Sourcing I suggest reading some great blogs by Greg Young, Udi Dahan et al. As they will give a much more comprehensive understanding of these concepts than I ever could.

In this and other upcoming blog articles I hope to outline some of the challenges I face as I design this system and how I’ve decided upon which solution to choose.

Challenge #1: A Problem of Identity

When a bug is created on a phone that is disconnected from the “Master” server, how do you identify that bug? How do other entities in the system refer to that bug? You can’t use the master’s identity because it hasn’t been assigned yet. So you need some sort of indirection, a temporary identity if you will. One interesting technique I saw in an article by Jeremie Chassaing used strongly typed Ids for identifying aggregate roots. While this on it’s own doesn’t completely solve my problem it’s a good first step to addressing it.

Solution #1 – Mutable Identity

With this technique we have Id’s as mutable reference types where the external system’s Id is swapped under the covers when synchronizing. All referrers must refer to a single object reference for each unique Id, this way when the mapping to the external id is made, then all parties have the new information immediately. This necessitates, however, a central repository for storing Id instances in a single location, which makes storing and loading them much more difficult for other Aggregate Roots in the system.

public class MutableBugId
{
    internal MutableBugId()
    {
        this.InternalBugId = Guid.NewGuid();
    }

    public Guid InternalBugId
    {
        get; 
        private set;
    }
        
    internal int ExternalBugId
    {
        get;
        private set;
    }

    internal void MapTo(int externalBugId)
    {
        this.ExternalBugId = externalBugId;
    }
}

Pros:

  • Centralized mapping and swapping code.
  • Very simple Id class implementation.
  • Can use simple reference equality comparison for comparing id’s

Cons:

  • Have to have a central lookup of Id instances, which if static would mean that the external ids cannot be scoped without some kind of scope id e.g. if BugId’s aren’t unique across projects in the third party system.
  • Serialization of the identity by a third party becomes difficult as references must be centralized, this means that all referring parties need code serialize and de-serialize and then lookup each Id.
  • No notification that the External Id has changed, so if a component persists that ExternalId and it changes underneath them, then bugs could occur.

Using MutableBugId

Here we’ve created a simple multi-keyed repository called MutableBugIdRepository that just maintains the instances which can be looked up either by internal or external Id. The real mental gymnastics comes from storing and loading the Id’s which I’ve not listed here. Actually It’s kind of a sticking point I’m struggling to resolve.

public class MutableBugSynchronizer
{
    private MutableBugIdRepository bugIds = new MutableBugIdRepository();

    public void SaveNewBug(MutableBug bug)
    {
        var newExternalBug = this.CreateUsingWebService(bug.Summary);
        bugIds.MapBugId(bug.Id, newExternalBug.Id);
    }

    private ExternalBug CreateUsingWebService(string summary)
    {
        // TODO: Call the web service for real and get 
        // back the services version of the bug. 
        return new ExternalBug()
        {
            Id = new Random().Next(1, 10000),
            Summary = summary
        };
    }
}

Solution #2 – Immutable Identity

With this technique we use a struct that is swapped out when synchronizing with the external system, publishing an event such as BugIdReplaced when the temporary Id is swapped out with the real one. To make the code clearer I have referred to my systems BugId’s as Internal BugId’s and the third party bug tracking service id’s as External BugId’s.

public struct ImmutableBugId : IEquatable<ImmutableBugId>
{
    public static readonly ImmutableBugId Empty = new ImmutableBugId(Guid.Empty);

    private Guid id;

    private int externalId;

    private ImmutableBugId(Guid id)
    {
        this.id = id;
        this.externalId = 0;
    }

    private ImmutableBugId(int externalId)
    {
        this.id = Guid.Empty;
        this.externalId = externalId;
    }

    public static ImmutableBugId NewInternalBugId()
    {
        return new ImmutableBugId(Guid.NewGuid());
    }

    public static ImmutableBugId GetExternalBugId(int updatedExternalBugId)
    {
        return new ImmutableBugId(updatedExternalBugId);
    }

    public bool Equals(ImmutableBugId other)
    {
        if (this.externalId != 0 && other.externalId != 0)
        {
            return other.externalId.Equals(this.externalId);
        }

        return other.id.Equals(this.id);
    }

    public override bool Equals(object obj)
    {
        if (ReferenceEquals(null, obj))
        {
            return false;
        }

        if (obj.GetType() != typeof(ImmutableBugId))
        {
            return false;
        }

        return this.Equals((ImmutableBugId)obj);
    }

    public override int GetHashCode()
    {
        return this.id.GetHashCode();
    }

    public static bool operator ==(ImmutableBugId a, ImmutableBugId b)
    {
        return a.Equals(b);
    }

    public static bool operator !=(ImmutableBugId a, ImmutableBugId b)
    {
        return !a.Equals(b);
    }
}

public class ImmutableBugIdReplaced : IDomainEvent
{
    public ImmutableBugId OldBugId { get; set; }

    public ImmutableBugId NewBugId { get; set; }
}

public class ReplaceBugIdCommand
{
    public ImmutableBugId NewBugId { get; set; }
}

Pros:

  • Clean serialization code by third parties as references instances no longer matter, as long as the current id is persisted the code will still work.
  • Id mapping code is just treated as a standard event describing a property change, that subscribers have to handle (no special plumbing code in the framework specific to mapping and replacing Ids)
  • Relying entities can decide how to handle the transition or choose to ignore the event if they don’t rely on BugIds (such as statistics report queries etc.).
  • Enables peer-to-peer synchronization without connecting to the central server as the internal Id’s remain persisted across tiers and the point at which the transition to the new id occurs is listed in the event stream.

Cons:

  • Every object that references that Id is concerned with the replacement of the Id when it’s mapped to the external system.
  • Subtle bugs could occur due to some id’s not being updated by a relying party, this includes their serialized events which would also reference those Id’s
  • Much more complicated Id types which have to support equality comparison of id’s (rather than simply using reference equality) and conversion to and from external id’s.

Using ImmutableBugId

To put it all together I’ve created a simple synchronizer class that would get called when we sync with the server. This obviously is overly simplified and doesn’t used the replayed events related to bug creation which we haven’t covered yet, but you get the idea. We perform our update on the server and then call a command to replace it’s id just like we would if we wanted to modify the Bug’s Summary field etc. The Bug (our aggregate root) then is responsible for applying this change to itself, and for publishing a BugIdReplaced event to all subscribers.

public class ImmutableBugSynchronizer
{
    public void SaveNewBug(ImmutableBug bug)
    {
        var newExternalBug = this.CreateUsingWebService(bug.Summary);
        var mappedId = ImmutableBugId.GetExternalBugId(newExternalBug.Id);
        bug.ReplaceBugId(new ReplaceBugIdCommand() { NewBugId = mappedId });
    }

    private ExternalBug CreateUsingWebService(string summary)
    {
        // TODO: Call the web service for real and get 
        // back the services version of the bug. 
        return new ExternalBug()
            {
                Id = new Random().Next(1, 10000), 
                Summary = summary
            };
    }
}

Conclusion

At the moment I am leaning towards using the immutable technique as the code does seem very clean and fits well with the Tell don’t Ask approach to object design. This design also appears to require less specialized work despite the fact that all event handling classes that refer to the BugId must now handle one more event.

Leave a comment

Fixing the Microphone Audio cut-off issue in Windows Phone 7 Silverlight.

Thanks to some samples online it’s relatively straightforward to access the Microphone from a Silverlight application on Windows Phone 7. However when using these samples I noticed that sometimes the last half second of audio wasn’t

Getting Started

  • First add a reference to Microsoft.Xna.Framework to your Silverlight application. Words cannot describe how much this rubs me the wrong way, why it’s not part of the System namespaces or a separate shared Microsoft assembly is beyond me.
  • Add the dispatcher timer for XNA as described in this blog post.
  • Add two buttons to a page (Start and Stop). Call the StartRecording and StopRecording methods in the appropriate click event handlers.
 private bool isRecording;
 private MemoryStream ms;
 private readonly  Microphone mic = Microphone.Default;
 private byte[] buffer;
 
 private void StartRecording()
 {
     int bufferSize = mic.GetSampleSizeInBytes(mic.BufferDuration);
     this.buffer = new byte[bufferSize];
     mic.BufferReady += this.MicrophoneBufferReady;
     SoundEffect.MasterVolume = 1.0f;
 
     if (ms != null)
     {
         ms.Close();
     }
 
     ms = new MemoryStream();
     mic.Start();
     this.isRecording = true;
 }
 
 private void MicrophoneBufferReady(object sender, EventArgs e)
 {
     int bytesRead = 0;
     while ((bytesRead = mic.GetData(buffer, 0, buffer.Length)) > 0)
     {
         ms.Write(buffer, 0, bytesRead);
     }
 
     if (!this.isRecording)
     {
         ms.Flush();
         mic.Stop();
     }
 }
 
 private void StopRecording()
 {
     if (mic.State != MicrophoneState.Stopped)
     {
         // Calling stop here would stop receiving before  
         // the last buffer was complete. 
         //// mic.Stop(); 
         this.isRecording = false ;
     }
 }
 

Fixing the Audio cut-off issue.

The problem is that as buffers are being pushed to your application, the user pushes the stop button. Calling mic.Stop() immediately causes the buffers to stop being sent to your application and the last buffer that hadn’t completed is lost (leaving the audio chopped off).

The easiest fix for this is to have the Stop button request that the next buffer be the last to be recorded simply by setting a flag this.isRecording = false; and the event handler checks it after processing the last buffer. Viola! Problem solved.

Update: Added a link to blog regarding XNA dispatcher.

2 Comments

ToList() or not ToList(), that is the query.

A lot has been written about the nature of deferred execution with regards to LINQ queries, so I won’t go into it in detail here, suffice to say it’s nice when you can retain this behaviour as it usually makes your code more efficient. It can however come at a cost and here’s an example of why:

var query = from customer in db.Customers 
            where customer.City == "Paris" 
            select customer;
if (!query.Any()) << Query Executes Here
{
    Console.WriteLine("No results found");
    return;
}

foreach (var customer in query)  << And Query Executes Again Here
{
    Console.WriteLine(customer.CompanyName);
}

You have to make a choice here, read all results of query into a list up front with a call to ToList(), which could take a while before it starts writing out to the console (due to network IO or whatever), or you can execute the query twice. (as seen above)

Why not have the best of both worlds?

What we really want is an alternative to ToList() which still defers the execution and reads from the query as it iterates, but persists the results in subsequent iterations so that the heavy lifting only happens once (just like ToList()).

To achieve this I implemented an IEnumerator<T> called BufferedEnumerator<T> which wraps an inner enumerator and buffers the results as calls to MoveNext() are made against it, until the inner enumerator has been fully buffered and we can dispose it. To provide the BufferedEnumerator<T> I implemented an enumerable called BufferedEnumerable<T>.

One thing to note is that we don’t dispose the inner enumerator because we might not be finished with it. This raises an issue, if we don’t Dispose it when the Dispose() method is called, then when do we Dispose? The way I chose to get around this was to make the BufferedEnumerable<T> disposable and have it call the inner dispose, this means to be clean and tidy you should really wrap the buffered result in a using statement or call Dispose on it when you’re done (to ensure any underlying query is not left open etc).

Using the new Buffered() extension method

var query = from customer in db.Customers 
            where customer.City == "Paris" 
            select customer;

using (var buffer = query.Buffered())
{
    if (!buffer.Any()) << Query starts executing here
    {
        Console.WriteLine("No results found");
        return;
    }
    foreach (var customer in buffer) << Same query continues executing here
    {
        Console.WriteLine(customer.CompanyName);
    }
}

The good news is this can now be used as a 100% replacement to ToList(), you get none of the negative side effects of ToList() executing when it may not be necessary or reading all of the results when only the first result will do. Just remember that it is truly deferred, if you call query.Buffered() and never iterate over it, the query will never be executed.

Enjoy!

Download Source

Leave a comment

Delivering on the Natural User Interface Promise

The Problem with Natural User Interface Paradigm

A fair bit of talk is bubbling about this new term Natural User Interface or NUI for short. I’ve been scouring the net about this topic because I truly believe it is the future of human/computer interaction, and there are some excellent resources out there on the subject. A large part of Natural User Interfaces is multi-touch, but one thing about it has been really pissing me off: Pinch to Zoom.

Almost every time I see a demonstration of a “Natural User Interface” usually it involves multi-touch, and that inevitably reduces to the Pinch to Zoom manipulation or some variation, where by the user puts two fingers on a screen and scatters some photos around, or zooms in on a map.

Is this the best we as developers can come up with? I mean come on, this is no better than the mouse, how is raising my arm to pinch on the screen more productive than flicking my middle finger on a mouse wheel. To quote the legend Bill Buxton “something is always good for one thing, but worst for something else”. Multi-touch is crap for zooming, sorry, but there I said it, a mouse with a wheel wins for that and you probably won’t beat it. If you want to rotate a picture at the same time then fine (clicking a mouse wheel down seems to work pretty well too, but perhaps is less intuitive).

You see the temptation with the term Natural User Interface is for designers and developers to shy away from creating interactions that we has humans are unfamiliar with in our daily lives, and I don’t think that’s right. Certainly we want our applications to be easy to use, but I want multi-touch applications that are productive,, with interactions that increase the bandwidth between me and the computer, and I think I know why I haven’t seen any examples of it yet.

Most developers build using the metaphors they are familiar with, most developers don’t have the lease of time and resources to innovate and fail, they have to deliver something that’s proven to work and proven to be useful. Multi-touch will never become as ubiquitously used as the mouse and keyboard unless someone has pioneered great interactions, that have proven to be productive, and someone has made it easy for developers to incorporate it into their products. Right now there just aren’t the tools to quickly and easily make a multi-touch application beyond a scatter photo viewer.

Delivering on the NUI Promise

I’m on a mission to remedy this issue, and I don’t know if I’ll be able to do it on my own, but my plan is to develop a single application that does two things:

  1. Deliver a natural user interface that inspires developers to embrace NUI not just because it’s cool, but because it’s useful!
  2. Deliver a tool that helps developers build their own NUI applications and innovate in this space (because what I come up with may turn out to suck and that’s ok).

I’m calling this application Rush and it will be a WPF / Silverlight design and development tool (because it’s what I know). In an upcoming post I hope to journal my plans for this application, but if I’m going to deliver on this grand plan I’ll need people urging me on, so please post comments, I will need all the encouragement I can get!

Leave a comment

Quick way of checking if you’ve localised your Silverlight or WPF Xaml files

I typically use data binding to perform localisation of labels but I typically start out just by having plain text and go back later and replace these with bindings at a later stage.

e.g. <TextBlock Text=”{Binding Strings.Label_ContactDetails}” />

Here’s a really quick and easy way to make sure you’ve got your Xaml files completely localised.

1. Make sure you have downloaded LINQPad and installed it.

2. Start LINQPad and change the Language dropdown to C# Statement(s)

3. Paste this query into LINQPad and change the path to point to your project or solution folder

Code Snippet
  1. string path = @"C:\MyProjectFolder";
  2. XNamespace ns = "http://schemas.microsoft.com/winfx/2006/xaml/presentation&quot;;
  3.  
  4. var attributes = new List<XName> { "Text", "Tag", "Content", "Header", "ToolTipService.ToolTip" };
  5.  
  6. var files = System.IO.Directory.GetFiles(path, "*.xaml", System.IO.SearchOption.AllDirectories);
  7. var q = from eachFile in files
  8.         let fileContent = XElement.Load(eachFile)
  9.         from eachNode in fileContent.Descendants()
  10.         from eachAttribute in eachNode.Attributes()
  11.         where attributes.Contains(eachAttribute.Name)
  12.         where !eachAttribute.Value.StartsWith("{")
  13.         select new
  14.         {
  15.             Text = eachAttribute.Value,
  16.             Node = eachNode.Name.LocalName,
  17.             Attribute = eachAttribute.Name,
  18.             FileName = eachFile.Substring(path.Length)
  19.         };
  20.  
  21. q.Dump();

 

4. Click Run and viola you have a list of all the nodes in all Xaml files that need localising!

Hope this helps!

Leave a comment

Spherical Glass Buttons in WPF and Silverlight

I long while back I post an article about Making a Glass Ball act like a Glass Button using Expression Blend and WPF. Well a reader asked me if I could share the code for these glass buttons so I’ve added a resource dictionary to my SkyDrive with the following Button templates:

image

Unfortunately this version is out of date, and I seem to have lost the original code which was more functionally complete, as you can see disabled buttons don’t fade, they used to turn to transparent glass buttons. Looking back on this my main criticism would be that the colour of the button is difficult to change, I’d have preferred if the background could be adjusted just by changing the BackColor property on a button but the gradients are just too complex to achieve this at the moment.

Hope this helps someone!

http://cid-17124d03a9a052b0.skydrive.live.com/embedicon.aspx/Public/GlassButtonResources.xaml

1 Comment

Outputting Visual Studio Logs from Silverlight Unit Test Framework

After much tearing of my hair out trying to understand how the Silverlight Unit Testing framework can output Visual Studio Test Result files (TestResults.trx), I have come up with the simplest solution I could think of, that I’m sure anyone can follow:

Setting up the Silverlight Unit Test Project

  1. Create a new Silverlight Application project in Visual Studio and add the following references:
    • Microsoft.Silverlight.Testing
    • Microsoft.VisualStudio.QualityTools.UnitTesting.Silverlight
  2. Add the following code to the Application_Startup function in App.xaml.cs:
  3. Code Snippet
    1. private void Application_Startup(object sender, StartupEventArgs e)
    2. {
    3.     var settings = UnitTestSystem.CreateDefaultSettings();
    4.     settings.TestService.UniqueTestRunIdentifier = Guid.NewGuid().ToString();
    5.     this.RootVisual = UnitTestSystem.CreateTestPage(settings);
    6. }

The Silverlight test project is now ready to have test classes added to it.

Setting up the Log Handler Web Service

The Silverlight’s Unit Testing Framework test runner has an in-built VisualStudioLogProvider that is automatically configured to output to http://localhost:8000/externalInterfaces

To support accepting the log content, you’ll need to create a special web site or Http Service to listen on this port, at this address. Stupidly this not configurable (yet), of course you could possibly implement your own logging service (and if you do I wish you luck).

The easiest way I could think of accepting the information and providing the correct responses for the TestHarness was to create an ASP.NET MVC web site.

  1. Create a new C# ASP.NET MVC web application
  2. Right click the ASP.NET MVC project, select the Web tab and change the development server to host on a specific port: 8000

    image

  3. Add a file called clientaccesspolicy.xml to the root of the web site and add the following xml to it:
  4. Code Snippet
    1. <?xml version="1.0" encoding="utf-8" ?>
    2. <access-policy>
    3.     <cross-domain-access>
    4.         <policy>
    5.             <allow-from http-request-headers="*">
    6.                 <domain uri="*"/>
    7.             </allow-from>
    8.             <grant-to>
    9.                 <resource path="/"
    10.                           include-subpaths="true"/>
    11.             </grant-to>
    12.         </policy>
    13.     </cross-domain-access>
    14. </access-policy>

  5. Add the following code to the RegisterRoutes function in the Global.asax.cs file:
  6. Code Snippet
    1. public static void RegisterRoutes(RouteCollection routes)
    2. {
    3.     routes.IgnoreRoute("{resource}.axd/{*pathInfo}");
    4.  
    5.     routes.MapRoute(
    6.         "Default",                              // Route name
    7.         "{controller}/{action}/{*pathInfo}",    // URL with parameters
    8.         new
    9.         {
    10.             controller = "Home",
    11.             action = "Index",
    12.             pathInfo = ""
    13.         }  // Parameter defaults
    14.     );
    15. }

  7. Create a new controller under the \Controllers folder called ExternalInterfacesController.cs
  8. Add the following code to the controller:
  9. Code Snippet
    1. public class ExternalInterfaceController : Controller
    2. {
    3.     [AcceptVerbs(HttpVerbs.Get)]
    4.     public ActionResult Ping()
    5.     {
    6.         return View();
    7.     }
    8.  
    9.     [AcceptVerbs(HttpVerbs.Get)]
    10.     public ActionResult GetRunParameters(string pathInfo)
    11.     {
    12.         return View();
    13.     }
    14.  
    15.     [AcceptVerbs(HttpVerbs.Get)]
    16.     public ActionResult ReportTestResults(string pathInfo)
    17.     {
    18.         return View();
    19.     }
    20.  
    21.     [ValidateInput(false)]
    22.     [AcceptVerbs(HttpVerbs.Get | HttpVerbs.Post)]
    23.     public ActionResult SaveLogFile(string pathInfo)
    24.     {
    25.         var fileName = pathInfo.Split(‘/’).SkipWhile(part => !part.StartsWith("logName")).Skip(1).FirstOrDefault();
    26.         if (!string.IsNullOrEmpty(fileName))
    27.         {
    28.             this.Request.SaveAs(Server.MapPath(Path.Combine(@"\App_Data", fileName)), false);
    29.         }
    30.  
    31.         return View();
    32.     }
    33.  
    34.     protected override void Execute(System.Web.Routing.RequestContext requestContext)
    35.     {
    36.         // HACK: To overcome Http Form Validation (ValidateInput attribute doesn’t seem to work).
    37.         string pathInfo = requestContext.HttpContext.Request.Url.PathAndQuery;
    38.         if (pathInfo.StartsWith("/externalInterface/saveLogFile/", StringComparison.OrdinalIgnoreCase))
    39.         {
    40.             var fileName = pathInfo.Split(‘/’).SkipWhile(part => !part.StartsWith("logName")).Skip(1).FirstOrDefault();
    41.             if (!string.IsNullOrEmpty(fileName))
    42.             {
    43.                 requestContext.HttpContext.Request.SaveAs(requestContext.HttpContext.Server.MapPath(Path.Combine(@"\App_Data", fileName)), false);
    44.             }
    45.         }
    46.         else
    47.         {
    48.             base.Execute(requestContext);
    49.         }
    50.     }
    51. }

  10. Add the following Views to the Views folder (none of them should use a master page):
    • Ping.aspx
    • GetRunParameters.aspx
    • ReportTestResults.aspx
    • SaveLogFile.aspx
  11. Then replace contents of each of these views except GetRunParameters.aspx.

    Code Snippet
    1. <%@ Page Title="" Language="C#" Inherits="System.Web.Mvc.ViewPage" %><rsp stat="ok"/>

  12. Add the following content to GetRunParameters.aspx which tells the Silverlight Test Run the name of the computer that is running the tests.
  13. Code Snippet
    1. <%@ Page Language="C#" Inherits="System.Web.Mvc.ViewPage" %><rsp stat="ok">
    2. <option name="ComputerName" value="<%= Environment.MachineName %>" />
    3. </rsp>

 

That’s it! Now your Silverlight Test Results will be output the the \App_Data\TestResults.trx file on the web server, the file will be replaced each time you run the tests. If you want to change the filename to something unique each time, change the SaveLogFile function accordingly.

Hope this helps!

Leave a comment

Follow

Get every new post delivered to your Inbox.