Archive for July, 2011

Integrating with third party systems using CQRS + Event Sourcing – Part 1

I’m developing a mobile application for tracking bugs called PhoneBugz. Instead of re-inventing the wheel I decided I would integrate with as many third party bug tracking systems as I could. This of course is a massive undertaking but I’ve decided to pick a few fairly popular ones and then expand from there in future versions. A major design goal of my application is a fast reliable way of logging bugs and doing bug “triage” even while disconnected from the server. In order to achieve this goal I need to make sure I can defer the work of synchronizing the server to a single operation that will happen in the background while the user continues to work. This enables scenarios where a user is offsite logging bugs they found at a customer site, and then can connect to their corporate Wi-Fi to sync the bugs once they get back to the office and synchronize with the internal bug tracking system.

This poses some major design challenges:

  • How do I ensure that changes merge seamlessly in a disconnected scenario?
  • How do I make the code flexible enough that it will work with almost any bug tracking system, especially when they have different data models?

This is where CQRS + Event Sourcing can help, at least a little. If you haven’t read up about CQRS + Event Sourcing I suggest reading some great blogs by Greg Young, Udi Dahan et al. As they will give a much more comprehensive understanding of these concepts than I ever could.

In this and other upcoming blog articles I hope to outline some of the challenges I face as I design this system and how I’ve decided upon which solution to choose.

Challenge #1: A Problem of Identity

When a bug is created on a phone that is disconnected from the “Master” server, how do you identify that bug? How do other entities in the system refer to that bug? You can’t use the master’s identity because it hasn’t been assigned yet. So you need some sort of indirection, a temporary identity if you will. One interesting technique I saw in an article by Jeremie Chassaing used strongly typed Ids for identifying aggregate roots. While this on it’s own doesn’t completely solve my problem it’s a good first step to addressing it.

Solution #1 – Mutable Identity

With this technique we have Id’s as mutable reference types where the external system’s Id is swapped under the covers when synchronizing. All referrers must refer to a single object reference for each unique Id, this way when the mapping to the external id is made, then all parties have the new information immediately. This necessitates, however, a central repository for storing Id instances in a single location, which makes storing and loading them much more difficult for other Aggregate Roots in the system.

public class MutableBugId
{
    internal MutableBugId()
    {
        this.InternalBugId = Guid.NewGuid();
    }

    public Guid InternalBugId
    {
        get; 
        private set;
    }
        
    internal int ExternalBugId
    {
        get;
        private set;
    }

    internal void MapTo(int externalBugId)
    {
        this.ExternalBugId = externalBugId;
    }
}

Pros:

  • Centralized mapping and swapping code.
  • Very simple Id class implementation.
  • Can use simple reference equality comparison for comparing id’s

Cons:

  • Have to have a central lookup of Id instances, which if static would mean that the external ids cannot be scoped without some kind of scope id e.g. if BugId’s aren’t unique across projects in the third party system.
  • Serialization of the identity by a third party becomes difficult as references must be centralized, this means that all referring parties need code serialize and de-serialize and then lookup each Id.
  • No notification that the External Id has changed, so if a component persists that ExternalId and it changes underneath them, then bugs could occur.

Using MutableBugId

Here we’ve created a simple multi-keyed repository called MutableBugIdRepository that just maintains the instances which can be looked up either by internal or external Id. The real mental gymnastics comes from storing and loading the Id’s which I’ve not listed here. Actually It’s kind of a sticking point I’m struggling to resolve.

public class MutableBugSynchronizer
{
    private MutableBugIdRepository bugIds = new MutableBugIdRepository();

    public void SaveNewBug(MutableBug bug)
    {
        var newExternalBug = this.CreateUsingWebService(bug.Summary);
        bugIds.MapBugId(bug.Id, newExternalBug.Id);
    }

    private ExternalBug CreateUsingWebService(string summary)
    {
        // TODO: Call the web service for real and get 
        // back the services version of the bug. 
        return new ExternalBug()
        {
            Id = new Random().Next(1, 10000),
            Summary = summary
        };
    }
}

Solution #2 – Immutable Identity

With this technique we use a struct that is swapped out when synchronizing with the external system, publishing an event such as BugIdReplaced when the temporary Id is swapped out with the real one. To make the code clearer I have referred to my systems BugId’s as Internal BugId’s and the third party bug tracking service id’s as External BugId’s.

public struct ImmutableBugId : IEquatable<ImmutableBugId>
{
    public static readonly ImmutableBugId Empty = new ImmutableBugId(Guid.Empty);

    private Guid id;

    private int externalId;

    private ImmutableBugId(Guid id)
    {
        this.id = id;
        this.externalId = 0;
    }

    private ImmutableBugId(int externalId)
    {
        this.id = Guid.Empty;
        this.externalId = externalId;
    }

    public static ImmutableBugId NewInternalBugId()
    {
        return new ImmutableBugId(Guid.NewGuid());
    }

    public static ImmutableBugId GetExternalBugId(int updatedExternalBugId)
    {
        return new ImmutableBugId(updatedExternalBugId);
    }

    public bool Equals(ImmutableBugId other)
    {
        if (this.externalId != 0 && other.externalId != 0)
        {
            return other.externalId.Equals(this.externalId);
        }

        return other.id.Equals(this.id);
    }

    public override bool Equals(object obj)
    {
        if (ReferenceEquals(null, obj))
        {
            return false;
        }

        if (obj.GetType() != typeof(ImmutableBugId))
        {
            return false;
        }

        return this.Equals((ImmutableBugId)obj);
    }

    public override int GetHashCode()
    {
        return this.id.GetHashCode();
    }

    public static bool operator ==(ImmutableBugId a, ImmutableBugId b)
    {
        return a.Equals(b);
    }

    public static bool operator !=(ImmutableBugId a, ImmutableBugId b)
    {
        return !a.Equals(b);
    }
}

public class ImmutableBugIdReplaced : IDomainEvent
{
    public ImmutableBugId OldBugId { get; set; }

    public ImmutableBugId NewBugId { get; set; }
}

public class ReplaceBugIdCommand
{
    public ImmutableBugId NewBugId { get; set; }
}

Pros:

  • Clean serialization code by third parties as references instances no longer matter, as long as the current id is persisted the code will still work.
  • Id mapping code is just treated as a standard event describing a property change, that subscribers have to handle (no special plumbing code in the framework specific to mapping and replacing Ids)
  • Relying entities can decide how to handle the transition or choose to ignore the event if they don’t rely on BugIds (such as statistics report queries etc.).
  • Enables peer-to-peer synchronization without connecting to the central server as the internal Id’s remain persisted across tiers and the point at which the transition to the new id occurs is listed in the event stream.

Cons:

  • Every object that references that Id is concerned with the replacement of the Id when it’s mapped to the external system.
  • Subtle bugs could occur due to some id’s not being updated by a relying party, this includes their serialized events which would also reference those Id’s
  • Much more complicated Id types which have to support equality comparison of id’s (rather than simply using reference equality) and conversion to and from external id’s.

Using ImmutableBugId

To put it all together I’ve created a simple synchronizer class that would get called when we sync with the server. This obviously is overly simplified and doesn’t used the replayed events related to bug creation which we haven’t covered yet, but you get the idea. We perform our update on the server and then call a command to replace it’s id just like we would if we wanted to modify the Bug’s Summary field etc. The Bug (our aggregate root) then is responsible for applying this change to itself, and for publishing a BugIdReplaced event to all subscribers.

public class ImmutableBugSynchronizer
{
    public void SaveNewBug(ImmutableBug bug)
    {
        var newExternalBug = this.CreateUsingWebService(bug.Summary);
        var mappedId = ImmutableBugId.GetExternalBugId(newExternalBug.Id);
        bug.ReplaceBugId(new ReplaceBugIdCommand() { NewBugId = mappedId });
    }

    private ExternalBug CreateUsingWebService(string summary)
    {
        // TODO: Call the web service for real and get 
        // back the services version of the bug. 
        return new ExternalBug()
            {
                Id = new Random().Next(1, 10000), 
                Summary = summary
            };
    }
}

Conclusion

At the moment I am leaning towards using the immutable technique as the code does seem very clean and fits well with the Tell don’t Ask approach to object design. This design also appears to require less specialized work despite the fact that all event handling classes that refer to the BugId must now handle one more event.

Advertisements

Leave a comment