Cross Language Application Messaging with RabbitMQ and Protocol Buffers

There are many cases when an old application becomes very large and things rev up that no one could have predicted early on in the design. The business then grows, and we need a way to scale, work with newer technology or find a way to make the output of one application work with another. That other application may belong to a different business unit or could be written in a different language.

Let's put together a business case of what we're trying to solve:

  1. The application Alpha needs to notify other micro services/applications about some data
  2. Application Alpha and application Beta are written in different languages
  3. Application Beta is listening for specific messages
  4. The applications are on the Intranet

As you may have guessed, RabbitMQ is the message broker that is used to notify the other services and to scale out. Google's protocol buffer will be used to deal with cross language issues. A quick quote from their site on what it is sums it up nicely:

Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

Overview Of RabbitMQ and Protocol Buffers#

RabbitMQ and Protocol Buffer Overview

  1. We define our message class as an actual class with some simple properties. These properties should conform to the protocol buffer "language".
  2. Each language has its own implementation of protocol buffers and will be able to serialize and deserialize binary messages that conform. An important bit to note is that the message is serialized as binary to decrease size and increase efficiency. It's an Intranet application, there is no need to have a large JSON message.
  3. The messages are serialized by a publisher and then passed onto RabbitMQ in whatever topology you choose to use and then read by the consumers.

For each language the corresponding class file is made and as a nice bonus with a strongly typed language, you get strongly typed classes and all the wonders of intellisense.

Protobuf File##

In step one we first define the message class in a protocol buffer markup. There is a compiler for protobuf that will take this markup and generate Java/c++/python class files to use. Other languages have implemented protocol buffers as well so we can look up their compilers or their methods. In this case we're going to do 2, Java and C#. Additionally, I'm going to use the Java class files in ColdFusion just to show how we can take a legacy system and have it work. It would also work with something like Scala as it runs on the JVM and also has support for Java classes.
File: listing.proto

package tv;

option java_outer_classname="ListingProto";

message Listing{
	required string ProgramName = 1;
	required string EpisodeName = 2;
	required int32 EpisodeNumber = 3;
	required bool IsLive = 4;	

Protocol Buffer - Java File##

We can create the Java files and import it into our project with just this line:

protoc --java-out=[your output path] listing.proto

That should give us a bunch of Java files. Optionally, we can compile this Java file and get the binaries and package it into a jar. To do that you can grab the Java protocol buffer package from Maven and add it to your class path for compiling.

javac -cp [path to protobuf jar] *.java

That should give us a bunch of .class files.

The project webpage has a bunch of tutorials on describing the protocol and how to use the builder class so I won't go into too much detail there.

Send The Message to RabbitMQ via ColdFusion##

Now that we've got the class, we can instantiate an instance, serialize it and pass it onto RabbitMQ. This is application Alpha's job.

<cfset listing = createObject("java","tv.ListingProto$Listing")/>
<cfset builder=listing.newBuilder()>

<cfset message=builder
	.setProgramName("The Simpsons")
	.setEpisodeName("Bart Gets an F")


	factory = createObject("java","com.rabbitmq.client.ConnectionFactory").init();
	connection = factory.newConnection();	
	messageProperties = createObject("java","com.rabbitmq.client.MessageProperties").init();



Pretty small amount of code and we can make it a bit cleaner by putting it into a CFC instead and get all dependencies via dependency injection which I posted about earlier for ColdFusion.

The trick here was to call the static methods to generate the builder which finally gives us the object we want to work with. Cfdumping any of the intermediate objects just throws a CF Error.

com.rabbitmq.client.MessageProperties can also be pulled from Maven and added to CF's classpath.

In RabbitMQ's web management portal, I made a custom Exchange and Queue binding for all messages to go to the tv.* queue. You can read more about RabbitMQ's exchanges and queues. I happened to install RabbitMQ on an Ubuntu virtualbox image and enabled the web interface.

Consuming RabbitMQ Message in .NET##

Let's recap what we've done so far:

  1. Speced out our message class and generated the Java files
  2. Published the message in ColdFusion which used the Java files (Alpha)

Now we need another application to consume the message which is namely application Beta. Beta is newer and more up to date. It's written in C#. C# is not on the list of protocol buffer languages listed on its proejct page. But there is an implementation available we can just install via Nuget:

Install-Package protobuf-net

Unlike the first step though, we do not call a compiler on the .proto file. Instead we will define the class using attributes in our source code.
File: Listing.cs

public class Listing
    [ProtoMember(1, IsRequired=true)]
    public string ProgramName { get; set; }
    [ProtoMember(2, IsRequired = true)]
    public string EpisodeName { get; set; }
    [ProtoMember(3, IsRequired = true)]
    public Int32 EpisodeNumber { get; set; }

    [ProtoMember(4, IsRequired = true)]
    public bool IsLive { get; set; }

And our code to consume the message from RabbitMQ in a .NET console application:

class Program
    static void Main(string[] args)
        var factory = new ConnectionFactory() { HostName = "localhost", UserName="xxx",Password="yyy" };
        using (var connection = factory.CreateConnection())
            using (var channel = connection.CreateModel())
                channel.ExchangeDeclare("MY_EXCHANGE_NAME", "topic",true);
                var queueName = channel.QueueDeclare("TV Queue",true,false,false,null);

                if (args.Length < 1)
                    Console.Error.WriteLine("Usage: {0} [binding_key...]",
                    Environment.ExitCode = 1;

                foreach (var bindingKey in args)
                    channel.QueueBind(queueName,"MY_EXCHANGE_NAME", bindingKey);

                Console.WriteLine(" [*] Waiting for messages. " +
                                  "To exit press CTRL+C");

                var consumer = new QueueingBasicConsumer(channel);
                channel.BasicConsume(queueName, true, consumer);

                while (true)
                    var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
                    var routingKey = ea.RoutingKey;
                    var body = ea.Body;
                    MemoryStream stream=new MemoryStream(body,false);

                    stream.Position = 0;
                    var message = Serializer.Deserialize<Listing>(stream);

                    Console.WriteLine(" [x] Received '{0}':'{1}'",
                                      routingKey, message.ToString());

                    Console.WriteLine("Program Name: {0}", message.ProgramName);
                    Console.WriteLine("Episode Name: {0}", message.EpisodeName);
                    Console.WriteLine("Episode Number: {0}", message.EpisodeNumber);
                    Console.WriteLine("Is Live: {0}", message.IsLive);

This establishes a connection to RabbitMQ and starts listening for messages. You will get:

Program Name: The Simpsons
Episode Name: Bart Gets an F
Is Live: True

I plan to use TeamCity to automate much of the build to generate the dlls and jars to be more efficient. But this is a working proof and is a good way to start working with pub sub models on different languages/platforms.