Home > Uncategorized > Is AMQP for you?

Is AMQP for you?

September 25th, 2010 Leave a comment Go to comments

The beginning

I needed to evaluate cross-platform asynchronous messaging solutions, and now I would like to share my experience. Googling and stacking for message queue popped up names like RabbitMQ, ActiveMQ, ZeroMQ, MQ. After some reading I found that most of the mentioned (Rabbit, Active, etc.) are messaging brokers implementing the AMQP (Advanced Messaging Queue Protocol) protocol. Ok, maybe it should be called AMQ protocol but set that aside. AMQP seemed to bring the possibility of persistent and reliable messaging. On the contrary, ZeroMQ turned out to be an intelligent super-socket, but without built-in persistence. This article describes the directions I took towards AMQP and its implementations, and how I (at least partially) rejected using it in favor of ZeroMQ.

The messaging is required for (1) client-server communication and (2) communication between loosely coupled modules inside the server. Multiple languages are to be supported including C++, Python and Java. My first thoughts were to use ZeroMQ for (2) since we originally planned some socket or ipc-based communication anyway, and some form of reliable messaging for (1). But then it occured to me why should we use two different messaging systems, therefore pondered the opportunity to use this AMQP-thing for (2) too.

The Pythonic racers for AMQP

In a nutshell about AMQP: It has a broker to which users connect using client libraries, then they can declare message exchanges, queues, bindings (routing rules), set durability options and so on. There is a swarm of AMQP client libs on first sight, however this is where complications arise. I started out with testing Python clients:

  • py-amqplib was the first to stumble upon. It is nice and easy, but has no multithreading support (which turned out that non of the python libs have so this is not a big loss anyway), but more importantly it doesn’t have non-blocking reception of messages (at least I couldn’t find it).
  • txAMQP uses the twisted library, which is… twisted. But therefore it naturally has async support.
  • pika seemed to be a nice one too with async support
  • Qpid also had a python client

For broker I used RabbitMQ which seems very good in what it does (and it is written in Erlang, yay!). But then I was looking for a C++ client, and found the following:

  • RabbitMQ has a C-client, which is fair by C standards, but is far from being user-friendly
  • Only Qpid has a C++ client. And this is where things start to go downhill

C++ neglected?

The former clients and the RabbitMQ broker only support version 0.8 (or at most 0.9x, but it is quite near to 0.8) of the AMQP draft, while the Qpid C++ client supports the 0.10 draft only (the python client also supports it). Therefore if I wanted to stick with AMQP and C++, I had to choose Qpid, which however discarded the use of the RabbitMQ broker. No problemo, Qpid has its own broker. Then I downloaded the last stable-marked release of Qpid, 0.6, and started to code some basic examples. Then, I stumbled upon that the online API docs for 0.6 (which seemed to be very nice at first) are out of sync with the doc bundled in the downloaded package. Ok, use the docs from the package, but it is still fishy, isn’t it?

Qpid shot me in the heart

The next surprise is the two-fold nature of the Qpid API. There are similarly named classes both in the qpid::client and the qpid::messaging namespace. Turns out that client is the conventional AMQP API as we are used to deal with, and messaging is an abstraction over AMQP (or rather, AMQP is the currently single implementation of the interface defined by the abstraction), which uses JMS-like terminology and semantics. The lower-level API would have been fine, but it again doesn’t have async message reception (ok not such a big deal, introduce a thread, synchronize, etc. etc. but it just didn’t seem right). The high-level API (having nice async reception, yay!) would have finer control over the AMQP operations performed underneath, configurable via the address strings, but these have such an obscure and mal-documented syntax that I couldn’t squeeze out the things I wanted from it.

This concluded my forming opinion that Qpid is better to avoid for now, since it:

  • imposes lock-in (must use Qpid broker)
  • has a poor, and even worse, confusing documentation
  • not sure about the activity of development, and if I get stuck on a bug then suddenly I’m in the middle of no man’s land

Unfortunately that means no C++ client for AMQP.

Zero effort is the best effort

Then, I decided to revert to ZeroMQ for server-internal communication, and maybe just keep AMQP 0.8 with RabbitMQ and pals for client-server communication. But then the same argument arises, is mixing technologies worth it? In this case the only advantage of AMQP would be its reliable messaging, but in this premature phase we don’t even really know what are the reliability requirements for our case would be. Also, ZeroMQ seems to be a fine piece of software, so maybe we would choose it for client-server communication also, and if the reliability conditions are clear we could try to implement it over ZMQ (don’t expect too big reliability requirements, more like “if the client is gone, this update message should stay in a queue for a while; but then from what I learned ZeroMQ supports this kind of buffered retransmit already).

Further reading

Also, an interesting reading for the patient ones in [1], which reveals some detail about the connection of ZeroMQ and AMQP, and also the sad story AMQP. Coupled with [2] and [3] you can draw some conclusions if you would like. The summary is not that I’m against AMQP and wish it a good luck (in coherence with [1]) but it may be overblown for the need we have right now.

Update: It turned out the Qpid messaging API was an intermediate experimental version. Maybe better luck with upcoming releases [4].

[1] http://www.imatix.com/articles:whats-wrong-with-amqp
[2] http://www.h-online.com/open/features/The-Red-Hat-Patent-Problem-and-AMQP-746549.html
[3] http://www.ipocracy.com/blog:redhat-claims-amqp
[4] http://stackoverflow.com/questions/3789822/qpid-finer-amqp-control-with-the-high-level-api

Digg This
Reddit This
Stumble Now!
Buzz This
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Kick It on DotNetKicks.com
Shout it
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)
Categories: Uncategorized Tags: , , , ,
  1. James
    January 9th, 2011 at 12:23 | #1

    Seems like I just went through exactly the same elimination process as you did 4 months ago. We needed C#, C++ and Python support and wanted to introduce a message based architecture.

    I first used RabbitMQ as a prototype which seemed to work nicely in C# but obviously couldn’t be used due to the lack of C++ client. Then I tried to use the Qpid implementation (upon which I quickly discovered that any AMQP implementations dont seem to be interoperable with each other!) but their message based api was useless and confusing (no decent examples or explanations to map back to the raw AMQP concepts) – not to mention that I couldn’t get the precompiled C# dlls to load into my prototype projects.

    Now we’re banging on the door of 0mq to salvage the day – I hope we can use it to produce a fairly robust and fast messaged based architecture!

    Your article gives me more confidence that my conclusions on AMQP implementations thus far have been well based. Cheers!

  2. July 8th, 2012 at 21:31 | #2

    Hi Ron,
    I have PHP application that I want to talk to some python scripts internally. I was wondering if using amqp and some library like celery would be a better choice than running python scripts through php exec(). What do you think?

  1. No trackbacks yet.