Using AMQP with RabbitMQ as an example

The work of a shawarma kiosk is very similar to how a website works, because in both cases visitors perform similar actions: 1 they provide data that needs to be processed; 2 they receive some data in return. In everyday life, the equivalent of GET requests is the venue's menu and everything we see, the outer shell. POST requests consist of the desire to place an order, a request for the complaint book, and so on; everything we can ask the seller for. What is the downside of a regular menu?

When a customer walks up to the kiosk, they have no inventory feedback and do not know how many lavash wraps of each kind are in stock.

So it has to figure out whether there is shawarma in thick lavash or, on the contrary, in thin lavash.

The seller will sometimes answer yes and sometimes no, and it would be nice to automate all of this to eliminate unnecessary operations.

What is the simplest way to optimize a large number of orders, for example during the lunch rush?

Shawarma vendors are already using it extensively.

We need to give customers the ability to place orders by phone (order now and pick up in 15 minutes).

The downside is that we still need to process this queue, and the phone line may not handle the surge of calls.

Then customers will be left without orders, and we will lose revenue.

Another way is to invite friends and start making shawarma together, but then new problems arise.

That is because nobody manages the queue.

The simplest option is to put a senior waiter, manager, or keeper above all the workers, who will distribute the orders.

Next to each record, they set a status of 'done' / 'not done' and then take the working data from this table.

The most obvious drawbacks of this approach are: 1 we clearly do not lock the content, so several workers can take the same order; 2 if there are many orders and MySQL is on the same server as the application itself, which is usually the case, we can simply bring down our database and everything will go badly.

Orders, and not only orders, are reliably written to the queue.

Orders are distributed among the cooks.

If the cook, say, steps away or breaks an arm, the order stays in the queue. That cook can take it later, or another cook can take it.

The order is locked so that two cooks do not take the same order at the same time.

There are separate queues for pizza and rolls.

We now move smoothly to implementing a queue broker in the form of RabbitMQ.

This is a hybrid broker; it supports multiple protocols.

Today the most common protocol is AMQP. That is what we will mostly talk about.

Or the worker subscribes to messages and tries to process them as soon as they appear.

The broker can also return messages to the queue so they remain there when needed.

In most cases, developers write it like this: "Thank you for your request, we will create a PDF and email it to you."

If the project is small, this logic is usually handled in MySQL tables.

But as soon as scaling becomes an issue, MySQL starts to fail: locking is possible, queue management is missing, and processing is slower compared to memory-based solutions.

A relational database is fine, but it is better to store messages in faster storage that is more suited for that purpose.

A message is an entity unit that is not interpreted by the broker.

This can be just a string, or an object in JSON form, or any structure represented as a string. It does not matter what we put there; we can specify the message type, for example application/JSON, and then it will be easy to know during processing what was originally sent.

The message is not interpreted; it is simply stored.

Then it goes into the exchange, into the routing point.

The routing point's task is to determine which queue the message should go to.

We can send a key along with the message.

The key tells us which queue or queues the message should go to.

There are several routing types and modes of operation: 1 the simplest option is fanout, when a message goes to all available queues; 2 direct - an exact key match.

We can create one queue for one key and another queue for another; a queue can have multiple keys, just like in regular STP routing; 3 topic - the key matches the pattern (#*), where * is any non-empty single word and # is the same as *, except that an empty word is also allowed here; 4 headers - we can specify some type directly in the message and route based on it, which is used when there is no separate key. Suppose we have different routing keys made up of several words.

If we specify multiple keys, for example europe.weather, then this message needs to go to both europe and weather.

If usa.weather, then both in usa and in weather.

In this way, we can route all messages to several queues at once.

TTL & expiration With time to live, we can specify for each message or for all messages in the queue how long they will live. For example, an order comes in. We understand that after 60 minutes it will already be outdated, since in the stock market rates will change significantly within an hour, and in the shawarma example no customer will wait an hour for shawarma. We can set a TTL so that any message that sits for more than 60 minutes disappears from the queue.

Dead lettering Dead lettering is when our message returns to the queue with an error and we need to process it not immediately, but after a 5-10 second delay. This often happens when the data provider is unavailable, meaning the provider returns an error such as 404, 504, or something similar. There is nothing wrong with that: we return the message to the queue, and when we do, we can specify when this item will become available for retry.

To start working with Rabbit, we do not need any fancy extras, just plain composer.

Pull the Docker image and run it.

I recommend pulling by the 3-management tag.

This tag provides access to the latest stable RabbitMQ release; the management suffix means it comes with an administration panel as a Web UI, a user interface presented as a website that runs in a web browser.

This is very convenient, roughly like MySQL together with phpMyAdmin out of the box.

Here we can see all our queues, all our RabbitMQ connections, channels, and exchanges at once, in short, all the details.

Here we also see statistics: how many messages were processed; how many messages were not processed; which messages did not reach any queue. RabbitMQ has default queues where all messages that did not match any routing rule go, and we can handle them later. In PHP, everything connects quite easily.

When creating a connection, we always have to specify parameters, so there is no point in keeping a separate instance that would set Rabbit configuration parameters.

If such a queue, exchange, or anything else did not exist before, we always create it again, and RabbitMQ runs continuously, meaning it works persistently. We have the exchange name and the queue name, we create a connection to our host and port, use the authentication data, and a channel opens for work over that connection.

After setting up the queue, we need to create the exchange itself, the routing point, and bind it to one of the queues. The exchange also has a few parameters: type (there are four types, which we discussed earlier); passive; durable (whether we keep the unit after a restart); auto_delete (whether the queue is cleared when the connection closes).

After the new exchange is declared, we can bind our previous queue to our exchange.

Publishing is also quite simple: we can immediately specify the message text, the content type contained in the message, and delivery_mode, which determines how quickly the message appears in the queue.

We call basic_publish, and everything is sent to the queue immediately.

Then we can retrieve something from the queue.

For this we use basic_get, specify which queue we are reading from, and specify the message. This message also stores, in addition to our message and the data we set, information about which channel was used and which one is currently used, which routing_key was used, and generally everything that surrounded our message during processing.

Now let's talk about consume: it is a connection to a queue with the expectation that messages will arrive there. In principle, we could use basic_get, wrap it in a loop, and simply wait for a message from the queue.

But there may be nothing in the queue, and in addition we will be taking an inefficient path.

That is why consume was created: it is a long-lived connection through which you can receive things.

This results in a somewhat asynchronous working model.

Since we are calling process_message and specifying a callback function, we need to define and name that function.

It is not very complex: a message arrives, and we can do something with it.

We can also exit this loop, that is, pause consume from inside the callback.

To do this, as shown in the example, we need to check the body.

If quit comes in the body, as in the example, we need to exit.

To exit, we go back through message to the channel object stored in the message property, and call basic_cancel with some tag. Consumer_tag shows who took this message for processing. So we exit, and our loop stops and ends.

We will still need a while loop, but the consume approach is more optimal than constantly requesting messages, because a new connection is not created for every request.

Here register_shutdown_function is a standard PHP function in which we can define a callback function, for example shutdown, that will run when the script finishes.

If you need to send many messages at once, it is better to use publish_batch, which creates a transaction from multiple messages. It is very simple.

Best practices: Keep your queue short We try not to let our queue grow too large. Queue length is limited not only by RAM, but also by the number of messages. Rabbit is not built to store giant queues - it is built to deliver messages from something very simple, such as our frontend or some starting point, to something complex, like our backend, which may process messages for a very long time.

Lazy queues (3.9+) If we are worried about data durability, we use lazy queues. Although I do not consider this a best practice because performance suffers. We get disk persistence, which is not very fast, but it guarantees that data will never be lost.

Limit queue size with TTL or max-length We limit queue size by restricting message lifetime or maximum length. This is straightforward: since we work with memory, we do not want it to leak or be used improperly.