Sending Messages - RocketMQ Knowledge System (II)

[[410032]]

In the previous article, we got to know RocketMQ. This article will talk about the production side of RocketMQ and the principles of message sending.

Message sending process

RocketMQ client message sending can be divided into the following three layers:

Business layer: directly call the business code of the MQ Client sending API;

Message processing layer: After RocketMQ Client obtains the message object sent by the business, it performs a series of operations such as parameter checking, message sending preparation, parameter packaging, etc.

Communication layer: RocketMQ is an RPC communication service encapsulated by Netty. All the communications between RocketMQ components use this module.

The general process:

• When Broker starts, it registers information with NameServer
• When the client calls the producer to send a message, it will first obtain the routing information of the topic from the NameServer. The message header code is GET_ROUTEINFO_BY_TOPIC
• The routing information returned from the NameServer includes the queue list and broker list contained in the topic
• The Producer selects one of the queues based on the query strategy for subsequent message storage.
• Each message will generate a unique id and add it to the message attributes. The key of the attribute is UNIQ_KEY
• Do some special processing on the message, for example: if it exceeds 4M, the message will be compressed
• The producer sends an RPC request to the broker and saves the message to the broker. The code of the message header is SEND_MESSAGE or SEND_MESSAGE_V2 (the configuration file sets a special flag)

Message data structure

Message

The physical carrier of information transmitted by the message system, the smallest unit of production and consumption of data, each message must belong to a topic. Each message in RocketMQ has a unique Message ID and can carry a Key with a business identifier. The system provides the function of querying messages by Message ID and Key.

Core field configuration

Other Message Configuration

Producer Configuration

Message sending method

Rocketmq provides three ways to send common messages: synchronous, asynchronous, and one-way.

• Synchronous: After the sender sends a message, it will send the next message only after receiving a response from the server.
• Asynchronous: After sending a message, you can continue to send messages or process subsequent tasks without waiting for the server to return. The sender receives the server response through the callback interface and processes the response result.
• OneWay: The sender sends a message without waiting for the server to return a response and no callback function is triggered, that is, only a request is sent without a response.

Comparison of sending methods: sending throughput, one-way > asynchronous > synchronous. However, one-way sending has poor reliability and may cause message loss. The selection is determined according to actual needs.

2. Message Type

The message client provides multiple SDKs: normal, sequential, transactional, and delayed messages

Producer Load Balancing

When a producer sends a message, it polls all queues by default, and the message will be sent to different queues. Queues can be distributed on different brokers.

Producer high availability

【Application scenarios】

Suppose there is a cluster consisting of three broker nodes, with topic 1. By default, four queues are created on each broker, namely: master-a (q0, q1, q2, q3), master-b (q0, q1, q2, q3), master-c (q0, q1, q2, q3). The last time a message was sent to the q0 queue of master-a, master-a crashed. If you continue to send topic 1 messages, how can you avoid sending them to master-a again?

RocketMQ's solution:

Sending failure retry and Broker fault delay avoidance mechanism. The configuration item retryTimesWhenSendFailed is used to indicate the number of synchronous retries, which is 2 by default, plus 1 normal send, for a total of three opportunities; the queue selection method is controlled by the value of sendLatencyFaultEnable, which is false by default, and does not start the broker fault delay mechanism. When the value is true, the broker fault delay mechanism is enabled.

(1) Retry after sending failed

RocketMQ supports synchronous and asynchronous sending. Either method can be retried after failure. If a single Broker fails, the retry will select other Brokers to ensure normal message delivery.

Failure retry logic:

(2) Broker Avoidance Mechanism

RocketMQ Client maintains a "Broker-Send Delay" relationship and selects a Broker with a lower send delay level based on this relationship. This can maximize the use of Broker capabilities, eliminate Brokers that are down, unavailable, or have a high send delay level, and ensure that messages are sent normally as much as possible.

What to do if NameServer is down?

If Namesrv is down, the newly added production and consumption will not be able to obtain the topic routing information and will report MQExecption; if the production and consumption caches the routing information of the producer's cached Topic, if all NameServers are down, messages can still be sent at this time.