This chapter is from the book
This chapter is from the book
This chapter is from the book
6.9 Resequencer
By Sebastian Krueger
Messages can arrive out of order for many reasons. If these messages are required to be delivered in sequence to a downstream component, the easiest solution would be to make sure that they never get out of order in the first place. Such approaches were discussed in Chapter 4, "Message Exchange Patterns," section 4.8. However, there may be times when we don't have a choice of how the upstream components are implemented; for example, we may control only the receiving side. Thus, the need to implement a component that will reorder messages may arise.
A number of implementations of a resequencer are possible. Chapter 4, section 4.2, "Resequencer," in Part II, discusses and illustrates two implementations using examples: a simple buffered resequencer and a persisted resequencer, both of which are discussed in the remainder of this section.
The simple buffered resequencer operates as follows. When the resequencer receives a message, it adds that message to an internal buffer. It then sends all consecutive messages from the buffer.
In order to send all consecutive messages, the resequencer component needs to know the current sequence number index and whether a message with this index is in the buffer. If the index is not found in the buffer, then the message has not arrived yet. The resequencer will not send out buffered messages until at least the next message arrives.
A resequencer implementation requires all messages to have a unique sequence number. Not only do these sequence numbers have to be unique, they also have to be consecutive. That is, no gaps are allowed to exist in the sequence.
If a message gets lost and never arrives, messages would be queued up and would never be sent out because the resequencer component would be waiting for a message that will never arrive. To get around this issue, the designer could implement a solution whereby the resequencer only waits a set time for a message to arrive and then moves on to the next messages, effectively ignoring the message that never arrived. However, what if the message arrives late? What if a duplicate message arrives? Strategies for dealing with these conditions would have to be considered in designing a robust resequencer. The designer could, for example, discard the message, or send it to a Dead Letter Channel for alerting and auditing purposes.
When a simple resequencer starts, it expects the first message to have a sequence number of 0. However, what if this is not the case? For example, the resequencer might restart. Unless the sequence is persisted, the resequencer would expect the message sequence to start at 0 again. There are two ways to get around this problem. An initialization message could be sent that informs the resequencer which number is the start of the sequence. Alternatively, the sequence could be persisted so that it can be recovered in appropriate circumstances.
Another point that a simple resequencer does not handle is buffer overrun. If too many messages get queued up, the HashMap, used to store messages while assembling message sequences, may get too large to fit into the JVM allocated memory.
Implementation of a robust and scalable resequencer would require a significant amount of code and would likely be domain specific. An improved resequencer is discussed next. Implementation of a perfect resequencer is beyond the scope of this book.
An improvement to the previous resequencer would be to implement the buffer as a database table. By moving the message buffer to a persistent store, we solve two problems exhibited by the previous implementation. First, we effectively have an unlimited buffer, so no buffer overflow will occur. Second, in case of a server failure, buffered messages are not lost.
There are still unresolved issues with this persisted resequencer. The initialization of sequence numbers expects the first message to always start at 0. Also, we have not accounted for messages that never arrive. We briefly touched on the some of these issues in the previous section. They are out of the scope of this chapter.
While the two resequencers discussed in this chapter are by no means perfect, they do give examples of simple resequencers and give an indication of what is required to implement a robust resequencer.