Skip to content

Offset Tracking: Avoiding Message Loss

In this how-to we're going to expand the consumer loop script with the capability to keep track of the current stream position. This allows clients to resume messages consumption starting from their last received message across application restarts, system failures or network outages. Hence, this mechanism ensures no data is lost on the client side.


Prerequisites

You should have a working copy of a python consumer application that connects to and receives data from our SWIM service. While the example here is provided in Python it should be easy to translate to other programming languages. Please note that the underlying concepts are the same, since we only used Qpid Proton as a standard AMQP 1.0 client.

The Consumer Loop Script How-to is a good start for this. Make sure you have worked through it, before proceeding here.


Understanding Stream Position Indicators

The broker adds the key x-stream-offset to the AMQP message annotations. The value represents the current offset position and increases for each message. This allows us to use it as an indicator of the current position of the client we're using and re-attach to this position when re-connecting to the subscription.

The goal of this how-to is to extend the consumer loop to keep track of the offset position of the previously consumed messages and storing it persistently. Further, it reads the offset position on application restart and uses it as a starting offset when reconnecting to the broker in order to ensure that no data is lost during disconnects.

Step 1: Keep Track of the Current Stream Position

The on_message method of the PrintMessageHandler in the Consumer Loop Script can be extended in order to keep track of the current stream position.

The following code-example is examining the current message annotations for the stream position indicator and calls write_pos_file in order to store it to a cache file.

def on_message(self, event):
    msg = event.message
    pos = None
    if (msg.annotations is not None and 
        "x-stream-offset" is in msg.annotations):
        pos = msg.annotations["x-stream-offset"]
        write_pos_file(
          "/queues/de-dwd.weather.aviation.metar", pos)

Step 2: Store the Stream Position to a Persistent Cache

A stream position cache is used to store the positions for each consumed stream. This could be a file-based, database or memory-cached store in order to keep this information available, especially between application restarts.

Here, we utilise a json-File based cache. For this we define the variable pos_file containing the file name of our cache.

pos_file = Path("stream_position.json")

The exemplary write_pos_file looks as follows:

def write_pos_file(stream_name: str, pos: int):
    data: dict[str, int] = {}

    if pos_file.exists():
      with pos_file.open("r", encoding="utf-8") as f:
        data = json.load(f)

    data[stream_name] = pos

    with pos_file.open("w", encoding="utf-8") as f:
        json.dump(data, f, indent=2)

Step 3: Reloading Stream Positions on Application Restart

On a restart of the consumer app, we read the last stored stream position as an initial step.

def read_pos_file():
    if pos_file.exists():
      with pos_file.open("r", encoding="utf-8") as f:
        data = json.load(f)
        return data

    return None

Step 4: Reconnecting at the Last Stream Offset

Having retrieved the last consumed message position from our cache file, we can increment this position counter in order to continue consuming at the first not-consumed message, e.g. we use it as a filter on the create_receiver invocation.

offset_map = read_pos_file()
pos = "next"
if offset_map is not None:
    pos = offset_map.get("/queues/de-dwd.weather.aviation.metar", -1) + 1

receiver = conn.create_receiver(
    "/queues/de-dwd.weather.aviation.metar",
    handler=PrintMessageHandler(print_app_properties=False, print_message=True),
    options=Filter(
        {
            symbol("rabbitmq:stream-offset-spec"): Described(
                symbol("rabbitmq:stream-offset-spec"), pos
            ),
        },
    ),
)

Additional Steps

In a production environment, you might want to consider using a more elaborate caching backend, such as key/value stores (e.g. MinIO, Redis) or a database.

Further, You might want to use a different offset tracking method as detailed in our Offset Tracking Tutorial, such as absolute unix timestamps or using a duration to retrieve all messages sent since the specified period.


Appendix - complete script

Here is the complete listing of our working example from this lesson.

consumer_example.py
import json
from pathlib import Path

import typer
from dotenv import load_dotenv
from proton import Described, symbol
from proton.handlers import MessagingHandler
from proton.reactor import Filter
from proton.utils import BlockingConnection
from rich import print
from typing_extensions import Annotated

load_dotenv()
pos_file = Path("stream_position.json")


class PrintMessageHandler(MessagingHandler):
    def __init__(
        self,
        print_app_properties: bool = True,
        print_message: bool = False,
    ):
        super().__init__()
        self.print_app_properties = print_app_properties
        self.print_message = print_message

    def on_message(self, event):
        pos = None

        msg = event.message
        if msg.annotations is not None and "x-stream-offset" in msg.annotations:
            pos = msg.annotations["x-stream-offset"]
            print("stream-offset is", pos)  # (1)!
            write_pos_file("/queues/de-dwd.weather.aviation.metar", pos)  # (2)!

        if self.print_app_properties:
            print(msg.properties)

        if not self.print_message:
            return

        if isinstance(msg.body, memoryview):
            print(msg.body.tobytes().decode("utf-8"))
        elif isinstance(msg.body, str):
            print(msg.body)


def write_pos_file(stream_name: str, pos: int):
    data: dict[str, int] = {}

    if pos_file.exists():
        with pos_file.open("r", encoding="utf-8") as f:
            data = json.load(f)

    data[stream_name] = pos

    with pos_file.open("w", encoding="utf-8") as f:
        json.dump(data, f, indent=2)


def read_pos_file():
    if pos_file.exists():
        with pos_file.open("r", encoding="utf-8") as f:
            data = json.load(f)
            return data

    return None


def consume_message(
    user: Annotated[str, typer.Argument(envvar="SWIM_USER")],
    password: Annotated[str, typer.Argument(envvar="SWIM_PASSWORD")],
):
    conn = BlockingConnection(
        "amqps://swimt.dwd.de",
        user=user,
        password=password,
    )

    offset_map = read_pos_file()  # (3)!
    pos = "next"
    if offset_map is not None:
        pos = offset_map.get("/queues/de-dwd.weather.aviation.metar", -1) + 1  # (4)!
    print(f"starting to read from {pos}")

    receiver = conn.create_receiver(
        "/queues/de-dwd.weather.aviation.metar",
        handler=PrintMessageHandler(
            print_app_properties=False,
            print_message=True,
        ),
        options=Filter(
            {
                symbol("rabbitmq:stream-offset-spec"): Described(
                    symbol("rabbitmq:stream-offset-spec"), pos  # (5)!
                ),
            },
        ),
    )

    if receiver.container is None:
        conn.close()
        raise Exception("Failure in setting up the receiver")

    receiver.container.run()
    conn.close()


if __name__ == "__main__":
    typer.run(consume_message)
  1. Here we examine the current stream position and print it to stdout.
  2. Write the current position to a file. In this example, we do this on every received message. If I/O becomes a bottleneck, one could consider writing this on a timely basis, e.g. every N seconds, every N messages or a combination thereof.
  3. On application restart, we read the last stored position, or choose "next" if no position is found in the stored position file.
  4. Increment the position counter in order to consume the first non-consumed message.
  5. Contrary to the consume_loop.py example, we now use the filter every time we create a receiver using either the last-but-one stored stream position or next.