Skip to content

Using AMQP Offset Tracking to Retrieve Past Messages

This guide explains how to use AMQP offset tracking to retrieve past messages from the SWIM service. It basically follows the same steps as the connection to AMQP services tutorial but adds the necessary filter to retrieve past messages.

Install Prerequisites

For this tutorial we need to install the qpid proton library and rich for pretty printing. Using rich is optional but recommended, because it makes the xml output much more readable.

pip install python-qpid-proton==0.40.0 rich

Setup the Offset Tracking Filter

The most important part of this tutorial is the setup of the offset tracking filter. We use qpid proton's Filter class to create a filter with the appropriate offset tracking specification.

import json

from proton import Described, symbol, ProtonException
from proton.reactor import Filter
from proton.utils import BlockingConnection
from rich import print # (1)!

offset_spec = "30m"

OFFSET_FILTER_SPEC = "rabbitmq:stream-offset-spec"
offset_filter = Filter(
    {
        symbol(OFFSET_FILTER_SPEC): Described(
            symbol(OFFSET_FILTER_SPEC), offset_spec
        )
    }
)
  1. We use rich to print messages in a pretty way. This is optional but recommended for better readability.

Different Offset Tracking Options

The offset filter value (i.e., the variable offset_spec from above) defines from which point in time we want to receive messages. There are several options available:

  • next: Start from the message after the last one received on this subscription, i.e., all new messages after connecting. Default if no filter is provided.
  • first: Start from the first message in the stream.
  • last: Start from the last message in the stream.
  • Offset: A numerical value specifying the exact offset to start from. The offset of a message can be found in the message annotations under the key x-stream-offset, i.e., in qpid proton you can access it with msg.annotations['x-stream-offset'].
  • Timestamp: A numerical value representing the UNIX time (milliseconds since epoch) to start from.
  • Duration: A string value specifying the time interval relative to current time to receive messages from, e.g., 30m for the last 30 minutes, 2h for the last 2 hours, 1D for the last day (valid units Y, M, D, h, m, s).

This is adapted from the RabbitMQ documentation, which also contains a more specific view on the behaviour of each option.

Connect and Receive Messages

With the filter set up, we can now connect to the broker and receive messages from our desired offset. The code below is similar to the one in the connecting to AMQP services tutorial but includes the offset filter (the options parameter in create_receiver) when creating the receiver. If you have trouble understanding the code, please refer to the mentioned tutorial for a detailed explanation.

conn = BlockingConnection(
    "amqps://swimt.dwd.de",
    user="swim_user",
    password="your_password"
)

receiver = conn.create_receiver(
    "/queues/de-dwd.weather.aviation.metar",
    options=offset_filter
)
msg = receiver.receive(timeout=10)

if isinstance(msg.body, str):
    print(msg.body)
elif isinstance(msg.body, memoryview):
    print(json.loads(msg.body.tobytes().decode("utf-8")))

receiver.accept()
conn.close()

As an output you should receive a bunch of METAR's (~40) from the last 30 minutes (or whatever offset you specified).

Further Steps

  • The consumer loop script contains a fully featured consumer implementation including offset tracking and SQL filtering.

For further assistance or questions, contact swim.service@dwd.de.