Offset Tracking: Avoiding Message Loss¶
In this how-to we're going to expand the consumer loop script with the capability to keep track of the current stream position. This allows clients to resume messages consumption starting from their last received message across application restarts, system failures or network outages. Hence, this mechanism ensures no data is lost on the client side.
Prerequisites¶
You should have a working copy of a python consumer application that connects to and receives data from our SWIM service. While the example here is provided in Python it should be easy to translate to other programming languages. Please note that the underlying concepts are the same, since we only used Qpid Proton as a standard AMQP 1.0 client.
The Consumer Loop Script How-to is a good start for this. Make sure you have worked through it, before proceeding here.
Understanding Stream Position Indicators¶
The broker adds the key x-stream-offset to the AMQP message annotations.
The value represents the current offset position and increases for each message.
This allows us to use it as an indicator of the current position of the client
we're using and re-attach to this position when re-connecting to the subscription.
The goal of this how-to is to extend the consumer loop to keep track of the offset position of the previously consumed messages and storing it persistently. Further, it reads the offset position on application restart and uses it as a starting offset when reconnecting to the broker in order to ensure that no data is lost during disconnects.
Step 1: Keep Track of the Current Stream Position¶
The on_message method of the PrintMessageHandler in the
Consumer Loop Script
can be extended in order to keep track of the current stream position.
The following code-example is examining the current message annotations
for the stream position indicator and calls write_pos_file in order
to store it to a cache file.
def on_message(self, event):
msg = event.message
pos = None
if (msg.annotations is not None and
"x-stream-offset" is in msg.annotations):
pos = msg.annotations["x-stream-offset"]
write_pos_file(
"/queues/de-dwd.weather.aviation.metar", pos)
Step 2: Store the Stream Position to a Persistent Cache¶
A stream position cache is used to store the positions for each consumed stream. This could be a file-based, database or memory-cached store in order to keep this information available, especially between application restarts.
Here, we utilise a json-File based cache. For this we define the
variable pos_file containing the file name of our cache.
The exemplary write_pos_file looks as follows:
def write_pos_file(stream_name: str, pos: int):
data: dict[str, int] = {}
if pos_file.exists():
with pos_file.open("r", encoding="utf-8") as f:
data = json.load(f)
data[stream_name] = pos
with pos_file.open("w", encoding="utf-8") as f:
json.dump(data, f, indent=2)
Step 3: Reloading Stream Positions on Application Restart¶
On a restart of the consumer app, we read the last stored stream position as an initial step.
def read_pos_file():
if pos_file.exists():
with pos_file.open("r", encoding="utf-8") as f:
data = json.load(f)
return data
return None
Step 4: Reconnecting at the Last Stream Offset¶
Having retrieved the last consumed message position from our cache file, we
can increment this position counter in order to continue consuming at
the first not-consumed message, e.g. we use it as a filter
on the create_receiver invocation.
offset_map = read_pos_file()
pos = "next"
if offset_map is not None:
pos = offset_map.get("/queues/de-dwd.weather.aviation.metar", -1) + 1
receiver = conn.create_receiver(
"/queues/de-dwd.weather.aviation.metar",
handler=PrintMessageHandler(print_app_properties=False, print_message=True),
options=Filter(
{
symbol("rabbitmq:stream-offset-spec"): Described(
symbol("rabbitmq:stream-offset-spec"), pos
),
},
),
)
Additional Steps¶
In a production environment, you might want to consider using a more elaborate caching backend, such as key/value stores (e.g. MinIO, Redis) or a database.
Further, You might want to use a different offset tracking method as detailed in our Offset Tracking Tutorial, such as absolute unix timestamps or using a duration to retrieve all messages sent since the specified period.
Appendix - complete script¶
Here is the complete listing of our working example from this lesson.
import json
from pathlib import Path
import typer
from dotenv import load_dotenv
from proton import Described, symbol
from proton.handlers import MessagingHandler
from proton.reactor import Filter
from proton.utils import BlockingConnection
from rich import print
from typing_extensions import Annotated
load_dotenv()
pos_file = Path("stream_position.json")
class PrintMessageHandler(MessagingHandler):
def __init__(
self,
print_app_properties: bool = True,
print_message: bool = False,
):
super().__init__()
self.print_app_properties = print_app_properties
self.print_message = print_message
def on_message(self, event):
pos = None
msg = event.message
if msg.annotations is not None and "x-stream-offset" in msg.annotations:
pos = msg.annotations["x-stream-offset"]
print("stream-offset is", pos) # (1)!
write_pos_file("/queues/de-dwd.weather.aviation.metar", pos) # (2)!
if self.print_app_properties:
print(msg.properties)
if not self.print_message:
return
if isinstance(msg.body, memoryview):
print(msg.body.tobytes().decode("utf-8"))
elif isinstance(msg.body, str):
print(msg.body)
def write_pos_file(stream_name: str, pos: int):
data: dict[str, int] = {}
if pos_file.exists():
with pos_file.open("r", encoding="utf-8") as f:
data = json.load(f)
data[stream_name] = pos
with pos_file.open("w", encoding="utf-8") as f:
json.dump(data, f, indent=2)
def read_pos_file():
if pos_file.exists():
with pos_file.open("r", encoding="utf-8") as f:
data = json.load(f)
return data
return None
def consume_message(
user: Annotated[str, typer.Argument(envvar="SWIM_USER")],
password: Annotated[str, typer.Argument(envvar="SWIM_PASSWORD")],
):
conn = BlockingConnection(
"amqps://swimt.dwd.de",
user=user,
password=password,
)
offset_map = read_pos_file() # (3)!
pos = "next"
if offset_map is not None:
pos = offset_map.get("/queues/de-dwd.weather.aviation.metar", -1) + 1 # (4)!
print(f"starting to read from {pos}")
receiver = conn.create_receiver(
"/queues/de-dwd.weather.aviation.metar",
handler=PrintMessageHandler(
print_app_properties=False,
print_message=True,
),
options=Filter(
{
symbol("rabbitmq:stream-offset-spec"): Described(
symbol("rabbitmq:stream-offset-spec"), pos # (5)!
),
},
),
)
if receiver.container is None:
conn.close()
raise Exception("Failure in setting up the receiver")
receiver.container.run()
conn.close()
if __name__ == "__main__":
typer.run(consume_message)
- Here we examine the current stream position and print it to
stdout. - Write the current position to a file. In this example, we do this on every received message. If I/O becomes a bottleneck, one could consider writing this on a timely basis, e.g. every N seconds, every N messages or a combination thereof.
- On application restart, we read the last stored position, or choose "next" if no position is found in the stored position file.
- Increment the position counter in order to consume the first non-consumed message.
- Contrary to the
consume_loop.pyexample, we now use the filter every time we create a receiver using either the last-but-one stored stream position ornext.