Rebuilding the RFID integration on per-store queues

Context

La Casa de las Carcasas is a mobile accessories retailer with more than 1,200 stores worldwide. Its stores record stock movements through RFID, and an integration turns those readings into stock movements in the company's systems.

The volume is around 9,000 movements a day, with peaks of 20,000, and most of them arrive between 10:00 and 15:00 (Spanish peninsular time).

Problem

About 5,000 stock movements were lost every week. Two things caused it:

• Poor performance. The previous implementation did not perform well enough for the load.
• No visibility. There was no incident management, no view of the pipeline, and no ordering of the movements within the same store, which could lead to stock errors. When something went wrong, nobody could see where.

What I did

• Replaced the legacy code with FIFO queues per store, so each store's movements are processed in the order they happened.
• Moved the processing to Laravel Queues backed by Redis, monitored with Horizon.
• Gave the team full visibility of the pipeline, so every movement can be followed through the flow.
• Designed the failure handling so that about 90% of incidents correct themselves, with no human intervention.

Result

• Lost stock movements: from about 5,000 a week to practically zero.
• Processes around 9,000 movements a day, with peaks of 20,000, most of them between 10:00 and 15:00.
• Movements of the same store are always processed in order.
• About 90% of incidents are fixed automatically.

Stack

Laravel Queues, Redis, Horizon.

What I learned

What FIFO queues are really for. Used properly, they give excellent performance and good visibility at the same time.

It also taught me to go looking for a system's weak spots on purpose, and to design it so that about 90% of incidents correct themselves without anyone having to step in.