01 / replay / one simulated Saturday / 2 venues / 29,027 messages
A Saturday at two venues, as the data arrived.
Every time a ticket is scanned, a drink is ordered or someone taps the venue’s app, a message is sent. This pipeline takes them as they come, cleans them, and keeps the numbers a manager would want, a few seconds behind the real thing. Play the day: everything below shows only what had arrived by that moment.
A real run of the pipeline on my laptop, replayed. The venues’ data is simulated: no real venue’s systems were involved.
Saturday, on the simulated clock
1:00 pm
1 / At the venues
- Door scans0
- Food and drink orders0
- App taps0
Door scanners, tills and the app send a message for everything that happens.
2 / The queue (Kafka)
0arriving this minute
Holds every message in order until the pipeline takes it, so nothing is lost if it falls behind.
3 / Kept as it came (bronze)
0messages stored
Every message stored exactly as it arrived, broken ones and repeats included, so anything can be rebuilt.
4 / Cleaned (silver)
- clean
- 0
- repeats dropped
- 0
- turned away
- 0
Checked against the rules. Repeats are dropped; anything broken is set aside with the reason.
5 / ready to use (gold): the tables a manager would look at
Recalculated every few seconds, only for the minutes that changed.
- Now
- Saturday College Football
- Doors open, show at 2:00 pm
- Guests in for this show
- 0of 1,564 tickets sold
- Takings so far today
- $00 orders
- Late scans put right so far
- 0
Turned away so far: 0. The latest, each with its reason. Open one to see the message as it arrived.
Nothing broken has arrived yet.
02 / how it works
Six steps, every few seconds.
- 1A simulator plays a Saturday at two venues, three shows each. It misbehaves on purpose: it sends some messages twice, some late, some broken, and writes down exactly what it really sent.
- 2Every message goes into Kafka, a queue that keeps them in order, so nothing is lost if the next step falls behind.
- 3Spark reads the queue every few seconds and stores every message exactly as it came. If a rule further on is ever wrong, the originals are still there to rebuild from.
- 4Each message is checked against the rules. Repeats are dropped, and anything broken is set aside with the reason, never quietly thrown away.
- 5Only the minutes that changed are worked out again, into the finished tables: guests through each door, sales at each bar, and a summary of every show. A scan that arrives twenty minutes late puts its own minute right.
- 6At the end of the day, checks prove nothing was lost or counted twice, and that the totals match what the simulator really sent, to the cent.
03 / is it right
29,027 messages in, every one accounted for.
Of 29,027 messages, 28,308 were clean, 561 were repeats and 158 were broken, each set aside with its reason. 431 arrived late, the latest nearly 25 minutes after it happened, and every one was counted in the minute it belonged to. The takings came to $418,329 across the six shows, exactly what the venues sent. From the queue to the cleaned table usually took 7 to 10 seconds.
All 27 checks, and the three speed measurements
- pass · ticket_scans: clean rows match a full recount7,313 rows
- pass · ticket_scans: quarantined rows match a full recount37 rows
- pass · ticket_scans: every message is accounted for7,497 received = 7,313 clean + 37 quarantined + 147 duplicates removed
- pass · pos_orders: clean rows match a full recount8,491 rows
- pass · pos_orders: quarantined rows match a full recount53 rows
- pass · pos_orders: every message is accounted for8,711 received = 8,491 clean + 53 quarantined + 167 duplicates removed
- pass · app_events: clean rows match a full recount12,504 rows
- pass · app_events: quarantined rows match a full recount68 rows
- pass · app_events: every message is accounted for12,819 received = 12,504 clean + 68 quarantined + 247 duplicates removed
- pass · silver.ticket_scans: no repeated event_id0 repeats
- pass · silver.pos_orders: no repeated event_id0 repeats
- pass · silver.pos_order_items: no repeated order_id + line_no0 repeats
- pass · silver.app_events: no repeated event_id0 repeats
- pass · silver.ticket_scans: every row belongs to a known show0 orphans
- pass · silver.pos_orders: every row belongs to a known show0 orphans
- pass · silver.app_events: every row belongs to a known show0 orphans
- pass · gold.entry_flow_minute: equals a full recalculation from silver0 differing rows
- pass · gold.sales_minute: equals a full recalculation from silver0 differing rows
- pass · gold.show_summary: equals a full recalculation from silver0 differing rows
- pass · net sales agree across orders, order lines and both gold tables$348,883.00 / $348,883.00 / $348,883.00
- pass · ticket_scans: no message lost between Kafka and the lake7,497 messages
- pass · ticket_scans: each bad message caught for the right reason37 caught across 4 reasons
- pass · pos_orders: no message lost between Kafka and the lake8,711 messages
- pass · pos_orders: each bad message caught for the right reason53 caught across 7 reasons
- pass · app_events: no message lost between Kafka and the lake12,819 messages
- pass · app_events: each bad message caught for the right reason68 caught across 5 reasons
- pass · attendance, orders and revenue per show equal the source, to the cent6 shows, $418,329.20 revenue
- measured · silver.ticket_scans: seconds from Kafka to clean tabletypical 8.9s, 95% within 13.4s
- measured · silver.pos_orders: seconds from Kafka to clean tabletypical 9.5s, 95% within 17.7s
- measured · silver.app_events: seconds from Kafka to clean tabletypical 7.3s, 95% within 12.1s
04 / what the day looked like
Fight night filled the room. The film did not.
Championship Fight Night in Los Angeles was the fullest show of the day, at 92% of the room. Orbital: An Immersive Film in Dallas was the emptiest, at 48%. These are the finished tables the pipeline keeps, the ones a manager would open.
| Show | Venue | Doors | Guests | Room filled | Takings | Per guest |
|---|---|---|---|---|---|---|
| Saturday College Football | Los Angeles | 1:00 pm | 1,462 | 86% | $84,750 | $48 |
| Orbital: An Immersive Film | Los Angeles | 4:00 pm | 918 | 54% | $54,245 | $49 |
| Championship Fight Night | Los Angeles | 6:30 pm | 1,567 | 92% | $96,395 | $51 |
| Saturday College Football | Dallas | 1:00 pm | 1,149 | 77% | $66,298 | $48 |
| Orbital: An Immersive Film | Dallas | 4:00 pm | 713 | 48% | $43,035 | $50 |
| Championship Fight Night | Dallas | 6:30 pm | 1,223 | 82% | $73,606 | $50 |
Takings include tax and tips. Per guest is what each guest spent before them.
05 / what comes next
The same code, in the cloud.
This run used a real Kafka and a real Spark, on one laptop. The pipeline keeps where it stores data and where it reads from as settings, so the next step is running the same code on Databricks on AWS, with Amazon S3 holding the data and a managed Kafka in front of it, and pointing it at a venue’s real systems.