Asset Scan Processing Pipeline

When a device publishes an asset scan, an IoT Rule fans the message out to two independent processing paths simultaneously.

The two paths are independent. A failure in the archive path does not affect Tool Location Services submission, and vice versa.

Tool Location Services

The ingest Lambda validates the payload, then processes each BLE advertisement:

1. Parse advertisements — each xAd is inspected for the Milwaukee manufacturer company code (0x0165). Valid advertisements yield the scanned tool's MPBID and the scan metadata (RSSI, timestamp).

2. Resolve location — if the message includes a GNSS fix it is used directly. If only LTE tower data is present, the backend calls AWS IoT Wireless to derive a position estimate from the cell tower information.

3. Submit to TLS — the resolved locations are forwarded to the Tool Location Services API as tool location detail records. This is what drives last-known-location and proximity features in OneKey.

Advertisements that are malformed, oversized, or don't carry a Milwaukee payload are skipped.

Data Lake Archive

The archive Lambda runs in parallel with ingest. Before any processing occurs, the raw message is captured and written to the data lake. This serves two purposes:

• Replay — if the ingest Lambda fails or the Tool Location Services API is unavailable, the original messages can be replayed from the archive without data loss.
• Historical queries — the full scan history is queryable via Athena, including fields that aren't forwarded to TLS (raw advertisements, LTE tower details, session IDs, etc.).

Storage

Messages are written to Kinesis Firehose, which buffers and flushes to S3 in Parquet format (zstd compression). Each record in the archive includes the full original payload plus ingestion metadata:

Field	Notes
Original payload	Complete JSON as published by the device
Message ID	UUID generated at archive time
Receipt timestamp	When the IoT Rule received the message
MQTT topic	The topic the message was published to
Payload size (bytes)	For monitoring and anomaly detection

Partitioning: event_date=YYYY-MM-DD

Retention: 365 days. Failed/backup records are retained for 30 days.

Firehose buffering: Flushes when the buffer reaches 64 MB or 60 seconds, whichever comes first.

Querying with Athena

The S3 archive is registered in the Glue Catalog and is queryable via Athena. Glue runs automated optimizers (compaction, orphan file deletion, retention enforcement) to keep the Iceberg table healthy.

Contact the CPP team for the Glue database name, table name, and Athena workgroup for your environment.