Spotify's RAP Shows Advertisers a Faster Way to Use Data Lakes

Why Advertisers Should Care About Spotify's Data Architecture

Advertisers live and die by data. The faster you can pull a user's history, segment them, and serve a relevant ad, the better your campaign performs. But the systems that store all that data aren't always built for quick, single-record lookups. That's a problem Spotify just tackled with a new architecture called Random Access Parquet, or RAP.

Spotify's RAP lets you run low-latency point queries directly on a data lake, without copying data into a separate operational database. For advertisers, that could mean fresher audiences, more responsive bidding, and fewer data pipelines to maintain. It's not a tool you'll use tomorrow, but it shows where the industry is heading.

The Data Lake Bottleneck

Modern data lakes have become the central repository for analytics and AI workloads. They hold petabytes of data, often in open formats like Parquet. But retrieving a single record—say, a user's recent listening history—has been painfully slow. Query engines like Trino and BigQuery are optimized for scanning large datasets, not for key-based lookups.

Even though cloud storage like Google Cloud Storage now offers millisecond access times, the overhead of query planning, metadata traversal, and file discovery can swamp the actual read. Spotify says it stores petabytes of online data in Bigtable and exabytes in a Google Cloud Storage data lake. Copying that much data to a service database on a regular basis is expensive and slow.

What RAP Does Differently

RAP adds an external index layer on top of Apache Parquet files. This index maps query keys—like a user ID—directly to specific Parquet files and row positions. Instead of scanning thousands of files, the query engine resolves the key through the index and then performs a targeted range read on the object store.

As new data lands in Iceberg tables, an index builder generates append-only index fragments. The immutable Parquet files stay untouched. This means the same dataset can serve analytics, machine learning pipelines, notebooks, AI agents, and latency-sensitive online applications without duplicating storage.

For advertisers, the implication is clear: you can run both heavy analytical queries and real-time personalization on the same data. No more maintaining separate copies for batch and online use.

Ad-Tech Implications: Faster Targeting, Lower Costs

In ad tech, time is money. A bid request often has a hard deadline—sometimes under 100 milliseconds. If your data infrastructure can't fetch user preferences, past interactions, or contextual signals quickly, you lose the auction or serve a less relevant ad.

RAP-style architectures could let you keep all your user data in a data lake, yet still access it with sub-second latency. That means you can use the same dataset for training your ML models and for serving real-time ad decisions. You might even eliminate a dedicated serving database, cutting storage costs and reducing data pipeline complexity.

Spotify's approach also supports secondary indexes without rewriting Parquet files. For ad platforms, that means you can query by buyer ID, seller ID, or any other dimension you need, without changing your data pipelines. You just add a new index.

Storage Layout Tricks That Boost Query Speed

RAP isn't just about the index. Spotify also uses several storage layout optimizations to cut point-query latency:

• Sorting by query key reduces the number of files touched per query.
• Grouping related records together minimizes seeks.
• Interleaving value columns lets a single read fetch multiple attributes.
• Covering indexes can answer some queries without touching the Parquet file at all.

These techniques trade a bit of extra file size or index overhead for fewer storage operations. Some point queries end up as a single range read of just a few kilobytes.

For ad platforms, this could mean faster lookups of user profiles, ad impressions, or conversion events—all without spinning up a separate database.

The Open Data Ecosystem Angle

Spotify's RAP is part of a broader push to make open data lakes work for more than just analytics. Google Cloud recently described a lakehouse architecture based on Apache Iceberg that also aims to enable operational access while reducing data duplication. RAP takes a different route by adding a dedicated external index layer for point queries, while staying compatible with existing Parquet files and Iceberg tables.

Data engineers see RAP as a natural extension of open data formats. Andrew Lamb, for instance, views it as an example of pushing open formats to support interactive workloads. In a LinkedIn discussion, Vikas Singh noted that as cloud object storage gets faster, the bottleneck shifts to query planning and metadata access. RAP tackles exactly that by precomputing indexes.

What This Means for Advertisers and Ad Tech

If you're running an ad platform, you probably have a mix of batch processing for reporting and a real-time system for bidding. That split is expensive. Every time you copy data from your data lake to a serving database, you pay for storage, compute, and engineering time.

RAP-style architectures could let you unify those two worlds. Use the same Parquet data for both analytical queries and low-latency point lookups. That means fresher data for personalization, fewer pipeline failures, and lower infrastructure costs.

Of course, RAP isn't a drop-in solution for every ad tech stack. It's designed for point queries, not for complex joins or full-text search. But for the kind of lookups that drive ad targeting and frequency capping, it could be a game-changer.

Next Steps for Advertisers

You don't need to wait for Spotify to open-source RAP (if they ever do). The principles are already available: use open table formats like Iceberg, sort your data by your most common query keys, and consider adding an index layer for point lookups.

Think about which of your ad queries are really point queries: fetching a user's profile, checking if they've seen an ad, retrieving their last purchase. If you can answer those with a quick index lookup instead of a table scan, you're already on the right track.

The future of ad tech is data efficiency. The less time you waste copying and scanning data, the faster you can deliver relevant ads—and the better your bottom line.