Multiply that by a few million people on a Saturday night and you have one of the cleanest real-world examples of a Big Data problem I know of. The data is huge, it arrives continuously, it is messy, and the business wants answers from it in seconds, not tomorrow morning.

1. What the clickstream actually looks like

Before talking about tools, it helps to look at a single event. ReelStream's player emits JSON like this:

{
  "event_id": "9f1c2a7e-6b1d-4e0a-9c55-0d7e3b1f8a21",
  "user_id": "u_48213907",
  "session_id": "s_77a91c",
  "content_id": "t_10442",
  "event_type": "heartbeat",
  "position_sec": 1834,
  "bitrate_kbps": 2800,
  "buffer_ms": 6200,
  "rebuffer": false,
  "cdn_edge": "blr-edge-07",
  "device": "android_tv",
  "app_version": "7.3.1",
  "event_time": "2026-10-22T18:10:42.311Z"
}

There are roughly four families of events:

2. Why this is a Big Data problem: the 5 Vs, with numbers

"Big Data" gets thrown around loosely, so let's check ReelStream against the 5 Vs with actual arithmetic.

Volume. Say peak concurrency is 2 million viewers and the player sends a heartbeat every 10 seconds. That's 200,000 heartbeats per second at peak, before counting discovery and lifecycle events. At about 400 bytes per event:

200,000 events/s × 400 B = 80 MB/s ≈ 288 GB per peak hour

Averaged over a day (traffic is far from flat), ReelStream lands somewhere around 2–3 TB of raw JSON per day. Stored with HDFS's default 3x replication, that's 6–9 TB of disk per day. This alone rules out "just put it in MySQL".

Velocity. Some questions have a shelf life of seconds. If a CDN edge in Bengaluru starts dropping segments during a cricket final, ops needs to know within a minute, not in tomorrow's report.

Variety. JSON events from apps, CSV dumps from the billing system, a relational content catalog, subtitle files, thumbnails, and free-text search queries in Hinglish. Structured, semi-structured and unstructured all at once.

Veracity. Clients go offline and flush events hours later. Clocks on cheap Android TVs drift. Message queues deliver at-least-once, so duplicates are normal. Bots scrape the catalog. Any metric you compute has to survive this.

Value. The payoff is concrete: better recommendations raise watch time, early QoE alerts cut churn, and knowing which titles drive sign-ups shapes content buying decisions worth crores.

3. The architecture at a glance

Here's the overall design. Every box in it maps to something in the course syllabus, and the rest of the post goes through them one by one.

 Player / App SDKs
        │  (HTTPS, batched JSON)
        ▼
 ┌──────────────┐
 │ Ingest API   │──► Kafka topic: player-events (partitioned by user_id)
 └──────────────┘          │
                           ├──────────────► SPEED LAYER (Spark Structured Streaming)
                           │                 • Bloom filter: dedupe / "already watched"
                           │                 • DGIM: rebuffer counts per CDN edge
                           │                 • Flajolet-Martin / HLL: unique viewers per title
                           │                 • Trending titles (sliding windows)
                           │                       │
                           │                       ▼
                           │                 Redis (live counters)  +  Cassandra (watch history)
                           │
                           └──────────────► BATCH LAYER
                                             • Raw events land in HDFS (Parquet, partitioned by date)
                                             • MapReduce / Spark batch: daily views, genre joins
                                             • Similarity (Jaccard / cosine), CURE user segments
                                                   │
                                                   ▼
                                             Recommendation tables ──► Serving API ──► "Because you watched…"

This is essentially a Lambda architecture: a speed layer that gives fast approximate answers and a batch layer that recomputes exact answers from the full history. I'll come back to whether that's the right choice in the trade-offs section.

4. Storage layer, part 1: HDFS for the raw event lake

Every raw event ReelStream receives is kept. Storage is cheap; being unable to recompute a metric after you discover a bug is expensive. HDFS is the long-term home for this data.

How HDFS is organised

• NameNode: the master. It holds the filesystem namespace (directories, file names, permissions) and the mapping of each file to its blocks and of each block to the DataNodes holding it. It does not store the data itself. All of this metadata sits in memory, which is why HDFS hates millions of tiny files.
• DataNodes: the workers that store the actual blocks on local disks. They send heartbeats and block reports to the NameNode every few seconds.
• Secondary NameNode: despite the name, it is not a hot standby. It periodically merges the NameNode's edit log into the fsimage (a checkpoint) so restarts are fast. For real failover you run an HA pair with a Standby NameNode and JournalNodes.
• Blocks: files are split into 128 MB blocks (default), and each block is replicated 3 times.
• Rack awareness: with default placement, the first replica goes on the writer's node, the second on a node in a different rack, the third on another node in that same remote rack. You survive a whole rack dying without paying cross-rack bandwidth for every replica.

How ReelStream lays out data in HDFS

Two decisions here matter more than they look:

1. Partition by date and hour. A query about yesterday's Hindi originals reads one day's folders, not three years of data.
2. Compact into large Parquet files. The streaming job writes small files every minute. A nightly compaction job rewrites them into files close to the 128 MB block size. Parquet's columnar compression typically shrinks this kind of JSON by several times, so the 2–3 TB/day raw estimate becomes a few hundred GB on disk per replica.

The write path, step by step

When the compaction job writes a file:

1. The client asks the NameNode to create the file. The NameNode checks permissions and records it.
2. For each block, the NameNode returns a list of three DataNodes (rack-aware).
3. The client streams data to the first DataNode, which forwards to the second, which forwards to the third. This is the replication pipeline.
4. Acknowledgements flow back along the pipeline. Once all blocks are acknowledged, the client closes the file and the NameNode commits it.

5. Storage layer, part 2: NoSQL for things that must be fast

HDFS is great for scanning terabytes. It is terrible for "what's the resume position for user u_48213907 on title t_10442?" — a question the app asks every time someone opens it. Different access patterns get different stores. The four NoSQL families each earn a spot:

CAP theorem: choosing per feature, not per company

The CAP theorem says that when a network partition happens, a distributed store must choose between Consistency (every read sees the latest write) and Availability (every request gets a non-error response). Since partitions will happen, the real choice is C or A during one.

The interesting part for OTT is that different features want different answers:

• "Continue watching" row → AP. If a partition means you see a resume position that's 30 seconds stale, nobody cares. If the row fails to load, people do. Cassandra with consistency_level = ONE fits.
• Subscription and entitlement checks → CP. If someone just cancelled, or just paid, you'd rather return an error for a second than stream premium content to a non-paying account or block a paying one. This lives in a strongly consistent store (an RDBMS or HBase).
• Live view counters → AP, approximately. A "1.2M watching now" badge that's off by 0.5% is fine.

Cassandra also lets you tune this per query. With replication factor 3, reading and writing at QUORUM (2 of 3) guarantees R + W > N, so reads see the latest write. You pay for that with higher latency and lower availability, so ReelStream only uses QUORUM where it matters.

6. Batch processing with MapReduce

Every morning, the content team wants yesterday's plays per title and per genre. MapReduce is Word Count with a costume on.

The three phases: Map emits (content_id, 1) for every play_start. Shuffle groups by key. Reduce sums up.

Pass reducer.py as the combiner too — each mapper pre-aggregates locally before the shuffle, cutting network traffic dramatically. Addition is commutative and associative, so this is safe.

Genre join uses a reduce-side join: tag catalog rows with C and play-count rows with P, key both by content_id, and the reducer reassembles them. If the catalog fits in memory (~30K titles, a few MB), a map-side broadcast join is faster — no shuffle needed.

7. Spark: the same jobs, faster, plus streaming

MapReduce writes every intermediate result to HDFS. Spark keeps data in memory across stages — our three-job genre pipeline collapses into one program.

Key concepts. The driver builds a DAG; the cluster manager hands it executors on worker nodes. Nothing runs until you call an action. The DAG splits into stages at shuffle boundaries. Narrow transformations (map, filter) pipeline with no network I/O; wide ones (reduceByKey, join) trigger a shuffle and open a new stage.

8. Stream mining: approximate answers at scale

Some questions can't be answered exactly in a stream without storing everything. Three algorithms handle the common cases.

8.1 Bloom filter — "already watched?"

A bit array of m bits + k hash functions. Insert: set k bit positions. Check: any bit = 0 → definitely not seen; all 1s → probably seen. No false negatives; false-positive rate is tunable.

For 10M pairs at 1% FPR → m ≈ 12 MB, k = 7. An exact Python set would be ~1 GB.

Context matters: a false positive in recommendation filtering skips one valid title — harmless. In royalty-per-view deduplication it silently drops a real event — use exact dedupe in the batch layer instead.

8.2 DGIM — rebuffer count in a sliding window

Summarises a bit stream with buckets: each has a timestamp (of its latest 1) and a size (power of 2). Rules: ≤2 buckets per size; sizes non-decreasing toward the past; merge two oldest when a third of the same size appears. Estimate = sum of all sizes − half the oldest bucket.

Memory: O(log² N) vs. N bits exact. Worst-case error ≤ 50% (allow r buckets per size for ~`1/(r−1)` error).

8.3 Flajolet-Martin — distinct viewers per title

Intuition: hash each user ID and count trailing zeros. Half of hashes end in ≥1 zero, a quarter in ≥2, so the maximum trailing-zero count R implies ~`2^Rdistinct items. Duplicates hash identically — they don't moveR`.

Use 64 hash functions; average R; estimate 2^(mean R) / φ (φ ≈ 0.7735). Memory: ~256 bytes per title, regardless of audience size.

In production use HyperLogLog (approx_count_distinct in Spark): one hash, m registers, ~1–2% error. Sketches across partitions merge by element-wise max — no user IDs travel over the network.

8.4 Exact vs. approximate at a glance

9. From telemetry to taste: similarity metrics

Two titles are similar if the same people watch them. Pick the metric that fits your data shape.

Jaccard (sets) — did someone watch it or not?

J(A,B) = |A ∩ B| / |A ∪ B|

Asha = {Monsoon Files, Hostel Diaries, Chai Pe Charcha, Ward No. 9}, Rahul = {Hostel Diaries, Chai Pe Charcha, Night Shift} → J = 2/5 = 0.4.

Cosine (vectors) — how many minutes per genre?

cos(A,B) = (A·B) / (‖A‖ ‖B‖)

Asha = (120,30,0,60), Rahul = (90,0,10,45) → cos ≈ 0.97. Cosine compares direction, not volume — heavy and light viewers with the same taste still match.

At Spark scale — generate pairs only from co-watched titles; |A∪B| = |A|+|B|−|A∩B| avoids building unions; filter below 20 co-viewers to suppress noisy niche pairs:

Heads-up: users with 500+ titles explode the self-join. Cap at 100 recent titles or use MinHash + LSH (spark.ml.feature.MinHashLSH).

10. Understanding the audience: CURE clustering

Recommendations work per user; the business thinks in segments (night-owl bingers, live-sports loyalists, commute snackers). Clustering gives them that.

Why not k-means? Viewing clusters are rarely spherical, and k-means is wrecked by outliers (a test account at 22 hours/day).

CURE (Clustering Using REpresentatives) handles non-spherical clusters and outliers in five steps:

1. Sample ~200K users (fits in memory).
2. Hierarchically cluster the sample, merging by closest representative points.
3. Pick c scattered reps per cluster — they capture shape, not just the centroid.
4. Shrink reps toward the centroid: rep += α·(centroid−rep). Dampens edge outliers.
5. Assign all users to the nearest rep — embarrassingly parallel Spark map.

Illustrative segments:

11. Putting it together: one viewer, end to end

Asha opens the app at 11:40 pm:

1. app_open → Ingest API → Kafka (partitioned by user_id).
2. Speed layer (seconds): Spark updates trending windows, HLL sketch, DGIM rebuffer bucket for her CDN edge. Redis counters tick.
3. Home screen (milliseconds): Serving reads resume positions (Redis) + recent history (Cassandra), fetches candidates from the nightly Jaccard similarity table, re-ranks by her CURE segment ("night-owl binger" → full seasons up after 10 pm), filters finished titles via Bloom filter.
4. During playback: heartbeats every 10 s. DGIM spike → ops alert → traffic steered to another edge.
5. Overnight batch: all events in HDFS as Parquet. Spark recomputes exact plays, genre joins, similarity tables, CURE segments — correcting anything the speed layer approximated or dropped as late.

12. Trade-offs worth arguing about

A PostgreSQL + nightly job beats this architecture below ~50K users. Big Data tooling pays off only when volume and velocity demand it.

Lambda vs. Kappa: Lambda duplicates logic in streaming and batch. Kappa streams everything and replays Kafka for reprocessing. Spark makes the same DataFrame code work in both modes, so Kappa is attractive — but ReelStream keeps batch for exact royalty numbers and heavy iterative jobs (similarity, CURE).

MapReduce vs. Spark: MapReduce is predictable on huge one-pass jobs; Spark wins on anything iterative or multi-stage. New systems: write in Spark, but understand MapReduce first — it's what makes Spark's stages and shuffles make sense.

13. Closing thoughts

None of these techniques are exotic alone — a Bloom filter is a bit array and a few hashes; DGIM is a bookkeeping rule; Jaccard is a fraction. What makes an OTT analytics system work is picking the right tool for each question and knowing exactly how wrong each approximation is allowed to be.

Try it: generate a synthetic event stream in Python, push to local Kafka, run the Structured Streaming job from §7, and compare DGIM and FM estimates against exact counts as the stream grows. Watching approximation error stay flat while memory stays tiny is the moment these algorithms stop feeling like exam topics.

References

1. Leskovec, Rajaraman, Ullman — Mining of Massive Datasets, 3rd ed.
2. Dean & Ghemawat — "MapReduce," OSDI 2004
3. Shvachko et al. — "The Hadoop Distributed File System," MSST 2010
4. Zaharia et al. — "RDDs," NSDI 2012
5. Bloom — "Space/Time Trade-offs," CACM 1970
6. Datar, Gionis, Indyk, Motwani — "DGIM," SIAM 2002
7. Flajolet & Martin — "Probabilistic Counting," JCSS 1985
8. Guha, Rastogi, Shim — "CURE," ACM SIGMOD 1998
9. Gilbert & Lynch — "Brewer's Conjecture," ACM SIGACT 2002
10. Apache Spark — Structured Streaming Programming Guide

Written as part of the Big Data Analytics course. ReelStream is a fictional platform; all traffic figures are illustrative. Spot an error? Let me know in the comments.