All topics
84

The 10 biggest out of a billion, without sorting a billion

Tier 1Locked

Where it runs

1. The problem

A stream of events goes past: search queries, errors from logs, player scores. You need the 10 biggest values. The rest can be thrown away, and the order inside the top 10 does not matter.

The first idea is to collect everything, sort it and take the first 10. That works while the data fits in memory. But sorting a billion rows to keep 10 of them is wasteful. And a live stream cannot be held in memory at all.

At any moment you need only 10 values. Keep those 10, and let every new value try to take the place of one of them. A small heap does exactly that.

This step is part of the paid track

The first 33 steps are open to everyone: lesson, exercise and tests. This one opens with full access. Start with a free account and the free steps.

Create a free accountSee what is included

Recognise it

Binary heap / priority queue

Say these words and it is this pattern

  • always need the current smallest/largest
  • top k elements
  • merge k sorted lists
  • running median while items keep arriving
  • priority queue / most urgent next

Looks similar, isn't

n up to 1e6 gives O(log n) per push or pop. Reading the top alone is O(1).

On LeetCode

Problems that ask for the k largest, the k most frequent or the kth largest item. Keep a min-heap capped at k, so its root is the weakest item kept and the cost is O(n log k).

We use necessary cookies to run the site (sign-in and language). If you accept, we also load Google Analytics to see which pages are used, and Google reCAPTCHA to keep spam off the contact and bug-report forms. Privacy policy