Learn machine learning · deep dive D12 · optional

Capacity factor and token dropping

An expert has a fixed buffer, and a router that sends it more than that has to do something with the rest. This is the cost the active-parameter count in 10.1 leaves out.

Deep dive D12Harder targetAbout 45 minutesYour check: reach the stated target

This is optional. It opens once 10.2 Load balancing is done, and nothing in the track waits on it. The gate is a setting with a drop rate at or below 2 percent at no more than 1.2 times the capacity-1.0 cost, plus the four hand computations.

01 · The question

10.1 counted the parameters that run for one input and 10.2 made the router spread the batch. Both assume every input reaches an expert. On real hardware an expert is given room for a fixed number of inputs per batch, and anything beyond that room is not processed at all.

The buffer is ceil(capacity x inputs / experts). Everything else follows from that one line: how many are dropped, where they sit on the map, and what it costs to stop dropping them.

Commit first, on the balanced layer from 10.2 at a capacity factor of exactly 1.0.

02 · Predict

Before anything runs

At a capacity factor of 1.0, with the balanced router, how much of the held-out batch reaches no expert?

Nothing is dropped: the buffer is big enough for an average expert.A few percent are dropped.More than a fifth of the batch is dropped.

One line on why, if you want one

Choose something first.

03 · Try it

The arithmetic first

Four exact numbers, on a batch of 256 inputs across 8 experts. No tolerance on these: the buffer is a whole number of slots.

256 inputs, 8 experts, capacity 1.0: the buffer per expert

Attempts: 0

An expert receiving 40 inputs at capacity 1.0: how many it drops

Attempts: 0

The same layer at capacity 0.5: the buffer per expert

Attempts: 0

A perfectly uniform routing at capacity 0.5: how many of the 256 are dropped

Attempts: 0

The layer, under a buffer

Each expert fills its buffer in order of router score, strongest first. An input beaten to the last slot is not a worse input, it is a later one, and that is worth remembering when a drop rate is quoted without an ordering rule beside it.

The buffer, one setting at a time

trained offline by the course scriptrecord-runs.mjs · seed 7 · capacity-factor-and-token-dropping.json

Loading the two recorded routers.

The reading

One reading to choose

What happens to the drop rate when the same buffer meets the collapsed router from 10.2?

More inputs are dropped, because the collapsed router sends most of them to a few experts.Fewer inputs are dropped, because the buffers of the unused experts stay empty.The same number are dropped, because the buffer is the same size either way.

Choose one before locking it in.

Ask for a hint

Rungs open as you run the session, not as time passes. You have run it 0 times.

1. A question: available after 3 evaluated runs.

2. The answer: available after 5 evaluated runs.

04 · Compare

05 · Check your understanding

Your check: reach the stated target

not yet

On the balanced router: a drop rate at or below 2 percent with operations at most 1.2 times the capacity-1.0 cost. The four hand computations have to be right as well.

Not met yet. Keep working the session.

0 of 4 hand computations checked. No setting has cleared both numbers yet.

Save your takeaway

Your takeaway

Edit it into your own words, or keep the line as it is.

back to the track

10.2 Load balancing

Open the notebook