Value of a test set

The more you use a test set, the less valuable it becomes. In some cases, we can measure this value by something like information gain i.e how much information you have gained (in bits) from modifying your model using your test set. As a result, the best test set is only used once.

Written on June 20, 2026