Cross-validation and uncertainty of scores

A real-world problem

A team is comparing two approaches and must avoid selecting the one that merely looks good on a convenient example. Cross-validation and uncertainty of scores provides a reproducible way to compare outputs, expose failure cases, and explain what the measured score does and does not mean. The important question is not whether the first demonstration works; it is whether the same reasoning survives a boundary input, a changed requirement, or a partial failure. Keep that question visible throughout the lesson.

Learning outcomes

By the end of this lesson, you should be able to:

• Define the central concept and explain the contract it provides.
• Trace a worked example, including intermediate values or state transitions.
• Identify a boundary case and a likely failure mode.
• Select a technique for a realistic task and justify the choice with evidence.

The mental model

Keep the target, features, train/validation/test split, baseline, loss, metric, and deployment decision explicit. A score is meaningful only with a dataset, class balance, threshold, and error cost. For Cross-validation and uncertainty of scores, write down the inputs, output contract, and invariant that must remain true after each meaningful step.

For Cross-validation and uncertainty of scores, separate three questions. First, what are the allowed inputs and the promised output? Second, what mechanism transforms one into the other? Third, what evidence would show that the result is correct? A definition without a contract is difficult to apply; a code snippet without an expected result is difficult to test; and a measurement without a stated workload is difficult to compare.

Write down the preconditions before running an example. If the method depends on ordering, concurrency, randomness, class balance, a protocol version, or a particular runtime, state that dependency explicitly. This habit prevents an example from being misread as a universal rule.

Instructor walkthrough: worked example

Fit imputation, scaling, and feature selection using training data only, then apply learned transformations to validation and test data. If preprocessing learns from the test set, information leaks across the evaluation boundary and the reported score becomes optimistic.

Implementation or calculation

Input -> validate -> apply the rule -> inspect result

Predict the result before executing the code or finishing the calculation. Record the input, expected output, intermediate state, and the rule used at each step. Then compare the observed result with the prediction. If they disagree, find the earliest divergent step rather than changing several things at once.

Visual model

The diagram below shows the main dependency or state transition for this lesson. Read it from the starting state to the outcome. When studying, annotate the nodes with values from the worked example; when debugging, mark the first edge where actual behaviour differs from expected behaviour.

Failure analysis and trade-offs

For Cross-validation and uncertainty of scores, do not treat one successful run as a general guarantee. State version and input assumptions, test a boundary case, and verify the failure path.

A second review asks whether the assumptions of the worked example hold in the target environment. A technique may be correct only for sorted input, non-negative weights, a specific isolation level, a clean training split, a current request, or an approved tool. Name the dependency and design a test that makes it visible.

When several approaches are valid, compare them against the same workload. Consider correctness, time and space cost, maintainability, security, accessibility, and failure recovery where relevant. There is rarely a universally best solution. A strong engineering decision identifies the constraint that matters most, explains the trade-off, and states when the decision should be revisited.

Practice exercise: mini-lab

Build a small experiment around Cross-validation and uncertainty of scores using the opening scenario.

1. Baseline: state the expected behaviour and the input or environment needed to reproduce it.
2. Normal case: run one representative input and capture the output, trace, or calculation.
3. Boundary case: change one important variable, size, ordering, timing, missing data, or a limiting value, and predict the result before testing.
4. Failure case: introduce an invalid input or unavailable dependency. Verify that the system fails safely and does not silently report success.
5. Reflection: explain which assumption was tested, what the evidence supports, and what remains uncertain.

Keep the experiment small enough for another student to reproduce. Include the environment and version, commands or data, expected and observed output, and a short conclusion. For code, add a test that fails before the fix and passes afterwards. For mathematics, show each substitution and preserve units. For systems work, include a state trace, log, or packet evidence instead of only a screenshot of the final screen.

Check your understanding

1. Apply the core idea. Why must a scaler be fitted on the training split only?

Model answer: Using validation or test data to estimate scaling parameters leaks information into model fitting.

2. Explain a limitation. Which precondition would make your solution invalid, and what evidence would reveal that the precondition is not met?

Model answer: Identify a concrete condition such as unsorted input, a negative edge weight, a missing foreign key, a stale asynchronous response, data leakage, or an untrusted tool result. Describe the observable failure and the safe response expected from the system.

3. Evaluate the evidence. What is the difference between one successful demonstration and a reliable conclusion?

Model answer: One demonstration shows only that one path worked. A reliable conclusion combines a correct model or proof with representative normal, boundary, and failure tests. Record assumptions and versions, compare against a baseline, and state the limits of the evidence.

Transfer challenge

Adapt the worked example to a changed requirement. Preserve the original invariant, introduce one new constraint, and explain how the solution changes. Add one negative test and one sentence describing what the test cannot prove. If the original approach is unsuitable, justify a different choice using the workload and constraints rather than personal preference.

Further reading and source notes

Start with the machine learning reference material to verify terminology, version-specific behaviour, and edge cases. Use primary specifications or official documentation for guarantees. In submitted work, cite the exact page or section used, distinguish a normative specification from a tutorial, and do not cite a source for a claim it does not support.