Precision, recall, and F1 trade-offs

A real-world problem

Consider this real situation from machine learning: A support team predicts which tickets need urgent attention. A model may score well on familiar examples but fail on new tickets if data leaks across splits or the metric hides costly false negatives. The team must apply Precision, recall, and F1 trade-offs to the actual constraint, not merely repeat a definition. This lesson makes assumptions visible, traces an example, and tests a boundary where a shallow explanation would fail. The important question is not whether the first demonstration works; it is whether the same reasoning survives a boundary input, a changed requirement, or a partial failure. Keep that question visible throughout the lesson.

Learning outcomes

By the end of this lesson, you should be able to:

• Define the central concept and explain the contract it provides.
• Trace a worked example, including intermediate values or state transitions.
• Identify a boundary case and a likely failure mode.
• Select a technique for a realistic task and justify the choice with evidence.

The mental model

Keep the target, features, train/validation/test split, baseline, loss, metric, and deployment decision explicit. A score is meaningful only with a dataset, class balance, threshold, and error cost. For Precision, recall, and F1 trade-offs, write down the inputs, output contract, and invariant that must remain true after each meaningful step.

For Precision, recall, and F1 trade-offs, separate three questions. First, what are the allowed inputs and the promised output? Second, what mechanism transforms one into the other? Third, what evidence would show that the result is correct? A definition without a contract is difficult to apply; a code snippet without an expected result is difficult to test; and a measurement without a stated workload is difficult to compare.

Write down the preconditions before running an example. If the method depends on ordering, concurrency, randomness, class balance, a protocol version, or a particular runtime, state that dependency explicitly. This habit prevents an example from being misread as a universal rule.

Instructor walkthrough: worked example

For TP=42, FP=8, FN=6, TN=44, precision=42/50=0.84 and recall=42/48=0.875. Raising the threshold usually reduces predicted positives, often improving precision while reducing recall; choose based on error costs.

Implementation or calculation

Input -> validate -> apply the rule -> inspect result

Predict the result before executing the code or finishing the calculation. Record the input, expected output, intermediate state, and the rule used at each step. Then compare the observed result with the prediction. If they disagree, find the earliest divergent step rather than changing several things at once.

Visual model

The diagram below shows the main dependency or state transition for this lesson. Read it from the starting state to the outcome. When studying, annotate the nodes with values from the worked example; when debugging, mark the first edge where actual behaviour differs from expected behaviour.

Failure analysis and trade-offs

For Precision, recall, and F1 trade-offs, do not treat one successful run as a general guarantee. State version and input assumptions, test a boundary case, and verify the failure path.

A second review asks whether the assumptions of the worked example hold in the target environment. A technique may be correct only for sorted input, non-negative weights, a specific isolation level, a clean training split, a current request, or an approved tool. Name the dependency and design a test that makes it visible.

When several approaches are valid, compare them against the same workload. Consider correctness, time and space cost, maintainability, security, accessibility, and failure recovery where relevant. There is rarely a universally best solution. A strong engineering decision identifies the constraint that matters most, explains the trade-off, and states when the decision should be revisited.

Practice exercise: mini-lab

Build a small experiment around Precision, recall, and F1 trade-offs using the opening scenario.

1. Baseline: state the expected behaviour and the input or environment needed to reproduce it.
2. Normal case: run one representative input and capture the output, trace, or calculation.
3. Boundary case: change one important variable, size, ordering, timing, missing data, or a limiting value, and predict the result before testing.
4. Failure case: introduce an invalid input or unavailable dependency. Verify that the system fails safely and does not silently report success.
5. Reflection: explain which assumption was tested, what the evidence supports, and what remains uncertain.

Keep the experiment small enough for another student to reproduce. Include the environment and version, commands or data, expected and observed output, and a short conclusion. For code, add a test that fails before the fix and passes afterwards. For mathematics, show each substitution and preserve units. For systems work, include a state trace, log, or packet evidence instead of only a screenshot of the final screen.

Check your understanding

1. Apply the core idea. If false negatives are very expensive, which metric needs special attention?

Model answer: Recall, because it measures the share of actual positive cases the model identifies. Also inspect the confusion matrix and threshold.

2. Explain a limitation. Which precondition would make your solution invalid, and what evidence would reveal that the precondition is not met?

Model answer: Identify a concrete condition such as unsorted input, a negative edge weight, a missing foreign key, a stale asynchronous response, data leakage, or an untrusted tool result. Describe the observable failure and the safe response expected from the system.

3. Evaluate the evidence. What is the difference between one successful demonstration and a reliable conclusion?

Model answer: One demonstration shows only that one path worked. A reliable conclusion combines a correct model or proof with representative normal, boundary, and failure tests. Record assumptions and versions, compare against a baseline, and state the limits of the evidence.

Transfer challenge

Adapt the worked example to a changed requirement. Preserve the original invariant, introduce one new constraint, and explain how the solution changes. Add one negative test and one sentence describing what the test cannot prove. If the original approach is unsuitable, justify a different choice using the workload and constraints rather than personal preference.

Further reading and source notes

Start with the machine learning reference material to verify terminology, version-specific behaviour, and edge cases. Use primary specifications or official documentation for guarantees. In submitted work, cite the exact page or section used, distinguish a normative specification from a tutorial, and do not cite a source for a claim it does not support.