Uncertainty Wednesday: When Small Numbers Matter (Aircraft Safety)

Last week’s Uncertainty Wednesday was titled “Fooled by Small Numbers” and discussed why drawing conclusions from small samples is dangerous. A reader asked how that relates to grounding the Boeing 737 Max 8 airplane after two crashes? Isn’t that also drawing a conclusion from a small sample? No. Because here we are actually looking not at a sample but at the population of all flights.

Globally there are nearly 40 million commercial flights annually. That is over 100,000 flights every day! Each and everyone one of those flights is tracked. And in an amazing accomplishment of engineering, process and public safety regulations we have had years with zero  crashes. When people complain that we are going backwards on commercial flight because we used to have supersonic flight with the Concorde, what they are ignoring is that the real demand was not for speed but for safety.

With such a large number of successful flights, a crash is a strong indicator that something went wrong. At a high level there are two possible explanations: equipment failure or operator error (of course the two can interact with each other and compound). On any single crash it would be hard to have two much of a view as to which one caused it and one might rightly suggest that grounding a whole fleet of planes is an overreaction. But two crashes with the same type of equipment point strongly at the aircraft itself.

That argument is of course further strengthened when there is significant additional evidence pointing at the plane, such as complaints by pilots along with known changes to the plane. In summary then: when something normally goes right more than 100,000 times a day every day, and then goes wrong twice in a row with the same equipment, that’s a very strong signal.