Classification and Evaluation: Defects, Failures and Grades

Classification problems in engineering

A classifier assigns an input to one of several classes. Quality inspection separates good from defective parts, condition monitoring distinguishes fault modes, food engineering grades products, and geology assigns facies to depth intervals of a well. A classifier divides the input space into regions separated by decision boundaries. Most modern classifiers do more than name a class: They estimate a probability or a score for each class, and a decision rule, usually a threshold, turns the score into an action. Separating the model from the decision is essential in engineering, because the costs of different mistakes are rarely equal.

Models for classification

Logistic regression models the log-odds of a class as a linear function of the inputs, so its decision boundary is a straight line or a plane, and its coefficients show the direction in which each input pushes the decision [1]. The k-nearest-neighbour classifier takes a vote among the most similar training examples [2]. Support vector machines find the boundary with the largest margin between classes and use kernels to draw curved boundaries in the original space [3]. Decision trees ask a sequence of threshold questions and are easy to read [4, 5]; random forests and gradient boosting combine many trees and are among the strongest methods for tabular engineering data [6, 7].

The confusion matrix and its metrics

For a binary problem with a positive class, such as "failure", the confusion matrix counts true positives (TP), false positives (FP), false negatives (FN) and true negatives (TN). Accuracy, the share of correct decisions, is misleading when classes are imbalanced: If 3 percent of machines fail, a model that always predicts "no failure" is 97 percent accurate and useless. Precision, TP / (TP + FP), is the share of alarms that are real. Recall, TP / (TP + FN), is the share of real failures that are caught. The F1 score is their harmonic mean. For several classes, the metrics are computed per class and averaged, either equally for every class (macro average) or weighted by class size (weighted average); the macro average reveals poor performance on small classes.

Curves, thresholds and costs

Varying the threshold traces curves. The ROC curve plots the true positive rate against the false positive rate, and the area under it (AUC) equals the probability that a random positive example receives a higher score than a random negative one [8]. When positives are rare, the precision-recall curve is more informative, because the false positive rate can look small while alarms are still mostly false [9].

The threshold should come from the application. If a missed bearing failure costs twenty times more than an unnecessary inspection, the expected cost for each threshold on validation data can be computed, and the threshold with the lowest cost chosen. The default of 0.5 is only correct when both errors cost the same and the probabilities are well calibrated.

Imbalanced data

Rare events are the rule in engineering: failures, defects, accidents, rock bursts. Several remedies exist. Stratified splits keep the class proportions equal in training and test data. Class weights make errors on the rare class more expensive during training. Resampling duplicates rare examples or removes common ones, and SMOTE creates synthetic minority examples by interpolating between neighbouring minority samples [10]. None of these remedies creates information; the most effective step is often better features, more minority data or a problem definition that matches the decision.

Explaining classifiers

Engineers need to know why a model decides. Permutation importance measures how much a performance metric drops when the values of one feature are shuffled, which breaks its link to the target. SHAP values distribute a single prediction among the features according to a game-theoretic rule and are widely used for tree ensembles [11]. In predictive maintenance, explanations connect model output to physical mechanisms such as heat dissipation or tool wear [12]. Newer methods study the geometry of a model's decision function, for example GEMEX, which the instructor developed as an open Python package [13]. Explanations are models of models and can mislead, so they should be checked against domain knowledge and, where possible, against deliberately constructed test cases.

Review cards

Select a card to turn it over.

References