How a machine learning model is built and where it fails
A machine learning model is not given its function in code but learns it from data. What its reported performance means depends on how that learning was done: on the supervised training pipeline that runs from raw data to deployment, and on the points along it where bias, leakage, overfitting, a misleading metric or a changing world leave a performance figure meaning less than it seems.
Terms
Machine learning is an AI technique that teaches computers to learn from data. As a sub-domain of AI, it is the study of algorithms that can educate and improve themselves by analysing an input, usually a large amount of relevant data. Unlike traditional software, most machine learning does not use explicit or rule-based programming. Explicit programming gives the computer the function as lines of code, whereas machine learning uses statistical and mathematical modelling and large volumes of data to learn the relationships between variables, that is, it determines the function, and the model learns and refines itself as it ingests and processes data. An algorithm is a set of instructions, and in machine learning it learns and evolves without human intervention from the data it processes. It builds on common models such as linear and logistic regression, Bayesian algorithms and decision trees. The terms algorithm, model and tool are sometimes used interchangeably.
Deep learning is a kind of machine learning that uses artificial neural networks: layers of connected nodes, modelled on the brain and designed to emulate how humans process information. Each layer transforms information and passes it to the next. The term deep refers to the number of layers through which the model operates. The model can learn from itself to create new features. Ground truth information is fed back into it to enable this, ground truthing being the check of machine learning results for accuracy against the real world. Facial recognition is an example, and deep learning needs vast resources. It can learn nonlinear, high-dimensional relationships from data that are unstructured and multimodal, such as imaging, biometrics, audio, visual and time-series data. Because it creates its own algorithms, its models can lack transparency.
Kinds of machine learning
Three common kinds of machine learning model are supervised, unsupervised and reinforcement learning. Supervised learning trains on labelled data that pairs inputs with outputs; each pair is called an example, and a collection of examples is a data set. An application might, for example, predict from the size of a specific type of tumour whether it is likely to be malignant or benign. The data set is divided into training, validation and testing sets, and the share given to each depends on biostatistical calculations, which can resemble the sample size calculations of a clinical trial. The sets need to represent the population for the resulting model to generalise. Unsupervised learning uses unlabelled data in which the computer looks for patterns, as with images and pathology data, and segmentation, used for clustering, is one application. Reinforcement learning learns by reward and penalty.
What made machine learning practical and what holds it back
Recent developments that catalysed the development of machine learning models include big data, which provides the necessary data and resources, cloud storage for the large amounts of data that model development requires, powerful computing, since machine learning can be computationally intensive, parallel processing, which lets a model be deployed across multiple processors, and the maturation of the statistical and mathematical methods that underlie machine learning. The volume of medical data doubles every 8 to 12 months, which requires a lot of storage. Creating a machine learning model requires data and computing resources, and creating the complexity a valid model needs can require extensive resources. Trust that the model will make correct decisions for the benefit of the patient is also an issue, so data and privacy security need to be robust. Strong data and privacy security can help build trust in the model.
The supervised learning pipeline
Supervised learning resembles the way a child learns from a picture book which sound each animal makes. Learning starts from a problem to solve: given an observation or a measurement, what conclusion should be drawn from it? The child makes mistakes and is corrected (no, that is a dog and it says woof, not a cat saying miaow), and with continued feedback in the end gets everything right. It is important not to stop there but to check that the child can generalise: shown an image that does not look like the book's drawing, does it still recognise a dog, or did it remember the order in which the animals appear? A machine learning algorithm, in the same way, must be made to generalise to unseen data. The same loop can be pictured as a student doctor who learns how symptoms and measurements such as blood pressure relate to a condition, draws conclusions, sometimes draws the wrong one, sees the mistake and adjusts the model of the task held in their head.
In training, the model is fed input data and at first makes many mistakes. Its output is compared with the expected output, the distance between them is measured, and its parameters are tweaked to reduce the error a little. The process iterates, like reading the book again and again, until the model gets everything right.
In more detail, though still simplified, the process of machine learning training begins with raw data. Features may need to be extracted or re-engineered so that the data perform better for the specific task, and features not relevant to the problem may be excluded, since not every piece of information, such as a patient's name or weight, bears on every disease a model might be trained to diagnose. The result is pre-processed data holding only the wanted features, ideally without noise and without fault. Labelling follows if it has not already happened, recording for example that a patient had a given disease, and it produces the labelled data a supervised approach needs.
A model then has to be trained, and which exact model will be best for the task is not known in advance. Neural networks may seem the best choice, but they are not always; and once one is chosen, how deep and how wide to make it and how fast to train it still have to be decided. All of these hyperparameters, of the architecture and of the training process, can be tweaked and give different final accuracy. It is important to evaluate the candidate models on unseen data, to make sure the chosen one can generalise to data it was not trained on. The one that performs best on the validation data is trained completely, and its final evaluation runs on a further set used neither in training nor in validation, since otherwise test data leak into the training process. The model is then checked against its requirements, which go beyond accuracy: whether it can run on the available device, whether it needs too much computing power, and whether it is fast enough. An hour's wait for a result is unacceptable in some cases, and a couple of minutes in others. A model that meets its requirements is accepted and deployed.
What can go wrong at the steps of this pipeline applies generally to all uses of AI and to data analysis in general.
Bias in the training data
Bias can come in at any stage of training, which is why data governance is important.
Raw data are the first place it goes wrong: if the data are garbage, incorrect or full of noise, the model will be flawed. Data collected entirely in Europe for a system to be marketed to the whole world leave a large bias between the population the system was designed on and the population it will be used on. Because algorithms learn from the data set they are given and may not be able to detect the biases within it, bias can arise particularly where data from a particular group are limited. Certain demographics may then be underrepresented, and results may not be fair, calling the reliability of the data into question.
Garbage in, garbage out: output is only as good as input. Large language models are pre-trained on the internet, and most internet data are what is called WEIRD (western, educated, industrialised, rich and democratic), a bias nothing can be done about. Small language models are smaller models pre-trained on high-quality, well-crafted and curated data rather than giant ones, which highlights the importance of data quality in AI.
A recruiting model trained on all the CVs a company has received before, labelled with who was hired and who was not, will make similar judgement calls in future. If the recruiters behind that history were biased, against gender or age for example, the model will be biased in the same way.
Removing all data on a person's gender, name, ethnicity or age does not settle it. The model can still become biased through the wording someone uses in a CV or the language they use, something not necessarily thought of at once. Bias comes from humans: a model is not biased in itself, and it is the way it is trained or used that puts bias into it. It is very important that bias is tackled during development.
Leakage from the test set
A pre-processing pipeline should be designed on training data only, because test data that leak into it lead to an overestimate of the final model's quality and accuracy. Such pre-processing includes extracting specific features and rescaling inputs, and examples where test information must not leak in are principal component analysis and min-max standardisation.
Bias and disagreement in labels
Humans label the data, and humans are inherently biased and can introduce bias consciously or subconsciously. Data given the wrong label lead the model to make wrong decisions. Expert labellers also do not necessarily agree with each other, for example on which symptoms point to a diagnosis of autism. Where human experts disagree, the AI cannot be expected to agree either, because that inherent uncertainty sits in the training data.
Overfitting
A model that is too complex or too powerful for the simplicity of its task overfits: it learns the training data itself, such as which patient ID goes with which diagnosis or the order in which the answers come, and then performs worse than expected on a new population. Validation data are used to catch this. In training a neural network, the training error and the validation error can be followed separately, and early stopping is a technique sometimes used to tell when the model starts to overfit. The training error keeps falling while the validation error starts to rise, and from that point the model is learning noise that carries no information about the problem.
Even with validation data, testing many models against the same validation set, which is always limited in size, can give lower errors for particular models while very possibly overfitting that validation set as well. The European AI Act puts a lot of pressure towards benchmarks against which the models of different providers can be compared. If every provider tests against the same benchmark data set, society as a whole starts to overfit to it, and if that data set is biased, so are the models, a possible issue in the future that in 2024 was already an issue.
A misleading accuracy figure
Take a test population that is 90% healthy and 10% sick, and a model trained to say whether each person is healthy or sick that scores 90% accuracy, a figure most people would accept. Yet a model that always says the person is healthy gets that score, since it cannot score below 90% on that population. For medical devices and in vitro diagnostics, true positives and false negatives have to be seen, so more advanced metrics, and more than a single one, are needed to show how accurate a model is. Precision and recall are examples that combine the false positives and negatives to show better how the model works. Many more metrics exist, and this is only the simple case of binary classification; portraying how a model performs in real life calls for much more complex evaluations.
Loss of performance over time
A model put on the market is used fixed, while society changes. A model trained on old data that does not adapt to new data therefore loses performance, which is to be expected. Models degrade over time and need to be retrained, which in one practitioner's experience is a concern some application developers tend to neglect. Continuous learning can be used against this: where that setting is on, it feeds back into the model and keeps updating it so that it stays on track with the current real-world situation. A model that keeps receiving input from one specific environment, however, becomes biased towards that environment.