Model fit: the statistics of suits (20 December 2024)

What is model fit?

Much of statistical analysis involves ``fitting models to data.'' This means trying to create an algebraic formula that describes relationships between variables in the dataset. There is usually one variable that you are trying to predict or explain using other variables. That one is called the ``response,'' and the others are called ``predictors.''

But we want our equation to do more than just explain the relationship between our response and our predictors in our dataset. We want that equation to also, hopefully, explain that relationship in the broader population---out in the big wide world, in subjects like those in our sample, but who were not actually in it---those we did not recruit into our study.

A famous statistician, George Box, said, ``All models are wrong; some are useful.'' Every model is just an approximation of reality in the world. Some approximations are better and some are worse; some fit better and some fit worse. By ``fit,'' I mean how closely the model's predictions for the observed response values in the dataset are to the actual measured values. Because we have those measured values---got them right there on paper or in our datafile on the computer. So we can check them against what the model predicts. We can ask OK, here is the value of the response variable that our model predicts for item 12. Here is item 12's actual, measured value for the response variable. Now, how different are they? This difference is called the residual for item 12. And we can do this for all the items in the dataset; each of our subjects has a residual for this model. Residuals are numbers. There are various techniques you can apply to the residuals, collectively, to assess the ``goodness of fit.''

Some natural phenomena have very little intrinsic variability, so it is possible to build models of them with extremely good fit. Here is a model of how long a ball bearing takes to hit the ground when dropped from various heights:

This model fits extremely well. Tell me the height from which you plan to drop the ball bearing, and I can tell you very precisely how long it will take to hit the ground. As long as we measure time precisely, the residuals from many trials of dropping different ball bearings from different heights will all be essentially zero.

There is much more variability in biological and social systems, so model fit is never as good as it is with ball bearings. There may be multiple candidate models, each with a certain degree of fit. We generally want our conclusions, decisions, and actions to be based on the best model.

The fit of a suit

"Model fit" is one aspect of choosing the "best" model. But fit is a double-edged sword. Think of buying a suit. You could buy a custom-tailored suit that fits you exactly, both size and style (maybe you like double-breasted suits, for example.) That suit will serve your needs very well, but it probably won't serve other people's needs. It may not fit them. They may prefer single-breasted. It fits great in the sample (you) but not so well in the population (everyone else). So if you are a suit manufacturer, you may decide just to make suits in a small variety of standardized sizes and styles. They might not fit every customer's size and style exactly, but the fit will be good enough that customers will still buy them. So you sacrifice a little goodness-of-fit for some generalizability.

Your suit company might also sacrifice some goodness of fit for some utility. Say cashmere makes the best-fitting suits. But it is a fragile and persnickety fabric that is difficult for customers to care for at home, so they won't buy them. In statistics, we might call this ``interpretability.'' A fantastic fitting model (statistically) might be difficult for users of that model to interpret and apply to their own circumstances. If some model does a great job predicting the cube root of cholesterol level, what's a physician supposed to do with that?

Why this matters

Statisticians have a number of ways of assessing model fit. Some methods produce a number. Some methods produce graphs. Sometimes the methods agree with each other, sometimes they point in slightly different directions. Bottom line: there is usually judgement involved in building and choosing models of biopsychosocial phenomena. Nevertheless, the fit of a statistical model should always be assessed before drawing conclusions from it. Only draw conclusions from adequately-fitting models.