Generalized linear model

Part of a series on
Regression analysis
Models
Linear regression Simple regression Polynomial regression General linear model
Generalized linear model Discrete choice Binomial regression Binary regression Logistic regression Multinomial logit Mixed logit Probit Multinomial probit Ordered logit Ordered probit Poisson
Multilevel model Fixed effects Random effects Linear mixed-effects model Nonlinear mixed-effects model
Nonlinear regression Nonparametric Semiparametric Robust Quantile Isotonic Principal components Least angle Local Segmented
Errors-in-variables
Estimation
Least squares Linear Non-linear
Ordinary Weighted Generalized
Partial Total Non-negative Ridge regression Regularized
Least absolute deviations Iteratively reweighted Bayesian Bayesian multivariate
Background
Regression validation Mean and predicted response Errors and residuals Goodness of fit Studentized residual Gauss–Markov theorem
v t e

In statistics, a generalized linear model (GLM) is a flexible generalization of ordinary linear regression. The GLM generalizes linear regression by allowing the linear model to be related to the response variable via a link function and by allowing the magnitude of the variance of each measurement to be a function of its predicted value.

Generalized linear models were formulated by John Nelder and Robert Wedderburn as a way of unifying various other statistical models, including linear regression, logistic regression and Poisson regression.^[1] They proposed an iteratively reweighted least squares method for maximum likelihood estimation (MLE) of the model parameters. MLE remains popular and is the default method on many statistical computing packages. Other approaches, including Bayesian regression and least squares fitting to variance stabilized responses, have been developed.

Ordinary linear regression predicts the expected value of a given unknown quantity (the response variable, a random variable) as a linear combination of a set of observed values (predictors). This implies that a constant change in a predictor leads to a constant change in the response variable (i.e. a linear-response model). This is appropriate when the response variable can vary, to a good approximation, indefinitely in either direction, or more generally for any quantity that only varies by a relatively small amount compared to the variation in the predictive variables, e.g. human heights.

However, these assumptions are inappropriate for some types of response variables. For example, in cases where the response variable is expected to be always positive and varying over a wide range, constant input changes lead to geometrically (i.e. exponentially) varying, rather than constantly varying, output changes. As an example, suppose a linear prediction model learns from some data (perhaps primarily drawn from large beaches) that a 10 degree temperature decrease would lead to 1,000 fewer people visiting the beach. This model is unlikely to generalize well over differently-sized beaches. More specifically, the problem is that if the model is used to predict the new attendance with a temperature drop of 10 for a beach that regularly receives 50 beachgoers, it would predict an impossible attendance value of −950. Logically, a more realistic model would instead predict a constant rate of increased beach attendance (e.g. an increase of 10 degrees leads to a doubling in beach attendance, and a drop of 10 degrees leads to a halving in attendance). Such a model is termed an exponential-response model (or log-linear model, since the logarithm of the response is predicted to vary linearly).

Similarly, a model that predicts a probability of making a yes/no choice (a Bernoulli variable) is even less suitable as a linear-response model, since probabilities are bounded on both ends (they must be between 0 and 1). Imagine, for example, a model that predicts the likelihood of a given person going to the beach as a function of temperature. A reasonable model might predict, for example, that a change in 10 degrees makes a person two times more or less likely to go to the beach. But what does "twice as likely" mean in terms of a probability? It cannot literally mean to double the probability value (e.g. 50% becomes 100%, 75% becomes 150%, etc.). Rather, it is the odds that are doubling: from 2:1 odds, to 4:1 odds, to 8:1 odds, etc. Such a model is a log-odds or logistic model.

Generalized linear models cover all these situations by allowing for response variables that have arbitrary distributions (rather than simply normal distributions)^{[clarification needed]}, and for an arbitrary function of the response variable (the link function) to vary linearly with the predictors (rather than assuming that the response itself must vary linearly). For example, the case above of predicted number of beach attendees would typically be modeled with a Poisson distribution and a log link, while the case of predicted probability of beach attendance would typically be modelled with a Bernoulli distribution (or binomial distribution, depending on exactly how the problem is phrased) and a log-odds (or logit) link function.

In a generalized linear model (GLM), each outcome Y of the dependent variables is assumed to be generated from a particular distribution in an exponential family, a large class of probability distributions that includes the normal, binomial, Poisson and gamma distributions, among others. The conditional mean μ of the distribution depends on the independent variables X through:

${\displaystyle \mathrm{E}(\mathbf{𝐘}\mid \mathbf{𝐗})=\mathit{\mu }=g^{-1}(\mathbf{𝐗}\mathit{\beta }),}$

where E(Y | X) is the expected value of Y conditional on X; Xβ is the linear predictor, a linear combination of unknown parameters β; g is the link function.

In this framework, the variance is typically a function, V, of the mean:

${\displaystyle \mathrm{Var}(\mathbf{𝐘}\mid \mathbf{𝐗})=\mathrm{V}(g^{-1}(\mathbf{𝐗}\mathit{\beta })).}$

It is convenient if V follows from an exponential family of distributions, but it may simply be that the variance is a function of the predicted value.

The unknown parameters, β, are typically estimated with maximum likelihood, maximum quasi-likelihood, or Bayesian techniques.

The GLM consists of three elements:

1. A particular distribution for modeling ${\displaystyle Y}$ from among those which are considered exponential families of probability distributions,

2. A linear predictor ${\displaystyle \eta =X\beta }$, and

3. A link function ${\displaystyle g}$ such that ${\displaystyle \mathrm{E}(Y\mid X)=\mu =g^{-1}(\eta )}$.

An overdispersed exponential family of distributions is a generalization of an exponential family and the exponential dispersion model of distributions and includes those families of probability distributions, parameterized by ${\displaystyle \mathit{\theta }}$ and ${\displaystyle \tau }$, whose density functions f (or probability mass function, for the case of a discrete distribution) can be expressed in the form

${\displaystyle f_Y(\mathbf{𝐲}\mid \mathit{\theta },\tau )=h(\mathbf{𝐲},\tau )\exp \left(\frac{\mathbf{𝐛}(\mathit{\theta }{)}^{\mathrm{T}}\mathbf{𝐓}(\mathbf{𝐲})-A(\mathit{\theta })}{d(\tau )}\right).}$

The dispersion parameter, ${\displaystyle \tau }$, typically is known and is usually related to the variance of the distribution. The functions ${\displaystyle h(\mathbf{𝐲},\tau )}$, ${\displaystyle \mathbf{𝐛}(\mathit{\theta })}$, ${\displaystyle \mathbf{𝐓}(\mathbf{𝐲})}$, ${\displaystyle A(\mathit{\theta })}$, and ${\displaystyle d(\tau )}$ are known. Many common distributions are in this family, including the normal, exponential, gamma, Poisson, Bernoulli, and (for fixed number of trials) binomial, multinomial, and negative binomial.

For scalar ${\displaystyle \mathbf{𝐲}}$ and ${\displaystyle \mathit{\theta }}$ (denoted ${\displaystyle y}$ and ${\displaystyle \theta }$ in this case), this reduces to

${\displaystyle f_Y(y\mid \theta ,\tau )=h(y,\tau )\exp \left(\frac{b(\theta )T(y)-A(\theta )}{d(\tau )}\right).}$

${\displaystyle \mathit{\theta }}$ is related to the mean of the distribution. If ${\displaystyle \mathbf{𝐛}(\mathit{\theta })}$ is the identity function, then the distribution is said to be in canonical form (or natural form). Note that any distribution can be converted to canonical form by rewriting ${\displaystyle \mathit{\theta }}$ as ${\displaystyle {\mathit{\theta }}^{\prime }}$ and then applying the transformation ${\displaystyle \mathit{\theta }=\mathbf{𝐛}({\mathit{\theta }}^{\prime })}$. It is always possible to convert ${\displaystyle A(\mathit{\theta })}$ in terms of the new parametrization, even if ${\displaystyle \mathbf{𝐛}({\mathit{\theta }}^{\prime })}$ is not a one-to-one function; see comments in the page on exponential families.

If, in addition, ${\displaystyle \mathbf{𝐓}(\mathbf{𝐲})}$ and ${\displaystyle \mathbf{𝐛}(\mathit{\theta })}$ are the identity, then ${\displaystyle \mathit{\theta }}$ is called the canonical parameter (or natural parameter) and is related to the mean through

${\displaystyle \mathit{\mu }=\mathrm{E}(\mathbf{𝐲})={\mathrm{\nabla }}_{\mathit{\theta }}A(\mathit{\theta }).}$

For scalar ${\displaystyle \mathbf{𝐲}}$ and ${\displaystyle \mathit{\theta }}$, this reduces to

${\displaystyle \mu =\mathrm{E}(y)=A^{\prime }(\theta ).}$

Under this scenario, the variance of the distribution can be shown to be^[2]

${\displaystyle \mathrm{Var}(\mathbf{𝐲})={\displaystyle \underset{\mathit{\theta }}{\overset{2}{\mathrm{\nabla }}}}A(\mathit{\theta })d(\tau ).}$

For scalar ${\displaystyle \mathbf{𝐲}}$ and ${\displaystyle \mathit{\theta }}$, this reduces to

${\displaystyle \mathrm{Var}(y)=A^{″}(\theta )d(\tau ).}$

The linear predictor is the quantity which incorporates the information about the independent variables into the model. The symbol η (Greek "eta") denotes a linear predictor. It is related to the expected value of the data through the link function.

η is expressed as linear combinations (thus, "linear") of unknown parameters β. The coefficients of the linear combination are represented as the matrix of independent variables X. η can thus be expressed as

${\displaystyle \eta =\mathbf{𝐗}\mathit{\beta }.}$

The link function provides the relationship between the linear predictor and the mean of the distribution function. There are many commonly used link functions, and their choice is informed by several considerations. There is always a well-defined canonical link function which is derived from the exponential of the response's density function. However, in some cases it makes sense to try to match the domain of the link function to the range of the distribution function's mean, or use a non-canonical link function for algorithmic purposes, for example Bayesian probit regression.

When using a distribution function with a canonical parameter ${\displaystyle \theta ,}$ the canonical link function is the function that expresses ${\displaystyle \theta }$ in terms of ${\displaystyle \mu ,}$ i.e. ${\displaystyle \theta =g(\mu ).}$ For the most common distributions, the mean ${\displaystyle \mu }$ is one of the parameters in the standard form of the distribution's density function, and then ${\displaystyle g(\mu )}$ is the function as defined above that maps the density function into its canonical form. When using the canonical link function, ${\displaystyle g(\mu )=\theta =\mathbf{𝐗}\mathit{\beta },}$ which allows ${\displaystyle {\mathbf{𝐗}}^{\mathrm{T}}\mathbf{𝐘}}$ to be a sufficient statistic for ${\displaystyle \mathit{\beta }}$.

Following is a table of several exponential-family distributions in common use and the data they are typically used for, along with the canonical link functions and their inverses (sometimes referred to as the mean function, as done here).

Common distributions with typical uses and canonical link functions

Distribution	Support of distribution	Typical uses	Link name	Link function, ${\displaystyle \mathbf{𝐗}\mathit{\beta }=g(\mu )}$	Mean function
Normal	real: ${\displaystyle (-\mathrm{\infty },+\mathrm{\infty })}$	Linear-response data	Identity	${\displaystyle \mathbf{𝐗}\mathit{\beta }=\mu }$	${\displaystyle \mu =\mathbf{𝐗}\mathit{\beta }}$
Laplace
Exponential	real: ${\displaystyle (0,+\mathrm{\infty })}$	Exponential-response data, scale parameters	Negative inverse	${\displaystyle \mathbf{𝐗}\mathit{\beta }=-{\mu }^{-1}}$	${\displaystyle \mu =-(\mathbf{𝐗}\mathit{\beta }{)}^{-1}}$
Gamma
Inverse Gaussian	real: ${\displaystyle (0,+\mathrm{\infty })}$		Inverse squared	${\displaystyle \mathbf{𝐗}\mathit{\beta }={\mu }^{-2}}$	${\displaystyle \mu =(\mathbf{𝐗}\mathit{\beta }{)}^{-1/2}}$
Poisson	integer: ${\displaystyle 0,1,2,\dots }$	count of occurrences in fixed amount of time/space	Log	${\displaystyle \mathbf{𝐗}\mathit{\beta }=\ln (\mu )}$	${\displaystyle \mu =\exp (\mathbf{𝐗}\mathit{\beta })}$
Bernoulli	integer: ${\displaystyle \{0,1\}}$	outcome of single yes/no occurrence	Logit	${\displaystyle \mathbf{𝐗}\mathit{\beta }=\ln \left(\frac{\mu }{1-\mu }\right)}$	${\displaystyle \mu =\frac{\exp (\mathbf{𝐗}\mathit{\beta })}{1+\exp (\mathbf{𝐗}\mathit{\beta })}=\frac{1}{1+\exp (-\mathbf{𝐗}\mathit{\beta })}}$
Binomial	integer: ${\displaystyle 0,1,\dots ,N}$	count of # of "yes" occurrences out of N yes/no occurrences		${\displaystyle \mathbf{𝐗}\mathit{\beta }=\ln \left(\frac{\mu }{n-\mu }\right)}$
Categorical	integer: ${\displaystyle [0,K)}$	outcome of single K-way occurrence		${\displaystyle \mathbf{𝐗}\mathit{\beta }=\ln \left(\frac{\mu }{1-\mu }\right)}$
	K-vector of integer: ${\displaystyle [0,1]}$, where exactly one element in the vector has the value 1
Multinomial	K-vector of integer: ${\displaystyle [0,N]}$	count of occurrences of different types (1, ..., K) out of N total K-way occurrences

In the cases of the exponential and gamma distributions, the domain of the canonical link function is not the same as the permitted range of the mean. In particular, the linear predictor may be positive, which would give an impossible negative mean. When maximizing the likelihood, precautions must be taken to avoid this. An alternative is to use a noncanonical link function.

In the case of the Bernoulli, binomial, categorical and multinomial distributions, the support of the distributions is not the same type of data as the parameter being predicted. In all of these cases, the predicted parameter is one or more probabilities, i.e. real numbers in the range ${\displaystyle [0,1]}$. The resulting model is known as logistic regression (or multinomial logistic regression in the case that K-way rather than binary values are being predicted).

For the Bernoulli and binomial distributions, the parameter is a single probability, indicating the likelihood of occurrence of a single event. The Bernoulli still satisfies the basic condition of the generalized linear model in that, even though a single outcome will always be either 0 or 1, the