Skip to main content Skip to Footer

BLOG


November 19, 2015
The Three Most Commonly Used Regression Models
By: Billy Hou

Regression analysis is a statistical process for estimating the relationships among variables. It includes many techniques for modeling and analyzing several variables, when the focus is on the relationship between a dependent variable and one or more independent variables (predictors). More specifically, regression analysis helps one understand how the typical value of the dependent variable changes when any one of the independent variables is varied, while the other independent variables are held fixed.


Regression analysis is widely used in the business world for prediction and forecasting and nowaday overlaps with machine learning. We often use it in the predictive stage of the four levels of analytics. Practical applications include demand forecasting and price optimization that require understanding of the relationship between independent and dependent variables.

There are hundreds of types of regression techniques developed since 1805, when the first regression technique, method of least squares, was published by Adrien-Marie Legendre. How do you decide which one to use? We review the top three most used regression techniques to help you decide which to use depending on your context.

Linear Regression

Linear Regression or Linear Probability (LP) is the oldest type of regression technique and used extensively in practical applications such as forecasting, predicting dependent variables with one or more independent variable, and inference, quantifying the strength of the relationship between the dependent variable and one or more independent variable.

Usage of linear regression is limited by four principle assumptions, linearity, homoscedasticity, independence, and normality. It does not capture highly non-linear or chaotic patterns and highly correlated independent variables will cause the model to collapse. Practitioners will need to use factor reduction techniques or other constrained regression techniques to address these issues.

Linear Regression

Logistic Regression

Logistic regression is a regression model that predicts a binary dependent variable. This technique is very useful in social sciences, medical research, and fraud detection. Unlike linear regression, logistic regression does not require assumptions such as linearity (between dependent variable and independent variable), normality, and homoscedasticity. However, some assumptions still apply. Logistic regression requires a dependent variable to be binary or ordinal, error terms need to be independent, linearity of independent variables and log odds, and at least 30 observations for each parameter to be estimated.

Logistic Regression

Ridge Regression

Ridge regression is a modeling technique for analyzing multiple regression data that suffer from multi-colinearity. It is like the least squares method but shrinks the estimated coefficients towards zero. One major benefit is that ridge regression does not over fit when compared to linear regression, see graph below for example. In short, it is a more robust version of linear regression. It has the same assumptions as the linear regression model except normality is not a required assumption.

A major drawback of ridge regression is that it cannot zero out coefficients; thus you either end up including all the coefficients in the model, or none of them. The least absolute shrinkage and selection operator, or LASSO, is a technique that addresses this issue. As with ridge regression, it assumes the covariates are standardized. A practitioner must consider using a factor selection and reduction method before starting to use ridge regression or consider to use LASSO regression instead of ridge regression.

Ridge Regression

Regression analysis is at the core of forecasting and machine learning. However, there are hundreds of regression techniques and it is up to the practitioners to select the correct technique based on their context. They must understand the advantages and disadvantages of each technique to make the correct choice.


About Accenture

Accenture is a leading global professional services company, providing a broad range of services and solutions in strategy, consulting, digital, technology and operations. Combining unmatched experience and specialized skills across more than 40 industries and all business functions—underpinned by the world's largest delivery network—Accenture works at the intersection of business and technology to help clients improve their performance and create sustainable value for their stakeholders. With approximately 373,000 people serving clients in more than 120 countries, Accenture drives innovation to improve the way the world works and lives. Visit us at www.accenture.com.

Accenture Digital, comprised of Accenture Analytics, Accenture Interactive and Accenture Mobility, offers a comprehensive portfolio of business and technology services across digital marketing, mobility and analytics. From developing digital strategies to implementing digital technologies and running digital processes on their behalf, Accenture Digital helps clients leverage connected and mobile devices; extract insights from data using analytics; and enrich end-customer experiences and interactions, delivering tangible results from the virtual world and driving growth. Learn more about Accenture Digital at www.accenture.com/digital.

 

About Accenture Analytics

Accenture Analytics, part of Accenture Digital, delivers insight-driven outcomes at scale to help organizations improve their performance. With deep industry, functional, business process and technical experience, Accenture Analytics develops innovative consulting and outsourcing services for clients to help ensure they receive returns on their analytics investments. For more information follow us @ISpeakAnalytics and visit www.accenture.com/analytics.

This document makes descriptive reference to trademarks that may be owned by others. The use of such trademarks herein is not an assertion of ownership of such trademarks by Accenture and is not intended to represent or imply the existence of an association between Accenture and the lawful owners of such trademarks.

This blogpost is produced by consultants at Accenture as general guidance. It is not intended to provide specific advice on your circumstances. If you require advice or further details on any matters referred to, please contact your Accenture representative.

Popular Tags

    More blogs on this topic

      Archive