Moneyball: Sports Analytics in Soccer to Predict Performance and Outcomes

There is no doubt that soccer is the most popular sport in the world, and its popularity is growing in the US. Over 25 million fans watched U.S. Women’s FIFA World Cup 2015, and earned Fox over $40 million in ad revenue. Similarly, the United States’ 2-2 draw with Portugal in the 2014 World Cup was seen by an average of 24.7 million viewers on Univision and ESPN.
What's Sports Analytics?

Sports analytics is the processes that identify and acquire the knowledge and insight about potential players’ performances based on the use of a variety of data sources such as game data and individual player performance data. These advanced and sophisticated type of analytics should be able to extract valuable actionable insights for the coaches and managers to utilize.

Sports analytics can be utilized in various domains including:
Predicting the outcome of a game
Predicting the performances of teams or individual players
Building new strategies for upcoming competitions
Deciding the price of a player if a club was to rent/sell/buy him or her
Connecting players to brands and sponsors

Of course, not all teams use analytical tools. In addition to the costs involved, there’s also the problem of explaining complex analytical methods to coaches in ways they can understand. Thus, soccer analytics is more widespread in big clubs where they have the necessary financial power to utilize these methods.

But that should not necessarily be the case. Since traditional sports analysts do not reveal the logic behind their methods, we’ve decided to give you a flavor of what can be done with soccer data. While this post will not answer all the questions you have about soccer analytics, it can help you understand how to get started.

In this project, we have used a very limited number of player’s attributes (See the section under Player’s Features/Attributes) that are easy and not expensive to gather, to both reverse engineer the most advanced Rating and Performance index (The results are presented in Figures 1-12) and then to propose a more robust and easy model for future player ratings and performance prediction (See the section under Machine-Learning and AI Models). The program runs on Spark and Cloud Environment, and can be used for Terra-Petta scale of data, from multiple years, with thousands of players, with 100s of attributes.
What's Soccer Analytics?

Soccer Analytics is the art of creating insights and actionable decisions using soccer related data. While predictive analytics uses big data to determine the probability or the likelihood of a certain outcome, intelligent descriptive analytics looks at big data and analyzes it using machine learning and artificial intelligence methods to come of with suggestions that will improve the likelihood of a desired outcome.

Some important concepts to know while conducting this analysis are:
Game Modeling: Modeling of the game before, during and after the game using scientific techniques to match or predict a set of outcomes.
Expert Player Rating: Players ratings given by an expert. These ratings take a black-box approach, and they vary according to the prior knowledge of the expert.
Soccer Performance Analytics: Is a tool to help players, coaches and managers to quantitatively assess the players and team performance and help to improve both players and team performance and design a set of wining strategies for upcoming game(s).

Here are some questions to ask when running analysis on soccer players:
How does expert rating differ from ratings generated by data-driven Machine Learning ratings?
What are the most important players’ attributes linked to their performance?
What criteria do experts use when they evaluate Players? Is there a way to reverse engineer their criteria?
Which attributes are important for each specific position?
Can we use ratings and players’ attributes to predict the outcome of a game?
Can we aggregate the players’ rating to come up with a team rating?
Is there a way to correlate the team rating to the outcome of the game?
Do the outcomes of games influence expert ratings more than individual performance indicators do?
Can we predict the outcome of a new game given the past performance of the players?

Using advance analytics and visualization tools such as Machine Learning and network analytics to predict the outcomes of soccer games is becoming more and more popular as these methods continue to move into the mainstream with the help of tools that make it easier to conduct these advanced analytical methods.
Some insights to our approach:

In this project, we used our selected set of clustering and classification techniques and the best model was selected and ranked based on Train-Validation-Test process.
Companies such as OPTA, Prozone, Amisco, and WhoScored are now collecting rich soccer data. These can be utilized to conduct accurate assessments.
For our project, a rich data set containing more than 210 attributes of players including 198 performance statistics were used. To calculate the overall performance and ratings of the players, some or all of the attributes were being used. Some of the very advanced Expert Ratings include: Caapello Index, Castrol Index, and WhoScored.com. These Ratings include each player’s cumulative ratings and game-based ratings.
For classification-regression and clustering, there are many Machine learning models that can be used. For classification-regression model, you can use Machine Learning models (SVMs, logistic regression, linear regression), naive Bayes, Regression by Discretization using J48, Additive Regression with Decision Stump, decision trees, ensembles of trees (Random Forests and Gradient-Boosted Trees), isotonic regression, Multilayer Perceptron, RBF Network. For Clustering, you can use k-means, clustering using affinity propagation, Agglomerative Clustering (Ward, Average, and Complete), Gaussian mixture, power iteration clustering (PIC), latent Dirichlet allocation (LDA). Furthermore, you can use dimensionality reduction such as singular value decomposition (SVD) and principal component analysis (PCA) to reduce the feature space. In our case study, we tested all the models and the best results combined and are presented in Figures 1-12, without elaborating about specific model and how they can be aggregated.
Players' Features/Attributes:

In this project, we have used a subset of the Player’s Features/Attributes from the following list. Keep in mind that your model should be able to select and rank these attributes based on their importance. At this blog, we are not elaborating on what features have been selected by models, as these will depend on the specific approach you choose to take when building your model.
Nationality, Club, League, Age, Height, String Foot, Position (GK, CB, RB, LB, DM, CM, RM, LM, AM, RW, LW, SS, CF)
Attacking Prowess, Ball Control, Dribbling, Low Pass, Lofted Pass, Finishing
Place Kicking, Swerve, Header, Defensive Prowess, Ball Winning, Kicking Power, Speed, Explosive Power, Body Balance, Jump, Stamina, Goalkeeping, Saving, Form, Injury, Resistance, Weak Foot Use, Weak Foot Accuracy, Trickster, Mazing Run, Speeding Bullet,, Incisive Run, Long Ball Expert, Early Cross, Long Ranger
Scissors Feint, Flip Flap, Marseille Turn, Sombrero, Cut Behind & Turn, Scotch Move, Long Range Drive, Knuckle Shot, Acrobatic Finishing, First-time Shot, One-touch Pass, Weighted Pass, Pinpoint Crossing, Outside Curler, Low Punt Trajectory, Long Throw, GK Long Throw, Man Marking, Track Back, Captancy, Super-sub, Fighting Spirit

While, we do not have intention to disclose the final optimized features selected by our optimized model and strategy, we used the following ML-AL based aggregator operator model (Figure 13 shows the Structure of our Committee Machine).
Figures:

Finally, here's a list of the plots and figures created as a result of this analysis:



Figure 1. Clustering (similarities of players’ clusters) and prediction of the rating (plot: prediction vs. actual) for the Forward Players. Train, test and Validation. The similarities of the individual players are shown by the lines on clustering plots.



Figure 2. Clustering (similarities of players’ clusters) and prediction of the rating (plot: prediction vs. actual) for the Goalkeepers. Train, test and Validation. The similarities of the individual players are shown by the lines on clustering plots.



Figure 3. Clustering (similarities of players’ clusters) and prediction of the rating (plot: prediction vs. actual) for the Defensive Players. Train, test and Validation. The similarities of the individual players are shown by the lines on clustering plots.



Figure 4. Clustering (similarities of players’ clusters) and prediction of the rating (plot: prediction vs. actual) for the MidField Players. Train, test and Validation. The similarities of the individual players are shown by the lines on clustering plots.



Figure 5. Clustering (similarities of players’ clusters) using advanced Visual-Analytics-Clustering for Forward Players. The similarities of the individual players are shown by the lines and their strength with the width of lines on clustering plot.



Figure 6. Clustering (similarities of players’ clusters) using advanced Visual-Analytics-Clustering for Forward Players. Similar clusters are closer to each other.



Figure 7. Typical performance of our Regression model’s Prediction for the rating of the Forward players (Prediction Vs. Actual). Train data for Forward Players.



Figure 8. Typical performance of our Regression model’s Prediction for the rating of the Forward players (Prediction Vs. Actual). Validation data for Forward Players.



Figure 9. Typical performance of our Regression model’s Prediction for the rating of the Forward players (Prediction Vs. Actual). Test data for Forward Players.



Figure 10. Typical performance of our Regression model’s Prediction for the rating of the players (Prediction Vs. Actual). Test data for Forward-MidField-Defensive Players.



Figure 11. Typical performance of our Regression model’s Prediction for the rating of the players (Prediction Vs. Actual). Validation data for Forward-MidField-Defensive Players.




Figure 12. Typical performance of our Regression model’s Prediction for the rating of the players (Prediction Vs. Actual). Test data for Forward-MidField-Defensive Players.



Figure 13. Committee Machine, and Intelligent Multi-Level-Aggregator Tree (MAT) platform.

Everything you ever wanted or needed to know about Big Data #BigData

Big Data is a phrase that gets bandied about quite a bit in the media, the board room – and everywhere in between. It’s been used, overused and used incorrectly so many times that it’s become difficult to know what it really means. Is it a tool? Is it a technology? Is it just a buzzword used by data scientists to scare us? Is it really going to change the world? Or ruin it?

This post is all about demystifying the mess that has become Big Data, and more importantly demonstrating how you can use it to improve your bottom line.

What Is Big Data?

First of all, what is Big Data? In it’s purest form, Big Data is used to describe the massive volume of both structured and unstructured data that is so large it is difficult to process using traditional techniques. So Big Data is just what it sounds like – a whole lot of data.

The concept of Big Data is a relatively new one and it represents both the increasing amount and the varied types of data that is now being collected. Proponents of Big Data often refer to this as the “datification” of the world. As more and more of the world’s information moves online and becomes digitized, it means that analysts can start to use it as data. Things like social media, online books, music, videos and the increased amount of sensors have all added to the astounding increase in the amount of data that has become available for analysis.

Everything you do online is now stored and tracked as data. Reading a book on your Kindle generates data about what you’re reading, when you read it, how fast you read it and so on. Similarly, listening to music generates data about what you’re listening to, when how often and in what order. Your smart phone is constantly uploading data about where you are, how fast you’re moving and what apps you’re using.

What’s also important to keep in mind is that Big Data isn’t just about the amount of data we’re generating, it’s also about all the different types of data (text, video, search logs, sensor logs, customer transactions, etc.). In fact, Big Data has four important characteristics that are known in the industry as the 4 V’s:

• Volume – the increasing amount of data that is generated every second
• Velocity – the speed at which data is being generated
• Variety – the different types of data being generated
• Veracity – the messiness of data, ie. it’s unstructured nature

Based on the incredible amount, speed, variety and unstructuredness of the data we are now generating and storing, it’s no surprise that it quickly became unmanageable using traditional storing and analysis methods. This is where the term Big Data becomes confusing, because it is often used to refer to the new technologies, tools and processes that have sprung up to accommodate this vast amount of data.

Glossary of Big Data Terms

Inevitably, much of the confusion around Big Data comes from the variety of new (for many) terms that have sprung up around it. Here is a quick run-down of the most popular ones:

• Algorithm – mathematical formula run by software to analyze data
• Amazon Web Services (AWS) – collection of cloud computing services that help businesses carry out large-scale computing operations without needing the storage or processing power in-house
• Cloud (computing) – running software on remote servers rather than locally
• Data Scientist – an expert in extracting insights and analysis from data
• Hadoop – collection of programs that allow for the storage, retrieval and analysis of very large data sets
• Internet of Things (IoT) – refers to objects (like sensors) that collect, analyze and transmit their own data (often without human input)
• Predictive Analytics – using analytics to predict trends or future events
• Structured v Unstructured data – structured data is anything that can be organized in a table so that it relates to to other data in the same table. Unstructured data is everything that can’t.
• Web scraping – the process of automating the collection and structuring of data from web sites (usually through writing code)

Why Has It Become So Popular

Big Data’s recent popularity has been due in large part to new advances in technology and infrastructure that allow for the processing, storing and analysis of so much data. Computing power has increased considerably in the past five years while at the same time dropping in price – making it more accessible to small and midsize companies. In the same vein, the infrastructure and tools for large-scale data analysis has gotten more powerful, less expensive and easier to use. According to

As the technology has gotten more powerful and less expensive, numerous companies have emerged to take advantage of it by creating products and services that help businesses to take advantage of all Big Data has to offer.  According to Inc, in 2012 the Big Data industry was worth $3.2 billion and growing quickly. They went on to say that “Total [Big Data] industry revenue is expected to reach nearly $17 billion by 2015, growing about seven times faster than the overall IT market”. For more on the size and projected growth of the Big Data industry, check out this Forbes article.

Businesses have also started taking notice of the Big Data trend. In a recent survey, “Eighty-seven percent of enterprises believe big data analytics will redefine the competitive landscape of their industries within the next three years.”

Why Should Businesses Care?

Data has always been used by businesses to gain insights through analysis. The emergence of Big Data means that they can now do this on an even greater scale, taking into account more and more factors. By analyzing greater volumes from a more varied set of data, businesses can derive new insights with a greater degree of accuracy. This directly contributes to improved performance and decision making within an organization.

Big Data is fast becoming a crucial way for companies to outperform their peers. Good data analysis can highlight new growth opportunities, identify and even predict market trends, be used for competitor analysis, generate new leads and much more. Learning to use this data effectively will give businesses greater transparency into their operations, better predictions, faster sales and bigger profits.

Best Big Data Tools

Taking advantage of all that Big Data has to offer can seem like a daunting task, but there are a number of tools (both free and paid) that can help businesses to collect, store, analyze and derive insight from Big Data. Here are just a few…

OpenRefine

OpenRefine is a data cleaning software that allows you to pre-process your data for analysis. This is especially useful if you are analyzing unstructured data or combining multiple data sets into one for analysis.

WolframAlpha

WorlframAlpha provides detailed responses to technical searches and does very complex calculations. For business users, it presents information charts and graphs, and is excellent for high level pricing history, commodity information, and topic overviews.

import.io

import.io is allows you to turn the unstructured data displayed on web pages into structured tables of data that can be accessed over an API.

Tableau

Tableau is a visualization tool that makes it easy to look at your data in new ways. In the analytics process, Tableau’s visuals allow you to quickly investigate a hypothesis, sanity check your instincts or build a compelling infographic to convince your audience with.

Google Fusion Tables

Google Fusion Tables is a versatile tool for data analysis, large data set visualization and mapping.

Best Additional Resources (blog posts, case studies, books, videos, etc)

If you’re interested in learning more about Big Data and how you can use it, here are a few of our favorite resources:

Blogs

• No Free Hunch (kaggle) – Kaggle hosts a number of predictive modeling competitions. Their competition and data science blog, covers all things related to the sport of data science.
• SmartData Collective – SmartData Collective is an online community moderated by Social Media Today that provides information on the latest trends in business intelligence and data management.
• FlowingData – FlowingData explores the ways in which data scientists, designers, and statisticians use analysis, visualization, and exploration to understand data and ourselves.
• KDnuggets – KDnuggets is a comprehensive resource for anyone with a vested interest in the data science community.
• Data Elixir – Data Elixir is a great roundup of data news across the web, you can get a weekly digest sent straight to your inbox.

Online Courses/Learning Resources

• DataCamp – DataCamp is a resource for learning data analysis and R interactively.
• School of Data – School of Data offers a variety of courses designed for everyone, from the data science-newbie to the professional seeking inspiration.
• Udemy – Udemy is the world’s largest destination for online courses with many in the data science field.
• w3schools – W3schools is great online tutorials for learning basic coding and data analysis skills.

Videos

• The Data Science Revolution – an expert panel that considerations of the future of data science and the ethics involved with data analytics and enhanced predictive powers.
• Turning Big Data into Big Analytics – focuses on the opportunity businesses have when dealing correctly with their data and serves as a case study for data science professionals.

Books

• Big Data: A Revolution That Will Transform How We Live, Work and Think – a fascinating survey of big data’s growing effect on just about everything: business, government, science and medicine, privacy, and even on the way we think.
• Big Data: Using SMART Big Data, Analytics and Metrics To Make Better Decisions and Improve Performance – a clear understanding, blueprint, and step-by-step approach to building your own big data strategy.
• Data Science for Business: What you need to know about data mining and data-analytic thinking – introduces the fundamental principles of data science, and walks you through the “data-analytic thinking” necessary for extracting useful knowledge and business value from the data you collect.

Looking Ahead

What the future of Big Data really holds, no one can predict. The rapid development of new technologies, especially in the machine learning space, will undoubtedly usurp any predictions we try to make. What is certain, is that Big Data is here to stay. The amount of data we are producing is only going to increase and by analyzing it, we can learn and eventually be able to predict some pretty cool things. Very soon, Big Data will touch and transform every industry and every piece of your daily life.

Wrapping Up

Whether or not you believe the hype about whether Big Data will change the world, the fact remains that learning how to use the recent influx of data effectively can help you make better, more informed decisions. The thing to take away from Big Data isn’t it’s largeness, it’s the variety. You don’t necessarily need to analyze a lot of data to get accurate insights, you just need to make sure you are analyzing theright data. To really take advantage of this data revolution, you need to start thinking about new and varied data sources that can give you a more well rounded picture of your customers, market and competitors. With today’s Big Data technologies, everything can be used as data – giving you unparalleled access to market factors.

What’s your take on the future of Big Data? Leave a comment for us below!

HOW THE MODEL WORKS

At a high level, our approach is as follows. First, we estimate a regression model to predict the number of goals scored by a particular team (“team i”) against a particular opponent (“team j”) using the entire history of mandatory international matches since 1958, when the first European championship was played (a total of 4,719 matches).[1] Following the literature on predicting football matches, we assume that the number of goals scored by team i is described by a so-called Poisson distribution and explained by the following statistical factors:[2] 1. The difference in team performance as reflected in Elo ratings prior to the match. The Elo system was originally devised to rank chess players. It is a composite measure of national football team success that evolves depending on a team's results and the strength of its opponents. 2. The number of goals scored by team i in the last 10 competitive matches. 3. The number of goals conceded by team j in the last 2 competitive matches. 4. A home dummy. 5. A European Championship dummy to capture whether a team does systematically better at European Championships than in other competitive matches Second, we use these regression estimates and our assumed Poisson distribution in a Monte Carlo simulation with 100,000 draws to generate a distribution of outcomes for each of the 52 matches, from the opener between France and Romania on June 10 to the final on July 10. We use the rounded prediction of the goals scored to determine the outcome of each match during the group stage and the unrounded prediction to pick the winner in the knockout stage. Third, we use the estimation results to generate both a set of probabilities that a particular team reaches a particular stage of the tournament, up to and including the championship, and a modal—that is, single most likely—forecast for the outcome of each match, which we then run forward through the tournament until the final.


A SUMMARY OF PREDICTIONS
Our probabilities are shown in Exhibit 1. The model says that France has a 23% probability of winning the trophy, followed by Germany at 20%, Spain at 14%, and England at 11%. Although Germany has the highest Elo rating, France is favored because of its home advantage.




Exhibits 2 and 3 provide a different perspective by showing the modal prediction for the entire tournament. There are some interesting contrasts with the probabilities in Exhibit 1. For example, Exhibit 1 says that Germany is more likely than Spain to win the tournament because it is more likely to succeed across the entire range of possible tournament configurations. But Exhibit 3 says that in the single most likely case, Spain beats England in Semifinal 1, France beats Germany in Semifinal 2, and France then wins the final—i.e., Germany finishes behind Spain. Which approach is better, the probabilistic one in Exhibit 1 or the modal one in Exhibits 2 and 3? A modal forecast does have the advantage of being more “crisp.” The sentence “Goldman Sachs says France will win” has a better ring to it than “Goldman Sachs says France has a 23% probability of winning, with Germany close behind.” Nevertheless, we think that a probabilistic approach is more useful—for predicting the outcome of football tournaments and, increasingly, for our day-to-day work on economic forecasting.



Exhibit 4 provides more insight into the results by breaking down the probabilities of winning for the top four teams in a “waterfall chart” format. It shows that the most important factor is the Elo score, followed by home advantage and the European Championship dummy. The chart illustrates that the front-runner position for France derives largely from its home advantage, as its Elo rating is well below Germany’s and also a bit below Spain’s. Meanwhile, Germany benefits from the European Championship dummy, which picks up its historically strong tournament performance. 





HOW CONFIDENT CAN WE BE?

It is difficult to assess how much faith one should have in these predictions. On the plus side, our approach carefully considers the stochastic nature of the tournament using statistical methods, and we do think that the Elo rating—the most important input into our analysis—is a compelling summary of a team’s track record. On the minus side, we ignore a number of potentially important factors that are difficult to summarize statistically, including the quality of the individual players unless they are reflected in the team’s recent track record.[4] And there is no room for human judgment (which may not be such a bad thing given that none of us are really football experts but some are enthusiastic Germany supporters).[5]



One useful cross-check is to compare our results with bookmakers’ odds. Exhibit 5 plots our estimated championship probability against the average probability implied by the odds offered by five different bookmakers. The basic result is clear. Even though our model does not include bookmakers’ odds in any way, the probabilities are quite similar. A possible reason is that professional betting firms use many of the same inputs—such as Elo ratings—in their analysis and that they process the information in ways that are ultimately similar to ours. Another useful check is to evaluate the performance of our model for the 2014 World Cup, which followed an essentially identical approach to the one presented here.[6] It is safe to say that we had our hits and misses. First, performance in the group stage was not great. The model only identified 9 of the 16 advancing teams and failed to predict the elimination of heavyweights Spain and Italy, although it correctly anticipated that England would fly home early. Second, the model gave Brazil a 48% probability of winning the trophy, by far the highest of all the contestants. That failure illustrates a certain lack of imagination that is inherent in our approach. If the greatest football nation on earth—in terms of both past victories and its 2014 Elo rating—plays a World Cup at home, we are bound to project success. At least the probability was below 50%! Third, the model did correctly identify three of the four semifinalists before the start of the tournament, namely Argentina, Brazil, and Germany, although it incorrectly picked Spain over the Netherlands. Fourth, the fully updated version of the model—that is, the projection we sent out before each day of play on the basis of updated Elo ratings and other performance measures—was remarkably accurate during the knockout stage. It correctly predicted the winner of every match except the 7-1 semifinal between Germany and Brazil. But that was, by one estimate, the single most surprising result in World Cup history.[7] Ultimately, this last predictive failure might best capture the spirit of the exercise. As we said in our comment at the time: “Speaking as forecasters, we regret the miss. But, speaking as Germans, we would note that there are more important things than being right.” May the best team win and let’s hope that watching Euro 2016 is as much fun as it was to write this article!

THE ECONOMETRICIAN’S TAKE ON EURO 2016

We present a model for predicting the outcome of the 2016 European Football championship in France from June 10 to July 10. It is similar to our model for the 2014 World Cup. Using historical performance data for each team—most importantly the Elo rating system originally devised to rank chess players—we estimate a set of probabilities that a particular team will reach a particular round, up to and including the championship. We also provide a modal “most likely” case for how the tournament will unfold (although “most likely” does not mean “likely”). The model says that France has a 23% probability of winning the trophy, followed by Germany at 20%, Spain at 14%, and England at 11%. Although Germany has the highest Elo rating, France is slightly favored because of its home advantage. After each day of play, we will re-run the model using updated historical performance data in order to generate new probabilities and a new modal forecast. How much faith should we have in these predictions? On the plus side, our approach carefully considers the stochastic nature of the tournament using statistical methods; also, the predictions are not far from bookmakers’ odds. On the minus side, the environment is “stochastic” indeed, i.e., football is quite an unpredictable game! That charming unpredictability was on full display two years ago, when our model failed to anticipate the elimination of heavyweights Spain and Italy in the group stage and gave Brazil a 48% probability of winning the trophy. More encouragingly, it identified three of the four semifinalists before the start of the tournament, and the fully updated version predicted the winner of every match in the knockout stage except for the 7-1 semifinal between Germany and Brazil. Below we will introduce our statistical model for predicting the outcome of the 2016 European Football Championship in France from June 10 to July 10.

Python in one Picture

Predictive Analytics

Predictive analytics encompasses a variety of techniques from statistics, data mining and game theory that analyze current and historical facts to make predictions about future events.

In business, predictive models exploit patterns found in historical and transactional data to identify risks and opportunities. Models capture relationships among many factors to allow assessment of risk or potential associated with a particular set of conditions, guiding decision making for candidate transactions.

Predictive analytics is used in actuarial science, financial services, insurance, telecommunications, retail, travel, healthcare, pharmaceuticals and other fields.

One of the most well-known applications is credit scoring, which is used throughout financial services. Scoring models process a customer’s credit history, loan application, customer data, etc., in order to rank-order individuals by their likelihood of making future credit payments on time. A well-known example would be the FICO score.

Definition Predictive analytics is an area of statistical analysis that deals with extracting information from data and using it to predict future trends and behavior patterns. The core of predictive analytics relies on capturing relationships between explanatory variables and the predicted variables from past occurrences, and exploiting it to predict future outcomes. It is important to note, however, that the accuracy and usability of results will depend greatly on the level of data analysis and the quality of assumptions.

Types: Generally, the term predictive analytics is used to mean predictive modeling, "scoring" data with predictive models, and forecasting. However, people are increasingly using the term to describe related analytical disciplines, such as descriptive modeling and decision modeling or optimization. These disciplines also involve rigorous data analysis, and are widely used in business for segmentation and decision making, but have different purposes and the statistical techniques underlying them vary.

Predictive models: Predictive models analyze past performance to assess how likely a customer is to exhibit a specific behavior in the future in order to improve marketing effectiveness. This category also encompasses models that seek out subtle data patterns to answer questions about customer performance, such as fraud detection models. Predictive models often perform calculations during live transactions, for example, to evaluate the risk or opportunity of a given customer or transaction, in order to guide a decision. With advancement in computing speed, individual agent modeling systems can simulate human behavior or reaction to given stimuli or scenarios. The new term for animating data specifically linked to an individual in a simulated environment is avatar analytics.

Descriptive models: Descriptive models quantify relationships in data in a way that is often used to classify customers or prospects into groups. Unlike predictive models that focus on predicting a single customer behavior (such as credit risk), descriptive models identify many different relationships between customers or products. Descriptive models do not rank-order customers by their likelihood of taking a particular action the way predictive models do. Descriptive models can be used, for example, to categorize customers by their product preferences and life stage. Descriptive modeling tools can be utilized to develop further models that can simulate large number of individualized agents and make predictions.

Decision models: Decision models describe the relationship between all the elements of a decision — the known data (including results of predictive models), the decision and the forecast results of the decision — in order to predict the results of decisions involving many variables. These models can be used in optimization, maximizing certain outcomes while minimizing others. Decision models are generally used to develop decision logic or a set of business rules that will produce the desired action for every customer or circumstance.

Applications: Although predictive analytics can be put to use in many applications, we outline a few examples where predictive analytics has shown positive impact in recent years.

Analytical customer relationship management (CRM): Analytical Customer Relationship Management is a frequent commercial application of Predictive Analysis. Methods of predictive analysis are applied to customer data to pursue CRM objectives.

Clinical decision support systems: Experts use predictive analysis in health care primarily to determine which patients are at risk of developing certain conditions, like diabetes, asthma, heart disease and other lifetime illnesses. Additionally, sophisticated clinical decision support systems incorporate predictive analytics to support medical decision making at the point of care. A working definition has been proposed by Dr. Robert Hayward of the Centre for Health Evidence: "Clinical Decision Support systems link health observations with health knowledge to influence health choices by clinicians for improved health care."

Collection analytics: Every portfolio has a set of delinquent customers who do not make their payments on time. The financial institution has to undertake collection activities on these customers to recover the amounts due. A lot of collection resources are wasted on customers who are difficult or impossible to recover. Predictive analytics can help optimize the allocation of collection resources by identifying the most effective collection agencies, contact strategies, legal actions and other strategies to each customer, thus significantly increasing recovery at the same time reducing collection costs.

Cross-sell: Often corporate organizations collect and maintain abundant data (e.g. customer records, sale transactions) and exploiting hidden relationships in the data can provide a competitive advantage to the organization. For an organization that offers multiple products, an analysis of existing customer behavior can lead to efficient cross sell of products. This directly leads to higher profitability per customer and strengthening of the customer relationship. Predictive analytics can help analyze customers’ spending, usage and other behavior, and help cross-sell the right product at the right time.

Customer retention: With the number of competing services available, businesses need to focus efforts on maintaining continuous consumer satisfaction. In such a competitive scenario, consumer loyalty needs to be rewarded and customer attrition needs to be minimized. Businesses tend to respond to customer attrition on a reactive basis, acting only after the customer has initiated the process to terminate service. At this stage, the chance of changing the customer’s decision is almost impossible. Proper application of predictive analytics can lead to a more proactive retention strategy. By a frequent examination of a customer’s past service usage, service performance, spending and other behavior patterns, predictive models can determine the likelihood of a customer wanting to terminate service sometime in the near future. An intervention with lucrative offers can increase the chance of retaining the customer. Silent attrition is the behavior of a customer to slowly but steadily reduce usage and is another problem faced by many companies. Predictive analytics can also predict this behavior accurately and before it occurs, so that the company can take proper actions to increase customer activity.

Direct marketing: When marketing consumer products and services there is the challenge of keeping up with competing products and consumer behavior. Apart from identifying prospects, predictive analytics can also help to identify the most effective combination of product versions, marketing material, communication channels and timing that should be used to target a given consumer. The goal of predictive analytics is typically to lower the cost per order or cost per action.

Fraud detection: Fraud is a big problem for many businesses and can be of various types. Inaccurate credit applications, fraudulent transactions (both offline and online), identity thefts and false insurance claims are some examples of this problem. These problems plague firms all across the spectrum and some examples of likely victims are credit card issuers, insurance companies, retail merchants, manufacturers, business to business suppliers and even services providers. This is an area where a predictive model is often used to help weed out the “bads” and reduce a business's exposure to fraud.

Predictive modeling can also be used to detect financial statement fraud in companies, allowing auditors to gauge a company's relative risk, and to increase substantive audit procedures as needed.

The Internal Revenue Service (IRS) of the United States also uses predictive analytics to try to locate tax fraud.

Portfolio, product or economy level prediction: Often the focus of analysis is not the consumer but the product, portfolio, firm, industry or even the economy. For example a retailer might be interested in predicting store level demand for inventory management purposes. Or the Federal Reserve Board might be interested in predicting the unemployment rate for the next year. These type of problems can be addressed by predictive analytics using Time Series techniques (see below).

Underwriting: Many businesses have to account for risk exposure due to their different services and determine the cost needed to cover the risk. For example, auto insurance providers need to accurately determine the amount of premium to charge to cover each automobile and driver. A financial company needs to assess a borrower’s potential and ability to pay before granting a loan. For a health insurance provider, predictive analytics can analyze a few years of past medical claims data, as well as lab, pharmacy and other records where available, to predict how expensive an enrollee is likely to be in the future. Predictive analytics can help underwriting of these quantities by predicting the chances of illness, default, bankruptcy, etc. Predictive analytics can streamline the process of customer acquisition, by predicting the future risk behavior of a customer using application level data. Predictive analytics in the form of credit scores have reduced the amount of time it takes for loan approvals, especially in the mortgage market where lending decisions are now made in a matter of hours rather than days or even weeks. Proper predictive analytics can lead to proper pricing decisions, which can help mitigate future risk of default.

Statistical techniques: The approaches and techniques used to conduct predictive analytics can broadly be grouped into regression techniques and machine learning techniques.

Regression techniques: Regression models are the mainstay of predictive analytics. The focus lies on establishing a mathematical equation as a model to represent the interactions between the different variables in consideration. Depending on the situation, there is a wide variety of models that can be applied while performing predictive analytics. Some of them are briefly discussed below.

Linear regression model: The linear regression model analyzes the relationship between the response or dependent variable and a set of independent or predictor variables. This relationship is expressed as an equation that predicts the response variable as a linear function of the parameters. These parameters are adjusted so that a measure of fit is optimized. Much of the effort in model fitting is focused on minimizing the size of the residual, as well as ensuring that it is randomly distributed with respect to the model predictions.

The goal of regression is to select the parameters of the model so as to minimize the sum of the squared residuals. This is referred to as ordinary least squares (OLS) estimation and results in best linear unbiased estimates (BLUE) of the parameters if and only if the Gauss-Markov assumptions are satisfied.

Once the model has been estimated we would be interested to know if the predictor variables belong in the model – i.e. is the estimate of each variable’s contribution reliable? To do this we can check the statistical significance of the model’s coefficients which can be measured using the t-statistic. This amounts to testing whether the coefficient is significantly different from zero. How well the model predicts the dependent variable based on the value of the independent variables can be assessed by using the R² statistic. It measures predictive power of the model i.e. the proportion of the total variation in the dependent variable that is “explained” (accounted for) by variation in the independent variables.

Discrete choice models: Multivariate regression (above) is generally used when the response variable is continuous and has an unbounded range. Often the response variable may not be continuous but rather discrete. While mathematically it is feasible to apply multivariate regression to discrete ordered dependent variables, some of the assumptions behind the theory of multivariate linear regression no longer hold, and there are other techniques such as discrete choice models which are better suited for this type of analysis. If the dependent variable is discrete, some of those superior methods are logistic regression, multinomial logit and probit models. Logistic regression and probit models are used when the dependent variable is binary.

Logistic regression: For more details on this topic, see logistic regression.
In a classification setting, assigning outcome probabilities to observations can be achieved through the use of a logistic model, which is basically a method which transforms information about the binary dependent variable into an unbounded continuous variable and estimates a regular multivariate model (See Allison’s Logistic Regression for more information on the theory of Logistic Regression).

The Wald and likelihood-ratio test are used to test the statistical significance of each coefficient b in the model (analogous to the t tests used in OLS regression; see above). A test assessing the goodness-of-fit of a classification model is the –.

Multinomial logistic regression: An extension of the binary logit model to cases where the dependent variable has more than 2 categories is the multinomial logit model. In such cases collapsing the data into two categories might not make good sense or may lead to loss in the richness of the data. The multinomial logit model is the appropriate technique in these cases, especially when the dependent variable categories are not ordered (for examples colors like red, blue, green). Some authors have extended multinomial regression to include feature selection/importance methods such as Random multinomial logit.

Probit regressionProbit models offer an alternative to logistic regression for modeling categorical dependent variables. Even though the outcomes tend to be similar, the underlying distributions are different. Probit models are popular in social sciences like economics.

A good way to understand the key difference between probit and logit models, is to assume that there is a latent variable z.

We do not observe z but instead observe y which takes the value 0 or 1. In the logit model we assume that y follows a logistic distribution. In the probit model we assume that y follows a standard normal distribution. Note that in social sciences (example economics), probit is often used to model situations where the observed variable y is continuous but takes values between 0 and 1.

Logit versus probit: The Probit model has been around longer than the logit model. They look identical, except that the logistic distribution tends to be a little flat tailed. One of the reasons the logit model was formulated was that the probit model was difficult to compute because it involved calculating difficult integrals. Modern computing however has made this computation fairly simple. The coefficients obtained from the logit and probit model are also fairly close. However, the odds ratio makes the logit model easier to interpret.

For practical purposes the only reasons for choosing the probit model over the logistic model would be:

There is a strong belief that the underlying distribution is normal
The actual event is not a binary outcome (e.g. Bankrupt/not bankrupt) but a proportion (e.g. Proportion of population at different debt levels).
Time series models: Time series models are used for predicting or forecasting the future behavior of variables. These models account for the fact that data points taken over time may have an internal structure (such as autocorrelation, trend or seasonal variation) that should be accounted for. As a result standard regression techniques cannot be applied to time series data and methodology has been developed to decompose the trend, seasonal and cyclical component of the series. Modeling the dynamic path of a variable can improve forecasts since the predictable component of the series can be projected into the future.

Time series models estimate difference equations containing stochastic components. Two commonly used forms of these models are autoregressive models (AR) and moving average (MA) models. The Box-Jenkins methodology (1976) developed by George Box and G.M. Jenkins combines the AR and MA models to produce the ARMA (autoregressive moving average) model which is the cornerstone of stationary time series analysis. ARIMA (autoregressive integrated moving average models) on the other hand are used to describe non-stationary time series. Box and Jenkins suggest differencing a non stationary time series to obtain a stationary series to which an ARMA model can be applied. Non stationary time series have a pronounced trend and do not have a constant long-run mean or variance.

Box and Jenkins proposed a three stage methodology which includes: model identification, estimation and validation. The identification stage involves identifying if the series is stationary or not and the presence of seasonality by examining plots of the series, autocorrelation and partial autocorrelation functions. In the estimation stage, models are estimated using non-linear time series or maximum likelihood estimation procedures. Finally the validation stage involves diagnostic checking such as plotting the residuals to detect outliers and evidence of model fit.

In recent years time series models have become more sophisticated and attempt to model conditional heteroskedasticity with models such as ARCH (autoregressive conditional heteroskedasticity) and GARCH (generalized autoregressive conditional heteroskedasticity) models frequently used for financial time series. In addition time series models are also used to understand inter-relationships among economic variables represented by systems of equations using VAR (vector autoregression) and structural VAR models.

Survival or duration analysis: Survival analysis is another name for time to event analysis. These techniques were primarily developed in the medical and biological sciences, but they are also widely used in the social sciences like economics, as well as in engineering (reliability and failure time analysis).

Censoring and non-normality, which are characteristic of survival data, generate difficulty when trying to analyze the data using conventional statistical models such as multiple linear regression. The normal distribution, being a symmetric distribution, takes positive as well as negative values, but duration by its very nature cannot be negative and therefore normality cannot be assumed when dealing with duration/survival data. Hence the normality assumption of regression models is violated.

The assumption is that if the data were not censored it would be representative of the population of interest. In survival analysis, censored observations arise whenever the dependent variable of interest represents the time to a terminal event, and the duration of the study is limited in time.

An important concept in survival analysis is the hazard rate, defined as the probability that the event will occur at time t conditional on surviving until time t. Another concept related to the hazard rate is the survival function which can be defined as the probability of surviving to time t.

Most models try to model the hazard rate by choosing the underlying distribution depending on the shape of the hazard function. A distribution whose hazard function slopes upward is said to have positive duration dependence, a decreasing hazard shows negative duration dependence whereas constant hazard is a process with no memory usually characterized by the exponential distribution. Some of the distributional choices in survival models are: F, gamma, Weibull, log normal, inverse normal, exponential etc. All these distributions are for a non-negative random variable.

Duration models can be parametric, non-parametric or semi-parametric. Some of the models commonly used are Kaplan-Meier and Cox proportional hazard model (non parametric).

Classification and regression trees: Main article: decision tree learning
Classification and regression trees (CART) is a non-parametric decision tree learning technique that produces either classification or regression trees, depending on whether the dependent variable is categorical or numeric, respectively.

Decision trees are formed by a collection of rules based on variables in the modeling data set:

Rules based on variables’ values are selected to get the best split to differentiate observations based on the dependent variable
Once a rule is selected and splits a node into two, the same process is applied to each “child” node (i.e. it is a recursive procedure)
Splitting stops when CART detects no further gain can be made, or some pre-set stopping rules are met. (Alternatively, the data is split as much as possible and then the tree is later pruned.)
Each branch of the tree ends in a terminal node. Each observation falls into one and exactly one terminal node, and each terminal node is uniquely defined by a set of rules.

A very popular method for predictive analytics is Leo Breiman's Random forests or derived versions of this technique like Random multinomial logit.

Multivariate adaptive regression splines: Multivariate adaptive regression splines (MARS) is a non-parametric technique that builds flexible models by fitting piecewise linear regressions.

An important concept associated with regression splines is that of a knot. Knot is where one local regression model gives way to another and thus is the point of intersection between two splines.

In multivariate and adaptive regression splines, basis functions are the tool used for generalizing the search for knots. Basis functions are a set of functions used to represent the information contained in one or more variables. Multivariate and Adaptive Regression Splines model almost always creates the basis functions in pairs.

Multivariate and adaptive regression spline approach deliberately overfits the model and then prunes to get to the optimal model. The algorithm is computationally very intensive and in practice we are required to specify an upper limit on the number of basis functions.

Machine learning techniques. Machine learning, a branch of artificial intelligence, was originally employed to develop techniques to enable computers to learn. Today, since it includes a number of advanced statistical methods for regression and classification, it finds application in a wide variety of fields including medical diagnostics, credit card fraud detection, face and speech recognition and analysis of the stock market. In certain applications it is sufficient to directly predict the dependent variable without focusing on the underlying relationships between variables. In other cases, the underlying relationships can be very complex and the mathematical form of the dependencies unknown. For such cases, machine learning techniques emulate human cognition and learn from training examples to predict future events.

A brief discussion of some of these methods used commonly for predictive analytics is provided below. A detailed study of machine learning can be found in Mitchell (1997).

Neural networks. Neural networks are nonlinear sophisticated modeling techniques that are able to model complex functions. They can be applied to problems of prediction, classification or control in a wide spectrum of fields such as finance, cognitive psychology/neuroscience, medicine, engineering, and physics.

Neural networks are used when the exact nature of the relationship between inputs and output is not known. A key feature of neural networks is that they learn the relationship between inputs and output through training. There are two types of training in neural networks used by different networks, supervised and unsupervised training, with supervised being the most common one.

Some examples of neural network training techniques are backpropagation, quick propagation, conjugate gradient descent, projection operator, Delta-Bar-Delta etc. Some unsupervised network architectures are multilayer perceptrons, Kohonen networks, Hopfield networks, etc.

Radial basis functions: A radial basis function (RBF) is a function which has built into it a distance criterion with respect to a center. Such functions can be used very efficiently for interpolation and for smoothing of data. Radial basis functions have been applied in the area of neural networks where they are used as a replacement for the sigmoidal transfer function. Such networks have 3 layers, the input layer, the hidden layer with the RBF non-linearity and a linear output layer. The most popular choice for the non-linearity is the Gaussian. RBF networks have the advantage of not being locked into local minima as do the feed-forward networks such as the multilayer perceptron.

Support vector machines: Support Vector Machines (SVM) are used to detect and exploit complex patterns in data by clustering, classifying and ranking the data. They are learning machines that are used to perform binary classifications and regression estimations. They commonly use kernel based methods to apply linear classification techniques to non-linear classification problems. There are a number of types of SVM such as linear, polynomial, sigmoid etc.

Naïve Bayes: Naïve Bayes based on Bayes conditional probability rule is used for performing classification tasks. Naïve Bayes assumes the predictors are statistically independent which makes it an effective classification tool that is easy to interpret. It is best employed when faced with the problem of ‘curse of dimensionality’ i.e. when the number of predictors is very high.

K-nearest neighbours: The nearest neighbour algorithm (KNN) belongs to the class of pattern recognition statistical methods. The method does not impose a priori any assumptions about the distribution from which the modeling sample is drawn. It involves a training set with both positive and negative values. A new sample is classified by calculating the distance to the nearest neighbouring training case. The sign of that point will determine the classification of the sample. In the k-nearest neighbour classifier, the k nearest points are considered and the sign of the majority is used to classify the sample. The performance of the kNN algorithm is influenced by three main factors: (1) the distance measure used to locate the nearest neighbours; (2) the decision rule used to derive a classification from the k-nearest neighbours; and (3) the number of neighbours used to classify the new sample. It can be proved that, unlike other methods, this method is universally asymptotically convergent, i.e.: as the size of the training set increases, if the observations are independent and identically distributed (i.i.d.), regardless of the distribution from which the sample is drawn, the predicted class will converge to the class assignment that minimizes misclassification error. 
Geospatial predictive modeling: Conceptually, geospatial predictive modeling is rooted in the principle that the occurrences of events being modeled are limited in distribution. Occurrences of events are neither uniform nor random in distribution – there are spatial environment factors (infrastructure, sociocultural, topographic, etc.) that constrain and influence where the locations of events occur. Geospatial predictive modeling attempts to describe those constraints and influences by spatially correlating occurrences of historical geospatial locations with environmental factors that represent those constraints and influences. Geospatial predictive modeling is a process for analyzing events through a geographic filter in order to make statements of likelihood for event occurrence or emergence.

Tools: There are numerous tools available in the marketplace which help with the execution of predictive analytics. These range from those which need very little user sophistication to those that are designed for the expert practitioner. The difference between these tools is often in the level of customization and heavy data lifting allowed.

In an attempt to provide a standard language for expressing predictive models, the Predictive Model Markup Language (PMML) has been proposed. Such an XML-based language provides a way for the different tools to define predictive models and to share these between PMML compliant applications. PMML 4.0 was released in June, 2009.