Evaluating a Betting System's Performance (Part One)

It's been quite a while since I last had a blog entry which was not a system update so here is (hopefully) some food for thought.

Surfing the Internet, visiting various betting websites and forums (or should I say fora), I've stumbled across countless betting systems and strategies. These systems consist of not only the tipping part but also different, and sometimes quite elaborate staking plans. An obvious question which arises is "How does one compare these systems?"

I have always been a firm believer that forecasting / prediction models, irrespective of the way their predictions are formulated, should be monitored, evaluated and adjusted; in other words they should be judged. Only by judging a forecasting tool one can hope to improve it. What I am not exactly sure, and please if you have any views do share them, is the way this should be done. You see, everybody looks at the yield and of course this is a measure of how good the system is with this particular set of results. But it's also important to acknowledge that part of a system's performance is due to luck. In other words, if it was possible in a way to re-create all matches in a season, with the same parameters as at the time of the match, what would the system result in, given that not all matches would end up with the same outcome? Is there a way to utilize statistical theory in such a way to account for this uncertainty an create a more robust measure than yield itself? The million dollar question, I guess.

Internet research on this issue leads quite often to the notion of χ²-testing. The idea behind this test is to compare, with some sort of statistical confidence, the number of a system's actual (or observed) winners to the corresponding expected number. For example 15 bets at even odds would yield an expected number of 7.5 winners, but the actual number of winning bets might be 9. Is this sufficient evidence to suggest that the system is performing better than what would be expected by sheer chance? What if the actual winners were 10? 12? All 15?

Well, the χ²-test provides a way to measure this by looking at the probability of achieving the observed result (i.e. the number of actual winners) or a better result by chance. If for example there is a 30% probability of hitting 9 winners or more (given the odds at which the bets were taken) out of 15 simply by chance then the system is not performing necessarily well. As soon however, as this figure is reduced, it means that the system is hitting the winners not necessarily by chance but rather by exploiting value through mis-priced matches. Anyone who has studied any introductory statistics course will tell you that the figure that statisticians are interested in, when testing hypotheses, is the 5% so in effect, according to this test, one would be happy if the χ²-test returned a probability less than 5%.

My personal opinion is that the χ²-test is not necessarily a good indicator of whether the system is performing well. All it does is to check whether the number of winners is significantly higher than what one would expect by sheer chance, but it does not consider the odds at which these winners were picked. What if a system's winners came at very low odds, whereas the system underperformed in the high range of odds. This could lead to a case of achieving a high (as compared to the expected) level of winners, yet the yield could be very low or even negative! Some consolation, I hear you say ... Of course the opposite could also be true: a few longshot winners could be enough to lead to a positive yield yet the number of correct picks could be below the expected number. [For the doubters amongst you that such a case is not as implausible as it may sound, feel free to check the relevant discussion on Punters' Paradise forum.]

The search must carry on. In Part Two of this post (to be posted some time in the near future), I'll look into different ways of assessing whether one has a profitable system and to evaluate its performance. Until then take care!

Choosing the right tools

Having now established two of the types of models that can be used for this project, we now turn our attention to choosing one of them. The fundamental difference between the two is that in the case of the logistic regression model of the match outcome, a 5-0 thrashing is considered (for modeling purposes) as the same as an ugly 1-0 win, while an entertaining 3-3 draw does not differ in any way with a boring 0-0 stalemate. Alternatively, modeling the whole probability distribution of match scores (i.e. assigning a probability estimate for each and every one of the scorelines) allows for these differences at the potential cost of higher estimation burden (possibly accompanied by an increased estimation error).

In effect we have a simple model, and a more complex one. We don't know much about their forecasting ability yet, although one could argue that sufficient backtesting should have clarified if one is significantly better than the other. However, one consideration that should be taken into account is the fact that modelling the whole distribution of goals (i.e. using the complex model) would also highlight cases where other types of betting (rather than 1X2 betting) could be profitable such as the "Number of Goals" market, Asian handicap, or even correct score betting.

As stated above, the chosen model tries to mimic the behaviour of two variables, namely the goals scored by the home team and the ones scored by the away team. In the univariate case, goals in a match can be seen to reflect events taking place at an unknown rate within a time-period (in the football scenario, 90 minutes), assumed to be constant throughout. This is called the "Poisson Process". The probability distribution of goals, that is the probability of the number of goals scored by the end of the match being 0, 1 2 , ... is entirely dependent on the unknown scoring rate. The bivariate case is simply an extension of this scenario, with the two teams having possibly differing scoring rates. But since they are playing each other, they interact, hence it's probable that their scoring rates are not independent, so under this scenario some sort of dependence structure is needed.

Anyone reading this blog will excuse me for not divulging further details for now. When adjustments are needed, more information on the type of model used may be given out but for now, allow me to keep some things to myself... Until then we shall be focusing on what may or may not be useful in predicting football scores.

The project

Today's post is somewhat different. It's the beginning of a long but hopefully successful journey. You see, one of the reasons of having a blog for me was to have a chance of combining a few of my interests, namely football, betting and statistics. Of course, I could do this without putting my thoughts on the blog, but having them here would put some sort of pressure on me to carry on, even if the going gets tough! And who knows, if it proves to be profitable, somebody else may benefit too!

As mentioned before this is simply the beginning, an introduction if I may. The idea was to develop a football match prediction model which would estimate the probability of a Home win, a Draw and an Away win (HDA from now on) of football matches as accurately as possible. These estimated probabilities would be used to decide whether the offered odds by a bookmaker provide value. But I am getting ahead of myself here. Let's take things from the beginning.

In strict statistical terms, modeling the outcome of a football match is a classic example of logistic regression. The "event" (that's the match) can take any of three "values" (H, D or A) with some unknown probability which we are trying to estimate. These probabilities will depend on factors such as the teams playing, home advantage, and possibly other variables such as the form of each team etc. It is our mission to find that model which mimics the real results as closely as possible.

One could also suggest that since the outcome of a match depends on the number of goals scored by each team, a better idea would be to model the match score itself. Such an analysis would provide probability estimates for every single possible result i.e. 0-0, 0-1, 1-0, 2-1, or even 3-5. By adding the appropriate probabilities one would be able to infer the estimated probability of H, D or A.

So to recap, on the one hand we have some sort of a logistic regression model of the match outcome, while on the other a (bivariate) probability distribution model of the match score. There are pros and cons in either case, which probably brings me to George Box's "All models are wrong, but some are useful". Only time will tell, whether the chosen approach is useful.