Here I want to discuss about how dealing with uncertainty can be rather unintuitive even in a fairly straightforward example.

Suppose that we are facing questions like below:

Sohuld I make a specific investment where the outcome is uncertain to a certain degree x? How much uncertainty should I be comfortable with?

I want to stress the investment here is generic: buying stock, investing in a marketing campaign, hire personnel, give out a credit card etc.etc. But we are talking of a point in time Yes/No decision (I will cover sequential decisions at a later time).

At first glance this is a pretty easy question to answer, suppose that the expected income when the investment pays out is $1 and the loss is $5 when things go wrong, the equation we generally want to maximize is below ( P(x) being the probability of x):

• E(Income)*P(Pay out)>E(Loss)*P(Loss) which, considering P(Pay out) = 1 – P(Loss), becomes:
• E(Income)*[1-P(Loss)] > E(Loss)*P(Loss) which with some algebra becomes (including a function inversion):
  • P(Loss) < E(Income)/[E(Income) + E(Loss)]

Which is simply a statement that we should go ahead taking risks until the likelihood of gain is greater than the likelihood of a loss. The notation E(variable) means the expectation of the variable, i.e. the most likely outcome.

If we have a situation where the expected income when things go right is $1 and the loss is $5 when things go wrong, it looks like our best action should be to dare until the P(Loss) = 1/(1+5) = 16.6%.

This means that we demand 5 wins every one loss 83.4%/16.6% at least (which should be intuitively right, as the cost of a loss is 5 times that of a win).

Now, I want to point out that this analysis already relies on quite some assumptions:

• 1 – That the expected income is independent of the pay out probability. This is a fair assumption but, in general, the relationship plays a role, e.g., you buy a stock option which is priced low, yet the price is cheap since the pay out probability is considered low by the market. In this case pay out probability and expected income (conditional on pay out) are inversely related. But we also have other situations where the opposite is true (like in credit granting).
• 2 – We have a way to estimate a probability of pay out and loss. This is non trivial, how do we manage that? How would different individuals/models come up with different probabilities?
• 3 – We can make that investment multple times. It does not make sense to guide yourself with probabilities and maximising expected utility, if all you can do is play once. Although this is still the “rational” choice by the book, the topic is debated (see Theory of Decisions Under Uncertainty by Itzhak Gilboa – beautiful book though highly technical)

Ok, assuming the above let us continue with this, and I will show you a condundrum.

Nowadays our problem seems easy, let us get a machine learning model and set the threshold to 16.6%, so that whenever the model tells me that the P(Loss)> 16.6% I won’t go ahead with the investment (or bet, or prospect acquisition). I am glossing over the model details, but it will be a model trained on the useful data for a particular use case (e.g. probability of conversion for a marketing investment, probability of default when giving out credit etc.etc.).

Great, I have created a simulation and I have a classifier performing as below:

Let me explain the chart briefly:

Sensitivity tells, given a threshold, how good my model is at identifying Bad Biz opportunities, also called the True Positive Rate (where positive means “I am positive this investment is not a good idea”). So, sensitivity should increase as the threshold decreases as we make the model sensitive to fieble signals of Bad Biz. Specificity, on the converse, is how good is the model at being “specific” with its recommendation, meaning how good it is at avoiding collateral damange by giving me, wrongly, a high likelihood of Bad Biz when we are looking at good investments, specificity increases as the threshold increases, as we ask the model to pull the trigger only when the signals are strong.

Overall, in this example, we have a decent model predicting whether the investment I am considering is Bad Biz, i.e. not a good idea, something that will loose money. The Area Under the Curve (AUC) = 0.82 in the chart above (which is called an Receiver Operating Characteristic, or ROC, chart) tells us that the model is 82% likely to correctly rank two investment opportunities in terms of their risk of being a bad business (Bad Biz). This is a global assessment of the model performance across thresholds, and takes into account the sensitivity and specificity we have achieved. The higher the AUC the better the model (on average with respect to the thresholds).

Now, if I ask a common algorithm in R (a statistical programming language) to provide me with the best threshold to guide my strategy, I get below:

> # Best threshold for investment
> coord1 <- min(coords(
+   rocobj,
+   "best",
+   input = "threshold",
+   ret = "threshold",
+   best.weights = c(5, sample_prevalence)
+ ))
> coord1
[1] 0.2466861

So it tells me that I should go as far as 24.6%! That’s pretty different from the “theoretical” threshold we have previously calculated, that one was ~16.6%, what’s going on?

What is going on is that we were not thinking statistically earlier on, but rather probabilistically (mathematically), which is something to consider when it comes to any decision in the real world.

When it comes to actual decisions from models, we need to deal with the model as it is given its actual performance, which is tied to the sample of data we worked with, and how well that data represents “now”. In fact, thinking about it, what happens if your model changes or improves?, or if there are no bad investments out there in a given period? Should we always blindly use the 16.6% probability of Bad Biz? Shouldn’t we react to the actual facts at hand?

The underling issue is that the utility threshold derived theoretically is correct, as mathematics/probability is always correct, nevertheless reality is messy and the score our model gives us is not a probability, but rather a statistical estimate of a probability. In fact the model gives us estimated odds, and odds also need to know the underlying baseline (this is key for the initiated, classifiers scores are not probabilities, but rather they are random variables themselves!).

The clearest way to articulate the actual, empirical outcome, is to use the concept of a policy Net Benefit, which simply measures what value I can expect out of my policy in terms of benefits (from good decisions) and costs (when we get it wrong).

What is the key Net Benefit equation? For a classifier driven policy, depending on threshold x, it is the following:

Net Benefit(x) = TPR(x)*Prevalence*LossPrevented – FPR(x)*(1-Prevalence)*IncomeSuppressed

TPR stands for True Positive Rate, which is simply the number of hits on Bad Biz beyond model threshold x and FPR is, conversely the False Positive Rate, which means that those are investments that are good, but we let go at the threshold x. Prevalence is the general ratio of Bad to Good Biz, regardless of my policy, i.e. if I had no policy that’s what I would get in terms of average bad to good decisions.

So, what will the Net Benefit of my policy be whether we use the expected utility threshold, or the value that the statistical software (R) algorithm gave us? See below:

It turns out that R is actually maximizing the Net Benefit equation, which is great, as it is aiming at maximing our expected profits “all” things considered, including the model we are using, and its performance (the optimal threshold can also be derived analytically with some calculus, but won’t go there this time).

The difference in NetBenefit between the two thresholds is high +9% for the “empirical” threshold, this can be a lot of money in a scaled up context.

To recap the overall story is a bit bendy: we have maths, which is correct, and yet, the best action is rather a different one, which is derived statistically. What connects the two thresholds, Maths and Stats derived?

The connection is the classifier itself and “where” the classifier errors are along the utility derived threshold.

On average the classifier scores are perfect, the % of Bad Biz is 20% and the average classifier score for Bad Biz is exacly 20% but… see below:

Our “Theoretical” utility threshold happens to be around deciles 6 and 7, where the score (grey bar) overpredicts materally, and this pushes the threshold up. For the threshold determination it does not matter the “global” performance of the model, all that matters is how the model performs around the utility threshold. Nevertheless the issue I have presented above is real, even a well calibrated model will always present distortions, unless it is perfect, and the distortions will be larger when the “Bad” event is rare, therefore we need to deal with it “in practice”.

One thing that I can tell is that if your classifier improves, you can worry less about “Theory vs Reality”, see below how things improve if the classifier achieves 0.87 AUC (from 0.82):

The thresholds get much closer together, and the overall Net Benefit of the policy increases + 16%! (Yes investing in Decision Science is probably worth it).

It is of note that the policy value decreases much steeply to the left (when we are conservative) then to the right (when we take more risk) so -> take more risks when the Bad Biz outcome baseline is low.

It is also worth noticing that, in reality, we do not know that an average loss is going to be $5 and a win $1, nor how noisy things would be around those averages, so, things get much more interesting, and yet the relationship between Maths and Stats needs even more expertise to be dealt with.

Next time, I will try to share more about “sequential” decisions:

What if your choice “now” impacts your future ones? e.g. what if refuting a Biz opportunity from someone will preclude us to do some business with them in the future? Now you might need to rethink your thresholds, as turning down a certain Bad Biz, means turning down several future opportunities.

A key tool to deal with that question is Dynamic Programming (recently rebranded as Reinforcement Learning) a wonderful and powerful idea from the 50s.

Due to the nature of my profession I am often engaged in conversations that revolve around the definition, the role and the potential of data science and artificial intelligence. Often the conversation start out of the desire to understand how to use this new bag of tools called “data science”… typical case of a solution trying to find a problem. I am not criticising this approach per se, it’s only natural that a relatively new discipline will go through this exploratory phase, but I am observing that the conversation often starts away from data scientists and data professionals in general.

At the core of those conversations is the desire to outline a data science strategy in a business context.

The issue is that the knowledge necessary to have a fruitful discussion is, sometimes, technical in nature and, therefore, I will now share an important technical aspect of doing data science that I believe even scales up to be a guiding principle for more general business strategies: the bias-variance trade off.

It is a fact of statistical learning that the error of predicting models is the sum of an error due to bias, and an error due to variance (note: variance of the model not the data).

In other words our predictive models will either be too stable and be wrong because of strong assumptions (BIAS) or too unstable due to sensitivity to data points (VARIANCE).

You could also use the terms under-fitting and over-fitting, but fitting sounds technical as well.

The drawing below shows where some popular data science techniques stand in this two dimensional scale (bias & variance):

Old traditional methods often have strong assumptions (like simple linear models), yet provide decent and, most importantly, coherent results when the data is updated. Whilst some of the most modern methods try to dispense as much as possible of assumptions and rely completely on data, yet can produce nonsensical results if the data varies too much, or if the data does correctly represent nonsensical behaviour.

It is also then consequential that low-bias methods need much more data (unless the data is highly structured), and can’t be used lightly in contexts where it is the objective to model general behaviours (like forecasting).

Historically we see then that businesses have relied on the two types of approaches being performed by different units. Subject matter experts providing forecasts (for example) and insight analysts providing complex numerical tables and charts. The first category being (generally) formed by biased experts (e.g. the product will sell 10% more if priced down by a fifth) and the second by research types looking impartially at the data (the chart says that your customers are already unhappy with the price yet buying your product). Inevitably, in several occasions, it turns out that the SMEs are right if the price is changed exactly by a fifth but terribly wrong otherwise, and the researchers have drawn conclusions from a small sample of customers with unclear methodological issues.

The drawing below sort of summarises the above paragraph:

Now, when I think “what can data science do for this organisation?”, my technical brain somewhat answers: it can help the organisation strike a better bias-variance trade off! and this actually means business transformation. Data science can bring together the insight and product teams, data science can also bring management close to customers and partners, and all this by simply striking a better balance between a business instinct toward common practices/assumptions (bias) and a business need to react quickly to new information (variance).

We can easily see that the two dimensions are not bad at modelling decision making in several other aspects of life, e.g. will I vote the party I always voted for (bias/assumptions) or stick strictly to the manifesto commitments (variance/data).

It is also a good framework for prioritising items in a data science transformation (net of business value), e.g. if you don’t have customer data, start collecting it and keep relying on smart and experienced marketers, whilst if your products data is in a good place start developing advanced models on cost effective production and quality control.

Data science is pretty much a discipline born out of the growth of available data and, therefore the possibility to move away from bias. Ignoring this fact when designing a data science strategy is akin to ignoring what a long catapult could do in the middle ages.

You can lose the war for it.

When I was studying Physics at University, I deeply fell in love with one “quantity” or “concept”: that of the ENTROPY of a system.

Perhaps you have not heard of it, but it is so important that in some ways it relates to:

1. Time itself
2. Disorder
3. Information
4. Energy
5. Optimisation
6. Economics and much more…

I find it incredibly interesting that the time evolution of the universe in a direction (universe from the Latin means one direction…please do not think of the boy band!) rather than another, is still a fundamentally unknown property of reality. In fact it appears that systems processing information (like us), wouldn’t perceive time if it weren’t for a particular tendency of Entropy to increase (also, we would be immortal too!).

Closing the Physics parenthesis, in the context of this blog post, I can tell you that I will see Entropy as a measure of “disorder”, and also as a measure of “information” contained in “something”.

To be more explicit, I see something possessing a high level of disorder if, by changing it slightly, it doesn’t change the way I see it. For example, if your desk is a mess (high disorder), I can move some paper from the left to the right, and I still basically see a messy desk. If your desk is tidy (low disorder) and I put a t-shirt on it… I easily perceive that something isn’t as it should.

Also, when I refer to the information possessed by something I mean pretty much the number of “meaningful” answers to straight yes/no questions that something can give me. Here “meaningful” is not defined but I hope you can follow.

What does this have to do with fraud analytics?

Let me put it this way asking you a couple of questions:

1. Do you think that when a certain system (you could think of a payment processor infrastructure, or a trading desk) is being cheated, will the system be more or less tidy?
2. Do you think that the information of a system where fraud happens is larger or smaller than a normal system?

When fraudster do act, they do not act randomly, they act on purpose and follow patterns, therefore your system could show signs of being “more tidy” than usual. For example, processing several hundreds payments of the same amount, or seeing traders all following a particular investment strategy (perhaps suggesting insider trading?) might not be a natural state of affairs.

When you see something following a pattern, you instinctively think that there is SOMEONE behind it… in other words it cannot be random. This is equivalent to say that Entropy tend to be high, unless we work to make it low (e.g. sorting out your desk).

This is when Entropy can come in and help fraud analysts monitor the system and see if too many patterns are emerging.

It can get even more interesting since we can also calculate the Entropy of X given Y. Therefore we can also analyze relationships with all the weaponry that statisticians can use to establish relationships.

Let’s look at some numbers.

Here’s a vector of random numbers: 52  31  22  52 100  46  11  24  77  21

Here’s a vector of not so random numbers: 10  10  10  20  20  20  30  30  30 100 

Using R (the statistical computing language), for example, we can calculate the Entropy of the random collection of numbers. We get 2.16 (don’t care about the units for now).

If we calculate the Entropy of the not so random vector we get 1.3.

But let’s now add some additional information and context. Let’s consider the vector of not so random numbers as the amount that was withdrawn on certain days. Let’s also look at the days: Mon, Tue, Wed, Tue, Wed, Thu, Thu, Fri, Sat, Sun.

Now what’s the Entropy of those cash withdrawals given the days they were made?

o,4

That tells us that this is a pattern, not necessarily a fraudulent one, but if we have a hypothesis over the average Entropy of cash withdrawals we could somewhat understand if a Bot/Fraudster has brought order where there shouldn’t be!

Overall, thinking of utilising Entropy in any analytics related matter is a fascinating example of the variety of tools, and sources of inspiration, that can help data professionals (and data driven organisations) in achieving their objectives.