Today everybody is talking about predictive analytics, and itâs undeniably desirable to be able to predict the behavior of customers, employees, voters, or other sources of uncertainty. However, while predictive analytics are always difficult, they are particularly difficult in the current economic crisis. If youâre trying to predict the future behaviors of customers based on past behaviors, for example, keep in mind that customers are no longer behaving typically. The particular products, prices, promotions, and places that persuaded customers to buy in the past may no longer be effective. The fact that almost everyone agrees this is an unprecedented economic climate means that the models created during more normal times no longer apply. Of course, if you are using predictive models to predict things that donât involve humans or the economy, you can ignore most of what Iâve said in this paragraph.
In times of dramatic change, reactive analytics may be safer and more appropriate than predictive ones. Reactive analytics result from trying something, and seeing what happensâwhat the reaction is. The analytics come in because you canât be sure that what happens isnât random variation unless you confirm its statistical significance.
Some call this approach âtest and learn,â and indeed it often involves formal (or semi-formal) testing approaches. There must be a control group, and subjects are randomly assigned to the test or control groups. The number of subjects must be sufficiently large to produce high confidence that any differences between the test and control groups arenât random.
I wrote an article about these tests and how best to perform them in the February issue of Harvard Business Review. I didnât think to call them âreactive analyticsâ when I wrote the article several months ago, and I didnât realize how appropriate they are for bizarre times like these. But even reactive analytics have their limits. If you did tests several years agoâor anytime before this crisisâyou might want to perform them again. The âlearnâ part of âtest and learnâ only applies to circumstances that are similar to those when the initial testing was performed.
If what you learned from testing in the past has anything to do with human economic behavior, itâs time to re-test.