Analyzing the Analytics

With the explosion in sophisticated machine learning, managing, tracking, and monitoring models is going to be our most formidable challenge to date.

Yasmeen Ahmad
Yasmeen Ahmad
28 février 2017 3 min de lecture

So… the global financial meltdown.

With all of our sophisticated analytics and modelling, shouldn’t we have been able to avoid the crisis?

The license to print money - revoked

The financial industry had lauded the analytical model known as Gaussian copula function as breakthrough for the modelling of complex risk, leading to unprecedented growth. Then, suddenly, in 2008 the financial system collapsed creating huge losses and disruption – the level of which, even today, is hard to comprehend.

Was it really that sudden? Why didn’t the analytics see it coming?

The truth of the matter is that early-warning signs were ignored. Worse still, signals were not monitored and analytics were left unchecked for more than a decade while markets slow-marched towards the abyss.

How do we learn from this catastrophe?

By automating model execution with measurement and monitoring in place. By ensuring that slow-changing conditions are identified and acted upon. By being proactive.

And talking of proactivity, which would you rather have? A team of innovators or a team of mechanics? Mechanics respond to dashboard lights with diagnostics and repair programs. But R&D teams innovate, designing engineering solutions that prevent the glowing dashboard lights in the first place, reducing costs and maximizing performance. In the same way, analytical teams should be innovators, ensuring that analytical models are not outdated, underperforming or in the worst-case scenario, wrong.

Gaining control and implementing governance is essential. With access to more granular data and sophisticated machine-learning analytics, it’s possible to have a model factory running simulations or forecasts for each customer, each product, and each store. Companies like eBay run thousands of tests on their digital platforms, concurrently, with every customer becoming a test subject. Wells Fargo can predict behavior within seconds and respond with a personalized offer within minutes. This opens up the possibility of modelling the world like never before, making predictions and acting on insights.

However, with the explosion in sophisticated machine learning, managing, tracking, and monitoring models is going to be our most formidable challenge to date.

Measure, monitor, and maintain

The basis for any management and monitoring requires a clear understanding of what the expected outcome is (KPIs can help), predicted by our model. However, control metrics are also needed to measure the validity and accuracy of predictions. These measures provide a model interface; a communication channel that reports input-feed quality, algorithm performance and prediction accuracy, triggering proactive action and maintenance where required. This ensures models continue to deliver business results consistently.

The cause of remedial action is not usually a massive failing. Instead, slowly-changing business and environmental circumstances affect the underlying assumptions that went into building the original model. Dramatic changes such as the introduction of a new product or service will get enough traction to initiate corrective action, immediately. However, subtle changes don’t attract the same level of attention. Shifts in customer behavior and market trends reveal themselves slowly (it can take years sometimes), and these small levels of degradation provide below-par results which lose the company money.

Constant review and recalibration ensure models are refreshed with old, outdated variables removed and new datasets introduced, generating variables that increase the knowledge input to a model and improve predictions.

Weekly, monthly, or quarterly, report automation allows constant baseline comparison, enabling experimentation – a process of trying hundreds and thousands of models to find the most accurate algorithm. Comparison with models from the past may also show that the performance of previous algorithms has been improved by access to increased data granularity and new variables. Previous models which were regarded as being too far ahead of their time can be dusted off and reintroduced.

Agile campaign management and a commitment to continual trial & error are essential. Letting models run feral can be bad news though. Take Target for example. The retailer identified a teenage pregnancy through predictive analytics and sent the girl appropriate marketing material, leaving an unwitting father to discover the truth from a coupon and deal with the consequences. Or Knight Trading’s $440 million loss, a result of mistakenly launching an incorrect version of an algorithm. A measurement and monitoring framework should have interceded this model to create alerts and modify business actions.

Managing models through automation can prove enormously beneficial, but strict process and governance have to be observed to maintain control. The lineage of models and the understanding of their development explain the impact that models have on the bottom line.

Rather than being an afterthought, these activities should always be front of mind; repeatable processes, not just one-time exercises for model construction. They’re just as important as insight generation and therefore analytical teams should focus on validation to enable model execution to build better business outcomes.


À propos de Yasmeen Ahmad

A strategist and change leader, Yasmeen Ahmad has worked on executive teams with focus on defining and leading strategy, driving priorities with a sense of urgency and leading cross-functional initiatives. Yasmeen has held roles including VP of Enterprise Analytics, Head of Global Communications and Chief of Staff to a CEO. Her creativity, ideas and execution have supported organizations to move quickly to deliver on key transformation objectives, including pivots to analytics, as-a-service, subscription and cloud. 
Yasmeen is a strong communicator, well versed in connecting business and technical disciplines. Her keynote presentations, articles and published materials are demonstration of her thought leadership and ability to simplify complex concepts. She is regarded as an expert in the enterprise data and analytics domain, having successfully consulted to deliver multi-million dollars of value within Fortune 500 companies. Yasmeen leads with a passion for being customer obsessed and outcome focused. A strong people leader, Yasmeen has driven change management and people initiatives to foster a culture of growth and continuous improvement. Yasmeen is a strong proponent for transparency, diversity, inclusiveness and authentic leadership. 
Yasmeen has a PhD in Life Sciences from the Wellcome Trust Centre in Gene Regulation and Expression and has studied on executive programs related to Disruptive Innovative and Strategic IQ at Harvard Business School. Yasmeen has been named as one of the top 50 data leaders and influencers by Information Age and Data Scientist of the Year by Computing magazine, as well as being nominated as a Finalist for Innovator of the Year in the Women in IT Awards. Finally, Yasmeen is part of the exclusive Executive Development Program at Teradata.

Voir tous les articles par Yasmeen Ahmad

Restez au courant

Abonnez-vous au blog de Teradata pour recevoir des informations hebdomadaires

J'accepte que Teradata Corporation, hébergeur de ce site, m'envoie occasionnellement des communications marketing Teradata par e-mail sur lesquelles figurent des informations relatives à ses produits, des analyses de données et des invitations à des événements et webinaires. J'ai pris connaissance du fait que je peux me désabonner à tout moment en suivant le lien de désabonnement présent au bas des e-mails que je reçois.

Votre confidentialité est importante. Vos informations personnelles seront collectées, stockées et traitées conformément à la politique de confidentialité globale de Teradata.