The pitfalls of productionising Machine Learning…and how to overcome them
We know there is a lot of hype around machine learning (ML), and we’re all aware there’s great value to be generated from this technology. However, it is only worth pursuing if a demonstrable case can be made for its sustainable business value.
As with any new technology, realising that sustainable value isn’t straightforward. Gartner estimates organisations who have implemented ML solutions have quadrupled in the last 4 years but, 54% are struggling to get past the barriers to efficiently embed ML solutions.
At Mudano, we help organisations integrate ML into their workflows, productionising ML so they can see a long term return on their investments. Here’s how we do it.
If it was easy, everybody would be doing it
The quantity of data that organisations are creating is exploding. This is offering huge opportunities in analytics. Forward-thinking organisations are looking to exploit this data via Machine Learning.
Traditional approaches, however, are causing roadblocks that prevent organisations from gaining sustainable value from their ML investments. Here are some common problems and a few ideas about how an organisation can overcome them.
Problem 1: Lack of transparency
One barrier that has stalled ML investment and collaboration is that ML models are often developed by data scientists in isolation. This is one example of a wider problem for organisations who try to link technical work with ongoing design and product development.
Development in isolation can result in Black Box solutions. This means that when the product is handed over to platform engineering teams, ‘Glue Code’ is written to update the model, instead of updating the source code, due to limited knowledge and expertise. Additionally, a lack of transparency into the model training and testing process means users can’t explain and trust the results.
These issues make it difficult for organisations to justify outputs of ML solutions to customers, regulators and internal compliance functions. They create roadblocks for those seeking to make a demonstrable case embedding ML.
Solution 1: Embed best practice early
To counter the problem of siloing, organisations should embed ML best practices into their existing Data Science teams and develop explainability of model outputs at the individual decision level. This means that highlight features strongly contribute to each decision. It’s also important for bias and fairness metrics to be used that are in line with an organisation’s data ethics and privacy policies.
Data science teams should also be upskilled so they adhere to ML best practices and established standards. And design teams should record model execution at every stage, as it’s always better to have data and not need it than to need it and not have it.
Problem 2: Traditional models aren’t built to be connected
One major barrier to productionising ML models is the architectural differences between:
- The model training (development)
- The model execution (results prediction)
- The end-user applications platforms (results delivery).
If these three elements aren’t connected, the product falls short. This occurs because experimental models are often developed on sandbox infrastructure with advanced ML capabilities, but equivalent capabilities are not available in production environments. Also, non-ML infrastructures lag ML infrastructures, lacking the required sophistication.
Solution 2: Build for flexibility and scalability
To overcome a poor build, architectural design patterns should cater to both model execution and results consumption. These patterns should always be proved through business use cases, with data and results being tested, learned from and iterated upon.
Organisations can do this by establishing deployment patterns that allow engineering teams to select a production route based on business and technical requirements. And they can leverage cloud-based technologies for greater flexibility.
At Mudano, we believe that prototyping and iteration lead to sustainable, long-term results. It takes the guesswork out of productionising, proving new patterns for your data science and engineering teams to adopt, future-proofing your investments for the long term.
Problem 3: Incompatible methods lead to short term growth
It’s fair to say that developing basic ML applications is relatively straightforward, but maintaining them whilst in production is hard. Due to the distinct characteristics of ML applications, even well-known software engineering practices can soon be inadequate.
ML applications require different tools and workflows to tackle changes across data, model and code and, as software delivery methods evolve over time, ML delivery methods are evolving too. ML solutions often fall victim to the problem of ‘changing one thing and everything changes’, therefore model maintenance isn’t straightforward either.
Even with the right build in place, without ML expertise, applications can grow outdated and the return on your initial investment can be lost for good.
Solution 3: Across your organisation, think like DevOps
It’s vital that organisations establish cross-functional ways of working across the ML lifecycle. One way this can be done is by uplifting computer science skills and implementing proven DevOps processes that drive organisation-wide automation, quality and discipline.
In typical DevOps fashion, organisations should also seek to establish reusable deployment patterns for Data Science, Application Engineering & Platform Support teams. This approach should be implemented by DevOps and CI/CD best practices must be adopted by all teams, enabling all parts of an organisation to support production-grade ML solutions.
Finally, a “Release any time” mindset is vital. This ensures that models are always ready to be deployed, driven by a business decision rather than a technical one. Again, the key is to prototype quickly and learn fast, as this is what leads to long term sustainability and growth.
How we can help
At Mudano, we’re experienced ML practitioners, so we know that building an ML business case isn’t easy. We know it’s even harder to effectively implement ML best practises and procedures. Right now, most organisations are encountering the same pitfalls. But the good news is that we’ve located these pitfalls and we’re here to help your organisation avoid them. Whether it is referred to as ‘Productionalising Machine Learning’, ‘Scaling AI’, “Industrialised Artificial Intelligence” or anything else, the premise is the same. You need to go beyond proof of concept in order to realise value at scale.
The correct integration of ML can pay dividends for your organisation, especially when many of your competitors are struggling to gain the sustainable value that organisations crave.
By working with Mudano, you will overcome the key barriers to productionising ML models and gain sustainable business value from your investments. This will give you a competitive advantage, helping to reap rewards for years to come.
If you’d like to discuss any element of ML implementation, we’d love to hear from you. Drop us a line to talk about your organisation’s approach.