Véronique Van Vlasselaer, Joline Jammers, and Kaat Tastenhoye work as customer advisor decision scientists at SAS Belgium and Luxembourg. They help companies with the development, deployment, and governance of machine learning models. It is clear to them that one of the root causes for not having a model in production is that management lacks the experience to envision the launch of a machine learning project. Consequently, they are overlooking a crucial question: what steps do we need to take to complete a machine learning project successfully?
To create business value with machine learning, you have to take five critical steps:
Step 1: Perform a business value assessment
Step 2: Create an analytical base table
Step 3: Develop the machine learning model
Step 4: Deploy the machine learning model
Step 5: Machine learning model governance
Step 1: Perform a business value assessment
It is crucial to start a machine learning project with a business value assessment. Joline explains: “Together with the client, we brainstorm about all machine learning use cases that are interesting based on the pain points they want to solve. We plot the answers in a graph that consists of two axes, yields, and efforts. It immediately shows the customer the low hanging fruit: high yields, low effort. Besides, it also indicates where you need to invest to achieve higher returns. Additionally, we use a machine learning canvas to define: the customer's value proposition, the techniques, and data we will use, the features we will create, and the returns on investments and risks of the project. During the machine learning project, we regularly use the canvas to check if we are still on track.”
Step 2: Create an analytical base table
The analytical base table builds the story of your product or customer based on data. A manufacturer who wants to predict if the quality of a product is good or poor needs to have sufficient and accurate production process data. For example, what raw materials go into the product? What kind of sensor measurements are available during the production process of the product? Are these measurements correct? The same applies if you want to predict customer behavior, e.g., does the customer buy in a physical store or online? What did the customer purchase in the past?
The decision scientist needs to work closely with the business to build the analytical base table. Kaat: “From experience, we know that it takes time to construct an analytical base table. What are the process steps? Which data sources are available, and is the data quality good? Is data missing? After all, a decision scientist needs the right historical data to develop the machine learning model.”
Step 3: Develop the machine learning model
The development of the machine learning model is Joline's favorite project step. “We have access to an extensive library of models, and I enjoy to test which model performs best. It is vital to verify that you don't create a biased model due to data problems that were not yet detected, such as wrong data registration.” Véronique adds: “Building a robust machine learning model requires extensive and open communication between the project team members. What data does the model use? Is the information correct? What does the model predict, and are the outcomes right? Together with the customer, we create scores to determine, e.g., the quality of a product, or whether a bank transaction is fraudulent. We use these scores to predict the decisions of the model and to check if these are statistically and business-wise correct. Last but not least, the machine learning model needs to meet the criteria for deployment and monitoring.”
Step 4: Deploy the machine learning model
Deploying a machine learning model means taking the model into production and using it to make decisions based on new, unseen data. Decisions can be made in batch or in real-time. “A decision scientist builds the best model based on historical data. When this model goes into production, you receive sensor values about your process or data about your customers. It allows you to immediately decide: is the quality of this product right or not, or is this bank transaction fraudulent or not? When the machine learning model is in production, you have to monitor and improve it. A model that works well today, for example, may no longer meet your requirements next year due to business changes,” says Kaat.
Step 5: Machine learning model governance
A machine learning model in production needs maintenance. Véronique: “Machine learning model governance is about monitoring and improving the model continuously. Are the decisions, and the scores that come out of the model, correct or not? Fortunately, it is possible to automate most monitoring activities, and to generate reports so that the decision scientist understands what to adapt: is it the model or the data? In case it is the model, we can check whether one of the challenger models running in the back-end performs better. If so, we can easily swap the models. That means take out the current model and put into production the outperforming challenger model.”
If the model is not performing well due to a changing environment, you have the option to re-train the machine learning model automatically. For example, you discover that the decision taken by the model needs refinement because new business information might imply changes in the importance of the weight of the variables used in the model. In this case, re-training the model means re-evaluating and changing the importance of each of the variables, if necessary. Another option that, unfortunately, does not allow automation is the re-development of the machine learning model. That takes place when changes in the current variables of the model do not let it perform better. Then the decision scientist has to go back to the business and collect additional data sources before being able to develop a well-performing machine learning model.