Is your business intelligence up to scratch? Maintain your models

You can bet that your model needs maintenance

Although a good model can last for years, it is wise to continue to monitor its operation. That starts by asking questions like:

• Are there important changes in the outside world?
• Are all data sources still relevant and of sufficient quality?
• Is the model still in line with internal processes, or has the situation changed?
• Is the forecast still close to reality or is it starting to deviate?

This allows you to quickly notice when the operation of your model needs to be adjusted. If you want to be sure that your model is in order, arrange for a major maintenance service in the form of a complete Model Review. All important factors are examined, improved or replaced. But there is more.

Ultimately, it is about recognizing that data science and computational models have a lifecycle. With a beginning, middle, and end—or a redevelopment phase, where the cycle starts all over again.

What does the lifecycle of a decision-support model look like?

Whether it is a simulation, optimization, machine learning, or statistical model—or even an Excel-based calculation model—every model has a lifecycle. It is crucial to ensure that models remain relevant and accurate throughout their operational lifespan, especially when they support daily or weekly decision-making. This reduces the risk of errors, improves decision-making, and ensures your organization remains agile in a changing environment. Furthermore, it helps meet compliance requirements and minimizes risks associated with outdated or underperforming models.

In the lifecycle management of data science and computational models, we distinguish four steps:

1. (Re)development of the model
2. Implementation
3. Monitoring
4. Model review

Step 1: Development or redevelopment: Creating (or updating) the model

The cycle begins with the development or redevelopment of the model. It is important to make the model as complex as necessary but to keep it as simple as possible. Each model is also tailored to its specific use—and thus to the user:

• Which decisions need to be supported? What level of result accuracy is required?
• What assumptions are being made, and are they appropriate for the model's purpose and the environment in which it will be used?
• Which input data sources are being used, and how do you ensure they are clean, relevant, and representative of the reality the model is intended to reflect?

To conclude this phase, the model's operation, the assumptions made, and the source data used are carefully documented. Validation follows, involving tests using historical data and scenario analyses to assess performance. This marks the completion of the model's (re)development.

Step 2: Implementation – Integrating the model into workflows

Once the model is ready for use, the organization must prepare for its effective integration into operational processes. This can range from adopting a simple spreadsheet to implementing complex software. In either case, adequate user training is essential for success. A trial period—such as a pilot or parallel test—is recommended to evaluate the model's performance. Also important: keep stakeholders informed about progress and potential adjustments from the very beginning. This builds trust and ensures the model is used correctly.

Step 3: Monitoring – Keeping an eye on the model

After the model has been put into use, you want assurance that it is functioning correctly and will continue to do so. Monitoring the model's performance KPIs is useful for this purpose, as is staying alert to changes in the environment or input data that could impair the model's functionality. By regularly checking the output and comparing it with expected results, you can quickly identify deviations and take action.

Step 4: The Model Review – Evaluating and adjusting the model

If monitoring reveals that the model consistently underperforms against its KPIs, or if there are significant changes in the environment where the model is used, it is time for a Model Review. This might occur, for instance, if a predictive model’s forecasting accuracy drops. The model may need recalibration; this could involve adjusting parameters or revising the model itself to ensure it continues to perform as intended.

It is also wise to schedule Model Reviews periodically—for example, every three years, depending on usage. Think of it as preventive maintenance: you assess in a timely manner whether the model remains current or could be improved by retraining or refitting it using more up-to-date historical data.

Following the Model Review, the cycle is complete, and a decision is made regarding whether the model requires minor or major adjustments. Sometimes adjusting parameters suffices; at other times, the model may be too outdated, requiring a new model to replace the old one.

In short, proactive model maintenance is key to strategic success and sustained strong business results.

Schedule periodic management and maintenance for your models

Curious about the best way to maintain your models? Lieneke van Boxel would be happy to tell you more about the ideal monitoring and review strategy for your specific situation.

Want to stay up to date with the latest news from CQM? Follow us on LinkedIn or sign up for our digital newsletter. 

_{Image by WaveGenerics from Pixabay.}