CQM prepares NS for audit of new passenger numbers model
NS considers it important that a traveler is transported comfortably to their destination during a train journey. One of the KPIs that addresses this is the likelihood that a traveler can find a seat during their journey. This so-called Seat Availability KPI is a concession KPI for NS. This means that NS's performance is assessed by the government based on the value of this KPI. If NS wishes to use a new model for calculating passenger numbers, it must be able to demonstrate that the performance assessment will not be affected by this. CQM prepared NS for a crucial audit.
More
rely
Less
manual measurements
Finished
for the future
Over the years, CQM has built countless quantitative models for countless clients in diverse application areas. Increasingly, we are also asked to review models that clients have created themselves or commissioned. For instance, NS had been working with the ROCKT model for calculating passenger numbers for years. This model was built by NS's own data scientists. Over the years, various models for determining train passenger numbers were built in multiple locations within the NS organization. NS considered this an undesirable situation in itself. However, the most widely used model, ROCKT, was also due for replacement.
Calibrations
'ROCKT worked fine until recently,' says business consultant Sven Barmentloo of NS Network Development & Design. 'But it was no longer suitable for modern times. For example, the model could not properly handle the latest developments in the OV-chipkaart (public transport smart card). In addition, specific travel subscriptions, checking in with a debit card, and fare evasion could not be processed. To still be able to include those figures in the counts, we had measurements taken on the train. People from a research agency regularly walked through a number of trains to scan passengers' tickets. With these samples, we were able to calculate the passenger numbers on all trains.' However, Corona threw a spanner in the works for NS in 2020,' according to Sven: 'The researchers were no longer allowed on the train. As a result, we could no longer properly calibrate the model. This prompted a few creative minds to start developing a model without reliance on measurements on the train.’
FLUX
NS's data scientists built a new model that can be used throughout the organization and is prepared for future developments and travel products: FLUX. The main input for the model is all OV-chipkaart transactions: travelers checking in and out at the stations. Single ticket sales and conductor scans are used to improve the model. All travel products can be processed, eliminating the need for calibration. The model can even predict incomplete journeys. 'Counting travelers is not as simple as you might think,' says Sven: 'A journey does not always proceed from A to B in a standard way. Some people check in and forget to check out, or vice versa. Others have a cup of coffee at the station before or after the trip. FLUX is able to make the best estimate of what the complete journey would have been based on past travel patterns. And, for example, whether a traveler prefers to travel by Intercity or a Sprinter.' All this information helps us accurately calculate passenger numbers and adjust the deployment of our trains accordingly, so that we can guarantee the right chance of a seat.
Prepared together for a crucial audit
Just before the ministry audit, we were able to go through the steps and conclude that the new FLUX meets the requirements for calculating the Seat Chance KPI.
Sven BarmentlooBusiness Consultant Netwerkontwikkeling & Design, NS
Audit
The question is: if FLUX works so well, why did NS need CQM? Sven: ‘The Seat Availability for train passengers is a concession KPI with established floor and target values. It is therefore important that NS reports on this correctly and in the right manner. The Ministry of Infrastructure and Water Management (I&W) therefore wants to be certain that the Seat Availability KPI remains the same when using FLUX instead of ROCKT, and that NS's performance can continue to be assessed in the same way. We had to be able to demonstrate this during an audit by the Ministry at the end of 2023. Initially, we had the ambitious plan to conduct a large internal audit with CQM, because CQM has extensive experience with NS and was previously involved in the development of FLUX. However, this proved unfeasible in terms of scheduling, so CQM proposed an alternative: a step-by-step plan that we could execute ourselves once the system was ready. This allowed us to prepare perfectly for the audit.’
Step-by-step plan
The specialists from NS and CQM held joint sessions to develop the step-by-step plan. CQM's consultant Heleen van Beek was the project leader on behalf of CQM: 'First, we ensured we knew exactly how the Seat Availability KPI is calculated. This allowed us to analyze the impact of a different model. It was fine if the use of FLUX caused some changes to the calculations, as long as we could adjust the floor and target values accordingly so that the final assessment of the Seat Availability KPI remained the same.' Together, over four intensive months, NS and CQM developed a step-by-step plan with statistical methods, enabling NS to perform the calculations and comparisons itself. Sven: 'Just before the Ministry's audit in October and November, we were able to go through the steps and conclude that the new FLUX meets the requirements for calculating the Seat Availability KPI. In early December, we received the Ministry's approval, and since January 2024, we have been reporting Seat Availability based on FLUX. A wonderful result.'
New: Model Review
The benefits of the new model for NS are significant, says Sven: 'With FLUX, we are ready for the future. If we add more travel products and methods, such as checking in with your phone, that is possible in FLUX. FLUX is an AI model that incorporates changes in travel patterns over time into new predictions. Moreover, the new FLUX has a major advantage: we no longer need to take measurements on the train.'
For CQM, the success of the FLUX project has prompted the reviewing and, where necessary, improving of existing customer models into an official new CQM service. The logical name: Model Review. Ideal for organizations that want to know if their model still aligns with their goals and if there are ideas for improvement.
Do you also want certainty about the quality of your model?
Get in touch. We will create a step-by-step plan that fits your model and organization perfectly. Just as we did with great enthusiasm and as one team together with NS.
Make an appointment with Lieneke; she will be happy to help you further!
Additionally, if you would like to stay up to date with the latest news from CQM, follow us on LinkedIn or sign up for our digital newsletter.
