4 min Applications

How Microsoft test the quality of Windows updates

How Microsoft test the quality of Windows updates

Microsoft gives more insight into the tools it uses to improve the quality of its Windows updates. Data plays a central role in this.

The Windows 10 October 2018 Update (1809) did not run very smoothly. Four months after launch, the update still only runs on 21 percent of all Windows 10 systems. Because of all the problems, Microsoft was able to count on criticism about the quality requirements it imposes on its updates for release. In response, it now provides more insight into the process it uses for this purpose.

According to Jane Liles and Rob Mauceri of Microsoft’s Windows Data Science team, each release is approached from the same question: Is this Windows update ready for customers?

It is intended to confirm that automatic and manual tests have been carried out, before we evaluate the quality through diagnostic data and feedback-based readings, they write in a blog post.

Before a build reaches Windows Insiders, it is preceded by a process of internal quality testing. After an automatic check, Microsoft’s own engineers use the software to perform stress tests. Only when a Windows update passes these checks is it rolled out to the first public testers.

Repeatable metrics

Reliable and repeatable metrics are required to determine whether an update is ready. We look at the statistics several times a week as part of our normal rhythm to better understand the impact of code changes, the data scientists write. By the time we are ready to roll out to our customers, our statistics should be at least equal to or higher than the quality levels of the previous release.

To determine the quality of an update, the team looks at the unique number of monthly active devices. They evaluate how the upgrade process went and what the overall health of the user experience is. Then it looks at concrete events, such as the number of times that a successful wifi connection was made, a PDF was opened in Microsoft Edge or logged in with Windows Hello.

RQV Dashboard

The Release Quality View (RQV) dashboard, which maps out more than 1,000 different measurements, is central to the collection and analysis of all this data. The dashboard is used during the internal and public test phases as well as when an update is generally available.

Based on this dashboard it is determined to what extent an update is ready for launch and which bugs need to be fixed first. Several times a week, meetings are organized to evaluate the quality of the software. Starting with the problem areas with the lowest scores, Liles and Mauceri tell us that the list of items whose results are furthest away from their targets is a long one.

Between meetings, engineers and managers regularly evaluate their own measurement results. They search for errors using analysis tools to correlate cohorts of failure conditions with problems in the code. This allows them to diagnose and solve problems more easily.

Fixes that assign engineers to future builds are tracked by the system, so reviewers can see when a fix is delivered via a new build and monitor the impact as the build moves through its normal validation path, it sounds in the blog post.

Customer feedback

The data scientists say they are proud of the validation process they have set up, but at the same time acknowledge that there are still important points of work to be done. For example, they acknowledge the need to invest even more in customer feedback to help us identify gaps or inconsistencies in our diagnostic-based measures.

The fact that the processing of customer feedback can still be improved was also evident in the Windows 10 October 2018 Update. After launch, it was plagued by several bugs, some of which had been noticed by Windows Insiders months earlier. However, the reports did not get high enough on Microsoft’s feedback tracker, so they were overlooked or not addressed quickly enough.

This news article was automatically translated from Dutch to give Techzine.eu a head start. All news articles after September 1, 2019 are written in native English and NOT translated. All our background stories are written in native English as well. For more information read our launch article.