Microsoft creates data compression algorithm and hardware for the cloud open source

Microsoft has created its algorithm, hardware specification and Verilog source code for compressing data in its Azure cloud open source. All information is contributed to the Open Compute Project (OCP).

Large cloud providers today store huge amounts of data. To make this possible, the data is compressed, just like in the past, zip files were created from files on computers in order to place them on floppies or CD-ROMs. These systems are often well-kept secrets, but Microsoft has now decided to make its information open source.

Project Zipline

Specifically, it concerns Project Zipline, reports TechCrunch. In their own words, this project can achieve compression ratios twice as high as the standard Zlib-L4 64KB model. To achieve this, the algorithm and the hardware implementation were specifically tailored to the large datasets that Microsoft sees in its cloud.

Because the whole system works at system level, there is virtually no overhead. Microsoft argues that it is even able to achieve higher throughput rates and less delay than other algorithms can do now.

Microsoft further emphasizes that it also contributes the Verilog source code for registry transfer language (RTL). That’s the low-level code that makes everything work. “RTL’s open source contribution to OCP at this level of detail is industry leading,” said Kushagra Vaid, general manager for Azure hardware infrastructures.

“It sets a new precedent for promoting frictionless collaboration in the OCP ecosystem for new technologies and opening doors for hardware innovation at Silicon level.

Collaborations

The system is now used by Microsoft itself in its own Azure cloud. However, the company also starts collaborations with others in OCP. These include Intel, AMD, Ampere, Arm, Marvell, SiFive, Broadcom, Fungible, Mellanox, NGD Systems, Pure Storage, Synopsys and Cadence.

“Ultimately, we expect Project Zipline to find its way into various markets and user models, such as network data processing, smart SSDs, archive systems, cloud appliances, IoT and edge devices.