The market has faced a shortage of qualified data engineers for years. With the growing demand for analytics and AI, the need for such professionals only seems to be increasing. In response, various parties are exploring solutions to attract more data engineers. Nimbus Intelligence is responding with its Academy, which is aimed at addressing the talent shortage. We spoke with founders Rik van Schaik and Martijn Verstrepen about how they set up this traineeship, how it helps reduce the staff shortage, and how the Academy helps to become a data engineer.
The Academy is designed to benefit the trainees, the market, and ultimately Nimbus Intelligence. Participants do have an IT background but need to reach a higher level before they can solve complex data issues. For example, they have experience with languages like SQL and Python, which enables them to go deeper into the subject matter during the traineeship. Thus, Nimbus Intelligence does not start from scratch and avoids basic explanations of fundamental data concepts.
Van Schaik and Verstrepen see in the Academy the ideal path for IT talents to become data engineer. They then have bright career prospects. Demand for data engineers has long been high, and with the ongoing transition from on-premises databases to the cloud, this demand is expected to continue growing. Every company currently operating on-premises is expected to move to the cloud in the foreseeable future. This presents opportunities for future data engineers.
Story continues after frame below.
Helping companies with their data estate
Nimbus Intelligence was founded in 2023 to help companies set up their data estate. The ultimate goal is to help companies become data-driven, as this can increase efficiency and competitive advantage. Possible outcomes include optimized profit margins, better handling of patient data, and streamlining supply chains. Nimbus Intelligence can provide custom-built solutions to realize the ultimate business goal.
Data engineers are especially needed to help with the setup. These are hard to find in the market. Therefore, Nimbus Intelligence has the traineeship to have more data engineers to support customers eventually. The trainees are placed in client locations tailored to the needs of a project and the trainee’s requirements. Nimbus Intelligence wants to give that data-driven client a guarantee that the trainee is of a very high level. That is where deep training in, for example, Snowflake, dbt, or Python contributes. One or more of these skills may be crucial for a particular client project.
The market needs data engineers
The Academy is an essential initiative for that market, thus automatically Nimbus Intelligence. “We help companies extract data from various systems, clean it up, and prepare it for analysis. There is a huge shortage of data engineers, so we started the Academy in January 2023,” Van Schaik explains the market situation.
The shortage of qualified data engineers is a long-standing problem that has only worsened recently. Companies worldwide collect huge amounts of data, but effectively managing, analyzing, and leveraging this data remains challenging. The demand for data engineers is expected to continue to rise as more organizations optimize digital strategies, migrate to the cloud, and seek to base their decision-making on data-driven insights. Data engineers play a key role in collecting, cleaning, transforming, and (re)structuring data. They are the backbone of an organization’s data strategy.
This shortage of data engineers has led to a competitive market in which companies compete for talent. Technological advances require new skills and expertise. The Academy’s four-month in-house training program aims to address these challenges by training over 30 new data engineers annually. While this initiative will not completely solve the shortage, it can potentially train enough qualified professionals to be employed by Nimbus Intelligence. This allows companies that partner with Nimbus Intelligence to outsource their data operations and benefit from a modern data strategy.
Structure of the Academy
During the course, Nimbus Intelligence immerses trainees in data engineering full-time. “About half of the 30+ participants are trained for the Netherlands, which can vary per cohort,” Verstrepen says. The remaining trainees are educated in Madrid and Milan, where Nimbus Intelligence also has offices. The intensive training is tailored to the specific challenges and needs of the local market. Participants gain practical experience with several live use cases in addition to technical knowledge.
All program participants eventually achieve Snowflake certification, among others. The program strongly emphasizes Snowflake, with 80 per cent of the time spent on this tool and 20 per cent on other tools such as Google BigQuery. “We want to specialize in the platforms we support,” says Verstrepen. Van Schaik adds, “The main goal of this training is to provide the market with well-trained data engineers. Customers who deploy the trainees after the program do not perceive that they have just left school. We invest four months in the training, with numerous exercises and practical sessions with the participants.”
Within the program, Nimbus Intelligence also organizes sessions in partnership with Snowflake. Each cohort visits Snowflake’s office for presentations on new features and customer benefits. The collaboration with Snowflake ensures that the trainees become familiar with the technology and understand how it is used in real-world applications with customers. In addition, partnerships have been established with several tech vendors that work closely with Snowflake, such as Coalesce, dbt Labs, and Fivetran, which provide essential technology for data transformation. These partners also provide training within the program so that trainees are always up to date with the latest developments and best practices in the field.
Pathway to employment
According to the founders, what sets Nimbus Intelligence apart is its clear focus on specialization from existing traineeships. “We do not focus on general Azure or AWS, as many others do. Our strong focus on becoming specialists in a specific domain makes us unique,” says Verstrepen. Van Schaik adds that many other data engineering traineeships are often broad. “With us, it’s really about data engineering for effective data analysis. We train people to extract data from systems, transform it and analyze it properly. This dovetails perfectly with what our sister organization, The Information Lab, does on the analytics side.”
Van Schaik compares Nimbus Intelligence’s approach to a contractor renovating a house: working with experts from different fields. Nimbus Intelligence also works this way with experts in data visualization and core data infrastructure.
After completing the traineeship, participants work with Nimbus Intelligence clients, gaining experience with four clients over two years. “This builds a resume that shows they have worked in various customer environments,” Verstrepen explains. “We also have a large network through The Information Lab, with hundreds of employees who work with data daily.”
This hands-on experience with customers allows participants to become technically savvy and gain practical insight into different business environments and challenges. “If you encounter problems with a customer, we don’t expect you to solve them alone. We have the resources and the team to work with you to address these challenges,” Verstrepen says. “Sometimes that’s a push in the right direction, and sometimes it’s working together to see how we can solve the problem.”
Future for data engineers
Verstrepen and Van Schaik are optimistic about the future of data engineers. The continuing demand for qualified professionals in the data market, combined with Nimbus Intelligence’s hands-on training, offers Academy participants excellent career opportunities. “Our trainees are not just graduates with no practical experience,” Van Schaik emphasizes. “We work on projects that cover all aspects of data engineering.”
With its Academy, Nimbus Intelligence answers the growing demand for data engineers by combining theoretical knowledge, practical skills, and specialized training. The traineeship is a springboard for anyone aspiring to a data engineering career. This is through theoretical knowledge and practical experience, where all facets are included. This is not only about technical implementation but also stakeholder management, where a data engineer aligns what exactly had to happen. After all, a data engineer goes beyond technology and, with the right stepping stone, should be able to deal with the complex and dynamic environments they will be working in.