Software engineers experience challenges when building a copilot. This is partly due to the relatively new aspect of the technology, but concerns around security and privacy are also surfacing. What six challenges do software engineers face, and what are the solutions?
A copilot must assist in performing a specific task. For example, there are copilots explicitly developed for developers to help write code. This AI uses a specific LLM that contains information about development.
Building such a copilot presents software engineers with six challenges, concludes a paper made by several researchers at Microsoft. They interviewed 26 software engineers who have experience building copilots. Specific questions the interviews addressed were the process followed to build a copilot, the challenges during development and opportunities to adapt the process for tool publishers.
Six challenges
Building a copilot is never the same. This is partly because the phases in the development process are not neatly defined and will always follow a different order. We will discuss the software development process in more detail in a moment. It is important to note, however, that software engineers run into many more problems because of this confused process. The Microsoft researchers grouped these challenges under six problems.
First, creating prompts turns out to be a time-consuming task. This includes the entire development of a prompt, from getting an idea to testing and fine-tuning it. It proves difficult to test prompts due to a lack of data. The testing problem, by the way, is not only experienced by software engineers with prompts. Testing appears to be a general second problem that appears due to the inability to filter out one variable for testing. A test result may be influenced by the prompt or model. Engineers try to solve this with benchmarks, but this requires expertise and is expensive.
Managing and coordinating different data sources and prompts presents a third challenge. Engineers are challenged to create sophisticated workflows that copilots can follow when multiple user queries or prompts follow one another. A solution to this problem has yet to be found, as best practices for working with LLMs have yet to be written. This, together with the lack of benchmarks, constitutes the fourth challenge. The fifth problem is also linked to this, as engineers lack good guidelines to address concerns around copilot safety and compliance. Then again, building these elements from telemetry data is not an option because it causes privacy issues with user data.
The final challenge that the study described is actually a combination of all the previous aspects. For example, the study writes that “the overall developer experience is hampered by inadequate tooling and integration issues.”
Software development process changes
Building a copilot appears to change the normal course of the development process. Four overarching phases can be identified, but they alternate in different orders.
During the exploration phase, developers search for suitable use cases for implementing a copilot and a proof of concept is built. During implementation, inputs and outputs, among other things, are already being compared. There is also time to come up with prompts, which are commands to the AI tool conveyed in writing, visually or with audio. During evaluation, engineers test the prompts and other things from the copilot. During productization, issues such as setting limits on input and output and other safety measures come into play. Here, engineers also ask users how they experience the product and if they notice any areas of improvement.
Developers need more control and data
The study not only focuses on the problems of development but also looks for solutions. To this end, four opportunities were identified for the publishers of the tools for building copilots.
- The tools should support the development of a prompt. That consists in part of a “toolbox” that already contains examples of prompts for general tasks.
- More transparency and control over the information AI uses is needed.
- An automated method of collecting user feedback or measuring copilot performance helps software engineers with testing.
- Engineers need a tool to insert copilots into projects or existing codebases quickly.
Building copilots still faces several challenges to provide software engineers with an easy and enjoyable overall experience. Among the solutions are automation and transparency, but it is also seen that more experience is still needed in general with building copilots to create a community in which software engineers can share experiences. Therefore, the field still needs to grow over time.
Also read: AI coding tools divide developers’ opinions
2 days ago
