Microsoft has released an open-source AI agent that can independently generate and verify unit tests before developers use them. The new tool first analyzes a codebase, then determines the best approach for writing tests, and finally verifies whether those tests can actually detect software bugs.
The new “code-testing-generator” is part of the open-source dotnet/skills repository and serves as the engine behind the dotnet-test plugin. Although the name refers to .NET, Microsoft says the agent supports multiple programming languages.
Analyzing the repository first
Instead of immediately generating unit tests, the agent first examines the entire repository. In doing so, it automatically detects the programming language and test framework being used, analyzes existing tests to learn the project conventions, and determines which build and test commands should be used. According to InfoWorld, the agent thus first learns a project’s workflow before adding new tests.
Only then does the agent choose the most appropriate approach. For minor changes, it writes tests immediately, while larger tasks are handled through a more comprehensive Research-Plan-Implement process. For large-scale projects, this cycle can be repeated multiple times to achieve the desired test coverage.
From simple to complex code
The test generator acts as a coordinator. Behind the scenes, it enlists specialized sub-agents for tasks including code analysis, planning, implementation, compilation, test execution, error correction, and code formatting.
According to InfoWorld, when generating tests, the agent starts with relatively simple code and then works step-by-step toward components with more dependencies. In doing so, it attempts to explicitly link every relevant functionality to a test scenario. This allows coverage to be built up systematically.
Verifying test quality
It is noteworthy that Microsoft places a strong emphasis on the quality of the generated tests. The agent not only checks whether the test code compiles and runs successfully but also assesses whether the tests actually perform meaningful checks.
Among other things, it checks whether assertions are sufficiently specific, whether all required scenarios are covered, and whether minor changes to the production code would cause a test to fail. The latter is a mild form of mutation testing, a technique used to assess the effectiveness of test suites.
Microsoft cites as an example a test that would still pass even if a function returned only a default value. The agent considers such a test to be invalid, and it must be modified before the process is completed.
In addition, the agent builds the entire workspace, runs the complete test suite, and verifies that the repository’s standard test command actually includes the newly created tests. This is intended to prevent new tests from being left out of the regular CI/CD pipeline.
Unit tests only
The current version focuses exclusively on unit tests. Integration, end-to-end, browser, and performance tests are currently beyond the agent’s capabilities.
The instructions for the agent also include various security measures. External services must be mocked; tests must not establish network connections or depend on timing; and the agent must not modify production code to make tests pass. Existing test files must not be deleted or overwritten either.
The agent is available as part of the dotnet-test-plugin within the dotnet/skills repository on GitHub. Developers can use it via the GitHub Copilot CLI and through the plugin support in Visual Studio Code and VS Code Insiders, which is currently still in the preview phase.