Measuring developer experience has never been more urgent now that AI plays an increasingly important role. It has also never been more complex. At Atlassian Team ’26 Europe in Amsterdam, we got the chance to talk about this topic in depth with Justin Reock, deputy CTO of DX at Atlassian. Among other things, we discuss how DX’s Core 4 framework wants to redefine R&D productivity measurement. Furthermore, we look at what AI is really doing to the developer experience data.
DX, acquired by Atlassian a little over a year ago, was built on a foundation of academic research. Three prominent measurement frameworks existed for developer productivity: DORA, SPACE, and DevEx. According to Reock, having three separate frameworks was simply too many. Therefore, DX distilled them into a single unified system called the DX Core 4.
“We built a metric framework called the Core 4, and then we built a product around that,” Reock explains. The engine gathers two types of signals: qualitative signals collected through surveys with participation rates of 95 to 100%, and system metrics pulled from source control, CI/CD pipelines, and incident management tooling. These signals are normalized into primary and secondary Core 4 metrics. This gives engineering leaders an aligned, agreed-upon definition of R&D productivity across the entire organization.
The Developer Experience Index (DXI)
A key component of the Core 4 framework is the Developer Experience Index, or DXI. This is a composite metric built on 14 distinct drivers. These drivers cover areas such as documentation quality, cross-team collaboration, production debugging, local development environment, local iteration speed, and incremental delivery. Notably, each driver captures data about the system, process, culture, and environment in which engineers work.
The scoring methodology draws on W. Edwards Deming’s statistical process control theory. This theory holds that 90 to 95% of an organization’s productivity output is a result of the system, not the individual worker.
The DXI data tells us that for every one-point improvement an organization achieves in its DXI, it returns approximately 10 hours per year per engineer back to the organization in reduced waste and loss.
Context switching kills productivity
The qualitative signals are clear, Reock tells us. The inability to achieve deep work kills productivity. It doesn’t matter whether this is caused by excessive meetings, constant interruptions, long build times, or poor documentation. The fact is that context switching pulls engineers out of their flow state.
“Sometimes I’m in my flow state and I’m doing great, and then I have to wait an hour for this build to complete. That’s a context switch,” Reock notes. “Or I need to go information seeking, but documentation is terrible or hard to access.” Crucially, the majority of friction engineers encounter stems from the systems in which they work. It actually has very little to do with people.
DX is not an Atlassian ROI calculator
One of the questions we have for Reock is what the role of DX is in the bigger Atlassian picture. That is, given that Atlassian’s own platform makes up a large part of the systems being measured, is DX simply measuring the return on investment of Atlassian’s own tools? According to Reock, DX has been running for five years, beginning as a standalone company. The product breaks the moment it tries to limit customers.
“The minute we start trying to pigeonhole people into only using a particular set of tools, our product breaks,” he says. Teams using GitLab or GitHub still need their CI pipeline and source control metrics captured. Where synergies with Atlassian tools exist, such as understanding ROI from migrating to Atlassian Cloud or adopting Rovo, DX can measure that, but it is not the primary mission. The commitment to tool flexibility was made explicit all the way to executive leadership during the acquisition.
Complex impact of AI on developer experience
Of course, the impact of AI is one of the biggest talking points nowadays, also when it comes to developer experience. On the positive side, documentation has seen a clear improvement. Reock mentions an uplift of approximately 12% across the industry. Given that documentation is universally disliked and chronically out of date, this is a meaningful gain.
Two other drivers have created a new tension. Code maintainability has improved across the industry as AI assistants and agents make code more readable. However, change confidence, the comfort level engineers feel when making changes without accidentally breaking production, has dropped by almost twice as much as code maintainability has risen.
“The code is more maintainable. I can understand it because I have an assistant, I have an agent, but I don’t trust the output,” Reock explains. Additionally, pull request size has nearly tripled in the last year. All of the metrics need to be examined together to paint the actual picture: code maintainability, change confidence, PR size, and incremental delivery.
Engineers are not the bottleneck
DX data comparing cohorts of heavy AI users versus lighter AI users produces a counterintuitive finding. In many cases, the end-to-end delivery lifecycle is actually longer for heavy AI users, sometimes by 5 to 10%. The coding phase itself is compressed, but the review and deployment phases extend. The latter is driven by higher pull request revert rates and higher defect ratios.
This particular finding has led to examining the full Product Development Lifecycle (PDLC) more, not just the software delivery lifecycle. When the coding phase becomes near-instantaneous, it is clear that the actual delays are upstream. Think of things like ideation and requirements definition, to name a few.
The ROI of AI does not yet exist
Reock is very clear on what AI delivers at the moment. “The reason that we’re having such a difficult time finding the ROI of AI is because it does not exist. Not yet,” he states. In order to get there, we first need to identify and resolve the bottlenecks that prevent the gains from flowing through to delivery.
Reock calls out the code review bottleneck in particular. As the percentage of AI-generated code increases in an organization, DX data shows an almost perfectly linear relationship with time-to-first-review. That is, organizations where AI-generated code represents less than 50% of output see a time-to-first-review of around 35 to 40 minutes. Those approaching 80 to 90% AI-generated code see that figure climb to 80 to 90 minutes. Senior engineers, tasked with reviewing larger volumes of larger pull requests, are reporting exhaustion. This trend is also visible in open source communities, where some maintainers are now explicitly banning AI-generated code submissions.
Developer joy is measurable
The final topic we discuss with Reock is something he has been involved in since his days at Gradle: developer joy. Borrowed from the Mozilla Foundation, the principle holds that developer joy equals developer productivity. There is neuroscience behind it too, he says. Context switching does something to the prefrontal cortex, which accelerates cognitive fatigue. Reducing context switching keeps engineers’ brains in a productive state longer.
DX has data correlating the Developer Experience Index with metrics like change failure rate and defect ratio, showing that less pressure leads to fewer corners cut and higher quality output. Even AI can contribute to this joyfulness: the METR study, despite its methodological flaws, found that engineers using AI tools reported a subjective sense of flow, even when objective metrics told a different story.
As for whether developers will be replaced by AI, Reock’s position is consistent with his data-driven approach: “AI is not going to take your job, but somebody really good at AI will take your job.”
Also read/watch/listen: Williams uses Atlassian wall-to-wall to chase that extra tenth of a second