VS Code Dora Metrics: Cycle Time vs. Lead Time

Just like the number of followers and views on your social media is directly proportional to your fame and influence in the digital world, the count of active contributors, stars, forks, and views on your GitHub repo shows where you stand in the open-source world. And when talking about raging numbers, VSCode immediately jumps to the forefront of your mind.

With a large community of 2083 contributors, 163k stars, 28.8k forks, and 3.3k watching, this community is definitely thriving and doing something right.

What is VS Code?

VSCode is one of the best code editors that offers an exceptional development experience with its sleek design, robust features, and performance. It is built on GitHub’s Electron framework, and also boasts a vast library of extensions to enhance your coding journey.

Interest in this case study was especially piqued after we completed the case study on Electron Framework. As part of our 100 days100 Repos’ Case Studies initiative, I also analyzed the Dora Metrics of VSCode using Middleware OSS.

Also read: Getting Started with Middleware Open Source

What are Dora Metrics?

DORA metrics are a set of key performance indicators that measure software delivery performance, focusing on deployment frequency, lead time for changes, mean time to recovery, and change failure rate. Know more about Dora Metrics here.

VS Code Dora Metrics - Killer Cycle Time and Deployment Frequency

When your deployment frequency is spot on, everything falls into place. Any developer would agree! In an open-source world, where contributor availability can be unpredictable and review times are all over the place, achieving those stellar deployment numbers feels like a major win—and it definitely takes a lot of hard work and discipline to get there! 🙂

The VS Code repo consistently delivered impressive deployment numbers: 174, 125, and an astonishing 314 in July, August, and September 2024.

314 deployments?! Wow! Seriously?! That’s far beyond the standard benchmark for deployments established by the 2023 State of the DevOps Report.

What are they doing so right to achieve this?

To answer that question, I need to share the cycle time involved in taking the PRs from start to finish.

In July: Their cycle time was just 20.63 hours

In August: It was 18.82 hours

In September: It scaled a bit to 36.16 hours

However, even after a slight rise in the cycle time in the month of September they did a pretty good job and stayed within the elite range of Dora Standards set by the State of DevOps Report. A good cycle time ensures that tasks are completed and integrated quickly into the main codebase.

And a good cycle time is directly proportional to a perfect first response, rework, and merge times, which were unbelievably high.

Months	First Response	Rework Time	Merge Time
July	117	112	113
August	150	100	213
September	151	72	83

A relatively low cycle time enables rapid iteration, allowing the team to address issues and implement features quickly as reflected in PRs such as PR #219820 and PR #213233

Also read: Bootstrap: Strong Merge and Cycle Times

Lead Time Reveals Vulnerabilities in Certain PRs

While the overall average lead time aligned well with the standard benchmarks, there are some concerns regarding the lead times of the following PRs:

Enable rendering of math fenced code blocks by mjbvz : 8.60 days
Diagnostics performance telemetry by gabritto : 6.73 days
Update notebook comment view by rehmsen : 11.32 days

The variability indicates potential inefficiencies during initial PR review and subsequent revisions, which, if addressed, could significantly enhance their overall lead time.

What are the unruly factors for these delays?

Approval Delays: The time taken for reviews and approvals may be longer than expected.

Bottleneck in Testing: If automated or manual testing phases are lengthy or face issues, they can significantly slow down the overall lead time.

Dependency Issues: The pull request may depend on other changes or external factors that slow down its integration.

Complexity of Changes: If the changes involved in the PR are complex, they may require more thorough review and testing, extending the lead time.

Communication Gaps: Miscommunication or lack of clarity among team members can lead to delays in decision-making and implementation.

Also read: What Developer Productivity Metrics Should An Engineering Manager Really Track?

VS Code Dora Metrics: Good Cycle Time, Killer Deployment Frequency, Concerning Lead Time

The VS Code repo is on the right track, but they could significantly speed up their delivery by harnessing the strength of their community. Actively involving more contributors in the review process, bug fixes, and feature additions would help them work at a faster pace. Currently, they allocate only 10% of their resources to documentation; investing more time in updating it could attract more developers and streamline their processes. Additionally, implementing a CI/CD pipeline would automate their engineering workflow, eliminating bottlenecks.

If you are also facing such engineering dilemmas, then write to us at productivity@middlewarehq.com and we would be happy to help you provide actionable insights into your workflow or you can also try tracking your Dora metrics yourself using Middleware Open Source that too for free!

If you want to keep the conversation going about boosting engineering productivity or the engineering world in general, connect with fellow engineering leaders by joining The Middle Out Community. Don’t forget to subscribe to our newsletter for exclusive case studies and more!