How to develop comprehensive testing infrastructure

At Gitar, we rely on a comprehensive testing infrastructure to move quickly and with high quality. The core of our initial product is a code analysis and rewrite engine that works across programming languages and frameworks. This is challenging because although languages share common functionality they also have many idiosyncrasies that distinguish each of them. Developing a product that works across languages, programming styles, and frameworks requires a comprehensive testing infrastructure to ensure reliability, consistency, and performance.

The Importance of Testing at Gitar

In software development, testing is crucial to ensuring reliability and performance. At Gitar, this process is even more significant given the nature of our compiler-like tool and our rapid development pace. Production-grade compilers typically have massive quantities of tests to cover the different combinations of language constructs and combinations of transformations. Our testing infrastructure serves three main purposes:

1. Preventing Regressions: As we add new features or refactor existing code, we must ensure that these changes do not introduce regressions—errors that result in previously functioning features breaking. When we encounter a bug in our tools, we write a test to ensure it does not occur again.

2. Measuring Performance: Due to the structure of our rewrite engine, a single inefficient algorithm can disproportionately consume compute and slow down cleanups. Testing ensures that new updates do not degrade the performance of existing features.

3. High Test Coverage: Our tool supports a plethora of languages, including Java, Kotlin, Go, TypeScript, and more. Each language can be used with a variety of frameworks which offer very different programming paradigms. By using several kinds of source code as input, we ensure that we maintain coverage across many languages.

Fast Iterations and Continuous Feedback

To manage our fast-paced development environment effectively, we leverage continuous integration and deployment (CI/CD) pipelines. These pipelines automate the testing process, allowing us to:

• Catch Issues Early: By running tests automatically on every PR, we can identify issues before they reach production. Feedback is immediate and easy to iterate upon.

• Accelerate Development: Automated testing reduces the time spent on manual verification, allowing developers to focus on innovation.

Our pipeline features both a pre-commit and a post-commit test suite. They are designed to provide quick feedback and ensure that only high-quality code is merged into the main branch:

• Pre-commit flow: Runs locally on the developer’s machine, ensuring that code is linted and properly formatted.

• Post-commit flow: The post-commit flow is much more comprehensive and runs in the cloud. The new Pull Request (PR) is linted, tested with unit and integration tests, and put through our full end-to-end workflow.

Unit Testing: The Foundation of Our Testing Suite

At the core of our testing infrastructure are unit tests. These tests focus on small methods and structs that are used to compose larger cleanups. Unit tests are critical for:

• Isolating Issues: By testing individual components, we can quickly identify and fix issues without affecting the entire codebase.

• Facilitating Refactoring: As we improve our code, unit tests provide a safety net, ensuring that changes do not disrupt existing functionality.

Our unit testing suite is extensive, covering the various components of our cleanup tools. This ensures that each feature performs correctly in isolation, laying a solid foundation for more comprehensive testing.

Integration Testing: Ensuring End-to-End Functionality

While unit tests are crucial, they alone cannot guarantee that our entire system works as expected. Testing cleanup functionality in a very limited context is different from running it against production-grade code used in the real world. For this reason, we employ a suite of integration tests designed to validate our end-to-end workflow. Our integration tests run on larger units of code, ensuring that smaller pieces of code can be composed together properly. This is done by running our cleanup tool against code samples and verifying the output is as expected. These tests are more comprehensive and cover several aspects of our product, including:

1. Client-Facing API Tests

Our external API allows clients to integrate our functionality into their workflows. Integration tests for our API ensure that:

• Consistency: The API behaves consistently across updates, maintaining backward compatibility.

• Correctness: Each endpoint returns the expected results under various conditions.

2. GitHub Bot Tests

Our GitHub bot allows for cleanup PRs to be generated directly on Github. Integration tests for the bot ensure that:

• Functionality: The bot integrates smoothly with GitHub, automating tasks without errors.

• Reliability: The bot performs consistently, handling different scenarios such as errors in expected ways.

Conclusion

At Gitar, our testing infrastructure is a cornerstone of our development process, ensuring that we deliver a reliable, high-quality product to our users. By combining extensive integration tests with our end-to-end flow and leveraging CI/CD pipelines, we can quickly iterate on our polyglot tooling without compromising its integrity.