Navigating the Challenges of Third-Party API Integration: Lessons Learned from Personal Experience

Over the past three years, I have been working on aggregator apps. These apps essentially act as intermediaries between users and various systems. The main goal is to provide a seamless user experience by allowing them to interact with multiple systems through our app. The core tasks involve proxying user requests to third-party APIs, mapping the responses, returning success or error messages to users, and storing the responses in our database.

Types of third-party API providers

There are three types of third-party API providers that we can categorize:

1. Seamless API Providers: These providers specialize in offering APIs for public use. Since this is their primary business, they pay extra attention to this area. They focus on making their APIs seamless and easy to use by providing excellent documentation and support. Users can often self-register and obtain their own authentication keys, such as apiKey or secretKey. Stripe is a good example of this category.

2. Semi-Seamless API Providers: These providers also offer public APIs, but with certain legal agreements in place. Users cannot self-register, and documentation is provided during the agreement process. Although this is not their main focus, they have dedicated teams and good support for their API offerings. Bank integrations are an example of this category.

3. APIs Under Development/Manual Integration: This is the most challenging category to integrate with. These providers are relatively new to offering third-party APIs and lack dedicated teams and support for integration. Agreements are often informal rather than legally binding. In most cases, support from these providers is limited. Businesses that already have established processes and want to expand into third-party API provision fall into this category.

Determining who needs whom

In these collaborative efforts, the fundamental question is: Who needs whom? Ideally, both parties should benefit equally from the collaboration (mutualism). However, when this balance is absent, the party with the greater need is at a disadvantage. If a provider requires us to integrate their APIs rapidly into our system, they are likely to actively provide support and fulfill all requirements to ensure a fast and accurate integration. However, if we rely more on their APIs, the dynamics change. In such cases, we must proactively communicate our needs.

The Issues

Based on my experience, the following issues are likely to arise:

1. Lack of Documentation

   The lack of comprehensive documentation is a common problem. It may include issues such as unclear API requests, insufficient API responses, and unclear error conditions. Additionally, if there is no clear understanding of how an API works or which API needs to be integrated, valuable time is wasted seeking clarification.

2. Errors

   Inadequate documentation makes every bit of support from the provider invaluable, especially during production. If there are any errors caused by their APIs, users may still blame us. Therefore, having a dedicated support team from the provider can save us from significant trouble.

3. Lack of Representative Test Data

   The absence of representative test data makes end-to-end testing difficult in the development environment. Furthermore, when test data is available, discrepancies between the production and development environments often hinder reproduction of specific cases.

4. Breaking Changes

   Unforeseen breaking changes in third-party APIs can occur without prior notification, causing error to our production system.

What Went Well

If in the future I need to develop this kind of apps again, this action that will be keep and repeated:

1. Proactive Approach

   Despite the lack of documentation, our team remains proactive by seeking clarification and initiating discussions to address any issues or problems encountered during the documentation and development process. While the provider's response may not always be immediate, it is essential to make the best of the available information and proceed accordingly.

2. Unified Communication

   We have established dedicated WhatsApp groups for each party involved, including our team, the provider's team, and all stakeholders. This approach ensures transparency and enables us to share any errors, successes, problems, or smooth operations that occur.

3. Stakeholder Understanding

   Our stakeholders comprehend that integrating with third-party APIs presents challenges, including slow development and occasional disruptions to production. Their understanding stems from the transparency we maintain through the WhatsApp groups. Moreover, if the provider's response is delayed, we can escalate our requests through higher channels. Based on my experience, this approach usually yields a higher response rate.

Lessons Learned

If in the future I need to develop this kind of app again, I will do all this action:

1. Extensive Unit Testing

   Although my company follows a standard of achieving 80% code coverage with unit tests, I would emphasize the need for more comprehensive and tailored unit tests. This would enhance our confidence in the integration code and account for a wider range of possibilities.

2. Mock Data

   We have faced delays in releases due to inadequate testing options. Leveraging tools like Postman for mocking responses proved to be beneficial. Although having real example data from the source is preferable, utilizing mocking tools can expedite the testing process.

3. Effective Communication

   Maintaining effective communication channels between teams and stakeholders is crucial for successful collaboration and issue resolution.

4. Meaningful Logging

   Employing well-structured logging mechanisms allows us to track and analyze system behavior effectively. We have done some logging and monitoring on production, but sometime some error still hard to trace. Good logging system will avoid some headaches.

Finally, it may be worth reconsidering the necessity of being an aggregator of third-party APIs. Evaluating if our business can operate without reliance on such APIs might provide valuable insights and potential opportunities for change.

That covers the main points of the article. Hopefully, this article give you some insight on working with third-party APIs.