Basic Troubleshooting of WSO2 Applications with WSO2 MI (Micro Integrator) Part-03

Basic Troubleshooting of WSO2 Applications with WSO2 MI (Micro Integrator)

After addressing issues potentially arising from the Identity Server or Micro Gateway, the next critical component in your application architecture to troubleshoot is the Micro Integrator (MI). MI plays a pivotal role in integrating different backend systems and services, making it crucial to resolve its issues promptly.

Understanding Your Application Flow

Typical Application Flow:

IS (Identity Server) → MGW (Micro Gateway) → MI (Micro Integrator) → [Your backends (EX; database)]

Note: This flow is illustrative; your actual application setup may differ.

Troubleshooting WSO2 MI

The MI is designed to handle complex integrations and process high volumes of messages efficiently. However, problems such as network connectivity issues, endpoint suspensions,resource limitations and many more errors can arise. Here’s how to systematically identify and resolve these issues:

1. Error Connecting Backend

Symptoms:

• Logs indicate an error when trying to connect to the backend service, typically due to network issues or the backend service not responding in time.

Example Command to Identify Errors:

cat wso2carbon.log | grep "error connecting backend" | more

Example Output:

[2023-10-18 09:44:21,155] ERROR {LOGGER_V1} - {"timestamp":"2023-10-18 09:44:21,152","platform":"WSO2","loggerVersion":"1.0","correlationId":"18b4187320b90abc","channel":"SIMAPP","integrationName":"ESB Proxy API","integrationVersion":"1.0.0","messageType":"ESB_PROXY_API","clientMessageId":"339344a6-7ef1-4528-892d-92a8e370cf44","origin":"172.18.139.7","sequenceName":"proxy-api-v1-fault-sequence","appVersion":"1.0.0","appId":"Simbanking","httpMethod":"GET","metaTransportInUrl":"/esb-proxy/1.0.0","contentType":"application/json","context":"/esb-proxy","statusCode":500,"message":"ESB Error, Error connecting to the back end"}

Why This Happened:

• Network Issues: There could be a network configuration error such as incorrect proxy settings, DNS issues, or the backend service might be down.

• Firewall/Security Configurations: Security appliances (like firewalls) or network policies may be blocking the connection.

• Misconfigurations in Service URL: The endpoint URL might be incorrect or the service might not be responding as expected.

How to Solve:

• Verify Backend Services: Ensure that the backend services are up and running.

• Check Network Configurations: Review network settings, including firewall rules, DNS configurations, and proxy settings.

• Service URL: Double-check the service URL and other configurations in your proxy service definition.

2. Suspending Endpoint

Symptoms:

• The MI suspends an endpoint due to repeated connectivity failures, which is indicated in the logs.

Example Command to Identify Errors:

cat wso2carbon.log | grep "Suspending endpoint" | more

Example Output:

[2024-04-20 17:59:21,260] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : bulk-sms-sender-api-v1-inforbip-bulk-sms-rest-endpoint with address https://9zm33.api.infobip.com/sms/2/text/advanced - current suspend duration is : 2000ms - Next retry after : Sat Apr 20 17:59:23 EAT 2024

Why This Happened:

• Service Unavailability: The endpoint might be temporarily unavailable, which triggers the suspension.

• Repeated Failures: Endpoint suspension often follows several consecutive failures in reaching the endpoint, as defined in the Synapse configuration.

How to Solve:

• Increase Timeout Values: Consider increasing the timeout values and the retry duration in your endpoint configuration.

• Monitor Endpoint: Use administrative tools or scripts to monitor the availability of the endpoint.

• Review Endpoint Strategy: Implement a more robust error handling and retry mechanism, possibly with dynamic error thresholds.

3. Dropping Message After Endpoint Timeout

Symptoms:

• Messages are dropped after a specified timeout when an endpoint does not respond in time.

Example Command to Identify Errors:

cat wso2carbon.log | grep "dropping message after ENDPOINT_TIMEOUT" | more

Example Output:

[2023-10-27 12:15:08,086] WARN {org.apache.synapse.core.axis2.TimeoutHandler} - Expiring message ID : urn:uuid:6ac77ba7-7525-4abd-9df6-e7ac3420f449; dropping message after ENDPOINT_TIMEOUT of : 30 seconds for Endpoint [proxy-api-v1-service-endpoint], URI : http://172.32.1.57:9090/savvycore/card/cardrequest/createCardRequest, Received through API : proxy-api-v1-api:v1.0.0

Why This Happened:

• Timeout Exceeded: The message was not processed within the specified endpoint timeout period (30 seconds in this case).

Resolution Steps:

• Adjust Timeout Settings: Increase the endpoint timeout setting to allow more time for slower backend services to respond.

• Optimize Backend Performance: Ensure that the backend services are optimized to handle requests within the timeout period.

• Error Handling Strategies: Implement comprehensive error handling strategies in the integration logic to manage timeouts gracefully.

4. Pool is Exhausted Error

Symptoms:

• The server's thread pool is exhausted, which means no new requests can be processed at the moment; this is typically logged as an error.

Example Command to Identify Errors:

cat wso2carbon.log | grep "exhaust" | more

Example Output:

Could not get a PassThroughMessageProcessor thread to process the request message. The primary worker pool is exhausted

Why This Happened:

• High Concurrency: Too many concurrent requests are being processed, exceeding the capacity of the available worker threads.

• Resource Limits: The server's resource limits (CPU, memory, threads) are reached, causing it to deny new requests.

How to Solve:

• Increase Thread Pool Size: Increase the size of the worker pool in the MI’s server configurations (axis2.xml or passthru-http.properties).

• Optimize Performance: Review the performance of your integrations to ensure efficient use of resources.

• Load Balancing: Implement load balancing to distribute requests more evenly across available resources.

General Advice for All Issues:

• Monitoring and Logging: Enhance monitoring and logging practices to get real-time insights and react proactively to issues.

• Regular Testing: Regularly test endpoints and backend services to ensure they meet operational standards and performance benchmarks.

• Configuration Review: Periodically review and optimize configurations based on operational experience and evolving requirements.

Conclusion

Effectively troubleshooting WSO2 Micro Integrator (MI) is essential for maintaining a robust and reliable application architecture. By systematically identifying and resolving common issues such as backend connectivity errors, endpoint suspensions, message drops, and thread pool exhaustion, you can ensure smooth integrations between services. Key strategies include monitoring logs, optimizing configurations, and enhancing backend performance. Proactive measures, such as increasing timeout values, adjusting thread pool sizes, and implementing load balancing, help mitigate potential issues before they affect application performance. Regular monitoring, logging, and configuration reviews are vital in ensuring that the MI continues to handle complex integrations efficiently in a dynamic and demanding production environment.