Are You Ready for AI-Powered Test Automation?

Where are we With AI?

At current pace, we’re able to get so much content generated out of most of LLMs.

• Expanded context windows, allowing LLMs to process more information

• Improved accuracy, with companies releasing benchmarks for comparison

• Advanced video and image recognition capabilities

• Ongoing research in content generation

Where Can Test Automation Be Improve?

1. Improved test coverage: Test coverage is a tester’s best friend. This helps them understand the quality of the product in a better way and gives them more control.

   • Solution: AI can generate test cases that humans might overlook, potentially uncovering edge cases and rare scenarios.

2. Self-healing tests: Adapting to UI changes and modifying test cases to cover them is an important a tester has to take care. Test case maintenance and management.

   • Solution: AI-powered tools can adapt to minor UI changes, reducing test maintenance efforts.

3. Test Prioritisation: This is much needed when you have a test cases in thousands, and running test’s periodically is important to your product’s quality.

   • Solution: AI can identify patterns and anomalies that might indicate bugs more effectively than traditional automated tests.

   • Solution: This could also be done by tagging and grouping test cases, so AI can better understand which test belong to which suite and what is the Priority of the test. Further down the line, this information adds up into making better decisions to test case management and maintenance.

Potential Drawbacks and Solutions:

• Over-reliance on AI

  • Teams might be too dependent on AI-generated tests

  • Scenarios that require human insight

  • Solution: Have a strict procedure to produce AI results and humans to validate the AI generated test cases.

    • AI can help in manual test case generation only → leading to covering the original problem of just covering the manual test cases. And writing Automation test cases by the tester

• False positives/negatives | Lack of contextual understanding

  • Misinterpret certain behaviours and Hallucinations

  • Understanding the broader context

  • Not meeting User expectations

  • Heavy reliance on accurate prompt engineering

  • Solutions: Writing definitive PRDs and SRS documents make the AI more aware about the context of the automation.

    • Sharing knowledge base of the complete application, to give more context. And prompting AI to not hallucinate outside the given context

    • Making a RAG application with context of all documents for the application, and running test case generation on that

    • Better prompting techniques to get finer results and block unwanted hallucinations

• Data privacy concerns

  • Sensitive organisational data

  • Data storage on AI Provider Cloud Servers

  • Data potentially used in training other applications

  • Solutions: Utilise on-device nano LLMs to provide context for test case generation

    • Host on-premise instances of LLMs and connect tester PCs to a VPC for accessing AI test case generation

    • Leverage company-owned, compliance-reliant services:

      • Host AI models on the company's own cloud infrastructure

      • Implement robust encryption

      • Ensure data is not stored on AI providers' cloud storage

      • Store data on company-owned cloud where it can be deleted after the use case is fulfilled

How Is DevAssure Leveraging AI?

• DevAssure leverages AI to generate manual test cases, taking into consideration the:

  1. Figma Files or UI mockups

  2. Feature Spec Documents like Software Requirements Specification or Product Requirements Document

• This helps DevAssure to get the required context on the feature and generate manual test cases; covering base and edge cases for the scenario

• Further the Tester can start recording test cases with Inbuilt recorder and run them locally to test the application

• https://youtu.be/3LCMdWX0eoA?si=KfTfS_R-1krtuoS8

Summary

AI advancements are significantly enhancing test automation. Current large language models (LLMs) are generating more accurate content, recognising videos and images, and adapting to UI changes, improving test coverage and prioritisation. However, challenges include over-reliance on AI, false positives/negatives, contextual understanding issues, and data privacy concerns. Solutions involve strict validation procedures, better documentation, improved prompting techniques, and secure data handling practices. DevAssure leverages AI to generate comprehensive manual test cases by integrating design files and requirement documents, streamlining the testing process.

PS: Please let us know your thoughts in the comment section, and Are you ready for AI-Powered Test Automation?