Test Orchestration using HyperExecute: Mayank Bhola [Testμ 2022]

Abhishek Mohanty, Senior Manager — Partner Marketing at LambdaTest, hosted Mayank Bhola, Co-founder and Head of Engineering at LambdaTest, to discuss Test Orchestration using HyperExecute. Mayank Bhola has 8+ years of experience in the testing domain, working on various projects and collaborating with experts across the globe.

Here is the video link:

He started by explaining how the open source frameworks such as Selenium, Cypress, and Puppeteer have been the root source to conduct all tests.

According to Mayank, the problem is when test execution is moved to the cloud, they face latency. He spoke on how the zeal to provide a great testing infrastructure motivated them. This is what led them to build a new product called HyperExecute.

The main question that inspired them to build HyperExecute was, “Why can’t we meet local system execution speed?” HyperExecute can run on the cloud and match the speed of the local system.

Another reason is to accelerate faster feedback from the developer’s end.

Features of LambdaTest

LambdaTest provides a digital test execution infrastructure on Cloud. You can leverage our cloud-based platform to run your tests on a variety of browsers in an instant.

Mayank then shared the core features offered by LambdaTest:

• Live testing: With LambdaTest, you can test in a live, interactive environment with 3000+ browsers and operating systems. Test whichever machine you want to, anytime, anywhere, with live testing.

• Automation cloud: Plug your automation test scripts from your local machines into the cloud, providing you with the scalability to expand your test coverage without worrying about the number of systems you have.

• Real device cloud: To plug in your app development efforts and test it over various iOS and Android devices, you can use the real device cloud.

Challenges with conventional test infrastructure

He spoke about the challenges faced by traditional test infrastructures, such as slow test case execution on the cloud, flaky test results, easily breakable infrastructure, and testing platforms. He gave an example of how an enterprise might need to provide client-based certifications. A traditional cloud grid might only provide access to the browser.

Simulating such cases is complicated, with an inability to fiddle with the underlying operating system. He spoke on how all these force the testing team to create their software testing infrastructure. Things would go down due to the overhead of maintaining that infrastructure, writing good test cases, and lacking robust infrastructure.

How so slow?

Mayank explained how the traditional cloud-based test case execution is extremely slow.

There is a script running on the CI system sending out some requests. Every use case is sent via wire as an HTTP or network call. The remote testing infra has a request gateway component responsible for accepting some commands. They are being relayed to different microservices responsible for provisioning some remote infrastructure. If different people want to test their website on different devices, whatever the user sends to the request gateway gets replicated to the virtual machines housing the web browser. The web browser would communicate it to the application under the test case. Just to execute a single command, the data has to travel to and fro between different components. The microservices, aiming to provide more logging observability, are responsible for more delays.

In the Selenium test case, if using a cloud-based test execution platform, multiple hops exist before the command can reach the target website.

If you want to test a website in Australia from India, the script running on the local system will send out HTTP-based requests or other network-based requests. These APIs are received by a component called the hub that accepts all commands and schedules the virtual machine running from a browser. Now, the virtual machine running is geographically separated from the hub component to make these services horizontally scalable. The microservices can also cause a lot of delays. The hub will transmit all the commands from the client script back to the virtual machine with the web browser. Finally, the web browser will access the target website.

For a typical end-to-end Selenium test case, you have 100s of such commands. If you wait for the network response, you will waste time. The test cases would become flakier. The test case can fail even if one of the connections between two hops gets degraded. When test cases fail, the one to blame could be the code, website, or platform.

With the traditional cloud-based model, you have these three problems. Hence it can be prolonged.

As a developer or a tester, if you run end-to-end test cases, the script, the hub, the Selenium component, and the web browser, everything is on the same laptop. This led to the idea of simulating them over the benefit of the cloud.

Why HyperExecute?

HyperExecute puts all of the components to run a test under a single virtual component. You can get 60% of the performance benefits by doing this. You do not have to communicate over different geographical regions. The local host is the fastest network you can get.

With HyperExecute, you can utilize and harness that. The problem gets more profound when you consider how people perform tests and run test cases.

The next thing was getting the orchestration layer on the top. This gives much more space for optimizations. HyperExecute has re-engineered it for you. You can extract 70–80% of performance benefits. You can easily achieve it when you want to meet local execution speed.

If you try to run multiple test cases sequentially on a local system, the time between different test cases can range from 4–5 seconds. The browser running the test case will have to shut down and start a new instance.

They have achieved the latency between stopping the last test case and beginning a new test case up to 300 milliseconds.

Core Orchestration

Mayank also spoke about how complex workflows, debugging nightmares, confusion on what and when to run tests, and observability issues complicate testing.

“Core orchestration means the platform should be able to take the load of executing test cases smartly and distribute them intelligently across nodes, which could mean docker containers, virtual machines, emulators, simulators, and real devices.”

“You write tests, we take care of the rest.”– This is the policy of HyperExecute. You can get it done easily when you want to reduce the end-to-end testing time. Most of the time, there is parallelization of test cases.

A developer or QA person does when they are trying to develop something is authoring a test case and orchestrating it. Orchestration means deciding which test cases to run. If people have 100s of test cases, they parallelize it.

If you have to take manual control, you have to write your own code to orchestrate it. After orchestration, you have to execute the test and fire off that command to a remote cloud provider or maintain a whole infrastructure.

Apart from all this, there are many parts where people spend a lot of time on. This is where HyperExecute wants to take off the burden.

Mayank explains how the HyperExecute team does not want people to provision infrastructure, wondering how to orchestrate, execute them efficiently, and collect back all the data. That would be an utter waste of time. With HyperExecute, you can just want to achieve better results. Hence you can use the YAML file. Just by having the same phenomena of adding a configuration, you have a YAML file in which our user has to tell what to run.

According to Mayank, you write a test case and the YAML once. HyperExecute will take care of the rest of the processes since they want their users to focus on writing more and more test cases.

He explained how HyperExecute allows fine-grain control over your testing durations. Let’s say a mature organization has a policy not to run tests for more than 10 minutes. In that case, you can set a timeout, after which it will abort the test case.

He spoke on how HyperExecute allows Retry and Timeout. He also shared how HyperExecute allows you to work on any framework.

Apart from running test cases, HyperExecute allows you to run some commands required to set up and execute the test cases.

HyperExecute also enables you to auto-split your test cases and parallelize them.

He then explained how with CLI, you could know the current status, how many steps have been executed along with the time, and get a summary of how the test execution was performed.

Top features of HyperExecute

With HyperExecute, you only find logs for the pre-steps. No other logs. It will make it easier for testers.

He then explained the top features of HyperExecute:

• Test environment customization: According to Mayank, HyperExecute has made it extremely easy for the end customer to customize the test environment. You have the feature to install certificates, run Mysql instances, and generate some analytics after executing the test cases. You can also install desktop applications while running test cases.

• Dependency caching: Apart from these, Mayank revealed that you also have smart dependency caching. All languages and runtimes have package managers like pip for Python and npm for Node. All these run time packages are very heavy. If you download them each time, it’s going to take time. Next time you run the test case with the same manifest file, the pattern will cache all the dependencies automatically without you telling yourself how to get it done.

• Auto-splitting: HyperExecute also allows you to auto-split your test cases and parallelize them, enabling smart ordering and retries. The platform will be able to order consistently failing test cases to run first so that the end user gets the feedback very fast.

• Real-time logs: You can also get real-time logs to make your testing experience smoother. Also, Mayank spoke about how you can get automatic report generation to get the most insightful reports with details of a testing framework. For example, if you have a Cucumber-based framework, you can list the number of features, test cases, scenarios, and task analytics. You can also get the feature and scenario-level logs.

• Artefact management: You can easily manage and fetch videos, JSON files, and CSV files. Customers can generate and manage these artifacts automatically. You can access it on your dashboard.

• Deep analytics: Mayank spoke about how LambdaTest is working on it to ensure that tomorrow if any test cases fail, we don’t have to read through any logs.

• Smart CI Support: Mayank explained how they had added smart CI features with blazing fast speed.

He then gave a live demo of the tool:

“HyperExecute is going to distribute your test cases across all the available nodes.” he quoted.

He then explained a matrix strategy where you want to run the same test case or test suite with different parameters. Instead of writing separate code, you can define all the permutations and combinations. HyperExecute is going to run every single permutation in a single virtual machine.

Mayank also shared how you can use HyperExecute on the shared cloud, dedicated cloud, and on-prem.

Insights from Live Q&A session:

What are the key features that set it apart?

Mayank: It’s not just a plain execution platform. It’s an orchestration platform. You can specify what needs to be done before the test case starts. That’s the most flexible feature!

Can we configure customer runners in HyperExecute like API and performance tests?

Mayank: Absolutely! The end-user can configure it to run according to their needs.

Does it support HPA and Node auto-scaling?

Mayank: Yes, it does.

Is the documentation available?

Mayank: Yes, it’s available. We also have various GitHub repositories. You can find them on the official website.

What is the right count for the retry test executions?

Mayank: Some developers from the Netflix TV testing app have this magic number of 35. They try to test their test cases 35 times. For us, anything above 5 is a red flag. You need to fix them immediately.

Can you touch upon plans to release mobile support for HyperExecute?

Mayank: Yes! We are bringing in support for mobile emulators and simulators by the last quarter of this year.

Is it SaaS-based, or can we deploy it on our K8 clusters?

Mayank: Yes, it is. We also provide on-prem installations or public clouds like AWS, Azure, etc.

How to try out HyperExecute? What’s the license price? Can we use this for the execution of any client project?

Mayank: You can set up the master cluster of HyperExecute. This way, you can reap many benefits without doing everything by yourself. All you need to do is log in and try the platform. We have a dedicated page for pricing details.

LambdaTest has both an online Selenium Grid and a HyperExecute grid. When should someone opt for HyperExecute?

Mayank: Even though we have both CPUs and GPUs, we have different use cases for both. When you want to do cross-browser testing, back from XP to MacBook Ventura, you can use Selenium. When you need to conduct test execution fastly and debug soon, HyperExecute is the option.

Is there a predictability capability that makes the testing faster, like only executing the test cases depending on the code change but learning from the build management?

Mayank: That would be our end goal. It will be available on our platform by the end of this quarter.

It was indeed a knowledge-packed session with Mayank! Hope you found our HyperExecute platform fascinating!

Happy testing!