Our main use case for Selenium Grid in the Cloud is running the automation test cases which we have, as running that locally is not a possibility because there are too many, and we want to run them in parallel in a reliable way, making it a must for us to utilize Selenium Grid in the Cloud.
Selenium Grid in the Cloud
GridlasticExternal reviews
External reviews are not included in the AWS star rating for the product.
Parallel web testing has reduced delivery time and now needs better automation and alerts
What is our primary use case?
How has it helped my organization?
Selenium Grid in the Cloud has positively impacted my organization by decreasing the time to test and deliver in production, significantly improving our DORA metrics. We are able to run more test cases on the same application and before using Selenium Grid in the Cloud, the time taken would have been prolonged, limiting our ability to test on different versions or browsers concurrently. This improvement helps in decreasing incidents, increasing customer satisfaction, and significantly reducing the time to deployment and testing, thereby positively impacting my organization.
What is most valuable?
The best features Selenium Grid in the Cloud offers include the capability to run test cases in parallel, and we implement customizations for identifying flaky tests or tests that are not running fine. Whenever test cases fail, we receive full logs and screenshots, which help us significantly. Selenium Grid in the Cloud assists us in parallel execution and enables us to distribute the workload effectively; if the workload is light, we can free up machines, and if it's heavy, it scales up automatically with multiple machines. It also allows us to test our applications across various versions of the same browser and on different browsers, as well as test mobile websites to see performance across different platforms, features that are definitely helpful for us.
We manage the tests on Selenium Grid in the Cloud based on the rankings, so we do ranks and decide when we are going to run based on which intervals. For some applications, we maintain an Excel sheet in which we do the ranking and the intervals, while in other cases we have config map files that categorize particular test cases based on their priority. We write the priority using annotations as well, which is how we manage everything on Selenium Grid in the Cloud.
What needs improvement?
Sometimes Selenium Grid in the Cloud requires manual intervention for very long-running test cases, which can get stuck and block the machine. We attempt to set up some customizations that facilitate automatic failure, and giving alerts at that moment would be really beneficial for debugging. This expectation of features in Selenium Grid in the Cloud is important to reduce our manual workload; having more customization options without switching to a paid version would be advantageous.
There are areas where Selenium Grid in the Cloud can be improved; we need to undertake many customizations on our end. The iteration speed for new features should be faster, incorporating market feedback effectively. Additionally, enhancements around Kubernetes could improve our configuration ease, and features like KEDA could optimize the upscaling and downscaling processes. Opportunities for integrating observability metrics through OpenTelemetry also exist, which would be useful for analysis. I would appreciate seeing more rapid iterations aligned with industry advancements, and incorporating AI features like chatbots for information retrieval would be beneficial.
There are smaller improvements needed as well; for instance, better test management integrations that allow direct session pushes to TestRail, Xray, or Allure would be great. Notifications to teams via Slack or PagerDuty alerts regarding issues will also aid in informing developers about failures. Implementing AI-based resource prediction capabilities would provide valuable insights into resource utilization based on past run data, which would certainly enhance our operations.
For how long have I used the solution?
I've been using Selenium Grid in the Cloud for more than six years.
What do I think about the scalability of the solution?
Regarding the scalability of Selenium Grid in the Cloud, it is not particularly scalable when configured manually, as we need to set up several custom solutions to enhance scalability.
How are customer service and support?
The customer support for Selenium Grid in the Cloud is decent, but I would not describe it as great since we have to handle many tasks ourselves.
Which solution did I use previously and why did I switch?
We did not use a specific solution prior to adopting Selenium Grid in the Cloud; however, we tried various paid tools such as BrowserStack, which performed well but was not cost-effective or suitable for running high-scale tests. Thus, we shifted to the open-source option and configured everything ourselves.
Which other solutions did I evaluate?
Before choosing Selenium Grid in the Cloud, we evaluated several other options, including paid tools like BrowserStack, Sauce Labs, and Perfecto, but found them lacking for our needs. We also checked self-hosted options such as Selenide and Moon before deciding to set up Selenium Grid in the Cloud on our public cloud.
What other advice do I have?
My advice for others considering Selenium Grid in the Cloud is that if you have a capable DevOps engineer to manage scalability issues and can operate the platform independently, the open-source solution is the way to go. However, if you prefer low-maintenance options, I recommend looking into paid solutions like BrowserStack that offer significant benefits and CI/CD integrations. I would rate this product a seven out of ten.
Automated browser testing has improved release confidence but setup and support still need work
What is our primary use case?
My main use case for Selenium Grid in the Cloud is testing the applications that I'm developing in an integrated, user-facing sort of environment. I had set up Selenium Grid in the Cloud on which we were running our application, and we had automated some tests using a common framework. It was Node.js based and we used Selenium Grid in the Cloud to test that across various browsers and various versions of these browsers so that we could ensure compatibility for our end users and also ensure that our app worked as intended.
What is most valuable?
The best features Selenium Grid in the Cloud offers include being able to test the application from a user perspective. Testing from a user perspective stands out as a feature for me, and I find parallel execution and cross-browser support to be basic—not having those is a deal-breaker for me.
Selenium Grid in the Cloud by itself does not provide any reporting. Mostly it's the frameworks that do that for me. WebdriverIO was the one that we were using, and it allowed us to do reporting while keeping parallel execution and cross-browser support. It also allowed us to scale pretty effectively and gave us debuggability so that we could keep our tests stable.
Selenium Grid in the Cloud definitely saved a lot of time for my organization. We do not have a specific QA team that tested each and every release, so we followed test-driven development. We ensured that tests were written alongside development, and they helped guide our product lifecycle. It definitely reduced costs a lot. Additionally, being able to get the reliability of running the entire test suite at a whim and with each and every release gave us a lot of confidence in each and every release.
What needs improvement?
Selenium Grid in the Cloud can be improved by providing more ready-to-use solutions and one-click setup. Right now, I had to go through a lot of hoops just to get it set up in a dockerized, containerized environment. Having a single-click sort of solution right in the open-source projects would be a godsend feature.
Being previously in startups, support did not matter as much for Selenium Grid in the Cloud, but integration has definitely been done well because of the W3C standards and very standardized frameworks which implement that. Documentation has definitely been a strong suite for Selenium Grid in the Cloud. However, being open-source, support lacks a lot, which definitely stops a lot of organizations from picking up a solution such as an open-source Selenium Grid. Being open-source, there is no support.
For how long have I used the solution?
I have been using Selenium Grid in the Cloud for more than six years.
What do I think about the stability of the solution?
Selenium Grid in the Cloud is as stable as you make it, but overall, we were able to make it pretty stable.
What do I think about the scalability of the solution?
We were able to scale Selenium Grid in the Cloud as per our needs. We did not have to have anything specifically for our needs or do anything specifically. Selenium out of the box does not provide scalability, so because we used some open-source projects which allowed us to do that, we were able to scale it. However, out of the box, I would say it does not rank great on scalability by its own.
Which solution did I use previously and why did I switch?
I have dabbled in QTP before choosing Selenium Grid in the Cloud, but being a VB Script based and very old framework, it did not make sense to continue with it. Selenium was the perfect framework for our use since our application was web-based.
What was our ROI?
We saved about an entire team's worth of salaries with Selenium Grid in the Cloud, which came to about 100k per year. The release confidence is difficult to quantify, but the time saved was significant with each release. I would say at least three to four days were saved per release, and we were doing a release every week, if not every two or three days.
Having not had to hire an entire QA team to test each and every release with Selenium Grid in the Cloud definitely saved us about 100k per year in USD. Additionally, we saved a lot on release timelines. For each release, we saved up to two to three days of manual testing effort at the very least. While it is difficult to quantify that in money terms, this was definitely a huge return on investment.
What's my experience with pricing, setup cost, and licensing?
My experience with pricing for Selenium Grid in the Cloud involved just the infrastructure cost, as being open-source, there was no licensing or setup cost.
Which other solutions did I evaluate?
I looked at Puppeteer for a while before choosing Selenium Grid in the Cloud, but the cross-browser support wasn't there. Selenium provided me the familiarity because I had worked with it before, so it was an obvious choice.
What other advice do I have?
I would advise others looking into using Selenium Grid in the Cloud to get someone who understands it at a grassroots level. You would be in good standing as long as you're testing a web-based application. It does have support for native apps as well, up to a certain extent, especially if you're also using something like React Native for testing across Android and iOS for cross-platform development. It could be great in that as well, but for web-based applications, I would say this is the best tool out there right now. I would rate this product seven out of ten overall.
Which deployment model are you using for this solution?
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Cloud testing has boosted parallel runs and reduced time while still needing better reliability
What is our primary use case?
I used Selenium Grid in the Cloud around five to six years ago, and I have more than four to five years of experience with it.
My main use case for Selenium Grid in the Cloud is running parallel execution, and I want to ensure that my test cases run smoothly on a headless browser on a Linux system.
I use Selenium Grid in the Cloud for running my test cases by managing multiple instances for 5,000 to 10,000 test cases that must run on multiple locales. Since the test case count is huge, execution was taking around one or two days. I addressed that challenge by dividing my load across multiple environments, specifically multiple instances of AWS, where I created multiple instances and ran all scripts on individual machines with the power of Linux. This ultimately reduced the execution time to two hours, which was a success story for our execution by using the cloud for parallel testing.
What is most valuable?
The best features Selenium Grid in the Cloud offers include the hub and nodes setup, where the hub controls everything on a single machine as the main execution center, and nodes act as supportive machines for distributing load. Running test cases in parallel is a major feature. I also explored another tool called Zelenium, which has very stable features for running tests in parallel without any hassle by providing easy-to-run access.
Selenium Grid in the Cloud has positively impacted my organization by significantly reducing execution time. Previously it was taking around 30 to 35 hours, but now it is reduced to two to three hours, which allows more time for decision-making. The ability to collect reports from individual machines and execution data easily is very good, making it a proud moment for us with respect to speed and efficiency improvements.
What needs improvement?
There are several areas where Selenium Grid in the Cloud can be improved, particularly regarding connectivity issues. There are challenges with scripts getting stuck, which causes nodes to become unresponsive. Exploring features that Playwright has would be beneficial, such as threading of execution and load balancing, which would enhance performance. Furthermore, good integration with the latest tools and AI capabilities is needed for better functionality.
For how long have I used the solution?
I have been working in my current field for 15 years.
What do I think about the scalability of the solution?
When considering scalability, having a good Kubernetes cluster with an open-source cloud like AWS has fantastic scalability, but a physical structure for execution can be risky and labor-intensive.
How are customer service and support?
I have never tried the customer support. For customer support, I would rate it a six.
How would you rate customer service and support?
Positive
Which solution did I use previously and why did I switch?
Before choosing Selenium Grid in the Cloud, I created my own solution for my execution needs.
While Selenium Grid in the Cloud had many problems during development, I eventually stopped using it and created a customized solution, which yielded a good percentage of success.
What was our ROI?
I have seen a good return on investment, especially when skilled employees tackle real business problems. A skilled employee knows the business well. The importance of having skilled resources over generic tools is evident since tooling might not always yield a high success rate.
Which other solutions did I evaluate?
In my experience, many organizations prioritize open-source technology over pricing or initial setup costs to customize their systems based on needs. There are challenges with existing setups that can introduce rigidity, which makes businesses struggle.
What other advice do I have?
My advice for others looking into using Selenium Grid in the Cloud is to first understand your business scenario and custom needs. Conducting a trial period or proof of concept is essential before fully committing, as it may pose challenges for new businesses.
I always recommend using stable and open-source solutions. Skilled employees are beneficial for reducing costs, but if resources are lacking, investing in initial tools to solve business problems is critical, as finding skilled labor can pose challenges. I would rate this review overall a six.