14 min read

Artificial Intelligence Used for Testing, Needs Testing?

Featured Image

Artificial Intelligence (AI) and Machine Learning (ML) solutions for quality assurance are growing increasingly popular. Seen as the “next big thing”, AI/ML have become buzzwords in the industry. Many organisations have already begun implementing AI frameworks into their delivery lifecycles, and many are exploring the possibilities of using AI in the future. In fact, 88% of respondents to the 2020/21 World Quality Report stated that AI is now the strongest growth area of their test activities [1].  

It’s easy to see why AI/ML tools are in such high demand. The promise of reduced test maintenance, complete test automation and fast test creation is hard to pass onThe thought of AI magically solving all problems an organisation might face in testing makes AI tools an attractive option. This is again reflected in the World Quality Report. in which 86% of respondents stated that AI is now a key criterion when selecting new QA solutions, products and tools [1].  

AI might be seen as the future of quality assurance; however, setting expectations for the capabilities of AI tools is critical for organisations looking to invest in the technologyLet’s first consider the challenges associated with adopting AI in testing, before then discussing a solution. 

The Challenge of Using AI for Quality Assurance

Implementing AI tools isn’t as easy as pressing a few buttons and then letting the technology do the work. Developing and teaching AI is a complex task which requires substantial investment and resources. Additionally, there’s a lot of factors to consider when developing AI tools, that many organisations might be overlooking. 

Primarily, a factor often overlooked with AI tools is that they are very data dependant. To teach AI, an incredible amount of data is required. Without it, the tools are useless. Smaller organisations or QA teams lack the required data to develop capable AI tools. Furthermoreprocesses and tools used in the development lifecycle are often disconnected, meaning AI tools can’t collect the data needed to tell the whole storyIf you apply AI in this situation, you risk a “garbage in, garbage out” scenario.  

Secondly, AI tools are often used to identify elements of ‘how’ we can test, but less often focused on the harder question of knowing what we should test before the next release. For instance, AI tools might help you convert a web page into a bunch of test artifacts, but that doesn’t tell you what needs testing before the next release 

In other instances, using AI simply doesn’t make sense for the use case as organisations could more effectively and efficiently rely on rule-based logic to do all the work.  

Lastly, test data is also often overlooked in AI driven approaches to testing, but is crucial for effective test automation and AI tools.   

AI/ML technologies are rarely complete solutions to testing problems. Instead, they should be applied as tool within a larger solution. However, this leaves us with a key questionWhat does this larger solution do, and how does AI/ML feature within it? 

Complete Test and Data Automation: The Open Testing Platform

Curiosity’s Open Testing Platform leverages data from across the whole application delivery cycle to enable complete test automation in-sprint. This includes cutting-edge data analysis as one tool within a complete solution for prioritising and generating tests that matter before the next release. 

The Open Testing Platform collects and analyses data from across DevOps pipelines, identifying and creating the tests that need running in-sprint. This comprehensive DevOps data analysis combines with automation far beyond test execution, including both test script generation and on-the-fly test data allocation. This way, the Open Testing Platform exposes the impact of changing user stories and system change, prioritising and generating the tests that will have the greatest impact before the next release. 

 

An Open Testing Platform in turn embeds AI/ML technologies within an approach to in-sprint test automation. This approach is built on the following components: 

  1. Connect: An Open Testing Platform connects disparate technologies from across the development lifecycle, ensuring that there is sufficient data to identify and generate in-sprint tests. The Curiosity Open Testing Platform leverages a fully extendable DevOps integration engine to connect disparate tools. This gathers the data needed to inform in-sprint test generation, avoiding a “garbage in, garbage out” situation when adopting AI/ML technologies in testing. 
  1. Baseline: The rich data gathered from across DevOps toolchains feeds a Baseline” of real-time data. A Baseline aggregates, analyzes and converts observations into actionable insights. It exposes the latest changes in requirements, systems, environments, and user behaviors, informing testers of what needs testing in-sprint. The analysis might leverage AI/ML-based technologies where appropriate, but these are one tool in a robust toolbox. 
  1. In-Sprint: An Open Testing Platform not only identifies what needs testing, it also generates the test cases and automation scripts needed to run those tests. An Open Testing Platform doesn’t just tell you “how” to test; it tells you “what” to test in-sprint and provide accelerators to building those tests in short iterations. In Curiosity’s Open Testing Platform, Test Modeller generates the in-sprint tests. 
  1. Pathfinder: Lastly, an Open Testing Platform provides the data needed to run the in-sprint tests. Unlike outdated approaches to test data management, data is furthermore served up “just in time” as tests are generated and run. Test Data Automation provides this on-the-fly data resolution in Curiosity’s Open Testing Platform. 

Shifting Focus to In-Sprint Testing

In short, the Open Testing Platform creates central models to auto-generate test scripts for over 100 tools, complete with on-the-fly test data. This in-sprint automation might apply AI/ML where appropriate to identify tests, but in other scenarios alternative coverage algorithms might be more appropriate based on the data inputs.  

Overall, the driving goal of the Open Testing Platform is not to use AI/ML for the sake of using AI/ML. The driving goal is to minimise manual test maintenance, maximise the creation of new tests where they are needed, and equip all tests with "just in time" test data.

To learn more about Curiosity and the Open Testing Platformcheck out our latest eBook, Building sustainable test automation: Should you hire more people or buy “another testing tool”? 

Follow Curiosity on LinkedIn, Twitter, Facebook or subscribe to our YouTube channel. 

 

Footnote: 

[1] Capgemini (2020) World Quality Report 2020/21. https://www.capgemini.com/research/world-quality-report-wqr-20-21/