How to leverage

ChatGPT for Test

Automation?
 How to leverage ChatGPT for Test Automation?

Technologies like AI (Arti cial Intelligence), ML (Machine Learning), etc. can be easily

considered as one of the most in uential technologies that are having a monumental

impact on the technology ecosystem. Under the hood, AI-language models use complex

algorithms and learning techniques through which it understands and generates Natural

Language response.

 ChatGPT by OpenAI is one of the most signi cant advancements in the eld of NLP

(Natural Language Processing). Such was the excitement around the product that close

to a million users signed up on ChatGPT in just ve days after the launch. As a user, you

can literally ask anything in a natural language to ChatGPT and the engine would provide

human-like answers based on the data trained on a massive dataset from the internet.

ChatGPT is super-powerful since it has the potential to write poems, construct emails,

craft social media content, and much more. Not to forget, ChatGPT can also be used in

various aspects related to automated testing, It can be used for test case generation, test

case management, base test code creation, etc.

The fascinating thing about ChatGPT is its potential to craft human-readable, accurate,

working boilerplate code that can build the base for your test suite(s). It can leverage

popular test automation frameworks (e.g. Selenium, Cypress, Playwright, etc.) and

programming languages (e.g. Python, Java, etc.) to come up with the test code!

The icing on the cake is that ChatGPT can also create sample YAML les for running the

automation tests in a CI pipeline (e.g. Jenkins, CircleCI, etc.). In this blog, we deep dive

into the endless possibilities of harnessing ChatGPT for various activities related to test

automation. Let’s get started…

How to leverage ChatGPT for Test Automation?

 How to leverage ChatGPT for Test Automation?

What is ChatGPT?
Built by OpenAI, ChatGPT is a language model that is based on the GPT (Generative Pre-

trained Transformer)3.5 architecture. It has been ne-tuned using supervised and

reinforcement learning techniques (Source). Since ChatGPT is a dialogue-based model, it

has the capability to understand human-like questions and provide answers that can be

very-well understood by humans.

Since ChatGPT is pre-trained on a large corpus of data from the internet, it generates

responses that are ideally accurate and grammatically correct! In fact, OpenAI also offers

ChatGPT & Whisper APIs (Application Programming Interfaces), using which developers

can integrate ChatGPT & Whisper models in their applications (Source).

A number of developers who signed up for the OpenAI API Beta Program managed to

create some useful applications using the APIs. For instance, a user created a fully-

functional AMA (Ask Me Anything) search engine using GPT3.

The possibilities with ChatGPT and APIs offered by OpenAPI are endless but we would

restrict the discussion to ChatGPT and test automation. You can nd more such useful

applications here.

How does ChatGPT work?

As shown in the diagram below, the ChatGPT architecture consists of two major blocks –

Training and Answer.

How to leverage ChatGPT for Test Automation?

 How to leverage ChatGPT for Test Automation?

How ChatGPT works

The training block in turn comprises of the following stages:

How to leverage ChatGPT for Test Automation?

 How to leverage ChatGPT for Test Automation?

 Pre-training stage

 Fine-tuning stage

Reinforcement learning (or PPO optimization) is applied in the ne-tuning stage to ne-

tune the model so that ChatGPT provides a more accurate answer to the user(s).

As the name suggests, the response (or answer) is generated in the ‘Answer a prompt’

stage where the generated answer is shown to the user once it passes the safety and

moderation phase. We would recommend you to read How ChatGPT works to get more

granular information about the internal working of this amazing product!

Now that we have covered the essentials of ChatGPT, let’s look at some of the cases

where it can give a huge head start! Let’s look at the different types of testing (e.g. UI

testing, API testing, etc.) where it can be used.

ChatGPT and UI Automation Testing

Consider a scenario where you are planning to use popular test automation frameworks

like Selenium, Cypress, Playwright, etc. for testing interactions with elements in the DOM

(Document Object Model). Well, ChatGPT can be instrumental in generating the source

code that can be a starter for your detailed test suite.

Here are some of the questions that we tried asking to ChatGPT, the answers were bang

on target!

Q. Write a test using Selenium framework and Java to search “KiwiQA” on

Google. Click the rst result after the search.
Shown below is the code generated by ChatGPT. The code is accurate and works when

we executed the same on our system.

How to leverage ChatGPT for Test Automation?

How to leverage ChatGPT for Test Automation?

How to leverage ChatGPT for Test Automation?

 How to leverage ChatGPT for Test Automation?

As seen above, the code uses the CSS Selector for locating the elements in the DOM.

Though CSS Selector is a better selector than XPath, we still asked ChatGPT to provide us

with the code that uses the XPath selector for locating the WebElement.

And voila, it actually worked!

How to leverage ChatGPT for Test Automation?

How to leverage ChatGPT for Test Automation?

How to leverage ChatGPT for Test Automation?

 How to leverage ChatGPT for Test Automation?

Next we tried leveraging ChatGPT for creating an automation test that makes use of a

BDD (Behaviour-Driven Development) framework to perform the same task i.e. Search

“KiwiQA” on Google

Q. Can you provide the SpecFlow code which searches for “KiwiQA” on Google.
Shown below is the .feature le generated by ChatGPT. As seen in the snapshot, the le is

tagged as an SQL le which seems to be an error from the tool 🙁

Feature File

Here is the step de nition le where the tool has created steps corresponding to the

scenarios mentioned in the .feature le:

How to leverage ChatGPT for Test Automation?

How to leverage ChatGPT for Test Automation?

How to leverage ChatGPT for Test Automation?

 How to leverage ChatGPT for Test Automation?

Barring aside some minor issues that we listed earlier, the code looks perfectly ne. The

best part is that it also mentioned the dependencies that have to be installed as a part of

executing the above code.

Note that in order to use SpecFlow with C#, you will need to install the appropriate
packages – SpecFlow, SpecFlow.NUnit, and SpecFlow.Tools.MsBuild.Generation. You will
also need to download and con gure the ChromeDriver executable le as well, similar to
the Java example.

How to leverage ChatGPT for Test Automation?

 How to leverage ChatGPT for Test Automation?

We tried this after creating a project in Visual Studio and it worked like a charm! It also

gave the following note in the answer that acts as a “Word Of Caution”.

Many times, teams are confused on whether they should opt for BDD or TDD? Though

this opinion largely depends on the team’s expertise, ChatGPT can still be used to get

some inputs that can aid in making an informed decision.

Q. When to choose BDD framework over TDD?

Here is the answer from ChatGPT that very well summarizes the advantages of BDD over

TDD.

How to leverage ChatGPT for Test Automation?

 How to leverage ChatGPT for Test Automation?

In a nutshell, ChatGPT can provide you with code that can act as a good starting point for

kickstarting the activities in your automation project. Though the code needs to be

tweaked as per your project (or product) requirements, the boilerplate code provided by

ChatGPT can come handy in saving those extra hours that you (or your team) will

normally spend in creating the base code!

How to leverage ChatGPT for Test Automation?

 How to leverage ChatGPT for Test Automation?

Since ChatGPT uses the information being inputted by the users to train their models, it

is recommended not to upload/ask any con dential information to the platform! You can

always partner with an experienced software testing consulting services company so

that you are able to accelerate the project delivery using the combination of AI and

human intelligence!

ChatGPT and DevOps

Software projects (irrespective of size or scale) make use of DevOps like Jenkins, CircleCI,

Travis CI, etc. to run automation tests in a continuous manner. There is extensive usage of

YAML les for de ning the pipelines with a speci c DSL (Domain Speci c Language).

We asked ChatGPT to create a Jenkins pipeline for running Selenium tests in parallel.

Q. Create a Jenkins pipeline for running Selenium tests in

parallel
ChatGPT provided a Jenkins le with four stages: Checkout, Build, Run tests, and Publish

test results. It also gave details of every stage in the pipeline.

How to leverage ChatGPT for Test Automation?

 How to leverage ChatGPT for Test Automation?

Here is the complete Jenkins le generated by ChatGPT:

How to leverage ChatGPT for Test Automation?

 How to leverage ChatGPT for Test Automation?

We upped the ante a bit and asked ChatGPT to port an existing Jenkins pipeline le to

CircleCI.

Q. Can you help me convert a Jenkins pipeline script to

CircleCI script?

How to leverage ChatGPT for Test Automation?

 How to leverage ChatGPT for Test Automation?

In this case, we provided the earlier Jenkins script as an input to ChatGPT.

Here the con guration le (i.e. .circleci/con g.yml) was correctly identi ed as a YAML le.

The interesting part in the answer was ChatGPT recommending OpenJDK (instead of the

commercial Java variant).

How to leverage ChatGPT for Test Automation?

 How to leverage ChatGPT for Test Automation?

In a nutshell, ChatGPT can be used for creating pipelines for a range of CI tools like

Jenkins, CircleCI, Bamboo, Bitbucket, and more.

ChatGPT and Microservices Testing

Microservices architecture is a type of software architecture where the application is

developed as a collection of services that interact with each other using APIs. Here are

the major building blocks of a microservices-based architecture[1]:

 Microservices
 C ontainers

 Service mesh

 Service discovery

 API gateway

Since most of the software products are using microservices, we decided to try out

ChatGPT so that it can help us with microservices testing.

How to leverage ChatGPT for Test Automation?

 How to leverage ChatGPT for Test Automation?

Q. We have three microservices running in our system. Can you please provide test code
to verify them out. The rst microservice comprises of Selenium tests, second test makes
entries into the MySQL database, and the third one does API testing using SoapUI

ChatGPT provided us with a Python test suite that uses Selenium, MySQL, and SoapUI to

test three microservices.

How to leverage ChatGPT for Test Automation?

 How to leverage ChatGPT for Test Automation?

ChatGPT also goes on to mention about the details of the tests and instructs to run the

tests using popular runners like Pytest or Nose2.

How to leverage ChatGPT for Test Automation?

 How to leverage ChatGPT for Test Automation?

In case your project is making use of the Microservices architecture, ChatGPT can provide

you with the base code for testing the respective microservice. It is important to reiterate

that the generated code for microservices testing can be used for educational or

reference purposes.

However, ChatGPT can be instrumental in providing you guidance on how to modernize

an application by migrating from Monolithic to Microservices based architecture. We

came across an interesting question where a Telecom executive asked ChatGPT

regarding application modernization to Cloud-native Microservices architecture.

Now that we have looked into the broad usage of ChatGPT for test automation, let’s deep

dive into the Do’s & Don’ts of using this powerful tool/platform!

Advantages of ChatGPT for test automation

We had great fun leveraging ChatGPT to generate test scripts, de ne Jenkins pipeline,

create test code for microservices testing, and more. Though inputs provided by

ChatGPT gives you the information on which you can further build your code, it needs to

be used with caution. First, let’s look at some of the major advantages of ChatGPT:

How to leverage ChatGPT for Test Automation?

 How to leverage ChatGPT for Test Automation?

 C hatGPT can be instrumental in minimizing the learning curve when it comes to

automation testing, DevOps, etc. Consider an example where you (as an automation

tester) is given the task to create a Jenkins pipeline to run those tests in a

continuous manner. In such cases, you can leverage ChatGPT to get high-level

information on creating Jenkins pipeline for running automation tests. The output

can then be appropriately tweaked to suit your needs.

 C reating scalable and reliable tests is an important factor that can play a huge role in

minimizing the efforts involved in test code maintenance. AS seen earlier in the UI

automation testing example, ChatGPT could quickly generate code that used a

more reliable locator like CSS Selector in the source code. This is where you can use

ChatGPT to generate test code that is more resilient, maintainable, and reliable in

the short as well as long run.

A word of caution – ChatGPT is a conversational AI tool that stores your conversations to

improve its accuracy.

Accuracy of ChatGPT

How to leverage ChatGPT for Test Automation?

 How to leverage ChatGPT for Test Automation?

The accuracy level of ChatGPT is ~85 percent, a factor that should be enough to imbibe

con dence in this awesome tool! However, it is suggested not to upload any con dential

information (or code) to the tool; else it might have serious implications on your project &

organization.

Conclusion
ChatGPT has tremendous potential due to the depth in the answers provided by the

platform. If you are someone who wants to transition to automation testing/DevOps, etc.;

you can leverage ChatGPT to prepare a blueprint of your learning path.

ChatGPT is a boon to automation testers since it helps in providing the base/boilerplate

code on which you can further build the test cases (or test suite). Those worried about AI

taking away their QA jobs can relax, since an AI-tool like ChatGPT will help you in creating

well-designed test code at an accelerated pace.

AI can never replace human creativity which is why you should onboard an experienced

software QA consulting services company so that your make the best worlds – human

intelligence and AI intelligence 🔥

How to leverage ChatGPT for Test Automation?