Interview for QA position

Introduction

In this article you can find some ideas on evaluating candidate's answers on a certain pool of questions.

Recruitement Process Structure

Evaluating Answers

In most companies there is a certain pool of questions that recruitment team ask each interview.

To evaluate candidate’s answers more thoroughly team expectations should be discussed and formalized before the interview.

Simple and transparent system should be created to back up desigions with data.

Key parts of this system are question pool and evaluation points.

Question Pool

Questions should be prepared before the interview.

It is not necessary to ask same questions all the time. Naturally interview may turn to some specific direction due to candidates answers or other factors.

Question pool is a large set of question to pick from. Recruitment team usually starts with the same qustions from the pool and then dives into specific direction. Key point is that team is moving inside a pool of predefined quesions. Low entropy high understanding of each move.

Each question aims to reveal a specific skill or fetch some data about candidate's personality. Or it can be asked to move interview to a certain direction.

Preparing a question just to have one generally is not a good idea.

Evaluation

After large enough question pool is ready it is time to prepare for the answers.

Asking good questions is a half way to success. Second half is processing answers in a transparent and measurable manner.

Team should answer the following questions in advance:

What is the best answer we expect?

What are the OK answers?

Which answers are not fitting to companies values?

Basic example for evaluating system is to give 10 points for each best answer.

0 points for totally not acceptable answers. Having 0 is a no go.

from 1 to 9 points to everything else.

After the interview all points are multiplied - if product is not 0 then summ is calculated.

E.g. I like question

“What is more important finding bugs, or avoiding bugs”

It is relatively easy to “digitalize” possible answers.

10 points for the answer like:

Finding bugs is more important because it is almost impossible to actually avoid all bugs. So finding those bugs that are still in the software is of great importance. In additon usually Software Testing Engineer does not have power to affect all processes into a way where all bugs are avoided.

6-9 points for the part of this answer.

5 points for "Finding" answer without full argumentation

4 points for "Avoiding" answer with good argumentation

1 point for "both" or "I don know" answer

If in your company avoinding bugs is more important you can reverse the evaluation points.

It is not necessary that there is a “correct” answer to some questions. But even to those questions we can evaluate the way of candidates thinking. After the interview we calculate the overall percentage and discuss those details that are not “digitalized” with points.

I usually ask the following questions:

Please compare the software testing processes that were used in your previous companies. What did you like most. What you did not like a all.

There is no "correct" answer here. So what I am looking for are the details and level of involment of the candidate and candidate's critical thinking.

Example of the answer I would give 10 for

In company A we had both customer projects and our own product. They have been tested in a different manner:

Project testing was going in agile way from kick-off meeting until a deadline that was set by Project Manager.

Project Manager was creating basic test cases according to project specification. PM could also provide customer a prototype so our Project Team was able to get feedback and implement changes.

Each project had one dedicated testing engineer, but it was always possible to ask help from Testing Team Lead. Some projects were similar to each other for example there were RFID cabinets where factory workers were able to lease or buy products from a no-human fully automated cabinets. I was usually testing this type of projects but not only them also this and that type of projects.

Testing was done in collaboration with a dedicated Client and Server developers. Usually one developer for each side. Complexity varied based on the number of layers. Some customers were using only our companies client and server solutions. Others were using their own backend system so we had to interated our server solution with their backed. It was adding one more set of API tests.

Typicall project required hardware testing, client mobile or windows application and API testing.

Our Product was released on not strictly regular basis and was manually tested from UI perspective. Later on I created very simiple automation with Python and Selenium to reveal broken pages faster since it was a typicall issue that we did not want in our Project testing. That was the only automation available.

What does this answer reveal to me:

Candidate has something to say about previous place - means that candidate was involved with the testing process.

Understanding of Project and Product concepts.

Ability to see and formulate differences between projects.

Specified how work was distributed.

General technical details provided but not to much for this type of question.

Some explanation of their "agile" was provided.

Mentioned situation with automation.

Another example of 10

In company B we were testing our prouduct X in a two month sprints. Each sprint meant one major release.

Tesing was performend in agile way from kick-off meeting where the features that we were inculuding to next release were listed unitll a go/no go meeting where finall decision on release date was taken.

Product Manager and Team Lead were creating basic features related tickets in Jira.

Our QA team had 4 members. I was taking tickets from Jira board based on my experience and some tickets were assined to each tester by our QA lead.

Testing was done in collaboration with a developer who was the code owner of the feature or whole microservice.

Usually creating test setups was required. It was up to a tester to decide which envrionment to use. Typical options were

* Creating environments from automated tests in RobotFramework where host created automaticall due to very well organized framework made with OpenStack, Docker, Make, Python and RobotFramework.
* Creating environments manually in CentOS or Rocky linux on VirtualBox localy.
* Same as previous but in VMWare or AWS cloud.

Some tests were easier to perform in already existed autotest environments. Custom tests were typically in VMWare due to its good balance between ease and perfomance + it was possible to keep system running overnight.

Automated Tests were executed on nightly basis.

Our Product was released every two month. Last two weeks were dedicated to release testing. It included running manual tests listed in Zephyr Plugin. Failed tests were reexecuted after fix in a rerun.

Automated tests were switched to run against release branch.

What does this answer reveal to me:

Candidate has something to say about previous place - means that candidate was involved with the testing process.

Clear understanding of release cycle.

Specified how work was distributed.

Many technical details provided with some usage examples shows that candidate is envolved with technology.

Claims to have ability to choose correct tool for specific task.

Experience in test automation.

Next question:

If you have a project to test API and you are free to choose any tools what would you choose.

Obviously there is no right or wrong answer. The reason to ask the quesion is to reveal if candidate has been using some technology and if candidate has passion to it.

I will use pytest/RobotFramework/postman… because I like it most/I have been using it in some other company.

Any specific answer will work and will get about 6 points. Answers like "I don't know" or I will use anything reveal that probably candidate is not into technology.

Answer that will get 7 - 10 points is actually a question. I expect that candidate will ask if there is some existing framework or programming language in use and then he/she will provide answer.

This callback question revelas that candidate is aware that usually you should adapt you toolset to existing project. Good answer can reveal that candidate is familiar with different tools and is seniour (or at least is aiming to become seniour enough) enough to make a choise.

Article author: Andrei Olegovich