Please Let Manual Testers Be Manual Testers

This entry was posted by on Sunday, 24 February, 2013 at

The testing world seems to have entered a state of flux in the last couple of years where “Automated Testing” is the new nirvana, I suspect this in part due to more and more companies following Google’s lead and starting to hire developers in test. Now in my opinion having people performing these roles is not a bad thing. When you have somebody writing your test framework you want somebody with some experience writing code and making architectural decisions. A developer in test is a very useful and powerful resource in this situation. The problem is that people have seen how useful a developer in test is, and they have decided that every tester should now become a developer in test, even though the majority of testers are probably not going to be performing automated testing, or writing test frameworks.

I regularly frequent the Selenium User’s mailing list and day by day, I see more and more people coming to the list who just don’t seem to have a basic clue about programming. These people invariably want you to ‘urgently’ help them because they have to write an automated test/test framework and they have no idea how to do it. Now people wanting to learn Selenium and become automated testers is not a bad thing, but most of these requests seem to be people who have testing jobs and have suddenly had the job of automated tester thrust upon them, this is a bad thing!

The other thing I see more and more regularly now is test frameworks that have been written so that manual testers can easily use them without learning how to program. These products seem to come in two forms:

  1. Something that scans a page finding all the elements of interest to abstract away the logic of finding them.Let manual testers be manual testers
  2. An Excel spread sheet driven framework where you have to manually populate the spread sheet with locators/expected text to run the tests.

Now I can see some value in option 1, that could be useful for automated testers who don’t want to spend lots of time locating things that may be interesting on a page and just want to spend time interacting with elements. Personally however I would not want to use something like this, I prefer to locate elements of interest myself to keep my tests lean and mean.

Option 2 is something that should be killed right now in my opinion. It has exclusively been written to take manual testers and trick them with the promise that they will now become automated testers and eventually developers in test. It will not do this; they will spend all of their day filling in Excel spread sheets (Why do we still have such an obsession with Excel spread sheets anyway?) and then the rest of their time updating them as things go wrong.

This process really turns manual testers into data entry clerks. These frameworks are invariably brittle as hell and require a lot of manual effort to keep them up to date. The spread sheets that are used end up being very complex, because automated testing is a complex thing to do, and as you add more functionality they get worse. They are useless, soulless and worst of all, take up all the time that manual testers have so that they stop doing the thing that manual testers do best, manual testing!

In our eagerness to move testing forward we are actually forgetting what the point of automated testing was in the first place. Automated testing was designed to make boring and monotonous regression tests a thing of the past. If the machine can rerun a series of known tests by itself and check that nothing has broken in the current build that frees the manual testers up to do the thing they do best; manual exploratory testing which is where you find all the bugs.

Automated testing checks that the functionality you have written works as you expect it to. Manual testing starts to push the envelope and use the program in ways it wasn’t intended to be used. Manual testing has no barriers, it isn’t constrained, and it is what will find holes in your application.

Some of the best manual testers I have worked with knew nothing about programming; they could not write automated tests and quite frankly had no interest in it. If I was building a new test team and could pick either two of them, or a team of forty automated testers, I would pick the manual testers any day of the week. They will find bugs, they will exercise the system properly and they will be of much more benefit to the project. I would still want automated testers as well to write the automated test framework and write regression tests, the point is that they would be doing this to free up time for the manual testers to go in and break the system in new and inventive ways.

To finish off I would like to make a plea:

Please don’t try to turn manual testers into data entry clerks, and don’t try and force them to become programmers; all you are doing is destroying the testing profession.

Manual testers rock and are the heart and soul of testing, make sure you appreciate them!

14 Responses to “Please Let Manual Testers Be Manual Testers”

  1. Thanks for the post – and the appreciation of what a good tester can bring to the table.

    It is one of the topics that gets talked about a lot, some recent examples are:
    http://watirmelon.com/2013/02/23/do-software-testers-need-technical-skills/
    http://www.techwell.com/2013/02/do-testers-really-need-technical-skills
    http://thesocialtester.co.uk/testers-need-to-learn-to-code/

    and I did a blog a while ago about the new wave of Zombie testers churning out their Selenium scripts:

    http://expectedresults.blogspot.com/2012/02/new-zombie-wave.html

    The SQA Stackexchange site is just full of testers trying to learn Selenium just like you are finding on the mailing list :(

    Thanks again for the post

    • Ardesco

      Interesting reads, cheers for the links :)

  2. +1

    focus on automation is a trend – one solution. .. that does not fit 100% in every context. Even google uses manual testers (for hire and) in the “real test environment” (aka Prod).

    Jesper
    http://jlottosen.wordpress.com/2012/10/08/yes-non-tech-people-can-be-testers/

  3. I mostly agree but not entirely. The vast majority of our work is manual exploratory testing and we never write automated regression suites, but there is often value in using automation to support manual testing even if the tests are only going to be used once.

    As such, the automated scripts we write may be ugly but they are good enough – there’s no point making them better than that because they will be thrown away. Our aim is for all our testers to be able to write such automation and to use it when appropriate.

    I am pretty unimpressed with the test design skills of most of the testers I have interviewed, but I have to say that the testers who have chosen to specialise in automation are the worst – they seem to have forgotten anything they ever knew about testing. This is an appalling trend because it means they are just automating a lot of weak tests and presumably their current employers are themselves not skilled enough to notice (don’t get me started on useless test managers!).

    • Ardesco

      How does automation support manual testing when you only ever use the test once?

      I don’t have any context, but my first impression is that you are wasting your manual testers time by making them write an automated test that they are not going to reuse and potentially are not sure how to write.

      I find your final statement deeply disturbing, testers should be able to design good tests. Automated testers are no exception to this rule, I would expect then to be as good as any manual tester and probably better when it comes to writing regression tests (as this is an automated testers bread and butter). I’m hoping you have just had bad luck with your interviewees.

  4. Natalie Bennett

    What about other applications of scripting/automation in testing? Is “replicate what an exploratory tester would do with no tools with a browser automation script like Selenium” really what a company like Google does with it’s “Engineers in Test?” Any thoughts on that?

    (Sorry if this is something you’ve covered elsewhere on your blog– I just got here from a link on Google Plus. Taking a look around now, but I’m a more-or-less brand new tester who almost exclusively does exploratory testing and this is something I’ve been thinking about so I figured I’d drop a line. I have decent technical skills of the “power user” variety but testing makes me want to learn how to code so I can understand better what it is that I’m finding problems in.)

    • Ardesco

      What I’m saying is that in general automation is a tool to free up manual testers from doing the boring and monotonous regression testing so that they can do exploratory testing. Automation tools are not good at doing exploratory testing, they are good are performing a defined series of actions and asserting that these actions continue to work.

      There are efforts to try and make automated tests more reactive and exploratory, but I haven’t yet seen one that can come anywhere near a manual human tester.The problem is that computers are not good at thinking out of the box and trying new things, they never will be (well OK maybe never is a strong word, but AI has to come along in leaps and bounds before a program could replicate what a human could do).

  5. Søren Harder

    I always quote the ‘First rule of testing’: “If you are doing it in Excel, you are doing it wrong.” It may be a little blunt, but it is a good rule to live by.

  6. I really enjoyed reading your article and felt it was quite insightful to the current trends of testing.

    I too have noticed in the past two years a trend happening to segregate the once single testing role into single focused testing specialties, i.e agile, test automation, exploratory, performance, etc. This segregation is truly that, segregation in that each component of testing is often not truly working cohesively together hence breeding testing solutions that truly do not maximize nor compliment one another.

    Test automation, in my humble opinion, is an artistry of marriage between being a clever programmer and an innovative and sapient tester. Personally I do not like that people are hiring “developers” for the role of automation without training them to be “testers.” The main problem as I see it is the vision is still very limited on creating a process and a work flow that maximizes both the advantages of manual testing with the advantages of automated testing.

    Automation is not just regression testing. Automation can be used to support exploratory testing, the setup and configuration of a test environment, testing a configuration matrix of a system, test data loading, regression testing, functional testing, performance, reported bug management and the list goes on. By having automation fulfill these roles, it allows manual testing to truly enjoy the art of intelligently and perhaps even scientifically learn and identify methods for testing the quality of the system.

    With today’s application complexities, automation is required therefore we need workflow solutions to maximize the use of our talent pool. There is nothing wrong with Test Case Capture by Excel as long as it supports an intelligent process and often Excel is just a method used while growing up to more intelligent solutions, e.g. using Jira or Test Management to capture the tests.

    Thank you for sharing this article.

  7. Well done for hitting this nail squarely on the head. This is pervasive and you did Manual Testers a huge service by helping prevent their devaluation as a crucial part of the lifecycle.

  8. sonali

    Totally agree with you.I am having 5 years of manual testing experience and finding job from last 3 months.Most of the company wants programming skilled testers.I have worked on agile and good at exploratory testing but still the company wants programming knowledge.I don’t understand the logic because both of the roles are different.If i love programming then i should be a developer rather than a tester.I am a manual tester and company should take an advantage of my QA skills.The best use of manual testing is investigation in to the product or infrastructure before writing the automation code.

  9. Lukas

    Hi Ardesco
    Excellent article and thanks for the other about Download dialogs in Selenium. This points out what I was thinking quietly but wasn’t able to articulate it, I saw much of bad automation. Some people try to do automation with avoiding to learn programming completely. I do work as automation tester but always focused on improving programming skills and still see areas I should improve, this is infinite process and hope I do not belong to the category of forever-developer-in-test-to-be.
    I guess you are one of better automation testers after I saw your blog today.

    • Ardesco

      Glad you found it useful :)

  10. Mark

    How functional test automation sells itself as superior to sapient testing…

    https://www.atlassian.com/test-automation
    http://support.smartbear.com/articles/testcomplete/manager-overview/

    1) Automation allows you to run a lot of tests really fast…way faster than a human if they were to follow the same ‘steps’.

    Counter argument:
    While this may be true…what about the quality of the tests? Are they telling is anything useful aside asserting that certain static elements exist on a webpage or a certain HTTP message is received.

    I suggest that when there are many automated test cases, they are not that different from each other. The ‘number’ of tests does not indicate ‘quality’ of tests.

    In addition, if a human performed the same steps as an automated test they would notice problems the script is not looking for. If they find a problem they don’t just log it and move on…..they change their approach and push that problem and thus discover more useful information.

    2) Automated tests are more accurate than human tests because they perform the same steps every time

    Counter argument:
    Assuming this were always true (and I don’t think it is) I still say…..big deal.

    Automated tests are created by humans and are prone to mistakes. If there is an error in the script it will be repeated. If you missed something once, you will miss it every time.

    Assuming that complete and exact reproducibility is achievable, how effective is it to perform the exact same steps with the exact same data on the AUT. The pesticide paradox applies here.

    In addition…if the test code is changing then the test has changed. Yes, it might change very little but is not ‘exactly’ the same test as when you ran it before. Changed a locator, nature of assertion, new test data….all these things modify the test.

    In any case, I think exact reproducibility is not worth the effort in many contexts.

    Aren’t we living in an Agile world where requirements are supposedly always changing (another myth…requirements do change sometimes but mostly they don’t….our understanding/realization of the requirement changes because we didn’t get it right before).

    In such a case of constant change, how useful are static functional tests?

    Each time a human repeats a test they approach it differently, they may use more up to to date data, the may change the order in which the complete the data entry, they may change the flow to achieve the same desired end point.

    When I do manual regression, I don’t repeat my tests exactly, I run the test to verify the same test objective as before. I also don’t do full regression each time a line of code changes. I test the module that changed and the modules indirectly affected. My bosses would not be happy if I insisted on retesting everything each time a single line of code changes.

    3) Automation can increase test coverage (by magic?)

    Counter argument:
    This is an easy one. This claim is not backed up with any data. I suppose it is possible in theory but unless the cost of automating everthing is factored in we cannot say if it is better than manual.

    In many cases, it might be effective to automate 70% coverage (given the appropriate level of resources and care.) Trying to close the gap and automate everything is a different story, you may find that the last 30% of automation costs a kings ransom to create and maintain.

    ===================================================================
    Actually I am a proponent of functional and other test automation. I picked these arguments from people who are selling automation so their claims are very optimistic (or should I say…biased).

    I understand that real test automation people are more realistic about the benefits of automation.

    We need to appreciate the value of sapient testing and balance manual vs automation with realistic expectations about what they can achieve.

    Viva Sapient Testing!


Leave a Reply




seven + = sixteen