Why the need for Automation?
This topic has been explored by many authors in books and blogs. Automation exists in many industries so don't think that its a "IT" thing. The cookies you are enjoying right now, the milk you're drinking to the foods you are eating are highly automated. Automation, in my opinion, arose from a human trait - being more effective and productive.
I'm not going to write about industrial automation since its not something i do from day to day back then though i suspect they share similar traits; so i'm going to focus on automating software or software automation and there's an area called "automated software testing" which is a subset of software automation,imo.
One reality of Automation,i realized, is that it'll take time to setup the necessary processes, applications and would draw some of your best QA talent from their daily work but the benefits you will reap will far outweigh its initial drawbacks for the simple reason that you would have freed up the resources you previously held and hence you have more QA resources to work on the really big problems that need attention.
What made me decide to do Automation from QA?
Quite some time back, a company i used to work for recognize that over a long run QA couldn't effectively run its business when you consider a server solution that supported tens of thousands of 3D-clients connected to a back-bone of 10,000 servers.
It was an exciting opportunity for me, personally, since white-box testing was a largely manual process but there were many other QA tasks that could be automated and hence raise overall efficiency and i wanted to explore that.
What do you automate?
This is tricky stuff. Some folks would like to automate everything but that is not practical. Focus on the problem you're trying to solve that is not is not easily solvable but if you pull it off, you can solve all other problems.
Many of the stuff you can list as potential candidates often manifest themselves as problems you've been solving repeatedly so a couple of examples could be:
- Application startup, shutdown
- Application deployment (I know some orgs actually still rely on human efforts)
- Application build and release
- Application build, test and release
- ...many others
Back then, QA implemented many processes w.r.t testing, coding standards, code reviews and the like and it was a rigorous effort. But still bugs are reported by the users and that means that our test coverage was inadequate. The answer to this question is not simply to add more QA resources but to solve the problem in a smarter manner...
Don't get me wrong here, there was automated testing implemented but neither QA nor the Devs had any visibility into how the tests relate to one another, how they vary from release to release and its a tough thing to keep track of considering that you have tens of thousands of tests cases. QA couldn't effectively exhaust all possible testing scenarios.
In effect, what we/QA/Dev wanted was something like this
1) Build the app and run automated tests on a platform type. e.g. Win32, Win64, Linux32 etc; report success/failures
2) Pre-release (automated) testing to a controlled environment; report success/failures
3) Collect & archive test metrics for trending purposes (afaik, it wasn't implemented by the time i left that company)
The end result of a series of brain storming sessions revealed that we needed a new way of doing things! Continuous Integration and SCM tools help realized 1) and 2) :)
Source Control Management - How does that come into the picture?
A big part of CI is to set up several gateways where the quality of the code can be checked before the actual build happens. As devs, we all know that when your code base expands rapidly, the time it takes to build an application becomes longer (not to mention source code locking) so it's best not to check-in code that doesn't compile or work else you'll incur the wrath of your co-workers.
Hence, the first gateway you should consider is scrutinizing the code that's being checked in. Modern SCM tools like Mercurial have a simple regression test framework in python & shell-script driven testing that runs automated tests for your code. Another example would be SVN, read about it here.
Check-in code -> SCM runs first-line defense -> rejects code if tests fail, accepts code otherwise.
SCM is not enough ...
Of course, this scrutinizing is somewhat limited in this effectiveness - so you need to ensure that devs check in tests that accompany the codes. This is TDD or Test Driven Development and back then, the product was a mix of C++, Python, Perl and so you can imagine that we used lightweight regression test frameworks to incorporate tests.
Alright, you say, now you've seem to have all the tests incorporated but how does that solve the problem? Who will run the tests and report the results? Before that, how does anyone know that code's been checked-in and builds need to happen?
You need Continuous Integration.
Continuous Integration - How does that help?
Martin Fowler said it best. Click this to read about it. Key thing to remember about CI is that frequent code check-ins trigger automated builds and that flashes errors faster than the usual build-test-release cycle you are used to.
A typical CI tool would allow have the capability to pull sources codes out onto its base, compile and run the automated tests you've built in - of course, tests results would be published to the tool's dashboard and alert the QA/Dev that a build/test failed and require attention.
Another thing that a CI tool would do is to allow you to upload modules that measure and informs you the coverage of your tests!
That is a really cool thing to have :) because a tool like this allows both QA and Dev to see the same problem in the same language.
CI Tool detects code checked-in -> runs build -> runs automated tests -> runs coverage tests -> reports build pass/fail & publishes test artifacts to dashboard.
But is that all it can do? Fortunately, no. Read on.
Extended Automation Framework via Python
Previously, i mentioned that we had a CI tool that builds, tests the app. In addition, the tool archived/published test results so that Devs/QA can determine which code check-ins caused a particular test/build to fail.
Now, i want to share with you something else we did. Part of the problem we had was that black box QA were quite involved in manual testing of the 3D client because there was no automated way to interact with such an application, even less report errors in a concise manner.
In a nutshell, we build a Python framework that allows any Dev/QA to control the 3D client in Linux, Windows & Mac OS X. The idea was to allow our QA/Devs to write automated test scripts in python so that the manual testing could be relieved. The framework came with a small set of APIs back then, but i'm sure it has grown much bigger these days.
Can you imagine how our test coverage blew sky high?!!!!
That was a tough problem to crack but we did it :) and that's because we had a bunch of dedicated and passionate engineers working on it.
But, that's not the end of the story ;)
With such a framework in place, QA now had the capability to run automated tests with a freshly build app (from the CI tool) in an environment of their choice. So, it resembles something like this:
CI Tool -> builds app -> test the (native) app through automated test scripts in Python -> publishes test results back to CI Tool's dashboard.
So that's it.
0 comments:
Post a Comment