Archive for the ‘Software Testing’ Category

Bug Reporting

08/17/2010

Having been in the role of Test Lead on many projects both large and small I have seen many, many bug reports. I’ve seen good ones, bad ones and some that I wouldn’t call bug reports at all.

As a Test Lead I have been asked to coordinate the testing effort with experienced testers as well as inexperienced temp resources. From this experience I can say that one of the things that can eat up a lot of your time is bad bug reporting.

On smaller projects it was easier to deal with. There were less stakeholders and the time allotted for testing was manageable so I had time to work with the testers as the bugs started rolling in. On the large, enterprise-wide projects it was much harder. Usually the projects had an absolute deadline for production (for various reasons) and the product was almost always late to test. More time is spent managing the testing effort then triaging bugs in batches at a later time.

I think I should back up a bit and make a statement that these projects were not run in an Agile fashion. These were using a waterfall-type methodology for product delivery and if you’ve every worked with a waterfall-type method you may already understand the issues I faced in the last paragraph. But the issue at hand is bug reporting, which I believe to be the same across any methodology.

Bug reports, at the very least, should contain the following:

Clear and Concise Title
“Got an error” is about the most useless title I have witnessed. It tells me nothing. It tells the developer nothing. Since the title of the bug is usually the first thing a bug reviewer will see it should be a concise record of the problem.

“Java Runtime error when selecting Checkout in Shopping Cart” would be an example of a decent title. A great title could contain (but is not limited to) more information such as test area, test case name and test step. “Checkout – TC 1802 – BUY EGC – Step 4 – Java Runtime error when selecting Checkout in Shopping Cart” I can now glance at the title and I already understand what the problem is.

Of course, depending on what you are using to track bugs you may be limited to the amount of characters you can enter or that can be seen in a quick view of the bug list. If this is the case do your best to offer as much concise information in the title as possible.

Detailed Problem Definition

In the body of the bug report give as much detail about the issue witnessed as possible and tailor the wording to your audience. On some larger projects we would have daily bug review meetings where the priority of the bug would be determined not by developers but by the non-technical stakeholders.

Knowing this we would want to tailor the description of the problem in non-technical terms then drill down to the technical details. If we were on a small project where the bugs would go directly to the developers we might write the description of the problem in more technical terms.

As with the title, “Got an error” is not useful information to anyone. Having already put as much concise information in the title as we could, the description should contain a more detailed review of the problem. This could include:

  • Test Case Name
  • Test Steps with point of failure noted and/or steps to reproduce the issue
  • Detailed problem description – one example of how this differs from last two bullets is if the test step fails only on one type of hardware/OS/Browser etc. and not another. If the developer tries to reproduce the problem where the problem doesn’t exist the bug may get rejected causing project delays and/or bugs pushed to production that shouldn’t have.
  • Error text or detailed description of the problem witnessed if no error is thrown
  • Screen Captures or video of the problem

Every project is different in some way. As a Test Lead I would suggest evaluating what an acceptable bug report should be dependent on the makeup of the resources on hand and the information flow set forth by the project prior to the testing effort. Then be sure your testing team adheres to the template. It will save you and the project a lot of time.

Please feel free to discuss in the comments below.

Who Am I? – The Uncertainty of the Laid Off Software Tester

05/20/2010

I was perusing the uTest.com blog and came across the interview “Testing the Limits with Lanette Creamer”. Lanette talks about the uncertainty of a pending lay off from her software testing role asking the questions “Who am I if I’m not a tester? Are testing skills relevant in the marketplace? Is talent at finding bugs totally obsolete?”

Earlier this year I found myself in a similar mindset after having been laid off after 12 years in a tester/test management role. Questions circled in my head as well. “Did I stay at one place too long?” “Am I stale and therefore irrelevant in my field?” “What now?”

My first reaction was to find a job, any job and quickly. I applied for just about every position that was available even if I wasn’t sure I’d be happy working for the company. It was panic mode and in hindsight it was not helpful to my mental well-being.

One thing I had going for me was networking. I had been an early adopter of LinkedIn.com and my professional network contact list was fairly large. I could find out more about the companies I was applying to before hitting the submit button by talking to my contacts and making new contacts at that very company.

The other thing that helped was that I knew that I didn’t know what I didn’t know. All those questions about my relevance as a tester needed to be answered. So I started searching for them. I search the Google Tech Talk videos and watched every video that had to do with Software Testing.

I found James Bach’s tech talk and found it very inspiring. He held many of the same beliefs about testing and who a tester is that I did. “And look he’s successful! I can be too.” So sent him an email explaining my situation and asked him for a direction to follow. There’s so much diversity in software testing that I couldn’t possibly study it all before my first job interview. James was very helpful in getting me on my own path and providing me support.

One of the first things he did for me (besides run me through a Test Challenge over Skype!) was introduce me to Matt Heusser who lived in my regional area. The next day I was talking with Matt on the phone about my situation (Matt also set up a Test Challenge for me).

Between the two of them and a bit of time the fear of uncertainty that came with the layoff subsided and I built my “way forward”. I still don’t have a full-time job, but I’ve managed to get work through crowdsourcing companies and other opportunities. I’m not making as much as I was, but I’m making enough to be happy and also to be selective with the companies posting jobs.

So to Lanette and any other software tester who may be in a similar situation I offer this advice. Know who you are. Know that your skills are relevant. Lean on your community for support and direction and make your own direction.

Who am I? I’m a software tester and I’m a damn good one.

Competitive Testing: The Good, The Bad and The Ugly

05/01/2010

The notion of competitive testing has been spinning around in my brain for the past few days and I’m not sure how I feel about it. Maybe putting it in writing will help.

I’m naturally adverse to competition as a catalyst for self-importance. While I may feel good having done well in a competition, I feel the results don’t define me as a person. I think this goes back to high-school where a B was just as good as an A in my opinion. “So what if I didn’t know the date of <insert historical event>, at least I understood the concept of what happened and why it’s important!”

I suppose we should start with “The Good”. Competitive testing in the form of bug battles is a great way to sharpen your skills, meet other testers and maybe learn a thing or two along the way. I have no problem with this.

So we move on to “The Bad” and, of course, this is completely subjective depending on the person.

Recently I’ve been involved in a crowdsourcing movement where most of the pay is based on finding bugs through exploratory testing. Basically throw a bunch of testers into an arena with a thin test plan and broad scope and say “Go find bugs”. While I believe this is a great way for a company to find problems with its software for a much cheaper price than having a salaried QA department I’m looking at it from the other side.

Now, I don’t want to come across as being against the crowdsourcing movement. Quite the contrary in fact, it has been a boon to my personal finances and networking connections in a time when I truly needed it. I was able to navigate the system successfully in a short time and make it work for me. But I have concerns for those truly skilled testers that may be in a similar position that may not possess the ability to navigate as successfully as I did.

The problem I see is that the client will pay for the bug, but the tester needs to be the first to find the bug and report it to get paid. All others may also find the bug but they weren’t quick enough, or their internet speed wasn’t fast enough. Miss by a millisecond and you get nothing for the effort and in some cases will be reprimanded for entering a duplicate bug.  Does this mean that the tester who was faster is a better tester? I don’t think so and I can’t help but wonder how many testers take this situation as a hit to their credibility?

Now for “The Ugly”. I am a firm believer of being clear and concise in reporting a bug, building a test case, etc. Reporting “Got an error” doesn’t cut it for me. In my opinion a tester needs to give the maximum possible information about an issue that he/she can find within the time constraints forced upon them by the scope.

I have seen many bugs reported in these crowdsourcing projects where the first bug reported did not contain quality information about the problem. I’m sure the client or PM had to go back and request more information. But still this bug is accepted where someone else may have found the same bug but submitted it later with quality information that can be acted upon. Yet, this tester gets nothing for the effort of creating a quality bug report.

Now, of course these are the extreme examples and the exception to the rule. For the most part I’ve seen crowdsourcing work very well when it’s well-executed, witnessed the community self-regulate and provide an overall quality report on the maturity and market-readiness of an application. It’s these edge cases that have had me thinking.

Additionally, I should also mention that I’ve only been involved with this type of crowdsourced movement for just under two months now. And as I had suspected, more and more “high profile” projects are being made available based on my performance here. Even for a veteran tester, it’s great to have such exposure to a diverse set of projects (for example, I’m about to get a taste of mobile testing in the coming week).

I have raised my concerns with managers in the crowdsourced testing company I am working with prior to posting this blog entry and I’m pleased that they recognize the need for continued improvement, and have concrete plans to level the playing field for all testers, allow for quality testing to rise to the top thus improving the overall quality of the service they are selling. I truly believe that they care about quality and, just as importantly, fairness between testers and customers.  It’s refreshing to know that they genuinely strive to raise the bar.

Favorite Bugs

04/19/2010

I was watching the movie “Raiders of the Lost Ark” the other day and it reminded me of one of the first (and favorite) software bugs I ever found. “Raiders” features John Rhys-Davies in one of my favorite roles, the character Sallah.

One of the first projects I worked on as a green tester was a customer-facing inventory kiosk for a major book chain. I wouldn’t say I knew what I was doing at the time as I hadn’t yet honed my skills but I knew enough to try a few things to see how it would react.

One of the functions of this particular piece of software was the ability to look up in-store inventory based on various search criteria such as Author, Title, ISBN, etc. I chose to run a simple search for “John Rhys-Davies”.

So I entered the name and clicked “Search”. Lo and behold I killed the system. Not just this kiosk but the whole system went offline. If this had been production I would have brought down every kiosk in every store nationwide.

Now being green to testing I didn’t know why this particular search killed the system but I quickly learned the reason. The system had a real hard time dealing with special characters. The hyphenated name “John Rhys-Davies” brought this to our attention.

Shortly after finding the issue I printed out a photo of “John Rhys-Davies” and captioned him “The Patron Saint of Software Testing”. That photo remained on my desk for 10 years as a reminder (and a great topic of discussion).

My second favorite bug was found on the very same system. The company had decided to add a printing feature so customers could print out title information and shelf locations within the store.

I wasn’t technically on the project but the kiosk was in our QA lab and I had some time. So I went up to it and ran a search (probably John Rhys-Davies!). Once the search was returned I clicked the “print” icon.

The kiosk had been outfitted with a small thermal printer the kind you might find on a cash register. My expectation of the printer was it would print the information on a small 3×5 sheet. But when I clicked “print” that’s not quite what happened.

What happened was a 2 foot long sheet of paper shot passed my neck at about 100mph and landed 6 feet behind me! Surprisingly it contained the correct information, but it could have resulted in a nasty paper cut to my neck. Since it didn’t cause injury I just busted out laughing. One problem though, no one else witnessed it.

I made sure I could recreate the issue a few more times before calling my boss in to see the cool new bug. I didn’t tell him what it was I just said “You have to see this!” And when he did, he too about fell out of his chair laughing. He actually called the CIO down to the lab to witness it. After the CIO stopped laughing he calmly stated, “This is not going into my stores.” The issue, it turns out, was an incorrect driver for the printer.

I’m sure everybody has a favorite bug or two. What’s yours?

Insatiable Curiosity

04/06/2010

A long time ago, in house in the middle of nowhere, Michigan my parents finally relented to the pressure I was putting on them to buy an Atari 2600 for me. All my friends had one and you can’t get good at the games by playing against them for an hour at their house. So I kept bugging them and it finally paid off, sort of.

What I got was the Sears equivalent of the 2600. But hey, it looked similar and played all the games so I was happy. It came with the game Space Invaders, one of my favorites, and played for hours. But like all things the novelty wore off in time.

Then something happened. I got bored with the game. I mean, it doesn’t really change that much and once you figure out the patterns it gets pretty easy. Sure the aliens keep coming at you faster and faster while you still move at the same pace and can only shoot one missile at a time, but it’s the same thing over and over. It was at this time my attention turned to the console itself.

Not really sure what I was doing I just started playing with the on/off and reset buttons. What happens if I switch the console on and off repeatedly? Let’s find out!

What happened was a wondrous thing! After a few attempts goofing around with the on/off switch I noticed that now, instead of one missile, I could fire two at a time! SCORE! The game all of a sudden became interesting again and I could reach high scores I had only dreamed about.

So what next? Let’s try it with the Pac-Man game. Same procedure, different game cartridge. What will happen? I have no idea, let us find out. Well guess what? I found you could make yourself immune to the ghosts that were your enemies. In fact you became a ghost-of-a-Pac-Man yourself. Slightly transparent, immune to attack and able to rack up points. Just don’t enter the tunnel, it had turned into a black hole that would throw you out of the tunnel on the opposite end that you went into and right back in. Forever. Game over.

So I guess that insatiable curiosity that manifested in me at a young age has guided my life and career. I taught myself to play the guitar at age 16 just to see if I could do it. Now when I play guitar I might hear a piece of music in a style I’ve never played before and instead of trying to learn the piece, I try to write an original composition in the same style. Just to see if I can do it. I’ve learned I get a lot more out of this than just copying what someone else has already written.

Somewhere along my lifeline I fell into Software Testing. I jumped in with no real knowledge of the craft but I had that insatiable curiosity within me. Give me an application. Give an idea of what it’s supposed to do and I’ll jump on it. Not only will I prove it does what it’s supposed to do but I’ll click the on/off switch repeatedly to see if I can make it do something unintended.