Bug Reporting

08/17/2010

Having been in the role of Test Lead on many projects both large and small I have seen many, many bug reports. I’ve seen good ones, bad ones and some that I wouldn’t call bug reports at all.

As a Test Lead I have been asked to coordinate the testing effort with experienced testers as well as inexperienced temp resources. From this experience I can say that one of the things that can eat up a lot of your time is bad bug reporting.

On smaller projects it was easier to deal with. There were less stakeholders and the time allotted for testing was manageable so I had time to work with the testers as the bugs started rolling in. On the large, enterprise-wide projects it was much harder. Usually the projects had an absolute deadline for production (for various reasons) and the product was almost always late to test. More time is spent managing the testing effort then triaging bugs in batches at a later time.

I think I should back up a bit and make a statement that these projects were not run in an Agile fashion. These were using a waterfall-type methodology for product delivery and if you’ve every worked with a waterfall-type method you may already understand the issues I faced in the last paragraph. But the issue at hand is bug reporting, which I believe to be the same across any methodology.

Bug reports, at the very least, should contain the following:

Clear and Concise Title
“Got an error” is about the most useless title I have witnessed. It tells me nothing. It tells the developer nothing. Since the title of the bug is usually the first thing a bug reviewer will see it should be a concise record of the problem.

“Java Runtime error when selecting Checkout in Shopping Cart” would be an example of a decent title. A great title could contain (but is not limited to) more information such as test area, test case name and test step. “Checkout – TC 1802 – BUY EGC – Step 4 – Java Runtime error when selecting Checkout in Shopping Cart” I can now glance at the title and I already understand what the problem is.

Of course, depending on what you are using to track bugs you may be limited to the amount of characters you can enter or that can be seen in a quick view of the bug list. If this is the case do your best to offer as much concise information in the title as possible.

Detailed Problem Definition

In the body of the bug report give as much detail about the issue witnessed as possible and tailor the wording to your audience. On some larger projects we would have daily bug review meetings where the priority of the bug would be determined not by developers but by the non-technical stakeholders.

Knowing this we would want to tailor the description of the problem in non-technical terms then drill down to the technical details. If we were on a small project where the bugs would go directly to the developers we might write the description of the problem in more technical terms.

As with the title, “Got an error” is not useful information to anyone. Having already put as much concise information in the title as we could, the description should contain a more detailed review of the problem. This could include:

  • Test Case Name
  • Test Steps with point of failure noted and/or steps to reproduce the issue
  • Detailed problem description – one example of how this differs from last two bullets is if the test step fails only on one type of hardware/OS/Browser etc. and not another. If the developer tries to reproduce the problem where the problem doesn’t exist the bug may get rejected causing project delays and/or bugs pushed to production that shouldn’t have.
  • Error text or detailed description of the problem witnessed if no error is thrown
  • Screen Captures or video of the problem

Every project is different in some way. As a Test Lead I would suggest evaluating what an acceptable bug report should be dependent on the makeup of the resources on hand and the information flow set forth by the project prior to the testing effort. Then be sure your testing team adheres to the template. It will save you and the project a lot of time.

Please feel free to discuss in the comments below.

The Art of Hunting

07/13/2010

I’m a hunter by nature. I grew up in a family of outdoorsmen who loved to hunt game and fish whenever they could get the chance. Most of our family vacations revolved around hunting or fishing. Even if they didn’t it was sure bet that a fishing pole was handy just in case the opportunity arose.

I was raised in a county in Northern Michigan where hunting the Whitetail Deer was more of a religion than a pastime. The preparation for the next year’s hunt began the day after the season closed.

I have many fond memories of scouting for sign with my dad before I was legally allowed to hunt on my own.  I was probably five or six years old at the time. We’d get in the jeep and head out to the hunting grounds and start looking for trails, bedding areas, food and water sources, cover and start to form a picture of the deer’s habits. You could see the traces of what they had been doing and there was a good chance they would continue to do so. Once the picture was complete you had a good idea where to set up for the hunt and the chances of it being a success was much greater than someone who just walks into a forest and sits down.

Now that’s not so say that just showing up on opening day and plopping your butt down at the base of a tree and waiting won’t garner a successful hunt. Luck plays a factor in any hunt, but if you took the time to prepare for it and had that mental picture of where and when the deer would appear then you have the advantage.

Deer come in all shapes and sizes. The ultimate goal of the hunt is to get the big buck with the trophy antlers, but bagging a smaller buck or a doe would mean a successful hunt as well. For the true outdoorsman not bagging anything would be considered a successful season for the sheer fact that they were out there in the woods.

Software testing is not much different. Knowledge and preparation can play a key role in hunting software bugs. A little luck on your side doesn’t hurt either.

With a little luck on your side you don’t need to know anything about software bugs to find one. Users do this every day across the globe. They log in to their apps and stumble on a bug. But for the software tester knowing a bit about your game can make you more successful.

The more you can find out about the application you are testing and how it’s intended to be used the better you chances of  assisting in the overall quality of the application by finding the bugs prior to a production release.

I mentioned earlier about finding deer trails while scouting with my dad for a place to hunt. One thing about this kind of sign is that it can be misleading. While it’s true that the majority of the deer have used this trail in the past, it’s also true that other factors play a role in the hunt. During the hunting season, with the added pressures of more humans in their territory and the added fact that “the rut”, the deer’s breeding season, is on the deer is more inclined to travel 10 to 20 yards off the main trail. So if you were to set up your hunt right next to the trail you saw while scouting, there is a good chance the deer will walk right behind you during the hunt.

The same can be said for “Happy Trails” software testing. I’ve worked on many projects where the business owners or project managers, for whatever reasons (usually deadline constraints), insisted that we only focus on the “happy trails” scenarios. Basically we’re talking about only testing the main paths to an applications function. This testing was meant to prove that the application worked, not to find bugs. The problem I have with limiting testing to “Happy Trails” is the same as setting up your stand directly over a deer trail, the chance of finding a significant problem (your game) are greatly reduced.

One real-life example I witnessed was with a retail company that wanted to have a survey printed on every 20th receipt that was printed. The testing consisted of making sure you could make a sale, accept all forms of tender and that every 20th receipt printed had the survey attached to it. Because this project was taking place in the fourth quarter (highest volume of sales for the year) thorough testing was disregard in favor of getting the survey out as quick as possible.

When the “Happy Trails” testing was complete all tests had passed and the release was sent into the field. If you’re reading this blog you probably have that feeling in your gut that something went horribly, horribly wrong. Your instincts are correct, because they set up their stand on the trail the bug walked right by them from behind.

About three weeks after the “successfully tested” release was in the field, the company started to notice a large discrepancy between the sales numbers and the actual revenue received. As it turned out, while the testers did test to make sure all tenders were accepted on sales, they neglected to test whether the type of tender used on that 20th sale had any affect on the changes made to the Point-of-Sale functions.

What was eventually discovered, through more testing, was that if every 20th transaction was paid for using a credit card, the transaction would complete, the receipt would be printed complete with the attached survey but the credit card information was never sent for settlement.

Now they had a real problem. Instead of making sales, they were giving away product free of charge and asked the people how enjoyable it was to get free stuff!

At least they realized the problem in time to save some money. The credit card reconciliation process took about a month before all was lost. If they could get these transactions to the credit card companies before this time, then the cards could be processed and the company would get its money.

But at what cost? All the stores were required to send their printed journals to the home office where a team of people had to go through each transaction, one-by-one, to find the problem transactions and get them to the credit company for approval.

In my opinion the cost of doing more thorough testing up front surely would have been less than the cost of finding the problem after the fact, the cost of sending all those journals to be reviewed and reviewing them. And who know how many transactions never got reconciled?

Know your game, be as prepared as possible and the odds are in your favor.

Who Am I? – The Uncertainty of the Laid Off Software Tester

05/20/2010

I was perusing the uTest.com blog and came across the interview “Testing the Limits with Lanette Creamer”. Lanette talks about the uncertainty of a pending lay off from her software testing role asking the questions “Who am I if I’m not a tester? Are testing skills relevant in the marketplace? Is talent at finding bugs totally obsolete?”

Earlier this year I found myself in a similar mindset after having been laid off after 12 years in a tester/test management role. Questions circled in my head as well. “Did I stay at one place too long?” “Am I stale and therefore irrelevant in my field?” “What now?”

My first reaction was to find a job, any job and quickly. I applied for just about every position that was available even if I wasn’t sure I’d be happy working for the company. It was panic mode and in hindsight it was not helpful to my mental well-being.

One thing I had going for me was networking. I had been an early adopter of LinkedIn.com and my professional network contact list was fairly large. I could find out more about the companies I was applying to before hitting the submit button by talking to my contacts and making new contacts at that very company.

The other thing that helped was that I knew that I didn’t know what I didn’t know. All those questions about my relevance as a tester needed to be answered. So I started searching for them. I search the Google Tech Talk videos and watched every video that had to do with Software Testing.

I found James Bach’s tech talk and found it very inspiring. He held many of the same beliefs about testing and who a tester is that I did. “And look he’s successful! I can be too.” So sent him an email explaining my situation and asked him for a direction to follow. There’s so much diversity in software testing that I couldn’t possibly study it all before my first job interview. James was very helpful in getting me on my own path and providing me support.

One of the first things he did for me (besides run me through a Test Challenge over Skype!) was introduce me to Matt Heusser who lived in my regional area. The next day I was talking with Matt on the phone about my situation (Matt also set up a Test Challenge for me).

Between the two of them and a bit of time the fear of uncertainty that came with the layoff subsided and I built my “way forward”. I still don’t have a full-time job, but I’ve managed to get work through crowdsourcing companies and other opportunities. I’m not making as much as I was, but I’m making enough to be happy and also to be selective with the companies posting jobs.

So to Lanette and any other software tester who may be in a similar situation I offer this advice. Know who you are. Know that your skills are relevant. Lean on your community for support and direction and make your own direction.

Who am I? I’m a software tester and I’m a damn good one.

Competitive Testing: The Good, The Bad and The Ugly

05/01/2010

The notion of competitive testing has been spinning around in my brain for the past few days and I’m not sure how I feel about it. Maybe putting it in writing will help.

I’m naturally adverse to competition as a catalyst for self-importance. While I may feel good having done well in a competition, I feel the results don’t define me as a person. I think this goes back to high-school where a B was just as good as an A in my opinion. “So what if I didn’t know the date of <insert historical event>, at least I understood the concept of what happened and why it’s important!”

I suppose we should start with “The Good”. Competitive testing in the form of bug battles is a great way to sharpen your skills, meet other testers and maybe learn a thing or two along the way. I have no problem with this.

So we move on to “The Bad” and, of course, this is completely subjective depending on the person.

Recently I’ve been involved in a crowdsourcing movement where most of the pay is based on finding bugs through exploratory testing. Basically throw a bunch of testers into an arena with a thin test plan and broad scope and say “Go find bugs”. While I believe this is a great way for a company to find problems with its software for a much cheaper price than having a salaried QA department I’m looking at it from the other side.

Now, I don’t want to come across as being against the crowdsourcing movement. Quite the contrary in fact, it has been a boon to my personal finances and networking connections in a time when I truly needed it. I was able to navigate the system successfully in a short time and make it work for me. But I have concerns for those truly skilled testers that may be in a similar position that may not possess the ability to navigate as successfully as I did.

The problem I see is that the client will pay for the bug, but the tester needs to be the first to find the bug and report it to get paid. All others may also find the bug but they weren’t quick enough, or their internet speed wasn’t fast enough. Miss by a millisecond and you get nothing for the effort and in some cases will be reprimanded for entering a duplicate bug.  Does this mean that the tester who was faster is a better tester? I don’t think so and I can’t help but wonder how many testers take this situation as a hit to their credibility?

Now for “The Ugly”. I am a firm believer of being clear and concise in reporting a bug, building a test case, etc. Reporting “Got an error” doesn’t cut it for me. In my opinion a tester needs to give the maximum possible information about an issue that he/she can find within the time constraints forced upon them by the scope.

I have seen many bugs reported in these crowdsourcing projects where the first bug reported did not contain quality information about the problem. I’m sure the client or PM had to go back and request more information. But still this bug is accepted where someone else may have found the same bug but submitted it later with quality information that can be acted upon. Yet, this tester gets nothing for the effort of creating a quality bug report.

Now, of course these are the extreme examples and the exception to the rule. For the most part I’ve seen crowdsourcing work very well when it’s well-executed, witnessed the community self-regulate and provide an overall quality report on the maturity and market-readiness of an application. It’s these edge cases that have had me thinking.

Additionally, I should also mention that I’ve only been involved with this type of crowdsourced movement for just under two months now. And as I had suspected, more and more “high profile” projects are being made available based on my performance here. Even for a veteran tester, it’s great to have such exposure to a diverse set of projects (for example, I’m about to get a taste of mobile testing in the coming week).

I have raised my concerns with managers in the crowdsourced testing company I am working with prior to posting this blog entry and I’m pleased that they recognize the need for continued improvement, and have concrete plans to level the playing field for all testers, allow for quality testing to rise to the top thus improving the overall quality of the service they are selling. I truly believe that they care about quality and, just as importantly, fairness between testers and customers.  It’s refreshing to know that they genuinely strive to raise the bar.

Favorite Bugs

04/19/2010

I was watching the movie “Raiders of the Lost Ark” the other day and it reminded me of one of the first (and favorite) software bugs I ever found. “Raiders” features John Rhys-Davies in one of my favorite roles, the character Sallah.

One of the first projects I worked on as a green tester was a customer-facing inventory kiosk for a major book chain. I wouldn’t say I knew what I was doing at the time as I hadn’t yet honed my skills but I knew enough to try a few things to see how it would react.

One of the functions of this particular piece of software was the ability to look up in-store inventory based on various search criteria such as Author, Title, ISBN, etc. I chose to run a simple search for “John Rhys-Davies”.

So I entered the name and clicked “Search”. Lo and behold I killed the system. Not just this kiosk but the whole system went offline. If this had been production I would have brought down every kiosk in every store nationwide.

Now being green to testing I didn’t know why this particular search killed the system but I quickly learned the reason. The system had a real hard time dealing with special characters. The hyphenated name “John Rhys-Davies” brought this to our attention.

Shortly after finding the issue I printed out a photo of “John Rhys-Davies” and captioned him “The Patron Saint of Software Testing”. That photo remained on my desk for 10 years as a reminder (and a great topic of discussion).

My second favorite bug was found on the very same system. The company had decided to add a printing feature so customers could print out title information and shelf locations within the store.

I wasn’t technically on the project but the kiosk was in our QA lab and I had some time. So I went up to it and ran a search (probably John Rhys-Davies!). Once the search was returned I clicked the “print” icon.

The kiosk had been outfitted with a small thermal printer the kind you might find on a cash register. My expectation of the printer was it would print the information on a small 3×5 sheet. But when I clicked “print” that’s not quite what happened.

What happened was a 2 foot long sheet of paper shot passed my neck at about 100mph and landed 6 feet behind me! Surprisingly it contained the correct information, but it could have resulted in a nasty paper cut to my neck. Since it didn’t cause injury I just busted out laughing. One problem though, no one else witnessed it.

I made sure I could recreate the issue a few more times before calling my boss in to see the cool new bug. I didn’t tell him what it was I just said “You have to see this!” And when he did, he too about fell out of his chair laughing. He actually called the CIO down to the lab to witness it. After the CIO stopped laughing he calmly stated, “This is not going into my stores.” The issue, it turns out, was an incorrect driver for the printer.

I’m sure everybody has a favorite bug or two. What’s yours?

Insatiable Curiosity

04/06/2010

A long time ago, in house in the middle of nowhere, Michigan my parents finally relented to the pressure I was putting on them to buy an Atari 2600 for me. All my friends had one and you can’t get good at the games by playing against them for an hour at their house. So I kept bugging them and it finally paid off, sort of.

What I got was the Sears equivalent of the 2600. But hey, it looked similar and played all the games so I was happy. It came with the game Space Invaders, one of my favorites, and played for hours. But like all things the novelty wore off in time.

Then something happened. I got bored with the game. I mean, it doesn’t really change that much and once you figure out the patterns it gets pretty easy. Sure the aliens keep coming at you faster and faster while you still move at the same pace and can only shoot one missile at a time, but it’s the same thing over and over. It was at this time my attention turned to the console itself.

Not really sure what I was doing I just started playing with the on/off and reset buttons. What happens if I switch the console on and off repeatedly? Let’s find out!

What happened was a wondrous thing! After a few attempts goofing around with the on/off switch I noticed that now, instead of one missile, I could fire two at a time! SCORE! The game all of a sudden became interesting again and I could reach high scores I had only dreamed about.

So what next? Let’s try it with the Pac-Man game. Same procedure, different game cartridge. What will happen? I have no idea, let us find out. Well guess what? I found you could make yourself immune to the ghosts that were your enemies. In fact you became a ghost-of-a-Pac-Man yourself. Slightly transparent, immune to attack and able to rack up points. Just don’t enter the tunnel, it had turned into a black hole that would throw you out of the tunnel on the opposite end that you went into and right back in. Forever. Game over.

So I guess that insatiable curiosity that manifested in me at a young age has guided my life and career. I taught myself to play the guitar at age 16 just to see if I could do it. Now when I play guitar I might hear a piece of music in a style I’ve never played before and instead of trying to learn the piece, I try to write an original composition in the same style. Just to see if I can do it. I’ve learned I get a lot more out of this than just copying what someone else has already written.

Somewhere along my lifeline I fell into Software Testing. I jumped in with no real knowledge of the craft but I had that insatiable curiosity within me. Give me an application. Give an idea of what it’s supposed to do and I’ll jump on it. Not only will I prove it does what it’s supposed to do but I’ll click the on/off switch repeatedly to see if I can make it do something unintended.