Rally Enterprise Edition for Test Case Management

I am looking into using Rally Enterprise Edition for Test Case Management since Rally is a popular SCRUM Life Cycle Management tool and so many companies use it. I know I’ll get the chance to continue to use it in the future. So, like I did in the late 90’s with HP’s buggy Quality Center, I will embrace it and work through the kinks with product support, as the product matures. I will utilize the community bulletin boards and vote up issues at the Rally Ideas site.  It took quite a while for MQC (old Mercury Test Director) to mature, it will take Rally a while to mature, or perhaps it won’t ever be a contender to QC and never needs to be. Time will tell.

I am finding on my current project, User Interface test automaton, which is a key value in agile development, is progressing nicely, but it is still in the early stages. The automation needs to be supplemented by standard manual functional test cases for each iteration. These in turn, will be marked and gathered for regressions and ultimately some of which will be automated.

I am looking into using Rally for test management as I would like to have visibility for the entire team to know testing status beyond what is illustrated in the standard burn down chart.  More than that, I would like to easily know what is left to execute without scouring the Google spreadsheets and multiple tabs within, to see which high priority test cases still need to pass test execution.

Here is what I found regarding utilizing Rally for test case management:

The good:

  1. You can create one or more test cases at the user story level to test all acceptance criteria within that user story.
  2. Test cases are created, updated, executed at the user story level.
  3. The status of test case execution per user story is shown at the test run level, under the user story in a simple graph, % passed and % covered (which means executed).
  4. You can pass/fail a test case, create a defect and edit the test case from the test run screen.
  5. You can filter by test cases to find which test cases at the use case level still need executing.
  6. You are able to print a detailed report from the results section at the use case level.
  7. You are able and save it as pdf with a tool such as CutePDF for your test execution reporting.

The not so good:

  1. You cannot perform step by step test execution, even though you can enter the test steps in one by one. This gave me the illusion that you could perform step by step execution and fail a test case at a particular step. This may not be such a big deal, in the long run, I like to keep it simple too. The tester looks at the main Rally screen while executing the tests against the application, and enters test execution information in the Test Case Results pop up window. The tester enters Pass/Fail information as well as Build, Verdict, Date (automatically populated) and Notes in the Test Case Result form which pops up when clicking the Pass/Fail icon for a test case.  It is in the Notes section that the user can document what step the test case failed on.
  2. You cannot roll up all your execution results into one report across user stories for an Iteration. You have to look at each user story and individually run a report. If your team typically has 6-8 user stories I suppose that isn’t too bad.
  3. You cannot move the test cases from the Use Case level into a regression test format, for instance a Regression test suite directory of folders. They have to be at the use case level. You can copy a test case into the same use case folder and edit, but from I can tell not across use cases.
  4. You cannot share a dashboard in Rally out of the box.  I yearn to have a standard dashboard the team looks at, similar to how you can create a query or dashboard in Jira and share it? What does this have to do with test case management, well I would like to share a dashboard simply so everyone has an idea where we are status wise.

So, to summarize, you can perform test case management out of Rally’s Enterprise edition. But is it any easier than using spreadsheets?   I don’t think so.  It is clunky and will take time to save off your reports from the individual user stories. You will not get good status reporting, except at the User Story level and you cannot save test cases to a regression suite, you will need the Unlimited edition for this. That is another story, I will save for another day to tell. I will also save for another day, the discussion of Rally’s API and if it’s worth it to customize Rally to meet your needs at the Enterprise edition.

Posted in Uncategorized | Leave a comment

What…..move from Spreadsheets to a Test Management System?

Well it may sound odd to even ask this question in the 21st Century, but in my experience as of late, is that there are still companies out there that use spreadsheets to track their legacy test cases. Also, for current development test cases and test case execution.

Now that we have Google docs, and Google spreadsheets, collaboration among the test team is free and simple. Still, there are several compelling reasons to move from manage by spreadsheet to a test management system:

  1. Ease of performing test execution
  2. History of test execution
  3. Test execution supporting documentation, results
  4. Ease of updating test cases and reuse of test cases
  5. Version control
  6. Mapping to test cases to user stories in tool
  7. Dashboards creation for testing visibility to team
  8. Able to import the spreadsheets into the test management tool, probably after some manipulation
  9. Allows visibility to the team of where you are, in planning and test execution
  10. Saves time on reporting, or gasp, reports for you if you don’t currently report

Sure there are costs to consider, but if you are going to do it eventually, why not start now? In the long run it will save you money and help you develop software a little easier.

Sometimes on projects it is difficult for managers and other team members to see what your doing, and where you are. How are we doing? Where are we going? Are we on track? I find that reporting in the SCRUM meeting manually is more difficult for the tester. As well, more difficult for the team to digest and understand than having a dashboard the team can easily find and gain the valuable information at a glance. Dashboard visibility, which comes from an integrated Test Case Management tool adds to the testers standard SCRUM update and visibility to the team’s progress immensely.

Posted in Uncategorized | Leave a comment

Just Do It

Just put One foot in front of the other, is what I say. Don’t be stifled.   ‘Just do it’  like Nike says, which I have to admit was my mantra before they started the campaign in 1988 (as far as I can tell).   Just begin.   Don’t focus on the job at hand, just start and good things will flow forth. As Lao-tzu said, ‘A journey of a thousand miles begins with a single step’.   I have a voice, and I want to give my blog my own voice, I want to be authentic and keep it real.  I write because I want to share, lift up others and inspire, as others have inspired me along the way….. ‘On the shoulders of giants   ‘.   If I aspire to being a writer I want to let my own voice be heard. I like my voice. I have a lot to say. This is the start of my journey of finding my voice and letting it be heard in the software testing community.

Posted in Uncategorized | Leave a comment