I am looking into using Rally Enterprise Edition for Test Case Management since Rally is a popular SCRUM Life Cycle Management tool and so many companies use it. I know I’ll get the chance to continue to use it in the future. So, like I did in the late 90’s with HP’s buggy Quality Center, I will embrace it and work through the kinks with product support, as the product matures. I will utilize the community bulletin boards and vote up issues at the Rally Ideas site. It took quite a while for MQC (old Mercury Test Director) to mature, it will take Rally a while to mature, or perhaps it won’t ever be a contender to QC and never needs to be. Time will tell.
I am finding on my current project, User Interface test automaton, which is a key value in agile development, is progressing nicely, but it is still in the early stages. The automation needs to be supplemented by standard manual functional test cases for each iteration. These in turn, will be marked and gathered for regressions and ultimately some of which will be automated.
I am looking into using Rally for test management as I would like to have visibility for the entire team to know testing status beyond what is illustrated in the standard burn down chart. More than that, I would like to easily know what is left to execute without scouring the Google spreadsheets and multiple tabs within, to see which high priority test cases still need to pass test execution.
Here is what I found regarding utilizing Rally for test case management:
- You can create one or more test cases at the user story level to test all acceptance criteria within that user story.
- Test cases are created, updated, executed at the user story level.
- The status of test case execution per user story is shown at the test run level, under the user story in a simple graph, % passed and % covered (which means executed).
- You can pass/fail a test case, create a defect and edit the test case from the test run screen.
- You can filter by test cases to find which test cases at the use case level still need executing.
- You are able to print a detailed report from the results section at the use case level.
- You are able and save it as pdf with a tool such as CutePDF for your test execution reporting.
The not so good:
- You cannot perform step by step test execution, even though you can enter the test steps in one by one. This gave me the illusion that you could perform step by step execution and fail a test case at a particular step. This may not be such a big deal, in the long run, I like to keep it simple too. The tester looks at the main Rally screen while executing the tests against the application, and enters test execution information in the Test Case Results pop up window. The tester enters Pass/Fail information as well as Build, Verdict, Date (automatically populated) and Notes in the Test Case Result form which pops up when clicking the Pass/Fail icon for a test case. It is in the Notes section that the user can document what step the test case failed on.
- You cannot roll up all your execution results into one report across user stories for an Iteration. You have to look at each user story and individually run a report. If your team typically has 6-8 user stories I suppose that isn’t too bad.
- You cannot move the test cases from the Use Case level into a regression test format, for instance a Regression test suite directory of folders. They have to be at the use case level. You can copy a test case into the same use case folder and edit, but from I can tell not across use cases.
- You cannot share a dashboard in Rally out of the box. I yearn to have a standard dashboard the team looks at, similar to how you can create a query or dashboard in Jira and share it? What does this have to do with test case management, well I would like to share a dashboard simply so everyone has an idea where we are status wise.
So, to summarize, you can perform test case management out of Rally’s Enterprise edition. But is it any easier than using spreadsheets? I don’t think so. It is clunky and will take time to save off your reports from the individual user stories. You will not get good status reporting, except at the User Story level and you cannot save test cases to a regression suite, you will need the Unlimited edition for this. That is another story, I will save for another day to tell. I will also save for another day, the discussion of Rally’s API and if it’s worth it to customize Rally to meet your needs at the Enterprise edition.