I'm trying to decide on a tool for test management, defect tracking and potentially integrate with an automation tool. The automation tool has not been chosen yet.
I work with a JAVA based desktop application with webservices connecting to a SQL server back end.
I would like to hear suggestions for choosing an automation tool and test management tool that can integrate well with the automation tool. What should we be considering? Why?
Given your predominantly Java environment you might consider a number of options - from fully integrated ALM stack to the "best-of-breed" combinations of tools (both free/open source and vendor supported). A number of considerations comes into play: compatibility with your current toolset and environment, whether you'd like to run it on-premises or as SaaS or a hybrid solution, scalability solution, costs (either upfront with an integrated stack or spent on configuration/integration/support with the best-of-the-breed components). You need to come up with a list of features (quality attributes) that are important to your team and evaluate each potential solution against it.
For example: extensibility, ability to run on/off premises, tight integration with IDE and current tools, scalability, security, TCO, traceability (requirement>>code>>test>>defect), reporting/dashboarding, management style (agile/waterfall, DevOps, Kanban etc) - to mention but a few.
Let's consider Atlassian (www.atlassian.com ) stack that somebody mentioned above. It has Issue tracking, Release management, version control, continuous integration, code coverage for Java and more. Notably, it lacks full blown Requirements Management (yes, I know - you can use Issue as a requirement but it is an afterthought) and test automation. You can add - depending on your third-party/open source components to fill the gaps (e.g Selenium for testing or, say, VersionOne for SaaS requirement management). But you always need to be mindful of the additional costs/complexity and perform a trade-off analysis for different architectures.
CollabNet TeamForge is a bit more rounded in its offering, but you might need to supplement test automation and need to integrate with third party CI (Jenkins/Hudson etc), code analysis tools and so on. www.collab.net
Both Atlassian and CollabNet offer on-premises and SaaS deployments.
You may choose to assemble your own stacks such as Rally, Serena, ThoughtWorks suites - even Microsoft Team Foundation Server (though it is not particularly suited for Java development)..
Or assembleopen source tools such as Mantiss or Bugzilla for defect tracking, OpenProject for project management activities, Jenkins for CI, Git/Subversion for version control, EMMA for code analysis, Selenium for testing... you get the idea :)
With Integrated ALM (Application Lifecycle Management) you might be looking into HP ALM 12 and IBM Jazz Platform. Both are capable, both are pricey (the last time I got quotes, IBM's was about 3x the HP quote, and both in 7 figures). But you have a turn-key solution that can be deployed both on premises and as SaaS. The test automation components (both functional and performance) are second to none (at least, in HP case - I do not have much experience with IBM's). Again, the integrated solutions will cost you upfront but might save you money on integration and code quality efforts.
For Defect tracking / management HPQC / JIRA is good tool. For Automation, vCloud Automotion (vCAC) is better if you have Cloud environment.
For Bug tracking JIRA would be a good fit (www.atlassian.com), and for test management she/he
can use (www.testlodge.com).
I propose Altassian-JIRA as a defect tracking tool and can be integrated with Jenkins & Zephyr.
You can use Zephyr for test management which can be integrated with JIRA for using it effectively. This plug-in was recently introduced and haven't heard anything negative on it.
Providing few links as below:
www.atlassian.com
www.youtube.com
JIRA is tool that makes an appearance anytime there is a discussion on any-management process- for all the right reasons. JIRA has 2 add-ons that support the test management process.
a) Zephyr: All the aspects that you would expect of a typical tool of this type are supported. You can create tests/test suites/test cycles/bugs/reports and so on. You could have an additional add-on, ZAPI for automation integration. Along with the initial JIRA license you would have to pay for Zephyr to use it. ($10 for 10 users a month). There is a free trail available too. Check out information about it at:
marketplace.atlassian.com
Choose HP's ALM for test management and Unified Functional Test for functional testing automation and LoadRunner for performance.
These are the best on the market and have native integration between them.
ALM and UFT combination would certainly help in your case, later on if you would need any performance testing then even HP perf center from HP suite can be leveraged. I am not saying this is the only option but it all really depends on what exactly are your needs and constraints (like budget etc. ). There are surely some open source tools in market which you can explore. But if you are aiming at long term tools strategy for your organization, then it would make sense to go with licensed and packaged solutions like HP (or other similar). Before your zero down on something it would be good to check which other applications your organization may need to test in near future and will your testing tool(automation) support your requirements.
As far as test management is concerned I had no complaints with ALM, and UFT is compatible and solves your current problem.
Hello,
Before going into application lifecycle management, few predominant question pop up. 1. Are you looking for open source or a licensed software? And you did mention that it is Java based desktop application with web service calls. If licensed software, then I would give HP UFT a try. It has an option to test web services as well. With UFT, you can combine HP ALM to make it an effective solution. With open source tools, you can mix and match most of the tools. Though a robust framework would be needed to support the entire testing process. Hope this helps.
Bugzilla is a good open source defect management
tool.
In case you are looking something keeping agile in mind Jira would be a
fantastic choice,and is also used in the industry widely now.
I am not expert at this but know of a couple of tools to use. One is Quality Center from HP for tracking requirements and test tracking which can connect to QTP which is there automation too. You can also get TFS from Microsoft which is a combination tool for testing and automation. Both of these are not the cheapest software solution to procure.
Douglas Johnson
HP ALM a proven Test Management tool. IBM Rational Quality Manager also can help. I see some suggestions on tools such as Zephyr and Tricentis as well. I've not used these tools, hence cant comment on them. It also depends on the budget being invested for a test management tool by your company, the size of your testing team and test suites (keeping future testing needs in mind) and the criticality of maintaining automated test suites and reporting. Two important things you need to check while selecting a test management tool. (1) If you can integrate your automation scripts so that execution can be triggered directly from the test management tool. (2) How good is the reporting feature and if reports would be generated automatically at the end of the execution. Also looks for all the basic features of a Defect management tool.
I am aware of some options (there are probably more).
The obvious one is HP Quality Centre/ALM, this product has the lion's share
of the market, is mature and used by most larger corporates. It integrates
with QTP. It is also expensive.
An alternative I have seen but not used is the Micro Focus/Borland product.
Silk Central which integrates with Silk Test. I cannot comment on how good
it is.
There will also be an IBM/Rational product as well, but these tools don't
have a very good reputation.
There are probably also some open source tools as well (Selenium is popular
for automation, but I don't know of a matching ALM tool).
What I suggest is implement IBM Rational Test manager tool for Test Management and IBM rational functional Tester as the automation tool. Since the product is in Java .IBM Rational tool will be able to integrate better with the Java source code. Another tool which I can suggest is Test complete which is a automation tool from big bear software. If you have good scripting skills then you can try selenium JIRA Test Management tool.
You can look at HP Application Lifecycle Management tool which ties in with QTP for automation and HP Performance Center for performance and Quality Center for defect tracking/ test requirements. The downside is that Jira is very expensive for the software.
Or you could go some Open source and look at Altassian Jira for issue/project tracking, Selina and Jmeter for performance. The downside is that there would be a lot more development work required
The problem is when you attempt to use the best of breed tools for testing, deployment automation, system configuration management, issue & defect tracking, is that you end up with a group of siloed point tools. You could consider an automation tool that will integrated and orchestrate with these "best of breed" tool, so you can have audit, visibility and traceability right through the entire life-cycle. Also by its nature you can define processes and metrics that do not entire simply end at the siloed tool but can give you that holistic view. Serena Deployment Automation can help you in that respect www.serena.com
I've worked with both IBM's Collaborative Lifecycle Management product and Inflectra's SpiraTeam.
I implemented the IBM suite at a large gas/electric utility company and SpiraTeam at a much smaller utility. In general, both products offered the same features and functions. I found IBM's product to be a bit more complex to use and administer than SpiraTeam. Both utilities had the same requirements. They needed a requirements repository, a testing repository, and an SDLC task management system. All integrated.
At the large utility, I wasn't involved in the decision to buy the software. But I alone made the buying decision at the smaller utility company. I had no budget restrictions at the smaller utility because we are in high-growth mode. My job was to buy the best software with the understanding that it must be able to scale up.
I bought SpiraTeam. It's been installed for a few months now. I have no regrets
www.inflectra.com
Inflectra is easy to do business with. Pricing is transparent. I was up and running within a few hours.
I suggest you do a proof-of-concept project. You can subscribe to SpiraTeam on a month-to-month basis. You have nothing to lose. It's easy to import or export to-from SpiraTeam so any work you do can be re-used. It integrates with many different requirements and testing products.
One of the leading tools for enterprise test automation is Tricentis Tosca (www.tricentis.com).
Tosca is able to provide test case design, test data management, and service virtualization, things you will appreciate during every test (automation) project. And yes it has an integration with Jira.
Hi
We have been using New Relic.. which has a Selenium testing portion.
While it doesn’t do everything, its really good a monitoring the application on both the server and the browser (including mobile apps). It ties them together and gives you a aggregate sample of the state of your application/servers over time. The graphs can be drilled down, and we have found it to be useful if you add your own variables to be tracked. They have a good sql type dashboard application, that allows you to set up various end user (read management vs client, vs programmer) dashboards, and tune it to your needs.
For issues we use Osticket which allows us to work through email with a large user community which is great at research for medicine, not so great at IT type stuff.
Technically this is application performance management, not so much development management (from the perspective of deliverables and deadlines).
If you have any questions, or want to know more, or wish to discuss another option, please feel free to write, I am always glad to help…