Try our new research platform with insights from 80,000+ expert users
PeerSpot user
Quality Assurance Practice Leader with 51-200 employees
Vendor
It reduced the regression timeline for a complex AUT, but there's an issue object mapping related to the need to modularize the test script code.

What is most valuable?

TestComplete is used for testing Windows desktop products; specifically the Embarcadero VCL interfaces created by Delphi/C++. All debug flags must be enabled during compilation that generates an output file required by TestComplete to identify and interact with the UI objects in the application.

An important feature of TestComplete is the ability to modularize testing. A lot of effort has been put into breaking the test script into reusable functions/methods that can be called from any test. A number of function libraries were created. This enabled reuse of code and kept the projects and project suites small. This is important because the size of these test artifacts have an impact on project loading time.

How has it helped my organization?

Test automation with TestComplete significantly (estimated 80%) reduced the regression timeline for a complex AUT with a large number of test cases. Automation with TestComplete has significantly shortened the feedback loop and the timeline to get a release production ready. A secondary benefit is that manual testers have begun thinking more technically about writing tests cases.

What needs improvement?

There are two major areas for improvement:

  1. Version control integration embeds information in the project suite, and project files that include a direct reference to the location of the project or project suite in Team Foundation Server. When branching a set of scripts for the next version of the AUT, TFS gets confused about where the file should be mapped into source control. A workaround is to replace the hardcoded paths with relative paths.
  2. There was also an issue object mapping related to the need to modularize the test script code. With the implementation of other automation test suites such at HP UFT, it is possible to have one object map for each function library. TestComplete has a limitation of only one object map per project. In order to modularize the code it was necessary to have a single shared object map used across multiple projects and project suites.

For how long have I used the solution?

I have used TestComplete for 20 months. It was chosen as an incumbent toolset could not interact with the product to be tested.

Buyer's Guide
SmartBear TestComplete
October 2024
Learn what your peers think about SmartBear TestComplete. Get advice and tips from experienced pros sharing their opinions. Updated: October 2024.
814,763 professionals have used our research since 2012.

What was my experience with deployment of the solution?

Integration with third-party products; specifically Microsoft Team Foundation Server and HP ALM could not be overcome. A custom integration to HP ALM was written using the HP ALM OTA API.

What do I think about the stability of the solution?

TestComplete has been a stable product.

How are customer service and support?

Customer Service can be slow to respond to electronic forms of communication and they do not have a way for a customer to speak directly to customer support. You create a ticket online, and request a phone call.

The team seems to be very knowledgeable when communication is established.

Which solution did I use previously and why did I switch?

TestComplete was added to the toolset. HP UFT is used for automated tests for other products.

How was the initial setup?

Setup is straightforward unless third-party tool integration is required. Integration with Microsoft Team Foundation Server is a little complex for initial configuration. Once it is understood the process is repeatable.

What about the implementation team?

Implemented in-house. Implementation is not difficult to implement or write tests especially if you have experience with other test automation tools.

What was our ROI?

We have not done a ROI calculation. However, automated testing with TestComplete has cut regression test time by months.

Which other solutions did I evaluate?

HP UFT was tried but the object recognition did not work with the implementation technology.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
PeerSpot user
Staff Test Architect at a tech services company with 1,001-5,000 employees
Real User
Multiple languages available, good technical support, but integration could improve
Pros and Cons
  • "The most valuable features of the SmartBear TestComplete are self-healing, they reduce the maintenance required. The different languages SmartBear TestComplete supports are good because some of our libraries are written in Python, JavaScript, and C#. It's very easy to put them all under one project and use them. The are other features that SmartBear TestComplete has but the competition widely has them as well."
  • "In SmartBear TestComplete the integration with Jenkins could be easier. Additionally, some of the controls could have better customization options. For example, if a grid is used and it contains multiple controls within it, it can be a checkbox, radio button, or any kind of control, the way the Object Spy is operating currently there is a lot of room for improvement."

What is most valuable?

The most valuable features of the SmartBear TestComplete are self-healing, they reduce the maintenance required. The different languages SmartBear TestComplete supports are good because some of our libraries are written in Python, JavaScript, and C#. It's very easy to put them all under one project and use them. The are other features that SmartBear TestComplete has but the competition widely has them as well.

What needs improvement?

In SmartBear TestComplete the integration with Jenkins could be easier. Additionally, some of the controls could have better customization options. For example, if a grid is used and it contains multiple controls within it, it can be a checkbox, radio button, or any kind of control, the way the Object Spy is operating currently there is a lot of room for improvement.

There are a lot of improvements that are required on the Object Spy, such as how we find the locators. That is a feature which I would love to see. 

It is currently very tightly coupled to the SmartBear ID. There should be a way that they could support the integration with external libraries and make it flexible to integrate with open-source frameworks. This would be a good option to add.

For how long have I used the solution?

I have been using SmartBear TestComplete for approximately eight years.

What do I think about the stability of the solution?

SmartBear TestComplete is a stable solution.

What do I think about the scalability of the solution?

When it comes to customization controls the scalability of SmartBear TestComplete is limited. However, the assets are scalable. It's easy to scale across the solutions within the same limited functionality.

How are customer service and support?

The technical support from SmartBear TestComplete is good.

How was the initial setup?

The initial setup of SmartBear TestComplete is easy. The process is almost instant, it is very easy.

What's my experience with pricing, setup cost, and licensing?

The price of SmartBear TestComplete could be less. The main challenge is when it comes to node-locked. They should use a subscription model, such as a monthly-based subscription or, a quarterly-based subscription. Their floating license is very expensive, and this high price should be reduced or provide, at a minimum, a subscription model.

What other advice do I have?

When it comes to automation, the fundamental aspects are the stabilization factor, how easy it is to automate, how much is the coverage, and how much are you able to cover with respect to an application. There will be always areas that cannot be automated. The coverage, stabilization, and ease to automate are the areas that have to be improved. The better the pricing and improved pricing models, the more it will attract more customers. 

I rate SmartBear TestComplete a seven out of ten.

Which deployment model are you using for this solution?

On-premises
Disclosure: I am a real user, and this review is based on my own experience and opinions.
PeerSpot user
Buyer's Guide
SmartBear TestComplete
October 2024
Learn what your peers think about SmartBear TestComplete. Get advice and tips from experienced pros sharing their opinions. Updated: October 2024.
814,763 professionals have used our research since 2012.
Senior Automation Test Analyst with 10,001+ employees
Real User
Allows us to test both desktop and web applications
Pros and Cons
  • "It allows us to test both desktop and web applications."
  • "It is very hard to read the test log generated by TestComplete Executor if the log file is very big. TestComplete Executor is a small tool for just running the TestComplete test framework (not for developing)."

What is most valuable?

It allows us to test both desktop and web applications. This is a very important feature for system integration tests, as we develop desktop applications, web applications, and a Web API using the same database.

What needs improvement?

It is very hard to read the test log generated by TestComplete Executor if the log file is very big. TestComplete Executor is a small tool for just running the TestComplete test framework (not for developing).

In fact, this is an issue regarding how to read the test log in MHT format generated by TestComplete.

We know a test log is always generated automatically by test tool after executing a test. TestComplete provides a standard IDE (Integrated Development Environment) interface for user to review the test log. We don’t have issue reviewing the test log within TestComplete regardless how big the test log is (Our test log record the test running for more than two days without stopping. The test log extension name is .tcLogs and the size of total test log is very big over 2.5 GB).

TestComplete also provides a function for the user to export the log into MHT format file via manually selecting each test log after complete a test or dynamically exporting the test log during test executing via command in test script. The MHT log file can be opened by IE. However, if the MHT file is a little big, such as >200M, the user could experience obvious performance issues. It takes a very long time for IE to open this file because IE loads all the contents into one page which could finally eat up all your system memory. I tried to find another tool to open the big MHT log file, but no luck. Even executing a small test (running for one to two hours), the MHT log file size can easily reach to 200M, if a screenshot is included, for example. Normally, each one round of our regression test runs for 30-48 hours. It is impossible for us to open the big MHT log file as the system runs out of memory and an error appears definitely after hours of waiting to just open this file by IE.

TestExecute is a small tool to just execute the test developed by TestComplete. But TestExecute only generates MHT log file rather than TCLOGS file. Obviously, we cannot open the MHT log file if it is very big. So we gave up using TestExectue and export MHT log file function in TestCompelete at all.

We have to buy six individual full license of TestComplete to run and review the test.

For how long have I used the solution?

We have used this for nine years.

What do I think about the stability of the solution?

Overall, TestComplete is quite stable. But the only issue affecting me is if TestComplete is terminated unexpectedly while test is still running (such as, terminating TestComplete without stopping the test first, window restarting, or shutting down unexpectedly), user could lose all the logs.

What do I think about the scalability of the solution?

There were no issues with scalability, but it may have a performance issue with expansion.

How is customer service and technical support?

Technical support is excellent.

How was the initial setup?

The initial setup was straightforward.

What's my experience with pricing, setup cost, and licensing?

The product is becoming more and more expensive. There are two types of licenses: locked and float. Locked license save more, but can only be used in a physical machine. Float licenses can be run in virtual machines.

Which other solutions did I evaluate?

QTP, was very expensive when compared to TestComplete, many years ago.

What other advice do I have?

This tool is very easy to use and very powerful. It has a short test development circle and good technical support. There is an expensive license, but it could save more, as testers may spend more time achieving a certain test goal if using an Open Source tool. You need to consider the testers' script experience and the company’s budget to choose the right tool.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
PeerSpot user
PeerSpot user
QA Automation Manager at a comms service provider with 501-1,000 employees
Vendor
We use it to do the automation test on our product’s web UI. In total, over 500 test cases are written and the automation coverage is over 80%.

What is most valuable?

The name mapping function to manage the objects across different web pages and the support for different web browsers are the most valuable features for us.

How has it helped my organization?

We use TestComplete to do the automation test on our product’s web UI. In total, over 500 test cases are written and the automation coverage is over 80%. This helps to reduce manual effort dramatically, and by using the same script, we can run the test on both IE and Google Chrome.

What needs improvement?

The scripts are saved as binary files on disk. This makes it troublesome to perform version control and merge it with the work that is done by other team members. I am not sure what the reason is that SmartBear chose to save the files as binary, but if they can change to plain text for the scripts in the future, that would be helpful.

For how long have I used the solution?

I've used it for one year.

What do I think about the stability of the solution?

Some of the objects/parent objects are dynamically created in the web page. During the test, TestComplete will have problem to identify these kind of objects. This will cause a lot of false rejection and block the following test cases. Thus it requires some rework when running all the test cases as a batch.

How are customer service and technical support?

I have not used any customer service/tech support yet. I cannot provide more comments on that.

Which solution did I use previously and why did I switch?

TestComplete is the first tool that I use for automation on UI and I have not used any other tools yet.

How was the initial setup?

It is pretty straightforward. Just by following the instructions given in the document and do the corresponding configuration on the web browsers that you want to test. However, we also encounter some problems such as the Chrome blocking users from proceeding because of an unsafe SSL connection, but we found a work around eventually.

What about the implementation team?

We implement the automation by ourselves, not through any other vendor team. My suggestion is that you need to decide how to divide the automation work into corresponding components carefully. Otherwise, it will be hard for you to share the common scripts with other team members. Also, the code architecture design is very important as well.

What was our ROI?

I am not responsible with the licensing quotation/renewal, but the benefit that TestComplete brings to us is, apparently, acknowledged.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
PeerSpot user
it_user341001 - PeerSpot reviewer
Junior ASP/.Net Developer at a construction company with 501-1,000 employees
Vendor
Its environment exposes a significant portion of its functionality to the command line, where execution details can be left to .bat files or Windows Scripts.

Valuable Features

Object-based recording, and data-driven testing.

Separation of data into Excel files made tests modifiable by QA personnel with limited development experience, and object-based recording kept maintenance to a minimum.

Improvements to My Organization

TestComplete's environment exposes a significant portion of its functionality to the command line, where execution details can be left to .bat files or Windows Scripts.

This allowed us to schedule execution of lengthy tests for nighttime and non-core hours, and to synchronize tests with Jenkins build releases.

This freed up QA assets to perform more specialized testing and reduced redundancy.

Room for Improvement

Native test result reporting does not provide overview reporting methods for tests that span multiple project suites. Features that allow for flagging a test as dependent on the result of another in multi-project hierarchies while maintaining name-mapping segregation would be much esteemed.

Use of Solution

I've used it for approximately eight months.

Customer Service and Technical Support

Customer Service:

8/10 - SmartBear representatives are surprisingly responsive, and they go out of their way to offer assistance.

Technical Support:

8/10 - SmartBear representatives are surprisingly responsive, and they go out of their way to offer assistance.

Initial Setup

The initial set-up was exceedingly simple. There is a silent mode option available during set-up, which is very convenient when deploying to multiple machines or remotely.

Implementation Team

We implemented it through an in-house team.

ROI

We did not maximize our ROI until we put somebody full-time on our TestComplete endeavours. The tool does have a learning curve, and it wasn't until we had an in-house expert on it that we began to see the benefits of automated testing over traditional QA roles.

Pricing, Setup Cost and Licensing

The licensing options for TestComplete both running a licensing server. This prevents users from running more sessions simultaneously than purchased keys. This can prove problematic if you want multiple developers writing or running tests at once, and prevents you from using your key while a distributed test is running.

For pricing, carefully consider how many machines you want running the software, rather than the number of developers.

Other Solutions Considered

We also evaluated another SmartBear product called SoapUI. The change to TestComplete occurred because we changed our target from web applications to desktop.

Other Advice

My advice in regards to implementation would be to choose carefully which tests to automate, specifically focusing on lengthy procedures, tasks that require looping, or places where you want to test against multiple data sets.

Additionally, I found it beneficial to prefix my keyword tests with a character and number to provide logical ordering instead of alphabetic.

I also found it beneficial to record "undo" steps with each keyword test; this allows each test to be more stand-alone and prevents your test from being dependent on the state the previous test left the application in.

Finally, I would suggesting limiting the number of test applications per test suite to prevent bloated name-mapping schemes.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
PeerSpot user
it_user336978 - PeerSpot reviewer
it_user336978Multimedia Solutions Verification Engineer at a comms service provider with 1,001-5,000 employees
Vendor

Test Complete provides detailed reports since every profit software should do, since people pay for its reporting functionality also. On the other hand, some big and expensive products stil has the reporting gaps, for example there are third party reporting tools for HP QC.

See all 3 comments
it_user294939 - PeerSpot reviewer
Test Engineering Team Leader at a consultancy with 501-1,000 employees
Consultant
After installation, I was able to immediately begin coding regression test for available functionality.

What is most valuable?

  • Initially support for flex/flash technologies, but since that project ended my primary use is Functional Automation against a web front-end built on AngularJS.
  • I use VBScript for a majority of my tests; it is one of many language choices available when writing scripts.
  • Out of the box support for desktop, mobile and web.

How has it helped my organization?

I am the only test engineer on the project, so time is limited in terms of automation development. Using TestComplete I was able to immediately (after installation) begin coding regression tests for available functionality.

My current suite of tests numbers in the hundreds, and each test involves several UI elements. Depending on how your organization defines a test you may call this one test or many. An example to quantify this, is as follows - Automated Build Viability (smoke test) execution:

  • Number of tests = 75
  • Automated Time to execute = 15 minutes
  • Manual Time to execute = 360 minutes (approx.)
  • Automation time saved = 345 minutes (5+ hours)
  • 2300% improvement

I have substantially reduced the amount of time it takes for regression and build viability activities.

What needs improvement?

I pay for support and maintenance; having used the “support” functions through online forums, I can say there is room for improvement. I would like to have access directly to TestComplete developers as opposed to “users” who have some level of expertise with the platform.

For how long have I used the solution?

I've used it for two years.

What was my experience with deployment of the solution?

No issues encountered.

What do I think about the stability of the solution?

No issues encountered.

What do I think about the scalability of the solution?

No issues encountered.

How are customer service and technical support?

5/10, there are knowledgeable users in the forums, but I would like to go straight to the TestComplete staff with solution questions. That said, I have had very little need for product or development specific support.

Which solution did I use previously and why did I switch?

Other tools were evaluated which supported Flex/Flash web apps, but we chose TestComplete for its rich feature set as well as Smartbear’s available suite of tools outside of Automated Testing.

How was the initial setup?

It was very straightforward! You just need to install and begin, no special plugins or out of the ordinary system requirements were necessary.

What about the implementation team?

We implemented it in-house. As with any automation platform, carefully planning the intended usage, understanding the technology under test and determining which features will be used are critical to success.

What was our ROI?

Given the cost was minimal, the best representation of ROI from my perspective is time saved using automated tests vs. manually execution.

My current suite of tests numbers in the hundreds, and each test involves several UI elements. Depending on how your organization defines a test you may call this one test or many. An example to quantify this, is as follows - Automated Build Viability (smoke test) execution:

  • Number of tests = 75
  • Automated Time to execute = 15 minutes
  • Manual Time to execute = 360 minutes (approx.)
  • Automation time saved = 345 minutes (5+ hours)
  • 2300% improvement

What's my experience with pricing, setup cost, and licensing?

Our recent purchase was a single node locked license for one user which was very inexpensive. That fit our project needs and the automation strategy. The solution resides on inexpensive laptop for development and execution.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
PeerSpot user
Test Automation Specialist at a tech vendor with 201-500 employees
Real User
The tool has the ability for a non-developer to develop intelligent, robust, data-driven tests

What is most valuable?

The most valuable feature is the ability for a non-developer to develop intelligent, robust, data-driven tests.

How has it helped my organization?

TestComplete has greatly improved our organization functions by allowing us to develop, manage, and execute many forms of testing in one central repository. Being able to develop, manage, and execute JUnit, NUnit, PHPUnit, PyUnit, Selenium, functional, and nonfunctional tests has given our company great insight as to the health of our testing efforts.

For how long have I used the solution?

I have used this solution for four years.

What do I think about the stability of the solution?

No issues.

What do I think about the scalability of the solution?

No issues.

How are customer service and technical support?

The level of technical support leaves a lot to be desired. The only way to get support is via email, therefore problems that could take a matter of minutes to solve by phone or web conference can sometimes take several days.

Which solution did I use previously and why did I switch?

We previously used IBM Rational Functional Tester, Segue/Borland/Micro Focus SilkTest, HPE Mercury Quick Test Professional, and IBM Rational Robot.

We switched because of the ability for a non-developer to develop intelligent, robust, data-driven tests.

How was the initial setup?

The initial setup was very straightforward and took very little time.

What's my experience with pricing, setup cost, and licensing?

No advice. I did not deal with the licensing and pricing.

Which other solutions did I evaluate?

I did compare the tool to IBM Rational Functional Tester, Segue/Borland/Micro Focus SilkTest, and HPE Mercury Quick Test Professional.

What other advice do I have?

TestComplete is a great product. It is a perfect fit for an organization that has a hard time finding testers with a development background and want to centralize the development, management, and execution of their tests.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
PeerSpot user
PeerSpot user
Mobile QA Tester at a comms service provider with 1,001-5,000 employees
Vendor
It's enabled us to automate a lot of our tests for Android devices.

Valuable Features

I found the Keyword Test feature very valuable. Even with my limited programming knowledge, I was able to create automation tests using the Keyword Test feature. Also, the ability to create image based tests was extremely helpful, especially when automating tests for mobile devices.

Improvements to My Organization

TestComplete has enabled us to automate a lot of our tests for Android devices.

Use of Solution

I've used it for three months.

Customer Service and Technical Support

I would rate it 10/10. Every time I had a question or issue I couldn’t figure out, they were right there to help.

Initial Setup

Set-up was pretty straightforward.

Implementation Team

We implemented it through an in-house team.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
PeerSpot user
Buyer's Guide
Download our free SmartBear TestComplete Report and get advice and tips from experienced pros sharing their opinions.
Updated: October 2024
Buyer's Guide
Download our free SmartBear TestComplete Report and get advice and tips from experienced pros sharing their opinions.