On this podcast we have another company spotlight for you on Testplant who specialise in test automation tools. Anthony Edwards, CTO of Testplant talks us through their products including the new ‘Eggplant’ product to help mobile app developers automatically test, improve and assure their applications before putting them to market.
For more related podcasts click here.
To subscribe to receive new podcast episodes for free click here.
Firstly Anthony set the scene for us with some background on Testplant?
We started off 4 years ago looking at test automation. The objectives that we have had on test automation over the last 10 to 15 years include the idea of repeatable tests with low incremental effort each time. Better quality and getting products out faster. That was a great goal but we didn’t see a lot of people achieving that goal really well. We had people frustrated with the existing set of tools which were largely object code based tools and we decided that there was a space there for innovation in a new approach on testing, and we came up with Eggplant.
Eggplant is a test automation tool that is based on image search and image capture. Whereas traditional tests would go in on the code framework level where they interpret the code and they look for buttons or test fields at the dot net or the piper level.
Eggplant interacts with the system exactly how the user will, it looks at the screen and it identifies objects through recognition and search techniques. So to clarify it is not a coordinate tool, it doesn’t go through a pre-determined coordinate to look for your okay button, what it will do is it knows what a button looks like on your screen and it will search for a button that looks like the okay button and then it will interact with that so this approach has a lot of benefits.
Firstly it is a much more intuitive way for testing rather than have a test case that describes the user story and then having to translate that into object properties. Testplant allows you to execute a test case to fit exactly as the user story would. It is also independent of the technology underlined so one of the questions we always get when we first start talking to people is they ask us questions like, does it work with dot net? Does it work with flash? Does it work with objective C? And our answer is always yes as long as it is rendering to the screen, as long as it has got some sort of user credentials to act with then we can test it. We came up with this approach about 4 years ago and we have had a lot of success.
Let’s set the scene for this then, what is the size of the market we are talking about here currently and where has the growth in mobile apps come from?
I think that this is a really interesting time in terms of apps because we are in almost a second wave of app development and I think about 4 or 5 years ago when the iPhone 1 really hit the streets then came the big surge of interest. Most of that interest came on the wave of advertising, media apps and a lot of apps that were really just about giving people phone access to existing websites also novelty apps and games. I think this was the first wave. I think we saw a decrease in that about a year and a half ago. I think that initial wave started to decrease especially on the media and on the advertising side of things; games and some of the novelty applications kept going.
The really interesting thing over the last 12 months is that enterprises have really started going into more the implementation phase with their apps. I think about 3 years ago enterprise software companies got a bit of a wakeup call when everyone had to have a mobile strategy, they had been thinking about that and they had come up with their plans and they have now hired their teams and they are really getting going.
A lot of the people we are talking to are enterprise software companies or traditional software companies who are now building their mobile applications and their mobile client rather than pure mobile app development shops. I think this is really interesting because we have got a lot of new companies coming in which is leading to a big surge in the number of mobile apps.
It is leading to much more complicated mobile apps, these are not just loading an HTML page and then really letting the browser framework to do all the work, they are actually real applications and they have got business logic in there. They have sometimes got more sophisticated UI techniques in there. It also means that they are not entirely applications in their own right so to test the mobile application you are not just testing the mobile application you have maybe got to interact with a backend and may be you have got to interact with a desktop as well. This is particularly relevant to the test industry because for the last few years I think tests for mobile apps have evolved separately to traditional testers on the desktop. This means that a lot of teams have their mobile testing app and their desktop testing app. This of course creates problems if you are trying to test one application which has a mobile part, a server part and a desktop part. These are a lot of the dynamics that are making testing mobile applications interesting and a challenge.
What are the types of testing that all apps must go through before going to market?
It is less about the skills required to test the mobile application I think it is more the mindset that strikes me as being really different because if I think about traditional enterprise desktop software then I am thinking about SAP, Oracle, ERP systems where the focus is very much on functional testing. It was all about: “I am inputting these values and I am pressing a button and that gets the database sorted correctly and then when I retrieve it to get the right information.”
As a result of the fact that the first wave of apps was about advertising and media and also because of what influence Apple have had. Mobile applications are more about user experience and a lot more about the interaction with the device than maybe performance and just usability. So when people come to test mobile applications there is of course the functional testing but there is a lot more performance testing. Most teams I know testing mobile applications have proper performance responsiveness criteria and that is not something I see on the desktop. There is also a lot more behaviour driven testing so testing from the users story and user experience rather than function points.
This is what Eggplant does quite cleverly isn’t it, mimicking human behaviour?
Exactly this is why we think we are seeing a lot of success in mobile because the fact that we are a visual tool and we are based on exactly replicating how the user works rather than looking at the code I think that is a very natural thing for us. It is also not just about encoding what a user would do, it now means you don’t have to be a developer to use Eggplant whereas to use some of the traditional heavy weight tools you have really got to be quite a scripter and understand how UI frameworks work.
The way it uses image capture technology means that you no longer need any additional hardware anymore either?
Absolutely not we mostly run over Wi-Fi and some people run us over USB but that is all the equipment you need. The other thing is we are really not invasive as in some of the testing solutions we have seen especially on mobile. Because of the signing restrictions they usually need you to compile something into your application or modify your application in some way or they need you to jailbreak the device and of course a lot of people are uncomfortable with that. You really need to be testing in the environment the software is going to run in, so we spend a lot of effort trying to make sure that we have a solution that does not require you to modify your application in any way or jailbreak the device.
An obvious question then would be: why not get a bunch of people to sit around for a bit testing these apps manually on the devices that they are intended for?
There are 2 or 3 reasons for that and the main one is just time and effort. I was talking to a company the other day doing mobile development and every time they make a release they like to test across at least 3 or 4 different iPhone variants and then 10 android variants.
What the development manager said to me was he has actually got to a point where he can’t really make changes to his applications so they get feedback and it would be one line code change for a developer but they are not doing it because it would be two weeks to test it with manual testing to get that release.
I think the original test automation was a more a progression test from release to release whereas in mobile testing and I guess with the more platforms we have it is not just from release to release but also across these different platforms and the test matrix is really expanding and if you are doing manual testing just your time for releases and getting the releases out regularly and rapidly is going to be incredibly difficult. Then there is the cost of course because that is relevant and having an army of people to test every android variant is just not practical. Then finally there is the reliability of the whole thing, there is certainly an argument that test automation is more reliable and is more thorough and more repeatable than human testing. Test scripts don’t miss things even the hundredth time they test. I think it is all about cost, time to market and quality.
App developers are quite rightly protective over privacy around their applications in the testing stage – how do you ensure this through Eggplant in comparison with some of the concerns over web based testing systems?
Eggplant is primarily a product rather than a service. We deliver Eggplant to you and you run that within your own environment, within your own firewall and you never send your applications to us. The more people we talk to not only do you have the concerns around the IP of the application itself but especially when you go into the finance industry, retail, Ecommerce, people have legal restrictions around sending that data to a third party from say sending their customer data for example to a third party for testing so we basically provide a product that runs inside the customers firewall and that seems to keep everyone happy.
You must have to keep a close eye on the mobile app market, what developments will we see across the market over the next few years do you think?
The current trend going on all around extending existing software and existing products onto the mobile I think that will continue, I think that integrating mobile devices , lifestyle apps are going to be the obvious next thing so the smart house is very much a topic coming up at the moment so the ability to integrate your phone with your television with your stereo and then with your fridge I guess it is the internet of things and I think the mobile phone is really at the centre of that as peoples primary computing device. I think it is all about mobile apps just becoming more integrated with the environment around them.