Frank Puranik, Senior Technical Specialist, iTrinegy, comments on conversations and learning from The National Software Testing Conference.

It was great to attend The National Software Testing Conference held in London recently, both as an exhibitor and a speaker. I really love this type of event as they provide a great opportunity to talk to professionals, in this case software test professionals, who are “at the coalface”.

Testing over networks is much more than just a check box exercise

Of course, my big thing, as ever, was “Will it [your software] work when you put in out into the real world network?” During conversations with several delegates who visited our (iTrinegy) stand I was encouraged to hear that they understood the concept (and saw the value and need) of including verifying application behaviour over networks as part of their test regime. A few years back, if you asked the question “So how are you checking that your application will work over in a production network?” It would not have been surprising to get a reply along the lines of “Oh, that’s not part of my remit, I leave that to the operations team/network team”.

Clearly, the adoption of DevOps has made software testers, and developers alike, much more aware that we need to look beyond the confines of functional testing if they are to ensure applications will work actually work properly when deployed into the real world, which is surely the point of testing.

As I’m ever curious to understand how they now incorporate testing over networks as part of the test programme (because it’s not at all clear how you put a real world network into the test or DevOps environment) I asked them how they were currently achieving this. Some testers of mobile applications, said “It’s tricky – we go into lifts and stairwells where the reception is poor. Others mentioned that “network emulation” (a topic close to my heart) was included in the browser or load testing product they were using. Great, but the more I spoke to them the more I realised the network test functionality they were being offered was extremely limited. “I can add a some latency”, said one, “Oh yes, and limit the bandwidth”. On chatting further it comes out that the latency on offer is fixed, and so is the bandwidth, and that’s not real world at all. I call this “checkbox” (US English) or “tick box” (UK English) functionality. More on that later…

Emulating (simulating) real world networks

Only unloaded networks offer fixed latencies (delays), and constant bandwidths, because busy networks are congested much of the time; real world latency comes from partly distance (fixed) and queuing (variable) and “available” bandwidth is what you get after others have taken the rest. Just like a road network really: At busy times you’ll be slowed down by other cars: queuing at roundabouts, traffic lights, junctions etc. and even the speed you can go on a straight road, like a motorway, goes down. The only time you can guarantee your journey time is at about 3am (night roadworks aside).

So emulating (simulating) real world networks properly requires the control and variation of lots of parameters at the same time, congestion (other traffic), variable queuing, variable data loss etc. That doesn’t mean it has to be complex for you. You just want to say “Give me a real world 3G network, for example, in a pretty poor state [i.e. busy or in a poor location] to test with” and we should take care of making it so, and we do. By the way, don’t think this is irrelevant in corporate WANs – they get congested with corporate – and social media traffic.

… but as more software testers and software engineers became savvy about the need to do real world network testing and demanded a method of doing that easily, vendors of other test & development tools:IDEs, functional test automation tools, performance test tools decided they had to provide it. So they offered – a bit of latency, or maybe restricted bandwidth, or a bit of loss. They offered checkbox functionality – by which I mean when you asked them for network emulation they can check the box on your requirements list to say they can provide it. So now they’re equal with all the other vendors you’re looking at on this important topic and you believe you have a method of “real world network testing”. You have not; your application will likely still fail when it meets the real world network.

Summary

Checkbox engineering is so easy to get into in product development. Chances are your company does it too: You have a missing feature which you’re getting beaten up on in the market, but it’s not core to what you do, so you put in the minimum effort to “check the box” which states that your product has it.

Don’t be a victim of checkbox engineering when it comes to real world network testing – you don’t just need “a bit of emulation” you need a real world network in the test environment, because you know that (just as a road network greatly affects your journey) real world networks pose a significant challenge to applications. We call producing real world network in your test or DevOps environment “a virtual test network” which is a much better description of the actual requirement, not just “a bit of emulation” – which only describes the proposed solution.

Edited for web by Cecilia Rehn.