Rob Hornby, Senior Test Manager, John Lewis, gives insight on the award-winning Peak Scalability and Click & Collect testing project at one of Britain’s largest retailers.
Click & Collect is a popular way for retailers to bridge the gap between their online and physical presences, often offering customers a quicker way to receive goods that they order over the internet compared with having them delivered to the home.
For John Lewis, the Christmas retail period is divided up into three peaks: Black Friday, Christmas and Clearance.
This article delves into the work of the testing department during the Black Friday 2014 period, summed up as the Peak project, which was the overall winner at The European Software Testing Awards in 2015.
This article will also give an overview of the retailer’s continued work in 2015.
In 2014, the five weeks in the lead up to Christmas including the phenomena of Black Friday was a significant period of heightened website activity when compared to the rest of the year.
Thanks to the success of the testing department’s Peak project work, John Lewis saw a successful trading period where:
- Online sales accounted for 36% – up from 32% the previous year.
- Click & Collect accounted for 56% of those sales.
- Supply chain picked up 54% more parcels than previous years, 87% during peak Black Friday sales delivering in to a Saturday.
Mark Lewis, Online Director at John Lewis, said in the company’s trading update for the week to 27 December 2014: “It was the week when our Click & Collect service broke records. We had billed 2014 as the Click & Collect Christmas and it certainly came to pass.”
Methodologies behind a successful project
The peak programme
John Lewis recognised that the peak period of Christmas is critical for trading success. Each year, the company prepares through a cross‑functional initiative to ensure that everything is ready from the first customer touchpoint – the website – through to the actual picking up of an ordered item from a Waitrose or John Lewis store.
A programme of work was created and run to support the run up to peak. The focus was in two key areas, the website, ensuring that it could meet the expected customer volume and order rates, and supply chain which focused on ensuring that the orders could then be successfully delivered to customers and Click & Collect orders to John Lewis and Waitrose stores. The key was for this to be delivered at scale.
The website
The website is an entry point to the John Lewis experience. The marketing teams drive traffic to the website through a co‑ordinated campaign for Peak. For 2014, this focused around ‘Monty the Penguin’. Traffic then builds throughout November with spikes in traffic for sales such as Black Friday. To ensure that the website could meet capacity John Lewis undertook several preparations.
The company formed a cross-functional engineering team made up of data scientists from the insight team, operations, test, networks and infrastructure. A test model was formed utilising site analytics (Adobe) and operational tools (Splunk), using a common currency across teams of TPS (transactions per second). The common currency allowed the team to clearly communicate website performance across both business and IT in a non‑technical manner.
The testers built a set of performance scripts which mirrored the live site behaviour and a scaling factor was agreed between the test and live environments. These tests were then run and analysed against actual live behaviour to ensure they were consistent with the measure of live and predicted peak TPS. The testing department worked closely in partnership with HP to ensure their SaaS platform enabled them to burst capabilities when required to the predicted live peak volumes.
A programme of exploratory testing of the website was run in different configurations and optimised changes made across networks, server and application (especially caching strategies). This testing also had to contend with new features and functionality being delivered to the live platform on a monthly basis. The monthly change was tracked and trended to ensure that performance was maintained and issues resolved over subsequent releases.
The commercial teams worked to manage webpage budgets to ensure page size and load were optimised for content without availability of a full CDN (content delivery network). The company also managed testing environments to ensure the testing day was maximised through a 20hr schedule running over six months.
John Lewis developed an email marketing campaign that would be linked to the final website performance results ensuring that traffic was driven to the website in a controlled manner while maximising volume.
Website outcome
The result was a successful peak trading demonstrated through trading over Black Friday. The marketing teams were able to release their most aggressive campaigns. The website remained up throughout the 2014 sales period with full availability. Sales throughout the week across John Lewis were significantly up on the previous year’s Black Friday.
The Guardian detailed this sales success: “Overall John Lewis sold an average of one tablet computer every second and a Flatscreen 40‑inch voice‑command TV every minute from the moment 24 hours of promotions began at midnight last Thursday. Internet traffic was up by more than 300% in the early hours of Friday as consumers logged on to snap up discounted clothing, handbags and electrical goods.”1
The success of the project team has led to this becoming a full time team with a focus on being the guardians of performance on the website working with project teams and ensuring performance is a key focus. TPS has become a key website KPI and is reviewed against business growth predictions and is then tested monthly as part of ongoing release change and also in monthly live tests.
The supply chain
The supply chain is integral to a successful order proposition, especially Click & Collect. There is no point in taking orders if you can’t fulfil them. This leads to a strain on stores, call centres and reduces customer retention. This was recognised early in the planning for the peak period. As with online, small cross functional teams were identified based on relevant skill sets and small projects established to ensure the supply chain was optimised. Some key testing activities were undertaken:
As with the website, peak tests were run against live supply chain systems to ensure performance of key systems based on predicted order volumes. This testing was in partnership with logistic partners such as Metapack. It was key that non‑John Lewis systems were part of the testing approach.
New testing approaches adopted risk based approaches and minimum path coverage techniques to rapidly test a number of small projects aimed at improving giving customers greater visibility of their order; purchase items later on the website with delivery guaranteed the next day and improved tracking of parcels within the internal fulfilment process.
Bottlenecks were removed through utilising automated tooling solutions to inject orders and run complex batch suites in test environments.
Time boxed testing based on priority and smart working patterns utilising John Lewis offshore test teams maximised the available time for testing.
Improving regression testing capabilities in the team through a regression scrum to build regression testing based on business processes, ensuring that tests were optimised and automated reducing execution times while maintaining and actually increasing coverage.
Supply chain outcomes
Dino Rocos, Operations Director, John Lewis commented: “There was clearly huge customer anticipation of Black Friday in 2014 and we knew we had high expectations to meet, both in terms of the products we had on offer through our ‘Never Knowingly Undersold’ commitment, and in ensuring that we fulfilled customer orders as promised.”
Should the website have crashed, run slowly during the 2014 Black Friday sales, or suffered a backlog of online order deliveries, John Lewis would have classed this as failing to meet customer expectations.
The avoidance of these potential issues highlighted how John Lewis succeeding through close working between teams across the business. From the marketing department through to the logistics team there is a recognition that optimisation of performance and functionality provided a much greater capacity to fulfil customer propositions such as Click & Collect.
“While the sales figures are attention‑grabbing, for me our biggest achievement was delivering an operation which ran like clockwork,” Rocos said.
“We picked and packed 87% more online parcels on Saturday than we did last year, and to have delivered successfully on customer expectation is a testament to the work of our Partners both on Black Friday itself and in our forward‑planning. Our website coped well with exceptional demand.”
Key testing success
Within the testing practice, it is all about how testing teams can work more closely with both commercial and IT areas. Although the company employs successful testing techniques such as performance engineering, testing automation, agile testing methods, it is more how they are employed in a team environment. The success of the Peak project and Click & Collect is down to how the organisation is self‑motivated through Partner (employee) ownership.
Morale is high at John Lewis, thanks to the unique Partner ownership. The Partnership ensures all employees are looked after and a transparent, democratic leadership instils a corporate culture of good work that pays off – all Partners receive % bonuses depending on the retailer’s overall performance.
Supporting the IT function
John Lewis has recognised that IT as a function is at the heart of how to deliver
great customer service. The goal is to become more agile, both in practice and in culture.
“Our challenge going forward is less testing,” Alex Wotton, Head of Testing and Environment Practice at John Lewis said. “We want to transform so that automation becomes the norm.”
It’s an engineering challenge as much as it is mind‑set. John Lewis is investigating and investing in continuous integration, delivery and rear end automation.
“We also need to change the skill sets of our testers,” Wotton added. “In order to make sure they can cope with an agile, DevOps approach.”
Looking ahead
Thanks to the peak programme in 2014, the retailer was more than ready to handle the same influx of orders in 2015.
There was not a spike in order volume, but thanks to the work carried out earlier and the continuous testing, the platform was much smoother and ran with few issues.
The trend in the industry is to take this performance engineered approach in to the wider delivery. Better non‑functional requirements, capacity planning and baselining of production systems are all areas of focus along with mind-set changes in teams to ensure they are thinking about performance early in the delivery lifecycle. Smaller more optimised changes are a focus within agile teams. In addition to this, testing needs to account for industry trends around the move to mobile apps, big data and heightened security all of which require a more engineered solution including how testing is carried out.
Summary
In summary, John Lewis delivered a successful Peak project in 2014. The retailer’s strategy, ensured successful delivery:
- Ensured the website remained operational through peak trading periods including the highly competitive Black Friday.
- Collaboration of teams across John Lewis ensured that the end to end journey was optimised and clear measures put in place for success.
- Cross-functional IT teams formed an engineered approach to improving performance across the website.
The delivery of many of the projects within a short period of time was testament to the hard work of the Partners and third parties within John Lewis. The Partner owned nature creates a unique culture and desire to drive success.
This article was first published in the May 2016 issue of TEST Magazine. Edited for web by Cecilia Rehn.
