Philip Clayson, Technology Developer at TalkTalk, reflects on how disastrous technical debt issues were taken care of after TalkTalk’s cyber attack 

Clayson’s journey at TalkTalk began three years ago with the acquisition of Blinkbox from Tesco. He then spent a year helping to remediate the TalkTalk technology portfolio after the well-publicised cyber attack in 2015. In the last year, he has been leading a massive software transformation which is addressing many many years of technical debt in the software estate – a complex problem and a sizeable challenge which has over 84 million lines of code, hundreds of software applications spanning every conceivable purpose, and nearly 1000 code bases. TalkTalk’s target is to reduce this by half in a year.

“I am passionate about inspiring technology teams to engage quickly to create, implement and operationalise transformational strategy, and I do this in complex environments where acquisition or under-investment has created a legacy technology estate that needs rationalising, modernising and improving. Many infrastructure-based companies have this challenge across sectors ranging from utilities to energy, and financial services to communications,” said Clayson.

Accelerating growth

Historically, TalkTalk has focused on accelerating growth, and achieved this through both organic and acquisitive means, the latter bringing with it duplicate technologies with each business. With so many acquisitions, the technical duplication is sometimes not always addressed in full and technical debt accumulates.

Clayson revealed: “I have started the software team at TalkTalk on the journey of a huge investment to start removing our software duplication, reducing obsolete code and re-engineering software stacks. We are also adopting new ways of working including a generational step change in software tooling, AI and cloud in order to achieve an improved operational business with far fewer software packages – we’re aiming to have the best and most effective software application portfolio in our industry.

“My team develops and maintains the software that runs TalkTalk’s business systems. The alignment of tool-chains has been on the“ to do” list for many years. Last year we reviewed both open source and commercial tool-chain solutions, but we made our first large-scale commitments to best in class tooling from HP, MicroFocus and Computer Associates in the summer of 2017. Each toolchain, from test-driven requirements to code scanning, brings different benefits that help our business and consumer customers to have a far better experience with TalkTalk.”

TalkTalk’s innovation

TalkTalk is a modern day blend of Financial Times Stock Exchange (FTSE) Company and a ‘grown-up start-up’; According to Clayson, the firm thrives in the combined environment, as it creates the right culture for innovation, while his team underpins some of the most complex and important software systems.

Clayson added: “My teams have the support and space to innovate in their day-jobs, and everyone is encouraged to provide solutions to the problems we solve daily, with funds and time set aside for the best ideas to grow. It is amazing to see how new employees (graduates and experienced) drop into the TalkTalk innovation model on day one, helping the wider organisation stay fresh on our innovation agenda. Having doubled the size of the UK software team in 9 months getting that cultural point working correctly is critical.

‘Providing the best customer service’

“We debated long and hard about whether to go open-source, or Commercial Off The Shelf software (COTS), but took a strong influence and reference from the trust our customers put in us in order to provide the best customer service we can, and to ensure great, reliable software, every time. To do that, we have largely chosen COTS-based solutions.

“The advantage we feel we can take from COTS is the suppliers have to remain competitive and therefore they invest in keeping their platforms current, if not always ‘bleeding edge’. This gives us, and our customers, some stability, certainty, and predictability – to ensure we can develop our end-user software cost-effectively and reliably. We, of course, continue to watch the open source community too, and the choices we make in the future will trade off value, stability, and responsiveness, of both COTS and open source options.”

Over the years’, TalkTalk has developed a capability to test everything from unit, system and end-to-end testing by partnering with the world’s leading software test partners. But, recently increased its automated testing, continuous integration, and continuous deployment expertise by investing around three million pounds in the last 12 months alone, to get less manual test activity, increase release cadence and reduce development cycle times as it moves to agiler and more automated development. The teams have already removed days of testing per cycle (in one case 80% in a release cycle) with these tools and are still tuning them at the moment.

‘Removing 50% of tech debt in one year’

Clayson continued: “We started with a truly ambitious shared goal in mind. We decided, as a team, that we would remove 50% of the technical debt in one year – that was it. Everything else fell in behind that, and everyone now has one single shared personal performance objective on this.

“However, with all simple objectives, the complexity is still there. As we got into the challenge of unpicking old hard-wired software applications that were difficult to evolve separately from one another, and with hundreds of them, the problem quickly spiralled.

“We decided to formally adopt the TM forum approach to categorising our application estate, something TalkTalk had never done before. We promoted our best internal people, and hired some great new people, to create a small but dedicated team of Subject Matter Experts (SME), each with a goal of addressing the applications with the highest trading risk first. Each expert worked to define the transformation needed and unpicked the technical debt in their area.

This spanned many areas including billing, assurance, data, databases, etc. There were over 20 of these groups, each one having an SME and a team of people defining and the right outcomes for TalkTalk. That structure helped to give a framework to the team, and from that, the team and the entire board could all see the direction, deliverables, and value of the approach.

Developing ‘duplicate feet’

The software teams at TalkTalk have committed everything to this intellectually, emotionally and physically, but according to Clayson, the hardest thing was balancing continuing business as usual roadmap deliveries and transforming away from technical debt without breaking the business operationally.

Clayson agreed: “We think of it as a bit like painting the ground your feet are stood on – it’s a massive logistical problem.

“Because of this, we had to develop ‘duplicate feet’ so the business didn’t need to know or worry about the fact the software was changing underneath them. We then had to carefully move the existing applications onto their new feet. This requires immense levels of thought, planning, and ingenuity. Doing this without disruption was the hardest challenge. The sheer volume of the unpicking required and intricate planning we had to do has slowed us down, but we’re still committed to the delivery of a 50% reduction in tech debt on schedule.

“I think we’re a long way ahead of the market here, I don’t know of anyone else across an infrastructure business like us in the UK that has made so much progress in such a short time. We have worked with and shared learning with a few European companies on our journey, companies who share our aspirations, speed, and commitment. That learning has helped us and we’ll soon be completing the first phase (the first year) of our software transformation, and as planned we’re rapidly approaching a reduction of over 50% of our software technical debt, a truly outstanding achievement by my teams.”

Pushing deeper into core software stacks

TalkTalk has solid plans for the next 12 months, and a strategy beyond that into 2019, but as the company moves to further consolidate software tools and processes together, and push deeper into the core software stacks, some of the hardest work is still to come.

Clayson added: “We have now really started on our journey to create the modern software estate we need, but we are not locked into any supplier or tool-chain and we will flex and change over time based on the performance of any one supplier or solution.

“We have adopted Computer Associates (CA) for the largest suite of our tooling, with other tools from MicroFocus and HPE. In the first wave of our journey, and whilst the deployment journey has been quick, we have had implementation challenges, where cloud solutions would have helped considerably.

“In the longer term I am only looking for cloud-based solutions for these types of products, so we can burst test at scale when we need to, and so we know when we pull a cloud instance down, the data is properly deleted, a key required for our GDPR compliance. We’ve made a great start with the choices we’ve made but COTS providers need to have cloud and old-style solutions to offer.”

‘Changing for the better’

Tooling is a rapidly evolving market. For all chosen tooling providers, TalkTalk ensures they carefully create business cases that work in the short and medium term, so it’s not locked into multiple years of investment returns.

“Well, you can tell this is a massive commitment for TalkTalk. TalkTalk’s software transformation journey has made amazing progress in its first phase, but there’s more to do in the next 12 months. I’m immensely proud and impressed at the way the entire business systems software team at TalkTalk has responded to the desire to change things for the better. The amount the teams have achieved in laying the foundations for an overall outcome will be nothing short of game-changing for TalkTalk, and noticeable by both our consumer, business and internal customers,” he commented.

Clayson also noted the journey gets tougher before it gets easier. TalkTalk software teams in the UK and internationally are ready for the next wave of this critical and exciting software transformation!

Written by Leah Alger