With double the number of users, comes double the amount of data, but how do you tackle it? Pamela Whitby finds out

From retail to online travel – in fact any industry with a web presence – the scale and complexity of analysing huge and growing volumes of data seems an insurmountable challenge. But it’s also an exciting opportunity to grow market share. That is certainly the view of Skyscanner’s, senior data analyst, Ewan Nicolson, who points to three major challenges that all firms are facing right now. However, at Skyscanner respect for what data can deliver is embedded in the DNA of the business and so it is committed to finding solutions to drive the business forward. 

Challenge 1: Finding the right mix of tools to analyse and store datasets

In the data field, one of the biggest challenges is both technological and architectural. “We’re getting to the stage that the tools we grew up with (SQL and Excel) aren’t really cutting it anymore,” says Ewan Nicolson, senior analyst, Skyscanner. The main challenge is figuring out right mix of tools that can deal with the volumes of data today, so that people don't have to sit around waiting for results.

Solution 1: Take an ensemble approach

In other words, talk to a range of vendors –“a very good source of info” - and suppliers from a range of different industries, as well as developers - in particular of different open source technologies. “There isn’t just one solution that is good for everyone,” says Nicolson. “You need to find the most suitable tool for dashboarding, the most suitable tool for large data processing and then plumb all those together.” Integration can pose a problem so there are two possible approaches:

·         Stick with the same stack. If you are using a Microsoft or IBM stack then all those tools talk to each other. The problem, however, is these lack flexibility to adopt new tools.

·         Avoid the walled garden: By integrating more widely adopted tools, like Cassandra, and the big open source tools that are coming out of companies like Yahoo, Facebook and Google and then it’s easier to integrate off-the-shelf solutions, says Nicolson.

On this front, Skyscanner is still trying to find its way and “doesn't have all the answers”. Still, Nicolson says they are taking a lot from open source sites. For example, it’s doing a lot of analysis using tools like Python, which are “quite good at talking to a wide variety of other systems” and can do more complicated things, like data visualisation, which is more difficult with, say, Microsoft or IBM.

Challenge 2: Data is doubling and getting far more complicated

In the past you would look at data to understand conversion rates, revenue generation and so on – essentially everything at a top level. Today, however, firms are increasingly looking at different segments of the customer base, and even at people as individuals. “We are trying not to look at everything in aggregate but at an atomic level because we see additional information in the data that we haven’t looked at in the past,” says Nicolson.

Solution 2: Adapt quickly

If something new is found in data – such as a new behaviour - then people at Skyscanner are willing to quickly shift from how they were doing things before. Recently the firm has, through data analysis, discovered fresh information about how customers use devices for different things. Whereas previously, the same campaign would go out across all platforms, now they are becoming far more granular and targeted.

Challenge 3: Communicating the message requires new skills from data teams

The third challenge facing data teams ties the first two together. “Because we doing more complicated things and finding new value from data, this requires a lot of communication,” explains Nicolson. “That’s an interesting challenge as we are really pushing boundaries of what has been achieved before.” Today, it’s essential that data teams can communicate effectively so that everyone grasps new ideas that can drive the business forward.

Solution 3: Put data at the centre of the organisation – literally and figuratively

Quite often analytics teams are hidden away in the IT department but today’s data teams are, or should be, very different from those of the past. “We’re not just sitting here doing spreadsheets for people and saying ‘there you go’,” says Nicolson. “We are trying to add value to people’s jobs, to tell them new things they didn’t know, so that we can steer the company in the right direction,” says Nicolson. At Skyscanner, the analyst team is centrally placed and deals directly with other departments to find out their problems. This works in both directions. “ People get questions answered and problems are solved more quickly. It also means analysts have better understanding of what is happening at the coalface. In a world where data is growing exponentially that has to be a good thing.

Ewan Nicolson, senior data analyst at Skyscanner will be speaking on how to monetise data at the Travel Distribution Summit, Europe (May 22-23)