It’s a little understood cyber threat but it could be the biggest threat to the travel industry today, writes guest columnist Martin Zetterlund

The wholesale theft of web site content known as web or screen scraping, data harvesting, or web data extraction is a growing risk. Many businesses are losing income and yet those same businesses are often completely unaware that they have been victims.

The travel industry and airlines in particular are identified in a recent ScrapeSentry report as being the victims of increased levels of scraping over the past year. They’re also the third most likely business sector to be scraped, after online directories and classified advertisers. Some key findings from the report are that the average Internet site experienced 10% more attacks in the year 2013. The worst hit found that 50% of Internet traffic in 2013 was in fact due to scrapers.

A definition: Web scraping is the removal of data and/or intellectual property from web pages. Web scraper as a term is used to describe both the automated software programme itself, also known as a web bot, and the person or organisation behind it. Usually a scraper will monitor the performance of his/her programme on each target site and will quickly notice when they are getting blocked. A malicious web bot is a programme that retrieves or posts information from/to a large number of sites more or less autonomously and is not likely to immediately notice a block on a specific website.

Yet despite the industry losing potentially billions of dollars to scrapers, how to handle this remains a controversial subject for airlines. There tend to be four responses to the risks of posed by scrapers:

• A lack of understanding of scraping as a threat/opportunity
   
• Knowledge of the issue but doing nothing about it
   
• Welcoming it or encouraging it with rebates
   
• Fierce and public opposition to it through PR campaigns, court room battles or technical measures

Ryanair is one that falls into the latter category. Back in 2012 it lashed out at scrapers calling them “a bunch of scam artists”, “the greatest source of passenger complaints” which “flip prices up 10%-15%, and they don’t pass on terms and conditions”. For those airlines, ferry operators and travel companies that value their brand relationships, this lack of communication is of course a huge frustration.

Another is Nick Aristou of Four Seasons Hotels who says the practice of scraping is real cause for concern for hotels.

Why should you care?

Understandably, many traditional airlines see the tickets sold through scrapers just as income generated (by any means), so scrapers are welcome on their websites. For some airlines, as an example, as much as 40% of their bookings come from scrapers, where travel aggregators collect information and drive traffic away from the website of the company that actually delivers the service and to the online travel agencies (OTA).

But those airlines using a global distribution system (GDS) on their back office systems will actually be bearing a cost caused by scrapers; exponentially they do more searches for flight availability than an independent customer.   

These excess web requests drive up the look-to-book ratio, and significantly increase the airline’s costs. The other financial consideration in allowing third-party bookings is that the opportunity to drive ancillary revenues from additional services is lost. Some airlines we know will earn £15-20 per ticket sold in ancillary revenues, and even more traditional airlines will also pay aggregators and scrapers for bookings or give them volume discounts. This adds an extra cost to bookings not made by the first line customer via the website.

In addition to the financial costs, there are less tangible, but still very important brand damage costs associated with allowing the travel aggregators to direct airline customers to third party bookers. This is a concern for easyJet because of the loss of control over the sales channel which can negatively impact on the brand relationship with the end customer. 

So, for example, without a direct relationship with the end client, problems can arise when communicating flight changes, whether those are welcomed or not. There is a real value in owning a direct relationship with the traveller with the ability to send emails directly to them with special offers and on-selling opportunities. 

By controlling that channel, travel brands are able to become more tailored and profitable in their approach to marketing. easyJet has taken action against scrapers and Jerry Dunn, the low-cost airline’s distribution development manager explains how: “By utilising an automated system for detecting scrapers gives us the information we need to act on each situation, and block individuals if we wish to.”

What are the options?

Having witnessed the success of several low cost airline’s internal dynamic packaging practices, many traditional full service airlines are now rethinking their own business models and in turn their position on scraping.

It’s important to remember that scrapers do not discriminate between ultra-low-cost, low cost, or full service traditional airlines.  Everyone is a likely target for scraping. The profits available for the dynamic packaging of ancillary services like insurance, car rentals, and hotels are enough financial motivation for scrapers to continually try to adapt their technology an bypass any measures put in place.  In my view thinking about a ‘captcha’ system that infuriates genuine customers and acts as no deterrent to scrapers is, effectively, a waste of time.

When addressing the scraping problem and how best to resolve it, the answer is often as much of a business issue as it is a technical issue. Left unattended OTAs will make up a large chunk of bookings through the use of web scraping tools and it could have a significant impact on seat sales if that were to stop overnight.

The best tactic would be to selectively target and block scrapers a few at a time, while at the same time communicating with them to address concerns. Establishing who they are in order to initiate a conversation can be a challenge for some organisations.

Some airlines choose to let scrapers book through APIs where they can be controlled. In such cases, the damages they normally cause to ancillary sales can be compensated by a referral fee or agreed cost-per-booking. This might be the most realistic option in that everybody wins (at least something). It’s an option that easyJet utilises with success. But of course it also requires the airlines to utilise anti-scraping measures, in order to push the OTAs through the APIs.

Or another option is to use anti-scraping solutions to selectively stop scraping OTAs from booking on specific routes that are easy to fill without their help.

Airlines, hotels and ferry companies will each handle scraping in their own way. The shortsighted ones probably won’t deal with it at all and the successful ones will make it work for them. 

Martin Zetterlund is a founding partner of ScrapeSentry. His views are his own