From build to buy: American Airlines changes modernization course midflight

The airline's still revamping core legacy apps, only in a different way.

 

Computerworld - American Airlines was well into a simultaneous revamp of its Passenger Services System (PSS) and Flight Operating System (FOS), its two most mission-critical families of applications, when the airline changed course last January. 

The plan still calls for a gradual migration off of an inflexible and outdated mainframe architecture in favor of a modern, distributed computing platform. But while the FOS focus has always been buy rather than build whenever possible, the focus for the PSS project has turned sharply away from rewriting all of the applications that make up the system in house in favor of buying existing software whenever possible and modifying it as needed.

The ramifications of that decision are still playing out.

American mobilizes its forces

In addition to modernizing its back-end infrastructure and applications, American is providing flight crews, airport agents and maintenance staff with access to operational information from a range of new mobile devices.

These devices provide new capabilities to different staff functions, including:

Flight attendants have been issued Samsung Galaxy Note tablets to access real-time customer information such as seat assignments, premium class food and beverage options, loyalty program status, connecting gate information, delay information and requests for special services.

Aviation maintenance technicians (AMTs) are using Samsung Galaxy Tab tablets to communicate with tech services, access technical information, receive and close tasks, and check maintenance history and parts availability -- all while working on the aircraft. Previously, AMTs had to return to their desktop computers or refer to manuals.

Airport agents use "Your Assistance Delivered Anywhere" mobile check-in devices to access current flight information, print boarding passes, check bags, print bag tags and perform other tasks. These devices are purpose-built, not off-the-shelf mobile hardware like tablets or smartphones.

Pilots will soon replace their flight bags with an electronic version that runs on an iPad. American estimates that eliminating the 35-lb bags from every cockpit will save the airline $1.2 million of fuel annually.

-- Robert L. Mitchell

For the last 47 years, both PSS and FOS have run on Sabre, a mainframe-based system that now lacks the flexibility and speed to market that the airline needs to compete. American began its modernization initiative with the launch of two projects several years ago.

The Jetstream project will replace the current PSS, which handles reservations and other customer-facing functions, while Horizon has already begun replacing many of the applications that make up the FOS, responsible for flight operations functions such as route planning and crew scheduling.

"Tackling these two important systems gives us the opportunity to rethink the customer experience as well as get away from massive, monolithic systems...that make it more difficult for us today to be responsive to customer needs in the marketplace," says Maya Leibman, American's CIO.

While American says that the projects are about cost savings and faster time to market, more is at stake here, says Henry Harteveldt, an airline and travel analyst with Atmosphere Research Group. "It's about how these new systems can help American generate more money," by creating new revenue-generating opportunities.

For her part, Leibman declined to go into specifics about the revenue-generation aspect.

The new systems interconnect applications using Web services and enterprise service buses (ESBs) that are part of the airline's service-oriented architecture (SOA). This will allow American to do things like share data in real time between applications, whether those run on or off the mainframe.

A Horizon application might need to know, for instance, that a specific plane has been delayed, or that weather is threatening one city, or that a particular series of jets needs FAA-ordered maintenance.

You see very few 47-year-old cars on the road these days. But with the airlines, some of their most important decisions are tied back to Mad Men-era technologies.

Henry Harteveldt, analyst, Atmosphere Research Group 

 

One ESB, Flight Hub, can pull data from the FOS on the mainframe, or directly from other applications designed to run in the new, distributed computing environment.

Like other major U.S. airlines, American has found itself falling behind foreign carriers in moving to a modern -- and flexible - architecture for its POS and FOS systems. A modern software and hardware infrastructure will allow American to move much more quickly in a world where a substantial percentage of business comes in through AA.com, the airline's website, as well as from travel partners around the Web.

To adapt, American has over the years bolted new features and functions onto the mainframe through the use of application front ends. But that still limits flexibility when it comes to distributing and bundling new fare offerings or creating different privilege options for customers, such as lower change fees or refundable tickets. Today's airlines need to be nimble, respond quickly to market changes, and be free to sell they way they need to sell.

"You see very few 47-year-old cars on the road these days. But with the airlines, some of their most important decisions are tied back to Mad Men-era technologies," says Harteveldt.

Former CIO Monte Ford initiated and managed the projects until 2011. Ford departed after 11 years as American's CIO for reasons that were not made public, and the airline tapped Leibman, the former president of American's AAdvantage Loyalty program, to fill the role in December 2011. That was a month after the airline filed for Chapter 11 bankruptcy protection.

   

"The new systems are designed to fundamentally improve both the customer and the employee experience, which will lead to increased loyalty and revenue," says Maya Leibman, American Airlines CIO.

 

It was Leibman who reversed the initial build versus buy strategy last January -- a move that Harteveldt calls "gutsy."

"It takes a lot for a CIO to go to management and say, 'I think we need to rethink this decision,' " he says. American IT execs won't comment on how far they had gotten, or how much they had spent. But the airline hadn't started coding to replace the Jetstream/PSS yet - that project was still in the business process evaluation stage. On the FOS side, work had already been completed on many programs.

The decision to change focus was driven not just by the need to reexamine priorities as the business reorganized, but by changes in the market that could not have been anticipated when American made its original plans. Initial planning for the projects began with American's outsourcer, EDS, before Hewlett-Packard acquired it in 2008.

But after the projects officially launched in 2010, HP laid off many airline-related programmers, project managers and others associated with the Jetstream project, according to Harteveldt. "That was one of the factors that contributed to the project's problems," he says. "It just became a very frustrating experience for American."

Liebman's response: "I can't comment or speculate on our business partners' internal matters, but HP remains an important and strategic business partner of American Airlines."

It was a good time to take a step back and rethink the strategy, Leibman says. "We questioned whether it made sense to make the tremendous investment of time and resources that go along with building something yourself -- or look for something that others had already built and tested. In this case we made the decision that we wanted to buy," she says.

Jetstream's slow take-off

Jetstream, the new PSS initiative, officially got off the ground in 2010. (Some early work, including the decision to dedicate resources to the project, had started in 2007.) Led by vice president of business technology services Daniel Henry, who reports to Leibman, Jetstream will eventually include modules and sub-modules to handle reservations, passenger itinerary management, shopping, ticketing and pricing, check-in and other functions.

 "Our ultimate goal is to have a customer-centric passenger services system, which will allow our agents to pull up a customer by name and have all their relevant information at their fingertips," he says.

Henry says American is intentionally taking it slow. "This is not a big bang project. This is the heart of the airline, so we are taking a very methodical approach," he says.

   

Daniel Henry, in charge of the Jetstream passenger services system, says American wants a "customer-centric" system "that will allow our agents to pull up a customer by name and have all their relevant information at their fingertips."

 

The sheer size of the Jetstream project makes that necessary. "I am not sure you can get more complex than a PSS," Henry says. "There's a lot of data transfer and transformation that goes on," with more than 100 applications sharing or synchronizing data with the core PSS software on the mainframe.

The PSS brokers the sharing of data, as well as data transformation and processing.
The new platform must do all of that and enable rapid change. "Our success will be defined by: Can we change it the next day? That's our number one success factor," he says.

Henry's team has made progress in reviewing and documenting business processes, policies and procedures, but programming plans stalled when American made public its decision to terminate its contract with HP in June.

American would say only that the buy versus build decision was not made at any one moment in time but was made over time as they evaluated different factors. "Thus, we made announcements and decisions about Jetstream as each decision point was reached," a PR spokesperson said.

 

With the new systems based on a service-oriented architecture, applications can subscribe to find out, for instance, the fullness status of specific flights. 

 

American fell into the trap of thinking that it had to build from scratch to get everything it wanted, says Harteveldt. "An airline the size of American will certainly need a lot of tailoring, but I was never convinced that they needed a totally custom-developed product." Changing to a buy strategy was a good move, he adds.

"How you manage an itinerary, that's commodity stuff," Henry explains. "But there's a competitive advantage in implementing different ways of checking in people. We want the flexibility to have [the] ability to modify [that]," he says. American also wants the ability to unbundle fares and offer new options, and has developed a merchandizing application that lets it offer such options as priority boarding.


But those functions are all still tied back to the PSS on the mainframe. Plugging those applications into Jetstream's new core transaction engine will offer greater back-end flexibility, the airline's IT executives hope.

Maya Leibman's focus

CIO Maya Leibman says the goal of American's IT modernization effort is to streamline the customer experience from beginning to end in an effort to increase customer loyalty and revenue. In leading the charge, Leibman says she is personally focused on three areas:

• "Collaboration across my technology organization, with our internal business partners and with our external partners and vendors."
• Transformation. "We are looking at everything we're doing and determining if there's a better, smarter way to be doing it."
• Speed to market. "We need to be more responsive, nimble and fast in order to meet all of the challenges we have."

-- Robert L. Mitchell

Now the question is which off-the-shelf software to choose. American is in the process of determining which product might be best, says Henry. Options include a new offering from Sabre (a former American Airlines division long since spun off); Amadeus, which makes software used by many European carriers; and ITA Software by Google.

For now, the PSS continues to run on the mainframe. American has put more modern user interface "wrappers" around its green-screen applications, but those applications are, for now, still locked into the Sabre system. "We need to get deeper into the stack and change the core engine and decouple the rules so we can change them rapidly," by leveraging American's SOA, Henry says. "We want to open up the flood of information that's locked into that system."

Ultimately, Jetstream will help American bring in new revenue through AA.com by enabling the airline to offer a mix-and-match menu of different options -- Leibman's main focus -- but it will also deliver cost savings by allowing passengers to do more things for themselves, thereby reducing the need for agent staffing at airports, says Harteveldt.

Horizon: Flight operations get a makeover

Horizon, the airline's name for its revamped FOS, is, if possible, even more complex than the PSS, says Tracy Hassell, managing director for the Horizon program and another direct report to Leibman. "The FOS let us run the airline, but every time we wanted new functionality it would take several months and a lot of money to go do it. We needed to do something differently and mitigate that," she says.

The FOS covers four broad functional areas: flights, crew management, cargo and maintenance, and engineering. The current mainframe runs an older version of IBM's Transaction Processing Facility software. "It is all one big database and it's not relational. That created constraints for us," she says.

The legacy-systems revamp will allow American to react quickly if, say, a particular series of jets needs FAA-ordered maintenance. 

 

The flat-file database makes tight integration between applications and the file system challenging, because there's no SQL interface to abstract the application layer from the database layer, as SQL-compliant databases do, she says. So rather than being able to define new SQL interfaces to the database, American has always had to create a custom application layer. "This has made the integration of new applications with the legacy FOS very complex and expensive," she says.

American began the initiative in 2007 with the development of an enterprise service bus (ESB) it dubbed Flight Hub. The ESB uses IBM products -- including WebSphere, Message Broker, MQSeries messaging and Application Server -- as its core components. It also uses IBM's Data Power for service management.

American built the system on instances of Red Hat Linux Enterprise running within VMware virtual machines hosted on x86-based blade servers, and has a few machines running Windows Server for off-the-shelf applications that require it.

   

Tracy Hassell is heading up the revamped Horizon flight operating system project for American. She says the overall goal of her group is to be able to add new functionality with a minimum of development time and effort.

 

At some point, Jetstream too will have its own ESB, Customer Hub, a new cargo system will have Cargo Hub, and so on. These ESBs serve as transport mechanisms that connect to all related applications and enable real-time data sharing between applications as well with other ESBs and with the mainframe.

Flight Hub is highly available, highly recoverable and runs simultaneously in more than one place -- unlike the mainframe, which offered redundancy within a single data center but had no backup site. "The program objective was to enable true disaster recovery," Hassell says.

The Flight Hub ESB consists of three main components. Applications can access data on demand, receive information on a regular schedule and send and receive data through topic-based publish/subscribe messaging. In the latter case, applications can subscribe to or publish information to update current flight times or let staff know when a crew can no longer legally continue to fly because they must rest, for instance. "The Horizon architecture uses standard distributed [computing] patterns using message queues and service calls between components," Hassell says.

"Flight Hub is a hybrid. It provides ESB capabilities and provides services to other applications," and passes data to and from the mainframe, Hassell says. In this way, it can pass data to and from any application and keep all data in synch.


After completing Flight Hub, American created a new relational data store to hold flight operations data, and began bringing in new applications. Hassell's team has been rewriting or replacing existing mainframe FOS applications for Flight Hub, one at a time, as well as writing and acquiring new ones. "Horizon was not keen on writing applications. Our strategy from the beginning was buy versus build," she says. "We continue to be thoughtful about which applications should be built."

So far American has 22 applications deployed, with 20 in development, and has connected a total of 55 applications to Flight Hub, including the AA.com Web site.

Around 20% of the completed applications were written from scratch, Hassell says; the rest were purchased and modified as needed.

The latest applications include:

• A revamp of the Aircraft Communications Addressing and Reporting System (ACARS), which allows text-based messaging between the cockpit and the personnel -- usually the dispatcher responsible for that flight, or operations. American runs ACARS on AIRCOM Server, an off-the-shelf product it acquired from Geneva-based SITA.
• A tool that shows aircraft locations on the taxiway.
• A mobile crew check-in app.
• An aircraft tracking application with routes of flight for dispatchers.

"We can seamlessly pass information to all of those applications and we don't have to modify them any more from an integration perspective," Hassell says.

American also deployed a new flight-following tool. Users, primarily dispatchers and the operations towers at some airports, "have the ability to talk to our airplanes in the air and report more efficiently any information regarding specific weather situations or something where they need to alert more than one aircraft," Hassell explains.

   

All American Airlines' pilots will soon have iPad-based versions of their flight bags.

In some cases American has been able to buy instead of build, with little or no customization required. One example is ACARS. Previously, those functions were written entirely on the mainframe in a proprietary language called SabreTalk. With the new software architecture, Hassell says, "We were able to deploy [new functionality] much faster and take advantage of the new technology."

American's partners, its Oneworld carriers, will also benefit from enhanced operational information sharing, as will their customers. "We are able to help them have a seamless experience," Hassell says, by doing development work to make sure American can share detailed flight information with each Oneworld airline's systems.

A matter of timing

American continues to chip away at both projects, one application at a time, with the ultimate goal of creating a more flexible and dynamic set of core systems and associated applications that can help the airline innovate. "The new systems are designed to fundamentally improve both the customer and the employee experience, which will lead to increased loyalty and revenue," Leibman says.

While work is progressing, American has no firm date for moving completely off the Sabre system and shutting off the mainframe. As new applications are added, "we pull things out of the mainframe," Hassell says. The airline can slow down or speed up development as resources -- especially money -- allow.

For American, it's a slow, deliberate and very collaborative process with the business that focuses on both business process change and the trade-offs involved when moving from a highly customized, homegrown system to buying off-the-shelf software. Collaboration with all stakeholders is essential, says Hassell. "That partnership is what makes us more effective."

Robert L. Mitchell is a national correspondent for Computerworld. Follow him on Twitter at twitter.com/rmitch, or email him at rmitchell@computerworld.com.
Johanna Ambrosio is managing editor for technologies at Computerworld. Follow her on Twitter at twitter.com/jambrosio, or email her at jambrosio@computerworld.com.
Read more about Applications in Computerworld's Applications Topic Center.