IT redundancy helps bond trader rebound

From...

By Jaikumar Vijayan

(IDG) -- Few companies build disaster recovery plans that will prepare them for the destruction of their primary data centers and the loss of every single person in them.

Neither did eSpeed, a business-to-business online marketplace spin-off and IT services arm of New York-based bond trading giant Cantor Fitzgerald LP, whose offices were destroyed in the World Trade Center attacks.

But despite the devastation that eSpeed suffered in the September 11 attacks -- which took the lives of 180 of its workers among a total of 733 Cantor employees who were killed -- Cantor was ready to trade again barely two days later, in time for the September 13 reopening of the U.S. Treasury markets.

IDG.net INFOCENTER Computerworld main page Computerworld's topic-oriented communities Computerworld's IT career center Free daily newsletters Related IDG.net Stories Comdisco completes sale of disaster recovery unit to SunGard Mapping out a career in disaster recovery Sept. 11 Web archive project a model for companies Features PC World.com Product Finder Mastering e-commerce by degrees Fast-track training puts nontechies in IT jobs Visit an IDG site Choose a site: IDG.net CIO Computerworld Darwin Dummies.com GamePro.com The Industry Standard ITWorld.com InfoWorld.com JavaWorld LinuxWorld Macworld Online Network World Fusion PC World Publish.com UnixInsider IDG.net search

That was possible largely because of the redundancy built into the eSpeed network and round-the-clock recovery efforts by the 300 remaining eSpeed IT workers in London and the U.S., said Matt Claus, the company's new chief technology officer.

"The reason it was possible was because of our previous strategy to build a concurrent computing center" that mirrored all the services in eSpeed's World Trade Center facilities, Claus said.

Since February, eSpeed had been building concurrent computing capabilities at its alternate data center here. The goal was to ensure uninterrupted uptime by having all services run simultaneously at two locations linked by a high-speed fiber-optic connection.

At any given time, half of eSpeed's trading and back-office applications ran live at its primary data center in the World Trade Center, while the other half ran live in the mirrored facility here, about 15 miles northwest. All applications running live at one location were periodically switched to the other site as part of the company's disaster preparedness exercises. Data replication software mirrored critical databases at both facilities.

In addition, eSpeed made it a point to use only 10 percent of the processing capacity on its servers so it could handle sudden spikes in volume, Claus said. ESpeed also used its London facilities to back up operations in New York.

So when the World Trade Center towers collapsed, all transactions running at eSpeed's New York facilities automatically kicked over to its sites in London and Rochelle Park.

Services to eSpeed customers in Europe and Asia were unaffected by the failure of the company's primary data center, but U.S. customers lost their connections to the eSpeed network. The attack also took down eSpeed's connections to banks, making it impossible for the company to settle and fulfill trades. Also destroyed were millions of dollars worth of servers, storage hardware and networking equipment.

One of the immediate tasks was to restore the company's trade settlement capability. A decision was quickly made to outsource the function to Automatic Data Processing Inc. (ADP) in Roseland, New Jersey Under that plan, eSpeed would send the output of its trading system to ADP through dedicated lines. ADP would then handle the fulfillment function.

The decision meant having to integrate eSpeed's trading systems with ADP's settlement systems within 48 hours. "We had never worked with them before, but we were able to leverage our development staff in London to work with ADP in designing and architecting a way for us to transmit data to ADP's systems," Claus said.

In less than two days, the teams in Rochelle Park and London also reconfigured U.S. customers with eSpeed's networks in London.

Despite the enormity of Cantor's losses, its fate on the IT front was similar to that of other financial services firms. Because of business and regulatory requirements, most affected companies had redundant architectures. "In most cases, the disaster recovery measures worked very well," said Larry Tabb, an analyst at TowerGroup in Needham, Massachusetts.