So it wasn't a malicious attack that overwhelmed Ellie Mae's servers after all. What exactly was it then?
The mortgage technology developer has backtracked from its previous explanation that a distributed denial-of-service attack overwhelmed its servers from March 31 to April 1. Ellie now says the outage "was triggered by a confluence of factors involving network, hardware, software and demand for service."
That raises the question of how a provider of a loan origination system and ancillary technology could have been inundated with service requests at a time when originations are in the doldrums.
Industry volume was off about 50% from a year earlier in the fourth quarter and at
There are at least two plausible alternative explanations. One theory is that the way Ellie's LOS stores data, using what's known as a flat-file database, is increasingly becoming difficult to scale as more volume flows through the system. Another possibility is that the recent addition of new correspondent investors and expanded features to its Total Quality Loan program could have been the source of the surge in demand.
The mystery of what brought Ellie Mae's systems to a screeching halt is far more than an academic question. The outage came at month- and quarter-end, a critical time for lenders, and delayed loan closings, which forced lenders to pay for rate lock extensions and hedging losses. Home closings funded by purchase mortgages were also delayed.
THE DDOS THEORY
The characteristics of the outage initially appeared to be
This much is known: The crash occurred after Ellie Mae's servers were overwhelmed with requests to a URL that is used to download an XML file containing a list of third-party technology vendors that integrate with the Encompass LOS via the Ellie Mae Network. The requests had the look and feel of legitimate communications, leading the company to initially
"It was a valid request with a normal signature; a very innocuous request, but something that created a tremendous amount of traffic," Corr said then.
The outage affected users of both the SaaS and self-hosted versions of the Encompass LOS. Even self-hosted users must connect with Ellie Mae's servers for authentication and to access the Ellie Mae Network, a communication medium and delivery channel for third-party underwriting services like document preparation, income and employment verification, and credit checks.
Some mortgage technology vendors are troubled by the potential long-term consequences of the incident and wonder if it will shake lenders' faith in SaaS-based technology.
"When there's a problem like this, it makes the customers nervous and that concerns me," says Sharon Matthews, CEO of the electronic document management system provider eLynx. "They need to be able to rely on us They need to know that we planned this stuff for throughput and quality with huge commitment, and when that doesn't happen, I think everybody hurts."
Others have been skeptical about Ellie Mae's initial claim that the outage was consistent with a DDoS attack.
"There's a much higher probability that they simply had internal issues related to the application and the architecture than a DDOS attack," DocuTech Chief Strategy Officer Scott Stucky said on April 3.
THE FLAT FILE THEORY
Rather, the outage could have been a result of the architecture of Ellie Mae's LOS, which uses a flat-file database instead of a relational one, Stucky and others have speculated.
Flat files are linear, meaning data is stored in one long string of text. Relational databases organize data into tables that allow for faster processing and multitasking. As more demand is put on a system, flat-file databases require more processing capacity and must be scaled using hardware and virtualized servers, while relational databases can be scaled using software.
"I think it's fair to ask, did that in fact contribute to the complications of recovery?" Stucky says. "Because the amount of processing overhead necessary to crunch flat files is exponentially more than it is to crunch files in a relational database."
Corr dismissed this notion as "absolutely, positively false."
"The issue had nothing to do with access to, or recovery of, flat files as we were coming back up. This was a load event that happened at the Web services layer," he said on April 3. "This was strictly related to an overwhelmed capacity, and even the calls that did take place were hitting our database."
THE TQL THEORY
Another possible explanation for the surge in demand may be tied to Ellie's Total Quality Loan program particularly given Corr's explanation that the outage was caused by a surge in requests for a file in the Ellie Mae Network.
TQL is a technology and services package that
On March 20, Ellie Mae released a series of updates and enhancements to TQL, including new tools designed to help lenders meet ability-to-repay and Qualified Mortgage regulations. The new features include a central repository for borrower verifications, additional quality control checks and dashboards and reports to provide increased organizational and vendor oversight.
TQL revenue doubled to approximately $5 million at the end of 2013, which accounts for less than 4% of Ellie's total revenue in 2013. But given the breadth of the new functions added to TQL, it could have been the source of the surge in traffic that crashed Ellie Mae's systems.
It is not known whether the XML file at the center of the crash is a component of TQL. When presented with this theory, a spokesman for Ellie Mae said the company would not elaborate on what it had said in Monday's press release.
A DEAL DERAILED
The crippling outage was an eye-opening experience for real estate agent Kira Kirk, who says she'd never really given much thought to the software that lenders use to process mortgages. Kirk, an agent at Coldwell Banker King Thompson in Upper Arlington, Ohio, represented a homebuyer whose closing was delayed.
Kirk, who's never experienced something like this before, says she "really didn't even really believe the lender at first," when told of the delay. After hours of trying to finalize the mortgage and close the home purchase on March 31, both sides agreed to a one-day extension of the sales contract, but it wasn't until late April 1 that the deal was complete.
Since the deal couldn't close on time, the seller, who was simultaneously purchasing another property, had to figure out a way to close on her new home before closing on the property Kirk's client was buying. The buyer was forced to change moving companies because of the delay, she says. In addition, Kirk says, her client's lender paid $750 to use a different service to prepare the final loan documents.
"I really don't know anything about that back side of lending," she says. "It did enlighten me a bit to the process in lending and how it can derail a deal when it goes awry."
