Thalasar Ventures

The recent pet food recall got me thinking about crisis management. Naturally I thought back to to my post on Godaddy’s failure in crisis management. In this post I compare the how the two companies handle their own recent crises.

Godaddy and Menu Foods are both privately held companies with very different profiles. Godaddy deals directly with consumers as the world’s largest registrar while Menu Foods is the largest wholesaler of dog food on the planet supplying Walmart and the premium brands on the market such as Nutro and Eukanuba. Just based on their markets, Godaddy should have been better able to handle it’s own crisis (an outage of some 600,000 customers – many of whom were out for 2-3 days). They have an active public relations department and sales reps that directly interact with customers EVERY day. Bob Parsons is a very effective marketer. It’s also clear that he wasn’t very effective in this situation. In the end Menu Food handled their situation with far fewer resources and far more effectively.
Godaddy’s outage crisis
The outage that Godaddy experienced on November 11th, 2007 seemed at first to be attributed to improperly applied DST patches. Given that Godaddy develops in house a large portion of their infrastucture, this seemed like a reasonable position to take at the time. Indeed the initial reports of the outage drastically understated the problem.
“After 4-5 hours of intermittent disruptions of various services this morning, including shared hosting and email, the attack was contained.”
Of course any Godaddy customer knows the problem was far more extensive than an a few hours. In fact one of my Godaddy hosted sites was done until Thursday and never came back up, so I canceled the service. This was due to the fact that Godaddy had changed settings on the shared hosting environment in response to the crisis (PHP Curl) and neglected to tell their customer base. Furthermore anyone that knows anything about network operations realizes that Godaddy’s preferred explanation of the problem sounds strange. DDOS attacks should be stopped at the router level and furthermore the size of the attack is dependent on the number of zombied machine attacking. How to respond to these sorts of attacks is SOP at any major ISP.
The problem was a large scale DDOS attack. Later the message changed somewhat as it became apparent that DOS wasn’t a standard DOS but a specific attack aimed at a underutilized service at Godaddy.
Furthermore Godaddy customer service representatives were completely clueless about the problem. Here’s one customer’s experience

9TimeZones.com was offline, from 5-11am on Sunday morning. At 10:21am, I sent an eMail to the hosting company, GoDaddy.com: “Is there something wrong with your servers?” The automated reply said I should expect a response within 24 hours.
A little over 27 hours later, they replied: “Thank you for contacting Online Support. We are unable to duplicate this issue. Please ensure you’re using the latest version of your browser. You can find the latest version at the browser developer’s website. Also, we recommend the following troubleshooting steps […]”
Then they proceeded to list 6 steps which hinted that the malfunction was probably caused by MY computer. So I wrote back:
Look, this is not an effective way to handle a problem. I know the outage occurred because of the eMail complaints I received, from all over the world, between the hours of 5am and 11am on Sunday. You should know by now that a customer (like me) does *not* want to hear that you’ve failed to duplicate the problem at a much later point in time. What the customer (me) wants to hear is this:
1) you’ve checked your logs, and your server was down between ___ and ___.
2) the reason your server malfunctioned was ______.
3) the outage will not happen again because you’ve taken this precautionary action: ________.
All you need to do is fill in the blanks. But if you persist in dealing with complaints by insisting that you can’t find and/or duplicate the problems, your customers will begin to think that you are ignorant idiots who don’t have the requisite skills to run an internet hosting system.

This highlights a completely inappropriate response to the crisis. Frontline CSRs and email communication should be your first line of response. Indeed it’s apparent that the sales organization and the technical side of the house simply don’t talk to each other. When I called in on Thursday of that week, the account rep I spoke with hadn’t even heard of it. His response to me was “I am in sales, I don’t hear about that stuff.”
Indeed this was the only message heard from Godaddy’s CTO was the following statement.

This morning, March 11, some of our Go Daddy services came under significant and sustained distributed denial of service attacks resulting in intermittent disruptions of various services, including shared hosting and email.
Our Internet Security and Network teams immediately invoked counter-measures to respond to these large scale, sophisticated attacks.
After 4-5 hours of intermittent disruptions of various services this morning, including shared hosting and email, the attack was contained.
Our Internet Security and Network teams will continue to analyze and assess the nature of today’s attacks and their characteristics to identify additional defense mechanisms that can be used in the ongoing efforts of Internet Security.
Go Daddy has made and will be continuing to make significant investments in our information security infrastructure to protect from these shifting types of attacks.
This in no way related to the switch to Daylight Savings Time, as some have speculated. With regard to DST, Go Daddy has been engaged in preparation and patching and worked closely with our vendors for some time leading up to the DST change. leading up to the DST change.

This message contains some good information namely a more accurate estimate of the outage but lacks specific action points that Godaddy is going to take to prevent future problems. You can reassure customers without giving away specific information. This message doesn’t do that. Instead it contains pretty much standard boilerplate such as “Our Internet Security and Network teams will continue to analyze and assess the nature of today’s attacks and their characteristics to identify additional defense mechanisms that can be used in the ongoing efforts of Internet Security.” which consumers have come to recognize as complete bullshit on the part of the company.
I suspect that the failure to handle this crisis in an appropriate and upfront manor stems from the personality of Bob Parsons. Bob Parson’s blog never mentioned the crisis. Indeed Bob had a post on Registerfly and after the crisis had another post on Registerfly without a single mention of the problem. What’s wrong with a post addressing the issue? Clearly Bob believes that by ignoring and being silent on a problem that the problem will blow over. That’s old school crisis management and it simply doesn’t work. Furthermore with such a n outspoken company president, consumers expect him to address the issue. Perhaps Bob thinks these sorts of outages are standard practice in the industry. They were 7 years ago but are no longer.
We can draw some larger lessons about crisis management by comparing the how Menu Food handled their own situation.
Menu Food’s Own Crisis
Menu Foods is a wholesaler to many of the largest dog food providers in the United States and Canada. As a wholesaler they have almost no public presence and almost no public visibility.
Furthermore the attachment that people have to their pets is high. People have a significant emotional investment to their pets. (And their web sites Godaddy!)
Menu Foods doesn’t have a direct sales team and doesn’t interact with the public too often. As a result when the crisis first broke the company was simply overwhelmed. Their website quickly went down and their phone lines were overloaded. In many ways a wholesaler like this is ill-equipped to handle this sort of very visible public crisis. Yet Menu Foods quickly adjusted their message and used the web to get their message out.
Once the crisis became public, Menu Foods quickly adjusted the content of their website. Instead of the typical business card web site, they shifted the site to quickly a single link to the product recall information. They realized that people coming to their web site for information on the recall and they would be best served by getting people the information they need.
Godaddy by contrast, never mentioned their outage on their web site or Bob Parson’s highly visible blog. Changing the content on their home page also meant they were reducing the calls incoming to their switchboard.
In a call to the company headquarters (905.826.3870), they address the issue of the pet food recall immediately and give a toll free number (866-895-2708) for dealing with the issue. Once again this can be contrasted with the Godaddy response which was to simply ignore the issue.
Let’s take a look at how they handled the initial stages of the crisis. Here’s the Menu Foods press release from March 16, 2007

Menu Foods Income Fund (the “Fund”) (TSX:MEW.UN) today announced the precautionary recall of a portion of the dog and cat food it manufactured between December 3, 2006 and March 6, 2007. The recall is limited to “cuts and gravy” style pet food in cans and pouches manufactured at two of the Fund’s United States facilities. These products are both manufactured and sold under private-label and are contract-manufactured for some national brands.
Over the past several days, the Fund has received feedback in the United States (none in Canada) raising concerns about pet food manufactured since early December, and its impact on the renal health of the pets consuming the products. Shortly after receipt of the first complaint, the Fund initiated a substantial battery of technical tests, conducted by both internal and external specialists, but has failed to identify any issues with the products in question. The Fund has, however, discovered that timing of the production associated with these complaints, coincides with the introduction of an ingredient from a new supplier. The Fund stopped using this ingredient shortly after this discovery and production since then has been undertaken using ingredients from another source.
At the same time, the Fund’s largest customer, for which it manufactures on a contract basis, received a small number of consumer complaints and has initiated its own recall. Furthermore, for the time being, the customer has put future orders for cuts and gravy products on hold. This customer’s cuts and gravy purchases in 2006 represented approximately 11% of the Fund’s annual revenue.
“We take these complaints very seriously and, while we are still looking for a specific cause, we are acting to err on the side of caution” said Paul K. Henderson, President and CEO, Menu Foods. “We will do whatever is necessary to ensure that our products maintain the very highest quality standards.”
While the number of complaints has been relatively small, Menu is taking this proactive step out of an abundance of caution, because the health and well-being of pets is paramount to the Fund.

Even thought Menu Foods doesn’t know the exact nature of the problem yet, it’s clear they were able to correlate to bringing in a new supplier. Furthermore note the specificity of the response from Menu Foods. They cite a specific response to the crisis namely discontinuing of the supplier and bringing in a new one. Contrast that response with the total lack of specificity in the Godaddy response to their own crisis, “Our Internet Security and Network teams will continue to analyze and assess the nature of today’s attacks and their characteristics to identify additional defense mechanisms that can be used in the ongoing efforts of Internet Security.”

YAWN. So the security team is going to do security stuff. It lacks sufficient details to assuage customers that the problem won’t reoccur. Furthermore it states the obvious. Of course your security team is going to do security analysis – that’s what your security team does.

Menu Foods also has kept the public informed in every stage of the process. When the problem with the pet food was eventually discovered, they held a press conference to inform the public. Here’s the opening statement at press conference on March 23, 2007. It’s important to highlight one very important fact. At that point in the crisis Menu Foods had spoken with 200,000 consumers about the issue. These aren’t Menu Foods Customers but their customer’s customers. Menu Foods just a few days before wouldn’t have had the mechanism for communicating with 200,000 customers, yet they responded to the crisis at hand. They then took questions for another twenty minutes on the issue.

Menu Foods was kept the public informed as developments in the situation continued. For example their latest release was dated Saturday March 24th, 2007 when they decided recall ALL WET FOOD as a precautionary measure.

“As a result of reports from the FDA and various media outlets that some recalled lots of “cuts and gravy” style wet pet food remain on store shelves, Menu Foods has asked all retail outlets to immediately remove all impacted varieties of wet pet food posted at www.menufoods.com , regardless of the date code. Menu Foods remains concerned that consumers are able to purchase recalled items. There is no known risk from items not listed on the recall list but an abundance of caution is called for in this situation. FDA has been apprised of this action.”

So what are the lessons we can learn from these two completely different approaches to crisis management?

Get in front of the problem
Proactive crisis management helps you retain your customers. Attempting to hide, dismiss or simply not talk about the problem won’t work. For example Godaddy’s hosting platform lacks the most basic tools for communication – a network status indicator where their customers can check on the status of various network services.

Give a customer your action plan for correcting the problem
Sometimes you cannot give the complete details of a problem or an exploit for security reasons during a crisis. That said you should be as specific as possible when communicating to the public. Simply saying “Our security team is working” is insufficient and quite frankly obvious. During an outage your security team certainly isn’t taking a vacation. Be as specific as possible and outline specific corrective courses of action.

Keep your customer service reps informed
There simply isn’t any excuse for not having your customer service reps informed of the problem and informed on what is going on. Menu Foods doesn’t have the infrastructure for speaking with 200,000 customers yet they able to respond quickly enough to do so. Godaddy has a much larger customer service footprint, yet they were incapable of keeping their CSR and sales reps informed.

Use the web. That’s what it’s for
Presumably Godaddy has sophisticated web presence, yet they seemed unable to utilize this platform for keeping their customers informed of the problem. Menu Foods just by updating their site when they have new information has kept their customers and consumers up to date on the issue.

Be honest and upfront about the issue
Wishing away customer perceptions about problems won’t work. Being evasive or in the case of Godaddy won’t fix customer perceptions that you don’t actually care about the customer. I spend something like $7,000 a year with Godaddy and I got no response on the issue.

Ultimately it’s my suspicion that Godaddy’s failed crisis management policies failed because of the culture of Godaddy, a culture that stems from Bob Parsons’ personal foibles. His complete lack of response in addressing the issue with this latest outage ( Here’s one that happened in 2005) is from Bob Parsons’ particular attitude. I suspect that it was this same set of personal foibles that lead Bob to withdraw Godaddy’s public offering. Bob Parsons runs the company like his personal fiefdom. Godaddy grew the business by being the cheapest registrar on the market, not by being the best registrar on the market. Perhaps this market positioning is why culturally Godaddy seems ill suited to handle moving up market into such value added areas such as hosting. It certainly would explain why for example they refused to offer such industry standard tools like Cpanel or Fantastico to their customers and instead have rolled their own control panel.

While Godaddy has come out this crisis with the loss of just a few customers, if they handle the next crisis the same way, it will certainly lead to continued customer loss. Godaddy could learn a thing or three from Menu Foods in terms of crisis management.

Update 3/272007
After reading thispost , it’s apparent that Menu Foods response to their supply chain was inefficient and incorrect. While Menu Foods did a good job notifying the public, they didn’t have the plan with corrective action in place for their supply chain. I also imagine that once they went public with this information, their response got overwhelmed with the response from the public. This highlights a serious issue – inform your business partners FIRST and have a clear plan for dealing with the issue with them. The above post highlights that floor people at Petcetera were recommending food that was potentially tainted.