Hi, I’m Mike Osier, head of IT Operations at Netflix. My team is responsible for the technology supporting our shipping, Web site, and streaming to PCs and TVs. Now that things are back to normal following last week’s shipping outage, I’d like to shed some light about what happened, why, and what we’re doing about it. On Monday, 8/11, our monitors flagged a database corruption event in our shipping system. Over the course of the day, we began experiencing similar problems in peripheral databases until our shipping system went down. It was going to be a long night.
We suspected hardware and moved the shipping system to an isolated environment, gradually getting DVD shipments moving again. Eventually the system was repaired and shipping returned to normal conditions. With some great forensic help from our vendors, root cause was identified as a key faulty hardware component. It definitively caused the problem yet reported no detectable errors. We’ve taken steps to fortify our shipping system with the acquisition of additional equipment and worked with our vendors to verify we’re in good shape elsewhere.
Hope this was helpful and thanks for your patience.
Friday, August 15, 2008
It’s been a long and challenging week, but it ends on a positive note. Today we shipped from all 55 Netflix distribution centers across the U.S. More than three million DVDs went into the mail, including roughly 95 percent of orders backlogged from Tuesday through Thursday. And the balance of backlogged orders will mail on Saturday. Importantly, all of us at Netflix want to offer special thanks to the scores of members who called, emailed and posted words of support at a tough and humbling moment for our company. Apologies to all once again and thanks for hanging in there with us.
Posted by Steve at 7:59 PM
Let’s get right to it. This morning, we’re happy to report that all of our shipping centers are resuming normal operations after experiencing three days of significant issues. Throughout the night and as we post this update, our distribution centers are processing customer orders and getting them into the mail. If a member should have been shipped a disc Tuesday, Wednesday or Thursday, with rare exception it will ship today (Friday). As a result, millions of our members will receive DVDs on Saturday, in time (we hope) for some weekend movie enjoyment. But we know that’s not enough. The service interruption we experienced this week inconvenienced many of our members and put us in the awful position of disappointing people. So we’re taking added measures. For all members whose shipments have been delayed, we’ll be automatically applying a 15% credit to their next billing statement. Or, for new members whose first shipments have been delayed, we’ll automatically extend their free trial by a week. Rest assured that we are taking aggressive steps to fully understand the root cause of this week’s problems and safeguard against issues like these in the future. For now, we’re just happy to be able to get back to doing what we love most – delighting our customers. Again, most sincere apologies and thanks for your understanding.
Posted by Steve at 7:25 AM
Thursday, August 14, 2008
By late this afternoon, we’d made enough progress on our system issues to begin shipping again from some of our distribution centers. We hope to bring the rest of our facilities back online overnight and be shipping from all of our distribution centers on Friday. But the issues we’ve faced over the last several days have been significant and there’s no guarantee at this point that our shipping operations will be fully restored by tomorrow. We’ll again be working throughout the night and will update you in the morning. Thanks for now and continued apologies.
Posted by Steve at 4:50 PM
Hi, it’s Andy again. Unfortunately, we continue to experience significant shipping issues as of Thursday morning. We were able to ship some DVDs from about half of our distribution centers yesterday but we haven’t yet been able to resume shipping this morning. Our engineers continue to work around the clock to restore normal operations. In the meantime, we’re notifying affected customers via personal email and we’ve posted a notice on the Netflix Web site. We’re as frustrated about this as you are and we once again apologize for the inconvenience. In any case, I’ll update you again later today. Thanks for your patience.
Posted by Steve at 8:19 AM
Wednesday, August 13, 2008
Hi, it’s Andy Rendich back with an update. As of this morning, some (but not all) of our distribution centers are back up and shipping DVDs. Our goal is to resume normal shipping from all of our facilities as soon as possible, but we are still experiencing technical issues with our shipping systems. We apologize again for the inconvenience and continue to appreciate your understanding. We’ll keep you updated as there’s more to say. Thanks for now.
Posted by Steve at 10:52 AM
Tuesday, August 12, 2008
Hi, Andy Rendich here. I head operations for Netflix, which consists of shipping DVDs to our members. We received and were able to process incoming DVDs this morning but, due to a technology issue, we weren't able to send emails confirming DVD receipt and we won’t ship any DVDs today (Tuesday). Our goal is to resume shipping tomorrow (Wednesday). Members who should have been shipped a DVD today will automatically receive a credit to their accounts, which we will communicate in personal emails. Our apologies for the delay.
Posted by Steve at 5:10 PM