Shipping Delay Recap

Hi, I’m Mike Osier, head of IT Operations at Netflix. My team is responsible for the technology supporting our shipping, Web site, and streaming to PCs and TVs. Now that things are back to normal following last week’s shipping outage, I’d like to shed some light about what happened, why, and what we’re doing about it. On Monday, 8/11, our monitors flagged a database corruption event in our shipping system. Over the course of the day, we began experiencing similar problems in peripheral databases until our shipping system went down. It was going to be a long night.

We suspected hardware and moved the shipping system to an isolated environment, gradually getting DVD shipments moving again. Eventually the system was repaired and shipping returned to normal conditions. With some great forensic help from our vendors, root cause was identified as a key faulty hardware component. It definitively caused the problem yet reported no detectable errors. We’ve taken steps to fortify our shipping system with the acquisition of additional equipment and worked with our vendors to verify we’re in good shape elsewhere.

Hope this was helpful and thanks for your patience.