Forum Home
PC World Chat
 
Thread ID: 103929 2009-10-11 02:04:00 AirNZ + DR somebody (208) PC World Chat
Post ID Timestamp Content User
819159 2009-10-13 06:40:00 they where doing UPS maintenance and bypassed the UPS and had the data centre on generator power.

the generator died (about 9:20am), the servers died.
power was restored quite quickly (about 9:30am)

clean up of the servers took longer than expected and so management called for DR about 10:30am

prod clean-up took till 10pm Sunday, not sure when they switch back from DR.


FYI, I don't work for air-NZ or IBM, but I do have friends in the right places to get an overview of what went wrong.

Thanks for the update - it's good to know that they actually have a DR environment.
somebody (208)
819160 2009-10-13 07:05:00 they where doing UPS maintenance and bypassed the UPS and had the data centre on generator power.

the generator died (about 9:20am), the servers died.
power was restored quite quickly (about 9:30am)

clean up of the servers took longer than expected and so management called for DR about 10:30am

prod clean-up took till 10pm Sunday, not sure when they switch back from DR.


FYI, I don't work for air-NZ or IBM, but I do have friends in the right places to get an overview of what went wrong.

Their switching circuitry should have an allowance for bypassing the UPS whilst remaining on a mains feed; that way you can cut the UPS out for a few minutes without having to boot your genny. Having both options is preferable though, particularly if you have a lumpy mains feed.

If they opted to run off the genny it must've been substantial maintenance!
nofam (9009)
1 2 3