SUMMARY: moving racks of equipment from datacenter A to datacenter B

All,

So our move has happenned between Feb 23rd and Feb 24th. It didn't go that
smoothly. It was supposed to start at 12am and be finished by 8am. We were
partially up by 2pm, fully operational by 4pm, and still had some
backend/local network issues to work out that lasted for another day
(after we got some sleep)

The plan was to start backups at midnight of databases, the rest was
easily recoverable from other sources. While backups were occurring some
of the equipment was to be shutdown and packed and 'vanned'. At
about 1:30 the rest of the equipment was to be packed. At 4:30 in the
morning we were supposed to be at the other datacenter to start brining
things up in a certain order, so that all services would be up.

At 5am we were already running about an hour late, which wasn't really
that big of a deal, until the loading dock slammed my finger, split it
wide open and crushed a bone. We called for backup (another guy, there
were 4 of us packing servers) and I went to the hospital. At 8am I was
getting out of cab and arriving at the new datacenter. The 'core' (main
web site) boxes were coming up as per plan.. the database server was not
nearly being brought up yet.

For the person recommending a van instead of a truck: you kinda caused me
an injury and caused us an extra 30-60 mins of moving equipment. We
could've just rolled in wood pellets into the truck. To be fair, we didn't
know wood pelets were available, nor did I inspect the loading dock
enough.

Other things that went wrong, and delays they caused:

The datacenter contract was only ready at the end of Dec (mine/vendor
fault), mgt added another 2.5 weeks of 'chewing' over it. This caused
precious time, and hasn't allowed us to bring more of the new boxes
'online' and into production mode prior to the move, which added about 3
hours for moving 'extra' boxes, which included the e450, whose bolts we
had to drill out of the rack to get it out.

The injury caused about 1-1.5hr delay

Our Managed Hosting partner in DC did not properly pre-cable all racks,
which caused us a major headache with the network setup of a few dell
boxes that were plugged into the network early (8am) but weren't fully
operational until 4pm. This also added at least 1-2hrs for cabling.

Damn dell and/or redhat. The boxes from OS looked like they had 2 network
cards plumbed, while in reality only 1 (eth0) was plugged in, and either
dell or redhat was doing some kind of weird redirection in the background.
This took us a while to figure out, and was caused by poor pre-networking
of the racks, as well as an oversight on the network diagrams/poor
documentation. The solution was to unplug the second network card. This
caused a good 8hr delay on these boxes, and intereferred with setting up
the rest.

The equipment lease was also signed late, only Jan 30th, this has also
delayed things up significantly (we had to rush things, and didn't have
enough time to test and/or configure additional equipment).

The amount of time it would take to disassemble and re-assemble EMC was
underestimated.

Last problem was the datacenter that took over a month if giving us a feed
into the racks, and in the end didn't give us an appropriate one (100M
instead of gigE). This gave us network and IPs only a day prior to the
move, which as you can imagine limited our testing and gave us a lot to do
in the last day, which limited my ability to check (e.g. wiring) that
everything was in place.

Unfortunately the move could not have been postponed, since our datacenter
was going out of business and gave us a firm 'get-out' date.

Only one thing broke during the move: a disk in the e450, which was a
mirror of the root disk, and which consequently took us a while to bring
up (it would refuse to boot until we removed the disk).

Due to all circumstances, I think the move went OK. There were a few
under-deliverances and oversights by all stakeholders including myself
which if didn't happen, could've kept the move within or very close to
deadlines.

-Andrey

  


Tips for 'pre-move' activities from various Sun Managers:

>From Stuart:

TIP before you shutdown anything take a full ufsdump to a third server.
Plus, if you need to re-IP your servers quickest way is to run
sys-unconfig before shutdown.

Also healthcheck the servers beforehand i.e. disks corrected memory errors
[even run explorers and send them off to sun].

Get sun to have spares on-site just in case

form a team just for this move based on the following skills

networks
cable management
power management
space planning
unix admin's
storage guru's


>From Stuart:
You may also want to consider pre-building the network and san.


>From David:

Almost any commercial moving firm (Mayflower, Allied Van, Kloke, etc.)
has a section that specializes in moving electronics equipment.

I'd check the yellow pages, contact at least three and get bids, along
with their sales pitch on how they would do it.

Northrop Grumman does a lot of IT service work, which probably includes
moving support.

from ian:

I would be staying away from the rags/blankets/boxes idea as much as
possible. If you are fortunate enough to own the racks that these items
are located in, and
the racks are portable (eg Sun Rack 900, or Sun StorEdge Expansion
Cabinet) then I would leave the systems racked.

Make sure you follow a reasonable process of bringing the servers down,
let them climatise to the 'switched off environement' (eg 15 minutes when
the OS off - ok
prompt - but fans running, then turn off and let sit for a further 30
minutes to 1 hour), then the common thing is to wrap the servers in the
rack in plastic wrap
to avoid rain damage. Transporting the systems in the racks is the best
option.

Once you are at the new site, you should leave the systems in place for 12
- 24 hours before power up. Let them climatise to the new environment,
which will
involve graduate temperature and humidity climatisation. Once this period
is over you would then be safe to run these up in full diag mode, log the
output for
errors, and then bring up the OS into Single User mode - sanity check
everything, and then bring it all the way up.

I have done this many times, and the only failures i've had were 2 disks
in two seperate E450's.

I guess it depends on the age of the disk array, but you shouldn't have
too much of a problem - make sure you have a full backup of every system
(or two) and
depending on how paranoid you are, transport these seperately to the
moving servers.

>From Lajber:

Our supplier sent cx700 with a standar 7.5t truck. In first two month, we
changed one storage proc, and four disk.

The heavy-duty truck suspension is tuned to spring under sever-ton load.
If the equipment less than this, there no real spring -> lot of
acceleration hit the sensitive parts. And they not designed like a laptop
hdd.

I suggest you to use a van or a pickup with some cover better
suited for this.

Or put everithing in paper boxes with lot of acceleration-absorbing thing
(foam, etc).

(my original Ms.C. is mech. engineer, specialized on  vehicles :)



There were a couple of 'professional movers' recommendations:

http://www.maintech.com
http://www.sun.com/service/sunmoves/
www.officemovers.com


Some advice on 'sun' (link above)
If money is no object, "Sun Moves" (Sun itself) will do the move, and
the nifty thing is, you get 2-4 hours parts delivery onsite onsite if
they break anything!



Original post:
We've been forced to move from one datacenter to another, due to
datacenter's closure. (for those curious, Intel/Savvis in DC area)

The datacenter we are moving to is actually nearby, within 10 miles or so.

We have used this opportunity to refresh some of our equipment, so about
half the equipment is going to stay behind. Still, some (rather
heavy) servers are going to come with us. They include 1xe450, a pair of
e250's, 3 or 4 e420's, 3 v40z'a, D1000 and an EMC CX500.

Right now the plan is to get some uhaul truck, lots of
rags/blankets/boxes, and just hope for the best. My main worry is the disk
array.

Can anyone recommend anything (including some specialized movers that'd do
it at night). We are in the DC Dulles airport area.

Thanks,
Andrey
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers