IT @ A Profitable Dot Com

You are at the end of hallway which extends back to the North. There is an unlocked door to the West, and a locked door to the East. There is a cart visible through the glass in the door to the West.

>East

The door is locked! You need a badge and a keycode to open the door.

>Swipe badge, enter 1-2-3-4 on keypad

The door beeps and unlocks.

>East

You proceed through the door and are now in what appears to be a datacenter placed in an old, concrete bunker. It is very cold, starkly lit by old flourscent lamps, and extremely loud. A small staircase leads down to the West. Four rows of white metal racks, spaced as if this were a bookstore or library, run north to south inside the room. There are 9 other people here; it appears as if they were waiting for you.

>Look people

The other people are dressed similarly to you: shorts, T-shirts, athletic shoes. All look like they are more comfortable seated in a chair than moving about. Only one or two of them look remotely athletic.

>Look racks

The white metal racks are familiar looking, only they are spaced closer than you’ve ever seen racks placed, leaving barely enough room for two people to pass in the aisles. The closest rack to you is labelled A01.

>Open rack door

The metal door opens, revealing the razor sharp rack standards, finger pinching rack rails, and 10 HP DL585s. With the door open, the sound level increases slightly, as if the invisble jet in the room is now taxiing.

>Remove DL585

There is not enough room to do that.

>Slide DL585 out of rack

The 585 slides out of the rack, attached to its rails.

>Release rails

The rails release, but there is not enough room to remove the server.

>Push rails and server back into rack

You carefully push the rails and server back into the rack, pinching your finger on the rails. A drop of blood is visible on your index finger.

>Pull server out without letting rails slide out

The 90# server comes out of the rack. Your fingers are already tired.

>Carry server down the stairs and place on waiting cart

You carry the server outside the data center and place it on the cart. It is remarkably peaceful and pleasantly warm out here.

>Define macro remove-server: (Remove server as above.)

Macro defined.

>Define macro move-server: (Push the cart 90 yards, down twisty passages, through 4 doors, find the server location in the new DC on the printouts, re-rack server, re-cable server, push cart 90 yards, down twisty passages, back to the old datacenter)

Macro defined.

>Repeat macros (remove-server move-server) for all 20 racks

Processing…
Processing……
Processing………

Exceptions noted during processing: “Happy New Years!” messages from numerous friends and family left at home hundreds of miles away during the holidays. (Not many people are trying to order business cards or marketing products as the various balls are dropping to usher in the new year, making this the perfect time to do this move.)

Processing completed. Execution time: just over 10 hours.

Significant damages: To humans: none. To equipment: none.

The equipment moved appears to total about 8 tons and several million dollars worth of gear. It represents the entire VistaPrint website.

>Check www.vistaprint.com

The site is operational and is taking orders at a normal rate for 5 AM on New Years Day. It is dark outside and you are hungry enough to eat a grue.

>Score

Your score is 314 [out of total of 585 points], in 6.022E+23 moves.
This gives you the rank of Grizzled IT Veteran.

That’s how 10 of us ushered in 2009; I hope your New Year’s observance was a little more celebratory, but I’m quite sure it was no more a bonding experience and that per-person, your fellow party-goers didn’t move as much computer equipment!

The New Data Center
We took a behind-the-scenes tour of all the new infrastructure and I have to say that, as much as we complained about the old DC rack spacing and cooling limitations, our colo provider has really built a first-rate DC here. We’re occupying a couple dozen 6kVA racks, though there is support in the DC for 10 and 12 kVA racks. All racks have fully redundant power (A&B 480V/3 feeds each with UPS, and 3 1MW generators supporting the A/B feeds), appropriate inter-row spacing, a hot-aisle/cold-aisle arrangement for airflow, and all racks have supplemental back-of-rack air-to-water heat exchangers to take some of the load off the air-based cooling units.

Next Steps
What’s our next challenge in that DC?

Power is a significant component of our hosting cost and, in the new DC, we’re now paying pass-through rates for our power (each rack has two intelligent metering PDUs). Prior to the move, we were paying a fixed (very high, but fixed) monthly power charge per rack, so there was no incentive to economize. Now that we’re paying based on consumption, we’re going to have to look into intelligently scaling our web and compute farms based on site traffic over the course of days and weeks. This is good for aligning consumption and payments, and giving us incentive to economize is good as a “green” initiative, but it’s another project my group to tackle this year.

More on how we’re planning to do that, and the process we go through to make sure that any page load time slowness isn’t costing us more than the electricity savings in future posts…