DeployLinux has received notification from the AIS San Diego Datacenter that emergency maintenance will be performed tonight on several key switches in the datacenter, and therefore those customers which are located within the SDTC may experience up to sixty minutes of downtime starting at 10pm tonight. 

AIS first notification to DeployLinux on this matter was on Friday evening, at which point DeployLinux expressed concern to AIS about the duration of the downtime, shortness of notice, and scheduling.

DeployLinux is currently working with AIS to minimize downtime, including testing of previously deployed BGP and HSRP across the multiple uplinks DeployLinux has installed in the SDTC.   It is hoped we will be able to reduce any downtime to only a few minutes of routing instability.

Customers with questions should feel free to open up tickets or call their Account Managers.  All questions will be treated with high priority.

Our lead consultant and CEO, Matthew Marlowe, had an entry on his blog slashdotted this morning.

Per wikipedia:

 "The Slashdot effect, also known as slashdotting, is the phenomenon of a popular website linking to a smaller site, causing the smaller site to slow down ... due to the increased traffic."

Matthew is an advocate of vmware's virtualization product, ESX, and became aware of a serious bug in vmware's latest release last night that could potientially bring down many business virtual machines w/o warning.  He quickly consolidated information about the bug over several hours and posted a summary to his blog and the #vmware channel at irc.freenode.net where many professional sysadmins work together to solve customer problems in order to limit the fallout and push vmware to release a quicker fix.

Slashdot linked to the blog entry several hours later and over 46,000 visits from ~32,000 visitors for a total of ~364,000 hits have been recorded so far this morning (~first four hours after being linked).

The blog is located at:  http://www.deploylinux.net/matt

DeployLinux upgraded all hosts in the production hosting cluster to ESX 3.5 U2 over the weekend.  

There was no user-noticeable downtime, or significant issues.

U2 is more of a maintenance release and focuses on bug fixes, but there were some minor feature additions.  From our perspective, the most interesting is the ability to fine-tune the HA algorithm to focus more on receiving the heartbeat of individual VM's than hosts....as we've certainly seen cases where an occasional Linux Kernel will lock up.  In these cases, we might be able to have VMware automatically restart the VM within minutes rather than waiting for the monitoring system to mark the host down and page our staff who who would investigate the problem and force a manual reboot(e.g. VM restarts in 2 minutes rather than 15?).  

We're not 100% convinced that we want VMware automatically restarting VM's as that might slightly impact our ability to track down the cause of a crash, but this may be desired for the most downtime sensitive VM's (e.g. databases).   On the other hand, if a database crashed ...there are reasons to want it to restart by hand.  There are always trade offs.

In any case, U2 also included several updates to the Virtual Center management application and dozens of bug fixes to ESX itself -- all of which we are very grateful for.