Matthew Marlowe's Blog

If you are running ipv4 only network with local nameservers and starting to see problems with ipv6 DNS

2010-07-21T12:55:44Z

Problem is caused by someone or some committee deciding that "gosh darn it! we can make ipv6 adoption go much faster if we bug the hell out of the ipv4 only network admins".

So they did......

On latest versions of RHEL, not only do you get ipv6 enabled by default, all dns requests go to ipv6 first, even if think you have told server it should process ipv4 only That means that if a particular domain has a large set of nameservers, it is possible in certain circumstances for your critical app which is trying to get a simple ipv4 ip will end up waiting while each nameserver is checked one by one for an ipv6 response.....which can take a really really looooooooong time.

Step 1: verifying what is going on -- use dig to query the nameservers directly for AAAA responses (if all the nameservers are responding refused), you got a candidate.

example: dig api-read.facebook.com/AAAA/IN @glb01.sf2p.tfbnw.net

Step 2: use "getent ahostsv4 hostname" and "getent ahostsv6 hostname" to confirm it is the ipv6 queries that are causing the problem.

You may also see something like the following in named (even though you have forwarding disabled):

named[2837]: FORMERR resolving 'api-read.facebook.com/AAAA/IN': 69.63.176.101#53

Step 3: redhat does provide a few solutions here (thankfully), but it is not as straightforward as it should be.

I assume you already have NETWORKING_IPV6=no in /etc/sysconfig/network.

You may even have been smart and put the following /etc/modprobe.conf:

alias net-pf-10 off
alias ipv6 off

And, of course, you've turned off all ipv6 daemons including iptables6.

No, that is not enough, it is very easy for ipv6 to be forced to load, especially in RHEL 5.5. You probably also need to create the file /etc/sysconfig/disable-ipv6 with the contents:

options ipv6 disable=1

step 4: pretend you're running a windows box that needs its daily reboot, go ahead.

When system comes back up, do lsmod | grep ipv6 ..and the ipv6 module should not be loaded.

Run getent ahosts hostname and see ipv4 only results come up very fast.

If you're still seeing problems, review lsmod to see what is forcing ipv6 to load and you may need to perform magic on your dns servers too -- such as adding to /etc/named.conf:

listen-on-v6 { none; };

Adding "-4" to the options field in /etc/sysconfig/named.

Restart named.

Performing steps above to kill ipv6 completely on the local resolving dns servers.

Note: RedHat has their own technote about the problem here at: http://kbase.redhat.com/faq/docs/DOC-17904 .

I suspect this is more specific to RHEL 5.4 than 5.5 RHEL 5.5 does seem to do the right thing when you go through the numerous hoops to disable the ipv6 module.

DeployLinux DNS Configuration - May 2010

2010-05-22T06:48:37Z

The primary authoritive DNS nameserver for our client domains is:

     thepostman.deploylinux.net

This is because unlike everyone else, I thought thepostman was a good movie.

Our secondary nameservers are:

     consul.deploylinux.net

     centurion.deploylinux.net

   ns5.dnsmadeeasy.com

   ns6.dnsmadeeasy.com

   ns7.dnsmadeeasy.com

Thepostman, consul, and centurion are located within our San Diego datacenter and also provide responses to recursive dns queries made from virtual machines.

The three dnsmadeeasy name servers are geographically distributed and built to handle significant DoS attacks.     The combination of both local and distributed servers allows us to maximize reliability w/o depending on any single platform.

To migrate a domain to DeployLinux hosted DNS, clients should:
     a) first open a ticket within our support portal at https://support.deploylinux.net/
     b) DeployLinux techs will verify that a new zone file is created for your domain and that any prior records are transfered. If you desire, we may also migrate your domain to our domain registry and handle all future domain renewals on your behalf.
     c) Update their domain register to point domains to the nameservers above.

Future changes to DNS should also be handled by opening up support tickets.

DeployLinux does also provide domain registration services at a flat rate of $15/yr/domain. We realize this is a little more expensive than $7.99 at godaddy/etc, but in our experience the cheaper domain registration vendors tend to have reliability issues which result in at least a few hours/yr of domain downtime. They may also have limits of how popular your site can get and how many dns responses they will process.   The $15/yr flat rate allows us to host the largest sites w/o any additional charges and with personal support/service.

Our hosting clients typically receive a limited number of free domain registrations and DNS hosting w/ each VM. Note that information above is specific to .com domains.    Other domain extensions may have different handling and rates.

Thoughts on Dell R810/R910 announcements

2010-03-30T20:58:20Z

Good:
R810 2U form factor w/ 32 possible cores (4 x 8) is extremely compelling
512GB max ram across 32 dimm slots in 2U ..wow. nice
Will probably start using ram raid in future
Overall design of the R910 seems amazing, very tempting
16 SAS drives in the R910 great
10 pcie slots in R910 ideal

Bad:
2 broadcom gigE in R810 annoying, should have been intel dual 10 gigE.
H200 default raid controller -- does not support non dell disks??
lack of info on how the 2->4 cpu socket occurs on R810, any limitations?
need more info on cooling of R810 in maxed out config
prior history suggests 4U are impossible to rackmount w/ 1 person
considering going 2U box only going forward due to safety/per-unit cost reasons
1100W power supplies :(
Looking for info on weight of new boxes...

Unsure:
1TB ram for R910??

Trying to get rid of a bunch of miscellaneous used cisco, datacenter, and office equipment

2010-02-09T00:15:22Z

We have a moderate amount of gear that was removed from production at our datacenter or corporate office over the last few years that has been waiting in boxes to be sold when he had time to get around to it. I'll be putting together an exhaustive list over the next few weeks, but here are some items to start with:

Genuine Cisco VWIC2-2MFT-T1, Qty 2
Genuine Cisco 2811 Router w/ upgraded ram+flash (See Specs in 2/11 Update Below)
HP ProCurve 2626
Avocent Autoview 2030
Kingston Rackmount Ultra SCSI Storage Array
Snap Server 210 ( 2 * 700GB, GuardianOS 5.2.056)
Snap Server 2200 (2 * 400GB, SnapOS 4.0.860)
Several Adaptec UltraSCSI Adapters
Intel 1000PT Dual Port PCIe GigE NICS Qty 3
Intel Quad Port NICS (ask for model #'s)
Qlogic QLA4052C Dual Port GigE PCI-X Server iSCSI Storage HBA's, Qty 2
Numerous rackmount kits for Dell PowerEdge 2800/2850
Rackmount Avocent LCD + Keyboard
Several APC Business SmartUPS 1000XL units with additional connected external battery packs for extended runtime (note that original batteries in these units exceeded their normal lifetime - buyer will need to procure new ones from APC to run - will be very heavy when shipped.)

There is also a maxed out Genuine Cisco 2821 that may be leaving production in the next few weeks if we can get a new 1941 installed to replace it.

Right now, we're looking for interested buyers and best offers. All HP, Avocent, or Cisco gear was purchased direct from CDW or PCNation with DeployLinux Consulting as the one/only owner.

UPDATE (03/01/2010):

It's been a quiet last two weeks - much more equipment still to list, but I have decided to focus on getting rid of the stuff in the current list before adding more. I would love to be able to sell everything this week one way or another -- trying to avoid the ebay route, but may end up going that way for some of it.
If you have any questions or need pictures/more info, please email me.

UPDATE (02/15/2010):

The highest offer we've received to date on the Cisco Router + Two Dual T-1 HWICS is $2K. This will be the selling price soon if we don't receive any higher offers (feel free to bid on just router or hwics).
Added numerous rackmount kits for Dell PowerEdge 2800/2850 Servers.
Added SmartUPS XL Units
Added Rackmount LCD/Keyboard
Updated Information on Snap Servers
Our desired price for HP switch is $200, but we'll sell for best offer.

UPDATE (02/11/2010):

We received an offer on the 2811 + T-1 VWIC2, but will wait a week to see if we get any higher bids before accepting.
We were asked what we're hoping to get price wise for our equipment and the best answer I could put forward was:

Our goal is to sell everything within the next 30-60 days, which means we will consider all offers. Pricing wise, we expect to receive bids as low as 25% of the current market price for new and as high as 75%. We'll sell to the higher bids immediately and take longer to accept the lower bids, but we do want to sell everything.

Most gear is in good to excellent condition. If you have a question about a particular item, I can give detailed answers or upload photos. In general, gear was taken out of production system sometime in the last 30 days to 2 years and placed in our office storage until we had time to get around to selling it.

UPDATE (2/10/2010):

Updating LinkedIn and other places where notice was sent to point here. All future updates to 'sell list' will occur here. I will keep list constantly updated whenever items are added/removed/questions answered/etc.
Specs gathered for Cisco 2811
Added Qlogic iSCSI HBA's to Sell List
Other minor edits to list

Cisco 2811 Details:

Appears to be a Cisco2821-HSEC-K9 w/ 786MB RAM + HWIC-4ESW Installed. Unit seems to be in good condition. I noticed 1 minor scratch in top paint and fan is loud, but I guess that is normal for 2811's. Configuration is not wiped, doing that now.

System Bootstrap, Version 12.3(8r)T7, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2004 by cisco Systems, Inc.

Initializing memory for ECC
...
c2811 processor with 786432 Kbytes of main memory
Main memory is configured to 64 bit mode with ECC enabled


Readonly ROMMON initialized
program load complete, entry point: 0x8000f000, size: 0xc5a0

Initializing ATA monitor library.......

program load complete, entry point: 0x8000f000, size: 0x231985c
Self decompressing the image : ####################################################################################################################################################################]

Smart Init is disabled. IOMEM set to: 5

Using iomem percentage: 5

              Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706

Cisco IOS Software, 2800 Software (C2800NM-ADVIPSERVICESK9-M), Version 12.4(10a), RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2006 by Cisco Systems, Inc.
Compiled Wed 11-Oct-06 19:01 by prod_rel_team
Image text-base: 0x40093180, data-base: 0x42B00000

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 2811 (revision 50.46) with 747520K/38912K bytes of memory.
Processor board ID FTX0902D42F
6 FastEthernet interfaces
2 Virtual Private Network (VPN) Modules
DRAM configuration is 64 bits wide with parity enabled.
239K bytes of non-volatile configuration memory.
251904K bytes of ATA CompactFlash (Read/Write)

cisco>show inventory
NAME: "2811 chassis", DESCR: "2811 chassis"
PID: CISCO2811         , VID:   NA, SN: FTX0902D42F

NAME: "4 Port FE Switch", DESCR: "4 Port FE Switch"
PID: HWIC-4ESW         , VID: VN/A, SN: FOC1040406C

NAME: "Virtual Private Network (VPN) Module", DESCR: "Encryption AIM Element"
PID: AIM-VPN/EPII-PLUS , VID:   NA, SN: FOC09024B38

Building a Durable Home Office Linux Workstation - YR 2010 Edition

2010-01-26T01:44:16Z

There are truly dozens if not hundreds of requirements and limitations that come into place when designing and building a long lasting and durable Linux Workstation for the home office.

The overall goal is to have a system that will be reliable, constantly upgradeable, and high value. Existing hardware MUST be incorporated where possible.

Here are some of the decisions I made:

Equipment binges in the 90's and 2000's resulted in an excess of external equipment (low cost home nas, Firewire/USB attached drives, USB Hubs, video cameras, etc). While the ability to add things externally makes incremental expansion easy, it does so at the cost of complexity, reliability, and overall noise/aesthetics - not to mention power costs which in San Diego are quite high. So, my rule going forward was to put everything in a large full tower ATX case.
Supporting multiple operating systems is no longer as important. In the way, way, way past, I used to dual boot between Linux and Windows. Later, I experimented with multiple monitors dedicated to each either running in virtual machines or separate boxes (sharing keyboard/mouse). Then, as hardware got more microsoft dependent, I ran windows as an OS host and had a RedHat Enterprise Linux VM bootup in windowed mode for most day-to-day work. We are at the point though, where there are only a few applications that can not be fully ported to Linux (Quickbooks + VMware Virtual Center Client + FrameMaker/Acrobat Extended/Visio + ). Therefore, the default now is to make Linux the host and access applications via unity mode of vmware workstation software.
I agonized over what Linux Distribution to depend on for the next many years and finally decided that it needed to be Fedora 12. RedHat Enterprise Linux is by far the predominant OS installed in the datacenters I manage and being able to easily access development versions of new software for RHEL in my off hours is critical. Ubuntu may be very popular now, but I didn't see it being the right fit here. Furthermore, RHEL Workstation itself while being more stable than fedora just didn't provide easy access to recent versions of desktop/home office applications I may need or want. As a former gentoo developer, I'd have installed Gentoo but honestly I just couldn't afford to deal with managing a constant flood of manual security issues and qa testing that using it would have required.
All data must be protected by RAID, and regularly backed up in a low noise/maintenance manner w/ the ability to occasionally perform off site backups.
I want lots of storage, ability to burn blue ray discs, and excellent graphics.

So, I grabbed the best full tower case + motherboard/cpu/ram combination I currently had in the house and then add any other hardware components I could find. And, only if absolutely necessary, ordering specific hardware from amazon. The idea was not to build the perfect system right away but to have a base to grow over the next 10 years.

At the moment, I have mostly running in production state:

Thermaltake Kandalf Full ATX Tower Case
Dual Core P4 3.4Ghz CPU
8GB ECC DDR2 RAM w/ 800 Mhz FSB
Asus Workstation Motherboard based on i975x chipset (P5WDG2-WS)
MB includes ~8 SATA Ports, 2 x PCIe, 2 x PCI-x and 2 PCI, Intel HD Sound
Dual Sky2 GigE Nics, Firewire, and lots of USB2 ports on MB
Nvidia GeoForce 8800 GTS 512 w/ Two Dual Link DVI Ports
Dell 30" LCD Monitor
VF0560 Live! Cam Optia AF Video Camera
Logitech Z520 Speaker System
3Ware 9550SX 8 Port SATA-II RAID Controller w/ 128MB Battery Backed Cache
5 x Seagate 7200RPM 250GB SATA-II Drives in RAID5 for /home filesystem
1 x Seagate 7200RPM 250GB SATA-II Drive as Hot Spare
2 x Western Digital 2TB "Black Cavier" Hard Drives in RAID1 for all other filesystems
1 x ReadyNAS NV+ NAS w/ 4 x 750GB SATA-II in RAID5 for Daily Incremental Bkups
Multiple Internal Hard Drive Cages w/ Integrated Fans for Tower Case
Additional Low-noise directed case fans and BIOS tuning for stability/performance
Plextor PX-Q840U External USB DVD/CD Burner for easy access to drive
Plextor PX-B940SA Internal SATA Blu-ray Burner for creating offsite backups
Honeywell Firesafe next to system for safe keeping of critical media

Desktop configured for KDE because KDE makes inspiring apps and the gnome equivalents are boring and life draining.

Thoughts on VMware's acquisition of Zimbra

2010-01-13T05:09:33Z

For the last decade, I have been of two minds about VMware. My roots are in open-source but I keep buying expensive proprietary software licenses for VMware ESX and vSphere because VMware's products are proven and continue to be the driving force of innovation in the virtualization and cloud computing space.

This is not completely out of tune with being a Linux evangelist. Linus Torvalds himself has said he nothing against commercial software, and that businesses should be pragmatic when choosing which software to deploy.

But, how the soul tears when a new CEO of VMware arrives from no less than Microsoft. Is this just the first prong in a multi-staged cultural invasion of sales/marketing over technical staff.

I ask myself whether I am really supporting the growth of what will be the Microsoft of the next generation?

Sometimes, one can certainly think so - Short of Amazon EC2, no one and nothing has gotten in the way of VMware's skyrocketing usage and to a certain extent (skyrocketing prices).

Of course, VMware has a wonderful ecosystem of enterprise tools to support deployment of virtualization in business, including stage manager, life cycle manager, etc. However, none of these products are open-source and almost all of them are out of the budget of most internet start-ups that I perform consulting projects with. Yes, I know vmware offers some "free" tools like vmplayer and vmware server, but I do not consider these really production business tools.

Nor is VMware exactly open about its development plans, progress towards fixing bugs, or even really support. Their are some awesome websites, forums, blogs, and irc channels. But these are mostly user run and you dont see the constant back and forth between developers and users that one might normally experience in an open-source project.

And, yet.....VMware acquires Spring Source, Hyperic, and now Zimbra in a relatively short period of time. VMware is not just acquiring open-source companies, they are acquiring the cream of the crop when one narrows the search to focus on open-source innovation in the business application market. I would not be surprised if VMware tried to acquire EnterpriseDB (makers of postgresql plus) next. With the sagging of mysql and resurgence of Oracle, this would be smart.....for VMware.

I am aware that RedHat hopes to take a lot of the virtualization market away from VMware, but I wonder if they playing last years war and thus distracted from the current battle being played out to determine the shape of the future of open source business application market place.

There was palpable excitement in the Linux community throughout the 1990's as we knew that while enterprises move slowly, eventually they must and do deploy the most innovative and reliable software and thus the age of supporting defective closed-built buggy software would eventually come to an end when open-source business applications matured. We are there. It is happening, but how much of that success will be reinvested back into creating new open-source products if the real cash out goal of all developers creating innovative business applications is merely to be acquired by VMware? Will VMware in the business applications space be like Cisco Systems is in Networking - a constant ever growing technical company that purchases anyone that out-innovates it?

At least, this will be a step up from Microsoft.

On the other hand, maybe VMware will surprise me and actually adopt more open development processes and become not just a consumer and beneficiary of open source, not even a major source of patches to open-source projects, but a real evangelist of the philosophy of being transparent, sharing by default, and communicating constantly with clients and customers.

I would hope so.

Apache Bug - RHSA-2009-1075

2009-05-27T21:58:13Z

We will be updating apache software for all managed hosting clients today to resolve the following bug: https://rhn.redhat.com/errata/RHSA-2009-1075.html

Follow us on Twitter

2009-05-27T21:57:08Z

http://twitter.com/deploylinux

Nightly Cloud Maintenance Schedule

2009-02-25T08:43:44Z

San Diego Datacenter:

9pm        - Midnight:    File System and OS Level Backups
Midnight -         4am:    Mysql Backups (Snapshot and/or Full Export)
4am       -          6am:                Vulnerability Scans (Nessus Professional)

Zero Downtime. However, virtual machine resource utilization will fluctuate as each task is initiated.

Daily Updates - 2/24/09

2009-02-24T09:33:58Z

Viso Stencils for VMware Deployments
VMware vCenter Server 2.5 Update 4 Released, Linux Technology Preview also available (not for production)

Vulnerability Scanning Update

2009-02-21T02:54:06Z

Gone: Performing monthly security scans via vendor security appliances and manually reviewing reports in their entirety for every host in the network.

Incoming: Daily security audits initiated via a RedHat Enterprise Linux 5 VM running Nessus Professional software, updated with latest plugins, and reports automatically compared against any previous for each specific ip address. Differences are also automatically prioritized and our sysadmin staff are notified when they first login for the day of any changes. The scan configuration is setup to be comprehensive, but non-destructive or dangerous. Also, all PCI DSS requirements are checked.

Vulnerability scanning will also now include the scanning device logging into a restricted user account in each target VM and running OS, Application, and Database level checks (not just network visible checks).

The scanner will also attempt to download the complete content of all target websites and run as much automated code sanity checks on applications as possible.

In summary, vulnerability scanning technology has advanced substantially and we're going to try to take as much advantage of it as possible -- with the caveat that with vulnerability scanning, hardware and the network is usually not the limiting factor (physical human time to review results is). Any checks that require intensive human review will still need to be performed monthly, but 99% of all checks should now be possible to be done daily.

Note that vulnerability scanning is a free service we provide to all our hosting clients in our San Diego Datacenter. This service would normally itself run from $25-$80/month from prominent security service providers, without being near as complete.

References:

http://www.nessus.org/nessus/
http://www.nessus.org/whitepapers/sec_test_sc3_nessus.pdf
http://www.nessus.org/news/#204
http://www.autonessus.com/home
http://cgi.tenablesecurity.com/demos/PCI_Audit/PCI_audit.htm
http://www.nessus.org/plugins/index.php?view=all
http://www.alertra.com/pricing.php
http://www.mcafeesecure.com/us/pci-intro.jsp

EnableResignature and/or DisallowSnapshotLUN » Yellow Bricks

2008-12-15T14:54:17Z

Excellent reference and a good document to review once a year:

http://www.yellow-bricks.com/2008/12/11/enableresignature-andor-disallowsnapshotlun/

HOWTO: Teaching Hobbit to Send Alerts via SMS using an Internet Skype Gateway and Linux Host

2008-11-07T08:42:23Z

Gone are the days when every sysadmin had to carry a pager because cell phones just didn't cut it. These days, SMS is just about as reliable in most environments and essentially free compared to the cost of a paging service and dealing with a 2nd device.

The hiccup, as nearly everyone knows now, is that you have to find a way to get your monitoring server to talk to your cell phone company -- and, honestly, you're cell phone company does not want to make this easy....they'll try to upgrade you to various unlimited SMS/messaging plans and then have an internet SMTP -> SMS gateway that fails constantly.

I've switched from provider to provider over the last 4-5 years and they are all the same. Alerts may be reliable for a few months on one or the other, but eventually there comes a time when you didn't get the call because the email -> SMS gateway was down, or worse -- delivers the message a day later.

So, screw the cell phone company, let's use Skype:

Steps I Used to Teach The Hobbit Monitor to Use Skype:

Setup Hobbit in a Dedicated Linux Virtual Machine
Ensure Hobbit is running under its own non-root userid (actually, no reason not to just use apache userid since hobbit and apache are going to share a great many files and the entire VM is dedicated to hobbit).
Setup VM to automatically bootup in runlevel 5 and autostart a new gnome session under apache
Install Skype for Linux and the Skype Command Line Tools at: http://www.oberle.org/blog/2007/06/11/sending-sms-with-skype-on-linux/
Do not use the standard init system to startup hobbit. Instead, setup gnome to initiate the Skype Client followed by a hobbit restart at the start of each session. This will ensure that hobbit has full access to the X authentication environment which it needs to communicate with the X-windows skype client.
Create a new script in /usr/local/bin owned by apache that executes the skype command line SMS send tool and passes it the phone number and BB environment variable for the error message itself.
Follow the instructions to hobbit-alert.conf to tell hobbit under what condition it should send SMS alerts
Make sure you setup a dedicated Skype account for the server, use skype out credit, and have it auto deposit another $10 into the account whenever it runs out of funds. Also, modify the Skype client settings to not accept any incoming calls/messages/etc.

Voila!

VMware Releases ESX 3.5U3

2008-11-07T05:31:37Z

Seems mostly to be a bugfix + new hardware support patch, although I did see the following nuggets:

Intel Pro/1000 gigabit Ethernet device drivers (e1000) in some guests allocate MTU bytes for rx buffers, but tell the device the size of the rx buffer is 2048 bytes. If these buffers fall on the edge of the guest physical memory range, the virtual e1000 device could wedge during rx with the following messages in the VMkernel logs:
WARNING: Alloc: ppn=0xc0000 out of range: 0x0-0xc0000 (count=3)
WARNING: P2MCache: GetPhysMemRange failed: PPN 0xc0000 canBlock 0 status Bad parameter.

This patch fixes this problem.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007041

Add experimental support for a new utility, the VMDK Recovery Tool.

More information available at: http://kb.vmware.com/kb/1007243

The official release notes are available at: http://www.vmware.com/support/vi3/doc/vi3_esx35u3_rel_notes.html

RHEL 5.3 Has Entered Beta

2008-11-06T15:22:16Z

Announcement: https://www.redhat.com/archives/rhelv5-announce/2008-October/msg00000.html
Release Notes: http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.3/html/Release_Notes/index.html

I don't see much that is earth-shattering for VMware ESX Server Deployments- but then, this is RHEL, and a .3 release so you wouldn't expect that anyway. The good stuff is probably all in Fedora now and being saved up for RHEL6.

Anyhow, I was just thinking this morning that the rate of security updates for the RHEL 5.2 kernel was getting annoying.... When kernel security notices are as frequent as phpmyadmin ones, you know something is not right.