My server (rickk.com) is actually two servers. One is the primary and the other is the backup, and if the primary goes down the backup automatically takes over and keeps the web sites and email running. The servers are on different providers in different locations for redundancy.
When I switched providers for the backup server from the new defunct Xelhosting to Slicehost, I ended up in the curious position of having the backup server be quite a bit faster than the primary server. Also, the backup runs a newer version of Linux.
So the day came. Today I swapped the primary and backup server functions. So far so good.
Things like web and IMAP servers always run on both servers for backup purposes so that was easy.
DNS entries needed to be swapped but that was not a big deal because that's what the auto-swap script does. When the backup can no longer see the primary, it swaps the DNS entries for the servers to point to the backup server. I just swapped the default.
A bunch of scripts used for synchronizing the servers needed to be swapped from one server to the other. No big deal.
Swapping the SQL server master and slave. NOT TRIVIAL. As it turns out, my SQL database replication was not actually working, and I nearly lost 6 months of blog posts. And I have to say, I think I might have become a little bandwidth spoiled. While I was transferring 115 MB database backup files between servers, I was thinking "this is taking a really long time!" Well, it kind of was (1:41), but I really was achieving 10 Mbits/sec between the hosts. I remember the days of 10-BASE-T where you wouldn't actually get that much throughput between hosts on the same LAN a hundred feet apart, let alone transmitting that data from Dallas to San Francisco over the public Internet!
