DRBD with MySQL on CentOS 6.0 in AWS with automatic failover.

Recently I had to figure out some automatic failover for MySQL. I came to the conclusion that DRBD would be the most effective method to do this, but I had quite a painful time getting it up and running in AWS. I figured for future referrals, or to help you do the same, I would make this blog post. Note that in this scenario we I am using VPC so static IP's have been assigned. I'm sure this could all be replicated in normal EC2

Lets get started. First build out your 2 machines using a base CentOS 6.0 AMI, whichever one you trust (or build your own)

First we need to get the EL Repo

rpm -Uvh http://elrepo.org/elrepo-release-6-4.el6.elrepo.noarch.rpm

After you have this new repository installed, lets disable it so it's not checked except when we want it. You will want to edit /etc/yum.repos.d/elrepo.repo and set

enabled=0

Lets install drbd

yum --enablerepo=elrepo install drbd83-utils kmod-drbd83

Edit the hosts file to include your IP's on both servers

mysql1 10.0.1.100

mysql2 10.0.1.101

On both servers we want to edit /etc/drbd.d/mysql.res to set up the MySQL Resource for DRBD To use

resource mysql {

on mysql1 {

device /dev/drbd0;

disk /dev/xvdf;

address 10.0.1.100:7789;

meta-disk internal;

}



on mysql2 {

device /dev/drbd0;

disk /dev/xvdf;

address 10.0.1.101:7789;

meta-disk internal;

}

}

Attach a new EBS Volume to /dev/sdf on both servers. Pick a size your comfortable with for the amount of growth you expect.

Initialize DRBD on both servers

drbdadm create-md mysql

Start up DRBD. Note: This should be done on both servers at the same time!

/etc/init.d/drbd start

Now we want to create the filesystem for DRBD. This should only be run on mysql1

drbdsetup /dev/drbd0 primary -o

mkfs.ext4 /dev/drbd0

Now we will make a directory for MySQL to run in (on MySQL1)

mkdir -p /mnt/drbd/mysql

You will need to edit /etc/fstab and add the following line

/dev/drbd0 /mnt/drbd/mysql ext4 defaults 0 0

Now we can mount the drive

mount /mnt/drbd/mysql

Now we can test that everything is working

On MySQL1:

echo "This is a test" > /mnt/drbd/mysql/test.txt

umount /dev/drbd0

On MySQL2:

drbdsetup /dev/drbd0 primary

mount /dev/drbd0

ls /mnt/drbd/mysql

(you should see test.txt in there)

Ensure that we start DRBD on system boot

Both nodes:

chkconfig drbd on

There we go, DRBD is installed, and working. Now we'll install MySQL.

On Both Servers:

yum install -y mysql-server mysql

mv /var/lib/mysql /mnt/drbd/mysql

ln -s /mnt/drbd/mysql/mysql /var/lib

On MySQL1

/etc/init.d/mysqld start

Now log in to MySQL and see that it's working.

On MySQL1:

mysql

create database drbdtest;

quit;



/etc/init.d/mysqld stop

umount /dev/drbd0

drbdadm secondary mysql

On MySQL2

ln -s /mnt/drbd/mysql/mysql /var/lib

drbdadm primary mysql

mount /dev/drbd0

/etc/init.d/mysqld start

mysql

show databases;

drop database drbdtest;

Great, MySQL is now working on DRBD, and can fail over, lets automate this now!

On Both servers:

yum install -y heartbeat

No we want to edit /etc/ha.d/authkeys

auth 3

3 md5 (put your MD5 password here)

chmod 600 /etc/ha.d/authkeys

Edit /etc/ha.d/ha.cf

 debugfile /var/log/ha-debug

logfile /var/log/ha-log

logfacility local0

keepalive 1

deadtime 10

warntime 3

udpport 694

ucast eth0 10.0.1.101 # the other node

auto_failback off

node mysql1

node mysql2

Note that the ucast line will be different on each server

Edit /etc/ha.d/haresources

mysql1 drbddisk::mysql Filesystem::/dev/drbd0::/mnt/drbd/mysql mysqld eni

Now we need to set up a new virtual network interface in the AWS Management console. (Note this assumes that you have the EC2 command line tools installed on the servers there are lots of tutorials out there for that, so I won't reiterate that)

edit /etc/init.d/eni

#!/bin/bash



MY_ID="i-XXXXXXXX" # the instance id of this node

OTHER_ID="i-XXXXXXXX" # the instance id of the other node

ENI="eni-XXXXXXXX"

ATTACH=`ec2-describe-network-interface-attribute $ENI -a|grep eni-attach|awk '{print $3}'`

case $1 in

start)

ec2-detach-network-interface $ATTACH

ec2-attach-network-interface $ENI -i $MY_ID -d 1 $ENI

ifup eth1

echo $0 started

;;

stop)

ec2-detach-network-interface $ATTACH

echo $0 stopped

;;

status)

ec2-describe-network-interface-attribute $ENI -a |grep "$MY_ID" > /dev/null

# grep will return true if this ip is mapped to this instance

[ $? -eq 0 ] && echo $0 OK || echo $0 FAIL

;;

esac

chmod 700 /etc/init.d/eni

edit /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1

BOOTPROTO=static

BROADCAST=10.0.1.255

IPADDR=10.0.1.110

NETMASK=255.255.255.0

ONBOOT=no

/etc/init.d/heartbeat start

Now you should have everything running and automatically failing over, with one static IP to use for your app servers to hit for MySQL. You can test this by stopping heartbeat on MySQL1, waiting a minute, and watching mysql start, and the network interface come up on MySQL2

• Tweet
• Share
• Share
• Share
• Share

Sign up here with your email