DRBD provides replication of storage across multiple servers. Its a software solution to perform block level replication. DRBD is able to perform real-time replication of data which is transparent to the upper layer applications. The replication can be done synchronously or asynchronously.
The DRBD block replication requires a raw disk partition on which the software creates its own meta-data. So for replication each server needs a separate partition which is added to the host other than the OS disk. Also the replication is highly dependant on network throughput and latency so its highly recommended to have a dedicated network interface which is used for storage replication.
The DRBD replication can be used in Single Primary or Dual Primary modes. In Single Primary only one node (Primary) is able to mount the DRBD partition and do read/write operations. The other node (Secondary) will only perform replication but will not make available the partition for any file-system operations. In this mode any standard filesystem can be used on top of the DRBD device, like EXT4 or XFS. In Dual Primary both nodes can read/write data on the DRBD partition. We need to configure clustering in the OS and use a cluster file system like GFS or OCFS for Dual Primary operations.
Single Primary setup is simple to deploy and manage. The following guide will provide details of Single Primary setup on two Ubuntu 20.04 servers. The failover will require manual intervention to change the state of servers, mount filesystems and start required application services.
Installation of DRBD
Add the DRBD repository for Ubuntu on both servers. Run apt update and install the DRBD kernel module and utilities.
$ sudo add-apt-repository ppa:linbit/linbit-drbd9-stack
$ sudo apt update
$ sudo apt install drbd-utils drbd-dkmsDRBD Resource Configuration
The main configuration file for DRBD is /etc/drbd.conf. The configuration for resources is under the directory /etc/drbd.d. The default parameters of DRBD need to be changed to improve the performance of the replication. The configuration needs to be done on both Primary and Secondary hosts.
DRBD requires exclusive access to the underlying storage. Stop any processes which can block the storage access like multipath daemon.
$ sudo systemctl stop multipathd
$ sudo systemctl disable multipathdConfigure global parameters for all storage resources in the file /etc/drbd.d/global_common.conf
global {
usage-count no;
}
common {
.
.
.
disk {
# Increase the activity log size to prevent frequent metadata updates
al-extents 6007;
# Disable Barriers and Disk Flushes
disk-barrier no;
disk-flushes no;
}
.
.Create a resources configuration file with the resource name (user given name) /etc/drbd.d/mirrordisk1.res. This will create a DRBD resource mirrordisk1 with block device /dev/drbd1 on top of raw partition /dev/vdb1.
resource "mirrordisk1" {
# Virtual Block Device Number /dev/drbdX
device minor 1;
# Low-Level Storage
disk /dev/vdb1;
# Store meta-data within the same storage
meta-disk internal;
on "server1" {
node-id 0;
}
on "server2" {
node-id 1;
}
connection {
host "server1" address 172.16.10.11:7790;
host "server2" address 172.16.20.22:7790;
}
net {
# Protocol A - Asynchronous replication
# Protocol B - Memory synchronous (semi-synchronous) replication
# Protocol C - Synchronous replication
protocol B;
# maximum number of buffers DRBD allocates for writing data to disk
max-buffers 8000;
# maximum number of write requests permitted between two write barriers
max-epoch-size 8000;
# TCP send buffer auto-tuning
sndbuf-size 0;
# Configure Checksum-based Synchronization
csums-alg md5;
}
}On the Primary server create meta-data on the DRBD resource disk
$ sudo drbdadm create-md mirrordisk1On both Primray and Secondary bring up the device
$ sudo drbdadm up mirrordisk1On the Primary server change the state of the resource to primary.
$ sudo drbdadm primary mirrordisk1On both Primary and Secondary bring up the device
$ sudo drbdadm up mirrordisk1Check the status of the resource on primary server.
$ sudo drbdadm status mirrordisk1
mirrordisk1 role:Primary
disk:UpToDate
server2 role:Secondary
peer-disk:UpToDateCheck the status of the resource on secondary server.
$ sudo drbdadm status mirrordisk1
mirrordisk1 role:Secondary
disk:UpToDate
server1 role:Primary
peer-disk:UpToDateFormat the drbd device with the required filesystem on the Primary node.
$ sudo mkfs.ext4 /dev/drbd1Mount the DRBD device which is now ready to use on the Primary node.
$ sudo mount /dev/drbd1 /somedirDRBD Resource Operations
Check detailed status and statistics of DRBD device
$ sudo drbdsetup status mirrordisk1 --verbose --statisticsCheck events related to DRBD device
$ sudo drbdsetup events2 mirrordisk1Check DRBD device ID of resource
$ sudo drbdadm sh-dev mirrordisk1Check low level device for DRBD resource
$ sudo drbdadm sh-ll-dev mirrordisk1If any configuration change is done in the drbd configurations for any resource apply the new configurations as follows.
$ sudo drbdadm adjust mirrordisk1DRBD Resource Failover
For performing failover when primary node is on operation follow below steps.
On Primary Node:
# Stop any running service which is using the mounted DRBD device
$ sudo systemctl stop email-service
# Unmount the DRBD device
$ sudo umount /somedir
# Change the DRBD device to secondary
$ sudo drbdadm secondary mirrordisk1On Secondary Node:
# Change the DRBD device to primary
$ sudo drbdadm primary mirrordisk1
# Mount the DRBD device
$ sudo mount /dev/drbd1 /somedir
# Start service which is required for application operations
$ sudo systemctl start email-serviceFor performing failover when primary node is misbehaving, shutdown the primary node or disconnect it from network and follow below steps.
On Secondary Node:
# Change the DRBD device to primary
$ sudo drbdadm primary mirrordisk1
# Mount the DRBD device
$ sudo mount /dev/drbd1 /somedir
# Start service which is required for application operations
$ sudo systemctl start email-serviceOn Primary Node:
Identify and fix any problems on the old primary node and then follow below steps.
# Disconnect the drbd resource
$ sudo drbdadm disconnect mirrordisk1
# Change the status to secondary
$ sudo drbdadm secondary mirrordisk1
# Connect with the new primary discarding own data
$ sudo drbdadm connect --discard-my-data mirrodisk1