您当前的位置： 首页 > 学无止境 > 心得笔记 网站首页心得笔记
通过corosync+heartbeat实现集群
发布时间：2019-05-02 18:50:20编辑：雪饮阅读（）
上次实现了heartbeatv2的mysql高可用。这次来实现下通过heartbeatv3+corosync搭建高可用集群。
和单纯heartbeat搭建高可用集群的前提条件相同：
时间同步、配置主机名、主机互信
那么我们之前配置好的时间服务器(同时也是nfs共享服务器)暂时先不动。
只需要重新配置下各节点的时间同步、主机名、主机互信
时间同步、主机名、主机互信的配置这里就不在多加赘述了，接下来直接进入正题
ip规划：
时间服务器（nfs共享服务器）：192.168.2.189
node1:192.168.2.173
node2:192.168.2.191
所需软件包：
cluster-glue-1.0.6-1.6.el5.i386.rpm
cluster-glue-libs-1.0.6-1.6.el5.i386.rpm
corosync-1.2.7-1.1.el5.i386.rpm
corosynclib-1.2.7-1.1.el5.i386.rpm
heartbeat-3.0.3-2.3.el5.i386.rpm
heartbeat-libs-3.0.3-2.3.el5.i386.rpm
libesmtp-1.0.4-5.el5.i386.rpm
openaislib-1.1.3-1.6.el5.i386.rpm
pacemaker-1.1.5-1.1.el5.i386.rpm
pacemaker-cts-1.1.5-1.1.el5.i386.rpm
pacemaker-libs-1.1.5-1.1.el5.i386.rpm
resource-agents-1.0.4-1.1.el5.i386.rpm
各节点先将以上软件包都安装上
[root@node2 ~]# yum --nogpgcheck localinstall /usr/local/src/*
Loaded plugins: katello, product-id, security, subscription-manager
Updating certificate-based repositories.
Unable to read consumer identity
Setting up Local Package Process
Examining /usr/local/src/cluster-glue-1.0.6-1.6.el5.i386.rpm: cluster-glue-1.0.6-1.6.el5.i386
Marking /usr/local/src/cluster-glue-1.0.6-1.6.el5.i386.rpm to be installed
Examining /usr/local/src/cluster-glue-libs-1.0.6-1.6.el5.i386.rpm: cluster-glue-libs-1.0.6-1.6.el5.i386
Marking /usr/local/src/cluster-glue-libs-1.0.6-1.6.el5.i386.rpm to be installed
Examining /usr/local/src/corosync-1.2.7-1.1.el5.i386.rpm: corosync-1.2.7-1.1.el5.i386
Marking /usr/local/src/corosync-1.2.7-1.1.el5.i386.rpm to be installed
Examining /usr/local/src/corosynclib-1.2.7-1.1.el5.i386.rpm: corosynclib-1.2.7-1.1.el5.i386
Marking /usr/local/src/corosynclib-1.2.7-1.1.el5.i386.rpm to be installed
Examining /usr/local/src/heartbeat-3.0.3-2.3.el5.i386.rpm: heartbeat-3.0.3-2.3.el5.i386
Marking /usr/local/src/heartbeat-3.0.3-2.3.el5.i386.rpm to be installed
Examining /usr/local/src/heartbeat-libs-3.0.3-2.3.el5.i386.rpm: heartbeat-libs-3.0.3-2.3.el5.i386
Marking /usr/local/src/heartbeat-libs-3.0.3-2.3.el5.i386.rpm to be installed
Examining /usr/local/src/libesmtp-1.0.4-5.el5.i386.rpm: libesmtp-1.0.4-5.el5.i386
Marking /usr/local/src/libesmtp-1.0.4-5.el5.i386.rpm to be installed
Examining /usr/local/src/openais-1.1.3-1.6.el5.i386.rpm: openais-1.1.3-1.6.el5.i386
Marking /usr/local/src/openais-1.1.3-1.6.el5.i386.rpm to be installed
Examining /usr/local/src/openaislib-1.1.3-1.6.el5.i386.rpm: openaislib-1.1.3-1.6.el5.i386
Marking /usr/local/src/openaislib-1.1.3-1.6.el5.i386.rpm to be installed
Examining /usr/local/src/pacemaker-1.1.5-1.1.el5.i386.rpm: pacemaker-1.1.5-1.1.el5.i386
Marking /usr/local/src/pacemaker-1.1.5-1.1.el5.i386.rpm to be installed
Examining /usr/local/src/pacemaker-cts-1.1.5-1.1.el5.i386.rpm: pacemaker-cts-1.1.5-1.1.el5.i386
Marking /usr/local/src/pacemaker-cts-1.1.5-1.1.el5.i386.rpm to be installed
Examining /usr/local/src/pacemaker-libs-1.1.5-1.1.el5.i386.rpm: pacemaker-libs-1.1.5-1.1.el5.i386
Marking /usr/local/src/pacemaker-libs-1.1.5-1.1.el5.i386.rpm to be installed
Examining /usr/local/src/resource-agents-1.0.4-1.1.el5.i386.rpm: resource-agents-1.0.4-1.1.el5.i386
Marking /usr/local/src/resource-agents-1.0.4-1.1.el5.i386.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package cluster-glue.i386 0:1.0.6-1.6.el5 set to be updated
--> Processing Dependency: libltdl.so.3 for package: cluster-glue
--> Processing Dependency: libopenhpi.so.2 for package: cluster-glue
--> Processing Dependency: perl-TimeDate for package: cluster-glue
---> Package cluster-glue-libs.i386 0:1.0.6-1.6.el5 set to be updated
---> Package corosync.i386 0:1.2.7-1.1.el5 set to be updated
---> Package corosynclib.i386 0:1.2.7-1.1.el5 set to be updated
--> Processing Dependency: libibverbs.so.1 for package: corosynclib
--> Processing Dependency: libibverbs.so.1(IBVERBS_1.0) for package: corosynclib
--> Processing Dependency: libibverbs.so.1(IBVERBS_1.1) for package: corosynclib
--> Processing Dependency: librdmacm.so.1 for package: corosynclib
--> Processing Dependency: librdmacm.so.1(RDMACM_1.0) for package: corosynclib
---> Package heartbeat.i386 0:3.0.3-2.3.el5 set to be updated
---> Package heartbeat-libs.i386 0:3.0.3-2.3.el5 set to be updated
---> Package libesmtp.i386 0:1.0.4-5.el5 set to be updated
---> Package openais.i386 0:1.1.3-1.6.el5 set to be updated
---> Package openaislib.i386 0:1.1.3-1.6.el5 set to be updated
---> Package pacemaker.i386 0:1.1.5-1.1.el5 set to be updated
--> Processing Dependency: libsensors.so.3 for package: pacemaker
---> Package pacemaker-cts.i386 0:1.1.5-1.1.el5 set to be updated
---> Package pacemaker-libs.i386 0:1.1.5-1.1.el5 set to be updated
---> Package resource-agents.i386 0:1.0.4-1.1.el5 set to be updated
--> Running transaction check
---> Package libibverbs.i386 0:1.1.3-2.el5 set to be updated
--> Processing Dependency: openib >= 1.0 for package: libibverbs
--> Processing Dependency: libibverbs-driver for package: libibverbs
---> Package librdmacm.i386 0:1.0.10-1.el5 set to be updated
---> Package libtool-ltdl.i386 0:1.5.22-7.el5_4 set to be updated
---> Package lm_sensors.i386 0:2.10.7-9.el5 set to be updated
---> Package openhpi-libs.i386 0:2.14.0-5.el5 set to be updated
---> Package perl-TimeDate.noarch 1:1.16-5.el5 set to be updated
--> Running transaction check
---> Package libnes.i386 0:0.9.0-2.el5 set to be updated
---> Package openib.noarch 0:1.4.1-6.el5 set to be updated
--> Finished Dependency Resolution
Dependencies Resolved
========================================================================================================================================================================================
Package Arch Version Repository Size
========================================================================================================================================================================================
Installing:
cluster-glue i386 1.0.6-1.6.el5 /cluster-glue-1.0.6-1.6.el5.i386 758 k
cluster-glue-libs i386 1.0.6-1.6.el5 /cluster-glue-libs-1.0.6-1.6.el5.i386 331 k
corosync i386 1.2.7-1.1.el5 /corosync-1.2.7-1.1.el5.i386 321 k
corosynclib i386 1.2.7-1.1.el5 /corosynclib-1.2.7-1.1.el5.i386 349 k
heartbeat i386 3.0.3-2.3.el5 /heartbeat-3.0.3-2.3.el5.i386 255 k
heartbeat-libs i386 3.0.3-2.3.el5 /heartbeat-libs-3.0.3-2.3.el5.i386 553 k
libesmtp i386 1.0.4-5.el5 /libesmtp-1.0.4-5.el5.i386 127 k
openais i386 1.1.3-1.6.el5 /openais-1.1.3-1.6.el5.i386 453 k
openaislib i386 1.1.3-1.6.el5 /openaislib-1.1.3-1.6.el5.i386 175 k
pacemaker i386 1.1.5-1.1.el5 /pacemaker-1.1.5-1.1.el5.i386 13 M
pacemaker-cts i386 1.1.5-1.1.el5 /pacemaker-cts-1.1.5-1.1.el5.i386 741 k
pacemaker-libs i386 1.1.5-1.1.el5 /pacemaker-libs-1.1.5-1.1.el5.i386 870 k
resource-agents i386 1.0.4-1.1.el5 /resource-agents-1.0.4-1.1.el5.i386 1.1 M
Installing for dependencies:
libibverbs i386 1.1.3-2.el5 Server 45 k
libnes i386 0.9.0-2.el5 Server 13 k
librdmacm i386 1.0.10-1.el5 Server 23 k
libtool-ltdl i386 1.5.22-7.el5_4 Server 37 k
lm_sensors i386 2.10.7-9.el5 Server 511 k
openhpi-libs i386 2.14.0-5.el5 Server 157 k
openib noarch 1.4.1-6.el5 Server 20 k
perl-TimeDate noarch 1:1.16-5.el5 Server 32 k
Transaction Summary
========================================================================================================================================================================================
Install 21 Package(s)
Upgrade 0 Package(s)
Total size: 19 M
Total download size: 838 k
Is this ok [y/N]: y
Downloading Packages:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 292 MB/s | 838 kB 00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : libtool-ltdl 1/21
Installing : openib 2/21
Installing : perl-TimeDate 3/21
Installing : openhpi-libs 4/21
Installing : lm_sensors 5/21
Installing : libesmtp 6/21
Installing : pacemaker-cts 7/21
Installing : cluster-glue-libs 8/21
Installing : libibverbs 9/21
Installing : cluster-glue 10/21
Installing : resource-agents 11/21
Installing : librdmacm 12/21
Installing : libnes 13/21
Installing : corosynclib 14/21
Installing : heartbeat-libs 15/21
Installing : corosync 16/21
Installing : pacemaker 17/21
Installing : openaislib 18/21
Installing : pacemaker-libs 19/21
Installing : openais 20/21
Installing : heartbeat 21/21
Cluster/productid | 1.7 kB 00:00
ClusterStorage/productid | 1.7 kB 00:00
Server/productid | 1.7 kB 00:00
Installed products updated.
Installed:
cluster-glue.i386 0:1.0.6-1.6.el5 cluster-glue-libs.i386 0:1.0.6-1.6.el5 corosync.i386 0:1.2.7-1.1.el5 corosynclib.i386 0:1.2.7-1.1.el5 heartbeat.i386 0:3.0.3-2.3.el5
heartbeat-libs.i386 0:3.0.3-2.3.el5 libesmtp.i386 0:1.0.4-5.el5 openais.i386 0:1.1.3-1.6.el5 openaislib.i386 0:1.1.3-1.6.el5 pacemaker.i386 0:1.1.5-1.1.el5
pacemaker-cts.i386 0:1.1.5-1.1.el5 pacemaker-libs.i386 0:1.1.5-1.1.el5 resource-agents.i386 0:1.0.4-1.1.el5
Dependency Installed:
libibverbs.i386 0:1.1.3-2.el5 libnes.i386 0:0.9.0-2.el5 librdmacm.i386 0:1.0.10-1.el5 libtool-ltdl.i386 0:1.5.22-7.el5_4 lm_sensors.i386 0:2.10.7-9.el5
openhpi-libs.i386 0:2.14.0-5.el5 openib.noarch 0:1.4.1-6.el5 perl-TimeDate.noarch 1:1.16-5.el5
Complete!
节点配置
配置corosync
[root@node1 ~]# cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf
[root@node1 ~]# vi /etc/corosync/corosync.conf
[root@node1 ~]# cat /etc/corosync/corosync.conf
# Please read the corosync.conf.5 manual page
compatibility: whitetank
totem {
version: 2
secauth: on
threads: 2
interface {
ringnumber: 0
bindnetaddr: 192.168.2.0
mcastaddr: 226.96.6.17
mcastport: 5405
}
}
logging {
fileline: off
to_stderr: no
to_logfile: yes
to_syslog: no
logfile: /var/log/cluster/corosync.log
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
amf {
mode: disabled
}
service {
ver:0
name:pacemaker
}
aisexec {
user:root
group:root
}
注意：
secauth是开启秘钥
threads是线程数，不能太贪心，根据自己的实际情况酌情调整
bindnetaddr是绑定地址，只填写网络地址即可
mcastaddr是广播地址，只要不冲突就行
to_syslog是否将日志写入系统日志
logfile当to_logfile为yes时表示将日志写入独立文件中，则logfile用来指定该独立文件
创建日志文件
[root@node1 ~]# mkdir -p /var/log/cluster/
[root@node1 ~]# touch /var/log/cluster/corosync.log
生成秘钥
[root@node1 ~]# corosync-keygen
#此时已经卡住了，说没有这么多个随机数，可以在打开一个终端，不断的敲击键盘，不过这么有一点的久，你可以在ftp下载一个大的文件，这样会产生大量的IO。
注意秘钥只需任意一个节点生成，其它节点同步过去即可
集群启动
在任意一个节点上启动集群
[root@node1 ~]# service corosync start
Starting Corosync Cluster Engine (corosync):
[ OK ]
[root@node1 ~]# ssh node2 '/etc/init.d/corosync start'
Starting Corosync Cluster Engine (corosync): [ OK ]
集群检查
corosync引擎检查
[root@node1 ~]# grep -e "Corosync Cluster Engine" -e "configuration file" /var/log/cluster/corosync.log
Apr 27 05:45:44 corosync [MAIN ] Corosync Cluster Engine ('1.2.7'): started and ready to provide service.
Apr 27 05:45:44 corosync [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
检查成员节点通知
[root@node1 ~]# grep TOTEM /var/log/cluster/corosync.log
Apr 27 05:45:44 corosync [TOTEM ] Initializing transport (UDP/IP).
Apr 27 05:45:44 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Apr 27 05:45:44 corosync [TOTEM ] The network interface is down.
Apr 27 05:45:45 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Apr 27 05:48:53 corosync [TOTEM ] Initializing transport (UDP/IP).
Apr 27 05:48:53 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Apr 27 05:48:53 corosync [TOTEM ] The network interface is down.
Apr 27 05:48:53 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Apr 27 05:54:02 corosync [TOTEM ] Initializing transport (UDP/IP).
Apr 27 05:54:02 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Apr 27 05:54:02 corosync [TOTEM ] The network interface is down.
Apr 27 05:54:02 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Apr 27 06:04:25 corosync [TOTEM ] Initializing transport (UDP/IP).
Apr 27 06:04:25 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Apr 27 06:04:25 corosync [TOTEM ] The network interface is down.
Apr 27 06:04:25 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Apr 27 09:46:25 corosync [TOTEM ] Initializing transport (UDP/IP).
Apr 27 09:46:25 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Apr 27 09:46:26 corosync [TOTEM ] The network interface [192.168.2.173] is now up.
Apr 27 09:46:26 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Apr 27 09:46:42 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
检查启动过程中是否有错误发生
[root@node1 ~]# grep ERROR: /var/log/cluster/corosync.log
Apr 27 09:51:26 node1.magedu.com pengine: [23144]: ERROR: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
Apr 27 09:51:26 node1.magedu.com pengine: [23144]: ERROR: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
Apr 27 09:51:26 node1.magedu.com pengine: [23144]: ERROR: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
Apr 27 09:51:41 node1.magedu.com pengine: [23144]: ERROR: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
Apr 27 09:51:41 node1.magedu.com pengine: [23144]: ERROR: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
Apr 27 09:51:41 node1.magedu.com pengine: [23144]: ERROR: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
这里的STONITH可以忽略，因为我们本来就没有配置STONITH
检查pacemaker是否正常启动
[root@node1 ~]# grep pcmk_startup /var/log/cluster/corosync.log
Apr 27 09:51:47 corosync [pcmk ] info: pcmk_startup: CRM: Initialized
Apr 27 09:51:47 corosync [pcmk ] Logging: Initialized pcmk_startup
Apr 27 09:51:47 corosync [pcmk ] info: pcmk_startup: Maximum core file size is: 4294967295
Apr 27 09:51:47 corosync [pcmk ] info: pcmk_startup: Service: 9
Apr 27 09:51:47 corosync [pcmk ] info: pcmk_startup: Local hostname: node1.magedu.com
用crm_mon命令查看检查集群状态
============
Last updated: Sat Apr 27 09:57:00 2019
Stack: openais
Current DC: node1.magedu.com - partition with quorum
Version: 1.1.5-1.1.el5-01e86afaaa6d4a8c4836f68df80ababd6ca3902f
2 Nodes configured, 2 expected votes
0 Resources configured.
============
Online: [ node1.magedu.com node2.magedu.com ]
所需软件包下载：
关键字词：corosync,heartbeat,集群
上一篇：在ha的基础上搭建mysql集群
下一篇：corosync资源及约束