初探OCFS2-配置服务
2016/11/28
目的：研究如何使用 iscsi 存储。 一、认识 据zstack的童鞋们介绍：OCFS2是主流对接 SAN 云存储的方式 存储的选择，通常有以下方式： 1、NAS 1）NFS 简单；自建有可靠性问题，商业的 NFS 产品更靠谱； 2）分布式文件系统，以 GlusterFS 为例； 易懂；建议使用 glusterfs 的接口来访问，但也支持通过 nfs 来访问； 2、IP SAN（iscsi） 1）使用iscsiadm 发现一下 target 即可 2）研究 ovirt 是如何使用 iscsi 存储的 看到一个讨论ovirt通过lvm来使用iscsi的方式 http://users.ovirt.narkive.com/XzD01akX/ovirt-users-shared-storage-with-iscsi 按照字面意思，我的理解（未实验）是： a、每个 host 挂载 iscsi 存储 b、通过 lvm 划分出 vm disk c、在运行了 vm 的 host 上 通过 lvchange 来激活指定的 vm disk lvchange -ay vol_group/logical_vol 3）通过集群文件系统（cluster filesystem: OCFS2,GFS,etc. ）来控制读写 iscsi 存储 这就是本文的初衷，接下来展开记录 OCFS2 是如何使用的。 二、配置 1、准备安装 oracle uek 内核（因为新版本的 ocfs2 内核模块被 uek 整合了） [root@n149 ~]# wget http://public-yum.oracle.com/public-yum-ol6.repo -O /etc/yum.repos.d/public-yum-ol6.repo [root@n149 ~]# rpm --import http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6 [root@n149 ~]# yum install yum-plugin-downloadonly -y [root@n149 ~]# mkdir /tmp/ocfs2 && cd /tmp/ocfs2/ [root@n149 ocfs2]# yum install --downloadonly --downloaddir=/tmp/ocfs2/ kernel-uek ocfs2-tools -y 2、安装 uek [root@n149 ocfs2]# yum localinstall kernel-uek* 使用新的内核，编辑 grub 调整启动顺序： [root@n149 ocfs2]# vim /etc/grub.conf 重启系统后查看： [root@n149 ocfs2]# uname -r 2.6.39-400.290.1.el6uek.x86_64 查看 ocfs2 模块是否加载： [root@n149 ocfs2]# modprobe ocfs2 [root@n149 ocfs2]# lsmod |grep oc ocfs2 1094439 0 ocfs2_nodemanager 223253 1 ocfs2 configfs 29308 2 ocfs2_nodemanager ocfs2_stackglue 11896 1 ocfs2 jbd2 93114 2 ocfs2,ext4 3、安装 ocfs2-tools [root@n149 ocfs2]# yum localinstall ocfs2-tools* -y 4、配置 ocfs2 服务 [root@n149 ~]# vim /etc/hosts 10.50.200.149 n149.ocfs2.test 10.50.200.150 n150.ocfs2.test [root@n149 ~]# service o2cb load checking debugfs... Mounting configfs filesystem at /sys/kernel/config: OK Loading stack plugin "o2cb": OK Loading filesystem "ocfs2_dlmfs": OK Creating directory '/dlm': OK Mounting ocfs2_dlmfs filesystem at /dlm: OK [root@n149 ~]# service o2cb online checking debugfs... Setting cluster stack "o2cb": OK Checking O2CB cluster configuration : Failed 符合预期，继续配置集群： [root@n149 ~]# o2cb_ctl -C -n prdcluster -t cluster -a name=prdcluster [root@n149 ~]# o2cb_ctl -C -n n149.ocfs2.test -t node -a number=0 -a ip_address=10.50.200.149 -a ip_port=11111 -a cluster=prdcluster [root@n149 ~]# o2cb_ctl -C -n n150.ocfs2.test -t node -a number=1 -a ip_address=10.50.200.150 -a ip_port=11111 -a cluster=prdcluster [root@n149 ~]# cat /etc/ocfs2/cluster.conf node: name = n149.ocfs2.test cluster = prdcluster number = 0 ip_address = 10.50.200.149 ip_port = 11111 node: name = n150.ocfs2.test cluster = prdcluster number = 1 ip_address = 10.50.200.150 ip_port = 11111 cluster: name = prdcluster heartbeat_mode = local node_count = 2 [root@n149 ~]# service o2cb configure [root@n149 ~]# service o2cb online 上述操作，在每个节点上执行。 其他可选操作： [root@n149 ~]# service o2cb status Driver for "configfs": Loaded Filesystem "configfs": Mounted Stack glue driver: Loaded Stack plugin "o2cb": Loaded Driver for "ocfs2_dlmfs": Loaded Filesystem "ocfs2_dlmfs": Mounted Checking O2CB cluster "prdcluster": Online Heartbeat dead threshold: 31 Network idle timeout: 30000 Network keepalive delay: 2000 Network reconnect delay: 2000 Heartbeat mode: Local Checking O2CB heartbeat: Not active Debug file system at /sys/kernel/debug: mounted 另外：确认防火墙放行了对应的端口，本例是 tcp port 11111 5、配置 iscsi 存储 [root@n149 ocfs2]# iscsiadm -m discovery -t st -p 10.50.200.41 [root@n149 ocfs2]# iscsiadm -m node -o delete -T iqn.1984-05.com.dell:powervault.md3800i.600a098000a64d8b0000000057a152ea -p 192.168.131.101 [root@n149 ocfs2]# iscsiadm -m node -o delete -T iqn.1984-05.com.dell:powervault.md3800i.600a098000a64d8b0000000057a152ea -p 192.168.131.102 [root@n149 ocfs2]# iscsiadm -m node -T iqn.1984-05.com.dell:powervault.md3800i.600a098000a64d8b0000000057a152ea --login [root@n149 ocfs2]# yum install device-mapper-multipath -y [root@n149 ocfs2]# modprobe dm-multipath [root@n149 ocfs2]# mpathconf [root@n149 ocfs2]# cat /etc/multipath.conf defaults { path_grouping_policy multibus path_checker readsector0 rr_min_io_rq 10 max_fds 8192 rr_weight priorities failback immediate no_path_retry fail user_friendly_names yes } multipaths { multipath { wwid 3600a098000a64d8b000002a4582305f8 alias test_block_a } } [root@n149 ocfs2]# service multipathd start [root@n149 ocfs2]# chkconfig multipathd on [root@n149 ocfs2]# multipath -ll test_block_a (3600a098000a64d8b000002a4582305f8) dm-0 DELL,MD38xxi size=100G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 4:0:0:0 sda 8:0 active ready running `- 5:0:0:0 sdb 8:16 active ready running 6、使用 iscsi 存储 挂载参数请参考：man mkfs.ocfs2 [root@n149 ~]# mkfs.ocfs2 -b 4K -C 64K -L testblocka -T datafiles -F /dev/mapper/test_block_a mkfs.ocfs2 1.8.6 Cluster stack: classic o2cb Overwriting existing ocfs2 partition. WARNING: Cluster check disabled. Proceed (y/N): y Filesystem Type of datafiles Label: testblocka Features: sparse extended-slotmap backup-super unwritten inline-data strict-journal-super xattr indexed-dirs refcount discontig-bg Block size: 4096 (12 bits) Cluster size: 65536 (16 bits) Volume size: 107374182400 (1638400 clusters) (26214400 blocks) Cluster groups: 51 (tail covers 25600 clusters, rest cover 32256 clusters) Extent allocator size: 41943040 (10 groups) Journal size: 33554432 Node slots: 8 Creating bitmaps: done Initializing superblock: done Writing system files: done Writing superblock: done Writing backup superblock: 4 block(s) Formatting Journals: done Growing extent allocator: done Formatting slot map: done Formatting quota files: done Writing lost+found: done mkfs.ocfs2 successful [root@n149 ~]# cat <<_EOF >>/etc/fstab /dev/mapper/test_block_a /mnt/tba ocfs2 defaults 0 0 _EOF [root@n149 ~]# mkdir /mnt/tba [root@n149 ~]# mount -a [root@n149 ~]# df -h /mnt/tba Filesystem Size Used Avail Use% Mounted on /dev/mapper/test_block_a 100G 586M 100G 1% /mnt/tba 在另一个节点上执行： [root@n150 ~]# cat <<_EOF >>/etc/fstab /dev/mapper/test_block_a /mnt/tba ocfs2 defaults 0 0 _EOF [root@n150 ~]# mkdir /mnt/tba [root@n150 ~]# mount -a [root@n150 ~]# df -h /mnt/tba Filesystem Size Used Avail Use% Mounted on /dev/mapper/test_block_a 100G 586M 100G 1% /mnt/tba 查看对应的日志： Nov 16 13:46:01 n149 kernel: o2hb: Heartbeat mode set to local Nov 16 13:46:01 n149 o2hbmonitor: Starting Nov 16 13:47:42 n149 kernel: OCFS2 1.8.0 Nov 16 13:47:43 n149 kernel: o2dlm: Joining domain 55663D9EA9B0466D80626459E2B6AE9B ( 1 ) 1 nodes Nov 16 13:47:43 n149 kernel: ocfs2: Mounting device (251,0) on (node 1, slot 0) with ordered data mode. Nov 16 13:48:16 n149 kernel: o2net: Connection to node n150.ocfs2.test (num 0) at 10.50.200.150:11111 shutdown, state 7 Nov 16 13:48:18 n149 kernel: o2net: Connected to node n150.ocfs2.test (num 0) at 10.50.200.150:11111 Nov 16 13:48:22 n149 kernel: o2dlm: Node 0 joins domain 55663D9EA9B0466D80626459E2B6AE9B ( 0 1 ) 2 nodes Nov 16 13:45:43 n150 kernel: o2hb: Heartbeat mode set to local Nov 16 13:45:43 n150 o2hbmonitor: Starting Nov 16 13:48:18 n150 kernel: o2net: Accepted connection from node n149.ocfs2.test (num 1) at 10.50.200.149:11111 Nov 16 13:48:22 n150 kernel: OCFS2 1.8.0 Nov 16 13:48:22 n150 kernel: o2dlm: Joining domain 55663D9EA9B0466D80626459E2B6AE9B ( 0 1 ) 2 nodes Nov 16 13:48:22 n150 kernel: ocfs2: Mounting device (251,0) on (node 0, slot 1) with ordered data mode. 2个节点的 heartbeat 的状态也都变成了 Active [root@n149 ~]# service o2cb status Driver for "configfs": Loaded Filesystem "configfs": Mounted Stack glue driver: Loaded Stack plugin "o2cb": Loaded Driver for "ocfs2_dlmfs": Loaded Filesystem "ocfs2_dlmfs": Mounted Checking O2CB cluster "prdcluster": Online Heartbeat dead threshold: 31 Network idle timeout: 30000 Network keepalive delay: 2000 Network reconnect delay: 2000 Heartbeat mode: Local Checking O2CB heartbeat: Active Debug file system at /sys/kernel/debug: mounted 7、测试文件读写 【2个节点同时写一个文件】 [root@n149 ~]# for i in `seq 1 9`;do echo "`hostname` -> [`date`]" >>/mnt/tba/1.log;sleep 1s;done [root@n150 ~]# for i in `seq 1 9`;do echo "`hostname` -> [`date`]" >>/mnt/tba/1.log;sleep 1s;done [root@n149 ~]# cat /mnt/tba/1.log n149.ocfs2.test -> [Wed Nov 16 13:55:20 CST 2016] n150.ocfs2.test -> [Wed Nov 16 13:55:21 CST 2016] n149.ocfs2.test -> [Wed Nov 16 13:55:21 CST 2016] n150.ocfs2.test -> [Wed Nov 16 13:55:22 CST 2016] n149.ocfs2.test -> [Wed Nov 16 13:55:22 CST 2016] n150.ocfs2.test -> [Wed Nov 16 13:55:23 CST 2016] n149.ocfs2.test -> [Wed Nov 16 13:55:23 CST 2016] n150.ocfs2.test -> [Wed Nov 16 13:55:24 CST 2016] n149.ocfs2.test -> [Wed Nov 16 13:55:24 CST 2016] n150.ocfs2.test -> [Wed Nov 16 13:55:25 CST 2016] n149.ocfs2.test -> [Wed Nov 16 13:55:25 CST 2016] n150.ocfs2.test -> [Wed Nov 16 13:55:26 CST 2016] n149.ocfs2.test -> [Wed Nov 16 13:55:26 CST 2016] n150.ocfs2.test -> [Wed Nov 16 13:55:27 CST 2016] n149.ocfs2.test -> [Wed Nov 16 13:55:27 CST 2016] n150.ocfs2.test -> [Wed Nov 16 13:55:28 CST 2016] n149.ocfs2.test -> [Wed Nov 16 13:55:28 CST 2016] n150.ocfs2.test -> [Wed Nov 16 13:55:29 CST 2016] 结论：符合预期。 【一个节点写，另一个节点读】 [root@n149 ~]# for i in `seq 1 9`;do echo "`hostname` -> [`date`]" >>/mnt/tba/1.log;sleep 1s;done [root@n150 ~]# tail -f /mnt/tba/1.log n149.ocfs2.test -> [Wed Nov 16 13:57:42 CST 2016] n149.ocfs2.test -> [Wed Nov 16 13:57:43 CST 2016] n149.ocfs2.test -> [Wed Nov 16 13:57:44 CST 2016] n149.ocfs2.test -> [Wed Nov 16 13:57:45 CST 2016] n149.ocfs2.test -> [Wed Nov 16 13:57:46 CST 2016] n149.ocfs2.test -> [Wed Nov 16 13:57:47 CST 2016] n149.ocfs2.test -> [Wed Nov 16 13:57:48 CST 2016] n149.ocfs2.test -> [Wed Nov 16 13:57:49 CST 2016] n149.ocfs2.test -> [Wed Nov 16 13:57:50 CST 2016] ^C 结论：符合预期。 三、问题解释 1、cluster 处于 Offline 的状态 [root@n150 ~]# mount -a mount.ocfs2: Internal logic failure while trying to join the group [root@n150 ~]# service o2cb status Checking O2CB cluster "prdcluster": Offline 则请检查各节点的域名是否能正常解析，ocfs2 要用到 hostname 来提供服务。有一种情况是： 上述节点的 hostname 为： n149.test.com n150.test.com 而配置 ocfs2 是在 hosts 中写死了： 10.50.200.149 n149.ocfs2.test 10.50.200.150 n150.ocfs2.test 而实际上，ocfs2 使用的却是 hostname，因而节点之间的连接异常。 2、heartbeat 处于 Not active 状态 挂载卷后 heartbeat 才会启动，因此，先操作后再观察。 ZYXW、参考 1、[ovirt-users] shared storage with iscsi http://users.ovirt.narkive.com/XzD01akX/ovirt-users-shared-storage-with-iscsi 2、Linux : Oracle unbreakable kernel and OCFS2 on CentOS/RHEL6 http://www.itechlounge.net/2014/08/linux-oracle-unbreakable-kernel-and-ocfs2-on-centosrhel6/ 3、Oracle Linux 6: Create an OCFS2 Cluster and Filesystem https://www.itfromallangles.com/2013/03/oracle-linux-6-create-an-ocfs2-cluster-and-filesystem/ 4、Installing And Configuring An OCFS2 Clustered File System http://www.shapeblue.com/installing-and-configuring-an-ocfs2-clustered-file-system/ 5、解决o2cb status时状态始终为offline(o2cb_ctl: Internal logic failure failed while adding RAC1) http://www.blogjava.net/KingKong/archive/2011/06/06/351826.html 6、O2CB heartbeat: Not active https://oss.oracle.com/pipermail/ocfs2-users/2009-April/003477.html另外有需要云服务器可以了解下创新互联cdcxhl.cn，海内外云服务器15元起步，三天无理由+7*72小时售后在线，公司持有idc许可证，提供“云服务器、裸金属服务器、高防服务器、香港服务器、美国服务器、虚拟主机、免备案服务器”等云主机租用服务以及企业上云的综合解决方案，具有“安全稳定、简单易用、服务可用性高、性价比高”等特点与优势，专为企业上云打造定制，能够满足用户丰富、多元化的应用场景需求。