sersync服务 – Tech 学习之路

一、功能

1.功能

sersync功能：实时同步；

2.组成

sersync==inotify+rsync

inotify:监控某个目录下面"文件/目录"是否发生变化;

3.工作原理

sersync工作原理：由inotify服务监控目录，当目录/文件发生变化时触发rsync服务进行推送，完成数据同步；

二、部署流程

使用Sersync服务之前，需在备份端部署Rsync守护进程模式

同时监控多个目录时，需增加多个配置文件，启动多个sersync进程

step0:上传Sersync安装包到NFS服务端指定目录下；
一般使用rz命令上传软件安装包

sersync安装包
百度云盘提取码：rfwy

[root@nfs01 /]# mkdir -p /server/tools/
[root@nfs01 /]# cd /server/tools/
[root@nfs01 /server/tools]# rz

[root@nfs01 /server/tools]# ls
sersync_installdir_64bit.zip
[root@nfs01 /server/tools]# unzip ./sersync_installdir_64bit.zip
Archive:  ./sersync_installdir_64bit.zip
   creating: sersync_installdir_64bit/
   creating: sersync_installdir_64bit/sersync/
   creating: sersync_installdir_64bit/sersync/bin/
  inflating: sersync_installdir_64bit/sersync/bin/sersync  
   creating: sersync_installdir_64bit/sersync/conf/
  inflating: sersync_installdir_64bit/sersync/conf/confxml.xml  
   creating: sersync_installdir_64bit/sersync/logs/
[root@nfs01 /server/tools]# cd ./sersync_installdir_64bit/
[root@nfs01 /server/tools/sersync_installdir_64bit]# ls
sersync
[root@nfs01 /server/tools/sersync_installdir_64bit]# mv ./sersync/ /application/
[root@nfs01 /server/tools/sersync_installdir_64bit]# ls ./ /application/
./:

/application/:
sersync

step1:为sersync/bin下的命令增加执行权限

chmod +x sersync/bin/sersync

[root@nfs01 /application]# tree sersync/
sersync/
├── bin
│   └── sersync
├── conf
│   └── confxml.xml
└── logs

3 directories, 2 files
[root@nfs01 /application]# cd ./sersync/bin/
[root@nfs01 /application/sersync/bin]# ls
sersync
[root@nfs01 /application/sersync/bin]# chmod +x sersync 
[root@nfs01 /application/sersync/bin]# ls
sersync

step2:将sersync命令创建软链接到/sbin目录下

ln -s sersync/bin/sersync /sbin/

创建软链接时一定要使用源文件的绝对路径，否则链接文件找不到源文件位置

[root@nfs01 /application/sersync/bin]# ln -s /application/sersync/bin/sersync /sbin/
[root@nfs01 /application/sersync/bin]# ll /sbin/sersync 
lrwxrwxrwx 1 root root 32 Jun 21 20:27 /sbin/sersync -> /application/sersync/bin/sersync

step3:修改sersync配置文件(sersync/conf/confxml.xml)

......
 <sersync>
 24 <localpath watch="监控目录路径">
 25 <remote ip="rsyncd服务端IP地址"name="模块名"/>
 ......
 <rsync>
 30 <commonParams params="-az"/>
 31 <auth start="true" users="rsync客户端用户名" passwordfile="rsync客户端密码文件"/>
 ......
 36 <failLog path="指定Sersync日志文件路径"`timeToExecute="60"/>......

配置sersync之前，先单独使用rsync服务推送一次，确定rsync服务正常以及各项参数的准确
相关路径写绝对路径，如密码文件的路径、监控目录的路径以及日志文件的路径

[root@nfs01 /application/sersync]# vim /application/sersync/conf/confxml.xml 
     1  <?xml version="1.0" encoding="ISO-8859-1"?>
     2  <head version="2.5">
     3      <host hostip="localhost" port="8008"></host>
     4      <debug start="false"/>
     5      <fileSystem xfs="false"/>
     6      <filter start="false">
     7      <exclude expression="(.*)\.svn"></exclude>
     8      <exclude expression="(.*)\.gz"></exclude>
     9      <exclude expression="^info/*"></exclude>
    10      <exclude expression="^static/*"></exclude>
    11      </filter>
    12      <inotify>
    13      <delete start="true"/>
    14      <createFolder start="true"/>
    15      <createFile start="false"/>
    16      <closeWrite start="true"/>
    17      <moveFrom start="true"/>
    18      <moveTo start="true"/>
    19      <attrib start="false"/>
    20      <modify start="false"/>
    21      </inotify>
    22  
    23      <sersync>
    24      <localpath watch="/upload">
    25          <remote ip="172.16.1.141" name="nfsbackup"/>
    26          <!--<remote ip="192.168.8.39" name="tongbu"/>-->
    27          <!--<remote ip="192.168.8.40" name="tongbu"/>-->
    28      </localpath>
    29      <rsync>
    30          <commonParams params="-az"/>
    31          <auth start="true" users="rsync_backup" passwordfile="/etc/rsync_client.password"/>
    32          <userDefinedPort start="false" port="874"/><!-- port=874 -->
    33          <timeout start="false" time="100"/><!-- timeout=100 -->
    34          <ssh start="false"/>
    35      </rsync>
    36      <failLog path="/var/log/sersync.log" timeToExecute="60"/><!--default every 60mins execute once-->
    37      <crontab start="false" schedule="600"><!--600mins-->
    38          <crontabfilter start="false">
    39          <exclude expression="*.php"></exclude>
    40          <exclude expression="info/*"></exclude>
    41          </crontabfilter>
    42      </crontab>
    43      <plugin start="false" name="command"/>
    44      </sersync>
    45  
    46      <plugin name="command">
    47      <param prefix="/bin/sh" suffix="" ignoreError="true"/>  <!--prefix /opt/tongbu/mmm.sh suffix-->
    48      <filter start="false">
    49          <include expression="(.*)\.php"/>
    50          <include expression="(.*)\.sh"/>
    51      </filter>
    52      </plugin>
    53  
    54      <plugin name="socket">
    55      <localpath watch="/opt/tongbu">
    56          <deshost ip="192.168.138.20" port="8009"/>
    57      </localpath>
    58      </plugin>
    59      <plugin name="refreshCDN">
    60      <localpath watch="/data0/htdocs/cms.xoyo.com/site/">
    61          <cdninfo domainname="ccms.chinacache.com" port="80" username="xxxx" passwd="xxxx"/>
    62          <sendurl base="http://pic.xoyo.com/cms"/>
    63          <regexurl regex="false" match="cms.xoyo.com/site([/a-zA-Z0-9]*).xoyo.com/images"/>
    64      </localpath>
    65      </plugin>
    66  </head>

step4:启动sersync服务
服务启动后，将过程文件中rsync服务命令粘贴出来单独执行一遍，检验配置文件信息是否填写正确

sersync -rdo sersync/conf/confxml.xml

sersync命令参数说明

sersync -h

set the system param

execute：echo 50000000 > /proc/sys/fs/inotify/max_user_watches

execute：echo 327679 > /proc/sys/fs/inotify/max_queued_events

parse the command param

参数-d:启用守护进程模式

参数-r:在监控前，将监控目录与远程主机用rsync命令推送一遍

参数-n: 指定开启守护线程的数量，默认为10个

参数-o:指定配置文件，默认使用confxml.xml文件

参数-m:单独启用其他模块，使用 -m refreshCDN 开启刷新CDN模块

参数-m:单独启用其他模块，使用 -m socket 开启socket模块

参数-m:单独启用其他模块，使用 -m http 开启http模块

不加-m参数，则默认执行同步程序

[root@nfs01 /upload]# sersync -rdo /application/sersync/conf/confxml.xml 
set the system param
execute：echo 50000000 > /proc/sys/fs/inotify/max_user_watches
execute：echo 327679 > /proc/sys/fs/inotify/max_queued_events
parse the command param
option: -r  rsync all the local files to the remote servers before the sersync work
option: -d  run as a daemon
option: -o  config xml name：  /application/sersync/conf/confxml.xml
daemon thread num: 10
parse xml config file
host ip : localhost host port: 8008
daemon start，sersync run behind the console 
use rsync password-file :
user is rsync_backup
passwordfile is     /etc/rsync_client.password
config xml parse success
please set /etc/rsyncd.conf max connections=0 Manually
sersync working thread 12  = 1(primary thread) + 1(fail retry thread) + 10(daemon sub threads) 
Max threads numbers is: 22 = 12(Thread pool nums) + 10(Sub threads)
please according your cpu ，use -n param to adjust the cpu rate
------------------------------------------
rsync the directory recursivly to the remote servers once
working please wait...
execute command: cd /upload && rsync -az -R --delete ./ rsync_backup@172.16.1.141::nfsbackup --password-file=/etc/rsync_client.password >/dev/null 2>&1 
run the sersync: 
watch path is: /upload
[root@nfs01 /upload]# rsync -az -R --delete ./ rsync_backup@172.16.1.141::nfsbackup --password-file=/etc/rsync_client.password

step5:查看sersync状态

ps -ef | grep 'sersync'

[root@nfs01 ~]# ps -ef | grep 'sersync'
root       7545      1  0 21:45 ?        00:00:00 sersync -rdo /application/sersync/conf/confxml.xml
root       7561   7502  0 21:45 pts/0    00:00:00 grep --color=auto sersync

step6:开机自启动sersync服务
向/etc/rc.d/rc.local文件追加内容前，先为文件添加执行权限

echo 'sersync -rdo /application/sersync/conf/confxml.xml' >>/etc/rc.local

[root@nfs01 ~]# chmod +x /etc/rc.d/rc.local 
[root@nfs01 ~]# vim /etc/rc.d/rc.local
#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.

touch /var/lock/subsys/local
sersync -rdo /application/sersync/conf/confxml.xml

step7:功能验证
[客户端-监控端]

[root@nfs01 /upload]# touch aspen{01..10}.txt
[root@nfs01 /upload]# ll
total 0
-rw-r--r-- 1 root root 0 Jun 21 21:55 aspen01.txt
-rw-r--r-- 1 root root 0 Jun 21 21:55 aspen02.txt
-rw-r--r-- 1 root root 0 Jun 21 21:55 aspen03.txt
-rw-r--r-- 1 root root 0 Jun 21 21:55 aspen04.txt
-rw-r--r-- 1 root root 0 Jun 21 21:55 aspen05.txt
-rw-r--r-- 1 root root 0 Jun 21 21:55 aspen06.txt
-rw-r--r-- 1 root root 0 Jun 21 21:55 aspen07.txt
-rw-r--r-- 1 root root 0 Jun 21 21:55 aspen08.txt
-rw-r--r-- 1 root root 0 Jun 21 21:55 aspen09.txt
-rw-r--r-- 1 root root 0 Jun 21 21:55 aspen10.txt

[服务端-备份端]

[root@backup /data]# cd /data/
[root@backup /data]# ll
total 0
-rw-r--r-- 1 rsync rsync 0 Jun 21 21:55 aspen01.txt
-rw-r--r-- 1 rsync rsync 0 Jun 21 21:55 aspen02.txt
-rw-r--r-- 1 rsync rsync 0 Jun 21 21:55 aspen03.txt
-rw-r--r-- 1 rsync rsync 0 Jun 21 21:55 aspen04.txt
-rw-r--r-- 1 rsync rsync 0 Jun 21 21:55 aspen05.txt
-rw-r--r-- 1 rsync rsync 0 Jun 21 21:55 aspen06.txt
-rw-r--r-- 1 rsync rsync 0 Jun 21 21:55 aspen07.txt
-rw-r--r-- 1 rsync rsync 0 Jun 21 21:55 aspen08.txt
-rw-r--r-- 1 rsync rsync 0 Jun 21 21:55 aspen09.txt
-rw-r--r-- 1 rsync rsync 0 Jun 21 21:55 aspen10.txt

一、功能

1.功能

2.组成

3.工作原理

二、部署流程

附：思维导图