MHA failover GTID ??????-??????-????????????-?????????

MHA failover GTID ??????

?????????masterha_master_switch??????????????????????????????????????????

????????????(???????????????)

host_1(host_1:3306) (current master)
 +--host_2(host_2:3306 slave[candidate master])
 +--host_3(host_3:3306 etl)

??????Master : MySQL down

1.1 etl ??????8??????

?????????????????????no_check_delay=0 ??????????????????

1.2 slave(??????master)???etl??????????????????

1.2.1 ???master?????????????????????????????????slave????????????master ??????MySQL??????

### ????????????????????????3???DB gtid??????

* master host_2

dba:lc> show master status;
+---------------------+----------+--------------+------------------+------------------------------------------------------------------------------------------+
| File                | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                                                                        |
+---------------------+----------+--------------+------------------+------------------------------------------------------------------------------------------+
| host_1.000002 |     2885 |              |                  | 0923e916-3c36-11e6-82a5-ecf4bbf1f518:1-16,
ebd9ff93-c5b2-11e6-b21d-ecf4bbf1f42c:1-446362 |
+---------------------+----------+--------------+------------------+------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)


* slave (candidate master) host_1

           Retrieved_Gtid_Set: ebd9ff93-c5b2-11e6-b21d-ecf4bbf1f42c:446353
            Executed_Gtid_Set: 0923e916-3c36-11e6-82a5-ecf4bbf1f518:1-16,
ebd9ff93-c5b2-11e6-b21d-ecf4bbf1f42c:1-446353
                Auto_Position: 1

* etl (other slave) host_3

           Retrieved_Gtid_Set: 0923e916-3c36-11e6-82a5-ecf4bbf1f518:4-16,
ebd9ff93-c5b2-11e6-b21d-ecf4bbf1f42c:446353-446356
            Executed_Gtid_Set: 0923e916-3c36-11e6-82a5-ecf4bbf1f518:1-16,
ebd9ff93-c5b2-11e6-b21d-ecf4bbf1f42c:1-446356
                Auto_Position: 1



### ????????????
masterha_master_switch --global_conf=/data/online/agent/MHA/conf/masterha_default.cnf --conf=/data/online/agent/MHA/conf/bak_mha_test.cnf  --dead_master_host=host_2  --dead_master_port=3306 --master_state=dead --interactive=0 --ignore_last_failover --ignore_binlog_server_error

Thu Nov  9 10:43:49 2017 - [info] MHA::MasterFailover version 0.56.
Thu Nov  9 10:43:49 2017 - [info] Starting master failover.
Thu Nov  9 10:43:49 2017 - [info]
Thu Nov  9 10:43:49 2017 - [info] * Phase 1: Configuration Check Phase..
Thu Nov  9 10:43:49 2017 - [info]
Thu Nov  9 10:43:50 2017 - [info] HealthCheck: SSH to host_2 is reachable.
Thu Nov  9 10:43:50 2017 - [info] Binlog server host_2 is reachable.
Thu Nov  9 10:43:50 2017 - [info] HealthCheck: SSH to host_1 is reachable.
Thu Nov  9 10:43:50 2017 - [info] Binlog server host_1 is reachable.
Thu Nov  9 10:43:50 2017 - [info] HealthCheck: SSH to host_3 is reachable.
Thu Nov  9 10:43:50 2017 - [info] Binlog server host_3 is reachable.
Thu Nov  9 10:43:51 2017 - [warning] SQL Thread is stopped(no error) on host_1(host_1:3306)
Thu Nov  9 10:43:51 2017 - [warning] SQL Thread is stopped(no error) on host_3(host_3:3306)
Thu Nov  9 10:43:51 2017 - [info] GTID failover mode = 1
Thu Nov  9 10:43:51 2017 - [info] Dead Servers:
Thu Nov  9 10:43:51 2017 - [info]   host_2(host_2:3306)
Thu Nov  9 10:43:51 2017 - [info] Checking master reachability via MySQL(double check)...
Thu Nov  9 10:43:51 2017 - [info]  ok.
Thu Nov  9 10:43:51 2017 - [info] Alive Servers:
Thu Nov  9 10:43:51 2017 - [info]   host_1(host_1:3306)
Thu Nov  9 10:43:51 2017 - [info]   host_3(host_3:3306)
Thu Nov  9 10:43:51 2017 - [info] Alive Slaves:
Thu Nov  9 10:43:51 2017 - [info]   host_1(host_1:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Thu Nov  9 10:43:51 2017 - [info]     GTID ON
Thu Nov  9 10:43:51 2017 - [info]     Replicating from host_2(host_2:3306)
Thu Nov  9 10:43:51 2017 - [info]     Primary candidate for the new Master (candidate_master is set)
Thu Nov  9 10:43:51 2017 - [info]   host_3(host_3:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Thu Nov  9 10:43:51 2017 - [info]     GTID ON
Thu Nov  9 10:43:51 2017 - [info]     Replicating from host_2(host_2:3306)
Thu Nov  9 10:43:51 2017 - [info]     Not candidate for the new Master (no_master is set)
Thu Nov  9 10:43:51 2017 - [info]  Starting SQL thread on host_1(host_1:3306) ..
Thu Nov  9 10:43:51 2017 - [info]   done.
Thu Nov  9 10:43:51 2017 - [info]  Starting SQL thread on host_3(host_3:3306) ..
Thu Nov  9 10:43:51 2017 - [info]   done.
Thu Nov  9 10:43:51 2017 - [info] Starting GTID based failover.
Thu Nov  9 10:43:51 2017 - [info]
Thu Nov  9 10:43:51 2017 - [info] ** Phase 1: Configuration Check Phase completed.
Thu Nov  9 10:43:51 2017 - [info]
Thu Nov  9 10:43:51 2017 - [info] * Phase 2: Dead Master Shutdown Phase..
Thu Nov  9 10:43:51 2017 - [info]
Thu Nov  9 10:43:51 2017 - [info] HealthCheck: SSH to host_2 is reachable.
Thu Nov  9 10:43:51 2017 - [info] Forcing shutdown so that applications never connect to the current master..
Thu Nov  9 10:43:51 2017 - [info] Executing master IP deactivation script:
Thu Nov  9 10:43:51 2017 - [info]   /data/online/agent/MHA/masterha/bak_mha_test/master_ip_failover_mha_test --orig_master_host=host_2 --orig_master_ip=host_2 --orig_master_port=3306 --command=stopssh --ssh_user=root
Thu Nov  9 10:43:53 2017 - [info]  done.
Thu Nov  9 10:43:53 2017 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Thu Nov  9 10:43:53 2017 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Thu Nov  9 10:43:53 2017 - [info]
Thu Nov  9 10:43:53 2017 - [info] * Phase 3: Master Recovery Phase..
Thu Nov  9 10:43:53 2017 - [info]
Thu Nov  9 10:43:53 2017 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Thu Nov  9 10:43:53 2017 - [info]
Thu Nov  9 10:43:53 2017 - [info] The latest binary log file/position on all slaves is host_1.000002:1115
Thu Nov  9 10:43:53 2017 - [info] Retrieved Gtid Set: 0923e916-3c36-11e6-82a5-ecf4bbf1f518:4-16,
Thu Nov  9 10:43:53 2017 - [info] Latest slaves (Slaves that received relay log files to the latest):
Thu Nov  9 10:43:53 2017 - [info]   host_3(host_3:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Thu Nov  9 10:43:53 2017 - [info]     GTID ON
Thu Nov  9 10:43:53 2017 - [info]     Replicating from host_2(host_2:3306)
Thu Nov  9 10:43:53 2017 - [info]     Not candidate for the new Master (no_master is set)
Thu Nov  9 10:43:53 2017 - [info] The oldest binary log file/position on all slaves is host_1.000002:230
Thu Nov  9 10:43:53 2017 - [info] Retrieved Gtid Set: ebd9ff93-c5b2-11e6-b21d-ecf4bbf1f42c:446353
Thu Nov  9 10:43:53 2017 - [info] Oldest slaves:
Thu Nov  9 10:43:53 2017 - [info]   host_1(host_1:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Thu Nov  9 10:43:53 2017 - [info]     GTID ON
Thu Nov  9 10:43:53 2017 - [info]     Replicating from host_2(host_2:3306)
Thu Nov  9 10:43:53 2017 - [info]     Primary candidate for the new Master (candidate_master is set)
Thu Nov  9 10:43:53 2017 - [info]
Thu Nov  9 10:43:53 2017 - [info] * Phase 3.3: Determining New Master Phase..
Thu Nov  9 10:43:53 2017 - [info]
Thu Nov  9 10:43:53 2017 - [info] Searching new master from slaves..
Thu Nov  9 10:43:53 2017 - [info]  Candidate masters from the configuration file:
Thu Nov  9 10:43:53 2017 - [info]   host_1(host_1:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Thu Nov  9 10:43:53 2017 - [info]     GTID ON
Thu Nov  9 10:43:53 2017 - [info]     Replicating from host_2(host_2:3306)
Thu Nov  9 10:43:53 2017 - [info]     Primary candidate for the new Master (candidate_master is set)
Thu Nov  9 10:43:53 2017 - [info]  Non-candidate masters:
Thu Nov  9 10:43:53 2017 - [info]   host_3(host_3:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Thu Nov  9 10:43:53 2017 - [info]     GTID ON
Thu Nov  9 10:43:53 2017 - [info]     Replicating from host_2(host_2:3306)
Thu Nov  9 10:43:53 2017 - [info]     Not candidate for the new Master (no_master is set)
Thu Nov  9 10:43:53 2017 - [info]  Searching from candidate_master slaves which have received the latest relay log events..
Thu Nov  9 10:43:53 2017 - [info]   Not found.
Thu Nov  9 10:43:53 2017 - [info]  Searching from all candidate_master slaves..
Thu Nov  9 10:43:53 2017 - [info] New master is host_1(host_1:3306)
Thu Nov  9 10:43:53 2017 - [info] Starting master failover..
Thu Nov  9 10:43:53 2017 - [info]
Thu Nov  9 10:43:53 2017 - [info]
Thu Nov  9 10:43:53 2017 - [info] * Phase 3.3: New Master Recovery Phase..
Thu Nov  9 10:43:53 2017 - [info]
Thu Nov  9 10:43:53 2017 - [info]  Waiting all logs to be applied..
Thu Nov  9 10:43:53 2017 - [info]   done.
Thu Nov  9 10:43:53 2017 - [info]  Replicating from the latest slave host_3(host_3:3306) and waiting to apply..
Thu Nov  9 10:43:53 2017 - [info]  Waiting all logs to be applied on the latest slave..
Thu Nov  9 10:43:53 2017 - [info]  Resetting slave host_1(host_1:3306) and starting replication from the new master host_3(host_3:3306)..
Thu Nov  9 10:43:53 2017 - [info]  Executed CHANGE MASTER.
Thu Nov  9 10:43:54 2017 - [info]  Slave started.
Thu Nov  9 10:43:54 2017 - [info]  Waiting to execute all relay logs on host_1(host_1:3306)..
Thu Nov  9 10:43:54 2017 - [info]  master_pos_wait(host_3.000049:18041) completed on host_1(host_1:3306). Executed 0 events.
Thu Nov  9 10:43:54 2017 - [info]   done.
Thu Nov  9 10:43:54 2017 - [info]   done.
Thu Nov  9 10:43:54 2017 - [info] -- Saving binlog from host host_2 started, pid: 150294
Thu Nov  9 10:43:54 2017 - [info] -- Saving binlog from host host_1 started, pid: 150295
Thu Nov  9 10:43:54 2017 - [info] -- Saving binlog from host host_3 started, pid: 150297
Thu Nov  9 10:43:54 2017 - [info]
Thu Nov  9 10:43:54 2017 - [info] Log messages from host_1 ...
Thu Nov  9 10:43:54 2017 - [info]
Thu Nov  9 10:43:54 2017 - [info] Fetching binary logs from binlog server host_1..
Thu Nov  9 10:43:54 2017 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=host_1.000002  --start_pos=1115 --output_file=/var/log/masterha/mha_test/saved_binlog_binlog2_20171109104349.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.7.13-log  --binlog_dir=/data/mysql.bin
Thu Nov  9 10:43:54 2017 - [error][/usr/share/perl5/vendor_perl/MHA/MasterFailover.pm, ln660] Failed to save binary log events from the binlog server. Maybe disks on binary logs are not accessible or binary log itself is corrupt?
Thu Nov  9 10:43:54 2017 - [info] End of log messages from host_1.
Thu Nov  9 10:43:54 2017 - [warning] Got error from host_1.
Thu Nov  9 10:43:54 2017 - [info]
Thu Nov  9 10:43:54 2017 - [info] Log messages from host_3 ...
Thu Nov  9 10:43:54 2017 - [info]
Thu Nov  9 10:43:54 2017 - [info] Fetching binary logs from binlog server host_3..
Thu Nov  9 10:43:54 2017 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=host_1.000002  --start_pos=1115 --output_file=/var/log/masterha/mha_test/saved_binlog_binlog3_20171109104349.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.7.13-log  --binlog_dir=/data/mysql.bin
Thu Nov  9 10:43:54 2017 - [error][/usr/share/perl5/vendor_perl/MHA/MasterFailover.pm, ln660] Failed to save binary log events from the binlog server. Maybe disks on binary logs are not accessible or binary log itself is corrupt?
Thu Nov  9 10:43:54 2017 - [info] End of log messages from host_3.
Thu Nov  9 10:43:54 2017 - [warning] Got error from host_3.
Thu Nov  9 10:43:55 2017 - [info]
Thu Nov  9 10:43:55 2017 - [info] Log messages from host_2 ...
Thu Nov  9 10:43:55 2017 - [info]
Thu Nov  9 10:43:54 2017 - [info] Fetching binary logs from binlog server host_2..
Thu Nov  9 10:43:54 2017 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=host_1.000002  --start_pos=1115 --output_file=/var/log/masterha/mha_test/saved_binlog_binlog1_20171109104349.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.7.13-log  --binlog_dir=/data/mysql.bin
Thu Nov  9 10:43:55 2017 - [info] scp from root@host_2:/var/log/masterha/mha_test/saved_binlog_binlog1_20171109104349.binlog to local:/var/log/masterha/mha_test/saved_binlog_host_2_binlog1_20171109104349.binlog succeeded.
Thu Nov  9 10:43:55 2017 - [info] End of log messages from host_2.
Thu Nov  9 10:43:55 2017 - [info] Saved mysqlbinlog size from host_2 is 6047 bytes.
Thu Nov  9 10:43:55 2017 - [info] Applying differential binlog /var/log/masterha/mha_test/saved_binlog_host_2_binlog1_20171109104349.binlog ..
Thu Nov  9 10:43:55 2017 - [info] Differential log apply from binlog server succeeded.
Thu Nov  9 10:43:55 2017 - [info] Getting new master's binlog name and position..
Thu Nov  9 10:43:55 2017 - [info]  tjtx-126-164.000053:3624
Thu Nov  9 10:43:55 2017 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='host_1', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Thu Nov  9 10:43:55 2017 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: tjtx-126-164.000053, 3624, 0923e916-3c36-11e6-82a5-ecf4bbf1f518:1-16,
Thu Nov  9 10:43:55 2017 - [info] Executing master IP activate script:
Thu Nov  9 10:43:55 2017 - [info]   /data/online/agent/MHA/masterha/bak_mha_test/master_ip_failover_mha_test --command=start --ssh_user=root --orig_master_host=host_2 --orig_master_ip=host_2 --orig_master_port=3306 --new_master_host=host_1 --new_master_ip=host_1 --new_master_port=3306 --new_master_user='xxx' --new_master_password='xxx'
Thu Nov  9 10:43:57 2017 - [info]  OK.
Thu Nov  9 10:43:57 2017 - [info] Setting read_only=0 on host_1(host_1:3306)..
Thu Nov  9 10:43:57 2017 - [info]  ok.
Thu Nov  9 10:43:57 2017 - [info] ** Finished master recovery successfully.
Thu Nov  9 10:43:57 2017 - [info] * Phase 3: Master Recovery Phase completed.
Thu Nov  9 10:43:57 2017 - [info]
Thu Nov  9 10:43:57 2017 - [info] * Phase 4: Slaves Recovery Phase..
Thu Nov  9 10:43:57 2017 - [info]
Thu Nov  9 10:43:57 2017 - [info]
Thu Nov  9 10:43:57 2017 - [info] * Phase 4.1: Starting Slaves in parallel..
Thu Nov  9 10:43:57 2017 - [info]
Thu Nov  9 10:43:57 2017 - [info] -- Slave recovery on host host_3(host_3:3306) started, pid: 155162. Check tmp log /var/log/masterha/mha_test/host_3_3306_20171109104349.log if it takes time..
Thu Nov  9 10:43:58 2017 - [info]
Thu Nov  9 10:43:58 2017 - [info] Log messages from host_3 ...
Thu Nov  9 10:43:58 2017 - [info]
Thu Nov  9 10:43:57 2017 - [info]  Resetting slave host_3(host_3:3306) and starting replication from the new master host_1(host_1:3306)..
Thu Nov  9 10:43:57 2017 - [info]  Executed CHANGE MASTER.
Thu Nov  9 10:43:58 2017 - [info]  Slave started.
Thu Nov  9 10:43:58 2017 - [info]  gtid_wait(0923e916-3c36-11e6-82a5-ecf4bbf1f518:1-16,
Thu Nov  9 10:43:58 2017 - [info] End of log messages from host_3.
Thu Nov  9 10:43:58 2017 - [info] -- Slave on host host_3(host_3:3306) started.
Thu Nov  9 10:43:58 2017 - [info] All new slave servers recovered successfully.
Thu Nov  9 10:43:58 2017 - [info]
Thu Nov  9 10:43:58 2017 - [info] * Phase 5: New master cleanup phase..
Thu Nov  9 10:43:58 2017 - [info]
Thu Nov  9 10:43:58 2017 - [info] Resetting slave info on the new master..
Thu Nov  9 10:43:58 2017 - [info]  host_1: Resetting slave info succeeded.
Thu Nov  9 10:43:58 2017 - [info] Master failover to host_1(host_1:3306) completed successfully.
Thu Nov  9 10:43:58 2017 - [info]
Thu Nov  9 10:43:58 2017 - [info] Sending mail..

1.2.2 ???master??????????????????????????????1???etl????????????master ??????MySQL??????


### ????????????????????????3???DB gtid??????

* master host_1

dba:lc> show master status;
+---------------------+----------+--------------+------------------+------------------------------------------------------------------------------------------+
| File                | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                                                                        |
+---------------------+----------+--------------+------------------+------------------------------------------------------------------------------------------+
| tjtx-126-164.000053 |     5229 |              |                  | 0923e916-3c36-11e6-82a5-ecf4bbf1f518:1-21,
ebd9ff93-c5b2-11e6-b21d-ecf4bbf1f42c:1-446362 |
+---------------------+----------+--------------+------------------+------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)



* slave (candidate master) host_2

           Retrieved_Gtid_Set:
            Executed_Gtid_Set: 0923e916-3c36-11e6-82a5-ecf4bbf1f518:1-16,
ebd9ff93-c5b2-11e6-b21d-ecf4bbf1f42c:1-446362
                Auto_Position: 1

* etl (other slave) host_3

           Retrieved_Gtid_Set: 0923e916-3c36-11e6-82a5-ecf4bbf1f518:17-21,
ebd9ff93-c5b2-11e6-b21d-ecf4bbf1f42c:446357-446362
            Executed_Gtid_Set: 0923e916-3c36-11e6-82a5-ecf4bbf1f518:1-21,
ebd9ff93-c5b2-11e6-b21d-ecf4bbf1f42c:1-446362
                Auto_Position: 1


### ????????????

masterha_master_switch --global_conf=/data/online/agent/MHA/conf/masterha_default.cnf --conf=/data/online/agent/MHA/conf/bak_mha_test.cnf  --dead_master_host=host_1  --dead_master_port=3306 --master_state=dead --interactive=0 --ignore_last_failover --ignore_binlog_server_error

Thu Nov  9 10:59:14 2017 - [info] MHA::MasterFailover version 0.56.
Thu Nov  9 10:59:14 2017 - [info] Starting master failover.
Thu Nov  9 10:59:14 2017 - [info]
Thu Nov  9 10:59:14 2017 - [info] * Phase 1: Configuration Check Phase..
Thu Nov  9 10:59:14 2017 - [info]
Thu Nov  9 10:59:15 2017 - [info] HealthCheck: SSH to host_2 is reachable.
Thu Nov  9 10:59:15 2017 - [info] Binlog server host_2 is reachable.
Thu Nov  9 10:59:15 2017 - [info] HealthCheck: SSH to host_1 is reachable.
Thu Nov  9 10:59:15 2017 - [info] Binlog server host_1 is reachable.
Thu Nov  9 10:59:15 2017 - [info] HealthCheck: SSH to host_3 is reachable.
Thu Nov  9 10:59:16 2017 - [info] Binlog server host_3 is reachable.
Thu Nov  9 10:59:16 2017 - [warning] SQL Thread is stopped(no error) on host_2(host_2:3306)
Thu Nov  9 10:59:16 2017 - [info] GTID failover mode = 1
Thu Nov  9 10:59:16 2017 - [info] Dead Servers:
Thu Nov  9 10:59:16 2017 - [info]   host_1(host_1:3306)
Thu Nov  9 10:59:16 2017 - [info] Checking master reachability via MySQL(double check)...
Thu Nov  9 10:59:16 2017 - [info]  ok.
Thu Nov  9 10:59:16 2017 - [info] Alive Servers:
Thu Nov  9 10:59:16 2017 - [info]   host_2(host_2:3306)
Thu Nov  9 10:59:16 2017 - [info]   host_3(host_3:3306)
Thu Nov  9 10:59:16 2017 - [info] Alive Slaves:
Thu Nov  9 10:59:16 2017 - [info]   host_2(host_2:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Thu Nov  9 10:59:16 2017 - [info]     GTID ON
Thu Nov  9 10:59:16 2017 - [info]     Replicating from host_1(host_1:3306)
Thu Nov  9 10:59:16 2017 - [info]     Primary candidate for the new Master (candidate_master is set)
Thu Nov  9 10:59:16 2017 - [info]   host_3(host_3:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Thu Nov  9 10:59:16 2017 - [info]     GTID ON
Thu Nov  9 10:59:16 2017 - [info]     Replicating from host_1(host_1:3306)
Thu Nov  9 10:59:16 2017 - [info]     Not candidate for the new Master (no_master is set)
Thu Nov  9 10:59:16 2017 - [info]  Starting SQL thread on host_2(host_2:3306) ..
Thu Nov  9 10:59:16 2017 - [info]   done.
Thu Nov  9 10:59:16 2017 - [info] Starting GTID based failover.
Thu Nov  9 10:59:16 2017 - [info]
Thu Nov  9 10:59:16 2017 - [info] ** Phase 1: Configuration Check Phase completed.
Thu Nov  9 10:59:16 2017 - [info]
Thu Nov  9 10:59:16 2017 - [info] * Phase 2: Dead Master Shutdown Phase..
Thu Nov  9 10:59:16 2017 - [info]
Thu Nov  9 10:59:16 2017 - [info] HealthCheck: SSH to host_1 is reachable.
Thu Nov  9 10:59:16 2017 - [info] Forcing shutdown so that applications never connect to the current master..
Thu Nov  9 10:59:16 2017 - [info] Executing master IP deactivation script:
Thu Nov  9 10:59:16 2017 - [info]   /data/online/agent/MHA/masterha/bak_mha_test/master_ip_failover_mha_test --orig_master_host=host_1 --orig_master_ip=host_1 --orig_master_port=3306 --command=stopssh --ssh_user=root
Thu Nov  9 10:59:20 2017 - [info]  done.
Thu Nov  9 10:59:20 2017 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Thu Nov  9 10:59:20 2017 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Thu Nov  9 10:59:20 2017 - [info]
Thu Nov  9 10:59:20 2017 - [info] * Phase 3: Master Recovery Phase..
Thu Nov  9 10:59:20 2017 - [info]
Thu Nov  9 10:59:20 2017 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Thu Nov  9 10:59:20 2017 - [info]
Thu Nov  9 10:59:20 2017 - [info] The latest binary log file/position on all slaves is tjtx-126-164.000053:5229
Thu Nov  9 10:59:20 2017 - [info] Retrieved Gtid Set: 0923e916-3c36-11e6-82a5-ecf4bbf1f518:17-21,
Thu Nov  9 10:59:20 2017 - [info] Latest slaves (Slaves that received relay log files to the latest):
Thu Nov  9 10:59:20 2017 - [info]   host_3(host_3:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Thu Nov  9 10:59:20 2017 - [info]     GTID ON
Thu Nov  9 10:59:20 2017 - [info]     Replicating from host_1(host_1:3306)
Thu Nov  9 10:59:20 2017 - [info]     Not candidate for the new Master (no_master is set)
Thu Nov  9 10:59:20 2017 - [info] The oldest binary log file/position on all slaves is tjtx-126-164.000053:3624
Thu Nov  9 10:59:20 2017 - [info] Oldest slaves:
Thu Nov  9 10:59:20 2017 - [info]   host_2(host_2:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Thu Nov  9 10:59:20 2017 - [info]     GTID ON
Thu Nov  9 10:59:20 2017 - [info]     Replicating from host_1(host_1:3306)
Thu Nov  9 10:59:20 2017 - [info]     Primary candidate for the new Master (candidate_master is set)
Thu Nov  9 10:59:20 2017 - [info]
Thu Nov  9 10:59:20 2017 - [info] * Phase 3.3: Determining New Master Phase..
Thu Nov  9 10:59:20 2017 - [info]
Thu Nov  9 10:59:20 2017 - [info] Searching new master from slaves..
Thu Nov  9 10:59:20 2017 - [info]  Candidate masters from the configuration file:
Thu Nov  9 10:59:20 2017 - [info]   host_2(host_2:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Thu Nov  9 10:59:20 2017 - [info]     GTID ON
Thu Nov  9 10:59:20 2017 - [info]     Replicating from host_1(host_1:3306)
Thu Nov  9 10:59:20 2017 - [info]     Primary candidate for the new Master (candidate_master is set)
Thu Nov  9 10:59:20 2017 - [info]  Non-candidate masters:
Thu Nov  9 10:59:20 2017 - [info]   host_3(host_3:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Thu Nov  9 10:59:20 2017 - [info]     GTID ON
Thu Nov  9 10:59:20 2017 - [info]     Replicating from host_1(host_1:3306)
Thu Nov  9 10:59:20 2017 - [info]     Not candidate for the new Master (no_master is set)
Thu Nov  9 10:59:20 2017 - [info]  Searching from candidate_master slaves which have received the latest relay log events..
Thu Nov  9 10:59:20 2017 - [info]   Not found.
Thu Nov  9 10:59:20 2017 - [info]  Searching from all candidate_master slaves..
Thu Nov  9 10:59:20 2017 - [info] New master is host_2(host_2:3306)
Thu Nov  9 10:59:20 2017 - [info] Starting master failover..
Thu Nov  9 10:59:20 2017 - [info]
Thu Nov  9 10:59:20 2017 - [info]
Thu Nov  9 10:59:20 2017 - [info] * Phase 3.3: New Master Recovery Phase..
Thu Nov  9 10:59:20 2017 - [info]
Thu Nov  9 10:59:20 2017 - [info]  Waiting all logs to be applied..
Thu Nov  9 10:59:20 2017 - [info]   done.
Thu Nov  9 10:59:20 2017 - [info]  Replicating from the latest slave host_3(host_3:3306) and waiting to apply..
Thu Nov  9 10:59:20 2017 - [info]  Waiting all logs to be applied on the latest slave..
Thu Nov  9 10:59:20 2017 - [info]  Resetting slave host_2(host_2:3306) and starting replication from the new master host_3(host_3:3306)..
Thu Nov  9 10:59:20 2017 - [info]  Executed CHANGE MASTER.
Thu Nov  9 10:59:21 2017 - [info]  Slave started.
Thu Nov  9 10:59:21 2017 - [info]  Waiting to execute all relay logs on host_2(host_2:3306)..
Thu Nov  9 10:59:21 2017 - [info]  master_pos_wait(host_3.000049:22035) completed on host_2(host_2:3306). Executed 0 events.
Thu Nov  9 10:59:21 2017 - [info]   done.
Thu Nov  9 10:59:21 2017 - [info]   done.
Thu Nov  9 10:59:21 2017 - [info] -- Saving binlog from host host_2 started, pid: 184482
Thu Nov  9 10:59:21 2017 - [info] -- Saving binlog from host host_1 started, pid: 184483
Thu Nov  9 10:59:21 2017 - [info] -- Saving binlog from host host_3 started, pid: 184487
Thu Nov  9 10:59:21 2017 - [info]
Thu Nov  9 10:59:21 2017 - [info] Log messages from host_2 ...
Thu Nov  9 10:59:21 2017 - [info]
Thu Nov  9 10:59:21 2017 - [info] Fetching binary logs from binlog server host_2..
Thu Nov  9 10:59:21 2017 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=tjtx-126-164.000053  --start_pos=5229 --output_file=/var/log/masterha/mha_test/saved_binlog_binlog1_20171109105914.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.7.13-log  --binlog_dir=/data/mysql.bin
Thu Nov  9 10:59:21 2017 - [error][/usr/share/perl5/vendor_perl/MHA/MasterFailover.pm, ln660] Failed to save binary log events from the binlog server. Maybe disks on binary logs are not accessible or binary log itself is corrupt?
Thu Nov  9 10:59:21 2017 - [info] End of log messages from host_2.
Thu Nov  9 10:59:21 2017 - [warning] Got error from host_2.
Thu Nov  9 10:59:21 2017 - [info]
Thu Nov  9 10:59:21 2017 - [info] Log messages from host_3 ...
Thu Nov  9 10:59:21 2017 - [info]
Thu Nov  9 10:59:21 2017 - [info] Fetching binary logs from binlog server host_3..
Thu Nov  9 10:59:21 2017 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=tjtx-126-164.000053  --start_pos=5229 --output_file=/var/log/masterha/mha_test/saved_binlog_binlog3_20171109105914.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.7.13-log  --binlog_dir=/data/mysql.bin
Thu Nov  9 10:59:21 2017 - [error][/usr/share/perl5/vendor_perl/MHA/MasterFailover.pm, ln660] Failed to save binary log events from the binlog server. Maybe disks on binary logs are not accessible or binary log itself is corrupt?
Thu Nov  9 10:59:21 2017 - [info] End of log messages from host_3.
Thu Nov  9 10:59:21 2017 - [warning] Got error from host_3.
Thu Nov  9 10:59:22 2017 - [info]
Thu Nov  9 10:59:22 2017 - [info] Log messages from host_1 ...
Thu Nov  9 10:59:22 2017 - [info]
Thu Nov  9 10:59:21 2017 - [info] Fetching binary logs from binlog server host_1..
Thu Nov  9 10:59:21 2017 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=tjtx-126-164.000053  --start_pos=5229 --output_file=/var/log/masterha/mha_test/saved_binlog_binlog2_20171109105914.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.7.13-log  --binlog_dir=/data/mysql.bin
Thu Nov  9 10:59:22 2017 - [info] scp from root@host_1:/var/log/masterha/mha_test/saved_binlog_binlog2_20171109105914.binlog to local:/var/log/masterha/mha_test/saved_binlog_host_1_binlog2_20171109105914.binlog succeeded.
Thu Nov  9 10:59:22 2017 - [info] End of log messages from host_1.
Thu Nov  9 10:59:22 2017 - [info] Saved mysqlbinlog size from host_1 is 800 bytes.
Thu Nov  9 10:59:22 2017 - [info] Applying differential binlog /var/log/masterha/mha_test/saved_binlog_host_1_binlog2_20171109105914.binlog ..
Thu Nov  9 10:59:22 2017 - [info] Differential log apply from binlog server succeeded.
Thu Nov  9 10:59:22 2017 - [info] Getting new master's binlog name and position..
Thu Nov  9 10:59:22 2017 - [info]  host_1.000003:1680
Thu Nov  9 10:59:22 2017 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='host_2', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Thu Nov  9 10:59:22 2017 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: host_1.000003, 1680, 0923e916-3c36-11e6-82a5-ecf4bbf1f518:1-21,
Thu Nov  9 10:59:22 2017 - [info] Executing master IP activate script:
Thu Nov  9 10:59:22 2017 - [info]   /data/online/agent/MHA/masterha/bak_mha_test/master_ip_failover_mha_test --command=start --ssh_user=root --orig_master_host=host_1 --orig_master_ip=host_1 --orig_master_port=3306 --new_master_host=host_2 --new_master_ip=host_2 --new_master_port=3306 --new_master_user='xxx' --new_master_password='xxx'
Thu Nov  9 10:59:24 2017 - [info]  OK.
Thu Nov  9 10:59:24 2017 - [info] Setting read_only=0 on host_2(host_2:3306)..
Thu Nov  9 10:59:24 2017 - [info]  ok.
Thu Nov  9 10:59:24 2017 - [info] ** Finished master recovery successfully.
Thu Nov  9 10:59:24 2017 - [info] * Phase 3: Master Recovery Phase completed.
Thu Nov  9 10:59:24 2017 - [info]
Thu Nov  9 10:59:24 2017 - [info] * Phase 4: Slaves Recovery Phase..
Thu Nov  9 10:59:24 2017 - [info]
Thu Nov  9 10:59:24 2017 - [info]
Thu Nov  9 10:59:24 2017 - [info] * Phase 4.1: Starting Slaves in parallel..
Thu Nov  9 10:59:24 2017 - [info]
Thu Nov  9 10:59:24 2017 - [info] -- Slave recovery on host host_3(host_3:3306) started, pid: 189393. Check tmp log /var/log/masterha/mha_test/host_3_3306_20171109105914.log if it takes time..
Thu Nov  9 10:59:25 2017 - [info]
Thu Nov  9 10:59:25 2017 - [info] Log messages from host_3 ...
Thu Nov  9 10:59:25 2017 - [info]
Thu Nov  9 10:59:24 2017 - [info]  Resetting slave host_3(host_3:3306) and starting replication from the new master host_2(host_2:3306)..
Thu Nov  9 10:59:24 2017 - [info]  Executed CHANGE MASTER.
Thu Nov  9 10:59:25 2017 - [info]  Slave started.
Thu Nov  9 10:59:25 2017 - [info]  gtid_wait(0923e916-3c36-11e6-82a5-ecf4bbf1f518:1-21,
Thu Nov  9 10:59:25 2017 - [info] End of log messages from host_3.
Thu Nov  9 10:59:25 2017 - [info] -- Slave on host host_3(host_3:3306) started.
Thu Nov  9 10:59:25 2017 - [info] All new slave servers recovered successfully.
Thu Nov  9 10:59:25 2017 - [info]
Thu Nov  9 10:59:25 2017 - [info] * Phase 5: New master cleanup phase..
Thu Nov  9 10:59:25 2017 - [info]
Thu Nov  9 10:59:25 2017 - [info] Resetting slave info on the new master..
Thu Nov  9 10:59:25 2017 - [info]  host_2: Resetting slave info succeeded.
Thu Nov  9 10:59:25 2017 - [info] Master failover to host_2(host_2:3306) completed successfully.
Thu Nov  9 10:59:25 2017 - [info]
Thu Nov  9 10:59:25 2017 - [info] Sending mail..

1.3 slave(??????master)???????????????????????????etl??????

1.3.1 ???master?????????????????????????????????slave????????????master ??????MySQL??????

masterha_master_switch --global_conf=/data/online/agent/MHA/conf/masterha_default.cnf --conf=/data/online/agent/MHA/conf/bak_mha_test.cnf  --dead_master_host=host_1  --dead_master_port=3306 --master_state=dead --interactive=0 --ignore_last_failover --ignore_binlog_server_error


Tue Nov  7 17:11:29 2017 - [info] MHA::MasterFailover version 0.56.
Tue Nov  7 17:11:29 2017 - [info] Starting master failover.
Tue Nov  7 17:11:29 2017 - [info]
Tue Nov  7 17:11:29 2017 - [info] * Phase 1: Configuration Check Phase..
Tue Nov  7 17:11:29 2017 - [info]
Tue Nov  7 17:11:29 2017 - [info] HealthCheck: SSH to host_2 is reachable.
Tue Nov  7 17:11:29 2017 - [info] Binlog server host_2 is reachable.
Tue Nov  7 17:11:29 2017 - [info] HealthCheck: SSH to host_1 is reachable.
Tue Nov  7 17:11:30 2017 - [info] Binlog server host_1 is reachable.
Tue Nov  7 17:11:30 2017 - [info] HealthCheck: SSH to host_3 is reachable.
Tue Nov  7 17:11:30 2017 - [info] Binlog server host_3 is reachable.
Tue Nov  7 17:11:30 2017 - [warning] SQL Thread is stopped(no error) on host_2(host_2:3306)
Tue Nov  7 17:11:30 2017 - [warning] SQL Thread is stopped(no error) on host_3(host_3:3306)
Tue Nov  7 17:11:30 2017 - [info] GTID failover mode = 1
Tue Nov  7 17:11:30 2017 - [info] Dead Servers:
Tue Nov  7 17:11:30 2017 - [info]   host_1(host_1:3306)
Tue Nov  7 17:11:30 2017 - [info] Checking master reachability via MySQL(double check)...
Tue Nov  7 17:11:30 2017 - [info]  ok.
Tue Nov  7 17:11:30 2017 - [info] Alive Servers:
Tue Nov  7 17:11:30 2017 - [info]   host_2(host_2:3306)
Tue Nov  7 17:11:30 2017 - [info]   host_3(host_3:3306)
Tue Nov  7 17:11:30 2017 - [info] Alive Slaves:
Tue Nov  7 17:11:30 2017 - [info]   host_2(host_2:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Tue Nov  7 17:11:30 2017 - [info]     GTID ON
Tue Nov  7 17:11:30 2017 - [info]     Replicating from host_1(host_1:3306)
Tue Nov  7 17:11:30 2017 - [info]     Primary candidate for the new Master (candidate_master is set)
Tue Nov  7 17:11:30 2017 - [info]   host_3(host_3:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Tue Nov  7 17:11:30 2017 - [info]     GTID ON
Tue Nov  7 17:11:30 2017 - [info]     Replicating from host_1(host_1:3306)
Tue Nov  7 17:11:30 2017 - [info]     Not candidate for the new Master (no_master is set)
Tue Nov  7 17:11:30 2017 - [info]  Starting SQL thread on host_2(host_2:3306) ..
Tue Nov  7 17:11:30 2017 - [info]   done.
Tue Nov  7 17:11:30 2017 - [info]  Starting SQL thread on host_3(host_3:3306) ..
Tue Nov  7 17:11:30 2017 - [info]   done.
Tue Nov  7 17:11:30 2017 - [info] Starting GTID based failover.
Tue Nov  7 17:11:30 2017 - [info]
Tue Nov  7 17:11:30 2017 - [info] ** Phase 1: Configuration Check Phase completed.
Tue Nov  7 17:11:30 2017 - [info]
Tue Nov  7 17:11:30 2017 - [info] * Phase 2: Dead Master Shutdown Phase..
Tue Nov  7 17:11:30 2017 - [info]
Tue Nov  7 17:11:30 2017 - [info] HealthCheck: SSH to host_1 is reachable.
Tue Nov  7 17:11:31 2017 - [info] Forcing shutdown so that applications never connect to the current master..
Tue Nov  7 17:11:31 2017 - [info] Executing master IP deactivation script:
Tue Nov  7 17:11:31 2017 - [info]   /data/online/agent/MHA/masterha/bak_mha_test/master_ip_failover_mha_test --orig_master_host=host_1 --orig_master_ip=host_1 --orig_master_port=3306 --command=stopssh --ssh_user=root
Tue Nov  7 17:11:33 2017 - [info]  done.
Tue Nov  7 17:11:33 2017 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Tue Nov  7 17:11:33 2017 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Tue Nov  7 17:11:33 2017 - [info]
Tue Nov  7 17:11:33 2017 - [info] * Phase 3: Master Recovery Phase..
Tue Nov  7 17:11:33 2017 - [info]
Tue Nov  7 17:11:33 2017 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Tue Nov  7 17:11:33 2017 - [info]
Tue Nov  7 17:11:33 2017 - [info] The latest binary log file/position on all slaves is tjtx-126-164.000051:13508
Tue Nov  7 17:11:33 2017 - [info] Retrieved Gtid Set: 0923e916-3c36-11e6-82a5-ecf4bbf1f518:3-8
Tue Nov  7 17:11:33 2017 - [info] Latest slaves (Slaves that received relay log files to the latest):
Tue Nov  7 17:11:33 2017 - [info]   host_2(host_2:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Tue Nov  7 17:11:33 2017 - [info]     GTID ON
Tue Nov  7 17:11:33 2017 - [info]     Replicating from host_1(host_1:3306)
Tue Nov  7 17:11:33 2017 - [info]     Primary candidate for the new Master (candidate_master is set)
Tue Nov  7 17:11:33 2017 - [info] The oldest binary log file/position on all slaves is tjtx-126-164.000051:11918
Tue Nov  7 17:11:33 2017 - [info] Retrieved Gtid Set: 0923e916-3c36-11e6-82a5-ecf4bbf1f518:2-3,
Tue Nov  7 17:11:33 2017 - [info] Oldest slaves:
Tue Nov  7 17:11:33 2017 - [info]   host_3(host_3:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Tue Nov  7 17:11:33 2017 - [info]     GTID ON
Tue Nov  7 17:11:33 2017 - [info]     Replicating from host_1(host_1:3306)
Tue Nov  7 17:11:33 2017 - [info]     Not candidate for the new Master (no_master is set)
Tue Nov  7 17:11:33 2017 - [info]
Tue Nov  7 17:11:33 2017 - [info] * Phase 3.3: Determining New Master Phase..
Tue Nov  7 17:11:33 2017 - [info]
Tue Nov  7 17:11:33 2017 - [info] Searching new master from slaves..
Tue Nov  7 17:11:33 2017 - [info]  Candidate masters from the configuration file:
Tue Nov  7 17:11:33 2017 - [info]   host_2(host_2:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Tue Nov  7 17:11:33 2017 - [info]     GTID ON
Tue Nov  7 17:11:33 2017 - [info]     Replicating from host_1(host_1:3306)
Tue Nov  7 17:11:33 2017 - [info]     Primary candidate for the new Master (candidate_master is set)
Tue Nov  7 17:11:33 2017 - [info]  Non-candidate masters:
Tue Nov  7 17:11:33 2017 - [info]   host_3(host_3:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Tue Nov  7 17:11:33 2017 - [info]     GTID ON
Tue Nov  7 17:11:33 2017 - [info]     Replicating from host_1(host_1:3306)
Tue Nov  7 17:11:33 2017 - [info]     Not candidate for the new Master (no_master is set)
Tue Nov  7 17:11:33 2017 - [info]  Searching from candidate_master slaves which have received the latest relay log events..
Tue Nov  7 17:11:33 2017 - [info] New master is host_2(host_2:3306)
Tue Nov  7 17:11:33 2017 - [info] Starting master failover..
Tue Nov  7 17:11:33 2017 - [info]
Tue Nov  7 17:11:33 2017 - [info]
Tue Nov  7 17:11:33 2017 - [info] * Phase 3.3: New Master Recovery Phase..
Tue Nov  7 17:11:33 2017 - [info]
Tue Nov  7 17:11:33 2017 - [info]  Waiting all logs to be applied..
Tue Nov  7 17:11:33 2017 - [info]   done.
Tue Nov  7 17:11:33 2017 - [info] -- Saving binlog from host host_2 started, pid: 54677
Tue Nov  7 17:11:33 2017 - [info] -- Saving binlog from host host_1 started, pid: 54681
Tue Nov  7 17:11:33 2017 - [info] -- Saving binlog from host host_3 started, pid: 54683
Tue Nov  7 17:11:33 2017 - [info]
Tue Nov  7 17:11:33 2017 - [info] Log messages from host_3 ...
Tue Nov  7 17:11:33 2017 - [info]
Tue Nov  7 17:11:33 2017 - [info] Fetching binary logs from binlog server host_3..
Tue Nov  7 17:11:33 2017 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=tjtx-126-164.000051  --start_pos=13508 --output_file=/var/log/masterha/mha_test/saved_binlog_binlog3_20171107171129.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.7.13-log  --binlog_dir=/data/mysql.bin
Tue Nov  7 17:11:33 2017 - [error][/usr/share/perl5/vendor_perl/MHA/MasterFailover.pm, ln660] Failed to save binary log events from the binlog server. Maybe disks on binary logs are not accessible or binary log itself is corrupt?
Tue Nov  7 17:11:33 2017 - [info] End of log messages from host_3.
Tue Nov  7 17:11:33 2017 - [warning] Got error from host_3.
Tue Nov  7 17:11:33 2017 - [info]
Tue Nov  7 17:11:33 2017 - [info] Log messages from host_2 ...
Tue Nov  7 17:11:33 2017 - [info]
Tue Nov  7 17:11:33 2017 - [info] Fetching binary logs from binlog server host_2..
Tue Nov  7 17:11:33 2017 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=tjtx-126-164.000051  --start_pos=13508 --output_file=/var/log/masterha/mha_test/saved_binlog_binlog1_20171107171129.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.7.13-log  --binlog_dir=/data/mysql.bin
Tue Nov  7 17:11:33 2017 - [error][/usr/share/perl5/vendor_perl/MHA/MasterFailover.pm, ln660] Failed to save binary log events from the binlog server. Maybe disks on binary logs are not accessible or binary log itself is corrupt?
Tue Nov  7 17:11:33 2017 - [info] End of log messages from host_2.
Tue Nov  7 17:11:33 2017 - [warning] Got error from host_2.
Tue Nov  7 17:11:33 2017 - [info]
Tue Nov  7 17:11:33 2017 - [info] Log messages from host_1 ...
Tue Nov  7 17:11:33 2017 - [info]
Tue Nov  7 17:11:33 2017 - [info] Fetching binary logs from binlog server host_1..
Tue Nov  7 17:11:33 2017 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=tjtx-126-164.000051  --start_pos=13508 --output_file=/var/log/masterha/mha_test/saved_binlog_binlog2_20171107171129.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.7.13-log  --binlog_dir=/data/mysql.bin
Tue Nov  7 17:11:33 2017 - [info] scp from root@host_1:/var/log/masterha/mha_test/saved_binlog_binlog2_20171107171129.binlog to local:/var/log/masterha/mha_test/saved_binlog_host_1_binlog2_20171107171129.binlog succeeded.
Tue Nov  7 17:11:33 2017 - [info] End of log messages from host_1.
Tue Nov  7 17:11:33 2017 - [info] Saved mysqlbinlog size from host_1 is 8578 bytes.
Tue Nov  7 17:11:33 2017 - [info] Applying differential binlog /var/log/masterha/mha_test/saved_binlog_host_1_binlog2_20171107171129.binlog ..
Tue Nov  7 17:11:33 2017 - [info] Differential log apply from binlog server succeeded.
Tue Nov  7 17:11:33 2017 - [info] Getting new master's binlog name and position..
Tue Nov  7 17:11:33 2017 - [info]  host_1.000001:5048
Tue Nov  7 17:11:33 2017 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='host_2', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Tue Nov  7 17:11:33 2017 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: host_1.000001, 5048, 0923e916-3c36-11e6-82a5-ecf4bbf1f518:1-16,
Tue Nov  7 17:11:33 2017 - [info] Executing master IP activate script:
Tue Nov  7 17:11:33 2017 - [info]   /data/online/agent/MHA/masterha/bak_mha_test/master_ip_failover_mha_test --command=start --ssh_user=root --orig_master_host=host_1 --orig_master_ip=host_1 --orig_master_port=3306 --new_master_host=host_2 --new_master_ip=host_2 --new_master_port=3306 --new_master_user='xxx' --new_master_password='xxx'
Tue Nov  7 17:11:36 2017 - [info]  OK.
Tue Nov  7 17:11:36 2017 - [info] Setting read_only=0 on host_2(host_2:3306)..
Tue Nov  7 17:11:36 2017 - [info]  ok.
Tue Nov  7 17:11:36 2017 - [info] ** Finished master recovery successfully.
Tue Nov  7 17:11:36 2017 - [info] * Phase 3: Master Recovery Phase completed.
Tue Nov  7 17:11:36 2017 - [info]
Tue Nov  7 17:11:36 2017 - [info] * Phase 4: Slaves Recovery Phase..
Tue Nov  7 17:11:36 2017 - [info]
Tue Nov  7 17:11:36 2017 - [info]
Tue Nov  7 17:11:36 2017 - [info] * Phase 4.1: Starting Slaves in parallel..
Tue Nov  7 17:11:36 2017 - [info]
Tue Nov  7 17:11:36 2017 - [info] -- Slave recovery on host host_3(host_3:3306) started, pid: 58422. Check tmp log /var/log/masterha/mha_test/host_3_3306_20171107171129.log if it takes time..
Tue Nov  7 17:11:37 2017 - [info]
Tue Nov  7 17:11:37 2017 - [info] Log messages from host_3 ...
Tue Nov  7 17:11:37 2017 - [info]
Tue Nov  7 17:11:36 2017 - [info]  Resetting slave host_3(host_3:3306) and starting replication from the new master host_2(host_2:3306)..
Tue Nov  7 17:11:36 2017 - [info]  Executed CHANGE MASTER.
Tue Nov  7 17:11:37 2017 - [info]  Slave started.
Tue Nov  7 17:11:37 2017 - [info]  gtid_wait(0923e916-3c36-11e6-82a5-ecf4bbf1f518:1-16,
Tue Nov  7 17:11:37 2017 - [info] End of log messages from host_3.
Tue Nov  7 17:11:37 2017 - [info] -- Slave on host host_3(host_3:3306) started.
Tue Nov  7 17:11:37 2017 - [info] All new slave servers recovered successfully.
Tue Nov  7 17:11:37 2017 - [info]
Tue Nov  7 17:11:37 2017 - [info] * Phase 5: New master cleanup phase..
Tue Nov  7 17:11:37 2017 - [info]
Tue Nov  7 17:11:37 2017 - [info] Resetting slave info on the new master..
Tue Nov  7 17:11:37 2017 - [info]  host_2: Resetting slave info succeeded.
Tue Nov  7 17:11:37 2017 - [info] Master failover to host_2(host_2:3306) completed successfully.
Tue Nov  7 17:11:37 2017 - [info]
Tue Nov  7 17:11:37 2017 - [info] Sending mail..

1.3.2 ???master???????????????????????????slave????????????master ??????MySQL??????

masterha_master_switch --global_conf=/data/online/agent/MHA/conf/masterha_default.cnf --conf=/data/online/agent/MHA/conf/bak_mha_test.cnf  --dead_master_host=host_2  --dead_master_port=3306 --master_state=dead --interactive=0 --ignore_last_failover --ignore_binlog_server_error


Tue Nov  7 15:56:11 2017 - [info] MHA::MasterFailover version 0.56.
Tue Nov  7 15:56:11 2017 - [info] Starting master failover.
Tue Nov  7 15:56:11 2017 - [info]
Tue Nov  7 15:56:11 2017 - [info] * Phase 1: Configuration Check Phase..
Tue Nov  7 15:56:11 2017 - [info]
Tue Nov  7 15:56:11 2017 - [info] HealthCheck: SSH to host_2 is reachable.
Tue Nov  7 15:56:12 2017 - [info] Binlog server host_2 is reachable.
Tue Nov  7 15:56:12 2017 - [info] HealthCheck: SSH to host_1 is reachable.
Tue Nov  7 15:56:12 2017 - [info] Binlog server host_1 is reachable.
Tue Nov  7 15:56:12 2017 - [info] HealthCheck: SSH to host_3 is reachable.
Tue Nov  7 15:56:13 2017 - [info] Binlog server host_3 is reachable.
Tue Nov  7 15:56:13 2017 - [warning] SQL Thread is stopped(no error) on host_1(host_1:3306)
Tue Nov  7 15:56:13 2017 - [warning] SQL Thread is stopped(no error) on host_3(host_3:3306)
Tue Nov  7 15:56:13 2017 - [info] GTID failover mode = 1
Tue Nov  7 15:56:13 2017 - [info] Dead Servers:
Tue Nov  7 15:56:13 2017 - [info]   host_2(host_2:3306)
Tue Nov  7 15:56:13 2017 - [info] Checking master reachability via MySQL(double check)...
Tue Nov  7 15:56:13 2017 - [info]  ok.
Tue Nov  7 15:56:13 2017 - [info] Alive Servers:
Tue Nov  7 15:56:13 2017 - [info]   host_1(host_1:3306)
Tue Nov  7 15:56:13 2017 - [info]   host_3(host_3:3306)
Tue Nov  7 15:56:13 2017 - [info] Alive Slaves:
Tue Nov  7 15:56:13 2017 - [info]   host_1(host_1:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Tue Nov  7 15:56:13 2017 - [info]     GTID ON
Tue Nov  7 15:56:13 2017 - [info]     Replicating from host_2(host_2:3306)
Tue Nov  7 15:56:13 2017 - [info]     Primary candidate for the new Master (candidate_master is set)
Tue Nov  7 15:56:13 2017 - [info]   host_3(host_3:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Tue Nov  7 15:56:13 2017 - [info]     GTID ON
Tue Nov  7 15:56:13 2017 - [info]     Replicating from host_2(host_2:3306)
Tue Nov  7 15:56:13 2017 - [info]     Not candidate for the new Master (no_master is set)
Tue Nov  7 15:56:13 2017 - [info]  Starting SQL thread on host_1(host_1:3306) ..
Tue Nov  7 15:56:13 2017 - [info]   done.
Tue Nov  7 15:56:13 2017 - [info]  Starting SQL thread on host_3(host_3:3306) ..
Tue Nov  7 15:56:13 2017 - [info]   done.
Tue Nov  7 15:56:13 2017 - [info] Starting GTID based failover.
Tue Nov  7 15:56:13 2017 - [info]
Tue Nov  7 15:56:13 2017 - [info] ** Phase 1: Configuration Check Phase completed.
Tue Nov  7 15:56:13 2017 - [info]
Tue Nov  7 15:56:13 2017 - [info] * Phase 2: Dead Master Shutdown Phase..
Tue Nov  7 15:56:13 2017 - [info]
Tue Nov  7 15:56:13 2017 - [info] HealthCheck: SSH to host_2 is reachable.
Tue Nov  7 15:56:13 2017 - [info] Forcing shutdown so that applications never connect to the current master..
Tue Nov  7 15:56:13 2017 - [info] Executing master IP deactivation script:
Tue Nov  7 15:56:13 2017 - [info]   /data/online/agent/MHA/masterha/bak_mha_test/master_ip_failover_mha_test --orig_master_host=host_2 --orig_master_ip=host_2 --orig_master_port=3306 --command=stopssh --ssh_user=root
Tue Nov  7 15:56:16 2017 - [info]  done.
Tue Nov  7 15:56:16 2017 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Tue Nov  7 15:56:16 2017 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Tue Nov  7 15:56:16 2017 - [info]
Tue Nov  7 15:56:16 2017 - [info] * Phase 3: Master Recovery Phase..
Tue Nov  7 15:56:16 2017 - [info]
Tue Nov  7 15:56:16 2017 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Tue Nov  7 15:56:16 2017 - [info]
Tue Nov  7 15:56:16 2017 - [info] The latest binary log file/position on all slaves is host_1.000049:11291
Tue Nov  7 15:56:16 2017 - [info] Retrieved Gtid Set: ebd9ff93-c5b2-11e6-b21d-ecf4bbf1f42c:3-446352
Tue Nov  7 15:56:16 2017 - [info] Latest slaves (Slaves that received relay log files to the latest):
Tue Nov  7 15:56:16 2017 - [info]   host_1(host_1:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Tue Nov  7 15:56:16 2017 - [info]     GTID ON
Tue Nov  7 15:56:16 2017 - [info]     Replicating from host_2(host_2:3306)
Tue Nov  7 15:56:16 2017 - [info]     Primary candidate for the new Master (candidate_master is set)
Tue Nov  7 15:56:16 2017 - [info] The oldest binary log file/position on all slaves is host_1.000049:10703
Tue Nov  7 15:56:16 2017 - [info] Retrieved Gtid Set: ebd9ff93-c5b2-11e6-b21d-ecf4bbf1f42c:3-446350
Tue Nov  7 15:56:16 2017 - [info] Oldest slaves:
Tue Nov  7 15:56:16 2017 - [info]   host_3(host_3:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Tue Nov  7 15:56:16 2017 - [info]     GTID ON
Tue Nov  7 15:56:16 2017 - [info]     Replicating from host_2(host_2:3306)
Tue Nov  7 15:56:16 2017 - [info]     Not candidate for the new Master (no_master is set)
Tue Nov  7 15:56:16 2017 - [info]
Tue Nov  7 15:56:16 2017 - [info] * Phase 3.3: Determining New Master Phase..
Tue Nov  7 15:56:16 2017 - [info]
Tue Nov  7 15:56:16 2017 - [info] Searching new master from slaves..
Tue Nov  7 15:56:16 2017 - [info]  Candidate masters from the configuration file:
Tue Nov  7 15:56:16 2017 - [info]   host_1(host_1:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Tue Nov  7 15:56:16 2017 - [info]     GTID ON
Tue Nov  7 15:56:16 2017 - [info]     Replicating from host_2(host_2:3306)
Tue Nov  7 15:56:16 2017 - [info]     Primary candidate for the new Master (candidate_master is set)
Tue Nov  7 15:56:16 2017 - [info]  Non-candidate masters:
Tue Nov  7 15:56:16 2017 - [info]   host_3(host_3:3306)  Version=5.7.13-log (oldest major version between slaves) log-bin:enabled
Tue Nov  7 15:56:16 2017 - [info]     GTID ON
Tue Nov  7 15:56:16 2017 - [info]     Replicating from host_2(host_2:3306)
Tue Nov  7 15:56:16 2017 - [info]     Not candidate for the new Master (no_master is set)
Tue Nov  7 15:56:16 2017 - [info]  Searching from candidate_master slaves which have received the latest relay log events..
Tue Nov  7 15:56:16 2017 - [info] New master is host_1(host_1:3306)
Tue Nov  7 15:56:16 2017 - [info] Starting master failover..
Tue Nov  7 15:56:16 2017 - [info]
Tue Nov  7 15:56:16 2017 - [info]
Tue Nov  7 15:56:16 2017 - [info] * Phase 3.3: New Master Recovery Phase..
Tue Nov  7 15:56:16 2017 - [info]
Tue Nov  7 15:56:16 2017 - [info]  Waiting all logs to be applied..
Tue Nov  7 15:56:16 2017 - [info]   done.
Tue Nov  7 15:56:16 2017 - [info] -- Saving binlog from host host_2 started, pid: 79759
Tue Nov  7 15:56:16 2017 - [info] -- Saving binlog from host host_1 started, pid: 79768
Tue Nov  7 15:56:16 2017 - [info] -- Saving binlog from host host_3 started, pid: 79770
Tue Nov  7 15:56:17 2017 - [info]
Tue Nov  7 15:56:17 2017 - [info] Log messages from host_1 ...
Tue Nov  7 15:56:17 2017 - [info]
Tue Nov  7 15:56:16 2017 - [info] Fetching binary logs from binlog server host_1..
Tue Nov  7 15:56:16 2017 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=host_1.000049  --start_pos=11291 --output_file=/var/log/masterha/mha_test/saved_binlog_binlog2_20171107155611.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.7.13-log  --binlog_dir=/data/mysql.bin
Tue Nov  7 15:56:17 2017 - [error][/usr/share/perl5/vendor_perl/MHA/MasterFailover.pm, ln660] Failed to save binary log events from the binlog server. Maybe disks on binary logs are not accessible or binary log itself is corrupt?
Tue Nov  7 15:56:17 2017 - [info] End of log messages from host_1.
Tue Nov  7 15:56:17 2017 - [warning] Got error from host_1.
Tue Nov  7 15:56:17 2017 - [info]
Tue Nov  7 15:56:17 2017 - [info] Log messages from host_3 ...
Tue Nov  7 15:56:17 2017 - [info]
Tue Nov  7 15:56:16 2017 - [info] Fetching binary logs from binlog server host_3..
Tue Nov  7 15:56:16 2017 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=host_1.000049  --start_pos=11291 --output_file=/var/log/masterha/mha_test/saved_binlog_binlog3_20171107155611.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.7.13-log  --binlog_dir=/data/mysql.bin
Tue Nov  7 15:56:17 2017 - [error][/usr/share/perl5/vendor_perl/MHA/MasterFailover.pm, ln660] Failed to save binary log events from the binlog server. Maybe disks on binary logs are not accessible or binary log itself is corrupt?
Tue Nov  7 15:56:17 2017 - [info] End of log messages from host_3.
Tue Nov  7 15:56:17 2017 - [warning] Got error from host_3.
Tue Nov  7 15:56:17 2017 - [info]
Tue Nov  7 15:56:17 2017 - [info] Log messages from host_2 ...
Tue Nov  7 15:56:17 2017 - [info]
Tue Nov  7 15:56:16 2017 - [info] Fetching binary logs from binlog server host_2..
Tue Nov  7 15:56:16 2017 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=host_1.000049  --start_pos=11291 --output_file=/var/log/masterha/mha_test/saved_binlog_binlog1_20171107155611.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.7.13-log  --binlog_dir=/data/mysql.bin
Tue Nov  7 15:56:17 2017 - [info] scp from root@host_2:/var/log/masterha/mha_test/saved_binlog_binlog1_20171107155611.binlog to local:/var/log/masterha/mha_test/saved_binlog_host_2_binlog1_20171107155611.binlog succeeded.
Tue Nov  7 15:56:17 2017 - [info] End of log messages from host_2.
Tue Nov  7 15:56:17 2017 - [info] Saved mysqlbinlog size from host_2 is 768 bytes.
Tue Nov  7 15:56:17 2017 - [info] Applying differential binlog /var/log/masterha/mha_test/saved_binlog_host_2_binlog1_20171107155611.binlog ..
Tue Nov  7 15:56:17 2017 - [info] Differential log apply from binlog server succeeded.
Tue Nov  7 15:56:17 2017 - [info] Getting new master's binlog name and position..
Tue Nov  7 15:56:17 2017 - [info]  tjtx-126-164.000051:11449
Tue Nov  7 15:56:17 2017 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='host_1', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Tue Nov  7 15:56:17 2017 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: tjtx-126-164.000051, 11449, 0923e916-3c36-11e6-82a5-ecf4bbf1f518:1,
Tue Nov  7 15:56:17 2017 - [info] Executing master IP activate script:
Tue Nov  7 15:56:17 2017 - [info]   /data/online/agent/MHA/masterha/bak_mha_test/master_ip_failover_mha_test --command=start --ssh_user=root --orig_master_host=host_2 --orig_master_ip=host_2 --orig_master_port=3306 --new_master_host=host_1 --new_master_ip=host_1 --new_master_port=3306 --new_master_user='xxx' --new_master_password='xxx'
Tue Nov  7 15:56:20 2017 - [info]  OK.
Tue Nov  7 15:56:20 2017 - [info] Setting read_only=0 on host_1(host_1:3306)..
Tue Nov  7 15:56:20 2017 - [info]  ok.
Tue Nov  7 15:56:20 2017 - [info] ** Finished master recovery successfully.
Tue Nov  7 15:56:20 2017 - [info] * Phase 3: Master Recovery Phase completed.
Tue Nov  7 15:56:20 2017 - [info]
Tue Nov  7 15:56:20 2017 - [info] * Phase 4: Slaves Recovery Phase..
Tue Nov  7 15:56:20 2017 - [info]
Tue Nov  7 15:56:20 2017 - [info]
Tue Nov  7 15:56:20 2017 - [info] * Phase 4.1: Starting Slaves in parallel..
Tue Nov  7 15:56:20 2017 - [info]
Tue Nov  7 15:56:20 2017 - [info] -- Slave recovery on host host_3(host_3:3306) started, pid: 85941. Check tmp log /var/log/masterha/mha_test/host_3_3306_20171107155611.log if it takes time..
Tue Nov  7 15:56:21 2017 - [info]
Tue Nov  7 15:56:21 2017 - [info] Log messages from host_3 ...
Tue Nov  7 15:56:21 2017 - [info]
Tue Nov  7 15:56:20 2017 - [info]  Resetting slave host_3(host_3:3306) and starting replication from the new master host_1(host_1:3306)..
Tue Nov  7 15:56:20 2017 - [info]  Executed CHANGE MASTER.
Tue Nov  7 15:56:21 2017 - [info]  Slave started.
Tue Nov  7 15:56:21 2017 - [info]  gtid_wait(0923e916-3c36-11e6-82a5-ecf4bbf1f518:1,
Tue Nov  7 15:56:21 2017 - [info] End of log messages from host_3.
Tue Nov  7 15:56:21 2017 - [info] -- Slave on host host_3(host_3:3306) started.
Tue Nov  7 15:56:21 2017 - [info] All new slave servers recovered successfully.
Tue Nov  7 15:56:21 2017 - [info]
Tue Nov  7 15:56:21 2017 - [info] * Phase 5: New master cleanup phase..
Tue Nov  7 15:56:21 2017 - [info]
Tue Nov  7 15:56:21 2017 - [info] Resetting slave info on the new master..
Tue Nov  7 15:56:21 2017 - [info]  host_1: Resetting slave info succeeded.
Tue Nov  7 15:56:21 2017 - [info] Master failover to host_1(host_1:3306) completed successfully.
Tue Nov  7 15:56:21 2017 - [info]
Tue Nov  7 15:56:21 2017 - [info] Sending mail..

1.4 slave(??????master???????????????????????????

1000s????????????

????????????????????????

flush tables with readlock


????????????????????????

1.5 binlog server ?????????????????????

dead_master???????????????????????????????????????slave???etl?????????, ??????slave??????????????????etl ??????????????????????????????

binlog server ???3???

masterha_master_switch --global_conf=/data/online/agent/MHA/conf/masterha_default.cnf --conf=/data/online/agent/MHA/conf/bak_mha_test.cnf  --dead_master_host=host_1  --dead_master_port=3306 --master_state=dead --interactive=0 --ignore_last_failover --ignore_binlog_server_error


Tue Nov  7 15:56:17 2017 - [info] Log messages from host_1 ...
Tue Nov  7 15:56:17 2017 - [info]
Tue Nov  7 15:56:16 2017 - [info] Fetching binary logs from binlog server host_1..
Tue Nov  7 15:56:16 2017 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=host_1.000049  --start_pos=11291 --output_file=/var/log/masterha/mha_test/saved_binlog_binlog2_20171107155611.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.7.13-log  --binlog_dir=/data/mysql.bin
Tue Nov  7 15:56:17 2017 - [error][/usr/share/perl5/vendor_perl/MHA/MasterFailover.pm, ln660] Failed to save binary log events from the binlog server. Maybe disks on binary logs are not accessible or binary log itself is corrupt?
Tue Nov  7 15:56:17 2017 - [info] End of log messages from host_1.
Tue Nov  7 15:56:17 2017 - [warning] Got error from host_1.
Tue Nov  7 15:56:17 2017 - [info]
Tue Nov  7 15:56:17 2017 - [info] Log messages from host_3 ...
Tue Nov  7 15:56:17 2017 - [info]
Tue Nov  7 15:56:16 2017 - [info] Fetching binary logs from binlog server host_3..
Tue Nov  7 15:56:16 2017 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=host_1.000049  --start_pos=11291 --output_file=/var/log/masterha/mha_test/saved_binlog_binlog3_20171107155611.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.7.13-log  --binlog_dir=/data/mysql.bin
Tue Nov  7 15:56:17 2017 - [error][/usr/share/perl5/vendor_perl/MHA/MasterFailover.pm, ln660] Failed to save binary log events from the binlog server. Maybe disks on binary logs are not accessible or binary log itself is corrupt?
Tue Nov  7 15:56:17 2017 - [info] End of log messages from host_3.
Tue Nov  7 15:56:17 2017 - [warning] Got error from host_3.
Tue Nov  7 15:56:17 2017 - [info]
Tue Nov  7 15:56:17 2017 - [info] Log messages from host_2 ...
Tue Nov  7 15:56:17 2017 - [info]
Tue Nov  7 15:56:16 2017 - [info] Fetching binary logs from binlog server host_2..
Tue Nov  7 15:56:16 2017 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=host_1.000049  --start_pos=11291 --output_file=/var/log/masterha/mha_test/saved_binlog_binlog1_20171107155611.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.7.13-log  --binlog_dir=/data/mysql.bin
Tue Nov  7 15:56:17 2017 - [info] scp from root@host_2:/var/log/masterha/mha_test/saved_binlog_binlog1_20171107155611.binlog to local:/var/log/masterha/mha_test/saved_binlog_host_2_binlog1_20171107155611.binlog succeeded.
Tue Nov  7 15:56:17 2017 - [info] End of log messages from host_2.
Tue Nov  7 15:56:17 2017 - [info] Saved mysqlbinlog size from host_2 is 768 bytes.
Tue Nov  7 15:56:17 2017 - [info] Applying differential binlog /var/log/masterha/mha_test/saved_binlog_host_2_binlog1_20171107155611.binlog ..
Tue Nov  7 15:56:17 2017 - [info] Differential log apply from binlog server succeeded.

binlog server ??????master

masterha_master_switch --global_conf=/data/online/agent/MHA/conf/masterha_default.cnf --conf=/data/online/agent/MHA/conf/bak_mha_test.cnf  --dead_master_host=host_2  --dead_master_port=3306 --master_state=dead --interactive=0 --ignore_last_failover --ignore_binlog_server_error


Thu Nov  9 11:20:04 2017 - [info] -- Saving binlog from host host_2 started, pid: 117389
Thu Nov  9 11:20:05 2017 - [info]
Thu Nov  9 11:20:05 2017 - [info] Log messages from host_2 ...
Thu Nov  9 11:20:05 2017 - [info]
Thu Nov  9 11:20:04 2017 - [info] Fetching binary logs from binlog server host_2..
Thu Nov  9 11:20:04 2017 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=host_1.000004  --start_pos=1115 --output_file=/var/log/masterha/mha_test/saved_binlog_binlog1_20171109111957.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.7.13-log  --binlog_dir=/data/mysql.bin
Thu Nov  9 11:20:05 2017 - [info] scp from root@host_2:/var/log/masterha/mha_test/saved_binlog_binlog1_20171109111957.binlog to local:/var/log/masterha/mha_test/saved_binlog_host_2_binlog1_20171109111957.binlog succeeded.
Thu Nov  9 11:20:05 2017 - [info] End of log messages from host_2.
Thu Nov  9 11:20:05 2017 - [info] Saved mysqlbinlog size from host_2 is 4444 bytes.
Thu Nov  9 11:20:05 2017 - [info] Applying differential binlog /var/log/masterha/mha_test/saved_binlog_host_2_binlog1_20171109111957.binlog ..
Thu Nov  9 11:20:05 2017 - [info] Differential log apply from binlog server succeeded.

binlog server ??????slave

### 3???????????????GTID??????

* master  host_1

dba:lc> show master status;
+---------------------+----------+--------------+------------------+------------------------------------------------------------------------------------------+
| File                | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                                                                        |
+---------------------+----------+--------------+------------------+------------------------------------------------------------------------------------------+
| tjtx-126-164.000055 |     6016 |              |                  | 0923e916-3c36-11e6-82a5-ecf4bbf1f518:1-31,
ebd9ff93-c5b2-11e6-b21d-ecf4bbf1f42c:1-446369 |
+---------------------+----------+--------------+------------------+------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

* slave host_2


            Executed_Gtid_Set: 0923e916-3c36-11e6-82a5-ecf4bbf1f518:1-21,
ebd9ff93-c5b2-11e6-b21d-ecf4bbf1f42c:1-446369
                Auto_Position: 1


* etl host_3

           Retrieved_Gtid_Set: 0923e916-3c36-11e6-82a5-ecf4bbf1f518:22-25,
ebd9ff93-c5b2-11e6-b21d-ecf4bbf1f42c:446366-446369
            Executed_Gtid_Set: 0923e916-3c36-11e6-82a5-ecf4bbf1f518:1-25,
ebd9ff93-c5b2-11e6-b21d-ecf4bbf1f42c:1-446369
                Auto_Position: 1


### ????????????

masterha_master_switch --global_conf=/data/online/agent/MHA/conf/masterha_default.cnf --conf=/data/online/agent/MHA/conf/bak_mha_test.cnf  --dead_master_host=host_1  --dead_master_port=3306 --master_state=dead --interactive=0 --ignore_last_failover --ignore_binlog_server_error

Thu Nov  9 15:00:09 2017 - [info] MHA::MasterFailover version 0.56.
Thu Nov  9 15:00:09 2017 - [info] Starting master failover.
Thu Nov  9 15:00:09 2017 - [info]
Thu Nov  9 15:00:09 2017 - [info] * Phase 1: Configuration Check Phase..
Thu Nov  9 15:00:09 2017 - [info]
Thu Nov  9 15:00:09 2017 - [info] HealthCheck: SSH to host_2 is reachable.
Thu Nov  9 15:00:09 2017 - [info] Binlog server host_2 is reachable.
Thu Nov  9 15:00:10 2017 - [warning] SQL Thread is stopped(no error) on host_2(host_2:3306)
Thu Nov  9 15:00:10 2017 - [warning] SQL Thread is stopped(no error) on host_3(host_3:3306)
Thu Nov  9 15:00:10 2017 - [info] GTID failover mode = 1
Thu Nov  9 15:00:10 2017 - [info] Dead Servers:
Thu Nov  9 15:00:10 2017 - [info]   host_1(host_1:3306)
Thu Nov  9 15:00:10 2017 - [info] Checking mast最后更新：2017-11-10 16:03:51
  上一篇：  【百度副总裁王海峰】AI基石知识图谱与百度AI布局（28PPT）
  下一篇：  Large-Scale Instant Messaging Hosting on Alibaba Cloud
相关内容
 全球97%智能手机采用美国造操作系统
  自动化测试 之 “好用例、坏用例” 
 C# 系统应用之窗体最小化至任务栏及常用操作
  史上最全解读WanaCrypt0r 2.0勒索软件及安全防护最佳实践 
  传统应用层逻辑分库DB迁移阿里云DRDS+RDS分布式数据库
  电脑不小心误删的文件怎么才能恢复哪个软件好 
 性能测试：自建数据库与RDS性能对比（SQL Server案例排查分析）
  tomcat连接mysql超时 
 中国人的败家子培训法
  《正则表达式经典实例（第2版）》——2.19　在替代文本中添加字面文本 
热门内容
 PostgreSQL 路径规划插件 pgruoting 介绍
 人生不过如此
  《STM32库开发实战指南：基于STM32F4》----导读
  解密区块链：从基础概念到技术实现的进阶攻略 
 ZED Board从入门到精通（三）：从传统ARM开发到PS开发的转变
 [图文]历届奥斯卡影帝(上)
  扎克伯格发6千字长文,谈Facebook未来愿景和世界该有的样子
 VS 添加jQuery智能提示
 python3学习
  突破吞吐限制，多NAS性能聚合方案，数据上传及读写
最新内容
  驻云Gartner联合发布人工智能时代的MSP白皮书
  云端安全之三：最佳实践
  突破吞吐限制，多NAS性能聚合方案，数据上传及读写
  如何同步TableStore数据到OSS
  百亿级全网舆情分析系统存储设计
  在ECS中访问各类阿里云云资源的正确姿势 
  不外传的seo计划教程方案 
  网站有收录没排名的原因与解决办法 
  燃！阿里的技术小哥们又又又拿冠军了！
  如何将个性化需求变成大数据解决方案

MHA failover GTID ??????-??????-????????????-?????????

MHA failover GTID ??????

????????????(???????????????)

??????Master : MySQL down

1.1 etl ??????8??????

1.2 slave(??????master)???etl??????????????????

1.3 slave(??????master)???????????????????????????etl??????

1.4 slave(??????master???????????????????????????

1.5 binlog server ?????????????????????

上一篇： 【百度副总裁王海峰】AI基石知识图谱与百度AI布局（28PPT）

下一篇： Large-Scale Instant Messaging Hosting on Alibaba Cloud

相关内容

热门内容

最新内容

上一篇：【百度副总裁王海峰】AI基石知识图谱与百度AI布局（28PPT）