Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

node status readable=0 ,after node online -> standby -> online #7

Open
zhaoshenju opened this issue Apr 9, 2013 · 4 comments
Open

node status readable=0 ,after node online -> standby -> online #7

zhaoshenju opened this issue Apr 9, 2013 · 4 comments

Comments

@zhaoshenju
Copy link

[root@jboss-203 ~]# crm status

Last updated: Fri Mar 29 23:39:09 2013
Stack: openais
Current DC: jboss-204 - partition with quorum
Version: 1.0.12-unknown
3 Nodes configured, 3 expected votes
#5 Resources configured.

Online: [ jboss-203 jboss-204 jboss-205 ]

reader_vip_1 (ocf::heartbeat:IPaddr2): Started jboss-203
reader_vip_2 (ocf::heartbeat:IPaddr2): Started jboss-204
reader_vip_3 (ocf::heartbeat:IPaddr2): Started jboss-205
writer_vip (ocf::heartbeat:IPaddr2): Started jboss-204
Master/Slave Set: ms_MySQL
Masters: [ jboss-204 ]
Slaves: [ jboss-203 jboss-205 ]

[root@jboss-203 ~]# crm node standby jboss-204
[root@jboss-203 ~]#
[root@jboss-203 ~]# crm node online jboss-204
[root@jboss-203 ~]#

after a few minutes.

[root@jboss-203 ~]# crm status

Last updated: Fri Mar 29 23:40:34 2013
Stack: openais
Current DC: jboss-204 - partition with quorum
Version: 1.0.12-unknown
3 Nodes configured, 3 expected votes
#5 Resources configured.

Online: [ jboss-203 jboss-204 jboss-205 ]

reader_vip_1 (ocf::heartbeat:IPaddr2): Started jboss-203
reader_vip_2 (ocf::heartbeat:IPaddr2): Started jboss-205
reader_vip_3 (ocf::heartbeat:IPaddr2): Started jboss-203
writer_vip (ocf::heartbeat:IPaddr2): Started jboss-203
Master/Slave Set: ms_MySQL
Masters: [ jboss-203 ]
Slaves: [ jboss-205 jboss-204 ]

[root@jboss-203 ~]# crm resource migrate reader_vip_1 jboss-204

[root@jboss-203 ~]# crm status

Last updated: Fri Mar 29 23:40:59 2013
Stack: openais
Current DC: jboss-204 - partition with quorum
Version: 1.0.12-unknown
3 Nodes configured, 3 expected votes
#5 Resources configured.

Online: [ jboss-203 jboss-204 jboss-205 ]

reader_vip_1 (ocf::heartbeat:IPaddr2): Started jboss-203
reader_vip_2 (ocf::heartbeat:IPaddr2): Started jboss-205
reader_vip_3 (ocf::heartbeat:IPaddr2): Started jboss-203
writer_vip (ocf::heartbeat:IPaddr2): Started jboss-203
Master/Slave Set: ms_MySQL
Masters: [ jboss-203 ]
Slaves: [ jboss-205 jboss-204 ]

[root@jboss-203 ~]# cibadmin -Q | grep readable | grep nvpair


but on node jboss-204, mysql slave is ok,seconds_behind_master=0
why readable=0?

[root@jboss-203 ~]# crm configure show
node jboss-203
attributes p_mysql_mysql_master_IP="192.168.10.203"
attributes standby="off"
node jboss-204
attributes p_mysql_mysql_master_IP="192.168.10.204"
attributes standby="off"
node jboss-205
attributes p_mysql_mysql_master_IP="192.168.10.205"
attributes standby="off"
primitive p_mysql ocf:percona:mysql
params config="/app/mysql/etc/my.cnf" pid="/data/mysql/data/mysql.pid" socket="/data/mysql/tmp/mysql.sock" replication_user="slave" replication_passwd="hGrXFdoEnWnW" max_slave_lag="180" evict_outdated_slaves="false" binary="/app/mysql/bin/mysqld" datadir="/data/mysql/data/" test_user="prm_test_user" test_passwd="XvrhWjtmIhzs" client_binary="/app/mysql/bin/mysql"
op monitor interval="5s" role="Master" OCF_CHECK_LEVEL="1"
op monitor interval="2s" role="Slave" OCF_CHECK_LEVEL="1"
op start interval="0" timeout="120s"
op stop interval="0" timeout="120s"
primitive reader_vip_1 ocf:heartbeat:IPaddr2
params ip="192.168.10.101" nic="eth0"
op monitor interval="10s"
primitive reader_vip_2 ocf:heartbeat:IPaddr2
params ip="192.168.10.102" nic="eth0"
op monitor interval="10s"
primitive reader_vip_3 ocf:heartbeat:IPaddr2
params ip="192.168.10.105" nic="eth0"
op monitor interval="10s"
primitive writer_vip ocf:heartbeat:IPaddr2
params ip="192.168.10.107" nic="eth0"
op monitor interval="10s"
ms ms_MySQL p_mysql
meta master-max="1" master-node-max="1" clone-max="3" clone-node-max="1" notify="true" globally-unique="false" target-role="Master" is-managed="true"
location loc-No-reader-vip-2 reader_vip_2
rule $id="rule-no-reader-vip-2" -inf: readable eq 0
location loc-No-reader-vip-3 reader_vip_3
rule $id="rule-no-reader-vip-3" -inf: readable eq 0
location loc-no-reader-vip-1 reader_vip_1
rule $id="rule-no-reader-vip-1" -inf: readable eq 0
colocation writer_vip_on_master inf: writer_vip ms_MySQL:Master
order ms_MySQL_promote_before_vip inf: ms_MySQL:promote writer_vip:start
property $id="cib-bootstrap-options"
dc-version="1.0.12-unknown"
cluster-infrastructure="openais"
expected-quorum-votes="3"
stonith-enabled="false"
no-quorum-policy="ignore"
property $id="mysql_replication"
p_mysql_REPL_INFO="192.168.10.203|mysql-bin.000012|1252"
@y-trudeau
Copy link
Collaborator

Hi,
create the file /tmp/mysql.ocf.ra.debug/log on 204 and look what is
going wrong with the monitor operation.

Regards,

Yves

Le 2013-04-09 07:22, zhaoshenju a écrit :

[root@jboss-203 ~]# crm status

Last updated: Fri Mar 29 23:39:09 2013
Stack: openais
Current DC: jboss-204 - partition with quorum
Version: 1.0.12-unknown
3 Nodes configured, 3 expected votes

5 Resources configured.

Online: [ jboss-203 jboss-204 jboss-205 ]

reader_vip_1 (ocf::heartbeat:IPaddr2): Started jboss-203
reader_vip_2 (ocf::heartbeat:IPaddr2): Started jboss-204
reader_vip_3 (ocf::heartbeat:IPaddr2): Started jboss-205
writer_vip (ocf::heartbeat:IPaddr2): Started jboss-204
Master/Slave Set: ms_MySQL
Masters: [ jboss-204 ]
Slaves: [ jboss-203 jboss-205 ]

[root@jboss-203 ~]# crm node standby jboss-204
[root@jboss-203 ~]#
[root@jboss-203 ~]# crm node online jboss-204
[root@jboss-203 ~]#

after a few minutes.

[root@jboss-203 ~]# crm status

Last updated: Fri Mar 29 23:40:34 2013
Stack: openais
Current DC: jboss-204 - partition with quorum
Version: 1.0.12-unknown
3 Nodes configured, 3 expected votes

5 Resources configured.

Online: [ jboss-203 jboss-204 jboss-205 ]

reader_vip_1 (ocf::heartbeat:IPaddr2): Started jboss-203
reader_vip_2 (ocf::heartbeat:IPaddr2): Started jboss-205
reader_vip_3 (ocf::heartbeat:IPaddr2): Started jboss-203
writer_vip (ocf::heartbeat:IPaddr2): Started jboss-203
Master/Slave Set: ms_MySQL
Masters: [ jboss-203 ]
Slaves: [ jboss-205 jboss-204 ]

[root@jboss-203 ~]# crm resource migrate reader_vip_1 jboss-204

[root@jboss-203 ~]# crm status

Last updated: Fri Mar 29 23:40:59 2013
Stack: openais
Current DC: jboss-204 - partition with quorum
Version: 1.0.12-unknown
3 Nodes configured, 3 expected votes

5 Resources configured.

Online: [ jboss-203 jboss-204 jboss-205 ]

reader_vip_1 (ocf::heartbeat:IPaddr2): Started jboss-203
reader_vip_2 (ocf::heartbeat:IPaddr2): Started jboss-205
reader_vip_3 (ocf::heartbeat:IPaddr2): Started jboss-203
writer_vip (ocf::heartbeat:IPaddr2): Started jboss-203
Master/Slave Set: ms_MySQL
Masters: [ jboss-203 ]
Slaves: [ jboss-205 jboss-204 ]

[root@jboss-203 ~]# cibadmin -Q | grep readable | grep nvpair

but on node jboss-204, mysql slave is ok,seconds_behind_master=0
why readable=0?

[root@jboss-203 ~]# crm configure show
node jboss-203
attributes p_mysql_mysql_master_IP="192.168.10.203"
attributes standby="off"
node jboss-204
attributes p_mysql_mysql_master_IP="192.168.10.204"
attributes standby="off"
node jboss-205
attributes p_mysql_mysql_master_IP="192.168.10.205"
attributes standby="off"
primitive p_mysql ocf:percona:mysql
params config="/app/mysql/etc/my.cnf" pid="/data/mysql/data/mysql.pid"
socket="/data/mysql/tmp/mysql.sock" replication_user="slave"
replication_passwd="hGrXFdoEnWnW" max_slave_lag="180"
evict_outdated_slaves="false" binary="/app/mysql/bin/mysqld"
datadir="/data/mysql/data/" test_user="prm_test_user"
test_passwd="XvrhWjtmIhzs" client_binary="/app/mysql/bin/mysql"
op monitor interval="5s" role="Master" OCF_CHECK_LEVEL="1"
op monitor interval="2s" role="Slave" OCF_CHECK_LEVEL="1"
op start interval="0" timeout="120s"
op stop interval="0" timeout="120s"
primitive reader_vip_1 ocf:heartbeat:IPaddr2
params ip="192.168.10.101" nic="eth0"
op monitor interval="10s"
primitive reader_vip_2 ocf:heartbeat:IPaddr2
params ip="192.168.10.102" nic="eth0"
op monitor interval="10s"
primitive reader_vip_3 ocf:heartbeat:IPaddr2
params ip="192.168.10.105" nic="eth0"
op monitor interval="10s"
primitive writer_vip ocf:heartbeat:IPaddr2
params ip="192.168.10.107" nic="eth0"
op monitor interval="10s"
ms ms_MySQL p_mysql
meta master-max="1" master-node-max="1" clone-max="3" clone-node-max="1"
notify="true" globally-unique="false" target-role="Master" is-managed="true"
location loc-No-reader-vip-2 reader_vip_2
rule $id="rule-no-reader-vip-2" -inf: readable eq 0
location loc-No-reader-vip-3 reader_vip_3
rule $id="rule-no-reader-vip-3" -inf: readable eq 0
location loc-no-reader-vip-1 reader_vip_1
rule $id="rule-no-reader-vip-1" -inf: readable eq 0
colocation writer_vip_on_master inf: writer_vip ms_MySQL:Master
order ms_MySQL_promote_before_vip inf: ms_MySQL:promote writer_vip:start
property $id="cib-bootstrap-options"
dc-version="1.0.12-unknown"
cluster-infrastructure="openais"
expected-quorum-votes="3"
stonith-enabled="false"
no-quorum-policy="ignore"
property $id="mysql_replication"
p_mysql_REPL_INFO="192.168.10.203|mysql-bin.000012|1252"

—
Reply to this email directly or view it on GitHub
#7.

@zhaoshenju
Copy link
Author

hi,y-trudeau:

Very happy to see your reply!

Not only in the jboss-204 this is the case,in all other slave such as jboss-205 is this

[root@jboss-203 /]# crm status

Last updated: Wed Apr 10 15:50:36 2013
Stack: openais
Current DC: jboss-203 - partition with quorum
Version: 1.0.12-unknown
3 Nodes configured, 3 expected votes

5 Resources configured.

Online: [ jboss-203 jboss-204 jboss-205 ]

reader_vip_1 (ocf::heartbeat:IPaddr2): Started jboss-203
reader_vip_2 (ocf::heartbeat:IPaddr2): Started jboss-203
reader_vip_3 (ocf::heartbeat:IPaddr2): Started jboss-203
writer_vip (ocf::heartbeat:IPaddr2): Started jboss-203
Master/Slave Set: ms_MySQL
Masters: [ jboss-203 ]
Slaves: [ jboss-204 jboss-205 ]
[root@jboss-203 /]#
[root@jboss-203 /]# cibadmin -Q | grep readable | grep nvpair



[root@jboss-203 /]#

I run the "bash -x /usr/lib/ocf/resource.d/percona/mysql monitor" command output is as follows

+/usr/lib/ocf/resource.d/percona/mysql:1508::date
Wed Apr 10 16:00:45 CST 2013
+/usr/lib/ocf/resource.d/percona/mysql:1509::echo monitor
monitor
+/usr/lib/ocf/resource.d/percona/mysql:1510::env
+/usr/lib/ocf/resource.d/percona/mysql:1510::grep OCF_
+/usr/lib/ocf/resource.d/percona/mysql:1510::sort
+/usr/lib/ocf/resource.d/percona/mysql:1511::set -x
+/usr/lib/ocf/resource.d/percona/mysql:1517::case "$1" in
+/usr/lib/ocf/resource.d/percona/mysql:1524::mysql_validate
+/usr/lib/ocf/resource.d/percona/mysql:945:mysql_validate:check_binary /usr/bin/safe_mysqld
/usr/lib/ocf/resource.d/percona/mysql: line 945: check_binary: command not found
+/usr/lib/ocf/resource.d/percona/mysql:946:mysql_validate:check_binary mysql
/usr/lib/ocf/resource.d/percona/mysql: line 946: check_binary: command not found
+/usr/lib/ocf/resource.d/percona/mysql:948:mysql_validate:'[' '!' -f /etc/my.cnf ']'
+/usr/lib/ocf/resource.d/percona/mysql:949:mysql_validate:ocf_log err 'Config /etc/my.cnf doesn'''t exist'
/usr/lib/ocf/resource.d/percona/mysql: line 949: ocf_log: command not found
+/usr/lib/ocf/resource.d/percona/mysql:950:mysql_validate:return
+/usr/lib/ocf/resource.d/percona/mysql:1525::rc=127
+/usr/lib/ocf/resource.d/percona/mysql:1526::LSB_STATUS_STOPPED=3
+/usr/lib/ocf/resource.d/percona/mysql:1527::'[' 127 -ne 0 ']'
+/usr/lib/ocf/resource.d/percona/mysql:1528::case "$1" in
+/usr/lib/ocf/resource.d/percona/mysql:1530::exit

Whether it is at master or at slave, are not implemented to mysql_monitor function position exit.

However, in the rest of the output of the file /tmp/mysql.ocf.ra.debug/log containing text:

  • __OCF_MSG='MySQL monitor succeeded (master)' (at master)
    or
  • __OCF_MSG='MySQL monitor succeeded' (at slave)

This indicates function mysql_monitor is executed

I need to provide what information?
Look forward to your reply!

@zhaoshenju
Copy link
Author 

      <nvpair id="status-jboss-203-readable" name="readable" value="1"/>
      <nvpair id="status-jboss-204-readable" name="readable" value="0"/>
      <nvpair id="status-jboss-205-readable" name="readable" value="0"/>

This is the output of the command "cibadmin -Q | grep readable | grep nvpair"

@y-trudeau
Copy link
Collaborator

Have you enable the trace file?

mkdir /tmp/mysql.ocf.ra.debug
touch /tmp/mysql.ocf.ra.debug/log

look in the "log" file and see what the monitor op is doing.

Le 2013-04-10 05:34, zhaoshenju a écrit :

|


|

This is the output of the command "cibadmin -Q | grep readable | grep
nvpair"

—
Reply to this email directly or view it on GitHub
#7 (comment).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@y-trudeau @zhaoshenju