[Ocfs2-users] 2 node OCFS2 clusters

[Thompson, Mark]

Hi,

This is my first post here so please be gentle with me.

My question is, can you have a 2 node OCFS2 cluster, disconnect one node
from the network, and have the remaining node continue to function
normally? Currently we have a 2 node cluster and if we stop the NIC that
has the OCFS2 o2cb net connection running on it, the other node will
reboot itself. I have researched having a 2 node OCFS2 cluster but so
far I have been unable to find a clear solution. I have looked at the
FAQ regarding quorum, and my OCFS2 init scripts are enabled etc.

Is this possible, or should we look at alternative solutions?

Regards,

Mark




This e-mail has come from Experian, the only business to have been twice named the UK's 'Business of the Year’

===================================================================================
Information in this e-mail and any attachments is confidential, and may not be copied or used by anyone other than the addressee, nor disclosed to any third party without our permission. There is no intention to create any legally binding contract or other binding commitment through the use of this electronic communication unless it is issued in accordance with the Experian Limited standard terms and conditions of purchase or other express written agreement between Experian Limited and the recipient.
Although Experian has taken reasonable steps to ensure that this communication and any attachments are free from computer virus, you are advised to take your own steps to ensure that they are actually virus free.
Companies Act information:
Registered name: Experian Limited
Registered office: Landmark House, Experian Way, NG2 Business Park, Nottingham, NG80 1ZZ, United Kingdom
Place of registration: England and Wales
Registered number: 653331

[Srinivas Eeda]

In a cluster with more than 2 nodes, if a network on one node goes down,
that node will evict itself but other nodes will survive. But in a two
node cluster, the node with lowest node number will survive no mater on
which node network went down.

thanks,
--Srini

Post by Thompson, Mark
Hi,
This is my first post here so please be gentle with me.
My question is, can you have a 2 node OCFS2 cluster, disconnect one
node from the network, and have the remaining node continue to
function normally? Currently we have a 2 node cluster and if we stop
the NIC that has the OCFS2 o2cb net connection running on it, the
other node will reboot itself. I have researched having a 2 node OCFS2
cluster but so far I have been unable to find a clear solution. I have
looked at the FAQ regarding quorum, and my OCFS2 init scripts are
enabled etc.
Is this possible, or should we look at alternative solutions?
Regards,
Mark
This e-mail has come from Experian, the only business to have been
twice named the UK's 'Business of the Year'
===================================================================================
Information in this e-mail and any attachments is confidential, and
may not be copied or used by anyone other than the addressee, nor
disclosed to any third party without our permission. There is no
intention to create any legally binding contract or other binding
commitment through the use of this electronic communication unless it
is issued in accordance with the Experian Limited standard terms and
conditions of purchase or other express written agreement between
Experian Limited and the recipient.
Although Experian has taken reasonable steps to ensure that this
communication and any attachments are free from computer virus, you
are advised to take your own steps to ensure that they are actually
virus free.
Registered name: Experian Limited
Registered office: Landmark House, Experian Way, NG2 Business Park,
Nottingham, NG80 1ZZ, United Kingdom
Place of registration: England and Wales
Registered number: 653331
------------------------------------------------------------------------
_______________________________________________
Ocfs2-users mailing list
http://oss.oracle.com/mailman/listinfo/ocfs2-users

[Thompson, Mark]

Hi Srini,



Thanks for the response.



So are the following statements correct:



If I stop the networking on node 1, node 0 will continue to allow OCFS2
filesystems to work and not reboot itself.



If I stop the networking on node 0, node 1 (now being the lowest node?)
will continue to allow OCFS2 filesystems to work and not reboot itself.



I guess I just need to know if it's possible to have a 2 node OCFS2
cluster that will cope with either one of the nodes dying, and have the
remaining node still provide service.



Regards,



Mark



From: Srinivas Eeda [mailto:***@oracle.com]
Sent: 16 November 2009 14:57
To: Thompson, Mark
Cc: ocfs2-***@oss.oracle.com
Subject: Re: [Ocfs2-users] 2 node OCFS2 clusters



In a cluster with more than 2 nodes, if a network on one node goes down,
that node will evict itself but other nodes will survive. But in a two
node cluster, the node with lowest node number will survive no mater on
which node network went down.

thanks,
--Srini

Thompson, Mark wrote:

Hi,

This is my first post here so please be gentle with me.

My question is, can you have a 2 node OCFS2 cluster, disconnect one node
from the network, and have the remaining node continue to function
normally? Currently we have a 2 node cluster and if we stop the NIC that
has the OCFS2 o2cb net connection running on it, the other node will
reboot itself. I have researched having a 2 node OCFS2 cluster but so
far I have been unable to find a clear solution. I have looked at the
FAQ regarding quorum, and my OCFS2 init scripts are enabled etc.

Is this possible, or should we look at alternative solutions?

Regards,

Mark

This e-mail has come from Experian, the only business to have been twice
named the UK's 'Business of the Year'



========================================================================
===========

Information in this e-mail and any attachments is confidential, and may
not be copied or used by anyone other than the addressee, nor disclosed
to any third party without our permission. There is no intention to
create any legally binding contract or other binding commitment through
the use of this electronic communication unless it is issued in
accordance with the Experian Limited standard terms and conditions of
purchase or other express written agreement between Experian Limited and
the recipient.

Although Experian has taken reasonable steps to ensure that this
communication and any attachments are free from computer virus, you are
advised to take your own steps to ensure that they are actually virus
free.

Companies Act information:

Registered name: Experian Limited

Registered office: Landmark House, Experian Way, NG2 Business Park,
Nottingham, NG80 1ZZ, United Kingdom

Place of registration: England and Wales

Registered number: 653331






________________________________

[Srinivas Eeda]

Post by Thompson, Mark
Hi Srini,
Thanks for the response.
If I stop the networking on node 1, node 0 will continue to allow
OCFS2 filesystems to work and not reboot itself.
If I stop the networking on node 0, node 1 (now being the lowest
node?) will continue to allow OCFS2 filesystems to work and not reboot
itself.

In both the cases node 0 will survive, because that's the node that has
lowest node number (defined in cluster.conf). This applies to the
scenario where interconnect went down but nodes are healthy and are
heartbeating to the disk.

Post by Thompson, Mark
I guess I just need to know if it's possible to have a 2 node OCFS2
cluster that will cope with either one of the nodes dying, and have
the remaining node still provide service.

If node 0 itself panics, reboots then node 1 will survive.

Post by Thompson, Mark
Regards,
Mark
*Sent:* 16 November 2009 14:57
*To:* Thompson, Mark
*Subject:* Re: [Ocfs2-users] 2 node OCFS2 clusters
In a cluster with more than 2 nodes, if a network on one node goes
down, that node will evict itself but other nodes will survive. But in
a two node cluster, the node with lowest node number will survive no
mater on which node network went down.
thanks,
--Srini
Hi,
This is my first post here so please be gentle with me.
My question is, can you have a 2 node OCFS2 cluster, disconnect one
node from the network, and have the remaining node continue to
function normally? Currently we have a 2 node cluster and if we stop
the NIC that has the OCFS2 o2cb net connection running on it, the
other node will reboot itself. I have researched having a 2 node OCFS2
cluster but so far I have been unable to find a clear solution. I have
looked at the FAQ regarding quorum, and my OCFS2 init scripts are
enabled etc.
Is this possible, or should we look at alternative solutions?
Regards,
Mark
This e-mail has come from Experian, the only business to have been
twice named the UK's 'Business of the Year'
===================================================================================
Information in this e-mail and any attachments is confidential, and
may not be copied or used by anyone other than the addressee, nor
disclosed to any third party without our permission. There is no
intention to create any legally binding contract or other binding
commitment through the use of this electronic communication unless it
is issued in accordance with the Experian Limited standard terms and
conditions of purchase or other express written agreement between
Experian Limited and the recipient.
Although Experian has taken reasonable steps to ensure that this
communication and any attachments are free from computer virus, you
are advised to take your own steps to ensure that they are actually
virus free.
Registered name: Experian Limited
Registered office: Landmark House, Experian Way, NG2 Business Park,
Nottingham, NG80 1ZZ, United Kingdom
Place of registration: England and Wales
Registered number: 653331
------------------------------------------------------------------------
_______________________________________________
Ocfs2-users mailing list
http://oss.oracle.com/mailman/listinfo/ocfs2-users

[Thompson, Mark]

Hi,



I have done some more tests today, and I observed the following:



Test 1:

node 0 - ifdown eth2

node 0 - OCFS2 filesystem stalls on both nodes

node 1 - Decides to reboot

node 0 - Resumes OCFS2 service (while still off the network) OCFS2
filesystem back online

node 1 - Cannot re-join cluster as node 0 is off the network and has the
fs lock (Transport endpoint error)

node 0 - ifup eth2

node 1 - Re-joins the clusters and re-mounts OCFS2 filesystem.



Test 2:

node 1 - ifdown eth2

node 0 - OCFS2 filesystem stalls on both nodes

node 1 - Decides to reboot

node 0 - Resumes OCFS2 service, OCFS2 filesystem back online

node 1 - Boots up, re-joins cluster and re-mounts OCFS2 filesystem.



Is this the expected behaviour? And if it is, is there anything we can
do avoid the loss of the OCFS2 filesystems?





Here's the messages file outputs.



Test 1 - Node 0

Nov 17 11:00:26 my_node0 kernel: ocfs2: Unmounting device (253,9) on
(node 0)

Nov 17 11:02:21 my_node0 modprobe: FATAL: Module ocfs2_stackglue not
found.

Nov 17 11:02:21 my_node0 kernel: OCFS2 Node Manager 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)

Nov 17 11:02:21 my_node0 kernel: OCFS2 DLM 1.4.4 Tue Sep 8 11:56:46 PDT
2009 (build e6e41b84c785deeea891e5873dbf19ab)

Nov 17 11:02:21 my_node0 kernel: OCFS2 DLMFS 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)

Nov 17 11:02:21 my_node0 kernel: OCFS2 User DLM kernel interface loaded

Nov 17 11:02:46 my_node0 kernel: OCFS2 1.4.4 Tue Sep 8 11:56:43 PDT
2009 (build 3a5bffa75b910d5bcdd5c607c4394b1e)

Nov 17 11:02:46 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0

Nov 17 11:02:46 my_node0 kernel: ocfs2: Mounting device (253,9) on (node
0, slot 0) with ordered data mode.

Nov 17 11:02:59 my_node0 kernel: ocfs2_dlm: Node 1 joins domain
21751145F96E45649324C9EEF5485248

Nov 17 11:02:59 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1

Nov 17 11:07:51 my_node0 kernel: (15,1):dlm_do_master_request:1334
ERROR: link to 1 went down!

Nov 17 11:07:51 my_node0 kernel: (15,1):dlm_get_lock_resource:917 ERROR:
status = -107

Nov 17 11:09:34 my_node0 kernel: (22108,1):ocfs2_dlm_eviction_cb:98
device (253,9): dlm has evicted node 1

Nov 17 11:09:34 my_node0 kernel: (29443,1):dlm_get_lock_resource:844
21751145F96E45649324C9EEF5485248:M000000000000000000001f96e7b609: at
least one node (1) to recover before lock mastery can begin

Nov 17 11:09:35 my_node0 kernel: (29443,1):dlm_get_lock_resource:898
21751145F96E45649324C9EEF5485248:M000000000000000000001f96e7b609: at
least one node (1) to recover before lock mastery can begin

Nov 17 11:09:36 my_node0 kernel: (15,1):dlm_restart_lock_mastery:1223
ERROR: node down! 1

Nov 17 11:09:36 my_node0 kernel: (15,1):dlm_wait_for_lock_mastery:1040
ERROR: status = -11

Nov 17 11:09:36 my_node0 kernel: (22167,0):dlm_get_lock_resource:844
21751145F96E45649324C9EEF5485248:$RECOVERY: at least one node (1) to
recover before lock mastery can begin

Nov 17 11:09:36 my_node0 kernel: (22167,0):dlm_get_lock_resource:878
21751145F96E45649324C9EEF5485248: recovery map is not empty, but must
master $RECOVERY lock now

Nov 17 11:09:36 my_node0 kernel: (22167,0):dlm_do_recovery:524 (22167)
Node 0 is the Recovery Master for the Dead Node 1 for Domain
21751145F96E45649324C9EEF5485248

Nov 17 11:09:46 my_node0 kernel: (29443,1):ocfs2_replay_journal:1183
Recovering node 1 from slot 1 on device (253,9)

Nov 17 11:12:27 my_node0 kernel: ocfs2_dlm: Node 1 joins domain
21751145F96E45649324C9EEF5485248

Nov 17 11:12:27 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1



Test 1 - Node 1

Nov 17 11:00:26 my_node1 kernel: ocfs2_dlm: Node 0 leaves domain
21751145F96E45649324C9EEF5485248

Nov 17 11:00:26 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 1

Nov 17 11:00:46 my_node1 kernel: ocfs2: Unmounting device (253,9) on
(node 1)

Nov 17 11:02:30 my_node1 modprobe: FATAL: Module ocfs2_stackglue not
found.

Nov 17 11:02:30 my_node1 kernel: OCFS2 Node Manager 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)

Nov 17 11:02:30 my_node1 kernel: OCFS2 DLM 1.4.4 Tue Sep 8 11:56:46 PDT
2009 (build e6e41b84c785deeea891e5873dbf19ab)

Nov 17 11:02:30 my_node1 kernel: OCFS2 DLMFS 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)

Nov 17 11:02:30 my_node1 kernel: OCFS2 User DLM kernel interface loaded

Nov 17 11:02:59 my_node1 kernel: OCFS2 1.4.4 Tue Sep 8 11:56:43 PDT
2009 (build 3a5bffa75b910d5bcdd5c607c4394b1e)

Nov 17 11:02:59 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1

Nov 17 11:02:59 my_node1 kernel: ocfs2: Mounting device (253,9) on (node
1, slot 1) with ordered data mode.

Nov 17 11:07:27 my_node1 kernel:
(7351,3):dlm_send_remote_convert_request:395 ERROR: status = -112

Nov 17 11:07:27 my_node1 kernel: (7351,3):dlm_wait_for_node_death:370
21751145F96E45649324C9EEF5485248: waiting 5000ms for notification of
death of node 0

Nov 17 11:07:57 my_node1 kernel:
(7351,3):dlm_send_remote_convert_request:395 ERROR: status = -107

Nov 17 11:07:57 my_node1 kernel: (7351,3):dlm_wait_for_node_death:370
21751145F96E45649324C9EEF5485248: waiting 5000ms for notification of
death of node 0

Nov 17 11:08:27 my_node1 kernel: (15,1):dlm_do_master_request:1334
ERROR: link to 0 went down!

Nov 17 11:08:27 my_node1 kernel:
(7351,3):dlm_send_remote_convert_request:395 ERROR: status = -107

Nov 17 11:08:27 my_node1 kernel: (7351,3):dlm_wait_for_node_death:370
21751145F96E45649324C9EEF5485248: waiting 5000ms for notification of
death of node 0

Nov 17 11:08:27 my_node1 kernel: (15,1):dlm_get_lock_resource:917 ERROR:
status = -107

Nov 17 11:11:31 my_node1 modprobe: FATAL: Module ocfs2_stackglue not
found.

Nov 17 11:11:32 my_node1 kernel: OCFS2 Node Manager 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)

Nov 17 11:11:32 my_node1 kernel: OCFS2 DLM 1.4.4 Tue Sep 8 11:56:46 PDT
2009 (build e6e41b84c785deeea891e5873dbf19ab)

Nov 17 11:11:32 my_node1 kernel: OCFS2 DLMFS 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)

Nov 17 11:11:32 my_node1 kernel: OCFS2 User DLM kernel interface loaded

Nov 17 11:11:40 my_node1 kernel: OCFS2 1.4.4 Tue Sep 8 11:56:43 PDT
2009 (build 3a5bffa75b910d5bcdd5c607c4394b1e)

Nov 17 11:12:06 my_node1 kernel: (6282,0):dlm_request_join:1036 ERROR:
status = -107

Nov 17 11:12:06 my_node1 kernel: (6282,0):dlm_try_to_join_domain:1210
ERROR: status = -107

Nov 17 11:12:06 my_node1 kernel: (6282,0):dlm_join_domain:1488 ERROR:
status = -107

Nov 17 11:12:06 my_node1 kernel: (6282,0):dlm_register_domain:1754
ERROR: status = -107

Nov 17 11:12:06 my_node1 kernel: (6282,0):ocfs2_dlm_init:2723 ERROR:
status = -107

Nov 17 11:12:06 my_node1 kernel: (6282,0):ocfs2_mount_volume:1437 ERROR:
status = -107

Nov 17 11:12:06 my_node1 kernel: ocfs2: Unmounting device (253,9) on
(node 1)

Nov 17 11:12:27 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1

Nov 17 11:12:27 my_node1 kernel: ocfs2: Mounting device (253,9) on (node
1, slot 1) with ordered data mode.



Test 2 - Node 0

Nov 17 11:16:37 my_node0 kernel: (22166,3):dlm_send_proxy_ast_msg:458
ERROR: status = -107

Nov 17 11:16:37 my_node0 kernel: (22166,3):dlm_flush_asts:600 ERROR:
status = -107

Nov 17 11:17:35 my_node0 kernel: (22108,1):ocfs2_dlm_eviction_cb:98
device (253,9): dlm has evicted node 1

Nov 17 11:17:35 my_node0 kernel: (6515,1):ocfs2_replay_journal:1183
Recovering node 1 from slot 1 on device (253,9)

Nov 17 11:17:36 my_node0 kernel: (22167,0):dlm_get_lock_resource:844
21751145F96E45649324C9EEF5485248:$RECOVERY: at least one node (1) to
recover before lock mastery can begin

Nov 17 11:17:36 my_node0 kernel: (22167,0):dlm_get_lock_resource:878
21751145F96E45649324C9EEF5485248: recovery map is not empty, but must
master $RECOVERY lock now

Nov 17 11:17:36 my_node0 kernel: (22167,0):dlm_do_recovery:524 (22167)
Node 0 is the Recovery Master for the Dead Node 1 for Domain
21751145F96E45649324C9EEF5485248

Nov 17 11:19:31 my_node0 kernel: ocfs2_dlm: Node 1 joins domain
21751145F96E45649324C9EEF5485248

Nov 17 11:19:31 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1



Test2 - Node 1

Nov 17 11:19:22 my_node1 modprobe: FATAL: Module ocfs2_stackglue not
found.

Nov 17 11:19:23 my_node1 kernel: OCFS2 Node Manager 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)

Nov 17 11:19:23 my_node1 kernel: OCFS2 DLM 1.4.4 Tue Sep 8 11:56:46 PDT
2009 (build e6e41b84c785deeea891e5873dbf19ab)

Nov 17 11:19:23 my_node1 kernel: OCFS2 DLMFS 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)

Nov 17 11:19:23 my_node1 kernel: OCFS2 User DLM kernel interface loaded

Nov 17 11:19:31 my_node1 kernel: OCFS2 1.4.4 Tue Sep 8 11:56:43 PDT
2009 (build 3a5bffa75b910d5bcdd5c607c4394b1e)

Nov 17 11:19:31 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1

Nov 17 11:19:31 my_node1 kernel: ocfs2: Mounting device (253,8) on (node
1, slot 1) with ordered data mode.





Regards,



Mark



From: Srinivas Eeda [mailto:***@oracle.com]
Sent: 16 November 2009 16:05
To: Thompson, Mark
Cc: ocfs2-***@oss.oracle.com
Subject: Re: [Ocfs2-users] 2 node OCFS2 clusters



Thompson, Mark wrote:

Hi Srini,



Thanks for the response.



So are the following statements correct:



If I stop the networking on node 1, node 0 will continue to allow OCFS2
filesystems to work and not reboot itself.



If I stop the networking on node 0, node 1 (now being the lowest node?)
will continue to allow OCFS2 filesystems to work and not reboot itself.

In both the cases node 0 will survive, because that's the node that has
lowest node number (defined in cluster.conf). This applies to the
scenario where interconnect went down but nodes are healthy and are
heartbeating to the disk.





I guess I just need to know if it's possible to have a 2 node OCFS2
cluster that will cope with either one of the nodes dying, and have the
remaining node still provide service.

If node 0 itself panics, reboots then node 1 will survive.





Regards,



Mark



From: Srinivas Eeda [mailto:***@oracle.com]
Sent: 16 November 2009 14:57
To: Thompson, Mark
Cc: ocfs2-***@oss.oracle.com
Subject: Re: [Ocfs2-users] 2 node OCFS2 clusters



In a cluster with more than 2 nodes, if a network on one node goes down,
that node will evict itself but other nodes will survive. But in a two
node cluster, the node with lowest node number will survive no mater on
which node network went down.

thanks,
--Srini

Thompson, Mark wrote:

Hi,

This is my first post here so please be gentle with me.

My question is, can you have a 2 node OCFS2 cluster, disconnect one node
from the network, and have the remaining node continue to function
normally? Currently we have a 2 node cluster and if we stop the NIC that
has the OCFS2 o2cb net connection running on it, the other node will
reboot itself. I have researched having a 2 node OCFS2 cluster but so
far I have been unable to find a clear solution. I have looked at the
FAQ regarding quorum, and my OCFS2 init scripts are enabled etc.

Is this possible, or should we look at alternative solutions?

Regards,

Mark

This e-mail has come from Experian, the only business to have been twice
named the UK's 'Business of the Year'



========================================================================
===========

Information in this e-mail and any attachments is confidential, and may
not be copied or used by anyone other than the addressee, nor disclosed
to any third party without our permission. There is no intention to
create any legally binding contract or other binding commitment through
the use of this electronic communication unless it is issued in
accordance with the Experian Limited standard terms and conditions of
purchase or other express written agreement between Experian Limited and
the recipient.

Although Experian has taken reasonable steps to ensure that this
communication and any attachments are free from computer virus, you are
advised to take your own steps to ensure that they are actually virus
free.

Companies Act information:

Registered name: Experian Limited

Registered office: Landmark House, Experian Way, NG2 Business Park,
Nottingham, NG80 1ZZ, United Kingdom

Place of registration: England and Wales

Registered number: 653331







________________________________

[Karim Alkhayer]

How about adding a third [FAKE] node? If this is a feasible workaround, then
only the troublesome node should fence itself.



Best regards,

Karim Alkhayer



From: ocfs2-users-***@oss.oracle.com
[mailto:ocfs2-users-***@oss.oracle.com] On Behalf Of Thompson, Mark
Sent: Tuesday, November 17, 2009 2:27 PM
To: Srinivas Eeda
Cc: ocfs2-***@oss.oracle.com
Subject: Re: [Ocfs2-users] 2 node OCFS2 clusters



Hi,



I have done some more tests today, and I observed the following:



Test 1:

node 0 - ifdown eth2

node 0 - OCFS2 filesystem stalls on both nodes

node 1 - Decides to reboot

node 0 - Resumes OCFS2 service (while still off the network) OCFS2
filesystem back online

node 1 - Cannot re-join cluster as node 0 is off the network and has the fs
lock (Transport endpoint error)

node 0 - ifup eth2

node 1 - Re-joins the clusters and re-mounts OCFS2 filesystem.



Test 2:

node 1 - ifdown eth2

node 0 - OCFS2 filesystem stalls on both nodes

node 1 - Decides to reboot

node 0 - Resumes OCFS2 service, OCFS2 filesystem back online

node 1 - Boots up, re-joins cluster and re-mounts OCFS2 filesystem.



Is this the expected behaviour? And if it is, is there anything we can do
avoid the loss of the OCFS2 filesystems?





Here's the messages file outputs.



Test 1 - Node 0

Nov 17 11:00:26 my_node0 kernel: ocfs2: Unmounting device (253,9) on (node
0)

Nov 17 11:02:21 my_node0 modprobe: FATAL: Module ocfs2_stackglue not found.

Nov 17 11:02:21 my_node0 kernel: OCFS2 Node Manager 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)

Nov 17 11:02:21 my_node0 kernel: OCFS2 DLM 1.4.4 Tue Sep 8 11:56:46 PDT
2009 (build e6e41b84c785deeea891e5873dbf19ab)

Nov 17 11:02:21 my_node0 kernel: OCFS2 DLMFS 1.4.4 Tue Sep 8 11:56:46 PDT
2009 (build e6e41b84c785deeea891e5873dbf19ab)

Nov 17 11:02:21 my_node0 kernel: OCFS2 User DLM kernel interface loaded

Nov 17 11:02:46 my_node0 kernel: OCFS2 1.4.4 Tue Sep 8 11:56:43 PDT 2009
(build 3a5bffa75b910d5bcdd5c607c4394b1e)

Nov 17 11:02:46 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0

Nov 17 11:02:46 my_node0 kernel: ocfs2: Mounting device (253,9) on (node 0,
slot 0) with ordered data mode.

Nov 17 11:02:59 my_node0 kernel: ocfs2_dlm: Node 1 joins domain
21751145F96E45649324C9EEF5485248

Nov 17 11:02:59 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1

Nov 17 11:07:51 my_node0 kernel: (15,1):dlm_do_master_request:1334 ERROR:
link to 1 went down!

Nov 17 11:07:51 my_node0 kernel: (15,1):dlm_get_lock_resource:917 ERROR:
status = -107

Nov 17 11:09:34 my_node0 kernel: (22108,1):ocfs2_dlm_eviction_cb:98 device
(253,9): dlm has evicted node 1

Nov 17 11:09:34 my_node0 kernel: (29443,1):dlm_get_lock_resource:844
21751145F96E45649324C9EEF5485248:M000000000000000000001f96e7b609: at least
one node (1) to recover before lock mastery can begin

Nov 17 11:09:35 my_node0 kernel: (29443,1):dlm_get_lock_resource:898
21751145F96E45649324C9EEF5485248:M000000000000000000001f96e7b609: at least
one node (1) to recover before lock mastery can begin

Nov 17 11:09:36 my_node0 kernel: (15,1):dlm_restart_lock_mastery:1223 ERROR:
node down! 1

Nov 17 11:09:36 my_node0 kernel: (15,1):dlm_wait_for_lock_mastery:1040
ERROR: status = -11

Nov 17 11:09:36 my_node0 kernel: (22167,0):dlm_get_lock_resource:844
21751145F96E45649324C9EEF5485248:$RECOVERY: at least one node (1) to recover
before lock mastery can begin

Nov 17 11:09:36 my_node0 kernel: (22167,0):dlm_get_lock_resource:878
21751145F96E45649324C9EEF5485248: recovery map is not empty, but must master
$RECOVERY lock now

Nov 17 11:09:36 my_node0 kernel: (22167,0):dlm_do_recovery:524 (22167) Node
0 is the Recovery Master for the Dead Node 1 for Domain
21751145F96E45649324C9EEF5485248

Nov 17 11:09:46 my_node0 kernel: (29443,1):ocfs2_replay_journal:1183
Recovering node 1 from slot 1 on device (253,9)

Nov 17 11:12:27 my_node0 kernel: ocfs2_dlm: Node 1 joins domain
21751145F96E45649324C9EEF5485248

Nov 17 11:12:27 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1



Test 1 - Node 1

Nov 17 11:00:26 my_node1 kernel: ocfs2_dlm: Node 0 leaves domain
21751145F96E45649324C9EEF5485248

Nov 17 11:00:26 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 1

Nov 17 11:00:46 my_node1 kernel: ocfs2: Unmounting device (253,9) on (node
1)

Nov 17 11:02:30 my_node1 modprobe: FATAL: Module ocfs2_stackglue not found.

Nov 17 11:02:30 my_node1 kernel: OCFS2 Node Manager 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)

Nov 17 11:02:30 my_node1 kernel: OCFS2 DLM 1.4.4 Tue Sep 8 11:56:46 PDT
2009 (build e6e41b84c785deeea891e5873dbf19ab)

Nov 17 11:02:30 my_node1 kernel: OCFS2 DLMFS 1.4.4 Tue Sep 8 11:56:46 PDT
2009 (build e6e41b84c785deeea891e5873dbf19ab)

Nov 17 11:02:30 my_node1 kernel: OCFS2 User DLM kernel interface loaded

Nov 17 11:02:59 my_node1 kernel: OCFS2 1.4.4 Tue Sep 8 11:56:43 PDT 2009
(build 3a5bffa75b910d5bcdd5c607c4394b1e)

Nov 17 11:02:59 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1

Nov 17 11:02:59 my_node1 kernel: ocfs2: Mounting device (253,9) on (node 1,
slot 1) with ordered data mode.

Nov 17 11:07:27 my_node1 kernel:
(7351,3):dlm_send_remote_convert_request:395 ERROR: status = -112

Nov 17 11:07:27 my_node1 kernel: (7351,3):dlm_wait_for_node_death:370
21751145F96E45649324C9EEF5485248: waiting 5000ms for notification of death
of node 0

Nov 17 11:07:57 my_node1 kernel:
(7351,3):dlm_send_remote_convert_request:395 ERROR: status = -107

Nov 17 11:07:57 my_node1 kernel: (7351,3):dlm_wait_for_node_death:370
21751145F96E45649324C9EEF5485248: waiting 5000ms for notification of death
of node 0

Nov 17 11:08:27 my_node1 kernel: (15,1):dlm_do_master_request:1334 ERROR:
link to 0 went down!

Nov 17 11:08:27 my_node1 kernel:
(7351,3):dlm_send_remote_convert_request:395 ERROR: status = -107

Nov 17 11:08:27 my_node1 kernel: (7351,3):dlm_wait_for_node_death:370
21751145F96E45649324C9EEF5485248: waiting 5000ms for notification of death
of node 0

Nov 17 11:08:27 my_node1 kernel: (15,1):dlm_get_lock_resource:917 ERROR:
status = -107

Nov 17 11:11:31 my_node1 modprobe: FATAL: Module ocfs2_stackglue not found.

Nov 17 11:11:32 my_node1 kernel: OCFS2 Node Manager 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)

Nov 17 11:11:32 my_node1 kernel: OCFS2 DLM 1.4.4 Tue Sep 8 11:56:46 PDT
2009 (build e6e41b84c785deeea891e5873dbf19ab)

Nov 17 11:11:32 my_node1 kernel: OCFS2 DLMFS 1.4.4 Tue Sep 8 11:56:46 PDT
2009 (build e6e41b84c785deeea891e5873dbf19ab)

Nov 17 11:11:32 my_node1 kernel: OCFS2 User DLM kernel interface loaded

Nov 17 11:11:40 my_node1 kernel: OCFS2 1.4.4 Tue Sep 8 11:56:43 PDT 2009
(build 3a5bffa75b910d5bcdd5c607c4394b1e)

Nov 17 11:12:06 my_node1 kernel: (6282,0):dlm_request_join:1036 ERROR:
status = -107

Nov 17 11:12:06 my_node1 kernel: (6282,0):dlm_try_to_join_domain:1210 ERROR:
status = -107

Nov 17 11:12:06 my_node1 kernel: (6282,0):dlm_join_domain:1488 ERROR: status
= -107

Nov 17 11:12:06 my_node1 kernel: (6282,0):dlm_register_domain:1754 ERROR:
status = -107

Nov 17 11:12:06 my_node1 kernel: (6282,0):ocfs2_dlm_init:2723 ERROR: status
= -107

Nov 17 11:12:06 my_node1 kernel: (6282,0):ocfs2_mount_volume:1437 ERROR:
status = -107

Nov 17 11:12:06 my_node1 kernel: ocfs2: Unmounting device (253,9) on (node
1)

Nov 17 11:12:27 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1

Nov 17 11:12:27 my_node1 kernel: ocfs2: Mounting device (253,9) on (node 1,
slot 1) with ordered data mode.



Test 2 - Node 0

Nov 17 11:16:37 my_node0 kernel: (22166,3):dlm_send_proxy_ast_msg:458 ERROR:
status = -107

Nov 17 11:16:37 my_node0 kernel: (22166,3):dlm_flush_asts:600 ERROR: status
= -107

Nov 17 11:17:35 my_node0 kernel: (22108,1):ocfs2_dlm_eviction_cb:98 device
(253,9): dlm has evicted node 1

Nov 17 11:17:35 my_node0 kernel: (6515,1):ocfs2_replay_journal:1183
Recovering node 1 from slot 1 on device (253,9)

Nov 17 11:17:36 my_node0 kernel: (22167,0):dlm_get_lock_resource:844
21751145F96E45649324C9EEF5485248:$RECOVERY: at least one node (1) to recover
before lock mastery can begin

Nov 17 11:17:36 my_node0 kernel: (22167,0):dlm_get_lock_resource:878
21751145F96E45649324C9EEF5485248: recovery map is not empty, but must master
$RECOVERY lock now

Nov 17 11:17:36 my_node0 kernel: (22167,0):dlm_do_recovery:524 (22167) Node
0 is the Recovery Master for the Dead Node 1 for Domain
21751145F96E45649324C9EEF5485248

Nov 17 11:19:31 my_node0 kernel: ocfs2_dlm: Node 1 joins domain
21751145F96E45649324C9EEF5485248

Nov 17 11:19:31 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1



Test2 - Node 1

Nov 17 11:19:22 my_node1 modprobe: FATAL: Module ocfs2_stackglue not found.

Nov 17 11:19:23 my_node1 kernel: OCFS2 Node Manager 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)

Nov 17 11:19:23 my_node1 kernel: OCFS2 DLM 1.4.4 Tue Sep 8 11:56:46 PDT
2009 (build e6e41b84c785deeea891e5873dbf19ab)

Nov 17 11:19:23 my_node1 kernel: OCFS2 DLMFS 1.4.4 Tue Sep 8 11:56:46 PDT
2009 (build e6e41b84c785deeea891e5873dbf19ab)

Nov 17 11:19:23 my_node1 kernel: OCFS2 User DLM kernel interface loaded

Nov 17 11:19:31 my_node1 kernel: OCFS2 1.4.4 Tue Sep 8 11:56:43 PDT 2009
(build 3a5bffa75b910d5bcdd5c607c4394b1e)

Nov 17 11:19:31 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1

Nov 17 11:19:31 my_node1 kernel: ocfs2: Mounting device (253,8) on (node 1,
slot 1) with ordered data mode.





Regards,



Mark



From: Srinivas Eeda [mailto:***@oracle.com]
Sent: 16 November 2009 16:05
To: Thompson, Mark
Cc: ocfs2-***@oss.oracle.com
Subject: Re: [Ocfs2-users] 2 node OCFS2 clusters



Thompson, Mark wrote:

Hi Srini,



Thanks for the response.



So are the following statements correct:



If I stop the networking on node 1, node 0 will continue to allow OCFS2
filesystems to work and not reboot itself.



If I stop the networking on node 0, node 1 (now being the lowest node?) will
continue to allow OCFS2 filesystems to work and not reboot itself.

In both the cases node 0 will survive, because that's the node that has
lowest node number (defined in cluster.conf). This applies to the scenario
where interconnect went down but nodes are healthy and are heartbeating to
the disk.



I guess I just need to know if it's possible to have a 2 node OCFS2 cluster
that will cope with either one of the nodes dying, and have the remaining
node still provide service.

If node 0 itself panics, reboots then node 1 will survive.



Regards,



Mark



From: Srinivas Eeda [mailto:***@oracle.com]
Sent: 16 November 2009 14:57
To: Thompson, Mark
Cc: ocfs2-***@oss.oracle.com
Subject: Re: [Ocfs2-users] 2 node OCFS2 clusters



In a cluster with more than 2 nodes, if a network on one node goes down,
that node will evict itself but other nodes will survive. But in a two node
cluster, the node with lowest node number will survive no mater on which
node network went down.

thanks,
--Srini

Thompson, Mark wrote:

Hi,

This is my first post here so please be gentle with me.

My question is, can you have a 2 node OCFS2 cluster, disconnect one node
from the network, and have the remaining node continue to function normally?
Currently we have a 2 node cluster and if we stop the NIC that has the OCFS2
o2cb net connection running on it, the other node will reboot itself. I have
researched having a 2 node OCFS2 cluster but so far I have been unable to
find a clear solution. I have looked at the FAQ regarding quorum, and my
OCFS2 init scripts are enabled etc.

Is this possible, or should we look at alternative solutions?

Regards,

Mark

This e-mail has come from Experian, the only business to have been twice
named the UK's 'Business of the Year'



============================================================================
=======

Information in this e-mail and any attachments is confidential, and may not
be copied or used by anyone other than the addressee, nor disclosed to any
third party without our permission. There is no intention to create any
legally binding contract or other binding commitment through the use of this
electronic communication unless it is issued in accordance with the Experian
Limited standard terms and conditions of purchase or other express written
agreement between Experian Limited and the recipient.

Although Experian has taken reasonable steps to ensure that this
communication and any attachments are free from computer virus, you are
advised to take your own steps to ensure that they are actually virus free.

Companies Act information:

Registered name: Experian Limited

Registered office: Landmark House, Experian Way, NG2 Business Park,
Nottingham, NG80 1ZZ, United Kingdom

Place of registration: England and Wales

Registered number: 653331












_____

[Srinivas Eeda]

What do you mean by loss of file system? Are you referring to the time
when file system is hung? If so, that is unavoidable. They have to talk
to each other whenever a file system metadata is changing. When network
went down, they were waiting for it to come back. Since it was taking
longer, quorum logic kicked in and took down one of the node(node with
higher node number). If network was backup before the quorum kicked in,
they would have been fine.

May I ask what your expectations are when the network goes down.

thanks,
--Srini

Post by Thompson, Mark
Hi,
node 0 - ifdown eth2
node 0 - OCFS2 filesystem stalls on both nodes
node 1 - Decides to reboot
node 0 - Resumes OCFS2 service (while still off the network) OCFS2
filesystem back online
node 1 - Cannot re-join cluster as node 0 is off the network and has
the fs lock (Transport endpoint error)
node 0 - ifup eth2
node 1 - Re-joins the clusters and re-mounts OCFS2 filesystem.
node 1 - ifdown eth2
node 0 - OCFS2 filesystem stalls on both nodes
node 1 - Decides to reboot
node 0 - Resumes OCFS2 service, OCFS2 filesystem back online
node 1 -- Boots up, re-joins cluster and re-mounts OCFS2 filesystem.
Is this the expected behaviour? And if it is, is there anything we can
do avoid the loss of the OCFS2 filesystems?
Here's the messages file outputs.
Test 1 - Node 0
Nov 17 11:00:26 my_node0 kernel: ocfs2: Unmounting device (253,9) on
(node 0)
Nov 17 11:02:21 my_node0 modprobe: FATAL: Module ocfs2_stackglue not
found.
Nov 17 11:02:21 my_node0 kernel: OCFS2 Node Manager 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)
Nov 17 11:02:21 my_node0 kernel: OCFS2 DLM 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)
Nov 17 11:02:21 my_node0 kernel: OCFS2 DLMFS 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)
Nov 17 11:02:21 my_node0 kernel: OCFS2 User DLM kernel interface loaded
Nov 17 11:02:46 my_node0 kernel: OCFS2 1.4.4 Tue Sep 8 11:56:43 PDT
2009 (build 3a5bffa75b910d5bcdd5c607c4394b1e)
Nov 17 11:02:46 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0
Nov 17 11:02:46 my_node0 kernel: ocfs2: Mounting device (253,9) on
(node 0, slot 0) with ordered data mode.
Nov 17 11:02:59 my_node0 kernel: ocfs2_dlm: Node 1 joins domain
21751145F96E45649324C9EEF5485248
Nov 17 11:02:59 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1
Nov 17 11:07:51 my_node0 kernel: (15,1):dlm_do_master_request:1334
ERROR: link to 1 went down!
Nov 17 11:07:51 my_node0 kernel: (15,1):dlm_get_lock_resource:917
ERROR: status = -107
Nov 17 11:09:34 my_node0 kernel: (22108,1):ocfs2_dlm_eviction_cb:98
device (253,9): dlm has evicted node 1
Nov 17 11:09:34 my_node0 kernel: (29443,1):dlm_get_lock_resource:844
21751145F96E45649324C9EEF5485248:M000000000000000000001f96e7b609: at
least one node (1) to recover before lock mastery can begin
Nov 17 11:09:35 my_node0 kernel: (29443,1):dlm_get_lock_resource:898
21751145F96E45649324C9EEF5485248:M000000000000000000001f96e7b609: at
least one node (1) to recover before lock mastery can begin
Nov 17 11:09:36 my_node0 kernel: (15,1):dlm_restart_lock_mastery:1223
ERROR: node down! 1
Nov 17 11:09:36 my_node0 kernel: (15,1):dlm_wait_for_lock_mastery:1040
ERROR: status = -11
Nov 17 11:09:36 my_node0 kernel: (22167,0):dlm_get_lock_resource:844
21751145F96E45649324C9EEF5485248:$RECOVERY: at least one node (1) to
recover before lock mastery can begin
Nov 17 11:09:36 my_node0 kernel: (22167,0):dlm_get_lock_resource:878
21751145F96E45649324C9EEF5485248: recovery map is not empty, but must
master $RECOVERY lock now
Nov 17 11:09:36 my_node0 kernel: (22167,0):dlm_do_recovery:524 (22167)
Node 0 is the Recovery Master for the Dead Node 1 for Domain
21751145F96E45649324C9EEF5485248
Nov 17 11:09:46 my_node0 kernel: (29443,1):ocfs2_replay_journal:1183
Recovering node 1 from slot 1 on device (253,9)
Nov 17 11:12:27 my_node0 kernel: ocfs2_dlm: Node 1 joins domain
21751145F96E45649324C9EEF5485248
Nov 17 11:12:27 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1
Test 1 -- Node 1
Nov 17 11:00:26 my_node1 kernel: ocfs2_dlm: Node 0 leaves domain
21751145F96E45649324C9EEF5485248
Nov 17 11:00:26 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 1
Nov 17 11:00:46 my_node1 kernel: ocfs2: Unmounting device (253,9) on
(node 1)
Nov 17 11:02:30 my_node1 modprobe: FATAL: Module ocfs2_stackglue not
found.
Nov 17 11:02:30 my_node1 kernel: OCFS2 Node Manager 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)
Nov 17 11:02:30 my_node1 kernel: OCFS2 DLM 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)
Nov 17 11:02:30 my_node1 kernel: OCFS2 DLMFS 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)
Nov 17 11:02:30 my_node1 kernel: OCFS2 User DLM kernel interface loaded
Nov 17 11:02:59 my_node1 kernel: OCFS2 1.4.4 Tue Sep 8 11:56:43 PDT
2009 (build 3a5bffa75b910d5bcdd5c607c4394b1e)
Nov 17 11:02:59 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1
Nov 17 11:02:59 my_node1 kernel: ocfs2: Mounting device (253,9) on
(node 1, slot 1) with ordered data mode.
(7351,3):dlm_send_remote_convert_request:395 ERROR: status = -112
Nov 17 11:07:27 my_node1 kernel: (7351,3):dlm_wait_for_node_death:370
21751145F96E45649324C9EEF5485248: waiting 5000ms for notification of
death of node 0
(7351,3):dlm_send_remote_convert_request:395 ERROR: status = -107
Nov 17 11:07:57 my_node1 kernel: (7351,3):dlm_wait_for_node_death:370
21751145F96E45649324C9EEF5485248: waiting 5000ms for notification of
death of node 0
Nov 17 11:08:27 my_node1 kernel: (15,1):dlm_do_master_request:1334
ERROR: link to 0 went down!
(7351,3):dlm_send_remote_convert_request:395 ERROR: status = -107
Nov 17 11:08:27 my_node1 kernel: (7351,3):dlm_wait_for_node_death:370
21751145F96E45649324C9EEF5485248: waiting 5000ms for notification of
death of node 0
Nov 17 11:08:27 my_node1 kernel: (15,1):dlm_get_lock_resource:917
ERROR: status = -107
Nov 17 11:11:31 my_node1 modprobe: FATAL: Module ocfs2_stackglue not
found.
Nov 17 11:11:32 my_node1 kernel: OCFS2 Node Manager 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)
Nov 17 11:11:32 my_node1 kernel: OCFS2 DLM 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)
Nov 17 11:11:32 my_node1 kernel: OCFS2 DLMFS 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)
Nov 17 11:11:32 my_node1 kernel: OCFS2 User DLM kernel interface loaded
Nov 17 11:11:40 my_node1 kernel: OCFS2 1.4.4 Tue Sep 8 11:56:43 PDT
2009 (build 3a5bffa75b910d5bcdd5c607c4394b1e)
status = -107
Nov 17 11:12:06 my_node1 kernel: (6282,0):dlm_try_to_join_domain:1210
ERROR: status = -107
status = -107
Nov 17 11:12:06 my_node1 kernel: (6282,0):dlm_register_domain:1754
ERROR: status = -107
status = -107
Nov 17 11:12:06 my_node1 kernel: (6282,0):ocfs2_mount_volume:1437
ERROR: status = -107
Nov 17 11:12:06 my_node1 kernel: ocfs2: Unmounting device (253,9) on
(node 1)
Nov 17 11:12:27 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1
Nov 17 11:12:27 my_node1 kernel: ocfs2: Mounting device (253,9) on
(node 1, slot 1) with ordered data mode.
Test 2 -- Node 0
Nov 17 11:16:37 my_node0 kernel: (22166,3):dlm_send_proxy_ast_msg:458
ERROR: status = -107
status = -107
Nov 17 11:17:35 my_node0 kernel: (22108,1):ocfs2_dlm_eviction_cb:98
device (253,9): dlm has evicted node 1
Nov 17 11:17:35 my_node0 kernel: (6515,1):ocfs2_replay_journal:1183
Recovering node 1 from slot 1 on device (253,9)
Nov 17 11:17:36 my_node0 kernel: (22167,0):dlm_get_lock_resource:844
21751145F96E45649324C9EEF5485248:$RECOVERY: at least one node (1) to
recover before lock mastery can begin
Nov 17 11:17:36 my_node0 kernel: (22167,0):dlm_get_lock_resource:878
21751145F96E45649324C9EEF5485248: recovery map is not empty, but must
master $RECOVERY lock now
Nov 17 11:17:36 my_node0 kernel: (22167,0):dlm_do_recovery:524 (22167)
Node 0 is the Recovery Master for the Dead Node 1 for Domain
21751145F96E45649324C9EEF5485248
Nov 17 11:19:31 my_node0 kernel: ocfs2_dlm: Node 1 joins domain
21751145F96E45649324C9EEF5485248
Nov 17 11:19:31 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1
Test2 -- Node 1
Nov 17 11:19:22 my_node1 modprobe: FATAL: Module ocfs2_stackglue not
found.
Nov 17 11:19:23 my_node1 kernel: OCFS2 Node Manager 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)
Nov 17 11:19:23 my_node1 kernel: OCFS2 DLM 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)
Nov 17 11:19:23 my_node1 kernel: OCFS2 DLMFS 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)
Nov 17 11:19:23 my_node1 kernel: OCFS2 User DLM kernel interface loaded
Nov 17 11:19:31 my_node1 kernel: OCFS2 1.4.4 Tue Sep 8 11:56:43 PDT
2009 (build 3a5bffa75b910d5bcdd5c607c4394b1e)
Nov 17 11:19:31 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1
Nov 17 11:19:31 my_node1 kernel: ocfs2: Mounting device (253,8) on
(node 1, slot 1) with ordered data mode.
Regards,
Mark
*Sent:* 16 November 2009 16:05
*To:* Thompson, Mark
*Subject:* Re: [Ocfs2-users] 2 node OCFS2 clusters
Hi Srini,
Thanks for the response.
If I stop the networking on node 1, node 0 will continue to allow
OCFS2 filesystems to work and not reboot itself.
If I stop the networking on node 0, node 1 (now being the lowest
node?) will continue to allow OCFS2 filesystems to work and not reboot
itself.
In both the cases node 0 will survive, because that's the node that
has lowest node number (defined in cluster.conf). This applies to the
scenario where interconnect went down but nodes are healthy and are
heartbeating to the disk.
I guess I just need to know if it's possible to have a 2 node OCFS2
cluster that will cope with either one of the nodes dying, and have
the remaining node still provide service.
If node 0 itself panics, reboots then node 1 will survive.
Regards,
Mark
*Sent:* 16 November 2009 14:57
*To:* Thompson, Mark
*Subject:* Re: [Ocfs2-users] 2 node OCFS2 clusters
In a cluster with more than 2 nodes, if a network on one node goes
down, that node will evict itself but other nodes will survive. But in
a two node cluster, the node with lowest node number will survive no
mater on which node network went down.
thanks,
--Srini
Hi,
This is my first post here so please be gentle with me.
My question is, can you have a 2 node OCFS2 cluster, disconnect one
node from the network, and have the remaining node continue to
function normally? Currently we have a 2 node cluster and if we stop
the NIC that has the OCFS2 o2cb net connection running on it, the
other node will reboot itself. I have researched having a 2 node OCFS2
cluster but so far I have been unable to find a clear solution. I have
looked at the FAQ regarding quorum, and my OCFS2 init scripts are
enabled etc.
Is this possible, or should we look at alternative solutions?
Regards,
Mark
This e-mail has come from Experian, the only business to have been
twice named the UK's 'Business of the Year'
===================================================================================
Information in this e-mail and any attachments is confidential, and
may not be copied or used by anyone other than the addressee, nor
disclosed to any third party without our permission. There is no
intention to create any legally binding contract or other binding
commitment through the use of this electronic communication unless it
is issued in accordance with the Experian Limited standard terms and
conditions of purchase or other express written agreement between
Experian Limited and the recipient.
Although Experian has taken reasonable steps to ensure that this
communication and any attachments are free from computer virus, you
are advised to take your own steps to ensure that they are actually
virus free.
Registered name: Experian Limited
Registered office: Landmark House, Experian Way, NG2 Business Park,
Nottingham, NG80 1ZZ, United Kingdom
Place of registration: England and Wales
Registered number: 653331
------------------------------------------------------------------------
_______________________________________________
Ocfs2-users mailing list
http://oss.oracle.com/mailman/listinfo/ocfs2-users

[Sunil Mushran]

Yes, it is working as expected.

Note that in a shared disk clustered file system, all nodes can
_write_ to the disk independently of the other nodes. They each
have a direct path to the disk. That's what allows such filesystems
to have a higher throughput than, say NFS, that directs all the IOs
to the disk via a NFS server.

But if we let all nodes write to the disk without any coordination,
we'll end up with a corrupted file system. To see this in action,
format the volume with mkfs.ext3 and mount it on both the nodes and
untar some tarballs on both nodes.

We use the interconnect for coordination. We send lots of very small
packets that allow nodes to take and drop locks on various resources.

Now when the interconnect fails, we have 3 choices:

1. Ignore the failure and continue without the coordination.
Result: Corrupted file system.

2. Wait for the sys-admin to fix the network problem.
Result: Cluster operations hang until the sys-admin fixes the issue.

3. Fence off node(s) that will allow the remaining nodes to continue.
Result: One or more nodes are reset while the remaining nodes continue
operating. The reset nodes restart and, with the help of startup
scripts, rejoin the cluster and continue doing their task.

OCFS2 chooses the third option.

And yes, adding a third node allows OCFS2 to triangulate... to better
determine the problematic node.

Sunil

Post by Thompson, Mark
Hi,
node 0 - ifdown eth2
node 0 - OCFS2 filesystem stalls on both nodes
node 1 - Decides to reboot
node 0 - Resumes OCFS2 service (while still off the network) OCFS2
filesystem back online
node 1 - Cannot re-join cluster as node 0 is off the network and has
the fs lock (Transport endpoint error)
node 0 - ifup eth2
node 1 - Re-joins the clusters and re-mounts OCFS2 filesystem.
node 1 - ifdown eth2
node 0 - OCFS2 filesystem stalls on both nodes
node 1 - Decides to reboot
node 0 - Resumes OCFS2 service, OCFS2 filesystem back online
node 1 – Boots up, re-joins cluster and re-mounts OCFS2 filesystem.
Is this the expected behaviour? And if it is, is there anything we can
do avoid the loss of the OCFS2 filesystems?
Here’s the messages file outputs.
Test 1 - Node 0
Nov 17 11:00:26 my_node0 kernel: ocfs2: Unmounting device (253,9) on
(node 0)
Nov 17 11:02:21 my_node0 modprobe: FATAL: Module ocfs2_stackglue not
found.
Nov 17 11:02:21 my_node0 kernel: OCFS2 Node Manager 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)
Nov 17 11:02:21 my_node0 kernel: OCFS2 DLM 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)
Nov 17 11:02:21 my_node0 kernel: OCFS2 DLMFS 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)
Nov 17 11:02:21 my_node0 kernel: OCFS2 User DLM kernel interface loaded
Nov 17 11:02:46 my_node0 kernel: OCFS2 1.4.4 Tue Sep 8 11:56:43 PDT
2009 (build 3a5bffa75b910d5bcdd5c607c4394b1e)
Nov 17 11:02:46 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0
Nov 17 11:02:46 my_node0 kernel: ocfs2: Mounting device (253,9) on
(node 0, slot 0) with ordered data mode.
Nov 17 11:02:59 my_node0 kernel: ocfs2_dlm: Node 1 joins domain
21751145F96E45649324C9EEF5485248
Nov 17 11:02:59 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1
Nov 17 11:07:51 my_node0 kernel: (15,1):dlm_do_master_request:1334
ERROR: link to 1 went down!
Nov 17 11:07:51 my_node0 kernel: (15,1):dlm_get_lock_resource:917
ERROR: status = -107
Nov 17 11:09:34 my_node0 kernel: (22108,1):ocfs2_dlm_eviction_cb:98
device (253,9): dlm has evicted node 1
Nov 17 11:09:34 my_node0 kernel: (29443,1):dlm_get_lock_resource:844
21751145F96E45649324C9EEF5485248:M000000000000000000001f96e7b609: at
least one node (1) to recover before lock mastery can begin
Nov 17 11:09:35 my_node0 kernel: (29443,1):dlm_get_lock_resource:898
21751145F96E45649324C9EEF5485248:M000000000000000000001f96e7b609: at
least one node (1) to recover before lock mastery can begin
Nov 17 11:09:36 my_node0 kernel: (15,1):dlm_restart_lock_mastery:1223
ERROR: node down! 1
Nov 17 11:09:36 my_node0 kernel: (15,1):dlm_wait_for_lock_mastery:1040
ERROR: status = -11
Nov 17 11:09:36 my_node0 kernel: (22167,0):dlm_get_lock_resource:844
21751145F96E45649324C9EEF5485248:$RECOVERY: at least one node (1) to
recover before lock mastery can begin
Nov 17 11:09:36 my_node0 kernel: (22167,0):dlm_get_lock_resource:878
21751145F96E45649324C9EEF5485248: recovery map is not empty, but must
master $RECOVERY lock now
Nov 17 11:09:36 my_node0 kernel: (22167,0):dlm_do_recovery:524 (22167)
Node 0 is the Recovery Master for the Dead Node 1 for Domain
21751145F96E45649324C9EEF5485248
Nov 17 11:09:46 my_node0 kernel: (29443,1):ocfs2_replay_journal:1183
Recovering node 1 from slot 1 on device (253,9)
Nov 17 11:12:27 my_node0 kernel: ocfs2_dlm: Node 1 joins domain
21751145F96E45649324C9EEF5485248
Nov 17 11:12:27 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1
Test 1 – Node 1
Nov 17 11:00:26 my_node1 kernel: ocfs2_dlm: Node 0 leaves domain
21751145F96E45649324C9EEF5485248
Nov 17 11:00:26 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 1
Nov 17 11:00:46 my_node1 kernel: ocfs2: Unmounting device (253,9) on
(node 1)
Nov 17 11:02:30 my_node1 modprobe: FATAL: Module ocfs2_stackglue not
found.
Nov 17 11:02:30 my_node1 kernel: OCFS2 Node Manager 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)
Nov 17 11:02:30 my_node1 kernel: OCFS2 DLM 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)
Nov 17 11:02:30 my_node1 kernel: OCFS2 DLMFS 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)
Nov 17 11:02:30 my_node1 kernel: OCFS2 User DLM kernel interface loaded
Nov 17 11:02:59 my_node1 kernel: OCFS2 1.4.4 Tue Sep 8 11:56:43 PDT
2009 (build 3a5bffa75b910d5bcdd5c607c4394b1e)
Nov 17 11:02:59 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1
Nov 17 11:02:59 my_node1 kernel: ocfs2: Mounting device (253,9) on
(node 1, slot 1) with ordered data mode.
(7351,3):dlm_send_remote_convert_request:395 ERROR: status = -112
Nov 17 11:07:27 my_node1 kernel: (7351,3):dlm_wait_for_node_death:370
21751145F96E45649324C9EEF5485248: waiting 5000ms for notification of
death of node 0
(7351,3):dlm_send_remote_convert_request:395 ERROR: status = -107
Nov 17 11:07:57 my_node1 kernel: (7351,3):dlm_wait_for_node_death:370
21751145F96E45649324C9EEF5485248: waiting 5000ms for notification of
death of node 0
Nov 17 11:08:27 my_node1 kernel: (15,1):dlm_do_master_request:1334
ERROR: link to 0 went down!
(7351,3):dlm_send_remote_convert_request:395 ERROR: status = -107
Nov 17 11:08:27 my_node1 kernel: (7351,3):dlm_wait_for_node_death:370
21751145F96E45649324C9EEF5485248: waiting 5000ms for notification of
death of node 0
Nov 17 11:08:27 my_node1 kernel: (15,1):dlm_get_lock_resource:917
ERROR: status = -107
Nov 17 11:11:31 my_node1 modprobe: FATAL: Module ocfs2_stackglue not
found.
Nov 17 11:11:32 my_node1 kernel: OCFS2 Node Manager 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)
Nov 17 11:11:32 my_node1 kernel: OCFS2 DLM 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)
Nov 17 11:11:32 my_node1 kernel: OCFS2 DLMFS 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)
Nov 17 11:11:32 my_node1 kernel: OCFS2 User DLM kernel interface loaded
Nov 17 11:11:40 my_node1 kernel: OCFS2 1.4.4 Tue Sep 8 11:56:43 PDT
2009 (build 3a5bffa75b910d5bcdd5c607c4394b1e)
status = -107
Nov 17 11:12:06 my_node1 kernel: (6282,0):dlm_try_to_join_domain:1210
ERROR: status = -107
status = -107
Nov 17 11:12:06 my_node1 kernel: (6282,0):dlm_register_domain:1754
ERROR: status = -107
status = -107
Nov 17 11:12:06 my_node1 kernel: (6282,0):ocfs2_mount_volume:1437
ERROR: status = -107
Nov 17 11:12:06 my_node1 kernel: ocfs2: Unmounting device (253,9) on
(node 1)
Nov 17 11:12:27 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1
Nov 17 11:12:27 my_node1 kernel: ocfs2: Mounting device (253,9) on
(node 1, slot 1) with ordered data mode.
Test 2 – Node 0
Nov 17 11:16:37 my_node0 kernel: (22166,3):dlm_send_proxy_ast_msg:458
ERROR: status = -107
status = -107
Nov 17 11:17:35 my_node0 kernel: (22108,1):ocfs2_dlm_eviction_cb:98
device (253,9): dlm has evicted node 1
Nov 17 11:17:35 my_node0 kernel: (6515,1):ocfs2_replay_journal:1183
Recovering node 1 from slot 1 on device (253,9)
Nov 17 11:17:36 my_node0 kernel: (22167,0):dlm_get_lock_resource:844
21751145F96E45649324C9EEF5485248:$RECOVERY: at least one node (1) to
recover before lock mastery can begin
Nov 17 11:17:36 my_node0 kernel: (22167,0):dlm_get_lock_resource:878
21751145F96E45649324C9EEF5485248: recovery map is not empty, but must
master $RECOVERY lock now
Nov 17 11:17:36 my_node0 kernel: (22167,0):dlm_do_recovery:524 (22167)
Node 0 is the Recovery Master for the Dead Node 1 for Domain
21751145F96E45649324C9EEF5485248
Nov 17 11:19:31 my_node0 kernel: ocfs2_dlm: Node 1 joins domain
21751145F96E45649324C9EEF5485248
Nov 17 11:19:31 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1
Test2 – Node 1
Nov 17 11:19:22 my_node1 modprobe: FATAL: Module ocfs2_stackglue not
found.
Nov 17 11:19:23 my_node1 kernel: OCFS2 Node Manager 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)
Nov 17 11:19:23 my_node1 kernel: OCFS2 DLM 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)
Nov 17 11:19:23 my_node1 kernel: OCFS2 DLMFS 1.4.4 Tue Sep 8 11:56:46
PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)
Nov 17 11:19:23 my_node1 kernel: OCFS2 User DLM kernel interface loaded
Nov 17 11:19:31 my_node1 kernel: OCFS2 1.4.4 Tue Sep 8 11:56:43 PDT
2009 (build 3a5bffa75b910d5bcdd5c607c4394b1e)
Nov 17 11:19:31 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1
Nov 17 11:19:31 my_node1 kernel: ocfs2: Mounting device (253,8) on
(node 1, slot 1) with ordered data mode.
Regards,
Mark
*Sent:* 16 November 2009 16:05
*To:* Thompson, Mark
*Subject:* Re: [Ocfs2-users] 2 node OCFS2 clusters
Hi Srini,
Thanks for the response.
If I stop the networking on node 1, node 0 will continue to allow
OCFS2 filesystems to work and not reboot itself.
If I stop the networking on node 0, node 1 (now being the lowest
node?) will continue to allow OCFS2 filesystems to work and not reboot
itself.
In both the cases node 0 will survive, because that's the node that
has lowest node number (defined in cluster.conf). This applies to the
scenario where interconnect went down but nodes are healthy and are
heartbeating to the disk.
I guess I just need to know if it’s possible to have a 2 node OCFS2
cluster that will cope with either one of the nodes dying, and have
the remaining node still provide service.
If node 0 itself panics, reboots then node 1 will survive.
Regards,
Mark
*Sent:* 16 November 2009 14:57
*To:* Thompson, Mark
*Subject:* Re: [Ocfs2-users] 2 node OCFS2 clusters
In a cluster with more than 2 nodes, if a network on one node goes
down, that node will evict itself but other nodes will survive. But in
a two node cluster, the node with lowest node number will survive no
mater on which node network went down.
thanks,
--Srini
Hi,
This is my first post here so please be gentle with me.
My question is, can you have a 2 node OCFS2 cluster, disconnect one
node from the network, and have the remaining node continue to
function normally? Currently we have a 2 node cluster and if we stop
the NIC that has the OCFS2 o2cb net connection running on it, the
other node will reboot itself. I have researched having a 2 node OCFS2
cluster but so far I have been unable to find a clear solution. I have
looked at the FAQ regarding quorum, and my OCFS2 init scripts are
enabled etc.
Is this possible, or should we look at alternative solutions?
Regards,
Mark
This e-mail has come from Experian, the only business to have been
twice named the UK's 'Business of the Year’
===================================================================================
Information in this e-mail and any attachments is confidential, and
may not be copied or used by anyone other than the addressee, nor
disclosed to any third party without our permission. There is no
intention to create any legally binding contract or other binding
commitment through the use of this electronic communication unless it
is issued in accordance with the Experian Limited standard terms and
conditions of purchase or other express written agreement between
Experian Limited and the recipient.
Although Experian has taken reasonable steps to ensure that this
communication and any attachments are free from computer virus, you
are advised to take your own steps to ensure that they are actually
virus free.
Registered name: Experian Limited
Registered office: Landmark House, Experian Way, NG2 Business Park,
Nottingham, NG80 1ZZ, United Kingdom
Place of registration: England and Wales
Registered number: 653331
------------------------------------------------------------------------
_______________________________________________
Ocfs2-users mailing list
http://oss.oracle.com/mailman/listinfo/ocfs2-users
------------------------------------------------------------------------
_______________________________________________
Ocfs2-users mailing list
http://oss.oracle.com/mailman/listinfo/ocfs2-users

[Luis Freitas]

Srinivas,

If this is true then I would sugest OCFS2 is not taking the best decision in this scenario.

The node that still has network connectivity should survive instead of the lowest node number. Oracle CRS has heuristics to detect if the network is down and in this scenario the node that lost network conectivy is evicted. That is why it is required to use a switch between the two nodes, instead of a cross cable.

OCFS2 should do the same.

Best Regards,
Luis Freitas

Post by Thompson, Mark
Subject: Re: [Ocfs2-users] 2 node OCFS2 clusters
Date: Monday, November 16, 2009, 2:05 PM
2 node OCFS2 clusters
Hi
Srini,
 
Thanks
for the response.
 
So
 
If
I stop the networking on node 1, node 0 will continue to
allow OCFS2 filesystems to work and not reboot itself.
 
If
I stop the networking on node 0, node 1 (now being the
lowest
node?) will continue to allow OCFS2 filesystems to work and
not reboot
itself.
In both the cases node 0 will survive, because that's
the node that has
lowest node number (defined in cluster.conf). This applies
to the
scenario where interconnect went down but nodes are healthy
and are
heartbeating to the disk.
 
I
guess I just need to know if it’s possible to have a
2
node OCFS2 cluster that will cope with either one of the
nodes dying,
and have
the remaining node still provide service.
If node 0 itself panics, reboots then node 1 will survive.
 
Regards,
 
Mark
 
Sent: 16 November 2009 14:57
To: Thompson, Mark
Subject: Re: [Ocfs2-users] 2 node OCFS2
clusters
 
In a cluster with more than 2 nodes,
if a
network on one
node goes down, that node will evict itself but other nodes
will
survive. But
in a two node cluster, the node with lowest node number
will survive no
mater
on which node network went down.
thanks,
--Srini
Hi,
This is my
first post here
so please be
gentle
with me.
My question is,
can you
have a 2 node OCFS2 cluster, disconnect one node from the
network, and
have the
remaining node continue to function normally? Currently we
have a 2
node
cluster and if we stop the NIC that has the OCFS2 o2cb net
connection
running
on it, the other
node
will reboot itself. I
have researched having a 2 node OCFS2 cluster but so far I
have been
unable to
find a clear solution. I have looked at the FAQ regarding
quorum,
and my OCFS2 init
scripts are
enabled etc.
Is this
possible, or should
we look at alternative solutions?
Regards,
Mark
This
e-mail
has come from Experian, the only business to have been
twice named the
UK's
'Business of the Year’
 
===================================================================================
Information
in this e-mail and any attachments is confidential, and may
not be
copied or
used by anyone other than the addressee, nor disclosed to
any third
party
without our permission. There is no intention to create any
legally
binding
contract or other binding commitment through the use of
this electronic
communication
unless it is issued in accordance with the Experian Limited
standard
terms and
conditions of purchase or other express written agreement
between
Experian
Limited and the recipient.
Although
Experian has taken reasonable steps to ensure that this
communication
and any
attachments are free from computer virus, you are advised
to take your
own
steps to ensure that they are actually virus free.
Companies
Act
Registered
name: Experian Limited
Registered
office: Landmark House, Experian Way, NG2 Business Park,
Nottingham,
NG80 1ZZ,
United Kingdom
Place
of
registration: England and Wales
Registered
number: 653331
 
 
 
_______________________________________________
Ocfs2-users mailing list
http://oss.oracle.com/mailman/listinfo/ocfs2-users
-----Inline Attachment Follows-----
_______________________________________________
Ocfs2-users mailing list
http://oss.oracle.com/mailman/listinfo/ocfs2-users

[Joel Becker]

Post by Luis Freitas
If this is true then I would sugest OCFS2 is not taking the best decision in this scenario.

It's doing the best it can. See, each node knows the other is
alive because of the disk heartbeat. Thus they know the error is
network related. But with only two nodes, there is no way to determine
who has the better network.

Post by Luis Freitas
The node that still has network connectivity should survive instead of the lowest node number. Oracle CRS has heuristics to detect if the network is down and in this scenario the node that lost network conectivy is evicted. That is why it is required to use a switch between the two nodes, instead of a cross cable.

ocfs2 is a general purpose filesystem that allows a myriad of
network configurations. It has never required a dedicated network
switch, nor will it, so relying on that isn't possible.

Joel
--
"The whole problem with the world is that fools and fanatics are always
so certain of themselves, and wiser people so full of doubts."
- Bertrand Russell

Joel Becker
Principal Software Developer
Oracle
E-mail: ***@oracle.com
Phone: [phone]

[Luis Freitas]

Joel,

Sorry but I don't agree on this.

For CRS there is no need for a dedicated switch, only a need for using switches instead of cross cables. Although it is not recommended you can use the same switch for the public and private networks, using different vlans. The network status can be checked by the link status, that is what CRS does, and also by pinging the router. This information could be used as part of the heuristics to decide which node should survive. Of course it doesn't cover all network topologies, but it is sure better than node 0 always survive when network is down.

I see this as a problem in a RAC implementation, since there are two different cluster stacks running (O2CB and CRS), they are not integrated and take decisions with a different heuristic. For me it would make more sense if they were integrated and one of the cluster stacks was in control, in the same way that happens when you use RAC with Veritas/HP ServiceGuard/Sun Cluster Suite, or OCFS2 with heartbeat2, for example.

Best Regards,
Luis Freitas

Post by Thompson, Mark
Subject: Re: [Ocfs2-users] 2 node OCFS2 clusters
Date: Monday, November 16, 2009, 5:38 PM
On Mon, Nov 16, 2009 at 08:55:38AM

   If this is true then I would sugest

OCFS2 is not taking the best decision in this scenario.
    It's doing the best it can.  See,
each node knows the other is
alive because of the disk heartbeat.  Thus they know
the error is
network related.  But with only two nodes, there is no
way to determine
who has the better network.

   The node that still has network

connectivity should survive instead of the lowest node
number. Oracle CRS has heuristics to detect if the network
is down and in this scenario the node that lost network
conectivy is evicted. That is why it is required to use a
switch between the two nodes, instead of a cross cable.
    ocfs2 is a general purpose filesystem
that allows a myriad of
network configurations.  It has never required a
dedicated network
switch, nor will it, so relying on that isn't possible.
Joel
--
"The whole problem with the world is that fools and
fanatics are always
so certain of themselves, and wiser people so full of
doubts."
    - Bertrand Russell
Joel Becker
Principal Software Developer
Oracle
Phone: [phone]

[Joel Becker]

Post by Luis Freitas
For CRS there is no need for a dedicated switch, only a need for using switches instead of cross cables. Although it is not recommended you can use the same switch for the public and private networks, using different vlans. The network status can be checked by the link status, that is what CRS does, and also by pinging the router. This information could be used as part of the heuristics to decide which node should survive. Of course it doesn't cover all network topologies, but it is sure better than node 0 always survive when network is down.

How does node1 know the link status of node0? That's the
fundamental problem of self-fencing. You have to assume the other guy
is going to make a predictable decision. node1 has no way of knowing
that node0 is going to reboot.
What if the switch chip between node0 and node1 is down? Both
see their links as up?
Which link status do you check? Do you consider your link
status down if the interconnect link is down or all links are down?
What if you have a separate public and private network, node1 has lost
public network and node0 has lost private? In the current scheme, node1
resets and node0 continues talking on the public. The web service is
working. In your scheme node0 resets and node1 can't talk to the
public. The web service is down.
Self-fencing is hard and never perfect. The two node case is
the worst because there is no difference between a majority of nodes and
all nodes. The easiest way to alleviate it is to add a third node. Now
you have a majority and much easier decisions.

Post by Luis Freitas
I see this as a problem in a RAC implementation, since there are two different cluster stacks running (O2CB and CRS), they are not integrated and take decisions with a different heuristic. For me it would make more sense if they were integrated and one of the cluster stacks was in control, in the same way that happens when you use RAC with Veritas/HP ServiceGuard/Sun Cluster Suite, or OCFS2 with heartbeat2, for example.

The standard install documentation makes sure that o2cb and crs
behave well together. crs won't make a decision before o2cb does, thus
giving o2cb precedence.

Joel
--
Joel's First Law:

Nature abhors a GUI.

Joel Becker
Principal Software Developer
Oracle
E-mail: ***@oracle.com
Phone: [phone]