Galera cluster replication stalls when using wsrep_provider_options repl.commit_order=2
we are currently testing if moving from mariaDB master/slave replication to a 3-node-galera-cluster is an option for us.
General replication in the cluster works fine like expected. Load test shows that write-performance does not meet our requirements though.
To migrate data from the master/slave-system to the cluster one cluster node is set up as slave to that m/s-system (backup from master is imported, slave replication on the cluster node starts from the point where the backup is taken). The cluster node never catches up to the master (seconds-behind-master is increasing).
So we looked for tuning-options for our cluster. Setting repl.commit_order=2 improved performance, replication from the master to the cluster node went fine. But we then experienced that the other cluster nodes weren't replicating any more (though their state was synced and the wsrep%-variables did not show anything unusual, no throttling), replication within the cluster seemed to have stalled entirely.
When resetting repl.commit_order replication worked again.
Our cluster was not replicating neither when we had just the cluster operational (without replicating from the master node) with repl.commit_order=2.
Here excerpts from the server.cnf:
[galera]
# Mandatory settings
wsrep_provider = /usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address = "gcomm://10.0.0.10,10.0.0.8,10.0.0.11"
wsrep_cluster_name = foobar
wsrep_node_name = foobar-db1
wsrep_node_address = 10.0.0.10
binlog_format = row
default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2
wsrep_slave_threads = 16
wsrep_provider_options = 'gcs.fc_limit=80;gcs.fc_factor=0.8;repl.commit_order=2'
# SST settings
wsrep_sst_method = xtrabackup-v2
wsrep_sst_receive_address= 10.0.0.10
wsrep_sst_auth = galera:somepw
wsrep_retry_autocommit = 10
# required since 10.1
wsrep_on=ON
Any ideas?
mysql replication mariadb galera
bumped to the homepage by Community♦ 9 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
we are currently testing if moving from mariaDB master/slave replication to a 3-node-galera-cluster is an option for us.
General replication in the cluster works fine like expected. Load test shows that write-performance does not meet our requirements though.
To migrate data from the master/slave-system to the cluster one cluster node is set up as slave to that m/s-system (backup from master is imported, slave replication on the cluster node starts from the point where the backup is taken). The cluster node never catches up to the master (seconds-behind-master is increasing).
So we looked for tuning-options for our cluster. Setting repl.commit_order=2 improved performance, replication from the master to the cluster node went fine. But we then experienced that the other cluster nodes weren't replicating any more (though their state was synced and the wsrep%-variables did not show anything unusual, no throttling), replication within the cluster seemed to have stalled entirely.
When resetting repl.commit_order replication worked again.
Our cluster was not replicating neither when we had just the cluster operational (without replicating from the master node) with repl.commit_order=2.
Here excerpts from the server.cnf:
[galera]
# Mandatory settings
wsrep_provider = /usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address = "gcomm://10.0.0.10,10.0.0.8,10.0.0.11"
wsrep_cluster_name = foobar
wsrep_node_name = foobar-db1
wsrep_node_address = 10.0.0.10
binlog_format = row
default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2
wsrep_slave_threads = 16
wsrep_provider_options = 'gcs.fc_limit=80;gcs.fc_factor=0.8;repl.commit_order=2'
# SST settings
wsrep_sst_method = xtrabackup-v2
wsrep_sst_receive_address= 10.0.0.10
wsrep_sst_auth = galera:somepw
wsrep_retry_autocommit = 10
# required since 10.1
wsrep_on=ON
Any ideas?
mysql replication mariadb galera
bumped to the homepage by Community♦ 9 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
we are currently testing if moving from mariaDB master/slave replication to a 3-node-galera-cluster is an option for us.
General replication in the cluster works fine like expected. Load test shows that write-performance does not meet our requirements though.
To migrate data from the master/slave-system to the cluster one cluster node is set up as slave to that m/s-system (backup from master is imported, slave replication on the cluster node starts from the point where the backup is taken). The cluster node never catches up to the master (seconds-behind-master is increasing).
So we looked for tuning-options for our cluster. Setting repl.commit_order=2 improved performance, replication from the master to the cluster node went fine. But we then experienced that the other cluster nodes weren't replicating any more (though their state was synced and the wsrep%-variables did not show anything unusual, no throttling), replication within the cluster seemed to have stalled entirely.
When resetting repl.commit_order replication worked again.
Our cluster was not replicating neither when we had just the cluster operational (without replicating from the master node) with repl.commit_order=2.
Here excerpts from the server.cnf:
[galera]
# Mandatory settings
wsrep_provider = /usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address = "gcomm://10.0.0.10,10.0.0.8,10.0.0.11"
wsrep_cluster_name = foobar
wsrep_node_name = foobar-db1
wsrep_node_address = 10.0.0.10
binlog_format = row
default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2
wsrep_slave_threads = 16
wsrep_provider_options = 'gcs.fc_limit=80;gcs.fc_factor=0.8;repl.commit_order=2'
# SST settings
wsrep_sst_method = xtrabackup-v2
wsrep_sst_receive_address= 10.0.0.10
wsrep_sst_auth = galera:somepw
wsrep_retry_autocommit = 10
# required since 10.1
wsrep_on=ON
Any ideas?
mysql replication mariadb galera
we are currently testing if moving from mariaDB master/slave replication to a 3-node-galera-cluster is an option for us.
General replication in the cluster works fine like expected. Load test shows that write-performance does not meet our requirements though.
To migrate data from the master/slave-system to the cluster one cluster node is set up as slave to that m/s-system (backup from master is imported, slave replication on the cluster node starts from the point where the backup is taken). The cluster node never catches up to the master (seconds-behind-master is increasing).
So we looked for tuning-options for our cluster. Setting repl.commit_order=2 improved performance, replication from the master to the cluster node went fine. But we then experienced that the other cluster nodes weren't replicating any more (though their state was synced and the wsrep%-variables did not show anything unusual, no throttling), replication within the cluster seemed to have stalled entirely.
When resetting repl.commit_order replication worked again.
Our cluster was not replicating neither when we had just the cluster operational (without replicating from the master node) with repl.commit_order=2.
Here excerpts from the server.cnf:
[galera]
# Mandatory settings
wsrep_provider = /usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address = "gcomm://10.0.0.10,10.0.0.8,10.0.0.11"
wsrep_cluster_name = foobar
wsrep_node_name = foobar-db1
wsrep_node_address = 10.0.0.10
binlog_format = row
default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2
wsrep_slave_threads = 16
wsrep_provider_options = 'gcs.fc_limit=80;gcs.fc_factor=0.8;repl.commit_order=2'
# SST settings
wsrep_sst_method = xtrabackup-v2
wsrep_sst_receive_address= 10.0.0.10
wsrep_sst_auth = galera:somepw
wsrep_retry_autocommit = 10
# required since 10.1
wsrep_on=ON
Any ideas?
mysql replication mariadb galera
mysql replication mariadb galera
asked Apr 13 '17 at 9:04
Heiko GroßmannHeiko Großmann
1
1
bumped to the homepage by Community♦ 9 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 9 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
(Is your question mostly about the "migration"? Or about write performance on the resulting Cluster?)
Look at your transactions. Optimal is some sort of "medium" size. Too short leads to long latency. Too long leads to extra risk of deadlocks/failure at COMMIT
time.
Look at the network. You are dependent on how far apart the nodes are. If there are next to each other, you do not get the HA benefits. If they are too far apart, you have latency issues.
Look at the write queries. If they are likely to conflict, but you are sending them to different nodes, then you may be adding to the problem, relative to somehow focusing some sets of queries onto a single node.
Do you have "critical reads"? How are you handling them?
More Galera tips .
Thanks for your input, Rick! Migration is working fine (in theory, due to the write performance issues the cluster is either neither catching up to the replication master or the nodes are not replicating at all (one node receiving the replication data is doing just fine, the others are "stalled") - depending on the "repl.commit_order"-configuration.
– Heiko Großmann
Apr 19 '17 at 16:01
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "182"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f170905%2fgalera-cluster-replication-stalls-when-using-wsrep-provider-options-repl-commit%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
(Is your question mostly about the "migration"? Or about write performance on the resulting Cluster?)
Look at your transactions. Optimal is some sort of "medium" size. Too short leads to long latency. Too long leads to extra risk of deadlocks/failure at COMMIT
time.
Look at the network. You are dependent on how far apart the nodes are. If there are next to each other, you do not get the HA benefits. If they are too far apart, you have latency issues.
Look at the write queries. If they are likely to conflict, but you are sending them to different nodes, then you may be adding to the problem, relative to somehow focusing some sets of queries onto a single node.
Do you have "critical reads"? How are you handling them?
More Galera tips .
Thanks for your input, Rick! Migration is working fine (in theory, due to the write performance issues the cluster is either neither catching up to the replication master or the nodes are not replicating at all (one node receiving the replication data is doing just fine, the others are "stalled") - depending on the "repl.commit_order"-configuration.
– Heiko Großmann
Apr 19 '17 at 16:01
add a comment |
(Is your question mostly about the "migration"? Or about write performance on the resulting Cluster?)
Look at your transactions. Optimal is some sort of "medium" size. Too short leads to long latency. Too long leads to extra risk of deadlocks/failure at COMMIT
time.
Look at the network. You are dependent on how far apart the nodes are. If there are next to each other, you do not get the HA benefits. If they are too far apart, you have latency issues.
Look at the write queries. If they are likely to conflict, but you are sending them to different nodes, then you may be adding to the problem, relative to somehow focusing some sets of queries onto a single node.
Do you have "critical reads"? How are you handling them?
More Galera tips .
Thanks for your input, Rick! Migration is working fine (in theory, due to the write performance issues the cluster is either neither catching up to the replication master or the nodes are not replicating at all (one node receiving the replication data is doing just fine, the others are "stalled") - depending on the "repl.commit_order"-configuration.
– Heiko Großmann
Apr 19 '17 at 16:01
add a comment |
(Is your question mostly about the "migration"? Or about write performance on the resulting Cluster?)
Look at your transactions. Optimal is some sort of "medium" size. Too short leads to long latency. Too long leads to extra risk of deadlocks/failure at COMMIT
time.
Look at the network. You are dependent on how far apart the nodes are. If there are next to each other, you do not get the HA benefits. If they are too far apart, you have latency issues.
Look at the write queries. If they are likely to conflict, but you are sending them to different nodes, then you may be adding to the problem, relative to somehow focusing some sets of queries onto a single node.
Do you have "critical reads"? How are you handling them?
More Galera tips .
(Is your question mostly about the "migration"? Or about write performance on the resulting Cluster?)
Look at your transactions. Optimal is some sort of "medium" size. Too short leads to long latency. Too long leads to extra risk of deadlocks/failure at COMMIT
time.
Look at the network. You are dependent on how far apart the nodes are. If there are next to each other, you do not get the HA benefits. If they are too far apart, you have latency issues.
Look at the write queries. If they are likely to conflict, but you are sending them to different nodes, then you may be adding to the problem, relative to somehow focusing some sets of queries onto a single node.
Do you have "critical reads"? How are you handling them?
More Galera tips .
answered Apr 13 '17 at 15:35
Rick JamesRick James
41.2k22258
41.2k22258
Thanks for your input, Rick! Migration is working fine (in theory, due to the write performance issues the cluster is either neither catching up to the replication master or the nodes are not replicating at all (one node receiving the replication data is doing just fine, the others are "stalled") - depending on the "repl.commit_order"-configuration.
– Heiko Großmann
Apr 19 '17 at 16:01
add a comment |
Thanks for your input, Rick! Migration is working fine (in theory, due to the write performance issues the cluster is either neither catching up to the replication master or the nodes are not replicating at all (one node receiving the replication data is doing just fine, the others are "stalled") - depending on the "repl.commit_order"-configuration.
– Heiko Großmann
Apr 19 '17 at 16:01
Thanks for your input, Rick! Migration is working fine (in theory, due to the write performance issues the cluster is either neither catching up to the replication master or the nodes are not replicating at all (one node receiving the replication data is doing just fine, the others are "stalled") - depending on the "repl.commit_order"-configuration.
– Heiko Großmann
Apr 19 '17 at 16:01
Thanks for your input, Rick! Migration is working fine (in theory, due to the write performance issues the cluster is either neither catching up to the replication master or the nodes are not replicating at all (one node receiving the replication data is doing just fine, the others are "stalled") - depending on the "repl.commit_order"-configuration.
– Heiko Großmann
Apr 19 '17 at 16:01
add a comment |
Thanks for contributing an answer to Database Administrators Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f170905%2fgalera-cluster-replication-stalls-when-using-wsrep-provider-options-repl-commit%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown