SQL Server i/o requests taking longer than 15 secs. and slow checkpoint
Last couple of weeks we've been working on getting to the root cause of what could likely be the cause of the occurrence of these i/o issues and slowdown of the checkpoints. At first glance it looks to be clearly an i/o subsystem error and the SAN admin was to be blamed for but recently we changed the SAN to utilize Full Flash but as of today the error still pops up and I have no clue as to why since every metric whether wait stats or any other metric that I run to check if SQL server is a possible culprit seems out to return normal. It doesn't really add up. It could also be very likely that something else is chewing the disk and SQL Server is becoming victimized here...but I am not able to find out what? Dbs are in AlwaysOn AG and as and when these events occur we do see role changes and flip overs occurring along with timeouts. Any help in figuring this out would be highly appreciated. Let me know if any further details is needed.
Error msgs. below
SQL Server has encountered 14212 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [E:MSSQLDATAABC.mdf] in database [ABC] (7). The OS file handle is 0x0000000000000D64. The offset of the latest long I/O is: 0x0000641262c000
SQL Server has encountered 5347 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [E:MSSQLDATAXYZ.mdf] in database [XYZ] (7). The OS file handle is 0x0000000000000D64. The offset of the latest long I/O is: 0x0000506c060000
FlushCache: cleaned up 111476 bufs with 62224 writes in 925084 ms (avoided 19 new dirty bufs) for db 7:0
average throughput: 0.94 MB/sec, I/O saturation: 55144, context switches 98407
last target outstanding: 10240, avgWriteLatency 14171
FlushCache: cleaned up 5616 bufs with 3126 writes in 248687 ms (avoided 3626 new dirty bufs) for db 6:0
average throughput: 0.18 MB/sec, I/O saturation: 10080, context switches 20913
last target outstanding: 2, avgWriteLatency 3
sql-server-2012 performance-tuning
asked 3 mins ago
FeivelFeivel
1509
add a comment |
Last couple of weeks we've been working on getting to the root cause of what could likely be the cause of the occurrence of these i/o issues and slowdown of the checkpoints. At first glance it looks to be clearly an i/o subsystem error and the SAN admin was to be blamed for but recently we changed the SAN to utilize Full Flash but as of today the error still pops up and I have no clue as to why since every metric whether wait stats or any other metric that I run to check if SQL server is a possible culprit seems out to return normal. It doesn't really add up. It could also be very likely that something else is chewing the disk and SQL Server is becoming victimized here...but I am not able to find out what? Dbs are in AlwaysOn AG and as and when these events occur we do see role changes and flip overs occurring along with timeouts. Any help in figuring this out would be highly appreciated. Let me know if any further details is needed.
Error msgs. below
SQL Server has encountered 14212 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [E:MSSQLDATAABC.mdf] in database [ABC] (7). The OS file handle is 0x0000000000000D64. The offset of the latest long I/O is: 0x0000641262c000
SQL Server has encountered 5347 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [E:MSSQLDATAXYZ.mdf] in database [XYZ] (7). The OS file handle is 0x0000000000000D64. The offset of the latest long I/O is: 0x0000506c060000
FlushCache: cleaned up 111476 bufs with 62224 writes in 925084 ms (avoided 19 new dirty bufs) for db 7:0
average throughput: 0.94 MB/sec, I/O saturation: 55144, context switches 98407
last target outstanding: 10240, avgWriteLatency 14171
FlushCache: cleaned up 5616 bufs with 3126 writes in 248687 ms (avoided 3626 new dirty bufs) for db 6:0
average throughput: 0.18 MB/sec, I/O saturation: 10080, context switches 20913
last target outstanding: 2, avgWriteLatency 3
sql-server-2012 performance-tuning
asked 3 mins ago
FeivelFeivel
1509
add a comment |
Last couple of weeks we've been working on getting to the root cause of what could likely be the cause of the occurrence of these i/o issues and slowdown of the checkpoints. At first glance it looks to be clearly an i/o subsystem error and the SAN admin was to be blamed for but recently we changed the SAN to utilize Full Flash but as of today the error still pops up and I have no clue as to why since every metric whether wait stats or any other metric that I run to check if SQL server is a possible culprit seems out to return normal. It doesn't really add up. It could also be very likely that something else is chewing the disk and SQL Server is becoming victimized here...but I am not able to find out what? Dbs are in AlwaysOn AG and as and when these events occur we do see role changes and flip overs occurring along with timeouts. Any help in figuring this out would be highly appreciated. Let me know if any further details is needed.
Error msgs. below
SQL Server has encountered 14212 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [E:MSSQLDATAABC.mdf] in database [ABC] (7). The OS file handle is 0x0000000000000D64. The offset of the latest long I/O is: 0x0000641262c000
SQL Server has encountered 5347 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [E:MSSQLDATAXYZ.mdf] in database [XYZ] (7). The OS file handle is 0x0000000000000D64. The offset of the latest long I/O is: 0x0000506c060000
FlushCache: cleaned up 111476 bufs with 62224 writes in 925084 ms (avoided 19 new dirty bufs) for db 7:0
average throughput: 0.94 MB/sec, I/O saturation: 55144, context switches 98407
last target outstanding: 10240, avgWriteLatency 14171
FlushCache: cleaned up 5616 bufs with 3126 writes in 248687 ms (avoided 3626 new dirty bufs) for db 6:0
average throughput: 0.18 MB/sec, I/O saturation: 10080, context switches 20913
last target outstanding: 2, avgWriteLatency 3
sql-server-2012 performance-tuning
asked 3 mins ago
FeivelFeivel
1509
Last couple of weeks we've been working on getting to the root cause of what could likely be the cause of the occurrence of these i/o issues and slowdown of the checkpoints. At first glance it looks to be clearly an i/o subsystem error and the SAN admin was to be blamed for but recently we changed the SAN to utilize Full Flash but as of today the error still pops up and I have no clue as to why since every metric whether wait stats or any other metric that I run to check if SQL server is a possible culprit seems out to return normal. It doesn't really add up. It could also be very likely that something else is chewing the disk and SQL Server is becoming victimized here...but I am not able to find out what? Dbs are in AlwaysOn AG and as and when these events occur we do see role changes and flip overs occurring along with timeouts. Any help in figuring this out would be highly appreciated. Let me know if any further details is needed.
Error msgs. below
SQL Server has encountered 14212 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [E:MSSQLDATAABC.mdf] in database [ABC] (7). The OS file handle is 0x0000000000000D64. The offset of the latest long I/O is: 0x0000641262c000
SQL Server has encountered 5347 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [E:MSSQLDATAXYZ.mdf] in database [XYZ] (7). The OS file handle is 0x0000000000000D64. The offset of the latest long I/O is: 0x0000506c060000
FlushCache: cleaned up 111476 bufs with 62224 writes in 925084 ms (avoided 19 new dirty bufs) for db 7:0
average throughput: 0.94 MB/sec, I/O saturation: 55144, context switches 98407
last target outstanding: 10240, avgWriteLatency 14171
FlushCache: cleaned up 5616 bufs with 3126 writes in 248687 ms (avoided 3626 new dirty bufs) for db 6:0
average throughput: 0.18 MB/sec, I/O saturation: 10080, context switches 20913
last target outstanding: 2, avgWriteLatency 3
sql-server-2012 performance-tuning
sql-server-2012 performance-tuning
asked 3 mins ago
FeivelFeivel
1509
asked 3 mins ago
FeivelFeivel
1509
asked 3 mins ago
FeivelFeivel
1509
asked 3 mins ago
FeivelFeivel
1509
asked 3 mins ago
FeivelFeivel
1509
1509
add a comment |
add a comment |
0
active
oldest
votes
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "182"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f233705%2fsql-server-i-o-requests-taking-longer-than-15-secs-and-slow-checkpoint%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Database Administrators Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f233705%2fsql-server-i-o-requests-taking-longer-than-15-secs-and-slow-checkpoint%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
