ibdata1 grows big again after using innodb_file_per_table and having shrunk it
After having set innodb_file_per_table=1, dumping and dropping all db's, deleting ibdata1, and restarting mysql ... My ibdata1 file is small at last...
However, directly after reading in my dump, and seeing that all tables now all have their own big ibd files (approx 15GB), but still the ibdata1 is over 2.6 GB again...
The way I dump my data is with:
mysqldump --tab="D:Mysql Databasesdumpsdump%%a" --opt --single-transaction --no-autocommit -u root -p
This dumps to .sql (DDL / table structures) and .txt (actual data) files. These files I can use to recover with with:
create database myDb;
use myDb;
SET autocommit=0;
SET unique_checks=0;
SET foreign_key_checks=0;
SOURCE *.sql; -- for each table
COMMIT;
SET autocommit=0;
LOAD DATA INFILE '*.txt'; -- for each table
COMMIT;
SET autocommit=1;
SET foreign_key_checks=1;
SET unique_checks=1;
Why is the ibdata1 file growing again after enabling innodb_file_per_table and having succesfully shrunk ibdata1? And is there a way now to reclaim again this data, seen as all ibd data is already present per table?
Can I for example copy the small ibdata1 after recreating the database with the *.sql files, and placing back that ibdata1 file after reading in the data from the *.txt files?
Here are the non-default items in my.ini:
sql-mode="NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
max_connections=151
tmp_table_size=256M
thread_cache_size=20
myisam_max_sort_file_size=1G
myisam_sort_buffer_size=32M
key_buffer_size=64M
innodb_log_buffer_size=8M
innodb_buffer_pool_size=6G
innodb_log_file_size=1G
innodb_thread_concurrency=16
innodb_open_files=1000
join_buffer_size=2M
max_allowed_packet=16M
group_concat_max_len = 10M
bulk_insert_buffer_size=512M
collation-server = utf8_unicode_ci
mysql
bumped to the homepage by Community♦ 4 hours ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
After having set innodb_file_per_table=1, dumping and dropping all db's, deleting ibdata1, and restarting mysql ... My ibdata1 file is small at last...
However, directly after reading in my dump, and seeing that all tables now all have their own big ibd files (approx 15GB), but still the ibdata1 is over 2.6 GB again...
The way I dump my data is with:
mysqldump --tab="D:Mysql Databasesdumpsdump%%a" --opt --single-transaction --no-autocommit -u root -p
This dumps to .sql (DDL / table structures) and .txt (actual data) files. These files I can use to recover with with:
create database myDb;
use myDb;
SET autocommit=0;
SET unique_checks=0;
SET foreign_key_checks=0;
SOURCE *.sql; -- for each table
COMMIT;
SET autocommit=0;
LOAD DATA INFILE '*.txt'; -- for each table
COMMIT;
SET autocommit=1;
SET foreign_key_checks=1;
SET unique_checks=1;
Why is the ibdata1 file growing again after enabling innodb_file_per_table and having succesfully shrunk ibdata1? And is there a way now to reclaim again this data, seen as all ibd data is already present per table?
Can I for example copy the small ibdata1 after recreating the database with the *.sql files, and placing back that ibdata1 file after reading in the data from the *.txt files?
Here are the non-default items in my.ini:
sql-mode="NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
max_connections=151
tmp_table_size=256M
thread_cache_size=20
myisam_max_sort_file_size=1G
myisam_sort_buffer_size=32M
key_buffer_size=64M
innodb_log_buffer_size=8M
innodb_buffer_pool_size=6G
innodb_log_file_size=1G
innodb_thread_concurrency=16
innodb_open_files=1000
join_buffer_size=2M
max_allowed_packet=16M
group_concat_max_len = 10M
bulk_insert_buffer_size=512M
collation-server = utf8_unicode_ci
mysql
bumped to the homepage by Community♦ 4 hours ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
How much RAM?..
– Rick James
Apr 8 '16 at 2:52
add a comment |
After having set innodb_file_per_table=1, dumping and dropping all db's, deleting ibdata1, and restarting mysql ... My ibdata1 file is small at last...
However, directly after reading in my dump, and seeing that all tables now all have their own big ibd files (approx 15GB), but still the ibdata1 is over 2.6 GB again...
The way I dump my data is with:
mysqldump --tab="D:Mysql Databasesdumpsdump%%a" --opt --single-transaction --no-autocommit -u root -p
This dumps to .sql (DDL / table structures) and .txt (actual data) files. These files I can use to recover with with:
create database myDb;
use myDb;
SET autocommit=0;
SET unique_checks=0;
SET foreign_key_checks=0;
SOURCE *.sql; -- for each table
COMMIT;
SET autocommit=0;
LOAD DATA INFILE '*.txt'; -- for each table
COMMIT;
SET autocommit=1;
SET foreign_key_checks=1;
SET unique_checks=1;
Why is the ibdata1 file growing again after enabling innodb_file_per_table and having succesfully shrunk ibdata1? And is there a way now to reclaim again this data, seen as all ibd data is already present per table?
Can I for example copy the small ibdata1 after recreating the database with the *.sql files, and placing back that ibdata1 file after reading in the data from the *.txt files?
Here are the non-default items in my.ini:
sql-mode="NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
max_connections=151
tmp_table_size=256M
thread_cache_size=20
myisam_max_sort_file_size=1G
myisam_sort_buffer_size=32M
key_buffer_size=64M
innodb_log_buffer_size=8M
innodb_buffer_pool_size=6G
innodb_log_file_size=1G
innodb_thread_concurrency=16
innodb_open_files=1000
join_buffer_size=2M
max_allowed_packet=16M
group_concat_max_len = 10M
bulk_insert_buffer_size=512M
collation-server = utf8_unicode_ci
mysql
After having set innodb_file_per_table=1, dumping and dropping all db's, deleting ibdata1, and restarting mysql ... My ibdata1 file is small at last...
However, directly after reading in my dump, and seeing that all tables now all have their own big ibd files (approx 15GB), but still the ibdata1 is over 2.6 GB again...
The way I dump my data is with:
mysqldump --tab="D:Mysql Databasesdumpsdump%%a" --opt --single-transaction --no-autocommit -u root -p
This dumps to .sql (DDL / table structures) and .txt (actual data) files. These files I can use to recover with with:
create database myDb;
use myDb;
SET autocommit=0;
SET unique_checks=0;
SET foreign_key_checks=0;
SOURCE *.sql; -- for each table
COMMIT;
SET autocommit=0;
LOAD DATA INFILE '*.txt'; -- for each table
COMMIT;
SET autocommit=1;
SET foreign_key_checks=1;
SET unique_checks=1;
Why is the ibdata1 file growing again after enabling innodb_file_per_table and having succesfully shrunk ibdata1? And is there a way now to reclaim again this data, seen as all ibd data is already present per table?
Can I for example copy the small ibdata1 after recreating the database with the *.sql files, and placing back that ibdata1 file after reading in the data from the *.txt files?
Here are the non-default items in my.ini:
sql-mode="NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
max_connections=151
tmp_table_size=256M
thread_cache_size=20
myisam_max_sort_file_size=1G
myisam_sort_buffer_size=32M
key_buffer_size=64M
innodb_log_buffer_size=8M
innodb_buffer_pool_size=6G
innodb_log_file_size=1G
innodb_thread_concurrency=16
innodb_open_files=1000
join_buffer_size=2M
max_allowed_packet=16M
group_concat_max_len = 10M
bulk_insert_buffer_size=512M
collation-server = utf8_unicode_ci
mysql
mysql
asked Apr 7 '16 at 14:22
nl-xnl-x
1011
1011
bumped to the homepage by Community♦ 4 hours ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 4 hours ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
How much RAM?..
– Rick James
Apr 8 '16 at 2:52
add a comment |
How much RAM?..
– Rick James
Apr 8 '16 at 2:52
How much RAM?..
– Rick James
Apr 8 '16 at 2:52
How much RAM?..
– Rick James
Apr 8 '16 at 2:52
add a comment |
1 Answer
1
active
oldest
votes
Ibdata1 file holds a lot of information and allocates space for different innodb functionality (double write buffer, undo logs, tablespace header, etc.). Therefore even with innodb_file_per_table you cannot drop/replace the ibdata1 file.
One common reason why ibdata can grow is because it has the undo space. If you have long running transactions the previous version of rows can pile up in the undo log easily. More about this can be read on person's blog:
https://www.percona.com/blog/2013/08/20/why-is-the-ibdata1-file-continuously-growing-in-mysql/
In MySQL 5.7 there are a lot of improvements around undo tablespace management including truncation and separate files to hold this.
So could it have to do with what I suspected... turning autocommit off during the reading of the dump? ... Is that what is causing the transactions to pile up ? ...
– nl-x
Apr 7 '16 at 14:50
And where can I read about how to truncate the tablespace (management) ?
– nl-x
Apr 7 '16 at 14:51
Innodb tablespace truncate (and other parameters here as well): dev.mysql.com/doc/refman/5.7/en/…
– Károly Nagy
Apr 7 '16 at 14:54
A post about the feature: mysqlserverteam.com/online-truncate-of-innodb-undo-tablespaces
– Károly Nagy
Apr 7 '16 at 14:54
mysqldump manages it's own transactions with--single-transactionas you specified in the command line. You can try without and see how big youribdata1file becomes.
– Károly Nagy
Apr 7 '16 at 14:57
|
show 3 more comments
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "182"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f134672%2fibdata1-grows-big-again-after-using-innodb-file-per-table-and-having-shrunk-it%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Ibdata1 file holds a lot of information and allocates space for different innodb functionality (double write buffer, undo logs, tablespace header, etc.). Therefore even with innodb_file_per_table you cannot drop/replace the ibdata1 file.
One common reason why ibdata can grow is because it has the undo space. If you have long running transactions the previous version of rows can pile up in the undo log easily. More about this can be read on person's blog:
https://www.percona.com/blog/2013/08/20/why-is-the-ibdata1-file-continuously-growing-in-mysql/
In MySQL 5.7 there are a lot of improvements around undo tablespace management including truncation and separate files to hold this.
So could it have to do with what I suspected... turning autocommit off during the reading of the dump? ... Is that what is causing the transactions to pile up ? ...
– nl-x
Apr 7 '16 at 14:50
And where can I read about how to truncate the tablespace (management) ?
– nl-x
Apr 7 '16 at 14:51
Innodb tablespace truncate (and other parameters here as well): dev.mysql.com/doc/refman/5.7/en/…
– Károly Nagy
Apr 7 '16 at 14:54
A post about the feature: mysqlserverteam.com/online-truncate-of-innodb-undo-tablespaces
– Károly Nagy
Apr 7 '16 at 14:54
mysqldump manages it's own transactions with--single-transactionas you specified in the command line. You can try without and see how big youribdata1file becomes.
– Károly Nagy
Apr 7 '16 at 14:57
|
show 3 more comments
Ibdata1 file holds a lot of information and allocates space for different innodb functionality (double write buffer, undo logs, tablespace header, etc.). Therefore even with innodb_file_per_table you cannot drop/replace the ibdata1 file.
One common reason why ibdata can grow is because it has the undo space. If you have long running transactions the previous version of rows can pile up in the undo log easily. More about this can be read on person's blog:
https://www.percona.com/blog/2013/08/20/why-is-the-ibdata1-file-continuously-growing-in-mysql/
In MySQL 5.7 there are a lot of improvements around undo tablespace management including truncation and separate files to hold this.
So could it have to do with what I suspected... turning autocommit off during the reading of the dump? ... Is that what is causing the transactions to pile up ? ...
– nl-x
Apr 7 '16 at 14:50
And where can I read about how to truncate the tablespace (management) ?
– nl-x
Apr 7 '16 at 14:51
Innodb tablespace truncate (and other parameters here as well): dev.mysql.com/doc/refman/5.7/en/…
– Károly Nagy
Apr 7 '16 at 14:54
A post about the feature: mysqlserverteam.com/online-truncate-of-innodb-undo-tablespaces
– Károly Nagy
Apr 7 '16 at 14:54
mysqldump manages it's own transactions with--single-transactionas you specified in the command line. You can try without and see how big youribdata1file becomes.
– Károly Nagy
Apr 7 '16 at 14:57
|
show 3 more comments
Ibdata1 file holds a lot of information and allocates space for different innodb functionality (double write buffer, undo logs, tablespace header, etc.). Therefore even with innodb_file_per_table you cannot drop/replace the ibdata1 file.
One common reason why ibdata can grow is because it has the undo space. If you have long running transactions the previous version of rows can pile up in the undo log easily. More about this can be read on person's blog:
https://www.percona.com/blog/2013/08/20/why-is-the-ibdata1-file-continuously-growing-in-mysql/
In MySQL 5.7 there are a lot of improvements around undo tablespace management including truncation and separate files to hold this.
Ibdata1 file holds a lot of information and allocates space for different innodb functionality (double write buffer, undo logs, tablespace header, etc.). Therefore even with innodb_file_per_table you cannot drop/replace the ibdata1 file.
One common reason why ibdata can grow is because it has the undo space. If you have long running transactions the previous version of rows can pile up in the undo log easily. More about this can be read on person's blog:
https://www.percona.com/blog/2013/08/20/why-is-the-ibdata1-file-continuously-growing-in-mysql/
In MySQL 5.7 there are a lot of improvements around undo tablespace management including truncation and separate files to hold this.
edited Apr 7 '16 at 14:46
answered Apr 7 '16 at 14:37
Károly NagyKároly Nagy
2,5101611
2,5101611
So could it have to do with what I suspected... turning autocommit off during the reading of the dump? ... Is that what is causing the transactions to pile up ? ...
– nl-x
Apr 7 '16 at 14:50
And where can I read about how to truncate the tablespace (management) ?
– nl-x
Apr 7 '16 at 14:51
Innodb tablespace truncate (and other parameters here as well): dev.mysql.com/doc/refman/5.7/en/…
– Károly Nagy
Apr 7 '16 at 14:54
A post about the feature: mysqlserverteam.com/online-truncate-of-innodb-undo-tablespaces
– Károly Nagy
Apr 7 '16 at 14:54
mysqldump manages it's own transactions with--single-transactionas you specified in the command line. You can try without and see how big youribdata1file becomes.
– Károly Nagy
Apr 7 '16 at 14:57
|
show 3 more comments
So could it have to do with what I suspected... turning autocommit off during the reading of the dump? ... Is that what is causing the transactions to pile up ? ...
– nl-x
Apr 7 '16 at 14:50
And where can I read about how to truncate the tablespace (management) ?
– nl-x
Apr 7 '16 at 14:51
Innodb tablespace truncate (and other parameters here as well): dev.mysql.com/doc/refman/5.7/en/…
– Károly Nagy
Apr 7 '16 at 14:54
A post about the feature: mysqlserverteam.com/online-truncate-of-innodb-undo-tablespaces
– Károly Nagy
Apr 7 '16 at 14:54
mysqldump manages it's own transactions with--single-transactionas you specified in the command line. You can try without and see how big youribdata1file becomes.
– Károly Nagy
Apr 7 '16 at 14:57
So could it have to do with what I suspected... turning autocommit off during the reading of the dump? ... Is that what is causing the transactions to pile up ? ...
– nl-x
Apr 7 '16 at 14:50
So could it have to do with what I suspected... turning autocommit off during the reading of the dump? ... Is that what is causing the transactions to pile up ? ...
– nl-x
Apr 7 '16 at 14:50
And where can I read about how to truncate the tablespace (management) ?
– nl-x
Apr 7 '16 at 14:51
And where can I read about how to truncate the tablespace (management) ?
– nl-x
Apr 7 '16 at 14:51
Innodb tablespace truncate (and other parameters here as well): dev.mysql.com/doc/refman/5.7/en/…
– Károly Nagy
Apr 7 '16 at 14:54
Innodb tablespace truncate (and other parameters here as well): dev.mysql.com/doc/refman/5.7/en/…
– Károly Nagy
Apr 7 '16 at 14:54
A post about the feature: mysqlserverteam.com/online-truncate-of-innodb-undo-tablespaces
– Károly Nagy
Apr 7 '16 at 14:54
A post about the feature: mysqlserverteam.com/online-truncate-of-innodb-undo-tablespaces
– Károly Nagy
Apr 7 '16 at 14:54
mysqldump manages it's own transactions with
--single-transaction as you specified in the command line. You can try without and see how big your ibdata1 file becomes.– Károly Nagy
Apr 7 '16 at 14:57
mysqldump manages it's own transactions with
--single-transaction as you specified in the command line. You can try without and see how big your ibdata1 file becomes.– Károly Nagy
Apr 7 '16 at 14:57
|
show 3 more comments
Thanks for contributing an answer to Database Administrators Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f134672%2fibdata1-grows-big-again-after-using-innodb-file-per-table-and-having-shrunk-it%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
How much RAM?..
– Rick James
Apr 8 '16 at 2:52