investigate contents of a sqlite3 file
I have created a new, clean firefox profile from scratch, and added couple of bookmarks. These, I have learned, are stored in places.sqlite file inside the profile directory. This file seems to be too big (1.2 MB) and I am wondering why:
$ ls -lh places.sqlite
-rw------- 1 martin martin 1.2M Sep 8 07:24 places.sqlite
When I open it with sqlite3 from the commandline, I see there are couple of tables:
$ sqlite3 places.sqlite
SQLite version 3.7.13 2012-06-11 02:05:22
sqlite> .tables
moz_anno_attributes moz_favicons moz_items_annos
moz_annos moz_historyvisits moz_keywords
moz_bookmarks moz_hosts moz_places
moz_bookmarks_roots moz_inputhistory
but none of them contain much data (I have tried SELECT * FROM <TABLE> ; for all of them).
Further, if I dump the entire database into file dump.sqlite:
sqlite> .output dump.sqlite
sqlite> .dump
The file has only couple of KB:
$ ls -lh
total 1.2M
-rw------- 1 martin martin 39K Sep 8 19:01 dump.sqlite
-rw------- 1 martin martin 1.2M Sep 8 07:24 places.sqlite
I am aware of the vacuum command in sqlite3. I did that, but it has no effect. The database file is still same size.
How can I investigate why the database file is so big, and what does it contain ?
sqlite dump
bumped to the homepage by Community♦ 12 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
I have created a new, clean firefox profile from scratch, and added couple of bookmarks. These, I have learned, are stored in places.sqlite file inside the profile directory. This file seems to be too big (1.2 MB) and I am wondering why:
$ ls -lh places.sqlite
-rw------- 1 martin martin 1.2M Sep 8 07:24 places.sqlite
When I open it with sqlite3 from the commandline, I see there are couple of tables:
$ sqlite3 places.sqlite
SQLite version 3.7.13 2012-06-11 02:05:22
sqlite> .tables
moz_anno_attributes moz_favicons moz_items_annos
moz_annos moz_historyvisits moz_keywords
moz_bookmarks moz_hosts moz_places
moz_bookmarks_roots moz_inputhistory
but none of them contain much data (I have tried SELECT * FROM <TABLE> ; for all of them).
Further, if I dump the entire database into file dump.sqlite:
sqlite> .output dump.sqlite
sqlite> .dump
The file has only couple of KB:
$ ls -lh
total 1.2M
-rw------- 1 martin martin 39K Sep 8 19:01 dump.sqlite
-rw------- 1 martin martin 1.2M Sep 8 07:24 places.sqlite
I am aware of the vacuum command in sqlite3. I did that, but it has no effect. The database file is still same size.
How can I investigate why the database file is so big, and what does it contain ?
sqlite dump
bumped to the homepage by Community♦ 12 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
Are you attempting to vacuum the file while Firefox is running? Mine is 10MB while Firefox is running, and would not change until I quit Firefox and ran the vaccum against the database while not in use. Immediately after starting Firefox, it again jumped to 10MB (from 3MB post-vacuum). I suspect that Firefox may be pre-allocating space for performance - see this recommendation they make for performance
– alroc
Sep 8 '16 at 17:20
@alroc - no, I am working with the file-copy offline. Firefox is not running.
– Martin Vegter
Sep 8 '16 at 17:38
The file is not just the data inside it. There's some overhead in there. Check the file format for SQLite databases: sqlite.org/fileformat.html
– Randolph West
Sep 10 '16 at 2:30
@Randolph West - thanks, but how can I investigate this "overhead" data ? How I see what is in the file?
– Martin Vegter
Sep 10 '16 at 19:12
I would use a hex editor, and the file format link I posted as a guide.
– Randolph West
Sep 11 '16 at 2:48
add a comment |
I have created a new, clean firefox profile from scratch, and added couple of bookmarks. These, I have learned, are stored in places.sqlite file inside the profile directory. This file seems to be too big (1.2 MB) and I am wondering why:
$ ls -lh places.sqlite
-rw------- 1 martin martin 1.2M Sep 8 07:24 places.sqlite
When I open it with sqlite3 from the commandline, I see there are couple of tables:
$ sqlite3 places.sqlite
SQLite version 3.7.13 2012-06-11 02:05:22
sqlite> .tables
moz_anno_attributes moz_favicons moz_items_annos
moz_annos moz_historyvisits moz_keywords
moz_bookmarks moz_hosts moz_places
moz_bookmarks_roots moz_inputhistory
but none of them contain much data (I have tried SELECT * FROM <TABLE> ; for all of them).
Further, if I dump the entire database into file dump.sqlite:
sqlite> .output dump.sqlite
sqlite> .dump
The file has only couple of KB:
$ ls -lh
total 1.2M
-rw------- 1 martin martin 39K Sep 8 19:01 dump.sqlite
-rw------- 1 martin martin 1.2M Sep 8 07:24 places.sqlite
I am aware of the vacuum command in sqlite3. I did that, but it has no effect. The database file is still same size.
How can I investigate why the database file is so big, and what does it contain ?
sqlite dump
I have created a new, clean firefox profile from scratch, and added couple of bookmarks. These, I have learned, are stored in places.sqlite file inside the profile directory. This file seems to be too big (1.2 MB) and I am wondering why:
$ ls -lh places.sqlite
-rw------- 1 martin martin 1.2M Sep 8 07:24 places.sqlite
When I open it with sqlite3 from the commandline, I see there are couple of tables:
$ sqlite3 places.sqlite
SQLite version 3.7.13 2012-06-11 02:05:22
sqlite> .tables
moz_anno_attributes moz_favicons moz_items_annos
moz_annos moz_historyvisits moz_keywords
moz_bookmarks moz_hosts moz_places
moz_bookmarks_roots moz_inputhistory
but none of them contain much data (I have tried SELECT * FROM <TABLE> ; for all of them).
Further, if I dump the entire database into file dump.sqlite:
sqlite> .output dump.sqlite
sqlite> .dump
The file has only couple of KB:
$ ls -lh
total 1.2M
-rw------- 1 martin martin 39K Sep 8 19:01 dump.sqlite
-rw------- 1 martin martin 1.2M Sep 8 07:24 places.sqlite
I am aware of the vacuum command in sqlite3. I did that, but it has no effect. The database file is still same size.
How can I investigate why the database file is so big, and what does it contain ?
sqlite dump
sqlite dump
edited Sep 8 '16 at 17:12
Martin Vegter
asked Sep 8 '16 at 17:05
Martin VegterMartin Vegter
4027
4027
bumped to the homepage by Community♦ 12 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 12 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
Are you attempting to vacuum the file while Firefox is running? Mine is 10MB while Firefox is running, and would not change until I quit Firefox and ran the vaccum against the database while not in use. Immediately after starting Firefox, it again jumped to 10MB (from 3MB post-vacuum). I suspect that Firefox may be pre-allocating space for performance - see this recommendation they make for performance
– alroc
Sep 8 '16 at 17:20
@alroc - no, I am working with the file-copy offline. Firefox is not running.
– Martin Vegter
Sep 8 '16 at 17:38
The file is not just the data inside it. There's some overhead in there. Check the file format for SQLite databases: sqlite.org/fileformat.html
– Randolph West
Sep 10 '16 at 2:30
@Randolph West - thanks, but how can I investigate this "overhead" data ? How I see what is in the file?
– Martin Vegter
Sep 10 '16 at 19:12
I would use a hex editor, and the file format link I posted as a guide.
– Randolph West
Sep 11 '16 at 2:48
add a comment |
Are you attempting to vacuum the file while Firefox is running? Mine is 10MB while Firefox is running, and would not change until I quit Firefox and ran the vaccum against the database while not in use. Immediately after starting Firefox, it again jumped to 10MB (from 3MB post-vacuum). I suspect that Firefox may be pre-allocating space for performance - see this recommendation they make for performance
– alroc
Sep 8 '16 at 17:20
@alroc - no, I am working with the file-copy offline. Firefox is not running.
– Martin Vegter
Sep 8 '16 at 17:38
The file is not just the data inside it. There's some overhead in there. Check the file format for SQLite databases: sqlite.org/fileformat.html
– Randolph West
Sep 10 '16 at 2:30
@Randolph West - thanks, but how can I investigate this "overhead" data ? How I see what is in the file?
– Martin Vegter
Sep 10 '16 at 19:12
I would use a hex editor, and the file format link I posted as a guide.
– Randolph West
Sep 11 '16 at 2:48
Are you attempting to vacuum the file while Firefox is running? Mine is 10MB while Firefox is running, and would not change until I quit Firefox and ran the vaccum against the database while not in use. Immediately after starting Firefox, it again jumped to 10MB (from 3MB post-vacuum). I suspect that Firefox may be pre-allocating space for performance - see this recommendation they make for performance
– alroc
Sep 8 '16 at 17:20
Are you attempting to vacuum the file while Firefox is running? Mine is 10MB while Firefox is running, and would not change until I quit Firefox and ran the vaccum against the database while not in use. Immediately after starting Firefox, it again jumped to 10MB (from 3MB post-vacuum). I suspect that Firefox may be pre-allocating space for performance - see this recommendation they make for performance
– alroc
Sep 8 '16 at 17:20
@alroc - no, I am working with the file-copy offline. Firefox is not running.
– Martin Vegter
Sep 8 '16 at 17:38
@alroc - no, I am working with the file-copy offline. Firefox is not running.
– Martin Vegter
Sep 8 '16 at 17:38
The file is not just the data inside it. There's some overhead in there. Check the file format for SQLite databases: sqlite.org/fileformat.html
– Randolph West
Sep 10 '16 at 2:30
The file is not just the data inside it. There's some overhead in there. Check the file format for SQLite databases: sqlite.org/fileformat.html
– Randolph West
Sep 10 '16 at 2:30
@Randolph West - thanks, but how can I investigate this "overhead" data ? How I see what is in the file?
– Martin Vegter
Sep 10 '16 at 19:12
@Randolph West - thanks, but how can I investigate this "overhead" data ? How I see what is in the file?
– Martin Vegter
Sep 10 '16 at 19:12
I would use a hex editor, and the file format link I posted as a guide.
– Randolph West
Sep 11 '16 at 2:48
I would use a hex editor, and the file format link I posted as a guide.
– Randolph West
Sep 11 '16 at 2:48
add a comment |
1 Answer
1
active
oldest
votes
I had a file that was around 30 MB, and issued the vacuum full (not the normal VACUUM as it has the same effect that the Postgres behavior) for reducing its size. After the operation the file got sized 22 MB. I'm pasting the stats of each execution, which shows the differences in between (you will see more failures due to size for the VF).
sqlite> vacuum;
Successful lookaside attempts: 3690
Lookaside failures due to size: 913
Lookaside failures due to OOM: 0
sqlite> vacuum full;
Successful lookaside attempts: 6509
Lookaside failures due to size: 1621
Lookaside failures due to OOM: 0
The command .dbinfo will provide some information as the database page size and its count. Database pages are variable as specified in the file format doc page (this link contains all the internal structures).
For investigating the contents you can use sqlite_stats1 (the link contains the specifications of this table) table, which will be created after the ANALYZE:
analyze;
select * from sqlite_stat1;
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "182"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f149134%2finvestigate-contents-of-a-sqlite3-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I had a file that was around 30 MB, and issued the vacuum full (not the normal VACUUM as it has the same effect that the Postgres behavior) for reducing its size. After the operation the file got sized 22 MB. I'm pasting the stats of each execution, which shows the differences in between (you will see more failures due to size for the VF).
sqlite> vacuum;
Successful lookaside attempts: 3690
Lookaside failures due to size: 913
Lookaside failures due to OOM: 0
sqlite> vacuum full;
Successful lookaside attempts: 6509
Lookaside failures due to size: 1621
Lookaside failures due to OOM: 0
The command .dbinfo will provide some information as the database page size and its count. Database pages are variable as specified in the file format doc page (this link contains all the internal structures).
For investigating the contents you can use sqlite_stats1 (the link contains the specifications of this table) table, which will be created after the ANALYZE:
analyze;
select * from sqlite_stat1;
add a comment |
I had a file that was around 30 MB, and issued the vacuum full (not the normal VACUUM as it has the same effect that the Postgres behavior) for reducing its size. After the operation the file got sized 22 MB. I'm pasting the stats of each execution, which shows the differences in between (you will see more failures due to size for the VF).
sqlite> vacuum;
Successful lookaside attempts: 3690
Lookaside failures due to size: 913
Lookaside failures due to OOM: 0
sqlite> vacuum full;
Successful lookaside attempts: 6509
Lookaside failures due to size: 1621
Lookaside failures due to OOM: 0
The command .dbinfo will provide some information as the database page size and its count. Database pages are variable as specified in the file format doc page (this link contains all the internal structures).
For investigating the contents you can use sqlite_stats1 (the link contains the specifications of this table) table, which will be created after the ANALYZE:
analyze;
select * from sqlite_stat1;
add a comment |
I had a file that was around 30 MB, and issued the vacuum full (not the normal VACUUM as it has the same effect that the Postgres behavior) for reducing its size. After the operation the file got sized 22 MB. I'm pasting the stats of each execution, which shows the differences in between (you will see more failures due to size for the VF).
sqlite> vacuum;
Successful lookaside attempts: 3690
Lookaside failures due to size: 913
Lookaside failures due to OOM: 0
sqlite> vacuum full;
Successful lookaside attempts: 6509
Lookaside failures due to size: 1621
Lookaside failures due to OOM: 0
The command .dbinfo will provide some information as the database page size and its count. Database pages are variable as specified in the file format doc page (this link contains all the internal structures).
For investigating the contents you can use sqlite_stats1 (the link contains the specifications of this table) table, which will be created after the ANALYZE:
analyze;
select * from sqlite_stat1;
I had a file that was around 30 MB, and issued the vacuum full (not the normal VACUUM as it has the same effect that the Postgres behavior) for reducing its size. After the operation the file got sized 22 MB. I'm pasting the stats of each execution, which shows the differences in between (you will see more failures due to size for the VF).
sqlite> vacuum;
Successful lookaside attempts: 3690
Lookaside failures due to size: 913
Lookaside failures due to OOM: 0
sqlite> vacuum full;
Successful lookaside attempts: 6509
Lookaside failures due to size: 1621
Lookaside failures due to OOM: 0
The command .dbinfo will provide some information as the database page size and its count. Database pages are variable as specified in the file format doc page (this link contains all the internal structures).
For investigating the contents you can use sqlite_stats1 (the link contains the specifications of this table) table, which will be created after the ANALYZE:
analyze;
select * from sqlite_stat1;
answered Sep 19 '16 at 5:44
3manuek3manuek
1,193410
1,193410
add a comment |
add a comment |
Thanks for contributing an answer to Database Administrators Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f149134%2finvestigate-contents-of-a-sqlite3-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Are you attempting to vacuum the file while Firefox is running? Mine is 10MB while Firefox is running, and would not change until I quit Firefox and ran the vaccum against the database while not in use. Immediately after starting Firefox, it again jumped to 10MB (from 3MB post-vacuum). I suspect that Firefox may be pre-allocating space for performance - see this recommendation they make for performance
– alroc
Sep 8 '16 at 17:20
@alroc - no, I am working with the file-copy offline. Firefox is not running.
– Martin Vegter
Sep 8 '16 at 17:38
The file is not just the data inside it. There's some overhead in there. Check the file format for SQLite databases: sqlite.org/fileformat.html
– Randolph West
Sep 10 '16 at 2:30
@Randolph West - thanks, but how can I investigate this "overhead" data ? How I see what is in the file?
– Martin Vegter
Sep 10 '16 at 19:12
I would use a hex editor, and the file format link I posted as a guide.
– Randolph West
Sep 11 '16 at 2:48