investigate contents of a sqlite3 file












3















I have created a new, clean firefox profile from scratch, and added couple of bookmarks. These, I have learned, are stored in places.sqlite file inside the profile directory. This file seems to be too big (1.2 MB) and I am wondering why:



$ ls -lh places.sqlite 
-rw------- 1 martin martin 1.2M Sep 8 07:24 places.sqlite


When I open it with sqlite3 from the commandline, I see there are couple of tables:



$ sqlite3 places.sqlite 
SQLite version 3.7.13 2012-06-11 02:05:22
sqlite> .tables
moz_anno_attributes moz_favicons moz_items_annos
moz_annos moz_historyvisits moz_keywords
moz_bookmarks moz_hosts moz_places
moz_bookmarks_roots moz_inputhistory


but none of them contain much data (I have tried SELECT * FROM <TABLE> ; for all of them).



Further, if I dump the entire database into file dump.sqlite:



sqlite> .output dump.sqlite
sqlite> .dump


The file has only couple of KB:



$ ls -lh
total 1.2M
-rw------- 1 martin martin 39K Sep 8 19:01 dump.sqlite
-rw------- 1 martin martin 1.2M Sep 8 07:24 places.sqlite


I am aware of the vacuum command in sqlite3. I did that, but it has no effect. The database file is still same size.



How can I investigate why the database file is so big, and what does it contain ?










share|improve this question
















bumped to the homepage by Community 12 mins ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
















  • Are you attempting to vacuum the file while Firefox is running? Mine is 10MB while Firefox is running, and would not change until I quit Firefox and ran the vaccum against the database while not in use. Immediately after starting Firefox, it again jumped to 10MB (from 3MB post-vacuum). I suspect that Firefox may be pre-allocating space for performance - see this recommendation they make for performance

    – alroc
    Sep 8 '16 at 17:20











  • @alroc - no, I am working with the file-copy offline. Firefox is not running.

    – Martin Vegter
    Sep 8 '16 at 17:38











  • The file is not just the data inside it. There's some overhead in there. Check the file format for SQLite databases: sqlite.org/fileformat.html

    – Randolph West
    Sep 10 '16 at 2:30











  • @Randolph West - thanks, but how can I investigate this "overhead" data ? How I see what is in the file?

    – Martin Vegter
    Sep 10 '16 at 19:12











  • I would use a hex editor, and the file format link I posted as a guide.

    – Randolph West
    Sep 11 '16 at 2:48
















3















I have created a new, clean firefox profile from scratch, and added couple of bookmarks. These, I have learned, are stored in places.sqlite file inside the profile directory. This file seems to be too big (1.2 MB) and I am wondering why:



$ ls -lh places.sqlite 
-rw------- 1 martin martin 1.2M Sep 8 07:24 places.sqlite


When I open it with sqlite3 from the commandline, I see there are couple of tables:



$ sqlite3 places.sqlite 
SQLite version 3.7.13 2012-06-11 02:05:22
sqlite> .tables
moz_anno_attributes moz_favicons moz_items_annos
moz_annos moz_historyvisits moz_keywords
moz_bookmarks moz_hosts moz_places
moz_bookmarks_roots moz_inputhistory


but none of them contain much data (I have tried SELECT * FROM <TABLE> ; for all of them).



Further, if I dump the entire database into file dump.sqlite:



sqlite> .output dump.sqlite
sqlite> .dump


The file has only couple of KB:



$ ls -lh
total 1.2M
-rw------- 1 martin martin 39K Sep 8 19:01 dump.sqlite
-rw------- 1 martin martin 1.2M Sep 8 07:24 places.sqlite


I am aware of the vacuum command in sqlite3. I did that, but it has no effect. The database file is still same size.



How can I investigate why the database file is so big, and what does it contain ?










share|improve this question
















bumped to the homepage by Community 12 mins ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
















  • Are you attempting to vacuum the file while Firefox is running? Mine is 10MB while Firefox is running, and would not change until I quit Firefox and ran the vaccum against the database while not in use. Immediately after starting Firefox, it again jumped to 10MB (from 3MB post-vacuum). I suspect that Firefox may be pre-allocating space for performance - see this recommendation they make for performance

    – alroc
    Sep 8 '16 at 17:20











  • @alroc - no, I am working with the file-copy offline. Firefox is not running.

    – Martin Vegter
    Sep 8 '16 at 17:38











  • The file is not just the data inside it. There's some overhead in there. Check the file format for SQLite databases: sqlite.org/fileformat.html

    – Randolph West
    Sep 10 '16 at 2:30











  • @Randolph West - thanks, but how can I investigate this "overhead" data ? How I see what is in the file?

    – Martin Vegter
    Sep 10 '16 at 19:12











  • I would use a hex editor, and the file format link I posted as a guide.

    – Randolph West
    Sep 11 '16 at 2:48














3












3








3








I have created a new, clean firefox profile from scratch, and added couple of bookmarks. These, I have learned, are stored in places.sqlite file inside the profile directory. This file seems to be too big (1.2 MB) and I am wondering why:



$ ls -lh places.sqlite 
-rw------- 1 martin martin 1.2M Sep 8 07:24 places.sqlite


When I open it with sqlite3 from the commandline, I see there are couple of tables:



$ sqlite3 places.sqlite 
SQLite version 3.7.13 2012-06-11 02:05:22
sqlite> .tables
moz_anno_attributes moz_favicons moz_items_annos
moz_annos moz_historyvisits moz_keywords
moz_bookmarks moz_hosts moz_places
moz_bookmarks_roots moz_inputhistory


but none of them contain much data (I have tried SELECT * FROM <TABLE> ; for all of them).



Further, if I dump the entire database into file dump.sqlite:



sqlite> .output dump.sqlite
sqlite> .dump


The file has only couple of KB:



$ ls -lh
total 1.2M
-rw------- 1 martin martin 39K Sep 8 19:01 dump.sqlite
-rw------- 1 martin martin 1.2M Sep 8 07:24 places.sqlite


I am aware of the vacuum command in sqlite3. I did that, but it has no effect. The database file is still same size.



How can I investigate why the database file is so big, and what does it contain ?










share|improve this question
















I have created a new, clean firefox profile from scratch, and added couple of bookmarks. These, I have learned, are stored in places.sqlite file inside the profile directory. This file seems to be too big (1.2 MB) and I am wondering why:



$ ls -lh places.sqlite 
-rw------- 1 martin martin 1.2M Sep 8 07:24 places.sqlite


When I open it with sqlite3 from the commandline, I see there are couple of tables:



$ sqlite3 places.sqlite 
SQLite version 3.7.13 2012-06-11 02:05:22
sqlite> .tables
moz_anno_attributes moz_favicons moz_items_annos
moz_annos moz_historyvisits moz_keywords
moz_bookmarks moz_hosts moz_places
moz_bookmarks_roots moz_inputhistory


but none of them contain much data (I have tried SELECT * FROM <TABLE> ; for all of them).



Further, if I dump the entire database into file dump.sqlite:



sqlite> .output dump.sqlite
sqlite> .dump


The file has only couple of KB:



$ ls -lh
total 1.2M
-rw------- 1 martin martin 39K Sep 8 19:01 dump.sqlite
-rw------- 1 martin martin 1.2M Sep 8 07:24 places.sqlite


I am aware of the vacuum command in sqlite3. I did that, but it has no effect. The database file is still same size.



How can I investigate why the database file is so big, and what does it contain ?







sqlite dump






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Sep 8 '16 at 17:12







Martin Vegter

















asked Sep 8 '16 at 17:05









Martin VegterMartin Vegter

4027




4027





bumped to the homepage by Community 12 mins ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.







bumped to the homepage by Community 12 mins ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.















  • Are you attempting to vacuum the file while Firefox is running? Mine is 10MB while Firefox is running, and would not change until I quit Firefox and ran the vaccum against the database while not in use. Immediately after starting Firefox, it again jumped to 10MB (from 3MB post-vacuum). I suspect that Firefox may be pre-allocating space for performance - see this recommendation they make for performance

    – alroc
    Sep 8 '16 at 17:20











  • @alroc - no, I am working with the file-copy offline. Firefox is not running.

    – Martin Vegter
    Sep 8 '16 at 17:38











  • The file is not just the data inside it. There's some overhead in there. Check the file format for SQLite databases: sqlite.org/fileformat.html

    – Randolph West
    Sep 10 '16 at 2:30











  • @Randolph West - thanks, but how can I investigate this "overhead" data ? How I see what is in the file?

    – Martin Vegter
    Sep 10 '16 at 19:12











  • I would use a hex editor, and the file format link I posted as a guide.

    – Randolph West
    Sep 11 '16 at 2:48



















  • Are you attempting to vacuum the file while Firefox is running? Mine is 10MB while Firefox is running, and would not change until I quit Firefox and ran the vaccum against the database while not in use. Immediately after starting Firefox, it again jumped to 10MB (from 3MB post-vacuum). I suspect that Firefox may be pre-allocating space for performance - see this recommendation they make for performance

    – alroc
    Sep 8 '16 at 17:20











  • @alroc - no, I am working with the file-copy offline. Firefox is not running.

    – Martin Vegter
    Sep 8 '16 at 17:38











  • The file is not just the data inside it. There's some overhead in there. Check the file format for SQLite databases: sqlite.org/fileformat.html

    – Randolph West
    Sep 10 '16 at 2:30











  • @Randolph West - thanks, but how can I investigate this "overhead" data ? How I see what is in the file?

    – Martin Vegter
    Sep 10 '16 at 19:12











  • I would use a hex editor, and the file format link I posted as a guide.

    – Randolph West
    Sep 11 '16 at 2:48

















Are you attempting to vacuum the file while Firefox is running? Mine is 10MB while Firefox is running, and would not change until I quit Firefox and ran the vaccum against the database while not in use. Immediately after starting Firefox, it again jumped to 10MB (from 3MB post-vacuum). I suspect that Firefox may be pre-allocating space for performance - see this recommendation they make for performance

– alroc
Sep 8 '16 at 17:20





Are you attempting to vacuum the file while Firefox is running? Mine is 10MB while Firefox is running, and would not change until I quit Firefox and ran the vaccum against the database while not in use. Immediately after starting Firefox, it again jumped to 10MB (from 3MB post-vacuum). I suspect that Firefox may be pre-allocating space for performance - see this recommendation they make for performance

– alroc
Sep 8 '16 at 17:20













@alroc - no, I am working with the file-copy offline. Firefox is not running.

– Martin Vegter
Sep 8 '16 at 17:38





@alroc - no, I am working with the file-copy offline. Firefox is not running.

– Martin Vegter
Sep 8 '16 at 17:38













The file is not just the data inside it. There's some overhead in there. Check the file format for SQLite databases: sqlite.org/fileformat.html

– Randolph West
Sep 10 '16 at 2:30





The file is not just the data inside it. There's some overhead in there. Check the file format for SQLite databases: sqlite.org/fileformat.html

– Randolph West
Sep 10 '16 at 2:30













@Randolph West - thanks, but how can I investigate this "overhead" data ? How I see what is in the file?

– Martin Vegter
Sep 10 '16 at 19:12





@Randolph West - thanks, but how can I investigate this "overhead" data ? How I see what is in the file?

– Martin Vegter
Sep 10 '16 at 19:12













I would use a hex editor, and the file format link I posted as a guide.

– Randolph West
Sep 11 '16 at 2:48





I would use a hex editor, and the file format link I posted as a guide.

– Randolph West
Sep 11 '16 at 2:48










1 Answer
1






active

oldest

votes


















0














I had a file that was around 30 MB, and issued the vacuum full (not the normal VACUUM as it has the same effect that the Postgres behavior) for reducing its size. After the operation the file got sized 22 MB. I'm pasting the stats of each execution, which shows the differences in between (you will see more failures due to size for the VF).



sqlite> vacuum;
Successful lookaside attempts: 3690
Lookaside failures due to size: 913
Lookaside failures due to OOM: 0

sqlite> vacuum full;
Successful lookaside attempts: 6509
Lookaside failures due to size: 1621
Lookaside failures due to OOM: 0


The command .dbinfo will provide some information as the database page size and its count. Database pages are variable as specified in the file format doc page (this link contains all the internal structures).



For investigating the contents you can use sqlite_stats1 (the link contains the specifications of this table) table, which will be created after the ANALYZE:



analyze;
select * from sqlite_stat1;





share|improve this answer























    Your Answer








    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "182"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: false,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f149134%2finvestigate-contents-of-a-sqlite3-file%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    0














    I had a file that was around 30 MB, and issued the vacuum full (not the normal VACUUM as it has the same effect that the Postgres behavior) for reducing its size. After the operation the file got sized 22 MB. I'm pasting the stats of each execution, which shows the differences in between (you will see more failures due to size for the VF).



    sqlite> vacuum;
    Successful lookaside attempts: 3690
    Lookaside failures due to size: 913
    Lookaside failures due to OOM: 0

    sqlite> vacuum full;
    Successful lookaside attempts: 6509
    Lookaside failures due to size: 1621
    Lookaside failures due to OOM: 0


    The command .dbinfo will provide some information as the database page size and its count. Database pages are variable as specified in the file format doc page (this link contains all the internal structures).



    For investigating the contents you can use sqlite_stats1 (the link contains the specifications of this table) table, which will be created after the ANALYZE:



    analyze;
    select * from sqlite_stat1;





    share|improve this answer




























      0














      I had a file that was around 30 MB, and issued the vacuum full (not the normal VACUUM as it has the same effect that the Postgres behavior) for reducing its size. After the operation the file got sized 22 MB. I'm pasting the stats of each execution, which shows the differences in between (you will see more failures due to size for the VF).



      sqlite> vacuum;
      Successful lookaside attempts: 3690
      Lookaside failures due to size: 913
      Lookaside failures due to OOM: 0

      sqlite> vacuum full;
      Successful lookaside attempts: 6509
      Lookaside failures due to size: 1621
      Lookaside failures due to OOM: 0


      The command .dbinfo will provide some information as the database page size and its count. Database pages are variable as specified in the file format doc page (this link contains all the internal structures).



      For investigating the contents you can use sqlite_stats1 (the link contains the specifications of this table) table, which will be created after the ANALYZE:



      analyze;
      select * from sqlite_stat1;





      share|improve this answer


























        0












        0








        0







        I had a file that was around 30 MB, and issued the vacuum full (not the normal VACUUM as it has the same effect that the Postgres behavior) for reducing its size. After the operation the file got sized 22 MB. I'm pasting the stats of each execution, which shows the differences in between (you will see more failures due to size for the VF).



        sqlite> vacuum;
        Successful lookaside attempts: 3690
        Lookaside failures due to size: 913
        Lookaside failures due to OOM: 0

        sqlite> vacuum full;
        Successful lookaside attempts: 6509
        Lookaside failures due to size: 1621
        Lookaside failures due to OOM: 0


        The command .dbinfo will provide some information as the database page size and its count. Database pages are variable as specified in the file format doc page (this link contains all the internal structures).



        For investigating the contents you can use sqlite_stats1 (the link contains the specifications of this table) table, which will be created after the ANALYZE:



        analyze;
        select * from sqlite_stat1;





        share|improve this answer













        I had a file that was around 30 MB, and issued the vacuum full (not the normal VACUUM as it has the same effect that the Postgres behavior) for reducing its size. After the operation the file got sized 22 MB. I'm pasting the stats of each execution, which shows the differences in between (you will see more failures due to size for the VF).



        sqlite> vacuum;
        Successful lookaside attempts: 3690
        Lookaside failures due to size: 913
        Lookaside failures due to OOM: 0

        sqlite> vacuum full;
        Successful lookaside attempts: 6509
        Lookaside failures due to size: 1621
        Lookaside failures due to OOM: 0


        The command .dbinfo will provide some information as the database page size and its count. Database pages are variable as specified in the file format doc page (this link contains all the internal structures).



        For investigating the contents you can use sqlite_stats1 (the link contains the specifications of this table) table, which will be created after the ANALYZE:



        analyze;
        select * from sqlite_stat1;






        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Sep 19 '16 at 5:44









        3manuek3manuek

        1,193410




        1,193410






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Database Administrators Stack Exchange!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f149134%2finvestigate-contents-of-a-sqlite3-file%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            Ronny Ackermann

            Köttigit

            MySQL 8.0.15 starts normally but any connection hangs