Vryjdfkk

I've been running MariaDB for some time now without problem. I did a minor version upgrade from 10.2.14 to 10.2.15 and now it refuses to start. If I'm reading the error logs correctly, it can't find the tables mysql.user and mysql.servers. I have confirmed that the data files exist, are not corrupted, and they appear to have the proper permissions.

MariaDB 10.2.15 running on Fedora 28.

This is the dump from the error log.

2018-06-19 17:12:41 140244707127552 [Note] RocksDB: 2 column families found

2018-06-19 17:12:41 140244707127552 [Note] RocksDB: Column Families at start:

2018-06-19 17:12:41 140244707127552 [Note]   cf=default

2018-06-19 17:12:41 140244707127552 [Note]     write_buffer_size=67108864

2018-06-19 17:12:41 140244707127552 [Note]     target_file_size_base=67108864

2018-06-19 17:12:41 140244707127552 [Note]   cf=__system__

2018-06-19 17:12:41 140244707127552 [Note]     write_buffer_size=67108864

2018-06-19 17:12:41 140244707127552 [Note]     target_file_size_base=67108864

2018-06-19 17:12:41 140244707127552 [Note] RocksDB: Table_store: loaded DDL data for 0 tables

2018-06-19 17:12:41 140244707127552 [Note] RocksDB: global statistics using get_sched_indexer_t indexer

2018-06-19 17:12:41 140244707127552 [Note] MyRocks storage engine plugin has been successfully initialized.

Tue Jun 19 17:12:41 2018 PerconaFT recovery starting in env /var/lib/mysql/

Tue Jun 19 17:12:41 2018 PerconaFT recovery scanning backward from 1319

Tue Jun 19 17:12:41 2018 PerconaFT recovery bw_end_checkpoint at 1319 timestamp 1529453469731997 xid 1315 (bw_newer)

Tue Jun 19 17:12:41 2018 PerconaFT recovery bw_begin_checkpoint at 1315 timestamp 1529453469731991 (bw_between)

Tue Jun 19 17:12:41 2018 PerconaFT recovery turning around at begin checkpoint 1315 time 6

Tue Jun 19 17:12:41 2018 PerconaFT recovery starts scanning forward to 1319 from 1315 left 4 (fw_between)

Tue Jun 19 17:12:41 2018 PerconaFT recovery closing 2 dictionaries

Tue Jun 19 17:12:41 2018 PerconaFT recovery making a checkpoint

Tue Jun 19 17:12:41 2018 PerconaFT recovery done

2018-06-19 17:12:41 140244707127552 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2018-06-19 17:12:41 140244707127552 [Note] InnoDB: Uses event mutexes

2018-06-19 17:12:41 140244707127552 [Note] InnoDB: Compressed tables use zlib 1.2.11

2018-06-19 17:12:41 140244707127552 [Note] InnoDB: Using Linux native AIO

2018-06-19 17:12:41 140244707127552 [Note] InnoDB: Number of pools: 1

2018-06-19 17:12:41 140244707127552 [Note] InnoDB: Using SSE2 crc32 instructions

2018-06-19 17:12:41 140244707127552 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M

2018-06-19 17:12:41 140244707127552 [Note] InnoDB: Completed initialization of buffer pool

2018-06-19 17:12:41 140243754858240 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

2018-06-19 17:12:41 140244707127552 [Note] InnoDB: Highest supported file format is Barracuda.

2018-06-19 17:12:41 140244707127552 [Note] InnoDB: 128 out of 128 rollback segments are active.

2018-06-19 17:12:41 140244707127552 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"

2018-06-19 17:12:41 140244707127552 [Note] InnoDB: Creating shared tablespace for temporary tables

2018-06-19 17:12:41 140244707127552 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...

2018-06-19 17:12:41 140244707127552 [Note] InnoDB: File './ibtmp1' size is now 12 MB.

2018-06-19 17:12:41 140244707127552 [Note] InnoDB: Waiting for purge to start

2018-06-19 17:12:41 140244707127552 [Note] InnoDB: 5.7.22 started; log sequence number 26491335405

2018-06-19 17:12:41 140243746465536 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool

2018-06-19 17:12:41 140244707127552 [Note] Plugin 'FEEDBACK' is disabled.

2018-06-19 17:12:41 140244707127552 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded

2018-06-19 17:12:41 140244707127552 [Note] Recovering after a crash using tc.log

2018-06-19 17:12:41 140244707127552 [Note] Starting crash recovery...

2018-06-19 17:12:41 140244707127552 [Note] Crash recovery finished.

2018-06-19 17:12:41 140244707127552 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist

2018-06-19 17:12:41 140244707127552 [Note] Server socket created on IP: '::'.

2018-06-19 17:12:41 140244707127552 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.user' doesn't exist

This the systemd output:

Starting MariaDB 10.2 database server...

mysql-prepare-db-dir[10435]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.

mysql-prepare-db-dir[10435]: If this is not the case, make sure the /var/lib/mysql is empty before running mysql-prepare-db-dir.

[Note] /usr/libexec/mysqld (mysqld 10.2.15-MariaDB) starting as process 10472 ...

systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE

systemd[1]: mariadb.service: Failed with result 'exit-code'.

systemd[1]: Failed to start MariaDB 10.2 database server.

I've run myisamchk on all the tables in the data directory, and I've uninstalled and reinstalled the mariadb-server package. As far as I can tell it hasn't made a difference. Any thoughts?

UPDATE: this is the error message from running mysql_upgrade

mysql_upgrade --upgrade-system-tables

The --upgrade-system-tables option was used, user tables won't be touched.

This installation of MySQL is already upgraded to 10.2.15-MariaDB, use --force if you still need to run mysql_upgrade

So it shows them as having been upgraded. If I run with --force, I get this:

mysql_upgrade --upgrade-system-tables --force

The --upgrade-system-tables option was used, user tables won't be touched.

Phase 1/7: Checking and upgrading mysql database

Processing databases

mysql

mysql.column_stats                                 OK

mysql.columns_priv                                 OK

mysql.db                                           OK

mysql.event                                        OK

mysql.func                                         OK

mysql.help_category                                OK

mysql.help_keyword                                 OK

mysql.help_relation                                OK

mysql.help_topic                                   OK

mysql.host                                         OK

mysql.index_stats                                  OK

mysql.plugin                                       OK

mysql.proc                                         OK

mysql.procs_priv                                   OK

mysql.proxies_priv                                 OK

mysql.roles_mapping                                OK

mysql.servers                                      OK

mysql.table_stats                                  OK

mysql.tables_priv                                  OK

mysql.time_zone                                    OK

mysql.time_zone_leap_second                        OK

mysql.time_zone_name                               OK

mysql.time_zone_transition                         OK

mysql.time_zone_transition_type                    OK

mysql.user                                         OK

Phase 2/7: Installing used storage engines... Skipped

Phase 3/7: Fixing views... Skipped

Phase 4/7: Running 'mysql_fix_privilege_tables'

ERROR 1813 (HY000) at line 64: Tablespace for table '`mysql`.`innodb_table_stats`' exists. Please DISCARD the tablespace before IMPORT

ERROR 1813 (HY000) at line 68: Tablespace for table '`mysql`.`innodb_index_stats`' exists. Please DISCARD the tablespace before IMPORT

ERROR 1813 (HY000) at line 136: Tablespace for table '`mysql`.`gtid_slave_pos`' exists. Please DISCARD the tablespace before IMPORT

ERROR 1146 (42S02) at line 588: Table 'mysql.innodb_index_stats' doesn't exist

ERROR 1243 (HY000) at line 589: Unknown prepared statement handler (stmt) given to EXECUTE

ERROR 1146 (42S02) at line 591: Table 'mysql.innodb_table_stats' doesn't exist

ERROR 1243 (HY000) at line 592: Unknown prepared statement handler (stmt) given to EXECUTE

ERROR 1146 (42S02) at line 596: Table 'mysql.innodb_index_stats' doesn't exist

ERROR 1146 (42S02) at line 600: Table 'mysql.innodb_table_stats' doesn't exist

ERROR 1146 (42S02) at line 603: Table 'mysql.innodb_table_stats' doesn't exist

FATAL ERROR: Upgrade failed

_{Answer left by the question author in a comment}

The frm files for the affected tables were missing (such as innodb_table_stats.frm) . I was able to retrieve copies from a backup directory. I stopped the server, copied the files over, and started the server again.

I was also able to copy over files for the tables user and servers. I did have to run restorecon on them to make them play nice with SELinux. After that I wound up having to reauthorize all the database permissions for each user other than root.

As far as I can tell, everything is working fine now.