Hello everyone,
I need your help to deal with this problem on a production server.
I was trying to delete a seemingly corrupted crm database when mariadb crashed.
Since then mariadb.service has been unavailable.
.....server details......
openSUSE Leap 15.1
mariadb 10.2.31-lp151.2.12.1 x86_64 from opensuse
php 7.2.5
php7-MySQL Mysql client for php (mysql 5.0.2)
..some commands executed...........
#systemctl stop mariadb.service (No error thrown probably because mariadb wasn't running anyway)
.......
#systemctl start mariadb.service
Job for mariadb.service failed because a fatal signal was delivered to the control process.
.......
#systemctl status mariadb.service
â mariadb.service - MySQL server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disa>
Active: failed (Result: protocol) since Sun 2024-06-23 16:35:52 CEST; 19min ago
Process: 2895 ExecStart=/usr/lib/mysql/mysql-systemd-helper start (code=killed, signal>
Process: 2888 ExecStartPre=/usr/lib/mysql/mysql-systemd-helper upgrade (code=exited, s>
Process: 2882 ExecStartPre=/usr/lib/mysql/mysql-systemd-helper install (code=exited, s>
Main PID: 2895 (code=killed, signal=SEGV)
Status: "Starting final batch to recover 87 pages from redo log"
Jun 23 16:35:46 smooths systemd[1]: Starting MySQL server...
Jun 23 16:35:47 smooths mysql-systemd-helper[2895]: 2024-06-23 16:35:47 14071001006>
Jun 23 16:35:52 smooths systemd[1]: mariadb.service: Main process exited, code=kill>
Jun 23 16:35:52 smooths systemd[1]: Failed to start MySQL server.
Jun 23 16:35:52 smooths systemd[1]: mariadb.service: Unit entered failed state.
Jun 23 16:35:52 smooths systemd[1]: mariadb.service: Failed with result 'protocol'.
[4]+ Stopped systemctl status mariadb.service
..........
#journalctl -xe
-- Unit UNIT has finished shutting down.
Jun 24 10:20:02 smooths systemd[13641]: Reached target Shutdown.
-- Subject: Unit UNIT has finished start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit UNIT has finished starting up.
--
-- The start-up result is done.
Jun 24 10:20:02 smooths systemd[13641]: Starting Exit the Session...
-- Subject: Unit UNIT has begun start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit UNIT has begun starting up.
Jun 24 10:20:02 smooths systemd[13641]: Received SIGRTMIN+24 from PID 13648 (kill).
Jun 24 10:20:02 smooths systemd[13642]: pam_unix(systemd-user:session): session clo>
Jun 24 10:20:02 smooths systemd[1]: Stopped User Manager for UID 72.
-- Subject: Unit user@72.service has finished shutting down
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit user@72.service has finished shutting down.
Jun 24 10:20:02 smooths systemd[1]: Removed slice User Slice of UID 72.
-- Subject: Unit user-72.slice has finished shutting down
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit user-72.slice has finished shutting down.
[6]+ Stopped journalctl -xe
............
#ps auxw |grep mysql
root 14238 0.0 0.0 8692 920 pts/0 S+ 11:11 0:00 grep --color=auto mysql
#mysql -u root -p
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/run/mysql/mysql.sock' (111)
.....last bits in mysql.log...(I could post more from the log if required)........
2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
2024-06-24 10:19:37 140453265478208 [Note] Plugin 'FEEDBACK' is disabled.
2024-06-24 10:19:37 140453265478208 [Note] Recovering after a crash using tc.log
2024-06-24 10:19:37 140453265478208 [Note] Starting crash recovery...
2024-06-24 10:19:37 140453265478208 [Note] Crash recovery finished.
2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Page [page id: space=86, page number=102] log sequence number 75258578838 is in the future! Current system log sequence number 75111066143.
2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Page [page id: space=86, page number=103] log sequence number 75769951403 is in the future! Current system log sequence number 75111066143.
2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Page [page id: space=160, page number=3] log sequence number 75133061380 is in the future! Current system log sequence number 75111066143.
2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
/lib64/libc.so.6(abort+0x151)[0x7fbdd0279b01]
/usr/sbin/mysqld(+0x44616a)[0x55c5418b916a]
/usr/sbin/mysqld(+0x8fd0b8)[0x55c541d700b8]
/usr/sbin/mysqld(+0x8fd85a)[0x55c541d7085a]
/usr/sbin/mysqld(+0x8fe1bb)[0x55c541d711bb]
/usr/sbin/mysqld(+0x8e1693)[0x55c541d54693]
/lib64/libpthread.so.0(+0x84f9)[0x7fbdd0edc4f9]
2024-06-24 10:19:37 140453265478208 [Note] Server socket created on IP: '0.0.0.0'.
/lib64/libc.so.6(clone+0x3f)[0x7fbdd033af2f]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
We think the query pointer is invalid, but we will try to print it anyway.
Query:
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 35758 35758 processes
Max open files 4184 4184 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 35758 35758 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: |/bin/false
.................
Please note the following:
- I renamed both ib_logfile0 and ib_logfile1 and they both got recreated when service.mysql command was ran.
- There is mysql.sock in /run/mysql
- The corrupt database I was trying to delete without success is still in /var/lib/mysql/
What do I need to do to get mariadb working again? Thank you.