Mariadb crashed while deleting a database

Hello everyone,

I need your help to deal with this problem on a production server.

I was trying to delete a seemingly corrupted crm database when mariadb crashed.
Since then mariadb.service has been unavailable.

.....server details......
openSUSE Leap 15.1
mariadb 10.2.31-lp151.2.12.1 x86_64 from opensuse
php 7.2.5
php7-MySQL Mysql client for php (mysql 5.0.2)

..some commands executed...........
#systemctl stop mariadb.service  (No error thrown probably because mariadb wasn't running anyway)
.......
#systemctl start mariadb.service
Job for mariadb.service failed because a fatal signal was delivered to the control process.
.......
#systemctl status  mariadb.service
● mariadb.service - MySQL server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disa>
   Active: failed (Result: protocol) since Sun 2024-06-23 16:35:52 CEST; 19min ago
  Process: 2895 ExecStart=/usr/lib/mysql/mysql-systemd-helper start (code=killed, signal>
  Process: 2888 ExecStartPre=/usr/lib/mysql/mysql-systemd-helper upgrade (code=exited, s>
  Process: 2882 ExecStartPre=/usr/lib/mysql/mysql-systemd-helper install (code=exited, s>
 Main PID: 2895 (code=killed, signal=SEGV)
   Status: "Starting final batch to recover 87 pages from redo log"

Jun 23 16:35:46 smooths systemd[1]: Starting MySQL server...
Jun 23 16:35:47 smooths mysql-systemd-helper[2895]: 2024-06-23 16:35:47 14071001006>
Jun 23 16:35:52 smooths systemd[1]: mariadb.service: Main process exited, code=kill>
Jun 23 16:35:52 smooths systemd[1]: Failed to start MySQL server.
Jun 23 16:35:52 smooths systemd[1]: mariadb.service: Unit entered failed state.
Jun 23 16:35:52 smooths systemd[1]: mariadb.service: Failed with result 'protocol'.

[4]+  Stopped                 systemctl status mariadb.service

..........
#journalctl -xe
-- Unit UNIT has finished shutting down.
Jun 24 10:20:02 smooths systemd[13641]: Reached target Shutdown.
-- Subject: Unit UNIT has finished start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit UNIT has finished starting up.
--
-- The start-up result is done.
Jun 24 10:20:02 smooths systemd[13641]: Starting Exit the Session...
-- Subject: Unit UNIT has begun start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit UNIT has begun starting up.
Jun 24 10:20:02 smooths systemd[13641]: Received SIGRTMIN+24 from PID 13648 (kill).
Jun 24 10:20:02 smooths systemd[13642]: pam_unix(systemd-user:session): session clo>
Jun 24 10:20:02 smooths systemd[1]: Stopped User Manager for UID 72.
-- Subject: Unit user@72.service has finished shutting down
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit user@72.service has finished shutting down.
Jun 24 10:20:02 smooths systemd[1]: Removed slice User Slice of UID 72.
-- Subject: Unit user-72.slice has finished shutting down
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit user-72.slice has finished shutting down.

[6]+  Stopped                 journalctl -xe
............

#ps auxw |grep mysql
root     14238  0.0  0.0   8692   920 pts/0    S+   11:11   0:00 grep --color=auto mysql

#mysql -u root -p
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/run/mysql/mysql.sock' (111)

.....last bits in mysql.log...(I could post more from the log if required)........

2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
2024-06-24 10:19:37 140453265478208 [Note] Plugin 'FEEDBACK' is disabled.
2024-06-24 10:19:37 140453265478208 [Note] Recovering after a crash using tc.log
2024-06-24 10:19:37 140453265478208 [Note] Starting crash recovery...
2024-06-24 10:19:37 140453265478208 [Note] Crash recovery finished.
2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Page [page id: space=86, page number=102] log sequence number 75258578838 is in the future! Current system log sequence number 75111066143.
2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Page [page id: space=86, page number=103] log sequence number 75769951403 is in the future! Current system log sequence number 75111066143.
2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Page [page id: space=160, page number=3] log sequence number 75133061380 is in the future! Current system log sequence number 75111066143.
2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
/lib64/libc.so.6(abort+0x151)[0x7fbdd0279b01]
/usr/sbin/mysqld(+0x44616a)[0x55c5418b916a]
/usr/sbin/mysqld(+0x8fd0b8)[0x55c541d700b8]
/usr/sbin/mysqld(+0x8fd85a)[0x55c541d7085a]
/usr/sbin/mysqld(+0x8fe1bb)[0x55c541d711bb]
/usr/sbin/mysqld(+0x8e1693)[0x55c541d54693]
/lib64/libpthread.so.0(+0x84f9)[0x7fbdd0edc4f9]
2024-06-24 10:19:37 140453265478208 [Note] Server socket created on IP: '0.0.0.0'.
/lib64/libc.so.6(clone+0x3f)[0x7fbdd033af2f]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

We think the query pointer is invalid, but we will try to print it anyway. 
Query: 

Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             35758                35758                processes 
Max open files            4184                 4184                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       35758                35758                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: |/bin/false
.................

Please note the following:

  • I renamed both ib_logfile0 and ib_logfile1 and they both got recreated when service.mysql command was ran.
  • There is mysql.sock in /run/mysql
  • The corrupt database I was trying to delete without success is still in /var/lib/mysql/

What do I need to do to get mariadb working again? Thank you.

As tag you say “leap-15.2-eol”, but here I see even 15.1! That became end-of-life five almost five years ago.

That means that a lot has changed in the mean time. It also means that people have no systems anymore to e.g. try to re-create your problem. So please be aware that, while people will try to help you, they may feel restricted in doing so.

Yeah, there was no leap-15.1 option on the list so I used the closest option available. Some of us have been around a while and can’t afford changing system as quickly as there were new releases. But we are still here.

I will appreciate any meaningful help and will put your statement into consideration going forward.

5 years is not “as quickly as there were new releases.” :wink:
Because it is a production server, use your backup.
If you don’t have a backup, follow the instructions from mariadb.

Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.

“5 years is not “as quickly as there were new releases.” :wink:”
Ok, I hear you. I am thankful for the amount of work put into constantly developing openSUSE. That people like me can’t keep up with new releases is not a complaint. So, regardless, show must go on.

In the meantime, I have tried to use innodb-force-recovery to force recovery of the databases. (InnoDB System Variables - MariaDB Knowledge Base)

I have put the option immediately under the [Mysqld] like: innodb-force-recovery=1 in the /etc/mycnf. I then tried to start mysql with that option on the command line like:

#mysql --innodb-force-recovery=1 --host=localhost --user=root --password=

but it threw an error like:

#mysql: unknown variable ‘innodb-force-recovery=1’

Please help answer the following questions:

  • Why is the variable unknown?

  • Is innodb-force-recovery not packaged with my version of openSUSE?

  • What am I doing wrong?

Thank you.

Did you have the option in both places (my.cnf and command line) at the same time?

If so, did you try to put the option only in the command line (i.e. no option in my.cnf)?

Did you have the option in both places (my.cnf and command line) at the same time?
Yes, I did.

If so, did you try to put the option only in the command line (i.e. no option in my.cnf)?
Yes, I did that as well. Played around with the levels also (0,1,2,etc) I received the same error in both occasions.

So you put

[mysqld]
innodb_force_recovery = 1

in your /etc/my.cnf file?

Did you then stop the server (systemctl stop mysql) and restart it again (systemctl start mysql)?

And what was the status (systemctl status mysql) after restart?

Beware I use openSUSE Tumbleweed so i don’t know whether the commands shown above will work on your system.

" So you put
[mysqld]
innodb_force_recovery = 1"

No. I put
“innodb_force_recovery=1”
And mysql started without error when “systemctl start mysql” is executed

As suggested by you:
[mysqld]
innodb_force_recovery = 1

smooths:~ # systemctl stop mysql
smooths:~ # systemctl start mysql
Job for mariadb.service failed because a fatal signal was delivered to the control process.
See “systemctl status mariadb.service” and “journalctl -xe” for details.

smooths:~ # systemctl status mysql
mariadb.service - MySQL server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disa>
Active: failed (Result: signal) since Wed 2024-06-26 23:25:10 CEST; 1min 41s ago
Process: 28420 ExecStart=/usr/lib/mysql/mysql-systemd-helper start (code=killed, signa>
Process: 28414 ExecStartPre=/usr/lib/mysql/mysql-systemd-helper upgrade (code=exited, >
Process: 28408 ExecStartPre=/usr/lib/mysql/mysql-systemd-helper install (code=exited, >
Main PID: 28420 (code=killed, signal=ABRT)

Jun 26 23:25:10 smooths systemd[1]: mariadb.service: Unit entered failed state.
Jun 26 23:25:10 smooths systemd[1]: mariadb.service: Failed with result ‘signal’.
Jun 26 23:25:10 smooths systemd[1]: mariadb.service: Service RestartSec=100ms expir>
Jun 26 23:25:10 smooths systemd[1]: Stopped MySQL server.
Jun 26 23:25:10 smooths systemd[1]: mariadb.service: Start request repeated too qui>
Jun 26 23:25:10 smooths systemd[1]: Failed to start MySQL server.
Jun 26 23:25:10 smooths systemd[1]: mariadb.service: Unit entered failed state.
Jun 26 23:25:10 smooths systemd[1]: mariadb.service: Failed with result ‘signal’.

Stopped systemctl status mysql

smooths:~ # mysql --innodb_force_recovery = 1 --host = localhost --user = root --password =
mysql: unknown option ‘–innodb_force_recovery’
smooths:~ # journalctl -xe
Jun 26 23:25:09 smooths mysql-systemd-helper[28420]: 2024-06-26 23:25:09 1398181865>
Jun 26 23:25:10 smooths systemd[1]: mariadb.service: Main process exited, code=kill>
Jun 26 23:25:10 smooths systemd[1]: Failed to start MySQL server.
– Subject: Unit mariadb.service has failed
– Defined-By: systemd
– Support: systemd-devel Info Page

– Unit mariadb.service has failed.

– The result is failed.
Jun 26 23:25:10 smooths systemd[1]: mariadb.service: Unit entered failed state.
Jun 26 23:25:10 smooths systemd[1]: mariadb.service: Failed with result ‘signal’.
Jun 26 23:25:10 smooths systemd[1]: mariadb.service: Service RestartSec=100ms expir>
Jun 26 23:25:10 smooths systemd[1]: Stopped MySQL server.
– Subject: Unit mariadb.service has finished shutting down
– Defined-By: systemd
– Support: systemd-devel Info Page

– Unit mariadb.service has finished shutting down.
Jun 26 23:25:10 smooths systemd[1]: mariadb.service: Start request repeated too qui>
Jun 26 23:25:10 smooths systemd[1]: Failed to start MySQL server.
– Subject: Unit mariadb.service has failed
– Defined-By: systemd
– Support: systemd-devel Info Page

– Unit mariadb.service has failed.

The result is failed.
Jun 26 23:25:10 smooths systemd[1]: mariadb.service: Unit entered failed state.
Jun 26 23:25:10 smooths systemd[1]: mariadb.service: Failed with result ‘signal’.

[13]+ Stopped journalctl -xe

Beware –innodb_force_recovery is only a valid option for the system daemon mariadbd (= mysqld). It will not work with mariadb (= mysql).

If you want to use it, you either have to place it in your my.cnf or you have to start mariadbd from the command line. The latter might be the better option for testing because it will send messages to the console.

So you could

  1. systemctl stop mysql

  2. remove the line that contains innodb_force_recovery from your my.cnf

  3. start the database daemon from the command line
    mariadbd --innodb_force_recovery=n

It might be a good idea to use a value greater 1 but smaller or equal 4 for n .

And please use the forums formatting tools (see here for more information on this topic) to give your posts a layout that is easier to read.

Thanks for your reply.

" please use the forums formatting tools (see here for more information on this topic)"

Absolutely. You missed attaching the actual link though. Please repost the link.

I have done the following:

smooths:~ # systemctl stop mysql
smooths:~ # mariadbd --innodb_force_recovery=2
If ‘mariadbd’ is not a typo you can use command-not-found to lookup the package that contains it, like this:
cnf mariadbd

smooths:~ # systemctl stop mysql
smooths:~ # systemctl start mariadb.service
Job for mariadb.service failed because a fatal signal was delivered to the control process.
See “systemctl status mariadb.service” and “journalctl -xe” for details.
smooths:~ # systemctl start mysql

Sorry! My fault.

Here it is.

(BTW the FAQ-Section of the forum has this link as well.)

As I said some posts ago: I use openSUSE Tumbleweed (mariadbd Ver 11.4.2). You need to find out what the database daemon is called on your system and how it gets started.

On my system a systemd-service (/etc/systemd/system/mysql.service) calls a script (/usr/libexec/mysql/mysql-systemd-helper) which will use the symlink /usr/sbin/mysqld to start the daemon /usr/sbin/mariadbd.

And on my system the parameter is called innodb-force-recovery not innodb_force_recovery.

# /usr/sbin/mariadbd --help --verbose | grep innodb
...
  --innodb-force-recovery=# 
...

Again check whats the case on your system.

And last but not least: To start the daemon as “root” from the command line may need some special procedure as well. You have to check the documentation to see what applies for your database version.

I guess it is best you wait for someone who uses your database version to chime in .

1 Like

" Here it is."
Thank you.

It looks like /usr/sbin/mysqld is where the variables are called from on mariadb 10.2.31.

smooths:~ # /usr/sbin/mysqld --innodb-force-recovery=1 --user=root --password=
2024-06-27 15:40:29 139844408709696 [Note] /usr/sbin/mysqld (mysqld 10.2.31-MariaDB) starting as process 6257 …
Aborted (core dumped)
smooths:~ # /usr/sbin/mysqld --innodb-force-recovery=2 --user=root --password=
2024-06-27 15:41:14 139774235189824 [Note] /usr/sbin/mysqld (mysqld 10.2.31-MariaDB) starting as process 6300 …
Aborted (core dumped)

** Finally innodb-force-recovery is called correctly** Thank you.
But it looks like the calls to dump were aborted above.

smooths:~ # /usr/sbin/mysqld --innodb-force-recovery=3 --user=root --password=
2024-06-27 15:41:36 140211052962368 [Note] /usr/sbin/mysqld (mysqld 10.2.31-MariaDB) starting as process 6333 …
smooths:~ # /usr/sbin/mysqld --innodb-force-recovery=4 --user=root --password=
2024-06-27 15:42:15 139872985178688 [Note] /usr/sbin/mysqld (mysqld 10.2.31-MariaDB) starting as process 6359 …

Looks like calls were not aborted but the processes were killed anyway. Any idea what happened?

Well, you were supposed to read it and layout your posts as described …

What makes you think that the processes were killed (when using –innodb-force-recovery=3 or –innodb-force-recovery=4)?

Did you read this?

There it says:

… Until MariaDB 10.2.7, mode 0 was the only mode permitting changes to the data. From MariaDB 10.2.7, write transactions are permitted with mode 3 or less.

To recover the tables, you can execute SELECTs to dump data, and DROP TABLE (when write transactions are permitted) to remove corrupted tables.

I did and I am trying to do my best here accordingly. Please do point out what I missed and I will correct the situation. No problem.

What makes you think that the processes were killed (when using –innodb-force-recovery=3 or –innodb-force-recovery=4 )? Blockquote

Because the 3 and 4 ended with the following error:

smooths:~ # /usr/sbin/mysqld --innodb-force-recovery=3 --user=root
2024-06-27 16:10:47 139654714872384 [Note] /usr/sbin/mysqld (mysqld 10.2.31-MariaDB) starting as process 6703 …
Aborted (core dumped)Blockquote

I started with level 1 and took it to level 4. I didn’t try level 0 as I thought the default level might not be any good.

Thankyou so much @susejunky

Since the levels all ended up with the same error it became impossible to get into mysql to execute select, dump and drop commands. Maybe I will try mysqld statement from the command line later when innodb-force-recovery=3 (rerun) finishes.

When innodb-force-recovery=3 and innodb-force-recovery=4 were running, the websites using the databases were accessible (online) with no error which means the databases are still good. The only site not accessible is the site with the suspected corrupted table. But once the innodb-force-recovery finished running with the error I posted above, the websites were refused access to their databases thus the front side error “No access to database” showed up again. I suspect the corrupted table is still forcing innodb script to end without being able to repair and upload the table or give access to the databases.

At the moment “innodb-force-recovery=3” is running. The websites are good except for the one with the corrupt table.

Extracts form mysqld.log below:

2024-06-28 12:21:00 140310246339136 [Warning] InnoDB: Using innodb_file_format is deprecated and the parameter may be removed in future releases. See InnoDB File Format - MariaDB Knowledge Base
2024-06-28 12:21:00 140310246339136 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2024-06-28 12:21:00 140310246339136 [Note] InnoDB: Uses event mutexes
2024-06-28 12:21:00 140310246339136 [Note] InnoDB: Compressed tables use zlib 1.2.11
2024-06-28 12:21:00 140310246339136 [Note] InnoDB: Using Linux native AIO
2024-06-28 12:21:00 140310246339136 [Note] InnoDB: Number of pools: 1
2024-06-28 12:21:00 140310246339136 [Note] InnoDB: Using generic crc32 instructions
2024-06-28 12:21:00 140310246339136 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
2024-06-28 12:21:00 140310246339136 [Note] InnoDB: Completed initialization of buffer pool
2024-06-28 12:21:00 140309563488000 [Note] InnoDB: page_cleaner coordinator priority: -20
2024-06-28 12:21:00 140310246339136 [Note] InnoDB: Highest supported file format is Barracuda.
2024-06-28 12:21:00 140310246339136 [Note] InnoDB: Starting crash recovery from checkpoint LSN=90299629329
2024-06-28 12:21:00 140310246339136 [Note] InnoDB: Starting final batch to recover 467 pages from redo log.
2024-06-28 12:21:01 140310246339136 [Note] InnoDB: 128 out of 128 rollback segments are active.
2024-06-28 12:21:01 140310246339136 [Note] InnoDB: Removed temporary tablespace data file: “ibtmp1”
2024-06-28 12:21:01 140310246339136 [Note] InnoDB: Creating shared tablespace for temporary tables
2024-06-28 12:21:01 140310246339136 [Note] InnoDB: Setting file ‘./ibtmp1’ size to 12 MB. Physically writing the file full; Please wait …
2024-06-28 12:21:01 140310246339136 [Note] InnoDB: File ‘./ibtmp1’ size is now 12 MB.
2024-06-28 12:21:01 140310246339136 [Note] InnoDB: 5.7.29 started; log sequence number 90302777296
2024-06-28 12:21:01 140310246339136 [Note] InnoDB: !!! innodb_force_recovery is set to 3 !!!
2024-06-28 12:21:01 140309459625728 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2024-06-28 12:21:01 140309459625728 [Note] InnoDB: Buffer pool(s) load completed at 240628 12:21:01
2024-06-28 12:21:01 140310246339136 [Note] Plugin ‘FEEDBACK’ is disabled.
2024-06-28 12:21:01 140310246339136 [Note] Recovering after a crash using tc.log
2024-06-28 12:21:01 140310246339136 [Note] Starting crash recovery…
2024-06-28 12:21:01 140310246339136 [Note] Crash recovery finished.
2024-06-28 12:21:01 140310246339136 [Note] Server socket created on IP: ‘0.0.0.0’.
2024-06-28 12:21:01 140310246339136 [Note] Reading of all Master_info entries succeeded
2024-06-28 12:21:01 140310246339136 [Note] Added new Master_info ‘’ to hash table
2024-06-28 12:21:01 140310246339136 [Note] /usr/sbin/mysqld: ready for connections.
Version: ‘10.2.31-MariaDB’ socket: ‘/run/mysql/mysql.sock’ port: 3306 SUSE package Blockquote

I am afraid innodb-force-recovery=3 will throw error at the end if it still can’t force-repair the corrupted table.

Is there a way to force the script to stop running now and then try and start mysql normally so that I can drop the corrupt database?

Thank you

To begin with: I never had to do that sort of recovery to any of my databases. So the only thing I can do is to refer you to the documentation.

The second red box in the documentation says

Please note that recovery mode does not repair corruption.

My understanding of the documentation is:

  1. You start the database daemon (mariadbd/mysqld) with lets say --innodb-force-recovery=1.

  2. You start up mariadb/mysql and try to SELECT tables in order to dump their data or try to DROP faulty tables.

  3. If that does not work you stop the database daemon and restart it with the next higher value for innodb-force-recovery. Then you try 2.) again. In the end: The higher the value which will allow you to dump data (or drop tables) the less data you will be able to recover.

1 Like