Categories: Linux

InnoDB: Page directory corruption

Ketika server mengalami mati mendadak, adakalanya database yang ada di server mengalami corrupt. Tanda-tanda yang dimunculkan adalah mysql tidak dapat dijalankan, dan menjadi state failed. Ada petunjuk untuk menjalankan 2 perintah berikut

systemctl status mysql.service
journalctl -xe

Ketika menjalankan systemctl status mysql.service , hanya muncul status error terakhir. Untuk melihat status eror secara lengkap, jalankan journalctl -xe . Berikut adalah pesan error yang didapatkan

-- Unit mariadb.service has begun starting up.
Dec 05 15:17:43 your.hostname kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Dec 05 15:17:43 your.hostname kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Dec 05 15:17:43 your.hostname kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Dec 05 15:17:43 your.hostname mysqld[2597]: 2016-12-05 15:17:43 140560138741952 [Note] /usr/sbin/mysqld (mysqld 10.1.19-MariaDB) starting as process 2597 ...
Dec 05 15:17:43 your.hostname mysqld[2597]: 2016-12-05 15:17:43 140560138741952 [Note] InnoDB: Using mutexes to ref count buffer pool pages
Dec 05 15:17:43 your.hostname mysqld[2597]: 2016-12-05 15:17:43 140560138741952 [Note] InnoDB: The InnoDB memory heap is disabled
Dec 05 15:17:43 your.hostname mysqld[2597]: 2016-12-05 15:17:43 140560138741952 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Dec 05 15:17:43 your.hostname mysqld[2597]: 2016-12-05 15:17:43 140560138741952 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
Dec 05 15:17:43 your.hostname mysqld[2597]: 2016-12-05 15:17:43 140560138741952 [Note] InnoDB: Compressed tables use zlib 1.2.7
Dec 05 15:17:43 your.hostname mysqld[2597]: 2016-12-05 15:17:43 140560138741952 [Note] InnoDB: Using Linux native AIO
Dec 05 15:17:43 your.hostname mysqld[2597]: 2016-12-05 15:17:43 140560138741952 [Note] InnoDB: Using SSE crc32 instructions
Dec 05 15:17:43 your.hostname mysqld[2597]: 2016-12-05 15:17:43 140560138741952 [Note] InnoDB: Initializing buffer pool, size = 128.0M
Dec 05 15:17:43 your.hostname mysqld[2597]: 2016-12-05 15:17:43 140560138741952 [Note] InnoDB: Completed initialization of buffer pool
Dec 05 15:17:43 your.hostname mysqld[2597]: 2016-12-05 15:17:43 140560138741952 [Note] InnoDB: Highest supported file format is Barracuda.
Dec 05 15:17:43 your.hostname mysqld[2597]: 2016-12-05 15:17:43 140560138741952 [Note] InnoDB: The log sequence numbers 3235735 and 3235735 in ibdata files do not 
Dec 05 15:17:43 your.hostname mysqld[2597]: 2016-12-05 15:17:43 140560138741952 [Note] InnoDB: Database was not shutdown normally!
Dec 05 15:17:43 your.hostname mysqld[2597]: 2016-12-05 15:17:43 140560138741952 [Note] InnoDB: Starting crash recovery.
Dec 05 15:17:43 your.hostname mysqld[2597]: 2016-12-05 15:17:43 140560138741952 [Note] InnoDB: Reading tablespace information from the .ibd files...
Dec 05 15:17:43 your.hostname mysqld[2597]: 2016-12-05 15:17:43 140560138741952 [Note] InnoDB: Restoring possible half-written data pages
Dec 05 15:17:43 your.hostname mysqld[2597]: 2016-12-05 15:17:43 140560138741952 [Note] InnoDB: from the doublewrite buffer...
Dec 05 15:17:44 your.hostname mysqld[2597]: 2016-12-05 15:17:44 140560138741952 [Note] InnoDB: 128 rollback segment(s) are active.
Dec 05 15:17:44 your.hostname mysqld[2597]: 2016-12-05 15:17:44 140560138741952 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.32-79.0 started; log se
Dec 05 15:17:44 your.hostname mysqld[2597]: 2016-12-05 15:17:44 140560138741952 [Note] InnoDB: !!! innodb_force_recovery is set to 3 !!!
Dec 05 15:17:44 your.hostname mysqld[2597]: 2016-12-05 15:17:44 140559402379008 [Note] InnoDB: Dumping buffer pool(s) not yet started
Dec 05 15:17:44 your.hostname mysqld[2597]: 2016-12-05 15:17:44 140560138741952 [Note] Plugin 'FEEDBACK' is disabled.
Dec 05 15:17:44 your.hostname mysqld[2597]: 2016-12-05 15:17:44 140560138741952 [Note] Recovering after a crash using tc.log
Dec 05 15:17:44 your.hostname mysqld[2597]: 2016-12-05 15:17:44 140560138741952 [Note] Starting crash recovery...
Dec 05 15:17:44 your.hostname mysqld[2597]: 2016-12-05 15:17:44 140560138741952 [Note] Crash recovery finished.
Dec 05 15:17:44 your.hostname mysqld[2597]: 2016-12-05 15:17:44 140560138741952 [Note] Server socket created on IP: '::'.
Dec 05 15:17:44 your.hostname mysqld[2597]: InnoDB: Page directory corruption: infimum not pointed to
Dec 05 15:17:44 your.hostname mysqld[2597]: 2016-12-05 15:17:44 7fd6b51ceb00 InnoDB: Page dump in ascii and hex (16384 bytes):
Dec 05 15:17:44 your.hostname mysqld[2597]: len 16384; hex 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 0000000000000000; asc
Dec 05 15:17:44 your.hostname mysqld[2597]: ;
Dec 05 15:17:44 your.hostname mysqld[2597]: InnoDB: End of page dump
Dec 05 15:17:44 your.hostname mysqld[2597]: 2016-12-05 15:17:44 7fd6b51ceb00 InnoDB: uncompressed page, stored checksum in field1 0, calculated checksums for field
Dec 05 15:17:44 your.hostname mysqld[2597]: InnoDB: page type 0 meaning ALLOCATED
Dec 05 15:17:44 your.hostname mysqld[2597]: InnoDB: Page may be a freshly allocated page
Dec 05 15:17:44 your.hostname mysqld[2597]: InnoDB: Page directory corruption: supremum not pointed to
Dec 05 15:17:44 your.hostname mysqld[2597]: 2016-12-05 15:17:44 7fd6b51ceb00 InnoDB: Page dump in ascii and hex (16384 bytes):
Dec 05 15:17:44 your.hostname mysqld[2597]: len 16384; hex 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dec 05 15:17:44 your.hostname mysqld[2597]: 0000000000000000; asc
Dec 05 15:17:44 your.hostname mysqld[2597]: ;
Dec 05 15:17:44 your.hostname mysqld[2597]: InnoDB: End of page dump
Dec 05 15:17:44 your.hostname mysqld[2597]: 2016-12-05 15:17:44 7fd6b51ceb00 InnoDB: uncompressed page, stored checksum in field1 0, calculated checksums for field
Dec 05 15:17:44 your.hostname mysqld[2597]: InnoDB: page type 0 meaning ALLOCATED
Dec 05 15:17:44 your.hostname mysqld[2597]: InnoDB: Page may be a freshly allocated page
Dec 05 15:17:44 your.hostname mysqld[2597]: 161205 15:17:44 [ERROR] mysqld got signal 11 ;
Dec 05 15:17:44 your.hostname mysqld[2597]: This could be because you hit a bug. It is also possible that this binary
Dec 05 15:17:44 your.hostname mysqld[2597]: or one of the libraries it was linked against is corrupt, improperly built,
Dec 05 15:17:44 your.hostname mysqld[2597]: or misconfigured. This error can also be caused by malfunctioning hardware.
Dec 05 15:17:44 your.hostname mysqld[2597]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Dec 05 15:17:44 your.hostname mysqld[2597]: We will try our best to scrape up some info that will hopefully help
Dec 05 15:17:44 your.hostname mysqld[2597]: diagnose the problem, but since we have already crashed,
Dec 05 15:17:44 your.hostname mysqld[2597]: something is definitely wrong and this may fail.
Dec 05 15:17:44 your.hostname mysqld[2597]: Server version: 10.1.19-MariaDB
Dec 05 15:17:44 your.hostname mysqld[2597]: key_buffer_size=134217728
Dec 05 15:17:44 your.hostname mysqld[2597]: read_buffer_size=131072
Dec 05 15:17:44 your.hostname mysqld[2597]: max_used_connections=0
Dec 05 15:17:44 your.hostname mysqld[2597]: max_threads=153
Dec 05 15:17:44 your.hostname mysqld[2597]: thread_count=0
Dec 05 15:17:44 your.hostname mysqld[2597]: It is possible that mysqld could use up to
Dec 05 15:17:44 your.hostname mysqld[2597]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467121 K  bytes of memory
Dec 05 15:17:44 your.hostname mysqld[2597]: Hope that's ok; if not, decrease some variables in the equation.
Dec 05 15:17:44 your.hostname mysqld[2597]: Thread pointer: 0x0x7fd685c12008
Dec 05 15:17:44 your.hostname mysqld[2597]: Attempting backtrace. You can use the following information to find out
Dec 05 15:17:44 your.hostname mysqld[2597]: where mysqld died. If you see no messages after this, something went
Dec 05 15:17:44 your.hostname mysqld[2597]: terribly wrong...
Dec 05 15:17:44 your.hostname mysqld[2597]: stack_bottom = 0x7fd6b51ce0f0 thread_stack 0x48400
Dec 05 15:17:44 your.hostname mysqld[2597]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7fd6b5d0fcde]
Dec 05 15:17:44 your.hostname mysqld[2597]: /usr/sbin/mysqld(handle_fatal_signal+0x2d5)[0x7fd6b5833005]
Dec 05 15:17:44 your.hostname mysqld[2597]: /lib64/libpthread.so.0(+0xf100)[0x7fd6b4e4d100]
Dec 05 15:17:44 your.hostname mysqld[2597]: mysys/stacktrace.c:268(my_print_stacktrace)[0x7fd6b5a2080e]
Dec 05 15:17:44 your.hostname mysqld[2597]: include/mach0data.ic:172(mach_read_from_2)[0x7fd6b5ae0932]
Dec 05 15:17:44 your.hostname mysqld[2597]: include/btr0pcur.ic:487(btr_pcur_open_with_no_init_func)[0x7fd6b5a786a9]
Dec 05 15:17:44 your.hostname mysqld[2597]: row/row0sel.cc:1465(row_sel)[0x7fd6b5a790de]
Dec 05 15:17:44 your.hostname mysqld[2597]: row/row0sel.cc:2127(row_sel_step(que_thr_t*))[0x7fd6b5a7ba8b]
Dec 05 15:17:44 your.hostname mysqld[2597]: que/que0que.cc:1067(que_thr_step)[0x7fd6b5a3bbe8]
Dec 05 15:17:44 your.hostname mysqld[2597]: que/que0que.cc:1277(que_eval_sql(pars_info_t*, char const*, unsigned long, trx_t*))[0x7fd6b5a3c402]
Dec 05 15:17:44 your.hostname mysqld[2597]: dict/dict0stats.cc:3092(dict_stats_fetch_from_ps)[0x7fd6b5b5b981]
Dec 05 15:17:44 your.hostname mysqld[2597]: include/dict0stats.ic:174(dict_stats_init)[0x7fd6b59b8c52]
Dec 05 15:17:44 your.hostname mysqld[2597]: sql/handler.cc:2501(handler::ha_open(TABLE*, char const*, int, unsigned int))[0x7fd6b5837363]
Dec 05 15:17:44 your.hostname mysqld[2597]: sql/table.cc:2930(open_table_from_share(THD*, TABLE_SHARE*, char const*, unsigned int, unsigned int, unsigned int, TABL
Dec 05 15:17:44 your.hostname mysqld[2597]: sql/sql_base.cc:2554(open_table(THD*, TABLE_LIST*, Open_table_context*))[0x7fd6b5669f2c]
Dec 05 15:17:44 your.hostname mysqld[2597]: sql/sql_base.cc:4067(open_and_process_table)[0x7fd6b566e0b0]
Dec 05 15:17:44 your.hostname mysqld[2597]: sql/sql_base.cc:5258(open_and_lock_tables(THD*, DDL_options_st const&, TABLE_LIST*, bool, unsigned int, Prelocking_stra
Dec 05 15:17:44 your.hostname mysqld[2597]: sql/rpl_rli.cc:1473(rpl_load_gtid_slave_state(THD*))[0x7fd6b5777fb0]
Dec 05 15:17:44 your.hostname mysqld[2597]: sql/slave.cc:316(handle_slave_background)[0x7fd6b5630e6d]
Dec 05 15:17:44 your.hostname mysqld[2597]: /lib64/libpthread.so.0(+0x7dc5)[0x7fd6b4e45dc5]
Dec 05 15:17:44 your.hostname mysqld[2597]: /lib64/libc.so.6(clone+0x6d)[0x7fd6b3267ced]
Dec 05 15:17:44 your.hostname mysqld[2597]: Trying to get some variables.
Dec 05 15:17:44 your.hostname mysqld[2597]: Some pointers may be invalid and cause the dump to abort.
Dec 05 15:17:44 your.hostname mysqld[2597]: Query (0x0):
Dec 05 15:17:44 your.hostname mysqld[2597]: Connection ID (thread ID): 1
Dec 05 15:17:44 your.hostname mysqld[2597]: Status: NOT_KILLED
Dec 05 15:17:44 your.hostname mysqld[2597]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge
Dec 05 15:17:44 your.hostname mysqld[2597]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Dec 05 15:17:44 your.hostname mysqld[2597]: information that should help you find out what is causing the crash.
Dec 05 15:17:44 your.hostname systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
Dec 05 15:17:44 your.hostname systemd[1]: Failed to start MariaDB database server.

Untuk melakukan recovery, hal yang dapat kita lakukan adalah innodb force recovery. Untuk melakukannya, tambahkan innodb_force_recovery = 1 di bawah [mysqld] pada file /etc/my.cnf atau /etc/my.cnf.d/server.cnf menjadi seperti berikut

[mysqld]
innodb_force_recovery = 1

Mulailah dari nilai 1, dan cobalah systemctl start mysql . Jika masih belum bisa, ganti nilai menjadi 2, dan cobalah lagi menjalankan mysql. Jika tidak bisa juga, ganti lagi menjadi 3, dan seterusnya. Semakin tinggi nilainya, semakin besar kemungkinan data yang hilang. Nilai maksimal yang dimungkinkan adalah 6.

[1]

Akhmad Fakhoni Listiyan Dede

Share
Published by
Akhmad Fakhoni Listiyan Dede

Recent Posts

Mengimplementasikan AI untuk Melakukan Review pada Pull Request Github

Halo semua Github memiliki fitur terkait Pull Request yang cukup luar biasa, yaitu dapat melakukan…

8 months ago

Install kubernetes cluster using kubeadm

Hello everyone! 👋 Welcome to our step-by-step guide on using kubeadm to install Kubernetes. It's…

1 year ago

Meningkatkan Efisiensi Memory Proxmox Lebih dari 20%

Proxmox adalah hypervisor yang populer digunakan karena penggunaannya yang cukup simpel jika dibandingkan dengan hypervisor…

4 years ago

Longhorn Failed Upgrade from v0.8.1 to v1.0.0 caused by pv created before v0.6.2

This post is mirror of https://forums.rancher.com/t/failed-upgrade-from-v0-8-1-to-v1-0-0-caused-by-pv-created-before-v0-6-2/17586 I scale down all pods that has vpc to…

5 years ago

Cara Setup Network Ubuntu Server 18.04 pada Proxmox

Ubuntu Konfigurasi network pada Ubuntu Server 18.04 berbeda dengan ubuntu versi sebelumnya. Versi sebelumnya menggunakan…

5 years ago

VSCode Terminal Font Fix on MacOS

For detailed problems, see https://notulensiku.com/2019/02/fix-integrated-terminal-font-vscode/ To solve on mac, you need to install powerline fonts…

5 years ago