SlideShare a Scribd company logo
9
Most read
10
Most read
11
Most read
Analyze corefile and get backtraces


with GDB for MySQL/MariaDB on Linux
Nilnandan Joshi,


Sr. Support Engineer, MariaDB


nilnandan@gmail.com


www.nilinfobin.com
Core Dump/File:




A core dump is a disk file containing an image of the process’s memory at the time of
termination. This image can be used in a debugger to inspect the state of the program at
the time that it terminated. The core dump includes key pieces of program state as
processor registers, memory management details, and other processor and operating
system flags and information.So a core dump is a file that can be very useful to understand
the context of a crash. 


GDB:


GDB, the GNU Project debugger, allows you to see what is going on `inside' another
program while it executes -- or what another program was doing at the moment it crashed.


i.e


sudo gdb -p `pidof mysqld`


sudo gdb /path/to/mysqld /path/to/coredump


sudo gdb --batch --eval-command="thread apply all bt" /usr/sbin/mysqld $(pgrep -xn
mysqld) > mysqld_bt_all_threads.txt


sudo gdb --batch --eval-command="thread apply all bt full" /usr/sbin/mysqld $(pgrep -xn
mysqld) > mysqld_full_bt_all_threads.txt


You can use GDB to study:


• InnoDB locks, table locks, metadata locks etc


• server variables, user variables at session level


• Threads and Statements
Backtraces:


A backtrace is a summary of how your program got where it is. It shows one line per
frame, for many frames, starting with the currently executing frame (frame zero), followed
by its caller (frame one), and on up the stack.


In a multi-threaded program, GDB by default shows the backtrace only for the current
thread. To display the backtrace for several or all of the threads, use the command thread
apply. For example, if you type thread apply all backtrace(thread apply all bt) , GDB will
display the backtrace for all the threads; this is handy when you debug a core dump of a
multi-threaded program.


Each line in the backtrace shows the frame number and the function name. The program
counter value is also shown—unless you use set print address off. The backtrace also
shows the source file name and line number, as well as the arguments to the function. The
program counter value is omitted if it is at the beginning of the code for that line number.
How to enable Coredump:


In order to enable core dumps, you need to set the core_file system variable either on the
command-line or in a relevant server option group in an option file.


[mariadb]


...


core_file


There are some additional details related to using core files on Linux. On some systems there
is a limit on the sizes of core files that can be dumped. So you have to disable some of the
restrictions. To check the system's current system-wide limit, run


ulimit -c


for the current limit of the mysqld process specifically:


sudo cat /proc/$(pidof mysqld)/limits | grep "core file"


If you need to change the core size limit, the method you use depends on how you
start mysqld. i.e mysqld using mysqld_safe


[mysqld_safe]


…


core_file_size=unlimited
- mysqld using systemd


sudo tee /etc/systemd/system/mariadb.service.d/limitcore.conf <<EOF


[Service]


LimitCORE=infinity


EOF


sudo systemctl daemon-reload


If you are using Linux, then it can be helpful to change a few settings to alter where the core
files is written and what file name is used. This is done by setting
the kernel.core_pattern and kernel.core_uses_pid attributes.


By default, core file will be generated in MySQL/MariaDB datadir. If you are using a
production system, you probably want to have the core files in a specific directory, not in
the data directory. (Because core file size could be very large somewhere in GBs) so for that
you can set something like,


sudo mkdir /tmp/corefiles


sudo chmod 777 /tmp/corefiles


sudo echo /tmp/corefiles/core > /proc/sys/kernel/core_pattern


sudo echo 1 > /proc/sys/kernel/core_uses_pid


Since  mysqld  executes  setuid, you may have to set  fs.suid_dumpable=2  to allow core
dumps on Linux. 


sudo echo 2 > /proc/sys/fs/suid_dumpable
In MariaDB 10.1.35, MariaDB 10.2.17, and MariaDB 10.3.9 and later, core_file has also
been made into a system variable. Previously it was just an option. It's value can be
checked at runtime by executing the following:


SHOW GLOBAL VARIABLES LIKE 'core_file';


However, in MariaDB 10.3.7 and later, some large buffers have been excluded from core
files on some systems as a way to reduce the size. The following buffers are excluded:


InnoDB buffer pool


InnoDB log buffer


Server recv buffer


Query cache


https://mariadb.com/kb/en/enabling-core-dumps/


https://dev.mysql.com/doc/refman/8.0/en/server-options.html#option_mysqld_core-file


https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
Other Requirements:
1. GDB need to be installed if it's not already


yum install gdb


2. debug info packages needs to be install for server and common atleast.


i.e


MariaDB-server-debuginfo-10.4.17_10-1.el8.x86_64


MariaDB-common-debuginfo-10.4.17_10-1.el8.x86_64


3. If you can’t run gdb + core dump on customer’s server then you must have the same
mysqld/mariadb binaries including debug-info packages + all dynamically linked libraries +
core file to run gdb in your local machine. Libraries can be checked by ldd command. i.e


[root@nilcentos7 ~]# ldd /usr/sbin/mysqld


	
linux-vdso.so.1 => (0x00007ffea8ddd000)


	
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f2149b93000)


	
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f214995c000)


	
liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f2149736000)


…


	
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f2148810000)


	
libdl.so.2 => /lib64/libdl.so.2 (0x00007f214860c000)
Problem:
190328 23:31:08 [ERROR] mysqld got signal 6 ;


...


Server version: 10.2.23-MariaDB-10.2.23+maria~stretch


...


Thread pointer: 0x7ff4d8001f28


Attempting backtrace. You can use the following information to find out


where mysqld died. If you see no messages after this, something went


terribly wrong...


stack_bottom = 0x7ff4dc62ccc8 thread_stack 0x49000


*** buffer overflow detected ***: /usr/sbin/mysqld terminated


======= Backtrace: =========


/lib/x86_64-linux-gnu/libc.so.6(+0x70bfb)[0x7ffa09af5bfb]


/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7ffa09b7e437]


/lib/x86_64-linux-gnu/libc.so.6(+0xf7570)[0x7ffa09b7c570]


/lib/x86_64-linux-gnu/libc.so.6(+0xf93aa)[0x7ffa09b7e3aa]


/usr/sbin/mysqld(my_addr_resolve+0xe2)[0x55ca42284922]


…


/usr/sbin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0x72)
[0x55ca41b4b992]


/usr/sbin/
mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15
enum_duplicatesb+0x1206)[0x55ca41b560f6]


/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3f68)[0x55ca41b6bee8]


/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x28a)
[0x55ca41b70e4a]


/usr/sbin/mysqld(+0x4c864f)[0x55ca41b7164f]


...
Problem:
2021-02-12 8:39:33 11039259 [ERROR] mysqld: Table '(temporary)' is
marked as crashed and should be repaired


2021-02-12 8:41:09 10757048 [ERROR] mysqld: Table '(temporary)' is
marked as crashed and should be repaired


free(): invalid next size (normal)


210212 8:41:09 [ERROR] mysqld got signal 6 ;


...


Server version: 10.4.17-10-MariaDB-enterprise-log


...


Thread pointer: 0x7f179d1103c8


Attempting backtrace. You can use the following information to
find out


where mysqld died. If you see no messages after this, something
went


terribly wrong...


stack_bottom = 0x7f0a020edbe8 thread_stack 0x49000
[root@centos8 nil]# gdb mysqld core.2751610


GNU gdb (GDB) Red Hat Enterprise Linux 8.2-12.el8


Copyright (C) 2018 Free Software Foundation, Inc.


License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/
gpl.html>


…


Type "apropos word" to search for commands related to "word"...


Reading symbols from mysqld...Reading symbols from /usr/lib/debug/
usr/sbin/mysqld-10.4.17_10-1.el8.x86_64.debug...done.


done.


[New LWP 2338279]


[New LWP 2751610]


…


…


Use the "info sharedlibrary" command to see the complete listing.


Do you need "set solib-search-path" or "set sysroot"?


[Thread debugging using libthread_db enabled]


Using host libthread_db library "/lib64/libthread_db.so.1".


…


Core was generated by `/usr/sbin/mysqld'.


Program terminated with signal SIGSEGV, Segmentation fault.


#0 0x00007f1908650de3 in __memmove_avx_unaligned_erms () from /
lib64/libc.so.6


[Current thread is 1 (Thread 0x7f0b74ee2700 (LWP 2338279))]
(gdb)


(gdb) thread 1 — this is for the crash


[Switching to thread 1 (Thread 0x7f0b74ee2700 (LWP 2338279))]


#0 0x00007f1908650de3 in __memmove_avx_unaligned_erms () from /
lib64/libc.so.6


(gdb) backtrace — to get the backtrace


#0 0x00007f1908650de3 in __memmove_avx_unaligned_erms () from /
lib64/libc.so.6


#1 0x0000563bd293aa84 in memcpy (__len=<optimized out>,
__src=<optimized out>, __dest=<optimized out>) at /usr/include/
bits/string_fortified.h:34


#2 write_block_record (info=info@entry=0x7f0aa8635b18,
old_record=old_record@entry=0x0, record=record@entry=0x7f0aa886ddd0
"347377003ngs",


row=row@entry=0x7f0aa8635bb0,
bitmap_blocks=bitmap_blocks@entry=0x7f0aa8635bb0,
head_block_is_read=<optimized out>, row_pos=0x7f0b74edbb30,
undo_lsn=1,


old_record_checksum=0) at /usr/src/debug/MariaDB-10.4.17-10/
src_0/storage/maria/ma_blockrec.c:2750
#3 0x0000563bd293c9f7 in allocate_and_write_block_record
(undo_lsn=1, row=0x7f0aa8635bb0, record=0x7f0aa886ddd0
"347377003ngs", info=0x7f0aa8635b18)


at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/maria/
ma_blockrec.c:3571


#4 _ma_write_init_block_record (info=0x7f0aa8635b18,
record=0x7f0aa886ddd0 "347377003ngs")


at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/maria/
ma_blockrec.c:3611


#5 0x0000563bd2947a81 in maria_write (info=0x7f0aa8635b18,
record=0x7f0aa886ddd0 "347377003ngs")


at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/maria/
ma_write.c:157


#6 0x0000563bd237b631 in handler::ha_write_tmp_row
(this=0x7f0aa8800e00, buf=0x7f0aa886ddd0 "347377003ngs")


at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_class.h:6645


#7 0x0000563bd2386134 in schema_table_store_record
(thd=0x7f0aa87efb18, table=0x7f0aa8268550)


at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_show.cc:3929


#8 0x0000563bd238ad2e in get_schema_column_record(THD*,
TABLE_LIST*, TABLE*, bool, st_mysql_const_lex_string const*,
st_mysql_const_lex_string const*) ()


at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_show.cc:6113
#9 0x0000563bd2391e04 in get_all_tables(THD*, TABLE_LIST*, Item*)
() at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_show.cc:5066


#10 0x0000563bd23930e4 in get_schema_tables_result(JOIN*,
enum_schema_table_state) () at /usr/src/debug/MariaDB-10.4.17-10/
src_0/sql/sql_show.cc:8919


#11 0x0000563bd2378bb7 in JOIN::exec_inner() () at /usr/src/debug/
MariaDB-10.4.17-10/src_0/sql/sql_select.cc:4438


#12 0x0000563bd2379113 in JOIN::exec (this=0x7f0aa81f9ee8) at /usr/
src/debug/MariaDB-10.4.17-10/src_0/sql/sql_select.cc:4264


#13 0x0000563bd2377541 in mysql_select(THD*, TABLE_LIST*, unsigned
int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*,
st_order*, unsigned long long, select_result*, st_select_lex_unit*,
st_select_lex*) () at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/
sql_select.cc:4699


#14 0x0000563bd2377e1e in handle_select (thd=0x7f0aa87efb18,
lex=0x7f0aa87f37d0, result=0x7f0aa81f9ec0,
setup_tables_done_option=0)


at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/
sql_select.cc:410


#15 0x0000563bd230f41c in execute_sqlcom_select(THD*, TABLE_LIST*)
() at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:6429


#16 0x0000563bd23177b3 in mysql_execute_command(THD*) () at /usr/
src/debug/MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:3926
#17 0x0000563bd231e74a in mysql_parse (thd=0x7f0aa87efb18,
rawbuf=<optimized out>, length=111, parser_state=0x7f0b74ee1310,
is_com_multi=<optimized out>,


is_next_command=<optimized out>) at /usr/src/debug/
MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:7967


#18 0x0000563bd2320bf1 in dispatch_command(enum_server_command,
THD*, char*, unsigned int, bool, bool) ()


at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_class.h:1179


#19 0x0000563bd23221f4 in do_command (thd=0x7f0aa87efb18) at /usr/
src/debug/MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:1357


#20 0x0000563bd2407973 in do_handle_one_connection
(connect=connect@entry=0x563c11465fe8) at /usr/src/debug/
MariaDB-10.4.17-10/src_0/sql/sql_connect.cc:1412


#21 0x0000563bd2407a6d in handle_one_connection
(arg=arg@entry=0x563c11465fe8) at /usr/src/debug/
MariaDB-10.4.17-10/src_0/sql/sql_connect.cc:1316


#22 0x0000563bd29f430b in pfs_spawn_thread (arg=0x563c1175fcd8)
at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/perfschema/
pfs.cc:1869


#23 0x00007f190a78c14a in start_thread () from /lib64/
libpthread.so.0


#24 0x00007f19085eff23 in clone () from /lib64/libc.so.6
(gdb) frame 17 — this is to know more details in that frame(line)
^^


#17 0x0000563bd231e74a in mysql_parse (thd=0x7f0aa87efb18,
rawbuf=<optimized out>, length=111, parser_state=0x7f0b74ee1310,
is_com_multi=<optimized out>,


is_next_command=<optimized out>) at /usr/src/debug/
MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:7967


7967
	
error= mysql_execute_command(thd);


(gdb) print thd->query_string


$1 = {string = {str = 0x7f0aa81f6520 "SELECT COLUMN_NAME FROM
INFORMATION_SCHEMA.Columns where TABLE_NAME = 'my_workflow' and
TABLE_SCHEMA='news'",


length = 111}, cs = 0x563bd342ad20 <my_charset_latin1>}


(gdb)
Some important links for the blog posts


by my friend and colleague Valerii Kravchuk


GDB basics for MySQL DB




https://www.slideshare.net/valeriikravchuk1/gdb-basics-for-my-sql-db-as-percona-live-europe-2019


Checking user threads with GDB


http://mysqlentomologist.blogspot.com/2021/01/checking-user-threads-and-temporary.html


http://mysqlentomologist.blogspot.com/2018/03/checking-user-threads-with-gdb-in-mysql.html


http://mysqlentomologist.blogspot.com/2017/07/how-to-find-processlist-thread-id-in-gdb.html


How to explore InnoDB locks with GDB


http://mysqlentomologist.blogspot.com/2015/03/using-gdb-to-understand-what-locks-and_31.html


http://mysqlentomologist.blogspot.com/2015/04/using-gdb-to-understand-what-locks-and.html
Thank you


Any Questions?
Ad

Recommended

MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11
Kenny Gryp
 
Differences between MariaDB 10.3 & MySQL 8.0
Differences between MariaDB 10.3 & MySQL 8.0
Colin Charles
 
Oracle Database on Docker - Best Practices
Oracle Database on Docker - Best Practices
gvenzl
 
Learning postgresql
Learning postgresql
DAVID RAUDALES
 
Android internals By Rajesh Khetan
Android internals By Rajesh Khetan
Rajesh Khetan
 
Lessons Learned: Troubleshooting Replication
Lessons Learned: Troubleshooting Replication
Sveta Smirnova
 
Oracle12c data guard farsync and whats new
Oracle12c data guard farsync and whats new
Nassyam Basha
 
Primeiros passos com a API do Zabbix
Primeiros passos com a API do Zabbix
Janssen Lima
 
Oracle Active Data Guard: Best Practices and New Features Deep Dive
Oracle Active Data Guard: Best Practices and New Features Deep Dive
Glen Hawkins
 
YOW2020 Linux Systems Performance
YOW2020 Linux Systems Performance
Brendan Gregg
 
Load Balancing MySQL with HAProxy - Slides
Load Balancing MySQL with HAProxy - Slides
Severalnines
 
LISA2019 Linux Systems Performance
LISA2019 Linux Systems Performance
Brendan Gregg
 
Hadoop Meetup Jan 2019 - Dynamometer and a Case Study in NameNode GC
Hadoop Meetup Jan 2019 - Dynamometer and a Case Study in NameNode GC
Erik Krogen
 
Deep dive into PostgreSQL statistics.
Deep dive into PostgreSQL statistics.
Alexey Lesovsky
 
Achieving High Availability in PostgreSQL
Achieving High Availability in PostgreSQL
Mydbops
 
MAA Best Practices for Oracle Database 19c
MAA Best Practices for Oracle Database 19c
Markus Michalewicz
 
MySQL InnoDB Cluster - A complete High Availability solution for MySQL
MySQL InnoDB Cluster - A complete High Availability solution for MySQL
Olivier DASINI
 
MySQL Group Replication - Ready For Production? (2018-04)
MySQL Group Replication - Ready For Production? (2018-04)
Kenny Gryp
 
Comparison of ACFS and DBFS
Comparison of ACFS and DBFS
DanielHillinger
 
eBPF Perf Tools 2019
eBPF Perf Tools 2019
Brendan Gregg
 
PostgreSQL Performance Tuning
PostgreSQL Performance Tuning
elliando dias
 
Alta disponibilidade com PostgreSQL
Alta disponibilidade com PostgreSQL
Leonardo Cezar
 
Linux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old Secrets
Brendan Gregg
 
The Oracle RAC Family of Solutions - Presentation
The Oracle RAC Family of Solutions - Presentation
Markus Michalewicz
 
OWASP AppSecCali 2015 - Marshalling Pickles
OWASP AppSecCali 2015 - Marshalling Pickles
Christopher Frohoff
 
Step by Step to Install oracle grid 11.2.0.3 on solaris 11.1
Step by Step to Install oracle grid 11.2.0.3 on solaris 11.1
Osama Mustafa
 
Docker storage drivers by Jérôme Petazzoni
Docker storage drivers by Jérôme Petazzoni
Docker, Inc.
 
Tuning Android for low RAM
Tuning Android for low RAM
Chris Simmonds
 
Gdb basics for my sql db as (percona live europe 2019)
Gdb basics for my sql db as (percona live europe 2019)
Valerii Kravchuk
 
Gdb basics for my sql db as (openfest 2017) final
Gdb basics for my sql db as (openfest 2017) final
Valeriy Kravchuk
 

More Related Content

What's hot (20)

Oracle Active Data Guard: Best Practices and New Features Deep Dive
Oracle Active Data Guard: Best Practices and New Features Deep Dive
Glen Hawkins
 
YOW2020 Linux Systems Performance
YOW2020 Linux Systems Performance
Brendan Gregg
 
Load Balancing MySQL with HAProxy - Slides
Load Balancing MySQL with HAProxy - Slides
Severalnines
 
LISA2019 Linux Systems Performance
LISA2019 Linux Systems Performance
Brendan Gregg
 
Hadoop Meetup Jan 2019 - Dynamometer and a Case Study in NameNode GC
Hadoop Meetup Jan 2019 - Dynamometer and a Case Study in NameNode GC
Erik Krogen
 
Deep dive into PostgreSQL statistics.
Deep dive into PostgreSQL statistics.
Alexey Lesovsky
 
Achieving High Availability in PostgreSQL
Achieving High Availability in PostgreSQL
Mydbops
 
MAA Best Practices for Oracle Database 19c
MAA Best Practices for Oracle Database 19c
Markus Michalewicz
 
MySQL InnoDB Cluster - A complete High Availability solution for MySQL
MySQL InnoDB Cluster - A complete High Availability solution for MySQL
Olivier DASINI
 
MySQL Group Replication - Ready For Production? (2018-04)
MySQL Group Replication - Ready For Production? (2018-04)
Kenny Gryp
 
Comparison of ACFS and DBFS
Comparison of ACFS and DBFS
DanielHillinger
 
eBPF Perf Tools 2019
eBPF Perf Tools 2019
Brendan Gregg
 
PostgreSQL Performance Tuning
PostgreSQL Performance Tuning
elliando dias
 
Alta disponibilidade com PostgreSQL
Alta disponibilidade com PostgreSQL
Leonardo Cezar
 
Linux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old Secrets
Brendan Gregg
 
The Oracle RAC Family of Solutions - Presentation
The Oracle RAC Family of Solutions - Presentation
Markus Michalewicz
 
OWASP AppSecCali 2015 - Marshalling Pickles
OWASP AppSecCali 2015 - Marshalling Pickles
Christopher Frohoff
 
Step by Step to Install oracle grid 11.2.0.3 on solaris 11.1
Step by Step to Install oracle grid 11.2.0.3 on solaris 11.1
Osama Mustafa
 
Docker storage drivers by Jérôme Petazzoni
Docker storage drivers by Jérôme Petazzoni
Docker, Inc.
 
Tuning Android for low RAM
Tuning Android for low RAM
Chris Simmonds
 
Oracle Active Data Guard: Best Practices and New Features Deep Dive
Oracle Active Data Guard: Best Practices and New Features Deep Dive
Glen Hawkins
 
YOW2020 Linux Systems Performance
YOW2020 Linux Systems Performance
Brendan Gregg
 
Load Balancing MySQL with HAProxy - Slides
Load Balancing MySQL with HAProxy - Slides
Severalnines
 
LISA2019 Linux Systems Performance
LISA2019 Linux Systems Performance
Brendan Gregg
 
Hadoop Meetup Jan 2019 - Dynamometer and a Case Study in NameNode GC
Hadoop Meetup Jan 2019 - Dynamometer and a Case Study in NameNode GC
Erik Krogen
 
Deep dive into PostgreSQL statistics.
Deep dive into PostgreSQL statistics.
Alexey Lesovsky
 
Achieving High Availability in PostgreSQL
Achieving High Availability in PostgreSQL
Mydbops
 
MAA Best Practices for Oracle Database 19c
MAA Best Practices for Oracle Database 19c
Markus Michalewicz
 
MySQL InnoDB Cluster - A complete High Availability solution for MySQL
MySQL InnoDB Cluster - A complete High Availability solution for MySQL
Olivier DASINI
 
MySQL Group Replication - Ready For Production? (2018-04)
MySQL Group Replication - Ready For Production? (2018-04)
Kenny Gryp
 
Comparison of ACFS and DBFS
Comparison of ACFS and DBFS
DanielHillinger
 
eBPF Perf Tools 2019
eBPF Perf Tools 2019
Brendan Gregg
 
PostgreSQL Performance Tuning
PostgreSQL Performance Tuning
elliando dias
 
Alta disponibilidade com PostgreSQL
Alta disponibilidade com PostgreSQL
Leonardo Cezar
 
Linux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old Secrets
Brendan Gregg
 
The Oracle RAC Family of Solutions - Presentation
The Oracle RAC Family of Solutions - Presentation
Markus Michalewicz
 
OWASP AppSecCali 2015 - Marshalling Pickles
OWASP AppSecCali 2015 - Marshalling Pickles
Christopher Frohoff
 
Step by Step to Install oracle grid 11.2.0.3 on solaris 11.1
Step by Step to Install oracle grid 11.2.0.3 on solaris 11.1
Osama Mustafa
 
Docker storage drivers by Jérôme Petazzoni
Docker storage drivers by Jérôme Petazzoni
Docker, Inc.
 
Tuning Android for low RAM
Tuning Android for low RAM
Chris Simmonds
 

Similar to Analyze corefile and backtraces with GDB for Mysql/MariaDB on Linux - Nilandan Joshi (MariaDB) (20)

Gdb basics for my sql db as (percona live europe 2019)
Gdb basics for my sql db as (percona live europe 2019)
Valerii Kravchuk
 
Gdb basics for my sql db as (openfest 2017) final
Gdb basics for my sql db as (openfest 2017) final
Valeriy Kravchuk
 
FOSDEM 2015: gdb tips and tricks for MySQL DBAs
FOSDEM 2015: gdb tips and tricks for MySQL DBAs
Valerii Kravchuk
 
More on gdb for my sql db as (fosdem 2016)
More on gdb for my sql db as (fosdem 2016)
Valeriy Kravchuk
 
Troubleshooting MySQL from a MySQL Developer Perspective
Troubleshooting MySQL from a MySQL Developer Perspective
Marcelo Altmann
 
Linux /proc filesystem for MySQL DBAs - FOSDEM 2021
Linux /proc filesystem for MySQL DBAs - FOSDEM 2021
Valeriy Kravchuk
 
MySQL Utilities -- Cool Tools For You: PHP World Nov 16 2016
MySQL Utilities -- Cool Tools For You: PHP World Nov 16 2016
Dave Stokes
 
How to migrate from MySQL to MariaDB without tears
How to migrate from MySQL to MariaDB without tears
Sveta Smirnova
 
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...
Valeriy Kravchuk
 
MySQL Server Backup, Restoration, and Disaster Recovery Planning
MySQL Server Backup, Restoration, and Disaster Recovery Planning
Lenz Grimmer
 
MySQL Backup and Security Best Practices
MySQL Backup and Security Best Practices
Lenz Grimmer
 
MySQL Server Backup, Restoration, And Disaster Recovery Planning Presentation
MySQL Server Backup, Restoration, And Disaster Recovery Planning Presentation
Colin Charles
 
Profiling of Oracle Function Calls
Profiling of Oracle Function Calls
Enkitec
 
Blazing Performance with Flame Graphs
Blazing Performance with Flame Graphs
Brendan Gregg
 
MySQL Utilities -- PyTexas 2015
MySQL Utilities -- PyTexas 2015
Dave Stokes
 
MariaDB: in-depth (hands on training in Seoul)
MariaDB: in-depth (hands on training in Seoul)
Colin Charles
 
Libmysqld Introduction
Libmysqld Introduction
papablues
 
Collaborate 2012 - Administering MySQL for Oracle DBAs
Collaborate 2012 - Administering MySQL for Oracle DBAs
Nelson Calero
 
E bpf and dynamic tracing for mariadb db as (mariadb day during fosdem 2020)
E bpf and dynamic tracing for mariadb db as (mariadb day during fosdem 2020)
Valeriy Kravchuk
 
Advanced Debugging with GDB
Advanced Debugging with GDB
David Khosid
 
Gdb basics for my sql db as (percona live europe 2019)
Gdb basics for my sql db as (percona live europe 2019)
Valerii Kravchuk
 
Gdb basics for my sql db as (openfest 2017) final
Gdb basics for my sql db as (openfest 2017) final
Valeriy Kravchuk
 
FOSDEM 2015: gdb tips and tricks for MySQL DBAs
FOSDEM 2015: gdb tips and tricks for MySQL DBAs
Valerii Kravchuk
 
More on gdb for my sql db as (fosdem 2016)
More on gdb for my sql db as (fosdem 2016)
Valeriy Kravchuk
 
Troubleshooting MySQL from a MySQL Developer Perspective
Troubleshooting MySQL from a MySQL Developer Perspective
Marcelo Altmann
 
Linux /proc filesystem for MySQL DBAs - FOSDEM 2021
Linux /proc filesystem for MySQL DBAs - FOSDEM 2021
Valeriy Kravchuk
 
MySQL Utilities -- Cool Tools For You: PHP World Nov 16 2016
MySQL Utilities -- Cool Tools For You: PHP World Nov 16 2016
Dave Stokes
 
How to migrate from MySQL to MariaDB without tears
How to migrate from MySQL to MariaDB without tears
Sveta Smirnova
 
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...
Valeriy Kravchuk
 
MySQL Server Backup, Restoration, and Disaster Recovery Planning
MySQL Server Backup, Restoration, and Disaster Recovery Planning
Lenz Grimmer
 
MySQL Backup and Security Best Practices
MySQL Backup and Security Best Practices
Lenz Grimmer
 
MySQL Server Backup, Restoration, And Disaster Recovery Planning Presentation
MySQL Server Backup, Restoration, And Disaster Recovery Planning Presentation
Colin Charles
 
Profiling of Oracle Function Calls
Profiling of Oracle Function Calls
Enkitec
 
Blazing Performance with Flame Graphs
Blazing Performance with Flame Graphs
Brendan Gregg
 
MySQL Utilities -- PyTexas 2015
MySQL Utilities -- PyTexas 2015
Dave Stokes
 
MariaDB: in-depth (hands on training in Seoul)
MariaDB: in-depth (hands on training in Seoul)
Colin Charles
 
Libmysqld Introduction
Libmysqld Introduction
papablues
 
Collaborate 2012 - Administering MySQL for Oracle DBAs
Collaborate 2012 - Administering MySQL for Oracle DBAs
Nelson Calero
 
E bpf and dynamic tracing for mariadb db as (mariadb day during fosdem 2020)
E bpf and dynamic tracing for mariadb db as (mariadb day during fosdem 2020)
Valeriy Kravchuk
 
Advanced Debugging with GDB
Advanced Debugging with GDB
David Khosid
 
Ad

More from Mydbops (20)

Scaling TiDB for Large-Scale Application
Scaling TiDB for Large-Scale Application
Mydbops
 
AWS MySQL Showdown - RDS vs RDS Multi AZ vs Aurora vs Serverless - Mydbops...
AWS MySQL Showdown - RDS vs RDS Multi AZ vs Aurora vs Serverless - Mydbops...
Mydbops
 
Mastering Vector Search with MongoDB Atlas - Manosh Malai - Mydbops MyWebinar 39
Mastering Vector Search with MongoDB Atlas - Manosh Malai - Mydbops MyWebinar 39
Mydbops
 
Migration Journey To TiDB - Kabilesh PR - Mydbops MyWebinar 38
Migration Journey To TiDB - Kabilesh PR - Mydbops MyWebinar 38
Mydbops
 
AWS Blue Green Deployment for Databases - Mydbops
AWS Blue Green Deployment for Databases - Mydbops
Mydbops
 
What's New In MySQL 8.4 LTS Mydbops MyWebinar Edition 36
What's New In MySQL 8.4 LTS Mydbops MyWebinar Edition 36
Mydbops
 
What's New in PostgreSQL 17? - Mydbops MyWebinar Edition 35
What's New in PostgreSQL 17? - Mydbops MyWebinar Edition 35
Mydbops
 
What's New in MongoDB 8.0 - Mydbops MyWebinar Edition 34
What's New in MongoDB 8.0 - Mydbops MyWebinar Edition 34
Mydbops
 
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Mydbops
 
Read/Write Splitting using MySQL Router - Mydbops Meetup16
Read/Write Splitting using MySQL Router - Mydbops Meetup16
Mydbops
 
TiDB - From Data to Discovery: Exploring the Intersection of Distributed Dat...
TiDB - From Data to Discovery: Exploring the Intersection of Distributed Dat...
Mydbops
 
MySQL InnoDB Storage Engine: Deep Dive - Mydbops
MySQL InnoDB Storage Engine: Deep Dive - Mydbops
Mydbops
 
Demystifying Real time Analytics with TiDB
Demystifying Real time Analytics with TiDB
Mydbops
 
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during Migration
Mydbops
 
Efficient MySQL Indexing and what's new in MySQL Explain
Efficient MySQL Indexing and what's new in MySQL Explain
Mydbops
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL Router
Mydbops
 
PostgreSQL Schema Changes with pg-osc - Mydbops @ PGConf India 2024
PostgreSQL Schema Changes with pg-osc - Mydbops @ PGConf India 2024
Mydbops
 
Choosing the Right Database: Exploring MySQL Alternatives for Modern Applicat...
Choosing the Right Database: Exploring MySQL Alternatives for Modern Applicat...
Mydbops
 
Mastering Aurora PostgreSQL Clusters for Disaster Recovery
Mastering Aurora PostgreSQL Clusters for Disaster Recovery
Mydbops
 
Navigating Transactions: ACID Complexity in Modern Databases- Mydbops Open So...
Navigating Transactions: ACID Complexity in Modern Databases- Mydbops Open So...
Mydbops
 
Scaling TiDB for Large-Scale Application
Scaling TiDB for Large-Scale Application
Mydbops
 
AWS MySQL Showdown - RDS vs RDS Multi AZ vs Aurora vs Serverless - Mydbops...
AWS MySQL Showdown - RDS vs RDS Multi AZ vs Aurora vs Serverless - Mydbops...
Mydbops
 
Mastering Vector Search with MongoDB Atlas - Manosh Malai - Mydbops MyWebinar 39
Mastering Vector Search with MongoDB Atlas - Manosh Malai - Mydbops MyWebinar 39
Mydbops
 
Migration Journey To TiDB - Kabilesh PR - Mydbops MyWebinar 38
Migration Journey To TiDB - Kabilesh PR - Mydbops MyWebinar 38
Mydbops
 
AWS Blue Green Deployment for Databases - Mydbops
AWS Blue Green Deployment for Databases - Mydbops
Mydbops
 
What's New In MySQL 8.4 LTS Mydbops MyWebinar Edition 36
What's New In MySQL 8.4 LTS Mydbops MyWebinar Edition 36
Mydbops
 
What's New in PostgreSQL 17? - Mydbops MyWebinar Edition 35
What's New in PostgreSQL 17? - Mydbops MyWebinar Edition 35
Mydbops
 
What's New in MongoDB 8.0 - Mydbops MyWebinar Edition 34
What's New in MongoDB 8.0 - Mydbops MyWebinar Edition 34
Mydbops
 
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Mydbops
 
Read/Write Splitting using MySQL Router - Mydbops Meetup16
Read/Write Splitting using MySQL Router - Mydbops Meetup16
Mydbops
 
TiDB - From Data to Discovery: Exploring the Intersection of Distributed Dat...
TiDB - From Data to Discovery: Exploring the Intersection of Distributed Dat...
Mydbops
 
MySQL InnoDB Storage Engine: Deep Dive - Mydbops
MySQL InnoDB Storage Engine: Deep Dive - Mydbops
Mydbops
 
Demystifying Real time Analytics with TiDB
Demystifying Real time Analytics with TiDB
Mydbops
 
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during Migration
Mydbops
 
Efficient MySQL Indexing and what's new in MySQL Explain
Efficient MySQL Indexing and what's new in MySQL Explain
Mydbops
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL Router
Mydbops
 
PostgreSQL Schema Changes with pg-osc - Mydbops @ PGConf India 2024
PostgreSQL Schema Changes with pg-osc - Mydbops @ PGConf India 2024
Mydbops
 
Choosing the Right Database: Exploring MySQL Alternatives for Modern Applicat...
Choosing the Right Database: Exploring MySQL Alternatives for Modern Applicat...
Mydbops
 
Mastering Aurora PostgreSQL Clusters for Disaster Recovery
Mastering Aurora PostgreSQL Clusters for Disaster Recovery
Mydbops
 
Navigating Transactions: ACID Complexity in Modern Databases- Mydbops Open So...
Navigating Transactions: ACID Complexity in Modern Databases- Mydbops Open So...
Mydbops
 
Ad

Recently uploaded (20)

9-1-1 Addressing: End-to-End Automation Using FME
9-1-1 Addressing: End-to-End Automation Using FME
Safe Software
 
ReSTIR [DI]: Spatiotemporal reservoir resampling for real-time ray tracing ...
ReSTIR [DI]: Spatiotemporal reservoir resampling for real-time ray tracing ...
revolcs10
 
" How to survive with 1 billion vectors and not sell a kidney: our low-cost c...
" How to survive with 1 billion vectors and not sell a kidney: our low-cost c...
Fwdays
 
You are not excused! How to avoid security blind spots on the way to production
You are not excused! How to avoid security blind spots on the way to production
Michele Leroux Bustamante
 
The Future of Technology: 2025-2125 by Saikat Basu.pdf
The Future of Technology: 2025-2125 by Saikat Basu.pdf
Saikat Basu
 
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
caoyixuan2019
 
OpenPOWER Foundation & Open-Source Core Innovations
OpenPOWER Foundation & Open-Source Core Innovations
IBM
 
Enhance GitHub Copilot using MCP - Enterprise version.pdf
Enhance GitHub Copilot using MCP - Enterprise version.pdf
Nilesh Gule
 
Quantum AI: Where Impossible Becomes Probable
Quantum AI: Where Impossible Becomes Probable
Saikat Basu
 
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
Edge AI and Vision Alliance
 
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Priyanka Aash
 
Securing Account Lifecycles in the Age of Deepfakes.pptx
Securing Account Lifecycles in the Age of Deepfakes.pptx
FIDO Alliance
 
cnc-processing-centers-centateq-p-110-en.pdf
cnc-processing-centers-centateq-p-110-en.pdf
AmirStern2
 
Oh, the Possibilities - Balancing Innovation and Risk with Generative AI.pdf
Oh, the Possibilities - Balancing Innovation and Risk with Generative AI.pdf
Priyanka Aash
 
OpenACC and Open Hackathons Monthly Highlights June 2025
OpenACC and Open Hackathons Monthly Highlights June 2025
OpenACC
 
Salesforce Summer '25 Release Frenchgathering.pptx.pdf
Salesforce Summer '25 Release Frenchgathering.pptx.pdf
yosra Saidani
 
EIS-Webinar-Engineering-Retail-Infrastructure-06-16-2025.pdf
EIS-Webinar-Engineering-Retail-Infrastructure-06-16-2025.pdf
Earley Information Science
 
PyCon SG 25 - Firecracker Made Easy with Python.pdf
PyCon SG 25 - Firecracker Made Easy with Python.pdf
Muhammad Yuga Nugraha
 
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdf
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdf
Priyanka Aash
 
Securing AI - There Is No Try, Only Do!.pdf
Securing AI - There Is No Try, Only Do!.pdf
Priyanka Aash
 
9-1-1 Addressing: End-to-End Automation Using FME
9-1-1 Addressing: End-to-End Automation Using FME
Safe Software
 
ReSTIR [DI]: Spatiotemporal reservoir resampling for real-time ray tracing ...
ReSTIR [DI]: Spatiotemporal reservoir resampling for real-time ray tracing ...
revolcs10
 
" How to survive with 1 billion vectors and not sell a kidney: our low-cost c...
" How to survive with 1 billion vectors and not sell a kidney: our low-cost c...
Fwdays
 
You are not excused! How to avoid security blind spots on the way to production
You are not excused! How to avoid security blind spots on the way to production
Michele Leroux Bustamante
 
The Future of Technology: 2025-2125 by Saikat Basu.pdf
The Future of Technology: 2025-2125 by Saikat Basu.pdf
Saikat Basu
 
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
caoyixuan2019
 
OpenPOWER Foundation & Open-Source Core Innovations
OpenPOWER Foundation & Open-Source Core Innovations
IBM
 
Enhance GitHub Copilot using MCP - Enterprise version.pdf
Enhance GitHub Copilot using MCP - Enterprise version.pdf
Nilesh Gule
 
Quantum AI: Where Impossible Becomes Probable
Quantum AI: Where Impossible Becomes Probable
Saikat Basu
 
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
Edge AI and Vision Alliance
 
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Priyanka Aash
 
Securing Account Lifecycles in the Age of Deepfakes.pptx
Securing Account Lifecycles in the Age of Deepfakes.pptx
FIDO Alliance
 
cnc-processing-centers-centateq-p-110-en.pdf
cnc-processing-centers-centateq-p-110-en.pdf
AmirStern2
 
Oh, the Possibilities - Balancing Innovation and Risk with Generative AI.pdf
Oh, the Possibilities - Balancing Innovation and Risk with Generative AI.pdf
Priyanka Aash
 
OpenACC and Open Hackathons Monthly Highlights June 2025
OpenACC and Open Hackathons Monthly Highlights June 2025
OpenACC
 
Salesforce Summer '25 Release Frenchgathering.pptx.pdf
Salesforce Summer '25 Release Frenchgathering.pptx.pdf
yosra Saidani
 
EIS-Webinar-Engineering-Retail-Infrastructure-06-16-2025.pdf
EIS-Webinar-Engineering-Retail-Infrastructure-06-16-2025.pdf
Earley Information Science
 
PyCon SG 25 - Firecracker Made Easy with Python.pdf
PyCon SG 25 - Firecracker Made Easy with Python.pdf
Muhammad Yuga Nugraha
 
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdf
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdf
Priyanka Aash
 
Securing AI - There Is No Try, Only Do!.pdf
Securing AI - There Is No Try, Only Do!.pdf
Priyanka Aash
 

Analyze corefile and backtraces with GDB for Mysql/MariaDB on Linux - Nilandan Joshi (MariaDB)

  • 1. Analyze corefile and get backtraces with GDB for MySQL/MariaDB on Linux Nilnandan Joshi, Sr. Support Engineer, MariaDB [email protected] www.nilinfobin.com
  • 2. Core Dump/File: A core dump is a disk file containing an image of the process’s memory at the time of termination. This image can be used in a debugger to inspect the state of the program at the time that it terminated. The core dump includes key pieces of program state as processor registers, memory management details, and other processor and operating system flags and information.So a core dump is a file that can be very useful to understand the context of a crash.  GDB: GDB, the GNU Project debugger, allows you to see what is going on `inside' another program while it executes -- or what another program was doing at the moment it crashed. i.e sudo gdb -p `pidof mysqld` sudo gdb /path/to/mysqld /path/to/coredump sudo gdb --batch --eval-command="thread apply all bt" /usr/sbin/mysqld $(pgrep -xn mysqld) > mysqld_bt_all_threads.txt sudo gdb --batch --eval-command="thread apply all bt full" /usr/sbin/mysqld $(pgrep -xn mysqld) > mysqld_full_bt_all_threads.txt You can use GDB to study: • InnoDB locks, table locks, metadata locks etc • server variables, user variables at session level • Threads and Statements
  • 3. Backtraces: A backtrace is a summary of how your program got where it is. It shows one line per frame, for many frames, starting with the currently executing frame (frame zero), followed by its caller (frame one), and on up the stack. In a multi-threaded program, GDB by default shows the backtrace only for the current thread. To display the backtrace for several or all of the threads, use the command thread apply. For example, if you type thread apply all backtrace(thread apply all bt) , GDB will display the backtrace for all the threads; this is handy when you debug a core dump of a multi-threaded program. Each line in the backtrace shows the frame number and the function name. The program counter value is also shown—unless you use set print address off. The backtrace also shows the source file name and line number, as well as the arguments to the function. The program counter value is omitted if it is at the beginning of the code for that line number.
  • 4. How to enable Coredump: In order to enable core dumps, you need to set the core_file system variable either on the command-line or in a relevant server option group in an option file. [mariadb] ... core_file There are some additional details related to using core files on Linux. On some systems there is a limit on the sizes of core files that can be dumped. So you have to disable some of the restrictions. To check the system's current system-wide limit, run ulimit -c for the current limit of the mysqld process specifically: sudo cat /proc/$(pidof mysqld)/limits | grep "core file" If you need to change the core size limit, the method you use depends on how you start mysqld. i.e mysqld using mysqld_safe [mysqld_safe] … core_file_size=unlimited
  • 5. - mysqld using systemd sudo tee /etc/systemd/system/mariadb.service.d/limitcore.conf <<EOF [Service] LimitCORE=infinity EOF sudo systemctl daemon-reload If you are using Linux, then it can be helpful to change a few settings to alter where the core files is written and what file name is used. This is done by setting the kernel.core_pattern and kernel.core_uses_pid attributes. By default, core file will be generated in MySQL/MariaDB datadir. If you are using a production system, you probably want to have the core files in a specific directory, not in the data directory. (Because core file size could be very large somewhere in GBs) so for that you can set something like, sudo mkdir /tmp/corefiles sudo chmod 777 /tmp/corefiles sudo echo /tmp/corefiles/core > /proc/sys/kernel/core_pattern sudo echo 1 > /proc/sys/kernel/core_uses_pid Since  mysqld  executes  setuid, you may have to set  fs.suid_dumpable=2  to allow core dumps on Linux.  sudo echo 2 > /proc/sys/fs/suid_dumpable
  • 6. In MariaDB 10.1.35, MariaDB 10.2.17, and MariaDB 10.3.9 and later, core_file has also been made into a system variable. Previously it was just an option. It's value can be checked at runtime by executing the following: SHOW GLOBAL VARIABLES LIKE 'core_file'; However, in MariaDB 10.3.7 and later, some large buffers have been excluded from core files on some systems as a way to reduce the size. The following buffers are excluded: InnoDB buffer pool InnoDB log buffer Server recv buffer Query cache https://mariadb.com/kb/en/enabling-core-dumps/ https://dev.mysql.com/doc/refman/8.0/en/server-options.html#option_mysqld_core-file https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
  • 7. Other Requirements: 1. GDB need to be installed if it's not already yum install gdb 2. debug info packages needs to be install for server and common atleast. i.e MariaDB-server-debuginfo-10.4.17_10-1.el8.x86_64 MariaDB-common-debuginfo-10.4.17_10-1.el8.x86_64 3. If you can’t run gdb + core dump on customer’s server then you must have the same mysqld/mariadb binaries including debug-info packages + all dynamically linked libraries + core file to run gdb in your local machine. Libraries can be checked by ldd command. i.e [root@nilcentos7 ~]# ldd /usr/sbin/mysqld linux-vdso.so.1 => (0x00007ffea8ddd000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f2149b93000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f214995c000) liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f2149736000) … libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f2148810000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f214860c000)
  • 8. Problem: 190328 23:31:08 [ERROR] mysqld got signal 6 ; ... Server version: 10.2.23-MariaDB-10.2.23+maria~stretch ... Thread pointer: 0x7ff4d8001f28 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff4dc62ccc8 thread_stack 0x49000 *** buffer overflow detected ***: /usr/sbin/mysqld terminated ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x70bfb)[0x7ffa09af5bfb] /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7ffa09b7e437] /lib/x86_64-linux-gnu/libc.so.6(+0xf7570)[0x7ffa09b7c570] /lib/x86_64-linux-gnu/libc.so.6(+0xf93aa)[0x7ffa09b7e3aa] /usr/sbin/mysqld(my_addr_resolve+0xe2)[0x55ca42284922] … /usr/sbin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0x72) [0x55ca41b4b992] /usr/sbin/ mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15 enum_duplicatesb+0x1206)[0x55ca41b560f6] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3f68)[0x55ca41b6bee8] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x28a) [0x55ca41b70e4a] /usr/sbin/mysqld(+0x4c864f)[0x55ca41b7164f] ...
  • 9. Problem: 2021-02-12 8:39:33 11039259 [ERROR] mysqld: Table '(temporary)' is marked as crashed and should be repaired 2021-02-12 8:41:09 10757048 [ERROR] mysqld: Table '(temporary)' is marked as crashed and should be repaired free(): invalid next size (normal) 210212 8:41:09 [ERROR] mysqld got signal 6 ; ... Server version: 10.4.17-10-MariaDB-enterprise-log ... Thread pointer: 0x7f179d1103c8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f0a020edbe8 thread_stack 0x49000
  • 10. [root@centos8 nil]# gdb mysqld core.2751610 GNU gdb (GDB) Red Hat Enterprise Linux 8.2-12.el8 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/ gpl.html> … Type "apropos word" to search for commands related to "word"... Reading symbols from mysqld...Reading symbols from /usr/lib/debug/ usr/sbin/mysqld-10.4.17_10-1.el8.x86_64.debug...done. done. [New LWP 2338279] [New LWP 2751610] … … Use the "info sharedlibrary" command to see the complete listing. Do you need "set solib-search-path" or "set sysroot"? [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". … Core was generated by `/usr/sbin/mysqld'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f1908650de3 in __memmove_avx_unaligned_erms () from / lib64/libc.so.6 [Current thread is 1 (Thread 0x7f0b74ee2700 (LWP 2338279))]
  • 11. (gdb) (gdb) thread 1 — this is for the crash [Switching to thread 1 (Thread 0x7f0b74ee2700 (LWP 2338279))] #0 0x00007f1908650de3 in __memmove_avx_unaligned_erms () from / lib64/libc.so.6 (gdb) backtrace — to get the backtrace #0 0x00007f1908650de3 in __memmove_avx_unaligned_erms () from / lib64/libc.so.6 #1 0x0000563bd293aa84 in memcpy (__len=<optimized out>, __src=<optimized out>, __dest=<optimized out>) at /usr/include/ bits/string_fortified.h:34 #2 write_block_record (info=info@entry=0x7f0aa8635b18, old_record=old_record@entry=0x0, record=record@entry=0x7f0aa886ddd0 "347377003ngs", row=row@entry=0x7f0aa8635bb0, bitmap_blocks=bitmap_blocks@entry=0x7f0aa8635bb0, head_block_is_read=<optimized out>, row_pos=0x7f0b74edbb30, undo_lsn=1, old_record_checksum=0) at /usr/src/debug/MariaDB-10.4.17-10/ src_0/storage/maria/ma_blockrec.c:2750
  • 12. #3 0x0000563bd293c9f7 in allocate_and_write_block_record (undo_lsn=1, row=0x7f0aa8635bb0, record=0x7f0aa886ddd0 "347377003ngs", info=0x7f0aa8635b18) at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/maria/ ma_blockrec.c:3571 #4 _ma_write_init_block_record (info=0x7f0aa8635b18, record=0x7f0aa886ddd0 "347377003ngs") at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/maria/ ma_blockrec.c:3611 #5 0x0000563bd2947a81 in maria_write (info=0x7f0aa8635b18, record=0x7f0aa886ddd0 "347377003ngs") at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/maria/ ma_write.c:157 #6 0x0000563bd237b631 in handler::ha_write_tmp_row (this=0x7f0aa8800e00, buf=0x7f0aa886ddd0 "347377003ngs") at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_class.h:6645 #7 0x0000563bd2386134 in schema_table_store_record (thd=0x7f0aa87efb18, table=0x7f0aa8268550) at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_show.cc:3929 #8 0x0000563bd238ad2e in get_schema_column_record(THD*, TABLE_LIST*, TABLE*, bool, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*) () at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_show.cc:6113
  • 13. #9 0x0000563bd2391e04 in get_all_tables(THD*, TABLE_LIST*, Item*) () at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_show.cc:5066 #10 0x0000563bd23930e4 in get_schema_tables_result(JOIN*, enum_schema_table_state) () at /usr/src/debug/MariaDB-10.4.17-10/ src_0/sql/sql_show.cc:8919 #11 0x0000563bd2378bb7 in JOIN::exec_inner() () at /usr/src/debug/ MariaDB-10.4.17-10/src_0/sql/sql_select.cc:4438 #12 0x0000563bd2379113 in JOIN::exec (this=0x7f0aa81f9ee8) at /usr/ src/debug/MariaDB-10.4.17-10/src_0/sql/sql_select.cc:4264 #13 0x0000563bd2377541 in mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) () at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/ sql_select.cc:4699 #14 0x0000563bd2377e1e in handle_select (thd=0x7f0aa87efb18, lex=0x7f0aa87f37d0, result=0x7f0aa81f9ec0, setup_tables_done_option=0) at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/ sql_select.cc:410 #15 0x0000563bd230f41c in execute_sqlcom_select(THD*, TABLE_LIST*) () at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:6429 #16 0x0000563bd23177b3 in mysql_execute_command(THD*) () at /usr/ src/debug/MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:3926
  • 14. #17 0x0000563bd231e74a in mysql_parse (thd=0x7f0aa87efb18, rawbuf=<optimized out>, length=111, parser_state=0x7f0b74ee1310, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /usr/src/debug/ MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:7967 #18 0x0000563bd2320bf1 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) () at /usr/src/debug/MariaDB-10.4.17-10/src_0/sql/sql_class.h:1179 #19 0x0000563bd23221f4 in do_command (thd=0x7f0aa87efb18) at /usr/ src/debug/MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:1357 #20 0x0000563bd2407973 in do_handle_one_connection (connect=connect@entry=0x563c11465fe8) at /usr/src/debug/ MariaDB-10.4.17-10/src_0/sql/sql_connect.cc:1412 #21 0x0000563bd2407a6d in handle_one_connection (arg=arg@entry=0x563c11465fe8) at /usr/src/debug/ MariaDB-10.4.17-10/src_0/sql/sql_connect.cc:1316 #22 0x0000563bd29f430b in pfs_spawn_thread (arg=0x563c1175fcd8) at /usr/src/debug/MariaDB-10.4.17-10/src_0/storage/perfschema/ pfs.cc:1869 #23 0x00007f190a78c14a in start_thread () from /lib64/ libpthread.so.0 #24 0x00007f19085eff23 in clone () from /lib64/libc.so.6
  • 15. (gdb) frame 17 — this is to know more details in that frame(line) ^^ #17 0x0000563bd231e74a in mysql_parse (thd=0x7f0aa87efb18, rawbuf=<optimized out>, length=111, parser_state=0x7f0b74ee1310, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /usr/src/debug/ MariaDB-10.4.17-10/src_0/sql/sql_parse.cc:7967 7967 error= mysql_execute_command(thd); (gdb) print thd->query_string $1 = {string = {str = 0x7f0aa81f6520 "SELECT COLUMN_NAME FROM INFORMATION_SCHEMA.Columns where TABLE_NAME = 'my_workflow' and TABLE_SCHEMA='news'", length = 111}, cs = 0x563bd342ad20 <my_charset_latin1>} (gdb)
  • 16. Some important links for the blog posts by my friend and colleague Valerii Kravchuk GDB basics for MySQL DB https://www.slideshare.net/valeriikravchuk1/gdb-basics-for-my-sql-db-as-percona-live-europe-2019 Checking user threads with GDB http://mysqlentomologist.blogspot.com/2021/01/checking-user-threads-and-temporary.html http://mysqlentomologist.blogspot.com/2018/03/checking-user-threads-with-gdb-in-mysql.html http://mysqlentomologist.blogspot.com/2017/07/how-to-find-processlist-thread-id-in-gdb.html How to explore InnoDB locks with GDB http://mysqlentomologist.blogspot.com/2015/03/using-gdb-to-understand-what-locks-and_31.html http://mysqlentomologist.blogspot.com/2015/04/using-gdb-to-understand-what-locks-and.html