MDEV-22689 MSAN use-of-uninitialized-value in decode_bytes()
This was not a user visible issue as the huffman code lookup tables would
automatically ignore any of the unitialized bits
Fixed by adding a end-zero byte to the bit-stream buffer.
Other things:
- Fixed a (for this case) wrong assert in strmov() for myisamchk
and aria_chk by removing the strmov()
Several macros such as sint2korr() and uint4korr() are using the
arithmetic + operator while a bitwise or operator would suffice.
GCC 5 and clang 5 and later can detect patterns consisting of
bitwise or and shifts by multiples of 8 bits, such as those used
in the InnoDB function mach_read_from_4(). They actually translate
that verbose low-level code into high-level machine language
(i486 bswap instruction or fused into the Haswell movbe instruction).
We should do the same for MariaDB Server code that is outside InnoDB.
Note: The Microsoft C compiler is lacking this optimization.
There, we might consider using _byteswap_ushort(), _byteswap_ulong(),
_byteswap_uint64(). But, those would lead to unaligned reads, which are
bad for reasons stated in MDEV-20277. Besides, outside InnoDB,
most data is already being stored in the native little-endian format
of that compiler.
Backport to 5.5
(external Bug#69407 Build warnings with mysql)
support-files/build-tags:
Run etags on sql_yacc.yy, ignore other .yy files
unittest/mysys/explain_filename-t.cc:
NO_PLAN seems to fail on some platforms, use the actual number instead.
mysql-test/suite/innodb/t/group_commit_crash.test:
remove autoincrement to avoid rbr being used for insert ... select
mysql-test/suite/innodb/t/group_commit_crash_no_optimize_thread.test:
remove autoincrement to avoid rbr being used for insert ... select
mysys/my_addr_resolve.c:
a pointer to a buffer is returned to the caller -> the buffer cannot be on the stack
mysys/stacktrace.c:
my_vsnprintf() is ok here, in 5.5
WITH MYISAM_USE_MMAP ENABLED
MySQL server can crash due to segmentation fault when
started with myisam_use_mmap.
The reason behind this being, while making a request to
unmap (munmap) the previously mapped memory (mmap), the
size passed was 7 bytes larger than the size requested at
the time of mapping. This can eventually unmap the adjacent
memory mapped block, belonging to some other memory-map pool.
Hence the subsequent call to mmap can map a region which was
still a valid memory mapped area.
Fixed by removing the extra 7-byte margin which was erroneously
added to the size, used for unmappping.
storage/myisam/mi_close.c:
Bug#11756764 48726: MYSQLD KEEPS CRASHING WITH SIGSEGV
WITH MYISAM_USE_MMAP ENABLED
Added a condition to call _mi_unmap_file() in case
of compressed records. mi_munmap_file() is called
otherwise.
storage/myisam/mi_packrec.c:
Bug#11756764 48726: MYSQLD KEEPS CRASHING WITH SIGSEGV
WITH MYISAM_USE_MMAP ENABLED
mi_dynmap_file() function, after successfully executing
mmap, stores the total size in info->s->mapped_length
variable. Now, if mi_dynmap_file() is invoked with a size
with an extra 7-byte margin (MEMMAP_EXTRA_MARGIN),
the margin will eventually also get stored in mapped_length.
So, un-mapping function can simply use the value stored in
mapped_length in order to unmap the previously mapped
region.
Essentially, the problem is that safemalloc is excruciatingly
slow as it checks all allocated blocks for overrun at each
memory management primitive, yielding a almost exponential
slowdown for the memory management functions (malloc, realloc,
free). The overrun check basically consists of verifying some
bytes of a block for certain magic keys, which catches some
simple forms of overrun. Another minor problem is violation
of aliasing rules and that its own internal list of blocks
is prone to corruption.
Another issue with safemalloc is rather the maintenance cost
as the tool has a significant impact on the server code.
Given the magnitude of memory debuggers available nowadays,
especially those that are provided with the platform malloc
implementation, maintenance of a in-house and largely obsolete
memory debugger becomes a burden that is not worth the effort
due to its slowness and lack of support for detecting more
common forms of heap corruption.
Since there are third-party tools that can provide the same
functionality at a lower or comparable performance cost, the
solution is to simply remove safemalloc. Third-party tools
can provide the same functionality at a lower or comparable
performance cost.
The removal of safemalloc also allows a simplification of the
malloc wrappers, removing quite a bit of kludge: redefinition
of my_malloc, my_free and the removal of the unused second
argument of my_free. Since free() always check whether the
supplied pointer is null, redudant checks are also removed.
Also, this patch adds unit testing for my_malloc and moves
my_realloc implementation into the same file as the other
memory allocation primitives.
client/mysqldump.c:
Pass my_free directly as its signature is compatible with the
callback type -- which wasn't the case for free_table_ent.
* Finished Monty and Jani's merge
* Some InnoDB tests still fail (because it's old xtradb code run against
newer testsuite). They are expected to go after mergning with the latest
xtradb.
Currently the memory map is being created
with a size that is greater than the size
of the underlying datafile. This can cause
varying behaviour,
e.g.
In windows the size of the datafile
is increased, while on linux it remains
the same.
This fix removes the increment margin to
the size that is used while creating the
memory map.
storage/myisam/mi_dynrec.c:
remove MEMMAP_EXTRA_MARGIN that is used as
the increment margin to the underlying
datafile size while creating the mmap.
storage/myisam/mi_packrec.c:
The size of the underlying datafile is
increased by MEMMAP_EXTRA_MARGIN when
using a packed record format. Hence in
this case the size of the memory map should
be incremented by the same factor.
- Remove bothersome warning messages. This change focuses on the warnings
that are covered by the ignore file: support-files/compiler_warnings.supp.
- Strings are guaranteed to be max uint in length
into a88-113-38-195.elisa-laajakaista.fi:/home/my/bk/mysql-maria.new
BitKeeper/etc/ignore:
auto-union
BUILD/compile-dist:
Auto merged
client/mysqladmin.cc:
Auto merged
client/mysqldump.c:
Auto merged
config/ac-macros/plugins.m4:
Auto merged
configure.in:
Auto merged
dbug/dbug.c:
Auto merged
include/keycache.h:
Auto merged
include/m_string.h:
Auto merged
include/my_global.h:
Auto merged
include/my_pthread.h:
Auto merged
include/my_sys.h:
Auto merged
include/mysql_com.h:
Auto merged
libmysql/CMakeLists.txt:
Auto merged
libmysql/Makefile.shared:
Auto merged
mysql-test/lib/mtr_cases.pl:
Auto merged
mysql-test/lib/mtr_report.pl:
Auto merged
mysql-test/mysql-test-run.pl:
Auto merged
mysql-test/r/innodb.result:
Auto merged
mysql-test/r/merge.result:
Auto merged
mysql-test/r/myisam.result:
Auto merged
mysql-test/r/mysqldump.result:
Auto merged
mysql-test/r/ps_2myisam.result:
Auto merged
mysql-test/r/ps_3innodb.result:
Auto merged
mysql-test/r/ps_4heap.result:
Auto merged
mysql-test/r/ps_5merge.result:
Auto merged
mysql-test/r/query_cache.result:
Auto merged
mysql-test/r/subselect.result:
Auto merged
mysql-test/r/view.result:
Auto merged
mysql-test/suite/ndb/r/ps_7ndb.result:
Auto merged
mysql-test/suite/rpl/r/rpl_switch_stm_row_mixed.result:
Auto merged
mysql-test/suite/rpl/t/rpl_switch_stm_row_mixed.test:
Auto merged
mysql-test/t/delayed.test:
Auto merged
mysql-test/t/disabled.def:
Auto merged
mysql-test/t/myisam.test:
Auto merged
mysql-test/t/mysqldump.test:
Auto merged
mysql-test/t/query_cache.test:
Auto merged
mysql-test/t/query_cache_notembedded.test:
Auto merged
mysql-test/t/subselect.test:
Auto merged
mysql-test/t/variables.test:
Auto merged
mysql-test/t/view.test:
Auto merged
mysys/CMakeLists.txt:
Auto merged
mysys/Makefile.am:
Auto merged
mysys/mf_keycache.c:
Auto merged
mysys/my_delete.c:
Auto merged
mysys/my_init.c:
Auto merged
mysys/my_symlink2.c:
Auto merged
mysys/safemalloc.c:
Auto merged
mysys/thr_lock.c:
Auto merged
sql/field.h:
Auto merged
sql/filesort.cc:
Auto merged
sql/ha_ndbcluster.cc:
Auto merged
sql/ha_partition.cc:
Auto merged
sql/handler.cc:
Auto merged
sql/handler.h:
Auto merged
sql/item.h:
Auto merged
sql/item_func.cc:
Auto merged
sql/item_func.h:
Auto merged
sql/item_strfunc.cc:
Auto merged
sql/item_xmlfunc.cc:
Auto merged
sql/lock.cc:
Auto merged
sql/log.cc:
Auto merged
sql/log_event_old.h:
Auto merged
sql/mysql_priv.h:
Auto merged
sql/net_serv.cc:
Auto merged
sql/opt_range.cc:
Auto merged
sql/set_var.h:
Auto merged
sql/slave.cc:
Auto merged
sql/slave.h:
Auto merged
sql/sql_class.cc:
Auto merged
sql/sql_class.h:
Auto merged
sql/sql_delete.cc:
Auto merged
sql/sql_insert.cc:
Auto merged
sql/sql_parse.cc:
Auto merged
sql/sql_plugin.cc:
Auto merged
sql/sql_select.cc:
Auto merged
sql/sql_show.cc:
Auto merged
sql/share/errmsg.txt:
Auto merged
sql/table.cc:
Auto merged
sql/table.h:
Auto merged
sql/udf_example.c:
Auto merged
storage/csv/ha_tina.cc:
Auto merged
storage/myisam/ft_boolean_search.c:
Auto merged
storage/myisam/ft_nlq_search.c:
Auto merged
storage/myisam/ft_parser.c:
Auto merged
storage/myisam/ha_myisam.cc:
Auto merged
storage/myisam/ha_myisam.h:
Auto merged
storage/myisam/mi_check.c:
Auto merged
storage/myisam/mi_dynrec.c:
Auto merged
storage/myisam/mi_open.c:
Auto merged
storage/myisam/mi_packrec.c:
Auto merged
storage/myisam/mi_write.c:
Auto merged
storage/myisam/myisamchk.c:
Auto merged
storage/myisam/myisampack.c:
Auto merged
storage/myisam/rt_index.c:
Auto merged
storage/myisam/sort.c:
Auto merged
storage/myisammrg/ha_myisammrg.cc:
Auto merged
storage/myisammrg/ha_myisammrg.h:
Auto merged
strings/llstr.c:
Auto merged
support-files/compiler_warnings.supp:
Auto merged
Makefile.am:
Manual merge from 5.1 to maria.
client/mysqltest.c:
Manual merge from 5.1 to maria.
include/my_base.h:
Manual merge from 5.1 to maria.
mysql-test/t/merge.test:
Manual merge from 5.1 to maria.
mysys/my_getopt.c:
Manual merge from 5.1 to maria.
mysys/thr_mutex.c:
Manual merge from 5.1 to maria.
sql/mysqld.cc:
Manual merge from 5.1 to maria.
sql/set_var.cc:
Manual merge from 5.1 to maria.
sql/sql_table.cc:
Manual merge from 5.1 to maria.
sql/sql_yacc.yy:
Manual merge from 5.1 to maria.
sql/unireg.cc:
Manual merge from 5.1 to maria.
into a88-113-38-195.elisa-laajakaista.fi:/home/my/bk/mysql-5.1-marvel
mysql-test/lib/mtr_report.pl:
Auto merged
mysys/my_delete.c:
Auto merged
sql/handler.cc:
Auto merged
sql/item_func.cc:
Auto merged
sql/log.cc:
Auto merged
sql/mysql_priv.h:
Auto merged
sql/mysqld.cc:
Auto merged
sql/sql_class.cc:
Auto merged
sql/sql_parse.cc:
Auto merged
sql/sql_select.cc:
Auto merged
sql/sql_show.cc:
Auto merged
sql/sql_table.cc:
Auto merged
sql/sql_yacc.yy:
Auto merged
sql/table.cc:
Auto merged
sql/table.h:
Auto merged
storage/myisam/ha_myisam.cc:
Auto merged
storage/myisam/mi_check.c:
Auto merged
storage/myisam/mi_open.c:
Auto merged
storage/myisam/mi_packrec.c:
Auto merged
storage/myisam/myisamchk.c:
Auto merged
storage/myisammrg/ha_myisammrg.cc:
Auto merged
include/my_base.h:
Manual merge from 5.1
into mysql.com:/home/svoj/devel/mysql/BUG31277/mysql-5.1-engines
include/mysql_com.h:
Auto merged
sql-common/client.c:
Auto merged
sql/mysql_priv.h:
Auto merged
storage/myisam/mi_open.c:
Auto merged
storage/myisam/mi_packrec.c:
Auto merged
mysql-test/mysql-test-run.pl:
Manual merge.
storage/myisam/mi_check.c:
Manual merge.
storage/myisam/myisamchk.c:
Manual merge.
