This commit updates default memory allocations size used with MEM_ROOT
objects to minimize the number of calls to malloc().
Changes:
- Updated MEM_ROOT block sizes in sql_const.h
- Updated MALLOC_OVERHEAD to also take into account the extra memory
allocated by my_malloc()
- Updated init_alloc_root() to only take MALLOC_OVERHEAD into account as
buffer size, not MALLOC_OVERHEAD + sizeof(USED_MEM).
- Reset mem_root->first_block_usage if and only if first block was used.
- Increase MEM_ROOT buffers sized used by my_load_defaults, plugin_init,
Create_tmp_table, allocate_table_share, TABLE and TABLE_SHARE.
This decreases number of malloc calls during queries.
- Use a small buffer for THD->main_mem_root in THD::THD. This avoids
multiple malloc() call for new connections.
I tried the above changes on a complex select query with 12 tables.
The following shows the number of extra allocations that where used
to increase the size of the MEM_ROOT buffers.
Original code:
- Connection to MariaDB: 9 allocations
- First query run: 146 allocations
- Second query run: 24 allocations
Max memory allocated for thd when using with heap table: 61,262,408
Max memory allocated for thd when using Aria tmp table: 419,464
After changes:
Connection to MariaDB: 0 allocations
- First run: 25 allocations
- Second run: 7 allocations
Max memory allocated for thd when using with heap table: 61,347,424
Max memory allocated for thd when using Aria table: 529,168
The new code uses slightly more memory, but avoids memory fragmentation
and is slightly faster thanks to much fewer calls to malloc().
Reviewed-by: Sergei Golubchik <serg@mariadb.org>
Heap tables are allocated blocks to store rows according to
my_default_record_cache (mapped to the server global variable
read_buffer_size).
This causes performance issues when the record length is big
(> 1000 bytes) and the my_default_record_cache is small.
Changed to instead split the default heap allocation to 1/16 of the
allowed space and not use my_default_record_cache anymore when creating
the heap. The allocation is also aligned to be just under a power of 2.
For some test that I have been running, which was using record length=633,
the speed of the query doubled thanks to this change.
Other things:
- Fixed calculation of max_records passed to hp_create() to take
into account padding between records.
- Updated calculation of memory needed by heap tables. Before we
did not take into account internal structures needed to access rows.
- Changed block sized for memory_table from 1 to 16384 to get less
fragmentation. This also avoids a problem where we need 1K
to manage index and row storage which was not counted for before.
- Moved heap memory usage to a separate test for 32 bit.
- Allocate all data blocks in heap in powers of 2. Change reported
memory usage for heap to reflect this.
Reviewed-by: Sergei Golubchik <serg@mariadb.org>
The limit of socket length on unix according to libc is 108, see
sockaddr_un::sun_path, but in the table it is a string of max length
64, which results in truncation of socket and failure to connect by
plugins using servers such as spider.
zlib-ng results in different compression length. The compression
length isn't that important as the test output examines the uncompressed
results.
fixes for zlib-ng
backport of 75488a57f2
This problem occured for statements like `INSERT INTO t1 SELECT 1`,
which do not have tables in the SELECT part. In such scenarios
SELECT_LEX::insert_tables was not properly set at `setup_tables()`,
and this led to either incorrect execution or a crash
Reviewer: Oleksandr Byelkin <sanja@mariadb.com>
Item_func_concat_ws::val_str():
- collects the result into the string "str" passed as a parameter.
- calls val_str(&tmp_buffer) to get arguments.
At some point due to heuristic it decides to swap the buffers:
- collect the result into &tmp_buffer
- call val_str(str) to get arguments
Item_func_password::val_str_ascii() returns a String pointing to its
member tmp_value[SCRAMBLED_PASSWORD_CHAR_LENGTH+1].
As a result, it's possible that both str and tmp_buffer in
Item_func_concat_ws::val_str() point to Item_func_password::tmp_value.
Then, memcmp() called on overlapping memory fragrments.
Fixing Item_func_password::val_str_ascii() to use Item::copy()
instead of Item::set().
add_special_frame_cursors() did not check the return
value offset_func->fix_fields(). It can return an error
if the data type does not support the operator "minus".
When UNION ALL is used with LIMIT ROWS EXAMINED, and when the limit is
exceeded for a SELECT that is not the last in the UNION, interrupt the
execution and call end_eof on the result. This makes sure that the
results are sent, and the query result status is conclusive rather
than empty, which would cause an assertion failure.
A prepared SELECT statement because of CF_REEXECUTION_FRAGILE needs to
check the table is the same definition as previously otherwise a
re-prepare of the statement can occur.
When running many 'SELECT DEFAULT(name) FROM table1_containing_sequence'
in parallel the TABLE_LIST::is_the_same_definition may be called when
m_table_ref_type is TABLE_REF_NULL because it hasn't been checked yet.
In this case populate the TABLE_LIST with the values determined by the
TABLE_SHARE and allow the execution to continue.
As a result of this, the main.ps_ddl test doesn't need to reprepare
as the defination hasn't changed. This is another case where
TABLE_LIST::is_the_same_definition is called when m_table_ref_type is
TABLE_REF_NULL, but that doesn't mean that the defination is different.
Fixing a wrong DBUG_ASSERT.
thd->start_time and thd->start_time_sec_part cannot be 0 at the same time.
But thd->start_time can be 0 when thd->start_time_sec_part is not 0,
e.g. after:
SET timestamp=0.99;
Under unknown circumstances, the SQL layer may wrongly disregard an
invocation of thd_mark_transaction_to_rollback() when an InnoDB
transaction had been aborted (rolled back) due to one of the following errors:
* HA_ERR_LOCK_DEADLOCK
* HA_ERR_RECORD_CHANGED (if innodb_snapshot_isolation=ON)
* HA_ERR_LOCK_WAIT_TIMEOUT (if innodb_rollback_on_timeout=ON)
Such an error used to cause a crash of InnoDB during transaction commit.
These changes aim to catch and report the error earlier, so that not only
this crash can be avoided but also the original root cause be found and
fixed more easily later.
The idea of this fix is from Michael 'Monty' Widenius.
HA_ERR_ROLLBACK: A new error code that will be translated into
ER_ROLLBACK_ONLY, signalling that the current transaction
has been aborted and the only allowed action is ROLLBACK.
trx_t::state: Add TRX_STATE_ABORTED that is like
TRX_STATE_NOT_STARTED, but noting that the transaction had been
rolled back and aborted.
trx_t::is_started(): Replaces trx_is_started().
ha_innobase: Check the transaction state in various places.
Simplify the logic around SAVEPOINT.
ha_innobase::is_valid_trx(): Replaces ha_innobase::is_read_only().
The InnoDB logic around transaction savepoints, commit, and rollback
was unnecessarily complex and might have contributed to this
inconsistency. So, we are simplifying that logic as well.
trx_savept_t: Replace with const undo_no_t*. When we rollback to
a savepoint, all we need to know is the number of undo log records
that must survive.
trx_named_savept_t, DB_NO_SAVEPOINT: Remove. We can store undo_no_t
directly in the space allocated at innobase_hton->savepoint_offset.
fts_trx_create(): Do not copy previous savepoints.
fts_savepoint_rollback(): If a savepoint was not found, roll back
everything after the default savepoint of fts_trx_create().
The test innodb_fts.savepoint is extended to cover this code.
Reviewed by: Vladislav Lesin
Tested by: Matthias Leich
The Write_rows_log_event originally allocated the m_rows_buf up-front, and
thus is_valid() checks that the buffer is allocated correctly. But at some
point this was changed to allocate the buffer lazily on demand. This means
that a a valid event can now have m_rows_buf==NULL. The is_valid() code was
not changed, and thus is_valid() could return false on a valid event.
This caused a bug for REPLACE INTO t() VALUES(), () which generates a
write_rows event with no after image; then the m_rows_buf was never
allocated and is_valid() incorrectly returned false, causing an error in
some other parts of the code.
Also fix a couple of missing special cases in the code for mysqlbinlog to
correctly decode (in comments) row events with missing after image.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Fractional part < 100000 microseconds was printed without leading zeros,
causing such timestamps to be applied incorrectly in mariadb-binlog | mysql
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
MDEV-32329 (patch) pushdown from having into where: Server crashes at sub_select
When generating an Item_equal with a Item_ref that refers to a field
outside of a subselect, remove_item_direct_ref() causes the dependency
(depended_from) on the outer select to be lost, which causes trouble
for code downstream that can no longer determine the scope of the Item.
Not calling remove_item_direct_ref() retains the Item's dependency.
Test cases from MDEV-32395 and MDEV-32329 are included.
Some fixes from other developers:
Monty:
- Fixed wrong code in Item_equal::create_pushable_equalities()
that could cause wrong item to be used if there was no matching items.
Daniel Black:
- Added test cases from MDEV-32329
Igor Babaev:
- Provided fix for removing call to remove_item_direct_ref() in
eliminate_item_equal()
MDEV-32395: update_depend_map_for_order: SEGV at /mariadb-11.3.0/sql/sql_select.cc:16583
Include test cases from MDEV-32329.
When binding to NULL, DEFAULT or IGNORE from an Item value, Item_param did not
change m_type_handler, so its value remained from the previous bind.
Thid led to DBUG_ASSERTs in Item_param::get_date() and
Timestamp_or_zero_datetime_native_null.
Fix:
Set Item_param::m_type_handler to &type_handler_null when
binding from an Item returning NULL.
This patch also fixes MDEV-35427.
Item_char_typecast::val_str_generic() uses Item::str_value as a buffer.
Item::val_str_ascii() also used Item::str_value as a buffer.
As a result, str_value tried to copy to itself.
Fixing val_str_ascii() to use a local buffer instead of str_value.
Item_{date|datetime}_typecase::get_date() erroneously passed the
TIME_INTERVAL_DAY flag from the caller to args[0] which made
CAST('100000:00:00' AS DATETIME) parse '100000:00:00' as TIME
rather that DATETIME.
Suppressing this flag.
Cursor protocol cannot handle select... into.
Disable this on loaddata.
For the grant_plugin/innodb_fts.fulltext changed
the tests to use a temporary table rather than a
user variable.
LEAST() and GREATEST() erroneously calcucalted the result as signed
for BIGINT UNSIGNED arguments.
Adding a new method for unsigned arguments:
Item_func_min_max::val_uint_native()
Item_func_substr::fix_length_and_dec() incorrecltly calculated its max_length
to 0 when a huge number was passed as the third argument:
substring('hello', 1, 4294967295)
Fixing this.
Limit only signed integer fields fields to LONGLONG_MAX.
Double and decimal fields do not need this limit, as they
can store integers up to ULONGLONG_MAX without problems.
Sys_var_typelib did not work when assigned to an expression
with character sets with mbminlen>1.
Using val_str_ascii() instead of val_str() to fix this.
Fixed main.mysql_upgrade to pass when unix_socket plugin is unavailable.
Also don't redefine _GNU_SOURCE, which was previously defined by command
line/environment. This fixes silent auth_socket build failure with
MYSQL_MAINTAINER_MODE=ERR.
Item_func_conv::fix_length_and_dec() incorrectly set maximum
length as 64 character. But for negative numbers it can
return up to 65 charcters (including the sign).
Also fixes
MDEV-35392 Assertion `!__asan_region_is_poisoned((vo id*) dest,templ->mysql_col_len)' failed in void row_sel_field_store_in_mysql_format_func(byte *, const mysql_row_templ_t *, const byte *, ulint)
Conversion from CHAR to VARCHAR must be done before the call
for create_length_to_internal_length_string().
Moving the conversion code from Column_definition::prepare_blob_field()
to Column_definition::prepare_stage1_string().
Reported in Debian bug #1084293, from the tzdata changelog:
* Upstream obsoleted the System V names CET, CST6CDT, EET, EST*, HST, MET,
MST*, PST8PDT, and WET. They are symlinks now. Move those zones to
tzdata-legacy and update /etc/localtime on package update to the new names.
Please use Etc/GMT* in case you want to avoid DST changes.
As such the timezone output started to output CET (or CEST) as the
current timezone. Due to the way the test was written, its only
possible to hit this error when running mtr from a package. The
internals of MTR fix the timezone so this will never be hit in a build.
As such, added Europe/Budapest as the Central Europe Standard Time
(per sql/win_tzname_data.h and its derived unicode.org source) as timezone,
hard fixed by timezone.opt file so it will always run. The
have_cet_timezone is there to check the zonedata is installed
(was absent on buildbot Ubuntu 22.04 and Windows).
As replace result to the CET output and treat MET/MEST as the
same while its on its way out.
Thanks Santiago Vila for the bug report and Otto for forwarding it.
The code in the can_eval_in_optimize() branch in
Item_func_pad::fix_length_and_dec() did not take into account
that the constant can be negative. So the function will return NULL.
This later crashed on DBUG_ASSERT() because a NOT NULL function returned NULL.
Adding set_maybe_null() into this branch if the constant is negative.
