We are trying to sort a lot of text/blob fields,
so the buffer is indeed too small.
Memory available = thd->variables.sortbuff_size = 262144
min_sort_memory = param.sort_length*MERGEBUFF2 = 292245
So the decision to abort the query is correct.
filesort() calls my_error(), the error is reported.
But, since we have DELETE IGNORE ... the error is converted to a warning by
THD::raise_condition
filesort currently expects an error to be recorded in the THD diagnostics
area.
If we lift this restriction (remove the assert) we end up in the familiar
void Protocol::end_statement()
default:
DBUG_ASSERT(0);
The solution seems to be to call my_error(ME_FATALERROR) in filesort,
so that the error is propagated as an error rather than a warning.
mysql-test/r/filesort_debug.result:
New test case.
mysql-test/t/filesort_debug.test:
New test case.
This patch is a backport of some of the cleanups/refactorings that were done
as part of WL#1393 Optimizing filesort with small limit.
mysql-test/t/bug13633383.test:
New test case.
mysql-test/valgrind.supp:
Changed signature for find_all_keys
Bug#13011410 CRASH IN FILESORT CODE WITH GROUP BY/ROLLUP
The assert in 13580775 is visible in 5.6 only,
but shows that all versions are vulnerable.
13011410 crashes in all versions.
filesort tries to re-use the sort buffer between invocations in order to save
malloc/free overhead.
The fix for Bug 11748783 - 37359: FILESORT CAN BE MORE EFFICIENT.
added an assert that buffer properties (num_records, record_length) are
consistent between invocations. Indeed, they are not necessarily consistent.
Fix: re-allocate the sort buffer if properties change.
mysql-test/r/partition.result:
New tests.
mysql-test/t/partition.test:
New tests.
sql/filesort.cc:
If we already have allocated a sort buffer in a previous execution,
then verify that it is big enough for the current one.
sql/table.h:
Add sort_keys_size; Number of bytes allocated for the sort_keys buffer.
GROUPING BY FUNCTIONS.... (PART
The bug was introduced in a patch for bug 49897.
Problem: The assertion inserted by the original patch to guard against
zero-lenght sort keys during merge phase triggers also when the whole
set fits in memory.
Fix: Move assert so that it does not trigger if the whole set is in
memory.
mysql-test/r/group_by.result:
Add test for bug#11765254
mysql-test/t/group_by.test:
Add test for bug#11765254
sql/filesort.cc:
Move assertion
Suppress the known warnings generated by filesort().
The real fix belongs to worklog 1509:
Pack values of non-sorted fields in the sort buffer
(which is basically the same issue, but in an optimization context:
We are writing the entire sort buffer to disk,
including un-used space for varchar columns.)
mysql-test/valgrind.supp:
Add new Memcheck suppressions for filesort.
sql/filesort.cc:
Remove the ifdef HAVE_purify/bzero code, use valgrind suppressions instead.
mysql-test/r/filesort_debug.result:
New test case.
mysql-test/t/filesort_debug.test:
New test case.
sql/filesort.cc:
thd->killed does not imply thd->is_error(), so test for that separately.
Write an additional warning message to the server log,
explaining why a sort operation is aborted.
The output in mysqld.err will look something like:
110127 15:07:54 [ERROR] mysqld: Sort aborted: Out of memory (Needed 24 bytes)
110127 15:07:54 [ERROR] mysqld: Out of sort memory, consider increasing server sort buffer size
110127 15:07:54 [ERROR] mysqld: Sort aborted: Out of sort memory, consider increasing server sort buffer size
110127 15:07:54 [ERROR] mysqld: Sort aborted: Incorrect number of arguments for FUNCTION test.f1; expected 0, got 1
If --log-warn=2 is enabled, we output information about host/user/query as well.
include/my_sys.h:
Update comment for ME_NOREFRESH
mysql-test/include/mtr_warnings.sql:
Remove global filtering of "Out of sort memory", let each individual test set it instead.
mysql-test/r/filesort_debug.result:
New test case.
mysql-test/r/order_by.result:
Ignore "Out of memory" for this test.
mysql-test/t/filesort_debug.test:
New test case.
mysql-test/t/order_by.test:
Ignore "Out of memory" for this test.
sql/filesort.cc:
Output an explanation using the error message from the THD Diagnostics_area.
sql/protocol.cc:
Do not DBUG_RETURN(function_call_with DBUG_RETURN)
as it messes up the call stack in the debug output.
sql/share/errmsg-utf8.txt:
Change error message for "Out of sort memory"
sql/unireg.h:
Remove unused/confusing ERRMAPP macro.
and 'THREAD_SAFE_CLIENT'.
As of MySQL 5.5, we no longer support non-threaded
builds. This patch removes all references to the
obsolete THREAD and THREAD_SAFE_CLIENT preprocessor
symbols. These were used to distinguish between
threaded and non-threaded builds.
This assert checks that the server does not try to send OK to the
client if there has been some error during processing. This is done
to make sure that the error is in fact sent to the client.
The problem was that view errors during processing of WHERE conditions
in UPDATE statements where not detected by the update code. It therefore
tried to send OK to the client, triggering the assert.
The bug was only noticeable in debug builds.
This patch fixes the problem by making sure that the update code
checks for errors during condition processing and acts accordingly.
Essentially, the problem is that safemalloc is excruciatingly
slow as it checks all allocated blocks for overrun at each
memory management primitive, yielding a almost exponential
slowdown for the memory management functions (malloc, realloc,
free). The overrun check basically consists of verifying some
bytes of a block for certain magic keys, which catches some
simple forms of overrun. Another minor problem is violation
of aliasing rules and that its own internal list of blocks
is prone to corruption.
Another issue with safemalloc is rather the maintenance cost
as the tool has a significant impact on the server code.
Given the magnitude of memory debuggers available nowadays,
especially those that are provided with the platform malloc
implementation, maintenance of a in-house and largely obsolete
memory debugger becomes a burden that is not worth the effort
due to its slowness and lack of support for detecting more
common forms of heap corruption.
Since there are third-party tools that can provide the same
functionality at a lower or comparable performance cost, the
solution is to simply remove safemalloc. Third-party tools
can provide the same functionality at a lower or comparable
performance cost.
The removal of safemalloc also allows a simplification of the
malloc wrappers, removing quite a bit of kludge: redefinition
of my_malloc, my_free and the removal of the unused second
argument of my_free. Since free() always check whether the
supplied pointer is null, redudant checks are also removed.
Also, this patch adds unit testing for my_malloc and moves
my_realloc implementation into the same file as the other
memory allocation primitives.
client/mysqldump.c:
Pass my_free directly as its signature is compatible with the
callback type -- which wasn't the case for free_table_ent.
This patch:
- Moves all definitions from the mysql_priv.h file into
header files for the component where the variable is
defined
- Creates header files if the component lacks one
- Eliminates all include directives from mysql_priv.h
- Eliminates all circular include cycles
- Rename time.cc to sql_time.cc
- Rename mysql_priv.h to sql_priv.h
column is used for ORDER BY
Problem: filesort isn't meant for null length sort data
(e.g. char(0)), that leads to a server crash.
Fix: disregard sort order if sort data record length is 0 (nothing
to sort).
mysql-test/r/select.result:
Fix for bug#49897: crash in ptr_compare when char(0) NOT NULL
column is used for ORDER BY
- test result.
mysql-test/t/select.test:
Fix for bug#49897: crash in ptr_compare when char(0) NOT NULL
column is used for ORDER BY
- test case.
sql/filesort.cc:
Fix for bug#49897: crash in ptr_compare when char(0) NOT NULL
column is used for ORDER BY
- assert added as filesort cannot handle null length sort data.
sql/sql_select.cc:
Fix for bug#49897: crash in ptr_compare when char(0) NOT NULL
column is used for ORDER BY
- don't sort null length data e.g. in case of ORDER BY CHAR(0).
No need to mask the error code returned by getting the next row to end of file when
doing filesort.
mysql-test/r/innodb_mysql_rbk.result:
Bug #41543: test case
mysql-test/t/innodb_mysql_rbk-master.opt:
Bug #41543: test case
mysql-test/t/innodb_mysql_rbk.test:
Bug #41543: test case
sql/filesort.cc:
Bug #41543: No need to mask the error code returned by getting the next row to end of file when
doing filesort.
- In QUICK_INDEX_MERGE_SELECT::read_keys_and_merge: when we got table->sort from Unique,
tell init_read_record() not to use rr_from_cache() because a) rowids are already sorted
and b) it might be that the the data is used by filesort(), which will need record rowids
(which rr_from_cache() cannot provide).
- Fully de-initialize the table->sort read in QUICK_INDEX_MERGE_SELECT::get_next(). This fixes BUG#35477.
(bk trigger: file as fix for BUG#35478).
sql/filesort.cc:
BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
- make find_all_keys() use quick->get_next() instead of init_read_record(r)/r.read_record() calls
- added dbug printout
sql/mysql_priv.h:
BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
- Added parameter to init_read_record
sql/opt_range.cc:
BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
- In QUICK_INDEX_MERGE_SELECT::read_keys_and_merge: when we got table->sort from Unique,
tell init_read_record() not to use rr_from_cache() because a) rowids are already sorted
and b) it might be that the the data is used by filesort(), which will need record rowids
(which rr_from_cache() cannot provide).
- Fully de-initialize the table->sort read in QUICK_INDEX_MERGE_SELECT::get_next().
sql/records.cc:
BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
- Added disable_rr_cache parameter to init_read_record
- Added comment
sql/sql_acl.cc:
BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
- Added parameter to init_read_record
sql/sql_delete.cc:
BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
- Added parameter to init_read_record
sql/sql_help.cc:
BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
- Added parameter to init_read_record
sql/sql_select.cc:
BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
- Added parameter to init_read_record
sql/sql_table.cc:
BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
- Added parameter to init_read_record
sql/sql_udf.cc:
BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
- Added parameter to init_read_record
sql/sql_update.cc:
BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
- Added parameter to init_read_record
into kaamos.(none):/data/src/opt/mysql-5.1-opt
client/mysql.cc:
Auto merged
client/mysqldump.c:
Auto merged
configure.in:
Auto merged
include/my_global.h:
Auto merged
libmysql/libmysql.c:
Auto merged
libmysqld/lib_sql.cc:
Auto merged
mysql-test/include/mix1.inc:
Auto merged
mysql-test/r/create.result:
Auto merged
mysql-test/r/func_str.result:
Auto merged
mysql-test/r/innodb.result:
Auto merged
mysql-test/r/innodb_mysql.result:
Auto merged
mysql-test/r/select.result:
Auto merged
mysql-test/r/subselect.result:
Auto merged
mysql-test/t/create.test:
Auto merged
mysql-test/t/disabled.def:
Auto merged
sql/filesort.cc:
Auto merged
sql/handler.cc:
Auto merged
sql/item.cc:
Auto merged
sql/item.h:
Auto merged
sql/item_cmpfunc.cc:
Auto merged
sql/item_func.cc:
Auto merged
sql/item_func.h:
Auto merged
sql/item_sum.cc:
Auto merged
sql/log.cc:
Auto merged
sql/mysql_priv.h:
Auto merged
sql/mysqld.cc:
Auto merged
sql/set_var.cc:
Auto merged
sql/sp.cc:
Auto merged
sql/sql_acl.cc:
Auto merged
sql/sql_base.cc:
Auto merged
sql/sql_insert.cc:
Auto merged
sql/sql_plugin.cc:
Auto merged
sql/sql_select.cc:
Auto merged
sql/sql_yacc.yy:
Auto merged
storage/ndb/src/kernel/blocks/backup/Backup.hpp:
Auto merged
tests/mysql_client_test.c:
Auto merged
mysql-test/r/func_time.result:
Manual merge.
mysql-test/r/view.result:
Manual merge.
mysql-test/t/view.test:
Manual merge.
scripts/mysql_config.sh:
Manual merge.
sql-common/client.c:
Manual merge.
sql/sql_parse.cc:
Manual merge.
into kaamos.(none):/data/src/opt/mysql-5.0-opt
client/mysql.cc:
Auto merged
sql/filesort.cc:
Auto merged
sql/item.cc:
Auto merged
sql/item.h:
Auto merged
sql/item_func.cc:
Auto merged
sql/mysqld.cc:
Auto merged
sql/sql_parse.cc:
Auto merged
sql/sql_yacc.yy:
Auto merged
tests/mysql_client_test.c:
Auto merged
mysql-test/r/view.result:
Manual merge.
mysql-test/t/view.test:
Manual merge.
scripts/mysql_config.sh:
Manual merge.
into kaamos.(none):/data/src/opt/mysql-5.1-opt
libmysqld/lib_sql.cc:
Auto merged
mysql-test/r/create.result:
Auto merged
mysql-test/r/innodb_mysql.result:
Auto merged
mysql-test/r/variables.result:
Auto merged
BitKeeper/deleted/.del-rpl_ndb_charset.result:
Auto merged
BitKeeper/deleted/.del-rpl_row_charset.result:
Auto merged
BitKeeper/deleted/.del-rpl_row_charset.test:
Auto merged
BitKeeper/deleted/.del-rpl_row_charset_innodb.result:
Auto merged
mysql-test/t/create.test:
Auto merged
mysql-test/t/sp.test:
Auto merged
mysql-test/t/variables.test:
Auto merged
sql/field.cc:
Auto merged
sql/filesort.cc:
Auto merged
sql/handler.cc:
Auto merged
sql/item.cc:
Auto merged
sql/item_cmpfunc.cc:
Auto merged
sql/item_func.cc:
Auto merged
sql/mysqld.cc:
Auto merged
sql/sql_acl.cc:
Auto merged
sql/share/errmsg.txt:
Auto merged
sql/sql_select.cc:
Auto merged
sql/sql_yacc.yy:
Auto merged
mysql-test/lib/mtr_report.pl:
Manual merge.
mysql-test/r/sp.result:
Manual merge.
The out of memory error was thrown when the sort buffer size were too small.
This led to a user confusion.
Now filesort throws the error message about sort buffer being too small.
mysql-test/t/order_by.test:
Added a test case for the bug#31590: Wrong error message on sort buffer being too small.
mysql-test/r/order_by.result:
Added a test case for the bug#31590: Wrong error message on sort buffer being too small.
sql/filesort.cc:
Bug#31590: Wrong error message on sort buffer being too small.
Now filesort throws the error message about sort buffer being too small
instead of out of memory error.
Fix by calling ha_release_temporary_latches() before ::filesort().
sql/filesort.cc:
Call ha_release_temporary_latches() before performing
a (possibly long) ::filesort().
into ramayana.hindu.god:/home/tsmith/m/bk/maint/51
client/mysqldump.c:
Auto merged
configure.in:
Auto merged
sql/field.cc:
Auto merged
sql/filesort.cc:
Auto merged
sql/item.cc:
Auto merged
sql/item_timefunc.cc:
Auto merged
sql/log.cc:
Auto merged
sql/repl_failsafe.cc:
Auto merged
sql/set_var.cc:
Auto merged
sql/sp.cc:
Auto merged
sql/sql_base.cc:
Auto merged
sql/sql_class.cc:
Auto merged
sql/sql_class.h:
Auto merged
sql/sql_select.cc:
Auto merged
sql/sql_show.cc:
Auto merged
sql/sql_yacc.yy:
Auto merged
sql/item_sum.cc:
Manual merge.
Bug#33675: Usage of an uninitialized memory by filesort in a subquery caused
server crash.
Free smaller buffer before allocating bigger one.
sql/filesort.cc:
Bug#33675: Usage of an uninitialized memory by filesort in a subquery caused
server crash.
Free smaller buffer before allocating bigger one.
into kaamos.(none):/data/src/opt/mysql-5.1-opt
sql/field.h:
Auto merged
sql/item_timefunc.cc:
Auto merged
sql/item_timefunc.h:
Auto merged
sql/set_var.cc:
Auto merged
sql/sql_base.cc:
Auto merged
mysql-test/r/subselect.result:
Manual merge.
mysql-test/t/subselect.test:
Manual merge.
sql/filesort.cc:
Manual merge.
server crash.
The filesort implementation has an optimization for subquery execution which
consists of reusing previously allocated buffers. In particular the call to
the read_buffpek_from_file function might be skipped when a big enough buffer
for buffer descriptors (buffpeks) is already allocated. Beside allocating
memory for buffpeks this function fills allocated buffer with data read from
disk. Skipping it might led to using an arbitrary memory as fields' data and
finally to a crash.
Now the read_buffpek_from_file function is always called. It allocates
new buffer only when necessary, but always fill it with correct data.
sql/filesort.cc:
Bug#33675: Usage of an uninitialized memory by filesort in a subquery caused
server crash.Now the read_buffpek_from_file function is always called. It allocates
new buffer only when necessary, but always fill it with correct data.
mysql-test/r/subselect.result:
Added a test case for the bug#33675: Usage of an uninitialized memory by
filesort in a subquery caused server crash.
mysql-test/t/subselect.test:
Added a test case for the bug#33675: Usage of an uninitialized memory by
filesort in a subquery caused server crash.
into zippy.cornsilk.net:/home/cmiller/work/mysql/mysql-5.1-maint
sql/field.cc:
Auto merged
sql/filesort.cc:
Auto merged
sql/ha_ndbcluster.cc:
Auto merged
sql/handler.cc:
Auto merged
sql/item_cmpfunc.cc:
Auto merged
sql/item_create.cc:
Auto merged
sql/item_func.cc:
Auto merged
sql/item_geofunc.cc:
Auto merged
sql/item_strfunc.cc:
Auto merged
sql/item_subselect.cc:
Auto merged
sql/item_sum.cc:
Auto merged
sql/item_timefunc.cc:
Auto merged
sql/log.cc:
Auto merged
sql/mysql_priv.h:
Auto merged
sql/mysqld.cc:
Auto merged
sql/net_serv.cc:
Auto merged
sql/opt_sum.cc:
Auto merged
sql/protocol.h:
Auto merged
sql/records.cc:
Auto merged
sql/set_var.cc:
Auto merged
sql/sp.cc:
Auto merged
sql/sp_head.h:
Auto merged
sql/sql_cache.cc:
Auto merged
sql/sql_class.cc:
Auto merged
sql/sql_class.h:
Auto merged
sql/sql_prepare.cc:
Auto merged
sql/sql_select.h:
Auto merged
sql/sql_trigger.cc:
Auto merged
sql/sql_update.cc:
Auto merged
sql/sql_view.cc:
Auto merged
sql/structs.h:
Auto merged
sql/unireg.h:
Auto merged
sql/item.cc:
manual merge
sql/log_event.cc:
manual merge
sql/protocol.cc:
manual merge
sql/sp_head.cc:
manual merge
sql/sql_base.cc:
manual merge
sql/sql_parse.cc:
manual merge
sql/sql_select.cc:
manual merge
