Problem was that a parallel open of a table, overwrote info->state that
was in used by repair.
Fixed by changing _ma_tmp_disable_logging_for_table() to use
a new state buffer state.no_logging to store the temporary state.
Other things:
- Use original number of rows when retrying repair to get rid of a
potential warning "Number of rows changed from X to Y"
- Changed maria_commit() to make it easier to merge with 10.4
- If table is not locked (like with show commands), use the global
number of rows as the local number may not be up to date.
(Minor not critical fix)
- Added some missing DBUG_RETURN
MDEV-10130 Assertion `share->in_trans == 0' failed in storage/maria/ma_close.c
MDEV-10378 Assertion `trn' failed in virtual int ha_maria::start_stmt
The problem was that maria_handler->trn was not properly reset
at commit/rollback and ha_maria::exernal_lock() could get confused
because.
There was some old code in ha_maria::implicit_commit() that tried
to take care of this, but it was not bullet proof.
Fixed by adding list of all tables that is part of the maria transaction to
TRN.
A nice side effect was of the fix is that loops in
ha_maria::implict_commit() got to be much simpler.
Other things:
- Fixed a bug in mysql_admin_table() where argument open_for_modify
was wrongly reset for the next table in the chain
- rollback admin command also in case of fatal error.
- Split _ma_set_trn_for_table() to three version to simplify code
and debugging.
- Several new asserts to detect the original problem (that file was
not properly removed from trn before calling ma_close())
For some simple benchmarks, a majority of time was
spend in find_head() which tries to find the best
place to put the record.
The result of this patch is a 2x or more speedup for
inserts without keys for format PAGE. All changes
are only related to how rows are stored
Should fix some of the problems mentioned in:
MDEV-8132 Temporary tables using Aria with very poor performance
MDEV-9079 Aria very slow for internal temporary tables
MDEV-5841 Mariadb very poor temporary performance
The following changes where done:
- For rows with a small row length that fits into
a page (818 bytes with 8192 pages), stop as soon as we
hit a match.
- Added markers full_head_size and full_tail_size that tells
us where to start searching on the bitmap page
- Ensure that page->used_size is correctly updated when
bitmap grows. This allows us to stop searching at used_size
- Added code to check that the bitmap variables are correct.
- Fixed a wrong test where we set "first_bitmap_with_space".
This shouldn't have caused any notable problems.
This bug happens when locking the same Aria "transactional" table
(page format) more then once with LOCK TABLES and inserting into one
of them with INSERT ... SELECT when the table is empty.
Fixed by ensuring we don't use fast bulk insert if table is opened
twice with LOCK TABLES (as this changes table->s->state)
Code changes:
- Added use_count to MARIA_USED_TABLES to be able to check if
table is opened twice for a statement/lock table
- Don't clear history or reset info->start_state if we
don't have versioning. One reason for the bug was
was that info->start_state was set to point to different
states for the two tables. If there is no versioning
info->start_state should always point to info->s->state.common.
Other things:
- Fixed also some typos that was noticed while scanning the code
- More DBUG_PRINT
Also, implement MDEV-11027 a little differently from 5.5 and 10.0:
recv_apply_hashed_log_recs(): Change the return type back to void
(DB_SUCCESS was always returned).
Report progress also via systemd using sd_notifyf().
- Used same fix as for MyISAM: High level collation byte stored in unused
bit_end position.
- Moved language from header to base_info
- Removed unused bit_end part in HA_KEY_SEG
Both aria and myisam storage engines feature a logic path in
thr_find_all_keys that leads to undefined behaviour by bypassing the
initialization code of variables after my_thread_init().
By refactoring the nested logic into a separate function, this problem
is resolved.
Problem was that insert-order (enforced by the optimizer) did not handle
the case where the bitmap changed to a new one.
Fixed by remembering the last bitmap page used and to force usage of this when inserting new rows
Due to a hack that has propagated to the maria storage engine, undefined
behaviour would result by bypassing the initialization code of variables
after my_thread_init().
By refactoring the nested logic into a separate function, this problem
is resolved.
This bug only happens in case of paritioned tables used in LOCK TABLES and implicit_commit() was called
(as part of trying to execute a CREATE TABLE withing lock tables)
The problem was that Aria could not move the tables from one transaction to the new one, as thd->open_tables contained
a partitioned tables and not an Aria table.
Fix:
- Store a list of all open tables that are part of a share in share->open_tables
- In maria::implict_commit() use transaction->used_tables & share->open_tables to find out which tables
was part of the current transaction instead of using thd->open_tables, which may contain partitioned tables.
mysql-test/suite/maria/maria_partition.result:
Added test case
mysql-test/suite/maria/maria_partition.test:
Added test case
storage/maria/ha_maria.cc:
Use trn->used tables and share->open_tables to find out which tables was part of the current transaction instead of using thd->open_tables.
storage/maria/ma_close.c:
Remove closed table from share->open_list
storage/maria/ma_open.c:
Add table to share->open_list
storage/maria/ma_state.c:
Added comment
storage/maria/maria_def.h:
Added share->open_list, a list of all tables that is using this share.
The problem was that a big record was allocated on the stack, which casued stack to run out.
Fixed by using my_safe_alloca() instead of my_alloca() when allocating records.
Now only records <= 16384 are allocated on the stack.
mysql-test/r/stack-crash.result:
Added test case
mysql-test/t/stack-crash.test:
Added test case
storage/maria/ma_blockrec.c:
Use my_safe_alloca() instead of my_alloca()
storage/maria/ma_dynrec.c:
Use my_safe_alloca() instead of my_alloca()
storage/maria/maria_def.h:
Added MARIA_MAX_RECORD_ON_STACK
storage/maria/maria_pack.c:
Use my_safe_alloca() instead of my_alloca()
The problem was that a big record was allocated on the stack, which casued stack to run out.
Fixed by using my_safe_alloca() instead of my_alloca() when allocating records.
Now only records <= 16384 are allocated on the stack.
mysql-test/r/stack-crash.result:
Added test case
mysql-test/t/stack-crash.test:
Added test case
storage/maria/ma_blockrec.c:
Use my_safe_alloca() instead of my_alloca()
storage/maria/ma_dynrec.c:
Use my_safe_alloca() instead of my_alloca()
storage/maria/maria_def.h:
Added MARIA_MAX_RECORD_ON_STACK
storage/maria/maria_pack.c:
Use my_safe_alloca() instead of my_alloca()
Update InnoDB to 5.6.14
Apply MySQL-5.6 hack for MySQL Bug#16434374
Move Aria-only HA_RTREE_INDEX from my_base.h to maria_def.h (breaks an assert in InnoDB)
Fix InnoDB memory leak
Fixed the following compilation errors and test failures:
- maria SE: "stage_waiting_for_a_resource" wasn't declared w/o PFS
- sql_repl.h: PSI_mutex_key is not available in non-PFS builds
- mysqld.cc: pfs_param is not available in non-PFS builds
- mysqld.cc: init_show_explain_psi_keys() is not available in non-PFS builds
- mysqld.cc: call net_before_header_psi, net_after_header_psi even if PFS
is not available so that thread enters stage_init at proper time. Fixes
sp-threads and a few tests in funcs_1.
- myisam_file_io.opt: added missing loose prefix
includes:
* remove some remnants of "Bug#14521864: MYSQL 5.1 TO 5.5 BUGS PARTITIONING"
* introduce LOCK_share, now LOCK_ha_data is strictly for engines
* rea_create_table() always creates .par file (even in "frm-only" mode)
* fix a 5.6 bug, temp file leak on dummy ALTER TABLE
This could happen when using Aria for internal temporary files (default case) and using DISTINCT.
_ma_scan_restore_block_record() didn't work correctly if there was rows inserted, updated or deleted on the handler
between calls to _ma_scan_remember_block_record() and _ma_scan_restore_block_record().
The effect was that some DISTINCT queries that used remove_dup_with_compare() could fail.
.bzrignore:
Ignore sql_yacc.hh
mysql-test/suite/maria/r/distinct.result:
Test case for MDEV-4280
mysql-test/suite/maria/t/distinct.test:
Test case for MDEV-4280
mysql-test/t/mysql.test:
Fixed test suite (we could get error -1 in some cases)
sql/sql_select.cc:
Break loop if restart_rnd_next() gives an error
storage/maria/ha_maria.cc:
scan_restore_pos() can return disk fault error.
storage/maria/ma_blockrec.c:
_ma_scan_remember_block_record() did incorrectly update scan.dir instead of scan_save.dir .
_ma_scan_restore_block_record() didn't work correctly if there was rows inserted,updated or deleted on the handler
between calls to _ma_scan_remember_block_record() and _ma_scan_restore_block_record().
Fixed by adding counters for row changes and reading the current scan page if changes had been made.
storage/maria/ma_blockrec.h:
scan_restore_pos() can return disk fault error.
storage/maria/ma_delete.c:
Increment row_changes
storage/maria/ma_scan.c:
scan_restore_pos() can return disk fault error.
storage/maria/ma_update.c:
Increment row_changes
storage/maria/ma_write.c:
Increment row_changes
storage/maria/maria_def.h:
scan_restore_pos() can return disk fault error.
- If one specifies --force twice to myisamchk and aria_chk, then we will try to finnish the repair even if sort_buffer would be too small.
This was done by dynamically allocate buffer handler objects as long as memory lasts.
- New option for myisamchk and aria_chk: create-missing-keys
- Changed default size of myisam_sort_buffer_size from 8M to 128M.
- Changed default size of sort_buffer_size in aria_chk from 128M to 256M.
- Increased information in error message about 'sort_buffer_size' beeing to small.
- Print also to 'show warnings' if repair was retried.
- Increased size of internal sort-buffer-readers from 16K to 128K
- Changed printing of 'number of records' to use %ll instead of casting to long
- Changed buffer sizes for myisam and aria to use MY_ALIGN_DOWN() to get same number of bytes allocated on different machines.
include/my_global.h:
Added MY_ALIGN_DOWN() to get previous alignment (for big memory areas)
include/myisam.h:
Increased size of types to be able to handle more records
include/myisamchk.h:
Increased size of types to be able to handle more records
Added T_FORCE_SORT_MEMORY to force repair to work even if sort_buffer would not be big enough
mysql-test/r/myisam.result:
Updated result
mysql-test/r/mysqld--help.result:
Updated result
mysql-test/r/repair.result:
Updated result
mysql-test/suite/maria/maria.result:
Added test cases for some fixed bugs in MyISAM to verify that Aria doesn't have them too.
mysql-test/suite/maria/maria.test:
Added test cases for some fixed bugs in MyISAM to verify that Aria doesn't have them too.
mysql-test/suite/maria/maria3.result:
Updated result after sort buffer size increase
mysql-test/suite/sys_vars/r/aria_sort_buffer_size_basic.result:
Updated result after sort buffer size increase
mysql-test/suite/sys_vars/r/myisam_sort_buffer_size_basic.result:
Updated result after sort buffer size increase
mysql-test/suite/sys_vars/t/myisam_sort_buffer_size_basic.test:
Updated result after sort buffer size increase
mysql-test/t/myisam.test:
Fixed error messages to not print system specific data
mysql-test/t/repair.test:
Fixed error messages to not print system specific data
storage/maria/ha_maria.cc:
Print also to 'show warnings' if repair was retried
Changed default size of sort_buffer_size from 128M to 256M (same as in mysqld)
storage/maria/ma_check.c:
Renamed USE_BUFFER_INIT -> PAGE_BUFFER_INIT
storage/maria/ma_sort.c:
Increased size of internal sort-buffer-readers from 16K to 128K
Increased size of types to be able to handle more records
Added support for T_FORCE_SORT_MEMORY
Don't allocate too many extra BUFFPEK at a time (they are probably not needed)
Improved error message for "sort_buffer_size is too small"
Changed printing of 'number of records' to use %ll instead of casting to long
Fixed bug where maria_update_key_parts() was called too early.
Fixed bug in detecting result from read_to_buffer().
Added 'out of memory' checking when calling 'alloc_dynamic()'.
storage/maria/maria_chk.c:
Added --create-missing-keys
If one specifies --force twice then we will try to finnish the repair even if sort_buffer would be too small.
check_param.sort_buffer_length varialble was used with wrong type.
storage/maria/maria_def.h:
Increased size of types to be able to handle more records
Use MY_ALIGN_DOWN() to get same number of bytes allocated on different machines
Renamed USE_BUFFER_INIT -> PAGE_BUFFER_INIT
storage/maria/maria_ftdump.c:
Renamed USE_BUFFER_INIT -> PAGE_BUFFER_INIT
storage/maria/maria_read_log.c:
Use PAGE_BUFFER_INIT for page cache
storage/myisam/ha_myisam.cc:
Changed default size of myisam_sort_buffer_size from 8M to 128M
storage/myisam/mi_check.c:
Renamed USE_BUFFER_INIT -> KEY_BUFFER_INIT
storage/myisam/myisam_ftdump.c:
Renamed USE_BUFFER_INIT -> KEY_BUFFER_INIT
storage/myisam/myisamchk.c:
Added --create-missing-keys
If one specifies --force twice then we will try to finnish the repair even if sort_buffer would be too small.
check_param.sort_buffer_length varialble was used with wrong type.
Renamed USE_BUFFER_INIT -> KEY_BUFFER_INIT
storage/myisam/myisamdef.h:
Increased SORT_BUFFER_INIT to 64M
(speeds up repair a lot and most machines have nowadays a lot of memory)
Use MY_ALIGN_DOWN() to get same number of bytes allocated on different machines
Renamed USE_BUFFER_INIT -> KEY_BUFFER_INIT
storage/myisam/sort.c:
Increased size of internal sort-buffer-readers from 16K to 128K
Increased size of types to be able to handle more records
Added support for T_FORCE_SORT_MEMORY
Don't allocate too many extra BUFFPEK at a time (they are probably not needed)
Improved error message for "sort_buffer_size is too small"
Changed printing of 'number of records' to use %ll instead of casting to long
Fixed bug in detecting result from read_to_buffer().
Added 'out of memory' checking when calling 'alloc_dynamic()'.
The patch lifts the limitation of the current implementation
of ALTER TABLE that does not allow to build unique/primary
indexes by sort for MyISAM and Aria engines.
and small collateral changes
mysql-test/lib/My/Test.pm:
somehow with "print" we get truncated writes sometimes
mysql-test/suite/perfschema/r/digest_table_full.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/perfschema/r/dml_handler.result:
host table is not ported over yet
mysql-test/suite/perfschema/r/information_schema.result:
host table is not ported over yet
mysql-test/suite/perfschema/r/nesting.result:
this differs, because we don't rewrite general log queries, and multi-statement
packets are logged as a one entry. this result file is identical to what mysql-5.6.5
produces with the --log-raw option.
mysql-test/suite/perfschema/r/relaylog.result:
MariaDB modifies the binlog index file directly, while MySQL 5.6 has a feature "crash-safe binlog index" and modifies a special "crash-safe" shadow copy of the index file and then moves it over. That's why this test shows "NONE" index file writes in MySQL and "MANY" in MariaDB.
mysql-test/suite/perfschema/r/server_init.result:
MariaDB initializes the "manager" resources from the "manager" thread, and starts this thread only when --flush-time is not 0. MySQL 5.6 initializes "manager" resources unconditionally on server startup.
mysql-test/suite/perfschema/r/stage_mdl_global.result:
this differs, because MariaDB disables query cache when query_cache_size=0. MySQL does not
do that, and this causes useless mutex locks and waits.
mysql-test/suite/perfschema/r/statement_digest.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/perfschema/r/statement_digest_consumers.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/perfschema/r/statement_digest_long_query.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/rpl/r/rpl_mixed_drop_create_temp_table.result:
will be updated to match 5.6 when alfranio.correia@oracle.com-20110512172919-c1b5kmum4h52g0ni and anders.song@greatopensource.com-20110105052107-zoab0bsf5a6xxk2y are merged
mysql-test/suite/rpl/r/rpl_non_direct_mixed_mixing_engines.result:
will be updated to match 5.6 when anders.song@greatopensource.com-20110105052107-zoab0bsf5a6xxk2y is merged
