Like the 10.2 version 1635686b50,
except C++ on internal functions for my_assume_aligned.
volatile != atomic.
volatile has no memory barrier schemantics, its for mmaped IO
so lets allow some optimizer gains and stop pretending it helps
with memory atomicity.
The MDEV lists a SEGV an assumption is made that an address was
partially read. As C packs structs strictly in order and on arm64 the
cache line size is 128 bits. A pointer (link - 64 bits), followed
by a hashnr (uint32 - 32 bits), leaves the following key (uchar *
64 bits), neither naturally aligned to any pointer and worse, split
across a cache line which is the processors view of an atomic
reservation of memory.
lf_dynarray_lvalue is assumed to return a 64 bit aligned address.
As a solution move the 32bit hashnr to the end so we don't get the
*key pointer split across two cache lines.
Tested by: Krunal Bauskar
Reviewer: Marko Mäkelä
Use my_thread_var::stack_ends_here inside lf_pinbox_real_free() for address
where thread stack ends.
Remove LF_PINS::stack_ends_here.
It is not safe to assume that mysys_var that was used during pin allocation,
remains correct during free. E.g with binlog group commit in Innodb,
that frees pins for multiple Innodb transactions, it does not work
correctly.
XID cache is now based on lock-free hash.
Also fixed lf_hash_destroy() to call alloc destructor.
Note that previous implementation had race condition when thread was accessing
XA owned by different thread. This new implementation doesn't fix it either.
Essentially, the problem is that safemalloc is excruciatingly
slow as it checks all allocated blocks for overrun at each
memory management primitive, yielding a almost exponential
slowdown for the memory management functions (malloc, realloc,
free). The overrun check basically consists of verifying some
bytes of a block for certain magic keys, which catches some
simple forms of overrun. Another minor problem is violation
of aliasing rules and that its own internal list of blocks
is prone to corruption.
Another issue with safemalloc is rather the maintenance cost
as the tool has a significant impact on the server code.
Given the magnitude of memory debuggers available nowadays,
especially those that are provided with the platform malloc
implementation, maintenance of a in-house and largely obsolete
memory debugger becomes a burden that is not worth the effort
due to its slowness and lack of support for detecting more
common forms of heap corruption.
Since there are third-party tools that can provide the same
functionality at a lower or comparable performance cost, the
solution is to simply remove safemalloc. Third-party tools
can provide the same functionality at a lower or comparable
performance cost.
The removal of safemalloc also allows a simplification of the
malloc wrappers, removing quite a bit of kludge: redefinition
of my_malloc, my_free and the removal of the unused second
argument of my_free. Since free() always check whether the
supplied pointer is null, redudant checks are also removed.
Also, this patch adds unit testing for my_malloc and moves
my_realloc implementation into the same file as the other
memory allocation primitives.
client/mysqldump.c:
Pass my_free directly as its signature is compatible with the
callback type -- which wasn't the case for free_table_ent.
include/atomic/generic-msvc.h:
prevent possible compiler warnings
include/lf.h:
comments, better definition for LF_HASH_OVERHEAD
include/maria.h:
define MARIA_CANNOT_ROLLBACK here
include/my_pthread.h:
avoid possible name clash
include/waiting_threads.h:
comments, const, move WT_RESOURCE to waiting_threads.c
mysql-test/suite/maria/r/maria_notembedded.result:
new test
mysql-test/suite/maria/t/maria_notembedded.test:
new test - 5-way deadlock
mysys/lf_hash.c:
better definition for LF_HASH_OVERHEAD
mysys/my_static.c:
comment
mysys/my_thr_init.c:
casts
mysys/waiting_threads.c:
comments, asserts, etc
server-tools/instance-manager/parse.cc:
fix my_init_dynamic_array() to follow new calling conventions
sql/mysqld.cc:
call wt_init after set_proper_floating_point_mode
sql/sql_class.h:
comment
storage/maria/ha_maria.cc:
move MARIA_CANNOT_ROLLBACK to a common header
storage/maria/ma_commit.c:
comment
storage/maria/ma_write.c:
comments, check for HA_ERR_FOUND_DUPP_KEY
storage/maria/trnman.c:
comments, assert
storage/maria/trnman.h:
comments
storage/maria/unittest/trnman-t.c:
be paranoid
unittest/mysys/lf-t.c:
comments
unittest/mysys/waiting_threads-t.c:
comments, safety, memory leak
client/mysqltest.c:
compiler warnings
configure.in:
remove old tests for unused programs
disable the use of gcc built-ins if smp assembler atomics were selected explictily.
add waiting_threads.o to THREAD_LOBJECTS
include/lf.h:
replace the end-of-stack pointer with the pointer to the end-of-stack pointer.
the latter could be stored in THD (mysys_vars) and updated in pool-of-threads
scheduler.
constructor/destructor in lf-alloc
include/my_pthread.h:
shuffle set_timespec/set_timespec_nsec macros a bit to be able to fill
several timeout structures with only one my_getsystime() call
include/waiting_threads.h:
waiting threads - wait-for graph and deadlock detection
mysys/Makefile.am:
add waiting_threads.c
mysys/lf_alloc-pin.c:
replace the end-of-stack pointer with the pointer to the end-of-stack pointer.
the latter could be stored in THD (mysys_vars) and updated in pool-of-threads
scheduler.
constructor/destructor in lf-alloc
mysys/lf_hash.c:
constructor/destructor in lf-alloc
mysys/my_thr_init.c:
remember end-of-stack pointer in the mysys_var
mysys/waiting_threads.c:
waiting threads - wait-for graph and deadlock detection
storage/maria/ha_maria.cc:
replace the end-of-stack pointer with the pointer to the end-of-stack pointer.
the latter could be stored in THD (mysys_vars) and updated in pool-of-threads
scheduler.
storage/maria/ma_commit.c:
replace the end-of-stack pointer with the pointer to the end-of-stack pointer.
the latter could be stored in THD (mysys_vars) and updated in pool-of-threads
scheduler.
storage/maria/trnman.c:
replace the end-of-stack pointer with the pointer to the end-of-stack pointer.
the latter could be stored in THD (mysys_vars) and updated in pool-of-threads
scheduler.
storage/maria/trnman_public.h:
replace the end-of-stack pointer with the pointer to the end-of-stack pointer.
the latter could be stored in THD (mysys_vars) and updated in pool-of-threads
scheduler.
storage/maria/unittest/trnman-t.c:
replace the end-of-stack pointer with the pointer to the end-of-stack pointer.
the latter could be stored in THD (mysys_vars) and updated in pool-of-threads
scheduler.
unittest/mysys/Makefile.am:
add waiting_threads-t
unittest/mysys/lf-t.c:
factor out the common code for multi-threaded stress unit tests
move lf tests to a separate file
unittest/mysys/my_atomic-t.c:
factor out the common code for multi-threaded stress unit tests
move lf tests to a separate file
unittest/mysys/thr_template.c:
factor out the common code for multi-threaded stress unit tests
unittest/mysys/waiting_threads-t.c:
wt tests
Added logging and pinning of pages to block format.
Integration of transaction manager, log handler.
Better page cache intergration
Split trnman.h into two files, so that we don't have to include my_atomic.h into C++ programs.
Renaming of structures, more comments, more debugging etc.
Fixed problem with small head block + long varchar.
Added extra argument to delete_record() and update_record() (needed for UNDO logging)
Small changes to interface of pagecache and log handler.
Change initialization of log_record_type_descriptors to not be depending on enum order.
Use array of LEX_STRING's to send data to log handler
Added 'dummy' transaction option to MARIA_INFO so that we can always assume 'trn' exists.
include/lf.h:
Interface fixes
Rename of structures
(Patch from Sergei via Sanja)
include/my_atomic.h:
More comments
include/my_global.h:
Added MY_ERRPTR
include/pagecache.h:
Added undo LSN when unlocking pages
mysql-test/r/maria.result:
Updated results
mysql-test/t/maria.test:
Added autocommit around lock tables
(Patch from Sanja)
mysys/lf_alloc-pin.c:
Post-review fixes, simple optimizations
More comments
Struct slot renames
Check amount of memory on stack
(Patch from Sergei)
mysys/lf_dynarray.c:
More comments
mysys/lf_hash.c:
More comments
After review fixes
(Patch from Sergei)
storage/maria/ha_maria.cc:
Split trnman.h into two files, so that we don't have to include my_atomic.h into the .cc program.
(Temporary fix to avoid bug in gcc)
Move out all deferencing of the transaction structure.
Transaction manager integrated (Patch from Sergei)
storage/maria/ha_maria.h:
Added prototype for start_stmt()
storage/maria/lockman.c:
Function call rename
storage/maria/ma_bitmap.c:
Mark deleted pages free from page cache
storage/maria/ma_blockrec.c:
Offset -> rownr
More debugging
Fixed problem with small head block + long varchar
Added logging of changed pages
Added logging of undo (Including only loggging of changed fields in case of update)
Added pinning/unpinning of all changed pages
More comments
Added free_full_pages() as the same code was used in several places.
fill_rows_parts() renamed as fill_insert_undo_parts()
offset -> rownr
Added some optimization of not transactional tables
_ma_update_block_record() has new parameter, as we need original row to do efficent undo for update
storage/maria/ma_blockrec.h:
Added ROW_EXTENTS_ON_STACK
Changed prototype for update and delete of row
storage/maria/ma_check.c:
Added original row to delete_record() call
storage/maria/ma_control_file.h:
Added ifdefs for C++
storage/maria/ma_delete.c:
Added original row to delete_record() call
(Needed for efficent undo logging)
storage/maria/ma_dynrec.c:
Added extra argument to delete_record() and update_record()
Removed not used variable
storage/maria/ma_init.c:
Initialize log handler
storage/maria/ma_loghandler.c:
Removed not used variable
Change initialization of log_record_type_descriptors to not be depending on enum order
Use array of LEX_STRING's to send data to log handler
storage/maria/ma_loghandler.h:
New defines
Use array of LEX_STRING's to send data to log handler
storage/maria/ma_open.c:
Added 'dummy' transaction option to MARIA_INFO so that we can always assume 'trn' exists.
Store in MARIA_SHARE->page_type if pages will have up to date LSN's
storage/maria/ma_pagecache.c:
Don't decrease number of readers when using pagecache_write()/pagecache_read()
In pagecache_write() decrement request count if page was left pinned
Added pagecache_delete_pages()
Removed some casts
Make trace output consistent with rest of code
Simplify calling of DBUG_ASSERT(0)
Only update LSN if the LSN is bigger than what's already on the page
Added LSN parameter pagecache_unpin_page(), pagecache_unpin(), and pagecache_unlock()
(Part of patch from Sanja)
storage/maria/ma_static.c:
Added 'dummy' transaction option to MARIA_INFO so that we can always assume 'trn' exists.
Added default page cache
storage/maria/ma_statrec.c:
Added extra argument to delete_record() and update_record()
storage/maria/ma_test1.c:
Added option -T for transactions
storage/maria/ma_test2.c:
Added option -T for transactions
storage/maria/ma_test_all.sh:
Test with transactions
storage/maria/ma_update.c:
Changed prototype for update of row
storage/maria/maria_def.h:
Changed prototype for update & delete of row as block records need to access the old row
Store in MARIA_SHARE->page_type if pages will have up to date LSN's
Added MARIA_MAX_TREE_LEVELS to allow us to calculate the number of possible pinned pages we may need.
Removed not used 'empty_bits_buffer'
Added pointer to transaction object
Added array for pinned pages
Added log_row_parts array for logging of field data.
Added MARIA_PINNED_PAGE to store pinned pages
storage/maria/trnman.c:
Added accessor functions to transaction object
Added missing DBUG_RETURN()
More debugging
More comments
Changed // comment of code to #ifdef NOT_USED
Transaction manager integrated.
Post review fixes
Part of patch originally from Sergei
storage/maria/trnman.h:
Split trnman.h into two files, so that we don't have to include my_atomic.h into the .cc program.
(Temporary fix to avoid bug in gcc)
storage/maria/unittest/ma_pagecache_single.c:
Added missing argument
Added SKIP_BIG_TESTS
(Patch from Sanja)
storage/maria/unittest/ma_test_loghandler-t.c:
Test logging with new LEX_STRING parameter
(Patch from Sanja)
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
Test logging with new LEX_STRING parameter
(Patch from Sanja)
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
Test logging with new LEX_STRING parameter
(Patch from Sanja)
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
Test logging with new LEX_STRING parameter
(Patch from Sanja)
storage/maria/unittest/trnman-t.c:
Stack overflow detection
(Patch from Sergei)
unittest/unit.pl:
Command-line options --big and --verbose
(Patch from Sergei)
unittest/mytap/tap.c:
Detect --big
(Patch from Sergei)
unittest/mytap/tap.h:
Skip_big_tests and SKIP_BIG_TESTS
(Patch from Sergei)
storage/maria/trnman_public.h:
New BitKeeper file ``storage/maria/trnman_public.h''
Removed compiler warnings
Fixed clashing function name in maria
Disable maria tests from MySQL level for now
BitKeeper/deleted/.del-ha_maria.cc:
Rename: libmysqld/ha_maria.cc -> BitKeeper/deleted/.del-ha_maria.cc
BitKeeper/etc/ignore:
added libmysqld/ha_maria.cc
---
added storage/maria/unittest/maria_control unittest/maria_control
---
added *.Tpo
---
added unittest/page_cache_test_file_1
---
added unittest/pagecache_debug.log
---
added unittest/mysys/mf_pagecache_consist_1k-t-big unittest/mysys/mf_pagecache_consist_1kHC-t-big unittest/mysys/mf_pagecache_consist_1kRD-t-big unittest/mysys/mf_pagecache_consist_1kWR-t-big unittest/mysys/mf_pagecache_consist_64k-t-big unittest/mysys/mf_pagecache_consist_64kHC-t-big unittest/mysys/mf_pagecache_consist_64kRD-t-big unittest/mysys/mf_pagecache_consist_64kWR-t-big
---
added unittest/mysys/mf_pagecache_single_64k-t-big
Makefile.am:
Don't run 'test-unit' by default (takes too long time)
client/mysqldump.c:
Fixed compiler warning
include/lf.h:
Remove compiler warnings about not used require_pins constant
include/pagecache.h:
LSN should be of type ulonglong
(This fixes some compiler warnings)
mysql-test/r/events_logs_tests.result:
Make test predictable
mysql-test/r/view.result:
Make test results predictable
mysql-test/t/disabled.def:
Disable maria tests for a while
mysql-test/t/events_logs_tests.test:
Make test predictable
mysql-test/t/view.test:
Make test results predictable
mysys/lf_alloc-pin.c:
#warning ->QQ
mysys/lf_hash.c:
#warning ->QQ
Removed compiler warnings
mysys/mf_pagecache.c:
Removed compiler warnings
mysys/my_rename.c:
Removed compiler warnings
plugin/daemon_example/daemon_example.c:
Remove compiler warning
sql/ha_ndbcluster.cc:
Remove compiler warning
sql/udf_example.c:
Remove compiler warning
storage/maria/lockman.c:
Changed #warnings to QQ comment
Removed compiler warnings
storage/maria/ma_blockrec.c:
Removed compiler warnings
storage/maria/ma_check.c:
After merge fixes
storage/maria/ma_key.c:
After merge fixes
storage/maria/ma_packrec.c:
After merge fixes
storage/maria/ma_rkey.c:
After merge fixes
storage/maria/ma_sort.c:
After merge fixes
storage/maria/ma_sp_defs.h:
Rename clashing function name
storage/maria/ma_sp_key.c:
Rename clashing function name
storage/maria/ma_test_all.res:
New test results
storage/maria/ma_unique.c:
Fixed compiler warning
storage/maria/tablockman.c:
#warning -> QQ
storage/maria/tablockman.h:
#warning -> QQ
storage/maria/trnman.c:
#warning -> QQ
storage/maria/unittest/lockman2-t.c:
Removed compiler warnings
storage/maria/unittest/ma_control_file-t.c:
Removed warning for 'maria_control' file not found
storage/maria/unittest/trnman-t.c:
Removed compiler warnings
storage/ndb/src/mgmapi/mgmapi.cpp:
Remove compiler warnings
unittest/mysys/mf_pagecache_consist.c:
Removed compiler warnings
unittest/mysys/my_atomic-t.c:
Removed compiler warnings
And reducing the time taken by my_atomic-t.
include/lf.h:
fix to be able to add a ";" at the end of the macro's invokation
(removing it here, removes a warning from "gcc -ansi" about a
standalone ";").
mysys/lf_hash.c:
";" is ok now after LF_REQUIRE_PINS
mysys/mf_pagecache.c:
comment fix
unittest/mysys/my_atomic-t.c:
decreasing number of iterations to make test take less time.
* merging changes done to the key cache since May 2006 into Maria
* enabling two small enough page cache's unit tests by default
* fix to have non-buffered output in unit tests (to not have a false
timeout killing in pushbuild) (patch given by Serg)
* removing some warnings of gcc -ansi
include/lf.h:
getting rid of "warning: ISO C does not allow extra `;' outside of a function"
(gcc -ansi)
mysys/lf_hash.c:
getting rid of "warning: ISO C does not allow extra `;' outside of a function"
(gcc -ansi)
mysys/mf_pagecache.c:
Cosmetic changes to minimize the diff with the key cache.
#define PAGECACHE_DEBUG_LOG is not needed (just define PAGECACHE_DEBUG
if you want) (this change removes "warning: 'pagecache_debug_print'
declared `static' but never defined").
Importing changes made to mf_keycache.c since May 2006, into the page cache.
Disabling online resizing in the page cache.
Fix for "warning: ISO C90 forbids mixed declarations and code".
unittest/mysys/Makefile.am:
Of the page cache's unit tests, two are small enough to run on pushbuild,
renaming them to a -t suffix.
unittest/mytap/tap.c:
pushbuild kills a test after seeing no output from it for 10 minutes;
so we set the mytap framework to not buffer output (patch given by
Serg) so that output is seen more frequently and not "all at the end
of the test".
storage/maria/trnman.c:
comments
include/my_dbug.h:
make DBUG_ASSERT always a statement
storage/maria/lockman.h:
comments
include/lf.h:
lf_pinbox - don't use a fixed-size purgatory.
mysys/lf_alloc-pin.c:
lf_pinbox - don't use a fixed-size purgatory.
mysys/lf_hash.c:
lf_pinbox - don't use a fixed-size purgatory.
storage/maria/lockman.c:
removed IGNORE_ME/UPGDARED matching - it was wrong in the first place.
updated for "lf_pinbox - don't use a fixed-size purgatory"
storage/maria/unittest/lockman-t.c:
IGNORE_ME/UPGRADED pair counting bugtest.
more tests
unittest/mysys/my_atomic-t.c:
lf_pinbox - don't use a fixed-size purgatory.
(lockmanager still fails unit tests)
BitKeeper/deleted/.del-Makefile.am~4375ae3d4de2bdf0:
Delete: unittest/maria/Makefile.am
configure.in:
silence up configure warnings, don't generate unittest/maria/Makefile
include/atomic/nolock.h:
s/LOCK/LOCK_prefix/
include/atomic/x86-gcc.h:
s/LOCK/LOCK_prefix/
include/atomic/x86-msvc.h:
s/LOCK/LOCK_prefix/
include/lf.h:
pin asserts, renames
include/my_atomic.h:
move cleanup
include/my_bit.h:
s/uint/uint32/
mysys/lf_dynarray.c:
style fixes, split for() in two, remove if()s
mysys/lf_hash.c:
renames, minor fixes
mysys/my_atomic.c:
run-time assert -> compile-time assert
storage/maria/Makefile.am:
lockman here
storage/maria/unittest/Makefile.am:
new unit tests
storage/maria/unittest/trnman-t.c:
lots of changes
storage/maria/lockman.c:
many changes:
second meaning of "blocker"
portability: s/gettimeofday/my_getsystime/
move mutex/cond out of LOCK_OWNER - it creates a race condition
that will be fixed in a separate changeset
increment lm->count for every element, not only for distinct ones -
because we cannot decrease it for distinct elements only :(
storage/maria/lockman.h:
move mutex/cond out of LOCK_OWNER
storage/maria/trnman.c:
move mutex/cond out of LOCK_OWNER
atomic-ops to access short_trid_to_trn[]
storage/maria/trnman.h:
move mutex/cond out of LOCK_OWNER
storage/maria/unittest/lockman-t.c:
unit stress test
include/lf.h:
few lf API changes
mysys/lf_alloc-pin.c:
few lf API changes
mysys/lf_dynarray.c:
few lf API changes
mysys/lf_hash.c:
few lf API changes
storage/maria/Makefile.am:
transaction manager
unittest/Makefile.am:
maria transaction manager
unittest/mysys/my_atomic-t.c:
ensure that values are positive
storage/maria/trxman.h:
New BitKeeper file ``storage/maria/trxman.h''
unittest/maria/Makefile.am:
New BitKeeper file ``unittest/maria/Makefile.am''
unittest/maria/trxman-t.c:
New BitKeeper file ``unittest/maria/trxman-t.c''
storage/maria/trxman.c:
comment clarified
lock-free alloc (WL#3229), lock-free hash (WL#3230)
bit functions made inline
include/Makefile.am:
lf.h added
mysys/Makefile.am:
lf_hash.c lf_dynarray.c lf_alloc-pin.c
include/atomic/nolock.h:
amd64 atomic ops
include/atomic/rwlock.h:
s/rw_lock/mutex/g
include/atomic/x86-gcc.h:
amd64 atomic ops
try PAUSE
include/my_global.h:
STATIC_INLINE
mysys/mf_keycache.c:
make bit functions inline
mysys/my_atomic.c:
STATIC_INLINE
mysys/my_bitmap.c:
make bit functions inline
sql/ha_myisam.cc:
make bit functions inline
sql/item_func.cc:
make bit functions inline
include/my_atomic.h:
STATIC_INLINE
mysys/my_bit.c:
make bit functions inline
sql/sql_select.cc:
make bit functions inline
storage/myisam/mi_create.c:
make bit functions inline
storage/myisam/mi_test2.c:
make bit functions inline
storage/myisam/myisamchk.c:
make bit functions inline
mysys/my_init.c:
thread_size moved to mysys
sql/mysql_priv.h:
thread_size moved to mysys
sql/set_var.cc:
thread_size moved to mysys
include/my_sys.h:
thread_size moved to mysys
sql/mysqld.cc:
thread_size moved to mysys
sql/sql_parse.cc:
thread_size moved to mysys
sql/sql_test.cc:
thread_size moved to mysys
include/lf.h:
dylf_dynarray refactored to remove 65536 elements limit
mysys/lf_alloc-pin.c:
dylf_dynarray refactored to remove 65536 elements limit
mysys/lf_dynarray.c:
dylf_dynarray refactored to remove 65536 elements limit
mysys/lf_hash.c:
dylf_dynarray refactored to remove 65536 elements limit
unittest/mysys/my_atomic-t.c:
fix to commit (remove debug code)
