Commit graph

156 commits

Author SHA1 Message Date
marko
d9317eb699 branches/zip: Non-functional change:
buf_block_dbg_add_level(block, level): Define as an empty macro when
UNIV_SYNC_DEBUG is not defined.  Remove #ifdef UNIV_SYNC_DEBUG around
all invocations.
2008-09-22 06:59:58 +00:00
marko
0d8f5237dd branches/zip: Introduce UNIV_LRU_DEBUG for debugging the LRU buffer pool
cache, especially buf_pool->LRU_old and bpage->old.

buf_LRU_old_adjust_len(), buf_LRU_remove_block(): Check that blocks in
buf_pool->LRU_old have the "old" flag set and the blocks preceding
buf_pool->LRU_old have the "old" flag clear.

buf_LRU_add_block_low(), buf_relocate(): Check that buf_pool->LRU_old
is the first block in the LRU list whose "old" flag is set.

buf_LRU_free_block(): When replacing a control block in the LRU list
with a control block for a compressed page, assert that the "old"
flags in the neighboring LRU list entries grow monotonically.

buf_page_set_old(): Assert that the "old" flags in the neighboring LRU
list entries grow monotonically.
2008-08-18 18:28:20 +00:00
marko
9c0d868a8d branches/zip: Correct some typos in comments. 2008-08-06 12:45:35 +00:00
marko
cd124b7524 branches/zip: Always display a newline after ut_print_buf(stderr, ...). 2008-08-06 08:10:21 +00:00
sunny
33c5bfc756 branches/zip: Cover the diagnostic print statement with a suitable lock. It
caused an assertion failure during testing.
2008-06-17 19:15:37 +00:00
vasil
b21c94dc01 branches/zip: Merge 2437:2485 from branches/5.1:
(r2478 was skipped for the obvious reason)

  ------------------------------------------------------------------------
  r2464 | vasil | 2008-05-19 17:59:42 +0300 (Mon, 19 May 2008) | 9 lines
  
  branches/5.1:
  
  Fix Bug#36600 SHOW STATUS takes a lot of CPU in buf_get_latched_pages_number
  
  by removing the Innodb_buffer_pool_pages_latched variable from
  SHOW STATUS output in non-UNIV_DEBUG compilation.
   
  Approved by:	Heikki
  
  ------------------------------------------------------------------------
  r2466 | calvin | 2008-05-20 01:37:14 +0300 (Tue, 20 May 2008) | 12 lines
  
  branches/5.1:
  
  Fix Bug#11894 innodb_file_per_table crashes w/ Windows .sym symbolic
  link hack
  
  The crash was due to un-handled error 3 (path not found). In the case
  of file per table, change the call to os_file_handle_error_no_exit()
  from os_file_handle_error(). Also, checks for full path pattern during
  table create (Windows only), which is used in symbolic link and temp
  table creation.
  
  Approved by:	Heikki
  ------------------------------------------------------------------------
  r2478 | sunny | 2008-05-23 08:29:08 +0300 (Fri, 23 May 2008) | 3 lines
  
  branches/5.1: Fix for bug# 36793. This is a back port from branches/zip. This
  code has been tested on a big-endian machine too.
  
  ------------------------------------------------------------------------
  r2480 | vasil | 2008-05-27 11:40:07 +0300 (Tue, 27 May 2008) | 11 lines
  
  branches/5.1:
  
  Fix Bug#36819 ut_usectime does not handle errors from gettimeofday
  
  by retrying gettimeofday() several times if it fails in ut_usectime().
  If it fails on all calls then return error to the caller to be handled
  at higher level.
  
  Update the variable innodb_row_lock_time_max in SHOW STATUS output only
  if ut_usectime() was successful.
  
  ------------------------------------------------------------------------
  r2482 | sunny | 2008-05-28 12:18:35 +0300 (Wed, 28 May 2008) | 5 lines
  
  branches/5.1: Fix for Bug#35602, "Failed to read auto-increment value from
  storage engine". The test for REPLACE was an error of ommission since it's
  classified as a simple INSERT. For REPLACE statements we don't acquire
  the special AUTOINC lock for AUTOINC_NEW_STYLE_LOCKING with this fix.
  
  ------------------------------------------------------------------------
  r2485 | vasil | 2008-05-28 16:01:14 +0300 (Wed, 28 May 2008) | 9 lines
  
  branches/5.1:
  
  Fix Bug#36149 Read buffer overflow in srv0start.c found during "make test"
  
  Use strncmp(3) instead of memcmp(3) to avoid reading past end of the string
  if it is empty (*str == '\0'). This bug is _not_ a buffer overflow.
  
  Discussed with:	Sunny (via IM)
  
  ------------------------------------------------------------------------
2008-06-06 15:24:58 +00:00
sunny
fd41c6a315 branches/zip: The block can't be in io fix state WRITE and we check for
that in the fix but the earlier check fails because it tries to be too
strict and assumes that if the block is on the clean list then it can
only be in state io fix NONE.
2008-04-12 10:04:28 +00:00
marko
77d8c98735 branches/zip: Rename ib_ulonglong to ib_uint64_t and ib_longlong to ib_int64_t. 2008-03-17 14:19:04 +00:00
marko
9d58e3ff99 branches/zip: Improve the LRU algorithm with a separate unzip_LRU list of
blocks that contains uncompressed and compressed frames.  This patch was
designed by Heikki and Inaam, implemented by Inaam, and refined and reviewed
by Marko and Sunny.

buf_buddy_n_frames, buf_buddy_min_n_frames, buf_buddy_max_n_frames: Remove.

buf_page_belongs_to_unzip_LRU(): New predicate:
bpage->zip.data && buf_page_get_state(bpage) == BUF_BLOCK_FILE_PAGE.

buf_pool_t, buf_block_t: Add the linked list unzip_LRU.  A block in the
regular LRU list is in unzip_LRU iff buf_page_belongs_to_unzip_LRU() holds.

buf_LRU_free_block(): Add a third return value to refine the case
"cannot free the block".

buf_LRU_search_and_free_block(): Update the documentation to reflect the
implementation.

buf_LRU_stat_t, buf_LRU_stat_cur, buf_LRU_stat_sum, buf_LRU_stat_arr[]:
Statistics for the unzip_LRU algorithm.

buf_LRU_stat_update(): New function: Update the statistics.  Called once
per second by srv_error_monitor_thread().

buf_LRU_validate(): Validate the unzip_LRU list as well.

buf_LRU_evict_from_unzip_LRU(): New predicate: Use the unzip_LRU before
falling back to the regular LRU?

buf_LRU_free_from_unzip_LRU_list(), buf_LRU_free_from_common_LRU_list():
Subfunctions of buf_LRU_search_and_free_block().

buf_LRU_search_and_free_block(): Reimplement.  Try to evict an uncompressed
page from the unzip_LRU list before falling back to evicting an entire block
from the common LRU list.

buf_unzip_LRU_remove_block_if_needed(): New function.

buf_unzip_LRU_add_block(): New function: Add a block to the unzip_LRU list.
2008-03-03 12:48:38 +00:00
marko
f7595d4278 branches/zip: buf_relocate(): Document and assert that bpage must be
in one of the states BUF_BLOCK_ZIP_DIRTY or BUF_BLOCK_ZIP_PAGE.
Note that the caller will have to relocate bpage->list.
2008-02-29 08:20:17 +00:00
marko
2c2b06ad75 branches/zip: Introduce UNIV_INTERN, a linkage specifier for InnoDB-global
symbols.  Use it for all definitions of non-static variables and functions.

lexyy.c, make_flex.sh: Declare yylex as UNIV_INTERN, not static.  It is
referenced from pars0grm.c.

Actually, according to
	nm .libs/ha_innodb.so|grep -w '[ABCE-TVXYZ]'
the following symbols are still global:

* The vtable for class ha_innodb
* pars0grm.c: The function yyparse() and the variables yychar, yylval, yynerrs

The required changes to the Bison-generated file pars0grm.c will be addressed
in a separate commit, which will add a script similar to make_flex.sh.

The class ha_innodb is renamed from class ha_innobase by a #define.  Thus,
there will be no clash with the builtin InnoDB.  However, there will be some
overhead for invoking virtual methods of class ha_innodb.  Ideas for making
the vtable hidden are welcome.  -fvisibility=hidden is not available in GCC 3.
2008-02-06 14:17:36 +00:00
marko
473d7c16a9 branches/zip: Introduce the page type code FIL_PAGE_TYPE_ZBLOB2 for
continuation pages containing compressed BLOBs.  The first compressed
BLOB page will be of type FIL_PAGE_TYPE_ZBLOB.
2008-01-24 08:12:02 +00:00
marko
66791e5a1b branches/zip: Free the buffer pool at shutdown.
buf_pool_free(): New function: Free all chunks of the buffer pool.

innobase_shutdown_for_mysql(): Call buf_pool_free() right before
ut_free_all_mem().
2008-01-16 12:44:44 +00:00
marko
7d18bba5e7 branches/zip: Add comments about the lock and latch protection of externally
stored columns (BLOBs).

btr_copy_blob_prefix(), btr_copy_zblob_prefix(),
btr_copy_externally_stored_field_prefix_low(),
btr_copy_externally_stored_field_prefix(),
btr_copy_externally_stored_field(),
btr_rec_copy_externally_stored_field():
Note that the page containing the clustered index record that points to
the BLOB must be latched.

btr_copy_zblob_prefix(): Note that there is no latch on the page, and thus
all accesses to a given page via this function must be covered by the same
set of locks or latches.

btr_copy_zblob_prefix(): Note that the block acquired by
buf_page_get_zip() is protected by an exclusive table lock or
or by a latch on the clustered index record.
2008-01-16 10:45:14 +00:00
marko
5c95ec4035 branches/zip: buf_pool_init(): Initialize buf_pool_zip_mutex before
acquiring buf_pool_mutex.  This avoids triggering the debug assertion
that was added in r2227.
2008-01-15 09:43:14 +00:00
marko
523a8a0e8d branches/zip: Add instrumentation for prohibiting the release of
the buffer pool mutex.  The instrumentation can be activated by
defining UNIV_DEBUG or UNIV_BUF_DEBUG at compilation time.

buf_pool_mutex_exit_forbidden: New variable.  When this is nonzero,
an assertion will fail in buf_pool_mutex_exit().

buf_pool_mutex_exit_forbid(): Macro for declaring that the buffer pool
mutex must not be released.  Calls may be nested.

buf_pool_mutex_exit_allow(): Macro for declaring that the buffer pool
mutex may be released.  Calls may be nested.

buf_LRU_search_and_free_block(): Prohibit buf_pool_mutex_exit() in the
scope of the function.

buf_LRU_free_block(): Prohibit buf_pool_mutex_exit() in buf_buddy_alloc()
and buf_buddy_free().

buf_LRU_block_remove_hashed_page(): Prohibit buf_pool_mutex_exit()
in buf_buddy_free().
2008-01-10 09:51:57 +00:00
marko
3bb405841f branches/zip: Implement wrappers for all operations on the buffer pool mutex.
buf_pool->mutex: Rename to buf_pool_mutex, so that the wrappers will have
to be used when changes are merged from other source trees.

buf_pool->zip_mutex: Rename to buf_pool_zip_mutex.

buf_pool_mutex_own(), buf_pool_mutex_enter(), buf_pool_mutex_exit():
Wrappers for buf_pool_mutex.
2008-01-10 09:37:13 +00:00
vasil
815bb2ccf4 branches/zip:
Non-functional change: add "out:" comment for the return value.
2007-12-17 10:03:15 +00:00
marko
a563fbc3d6 branches/zip: Clarify that buf_buddy_alloc() should only be used for
allocating compressed page frames or their control blocks.  Also note
that if buf_buddy_alloc() is used for allocating a control block,
it must be initialized before releasing buf_pool->mutex.

buf_page_init_for_read(): When the page hash check fails after
buf_buddy_alloc(), free the uninitialized control block before freeing
the compressed page frame.  This fixes a potential error in
buf_buddy_relocate_block().
2007-12-12 14:12:52 +00:00
marko
902615e374 branches/zip: Document how the data structures of the buddy allocator
are interfaced with the buffer pool.
2007-12-12 13:42:03 +00:00
marko
40dce922cc branches/zip: buf_page_get_gen(): Check the return status of
buf_zip_decompress() and return NULL on decompression failure.
2007-12-10 12:54:53 +00:00
marko
6f306fad26 branches/zip: buf_buddy_alloc(): Assign *lru = TRUE whenever the buffer pool
mutex is temporarily released.

buf_LRU_free_block(), buf_buddy_alloc_clean(): Add an output parameter that
will be assigned TRUE when the buffer pool mutex is released.

This bug was spotted by and fix provided by Sunny.
2007-12-10 09:48:28 +00:00
marko
7a1d5d91b7 branches/zip: Fix a bug that was introduced in r2123.
buf_block_is_uncompressed(): Check that the pointer is aligned.  Use the
C modulus operator % instead of ut_align_offset(), because sizeof(buf_block_t)
is not guaranteed to be a power of 2.
2007-11-28 13:45:22 +00:00
marko
f6a5cc1d47 branches/zip: buf_page_get_gen(): Note that the guessed block may also
point to a buffer pool chunk that has been released when resizing the
buffer pool.

buf_block_is_uncompressed(): Check that the pointer is aligned.  Thanks
to this check, it is safe to pass an arbitrary pointer as a guess
to buf_page_get_gen().
2007-11-28 11:31:12 +00:00
marko
7aedcf308e branches/zip: buf_page_get_gen(): Ignore "guess" if it does not point to
buf_pool->chunks[]->blocks[].

buf_block_is_uncompressed(): New function, to detect if a block points to
buf_pool->chunks[]->blocks[].
2007-11-05 10:31:41 +00:00
marko
bcd672cae7 branches/zip: buf_page_init_low(): Initialize also flush_type. 2007-10-31 22:02:23 +00:00
marko
7319444c08 branches/zip: Improve Valgrind instrumentation.
buf_LRU_free_block(): Check that the block descriptor contains valid data.

buf_buddy_relocate(): Check that the source block contains valid data.

buf_page_get_gen(): Do not dereference bpage after calling buf_relocate().
This avoids a bogus Valgrind warning; the memory itself was valid.

buf_page_hash_get(): Check that the returned block descriptor contains
valid data.
2007-10-30 09:27:09 +00:00
marko
ada2f71ab7 branches/zip: Reduce WAIT_FOR_READ from 20 to 5 milliseconds, to correspond
to the time in which modern disks can serve a random disk read.
2007-10-26 12:34:53 +00:00
marko
a4b26c4ffb branches/zip: buf_pool_t: Add n_pend_unzip. Display it in buf_print(). 2007-10-26 08:53:36 +00:00
marko
1890ec59c7 branches/zip: buf_page_get_gen(): Improve the comments about
wait_until_unfixed.
2007-10-26 08:48:01 +00:00
marko
aa8c8329f2 branches/zip: buf_pool_init(): Allocate buf_pool with mem_zalloc(). 2007-10-25 12:54:18 +00:00
marko
a14eb7445c branches/zip: buf_page_get_gen(): Reinitialize guess when re-entering the loop. 2007-10-25 12:51:18 +00:00
marko
37309546e0 branches/zip: Cast void* pointers to byte* when doing pointer arithmetics. 2007-10-25 07:03:02 +00:00
marko
0349b9d235 branches/zip: buf_chunk_init(): Note that the function does not always
return the requested amount of memory.
2007-10-15 11:26:58 +00:00
marko
3d57033064 branches/zip: Optimize buf_page_try_get_func().
buf_block_hash_get(): New function, similar to buf_page_hash_get().

buf_page_get_block(): Remove the const qualifiers.  This is a low-level
function, and the operations on block->mutex are non-const.

buf_page_try_get_func(): Implement with lower-level predicates, somewhat
similar to buf_page_get_known_nowait().

lock_rec_print(): Remove the unused variable zip_size and the
call to fil_space_get_zip_size().  Adapt to buf_page_try_get() returning
a const pointer.
2007-10-01 07:32:45 +00:00
marko
1110b563b2 branches/zip: Introduce the function buf_block_fix_dec(), similar to
buf_block_fix_inc(), and use it.
2007-10-01 07:25:02 +00:00
vasil
e05b8e86b9 branches/zip:
Move part of the code from lock_rec_print() in a separate function
buf_page_try_get() because the same functionality is needed in
INFORMATION SCHEMA code.

Approved by:	Heikki
2007-09-27 14:35:18 +00:00
inaam
ae1bf9e3e9 branches/zip: undo changes made in r1763.
These should have gone in branches/fts

Spotted by: Marko and Ken
2007-09-11 00:38:13 +00:00
inaam
8d96873f9d branches/zip: The freed_page_clock for both buf_pool and block starts ticking
only after the buffer_cache has warmed up. During the initial warmup
phase these values are set to zero. We should recommend to move the
block to the start of the LRU in this case.

Also added some comments to buf0buf.c about the LRU logic.

reviewed by: Sunny
2007-08-27 19:41:56 +00:00
marko
d662c3c656 branches/zip: Merge 1575:1664 from trunk. 2007-08-01 11:18:43 +00:00
marko
fbc6f37cb1 branches/zip: Fix two bugs.
mtr_commit(): Do not acquire the log mutex if no log records are to be written.

ibuf_set_free_bits_func(): Do not clear mtr.modifications, because that would
prevent the dirty insert buffer bitmap block from being moved to the flush
list.  Instead, assume that mtr_commit() will not acquire log_sys->mutex
because redo logging is disabled.

buf_page_init_for_read(): I/O-fix the block before buf_buddy_alloc(), because
the newly allocated block could otherwise be reused by the compressed page.
2007-05-07 05:53:42 +00:00
marko
dd529ee402 branches/zip: When reading pages for merging buffered inserts, request that
the pages be decompressed.

buf_read_page_low(), buf_page_init_for_read(): Add parameter "unzip".
2007-05-06 13:21:59 +00:00
marko
2b060715ef branches/zip: Split UNIV_IBUF_COUNT_DEBUG from UNIV_IBUF_DEBUG. 2007-02-27 11:56:38 +00:00
marko
37b0f7bf1c branches/zip: Enable the insert buffer on compressed tablespaces.
page_zip_max_ins_size(): New function.

btr_cur_optimistic_insert(), btr_cur_optimistic_delete(),
btr_page_split_and_insert(), btr_compress(): Do not update the
ibuf free bits for non-leaf pages or pages belonging to a clustered index.
The insert buffer only covers operations on leaf pages of secondary indexes.
For pages covered by the insert buffer, limit the max_ins_size to
page_zip_max_ins_size().

buf_page_get_gen(): Merge the insert buffer after decompressing the page.

buf_page_io_complete(): Relax the assertion about ibuf_count.  For
compressed-only pages, the insert buffer merge takes place
in buf_page_get_gen().

ibuf_index_page_calc_free_bits(), ibuf_index_page_calc_free_from_bits(),
ibuf_index_page_calc_free(), ibuf_update_free_bits_if_full(),
ibuf_update_free_bits_low(), ibuf_update_free_bits_for_two_pages_low(),
ibuf_set_free_bits_low(): Add the parameter zip_size.  Limit the maximum
insert size to page_zip_max_ins_size().
2007-02-19 20:32:06 +00:00
marko
d3a813fd69 branches/zip: buf_page_get_gen(): After allocating an uncompressed page for
decompression, check that the compressed block has not been
buffer-fixed or I/O-fixed.  This race condition was introduced in r1286.
2007-02-13 11:15:27 +00:00
marko
eb03297439 branches/zip: buf_page_init_for_read(): Defer the costly calls to
buf_buddy_alloc() until after the block has been found not to exist
in the buffer pool.

buf_buddy_alloc(), buf_buddy_alloc_low(): Make lru an output parameter.
Improve documentation.

buf_page_init_for_read(): Simplify the code.  Check buf_page_hash_get()
again if buf_buddy_alloc() released buf_pool->mutex.
2007-02-08 14:19:55 +00:00
marko
b36300e1ba branches/zip: buf_page_create(): Do not IO-fix or X-latch the block
when not invoking buf_buddy_alloc().
2007-02-08 12:33:54 +00:00
marko
5ac6f3217d branches/zip: buf_page_init_for_read(): Do not allocate an uncompressed page
frame when reading compressed pages, unless crash recovery is in progress.

buf_page_read_low(): Adapt for buf_page_init_for_read() returning
buf_page_t* instead of buf_block_t*.
2007-02-08 12:03:03 +00:00
marko
7315b06291 branches/zip: buf_page_init_for_read(), buf_page_create(): Note in the
function comments that buf_page_init_for_read() does not perform the
state transition to FILE_PAGE.  This should have been done in r1286.
2007-02-08 09:08:11 +00:00
marko
329c469e70 branches/zip: Do not decompress blocks in buf_page_init_for_read(),
but in buf_page_get_gen().  Adjust comments.  This prevents
unnecessary decompression in read-ahead when the compressed block
exists in the buffer pool.

If the block does not exist in the buffer pool, read-ahead will still
allocate an uncompressed page and decompress the block.

Move buf_block_init_low() and buf_zip_decompress() earlier in the file,
because some compilers are unable to inline functions that are defined
after the invocation.

buf_page_get_gen(): Decompress the block if needed.

buf_page_init_for_read(): Do not decompress.
2007-02-07 13:59:43 +00:00