When using concurrent insert with parallel index reads, it could
happen that reading sessions found keys that pointed to records
yet to be written to the data file. The result was a report of
a corrupted table. But it was false alert.
When inserting a record in a table with indexes, the keys are
inserted into the indexes before the record is written to the data
file. When the insert happens concurrently to selects, an
index read can find a key that references the record that is not
yet written to the data file. To avoid any access to such record,
the select saves the current end of file position when it starts.
Since concurrent inserts are always appended at end of the data
file, the select can easily ignore any concurrently inserted record.
The problem was that the ignore was only done for non-exact key
searches (partial key or using >, >=, < or <=).
The fix is to ignore concurrently inserted records also for
exact key searches.
No test case. Concurrent inserts cannot be tested with the test
suite. Test cases are attached to the bug report.
myisam/mi_rkey.c:
Bug#29838 - myisam corruption using concurrent select ... and update
Fixed mi_rkey() to always ignore records beyond saved eof.
tables
In case system doesn't have native pread/pwrite calls (e.g. Windows)
and there is CHECK TABLE runs concurrently with another statement that
reads from a table, the table may be reported as crashed.
This is fixed by locking file descriptor when my_seek is executed on
MyISAM index file and emulated pread/pwrite may be executed concurrently.
Affects MyISAM tables on platforms that do not have native
pread/pwrite calls (e.g. Windows).
No deterministic test case for this bug.
myisam/mi_check.c:
Key file descriptor is shared among threads and mixed set of
my_pread/my_pwrite and my_seek calls is possible. This is not
a problem on systems that have native pread/pwrite calls.
In case system doesn't have native pread/pwrite calls (e.g. Windows)
we must ensure that my_pread/my_pwrite are not executed at the same
time with my_seek. This is done by passing MY_THREADSAFE flag to
my_seek.
mysys/my_seek.c:
On platforms that do not have native pread/pwrite calls (e.g. Windows)
these calls are emulated as follow: lock fd, lseek, read, unlock fd.
In case file descriptor is shared among threads, where one thread
executes my_pread and another thread executes my_seek, we may read
from a wrong position. This may happen because my_seek doesn't lock
fd and thus may be executed by another thread after emulated pread
has done lseek and before it has done read.
To fix problem mentioned above we introduce new flag MY_THREADSAFE to
my_seek, which is meaningful only in case pread/pwrite calls are
emulated. If this flag is set, lseek operation is performed as follow:
lock fd, seek, unlock fd.
The message is gramatically wrong, and factually wrong.
Change it to refer to the myisam_sort_buffer_size variable and change
"to" to "too".
myisam/sort.c:
Change error messages to be gramatically correct and to refer to the
correct variable.
mysql-test/r/repair.result:
Refer to the correct variable. Message changed.
MERGE engine may return incorrect values when several representations
of equal keys are present in the index. For example "groß" and "gross"
or "gross" and "gross " (trailing space), which are considered equal,
but have different lengths.
The problem was that key length was not recalculated after key lookup.
Only MERGE engine is affected.
myisam/mi_rkey.c:
info->lastkey gets rewritten by mi_search. Later we recalculate found lastkey
length. This is done to make sure that mi_rnext_same gets true, found (not
searched) lastkey length. Searched and found key lengths may be different,
for example in case searched key is "groß" and found is "gross" or in case
a key has trailing spaces.
Unfortunately we recalculate found lastkey length only for first
underlying table. To recalculate found key length for non-first underlying
table we need to know how much key segments were used to create this key.
When mi_rkey is called for first underlying table of a merge table, store
offset to last used key segment.
Restore last_used_keyseg variable when mi_rkey is called for non-first
underlying table.
myisam/myisamdef.h:
Added last_used_keyseg variable to MI_INFO. It is used by merge engine to calculate
key length.
myisammrg/myrg_rkey.c:
Pass last used key segment returned by first table key read to other
table key reads.
mysql-test/r/merge.result:
A test case for bug#24342.
mysql-test/t/merge.test:
A test case for bug#24342.
when index is used
When the table contained TEXT columns with empty contents
('', zero length, but not NULL) _and_ strings starting with
control characters like tabulator or newline, the empty values
were not found in a "records in range" estimate. Hence count(*)
missed these records.
The reason was a different set of search flags used for key
insert and key range estimation.
I decided to fix the set of flags used in range estimation.
Otherwise millions of databases around the world would require
a repair after an upgrade.
The consequence is that the manual must be fixed, which claims
that TEXT columns are compared with "end space padding". This
is true for CHAR/VARCHAR but wrong for TEXT. See also bug 21335.
myisam/mi_range.c:
Bug#26231 - select count(*) on myisam table returns wrong value
when index is used
Added SEARCH_UPDATE to the search flags so that it compares
like write/update/delete operations do. Only so it expects
the keys at the place where they have been inserted.
myisam/mi_search.c:
Bug#26231 - select count(*) on myisam table returns wrong value
when index is used
Added some comments to explain how _mi_get_binary_pack_key()
works.
mysql-test/r/myisam.result:
Bug#26231 - select count(*) on myisam table returns wrong value
when index is used
Added a test.
mysql-test/t/myisam.test:
Bug#26231 - select count(*) on myisam table returns wrong value
when index is used
Added test result.
differences in tables
Certain merge tables were wrongly reported as having incorrect definition:
- Some fields that are 1 byte long (e.g. TINYINT, CHAR(1)), might
be internally casted (in certain cases) to a different type on a
storage engine layer. (affects 4.1 and up)
- If tables in a merge (and a MERGE table itself) had short VARCHAR column (less
than 4 bytes) and at least one (but not all) tables were ALTER'ed (even to an
identical table: ALTER TABLE xxx ENGINE=yyy), table definitions went ouf of
sync. (affects 4.1 only)
This is fixed by relaxing a check for underlying conformance and setting
field type to FIELD_TYPE_STRING in case varchar is shorter than 4
when a table is created.
myisam/mi_create.c:
Added a comment.
mysql-test/r/merge.result:
A test case for bug#26881.
mysql-test/t/merge.test:
A test case for bug#26881.
sql/ha_myisam.cc:
Relaxed some checks performed by check_definition():
As comparing of fulltext keys (and key segments) is not yet implemented,
only return an error in case one of keys is fulltext and other is not.
Otherwise, if both keys are fulltext, accept them as is.
As comparing of spatial keys (and key segments) is not yet implemented,
only return an error in case one of keys is spatial and other is not.
Otherwise, if both keys are spatial, accept them as is.
A workaround to handle situation when field is casted from FIELD_SKIP_ZERO
to FIELD_NORMAL. This could happen only in case field length is 1 and row
format is fixed.
sql/sql_parse.cc:
When a table that has varchar field shorter than 4 is created, field type is
set to FIELD_TYPE_VAR_STRING. Later, when a table is modified using alter
table, field type is changed to FIELD_TYPE_STRING (see Field_string::type).
That means HA_OPTION_PACK_RECORD flag might be lost and thus null_bit might
be shifted by alter table, in other words alter table doesn't create 100%
equal table definition.
This is usually not a problem, since when a table is created/altered,
definition on a storage engine layer is based on one that is passed from
sql layer. But it is a problem for merge engine - null_bit is shifted when
a table (merge or underlying) is altered.
Set field type to FIELD_TYPE_STRING in case FIELD_TYPE_VAR_STRING is shorter
than 4 when a table is created as it is done in Field::type.
incorrect key file for table
In certain cases it could happen that deleting a row could
corrupt an RTREE index.
According to Guttman's algorithm, page underflow is handled
by storing the page in a list for later re-insertion. The
keys from the stored pages have to be inserted into the
remaining pages of the same level of the tree. Hence the
level number is stored in the re-insertion list together
with the page.
In the MySQL RTree implementation the level counts from zero
at the root page, increasing numbers for levels down the tree.
If during re-insertion of the keys the tree height grows, all
level numbers become invalid. The remaining keys will be
inserted at the wrong level.
The fix is to increment the level numbers stored in the
reinsert list after a split of the root block during reinsertion.
myisam/rt_index.c:
Bug#25673 - spatial index corruption, error 126
incorrect key file for table
Added a loop in rtree_delete() to increment the level numbers
stored in the reinsert list after a split of the root block
during reinsertion.
Added comments and DBUG statements.
myisam/rt_key.c:
Bug#25673 - spatial index corruption, error 126
incorrect key file for table
Added DBUG statements.
myisam/rt_split.c:
Bug#25673 - spatial index corruption, error 126
incorrect key file for table
Added DBUG statements.
mysql-test/r/gis-rtree.result:
Bug#25673 - spatial index corruption, error 126
incorrect key file for table
Added the test result.
mysql-test/t/gis-rtree.test:
Bug#25673 - spatial index corruption, error 126
incorrect key file for table
Added a test.
into chilla.local:/home/mydev/mysql-4.1-axmrg
mysql-test/r/symlink.result:
Auto merged
mysql-test/t/symlink.test:
Auto merged
mysql-test/r/myisam.result:
Manual merged
mysql-test/t/myisam.test:
Manual merged
corrupted table
Accessing a table with corrupted column definition results in server
crash.
This is fixed by refusing to open such tables. Affects MyISAM only.
No test case, since it requires crashed table.
myisam/mi_open.c:
Refuse to open MyISAM table with summary columns length bigger than
length of the record.
Accessing fixed record format table with crashed key definition results
in server/myisamchk segmentation fault.
This is fixed by refusing to open such tables. Affects MyISAM only.
No test case, since it requires crashed table.
myisam/mi_open.c:
Refuse to open fixed record format table with key segment that includes
BLOB part (which is true only for tables with crashed key definition).
The function mi_get_pointer_length() computed too small
pointer size for very large tables.
Inserted missing 'else' between the branches for very
large tables.
myisam/mi_create.c:
Bug#24607 - MyISAM pointer size determined incorrectly
Inserted missing 'else' between the branches for very
large tables.
Harmonized literals "(longlong) 1" and "1L" to "ULL(1)"
where they are used for "ulonglong file_length".
mysql-test/r/myisam.result:
Bug#24607 - MyISAM pointer size determined incorrectly
Added the test result.
mysql-test/t/myisam.test:
Bug#24607 - MyISAM pointer size determined incorrectly
Added the test.
Compiler warnings due to non-matching conversion
specifications in format strings in DBUG_PRINT calls,
due to non-used parameters (in non-debug mode), and
due to seemingly uninitialized variables.
Initialized variables, declared parameters unused, and
casted DBUG_PRINT arguments to get rid of warnings.
myisam/mi_range.c:
Bug#25213 - Compiler warnings in MyISAM code
Initialized a variable to get rid of a compiler warning.
myisam/mi_test1.c:
Bug#25213 - Compiler warnings in MyISAM code
Declared an parameter unused to get rid of warnings.
myisam/mi_write.c:
Bug#25213 - Compiler warnings in MyISAM code
Initialized a variable to get rid of a compiler warning.
Casted arguments to DBUG_PRINT to match them with their
format string conversion specification.
myisam/rt_split.c:
Bug#25213 - Compiler warnings in MyISAM code
Initialized variables to get rid of compiler warnings.
Compiler warnings due to non-matching conversion
specifications in format strings in DBUG_PRINT calls.
Fixed DBUG_PRINT format specifiactions.
myisam/mi_packrec.c:
Bug#25208 - Warnings in mi_packrec.c
Fixed DBUG_PRINT format specifiactions.
storage engine returns errno 12
If there is not enough memory to store or update blob record
(while allocating record buffer), myisam marks table as crashed.
With this fix myisam attempts to roll an index back and return
an error, not marking a table as crashed.
Affects myisam tables with blobs only. No test case for this fix.
myisam/mi_dynrec.c:
If there is not enough memory to store or update blob record
(while allocating record buffer), return HA_ERR_OUT_OF_MEM
instead of ENOMEM. In this case storage engine can simply
roll an index back and return an error, not marking table
as crashed.
myisam/mi_update.c:
In some cases do not mark a table as crashed if we run out of
memory. Instead roll an index back and return an error. These
cases are signalled with my_errno set to HA_ERR_OUT_OF_MEM.
myisam/mi_write.c:
In some cases do not mark a table as crashed if we run out of
memory. Instead roll an index back and return an error. These
cases are signalled with my_errno set to HA_ERR_OUT_OF_MEM.
ALTER TABLE DISABLE KEYS doesn't work when modifying the table
ENABLE|DISABLE KEYS combined with another ALTER TABLE option, different
than RENAME TO did nothing. Also, if the table had disabled keys
and was ALTER-ed then the end table was with enabled keys.
Fixed by checking whether the table had disabled keys and enabling them
in the copied table.
myisam/mi_open.c:
Extend mi_indexes_are_disabled to implement return value
2 - Non-unique indexes are disabled
mysql-test/r/alter_table.result:
update result
mysql-test/t/alter_table.test:
update test
sql/sql_table.cc:
When ENABLE|DISABLE index is combined with another option
different than RENAME TO, we should ENABLE|DISABLE the keys of
the modified table. Also when modifying we should preserve the
previous state of the indices.
(This problem exists in 5.0 and 5.1 but since the codebase has
diverged, this fix won't automerge, but the fix will be quite
similar).
A corrupted compressed table could crash the server and
myisamchk.
The data file of an uncompressed table contains just the records.
There is no header in the data file.
However the data file of a compressed table has a header.
The header describes how the table was compressed. This
information is necessary to extract the records from the
compressed data file.
Part of the compressed data file header are the [de]code tables.
They are numeric representations of the Huffman trees used for
coding and decoding. A Huffman tree is a binary tree. Every
node has two childs. A child can be a leaf or a branch. Leaves
contain the decoded value. Branches point to another tree node.
Since the [de]code table is represented as an array of childs,
the branches need to point at a child within the same array.
The corruption of the compressed data file from the bug report
was a couple of branches that pointed outside their array.
This condition had not been correctly checked.
I added some checks for the pointers in the decode tables.
This type of corruption will no longer crash the server or
myisamchk.
No test case. A corrupted compressed table is required.
myisam/mi_packrec.c:
Bug#23139 - myisamchk and mysqld crash when trying to access table
Added some checks for the pointers in the decode tables.
Added comments, DBUG prints, style fixes.
into zippy.cornsilk.net:/home/cmiller/work/mysql/mysql-4.1-maint
configure.in:
Auto merged
mysql-test/t/ps.test:
Auto merged
sql/handler.cc:
Auto merged
sql/sql_delete.cc:
Auto merged
sql/sql_select.cc:
Auto merged
sql/table.cc:
Auto merged
tests/mysql_client_test.c:
Auto merged
myisam/sort.c:
Manual merge.
mysql-test/r/innodb_mysql.result:
Manual merge.
mysql-test/t/innodb_mysql.test:
Manual merge.
mysys/mf_iocache.c:
Manual merge.
When compiling with a default key block size greater than the
smallest key block size used in a table, checking that table
failed with bogus errors. The table was marked corrupt. This
affected myisamchk and the server.
The problem was that the default key block size was used at
some places where sizes less or equal to the block size of the
index in check was required.
We do now use the key block size of the particular index
when checking.
A test case is available for later versions only.
myisam/mi_check.c:
Bug#22119 - Changing MI_KEY_BLOCK_LENGTH makes a wrong myisamchk
Changed check_k_link() and chk_index_down() to use the block
size of the index in check or MI_MIN_KEY_BLOCK_LENGTH where
required. Formerly myisam_block_size or MYISAM_SHARE::blocksize
was used wrongly.
really damaged MyISAM tables
When unpacking a blob column from broken row server crash
could happen. This could rather happen when trying to repair
a table using either REPAIR TABLE or myisamchk, though it
also could happend when trying to access broken row using
other SQL statements like SELECT if table is not marked as
crashed.
Fixed ulong overflow when trying to extract blob from
broken row.
Affects MyISAM only.
myisam/mi_dynrec.c:
Fixed ulong overflow when trying to extract blob from
broken row. It happens when there are not enough bytes
to store blob length in `from' buffer. In this case
(ulong) (from_end - from) - size_length value is huge,
close to ULONG_MAX.
Repair table could crash a server if there is not sufficient
memory (myisam_sort_buffer_size) to operate. Affects not only
repair, but also all statements that use create index by sort:
repair by sort, parallel repair, bulk insert.
Return an error if there is not sufficient memory to store at
least one key per BUFFPEK.
Also fixed memory leak if thr_find_all_keys returns an error.
myisam/sort.c:
maxbuffer is number of BUFFPEK-s for repair. It is calculated
as records / keys. keys is number of keys that can be stored
in memory (myisam_sort_buffer_size). There must be sufficient
memory to store both BUFFPEK-s and keys. It was checked
correctly before this patch. However there is another
requirement that wasn't checked: there must be sufficient
memory for at least one key per BUFFPEK, otherwise repair
by sort/parallel repair cannot operate.
Return an error if there is not sufficient memory to store at
least one key per BUFFPEK.
Also fixed memory leak if thr_find_all_keys returns an error.
mysql-test/r/repair.result:
A test case for BUG#23175.
mysql-test/t/repair.test:
A test case for BUG#23175.
set. This has always worked because when flag is !=0 then
HA_VAR_LENGTH_KEY is always set. Therefore, a test case cannot
reveal a faulty behavior.
Fix for bug#23074: typo in myisam/sort.c
myisam/sort.c:
fix typo. Nevertheless, it has worked as expected
because when a bit in flag is set HA_VAR_LENGTH_KEY has
been always set too. Actually, no problem exposed through
DDL.
