a limitation of yesterday's implementation:
if there was an unfinished transaction (COMMIT not typed), and some MyISAM tables were
then updated, and then mysqld crashes, then at restart the server would use the too old
binlog offset known by InnoDB to cut the binlog, thus cutting the successful MyISAM
updates. We fix this by reporting the binlog offset into InnoDB even if InnoDB was not
affected at all by the update.
But the feature is still disabled until we decide if it can go into 4.1.3.
Final push. Printing some warnings at startup, as --innodb-safe-binlog requires
some other options, to work as expected.
Adding 6 new tests (3 pairs). If they fail on some platforms (so far they have been run only on my Linux),
they should be ignored for the 4.1.3 build (you can just
rm mysql-test/*/rpl_crash_*).
Now going to update doc.
Won't be pushed as is - separate email sent for internal review.
WL#1717 "binlog-innodb consistency".
Now when mysqld starts, if InnoDB does a crash recovery, we use the binlog name
and position retrieved from InnoDB (corresponding to the last transaction
successfully committed by InnoDB) to cut any rolled back transaction from
the binary log. This is triggered by the --innodb-safe-binlog option.
Provided you configure mysqld to fsync() InnoDB at every commit (using
flush_log_at_trx_commit) and to fsync() the binlog at every write
(using --sync-binlog=1), this behaviour guarantees that a master always has
consistency between binlog and InnoDB, whenever the crash happens.
6 tests to verify that it works.
Added basic per-thread time zone functionality (based on public
domain elsie-code). Now user can select current time zone
(from the list of time zones described in system tables).
All NOW-like functions honor this time zone, values of TIMESTAMP
type are interpreted as values in this time zone, so now
our TIMESTAMP type behaves similar to Oracle's TIMESTAMP WITH
LOCAL TIME ZONE (or proper PostgresSQL type).
WL#1266 "CONVERT_TZ() - basic time with time zone conversion
function".
Fixed problems described in Bug #2336 (Different number of warnings
when inserting bad datetime as string or as number). This required
reworking of datetime realted warning hadling (they now generated
at Field object level not in conversion functions).
Optimization: Now Field class descendants use table->in_use member
instead of current_thd macro.
New option --sync-binlog=x (and global settable variable) which will fsync the binlog
after every x-th disk write to it. That is, if in autocommit mode, after every x-th statement
written to the binlog; if using transactions, after every x-th transaction written to the binlog.
x==0 means no fsync. x==1 is the slowest.
There is no test added for this, I have just checked that it works as --sync-binlog=1 dramatically
slows down mysqld.
Made sync-frm a global settable variable.
by binlogging some SET ONE_SHOT CHARACTER_SETetc,
which will be enough until we have it more compact and more complete in 5.0. With the present patch,
replication will work ok between 4.1.3 master and slaves, as long as:
- master and slave have the same GLOBAL.COLLATION_SERVER
- COLLATION_DATABASE and CHARACTER_SET_DATABASE are not used
- application does not use the fact that table is created with charset of the USEd db (BUG#2326).
all of which are not too hard to fulfill.
ONE_SHOT is reserved for internal use of mysqlbinlog|mysql and works only for charsets,
so we give error if used for non-charset vars.
Fix for BUG#3875 "mysqlbinlog produces wrong ouput if query uses
variables containing quotes" and BUG#3943 "Queries with non-ASCII literals are not replicated
properly after SET NAMES".
Detecting that master and slave have different global charsets or server ids.
Fixed bugs in group_concat with ORDER BY and DISTINCT (Bugs #2695, #3381 and #3319)
Fixed crash when doing rollback in slave and the io thread catched up with the sql thread
Set locked_in_memory properly
* A more dynamic binlog format which allows small changes (1064)
* Log session variables in Query_log_event (1063)
It contains a few bugfixes (which I made when running the testsuite).
I carefully updated the results of the testsuite (i.e. I checked for every one,
if the difference between .reject and .result could be explained).
Apparently mysql-test-run --manager is broken in 4.1 and 5.0 currently,
so I could neither run the few tests which require --manager, nor check
that they pass nor modify their .result. But for builds, we don't run
with --manager.
Apart from --manager, the full testsuite passes, with Valgrind too (no errors).
I'm going to push in the next minutes. Remains: update the manual.
Note: by chance I saw that (in 4.1, in 5.0) rpl_get_lock fails when run alone;
this is normal at it makes assumptions on thread ids. I will fix this one day
in 4.1.
This is the main commit for Worklog tasks:
* A more dynamic binlog format which allows small changes (1064)
* Log session variables in Query_log_event (1063)
Below 5.0 means 5.0.0.
MySQL 5.0 is able to replicate FOREIGN_KEY_CHECKS, UNIQUE_KEY_CHECKS (for speed),
SQL_AUTO_IS_NULL, SQL_MODE. Not charsets (WL#1062), not some vars (I can only think
of SQL_SELECT_LIMIT, which deserves a special treatment). Note that this
works for queries, except LOAD DATA INFILE (for this it would have to wait
for Dmitri's push of WL#874, which in turns waits for the present push, so...
the deadlock must be broken!). Note that when Dmitri pushes WL#874 in 5.0.1,
5.0.0 won't be able to replicate a LOAD DATA INFILE from 5.0.1.
Apart from that, the new binlog format is designed so that it can tolerate
a little variation in the events (so that a 5.0.0 slave could replicate a
5.0.1 master, except for LOAD DATA INFILE unfortunately); that is, when I
later add replication of charsets it should break nothing. And when I later
add a UID to every event, it should break nothing.
The main change brought by this patch is a new type of event, Format_description_log_event,
which describes some lengthes in other event types. This event is needed for
the master/slave/mysqlbinlog to understand a 5.0 log. Thanks to this event,
we can later add more bytes to the header of every event without breaking compatibility.
Inside Query_log_event, we have some additional dynamic format, as every Query_log_event
can have a different number of status variables, stored as pairs (code, value); that's
how SQL_MODE and session variables and catalog are stored. Like this, we can later
add count of affected rows, charsets... and we can have options --don't-log-count-affected-rows
if we want.
MySQL 5.0 is able to run on 4.x relay logs, 4.x binlogs.
Upgrading a 4.x master to 5.0 is ok (no need to delete binlogs),
upgrading a 4.x slave to 5.0 is ok (no need to delete relay logs);
so both can be "hot" upgrades.
Upgrading a 3.23 master to 5.0 requires as much as upgrading it to 4.0.
3.23 and 4.x can't be slaves of 5.0.
So downgrading from 5.0 to 4.x may be complicated.
Log_event::log_pos is now the position of the end of the event, which is
more useful than the position of the beginning. We take care about compatibility
with <5.0 (in which log_pos is the beginning).
I added a short test for replication of SQL_MODE and some other variables.
TODO:
- after committing this, merge the latest 5.0 into it
- fix all tests
- update the manual with upgrade notes.
The constructor of Rotate_log_event used when we are rotating our binlog or
relay log, should not assume that there is a nonzero THD available.
For example, when we are reacting to SIGHUP, the THD is 0.
In fact we don't need to use the THD in this constructor;
we can do like for Stop_log_event, and use the minimal Log_event
constructor.
If we were allowed to put Unix-specific commands in the testsuite,
I'd add a test for this (<sigh>).
Fixed compiler warnings (a lot of hidden variables detected by the Forte compiler)
Added a lot of 'version_xxx' strings to 'show variables'
Prevent copying of TMP_TABLE_PARAM (This caused core dump bug on Solaris)
Fixed problem with printing sub selects to debug log
WL#604 Privileges in embedded library
code added to check privileges in embedded library
NO_EMBEDDED_ACCESS_CHECKS macros inserted in code so we can exclude
access-checking parts. Actually we now can exclude these parts from
standalone server as well. Do we need it?
Access checks are disabled in embedded server by default. One should
edit libmysqld/Makefile manually to get this working.
We definitely need the separate configure for embedded server
Bug #1392 "On Win, slave leaves one temp file after successf. replicating LOAD DATA INFILE"
Windows-specific bug (we forgot to close a file before deleting it).
Patch written by me, tested by Miguel (thanks!) and it works.
Add pack_bits to pack_reclength for dynamic rows. This solves buffer a possible buffer overflow on update.
(This will probably solve bug #563)
Fix test for available file descriptors in mysqltest
Fixed core dump bug in replication tests when running without transactional table support
For example the Binlog_dump thread (on the master) sometimes showed "Slave:".
And there were confusing messages where "binlog" was employed instead
of "relay log".
fix for BUG#1113 "INSERT into non-trans table SELECT ; ROLLBACK" does not send warning"
and
fix for BUG#873 "In transaction, INSERT to non-trans table is written too early to binlog".
Now we don't always write the non-trans update immediately to the binlog;
if there is something in the binlog cache we write it to the binlog cache
(because the non-trans update could depend on a trans table which was modified
earlier in the transaction); then in case of ROLLBACK, we write the binlog
cache to the binlog, wrapped with BEGIN/ROLLBACK.
This guarantees that the slave does the same updates.
For ROLLBACK TO SAVEPOINT: when we execute a SAVEPOINT command we write it
to the binlog cache. At ROLLBACK TO SAVEPOINT, if some non-trans table was updated,
we write ROLLBACK TO SAVEPOINT to the binlog cache; when the transaction
terminates (COMMIT/ROLLBACK), the binlog cache will be flushed to the binlog
(because of the non-trans update) so we'll have SAVEPOINT and ROLLBACK TO
SAVEPOINT in the binlog.
Apart from this rare case of updates of mixed table types in transaction, the
usual way is still clear the binlog cache at ROLLBACK, or chop it at
ROLLBACK TO SAVEPOINT (meaning the SAVEPOINT command is also chopped, which
is fine).
Note that BUG#873 encompasses subbugs 1) and 2) of BUG#333 "3 binlogging bugs when doing INSERT with mixed InnoDB/MyISAM".
"LOAD DATA INFILE is badly filtered by binlog-*-db rules".
There will probably be a second final one to merge Dmitri's changes
to rpl_log.result and mine.
2 new tests:
rpl_loaddata_rule_m : test of logging of LOAD DATA INFILE when the master has binlog-*-db rules,
rpl_loaddata_rule_s : test of logging of LOAD DATA INFILE when the slave has binlog-*-db rules and --log-slave-updates.
the first 4 bytes of the relay log. Indeed comments in mysys/mf_iocache.c
say we must always use my_b_append for such a cache.
This *could* avoid a very rare assertion failure which is:
030524 19:32:38 Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log '/
users/gbichot/4.1.1/mysql-test/var/log/slave-relay-bin.000001' position: 4
030524 19:32:38 next log '/users/gbichot/4.1.1/mysql-test/var/log/slave-relay-bin.000002' is currently active
mysqld: mf_iocache.c:701: _my_b_seq_read: Assertion `pos_in_file == info->end_of_file' failed.
and which seemed to happen always when the SQL thread and/or the I/O thread
were at position 4 in a relay log.
