This is the main commit for Worklog tasks:
* A more dynamic binlog format which allows small changes (1064)
* Log session variables in Query_log_event (1063)
Below 5.0 means 5.0.0.
MySQL 5.0 is able to replicate FOREIGN_KEY_CHECKS, UNIQUE_KEY_CHECKS (for speed),
SQL_AUTO_IS_NULL, SQL_MODE. Not charsets (WL#1062), not some vars (I can only think
of SQL_SELECT_LIMIT, which deserves a special treatment). Note that this
works for queries, except LOAD DATA INFILE (for this it would have to wait
for Dmitri's push of WL#874, which in turns waits for the present push, so...
the deadlock must be broken!). Note that when Dmitri pushes WL#874 in 5.0.1,
5.0.0 won't be able to replicate a LOAD DATA INFILE from 5.0.1.
Apart from that, the new binlog format is designed so that it can tolerate
a little variation in the events (so that a 5.0.0 slave could replicate a
5.0.1 master, except for LOAD DATA INFILE unfortunately); that is, when I
later add replication of charsets it should break nothing. And when I later
add a UID to every event, it should break nothing.
The main change brought by this patch is a new type of event, Format_description_log_event,
which describes some lengthes in other event types. This event is needed for
the master/slave/mysqlbinlog to understand a 5.0 log. Thanks to this event,
we can later add more bytes to the header of every event without breaking compatibility.
Inside Query_log_event, we have some additional dynamic format, as every Query_log_event
can have a different number of status variables, stored as pairs (code, value); that's
how SQL_MODE and session variables and catalog are stored. Like this, we can later
add count of affected rows, charsets... and we can have options --don't-log-count-affected-rows
if we want.
MySQL 5.0 is able to run on 4.x relay logs, 4.x binlogs.
Upgrading a 4.x master to 5.0 is ok (no need to delete binlogs),
upgrading a 4.x slave to 5.0 is ok (no need to delete relay logs);
so both can be "hot" upgrades.
Upgrading a 3.23 master to 5.0 requires as much as upgrading it to 4.0.
3.23 and 4.x can't be slaves of 5.0.
So downgrading from 5.0 to 4.x may be complicated.
Log_event::log_pos is now the position of the end of the event, which is
more useful than the position of the beginning. We take care about compatibility
with <5.0 (in which log_pos is the beginning).
I added a short test for replication of SQL_MODE and some other variables.
TODO:
- after committing this, merge the latest 5.0 into it
- fix all tests
- update the manual with upgrade notes.
Now the I/O thread (in flush_master_info()) flushes the relay log to disk
after reading every event. Slower but provides additionnal safety in case
of brutal crash.
I had to make the flush optional (i.e. add a if(some_bool_argument) in the function)
because sometimes flush_master_info() is called when there is no usable
relay log (the relay log's IO_CACHE is not initialized so can't be flushed).
- when we don't have in_addr_t, use uint32.
- a forgotten initialization of slave_proxy_id in sql/log_event.cc (was not really "forgot", was
"we needn't init it there", but there was one case where we needed...).
- made slave_proxy_id always meaningful in THD and Log_event, so we can
rely more on it (no need to test if it's meaningful). THD::slave_proxy_id
is equal to THD::thread_id except for the slave SQL thread.
- clean up the slave's temporary table (i.e. free their memory) when slave
server shuts down.
"Add a column "Timestamp_of_last_master_event_executed" in SHOW SLAVE STATUS".
Finally this is adding
- Slave_IO_State (a copy of the State column of SHOW PROCESSLIST for the I/O thread,
so that the users, most of the time, has enough info with only SHOW SLAVE STATUS).
- Seconds_behind_master. When the slave connects to the master it does SELECT UNIX_TIMESTAMP()
on the master, computes the absolute difference between the master's and the slave's clock.
It records the timestamp of the last event executed by the SQL thread, and does a
small computation to find the number of seconds by which the slave is late.
Now LOAD DATA FROM MASTER does not drop the database, instead it only tries to
create it, and drops/creates table-by-table.
* replicate_wild_ignore_table='db1.%' is now considered as "ignore the 'db1'
database as a whole", as it already works for CREATE DATABASE and DROP DATABASE.
Added proper options to CHANGE MASTER TO, new fields to SHOW SLAVE STATUS,
Honoring this parameters during connection to master.
Introduced new format of master.info file
- Bug #985: "Between RESET SLAVE and START SLAVE, SHOW SLAVE STATUS is wrong."
Now RESET SLAVE puts correct info in mi->host etc. A new test rpl_reset_slave
for that.
- Bug #986: "CHANGE MASTER & START SLAVE do not reset error columns in SHOW
SLAVE STATUS". Now these reset the errors.
I extended the task to cleaning error messages, making them look nicer,
and making the output of SHOW SLAVE STATUS (column Last_error) be as complete
as what's printed on the .err file;
previously we would have, for a failure of a replicated LOAD DATA INFILE:
- in the .err, 2 lines:
"duplicate entry 2708 for key 1"
"failed loading SQL_LOAD-5-2-2.info"
- and in SHOW SLAVE STATUS, only:
"failed loading SQL_LOAD-5-2-2.info".
Now SHOW SLAVE STATUS will contain the concatenation of the 2 messages.
and other replicate-*-table options in SHOW SLAVE STATUS.
Seems like it had not been done, so I push it now:
there's 4 new columns to SHOW SLAVE STATUS.
Added MAX_PASSWORD_LENGTH. This increased master-slave passwords to 32 bytes (Bug #766)
Fixed server crash on purge master logs or show master logs when binlog is off. (Bug #733)
- Comments for future devs.
- Start_log_event::exec_event() : when we hit it, do a rollback.
- We don't need LOG_EVENT_FORCED_ROTATE_F.
- Stop_log_event::exec_event() : when we hit it, we needn't clean anything.
- Removed LOG_EVENT_TIME_F and LOG_EVENT_FORCED_ROTATE_F.
- We don't need Stop events in the relay log.
- Now filtering of server id is done in the I/O thread first.
we now make a distinction between if the master is < 3.23.57, 3.23 && >=57, and 4.x
(before the 2 3.23 were one). This is because in 3.23.57 we have a way to distinguish between
a Start_log_event written at server startup and one written at FLUSH LOGS, so we
have a way to know if the slave must drop old temp tables or not.
Change: mi->old_format was bool, now it's enum (to handle 3 cases). However, functions
which had 'bool old_format' as an argument have their prototypes unchanged, because
the old old_format == 0 now corresponds to the enum value BINLOG_FORMAT_CURRENT which
is equal to 0, so boolean tests are left untouched. The only case were we use mi->old_format
as an enum instead of casting it implicitly to a bool, is in Start_log_event::exec_event,
where we want to distinguish between the 3 possible enum values.
Call pthread_mutex_destroy() on not used mutex.
Changed comments in .h and .c files from // -> /* */
Added detection of mutex on which one didn't call pthread_mutex_destroy()
Fixed bug in create_tmp_field() which causes a memory overrun in queries that uses "ORDER BY constant_expression"
Added optimisation for ORDER BY NULL
Fixed comparision of log-binary name to handle comparison when file name extension wraps from .999 to .1000
Don't replicate CREATE/DROP DATABASE if wild_xxx_table=database.% is used.
Improve mysql-test to be more robust.
Fix that GRANT doesn't delete SSL options
Change innobase_flush_log_at_trx_commit to uint.
Don't rotate logs if we read a rotate log entry from the master.
Changed option variables to my_bool (to avoid bugs in my_getopt())
Added new thread specific mutex LOCK_delete to be able to free LOCK_thread_count early.
Changed usage of LOCK_thread_count -> LOCK_status for statistics variables
More DBUG info for replication
Better error messages from replication
Fixed bug in replication code when connecting to 'localhost' (time was not released properly)
Block ALARM signal on Linux for signal handler thread (Fixes problem with running mysqld with --debug)
Removed warning when setting an AUTO_INCREMENT field to NULL
Fixed bugs in my last changeset that made MySQL hard to compile.
Added mutex around some data that could cause table cache corruptions when using OPTIMIZE TABLE / REPAIR TABLE or automatic repair of MyISAM tables.
Added mutex around some data in the slave start/stop code that could cause THD linked list corruptions
Extended my_chsize() to allow one to specify a filler character.
Extend vio_blocking to return the old state (This made some usage of this function much simpler)
Added testing for some functions that they caller have got the required mutexes before calling the function.
Use setrlimit() to ensure that we can write core file if one specifies --core-file.
Added --slave-compressed-protocol
Made 2 the minimum length for ft_min_word_len
Added variables foreign_key_checks & unique_checks.
Less logging from replication code (if not started with --log-warnings)
Changed that SHOW INNODB STATUS requre the SUPER privilege
More DBUG statements and a lot of new code comments
(All commit emails since 4.0.1 checked)
This had to be done now, before the 4.1 tree changes to much, to make it easy to propagate bug fixes to the 4.1 tree.
DBUG_ macro cleanup
buffer boundary cleanup
This changeset, although not fully tested, works for me better than
anything I've had so far, including what is in the repository. I will
push it unless something crashes while I am writing this :-)
of one of our users which replicates from a live server. So I now have a
lot of confidence in it and will push. It passes all tests, so I if I broke
somebody's code in 3.23 merge this is your fault - you should have had a good test
case :-)
fixed bug in LOAD DATA FROM MASTER
fixed rpl000001.result
Slave now replicates 3.23 master, with the exception of LOAD DATA INFILE,
which is still buggy. Will push this one after the pull/merge
Test of unsigned BIGINT values
Fixes for queries-per-hour
Cleanup of replication code (comments and portability fixes)
Make most of the binary log code 4G clean
Changed syntax for GRANT ... QUERIES PER HOUR
fixed too quick timeout in mysql-test-run which caused a race with the
new server getting started before the old one completely finished
shutdown. This should fix the pid warning we've been getting as well as
inconsistent results when running tests with the manager
first pull, merge,test, and get it to work.
The main change is the new replication code - now we have two slave threads
SQL thread and I/O thread. I have also re-written a lot of the code to
prepare for multi-master implementation.
I also documented IO_CACHE quite extensively and to some extend, THD class.
added extra/mysql_install.c - will work on it in 4.0, but it does not hurt
to have it sit in 3.23 tree for now since it will eventually be backported
to 3.23 anyway
Stop event and bugs the test suite could not catch
Did some big restructuring of binlog event classes - most important
change is that now each event class has exec_event method and one does
not need to modify slave core code to add a new event. Slave code is
now much smaller and easier to read
fixed test case to produce same results on txn-enabled binary.
temporary fix - permanent fix is it make mysqld behave the same with
MyISAM on auto_increment insert after delete from tbl_name regardless
of how the binary was compiled
sql/slave.cc
fixed concurrency bug with log_seq
sql/slave.h
last_log_seq needs to be updated atomically with pos
sql/sql_repl.cc
cleared up comment and fixed concurrency bug
slave will skip the first SQL_SLAVE_SKIP_COUNTER events on start -
this is needed to recover from broken replication when the next log
event(s) should just be skipped.
Rotate binlogs when max_binlog_size is reached
fixed slave to preserve temp tables on slave stop; slave start
added abort-slave-event-count option to mysqld to test the above
added a test case for temp tables with a contantly aborting slave
removed warning in sql_parse.cc
fixed thimble.cc to compile
send_file() is more stack-friendly - changed large static buffer to alloc_root()
fixed my_b_seek() to work with WRITE_CACHE
test case to make sure the slave starts correctly after being stopped
truncated words file so the replication tests will take less time
cleanup - remove unneeded daeamon restarts
sql/log_event.cc
fixed bug - do not confuse confused slaves even further if they try to
read less than LOG_EVENT_HEADER_LEN before the end of file
sql/mysqlbinlog.cc
give error if constructor fails
sql/mysqld.cc
--disconnect-slave-event-count for replication debugging
sql/slave.cc
--disconnect-slave-event-count
cleanup to remove unneeded sleeps when things go wrong - sleep only
when we have to
sql/slave.h
--disconnect-slave-event-count
test cases for ALTER TABLE ORDER BY and replication offset off by 22
confusion mystery bug (both pass)
this file needs to be deleted
sql/log_event.cc
fixed warnings
sql/log_event.h
fixed () #define bug
sql/mysqlbinlog.cc
fixed length argument - was supposed to be one less
sql/mysqld.cc
replicate-do/ignore-table now works, wild does not yet
sql/mysql_priv.h
updating argument to add_table_to_list() -- needed for replicate-do/ignore table
sql/slave.cc
changes fore replicate-do/ignore-table
close the socket before going to sleep sleeping after error
bad event was being freed too early
sql/slave.h
changes for replicate-do/ignore-table
sql/sql_class.cc
slave_thread variable to THD
sql/sql_class.h
added slave_thread to THD, fixed bug in end_time()
sql/sql_parse.cc
updating argument to add_tables_to_list()
sql/sql_table.cc
fixed bug in mysql_rm_table()
sql/sql_yacc.yy
fixed up add_table_to_list() calls to accept updating argument
sql/table.h
added updating to TABLE_LIST
