This is the main commit for Worklog tasks:
* A more dynamic binlog format which allows small changes (1064)
* Log session variables in Query_log_event (1063)
Below 5.0 means 5.0.0.
MySQL 5.0 is able to replicate FOREIGN_KEY_CHECKS, UNIQUE_KEY_CHECKS (for speed),
SQL_AUTO_IS_NULL, SQL_MODE. Not charsets (WL#1062), not some vars (I can only think
of SQL_SELECT_LIMIT, which deserves a special treatment). Note that this
works for queries, except LOAD DATA INFILE (for this it would have to wait
for Dmitri's push of WL#874, which in turns waits for the present push, so...
the deadlock must be broken!). Note that when Dmitri pushes WL#874 in 5.0.1,
5.0.0 won't be able to replicate a LOAD DATA INFILE from 5.0.1.
Apart from that, the new binlog format is designed so that it can tolerate
a little variation in the events (so that a 5.0.0 slave could replicate a
5.0.1 master, except for LOAD DATA INFILE unfortunately); that is, when I
later add replication of charsets it should break nothing. And when I later
add a UID to every event, it should break nothing.
The main change brought by this patch is a new type of event, Format_description_log_event,
which describes some lengthes in other event types. This event is needed for
the master/slave/mysqlbinlog to understand a 5.0 log. Thanks to this event,
we can later add more bytes to the header of every event without breaking compatibility.
Inside Query_log_event, we have some additional dynamic format, as every Query_log_event
can have a different number of status variables, stored as pairs (code, value); that's
how SQL_MODE and session variables and catalog are stored. Like this, we can later
add count of affected rows, charsets... and we can have options --don't-log-count-affected-rows
if we want.
MySQL 5.0 is able to run on 4.x relay logs, 4.x binlogs.
Upgrading a 4.x master to 5.0 is ok (no need to delete binlogs),
upgrading a 4.x slave to 5.0 is ok (no need to delete relay logs);
so both can be "hot" upgrades.
Upgrading a 3.23 master to 5.0 requires as much as upgrading it to 4.0.
3.23 and 4.x can't be slaves of 5.0.
So downgrading from 5.0 to 4.x may be complicated.
Log_event::log_pos is now the position of the end of the event, which is
more useful than the position of the beginning. We take care about compatibility
with <5.0 (in which log_pos is the beginning).
I added a short test for replication of SQL_MODE and some other variables.
TODO:
- after committing this, merge the latest 5.0 into it
- fix all tests
- update the manual with upgrade notes.
Now the I/O thread (in flush_master_info()) flushes the relay log to disk
after reading every event. Slower but provides additionnal safety in case
of brutal crash.
I had to make the flush optional (i.e. add a if(some_bool_argument) in the function)
because sometimes flush_master_info() is called when there is no usable
relay log (the relay log's IO_CACHE is not initialized so can't be flushed).
"CHANGE MASTER makes SQL thread restart from coordinates of I/O thread".
So, in CHANGE MASTER:
when it seems reasonable that the user did not want to discontinue
its replication (i.e. when he is not specifying host or port or master_log_file
or master_log_pos; this will be documented), set the coordinates of the
I/O thread to those of the SQL thread. This way, the SQL thread will see
no discontinuity in the relay log (i.e. will skip no events), because
the I/O thread will fill the brand new relay log with the events which
are just after the position where the SQL thread had stopped
(before CHANGE MASTER was issued).
And a new test for this bug.
"Add a column "Timestamp_of_last_master_event_executed" in SHOW SLAVE STATUS".
Finally this is adding
- Slave_IO_State (a copy of the State column of SHOW PROCESSLIST for the I/O thread,
so that the users, most of the time, has enough info with only SHOW SLAVE STATUS).
- Seconds_behind_master. When the slave connects to the master it does SELECT UNIX_TIMESTAMP()
on the master, computes the absolute difference between the master's and the slave's clock.
It records the timestamp of the last event executed by the SQL thread, and does a
small computation to find the number of seconds by which the slave is late.
WL#604 Privileges in embedded library
code added to check privileges in embedded library
NO_EMBEDDED_ACCESS_CHECKS macros inserted in code so we can exclude
access-checking parts. Actually we now can exclude these parts from
standalone server as well. Do we need it?
Access checks are disabled in embedded server by default. One should
edit libmysqld/Makefile manually to get this working.
We definitely need the separate configure for embedded server
Added proper options to CHANGE MASTER TO, new fields to SHOW SLAVE STATUS,
Honoring this parameters during connection to master.
Introduced new format of master.info file
For example the Binlog_dump thread (on the master) sometimes showed "Slave:".
And there were confusing messages where "binlog" was employed instead
of "relay log".
"LOAD DATA INFILE is badly filtered by binlog-*-db rules".
There will probably be a second final one to merge Dmitri's changes
to rpl_log.result and mine.
2 new tests:
rpl_loaddata_rule_m : test of logging of LOAD DATA INFILE when the master has binlog-*-db rules,
rpl_loaddata_rule_s : test of logging of LOAD DATA INFILE when the slave has binlog-*-db rules and --log-slave-updates.
- Bug #985: "Between RESET SLAVE and START SLAVE, SHOW SLAVE STATUS is wrong."
Now RESET SLAVE puts correct info in mi->host etc. A new test rpl_reset_slave
for that.
- Bug #986: "CHANGE MASTER & START SLAVE do not reset error columns in SHOW
SLAVE STATUS". Now these reset the errors.
Added MAX_PASSWORD_LENGTH. This increased master-slave passwords to 32 bytes (Bug #766)
Fixed server crash on purge master logs or show master logs when binlog is off. (Bug #733)
so I commit again in a fresh tree.
Fix for bug#763 (Relay_log_space too big by 4 bytes),
plus comments and DBUG_PRINT, and we don't start replication
if --bootstrap.
support issue with an unclear message which can have N reasons for appearing.
This should help us know at which point it failed, and get the errno when
my_open was involved (as the reason for the unclear message is often a
permission problem).
RESET SLAVE resets last_error and last_errno in SHOW SLAVE STATUS (without this,
rpl_loaddata.test, which is expected to generate an error in last_error, influenced
rpl_log_pos.test).
A small test update.
Added STOP SLAVE to mysql-test-run to get rid of several stupid error messages
which are printed while the master restarts and the slave attempts/manages to
connect to it and sends it nonsense binlog requests.
Plus a changeset which I had committed but forgot to push (and this changeset is lost on
another computer, so I recreate it here). This changeset is "user-friendly SHOW BINLOG EVENTS
and CHANGE MASTER TO when log positions < 4 are used.
Here is another pack of changes about gathering common client code in
sql-common/client.c.
Now i symlink the client.c from sql/ and libmysql/. These directories
have client_settings.h files to be included to client.c. It contains
defines and declarations to compile client.c in appropriate manner.
Also i've added include/sql_common.h, containing declarations of what
is exported from client.c
I removed as many #ifdef-s from client.c as i dared to. I think it's better
push it with some extra #ifdef-s now (of course, if everythihg besides it is
ok) so other people can check the code.
now by default, FLUSH, OPTIMIZE, ANALYZE, REPAIR commands are written to the
binlog, unless the new NO_WRITE_TO_BINLOG keyword was used :
OPTIMIZE NO_WRITE_TO_BINLOG table t;
Previously these commands were never written to the binlog, but there are
2 reasons to change this :
- the RENAME TABLE in MERGE table bug (#175) on slave
- the possible "differently optimised queries may lead to different
updates on the master and slave" bug, until we have automatic ORDER BY.
FLUSH LOGS/SLAVE/MASTER/TABLES WITH READ LOCK are never written to the binlog.
New test for the new logging behaviour.
Other small change : reload_acl_and_cache() and reset_slave() don't send their errors themselves,
this is more usual.
Use 0x.... as strings if 'new' mode. (bug 152)
Don't report -max on windows when InnoDB is enabled. (bug 332)
Reset current_linfo; This could cause a hang when doing PURGE LOGS.
Fix for row numbers in EXPLAIN (bug 322)
Fix that USE_FRM works for all table types (bug 97)
Added:
PURGE [MASTER] LOGS BEFORE date/date_expression
expire-logs-days option
With this option old files are deleted when
- mysqld is started
- log is rotated
- someone does FLUSH LOGS
Fixed comparision of log-binary name to handle comparison when file name extension wraps from .999 to .1000
Don't replicate CREATE/DROP DATABASE if wild_xxx_table=database.% is used.
Changed timestamp to return string in YYYY-MM-DD HH:MM:SS format.
DATE_ADD() and related functions now returns correct DATE/DATETIME type depending on argument types.
Now all tests passes, still some work left to remove warnings in log files from mysql-test-run
Class for sending data from server to client (Protocol)
This handles both the old ( <= 4.0 ) protocol and then new binary protocol that is used for prepared statements.
Let MySQL 4.0 read 4.1 .frm files without 4.1 specific extensions
New variables @@rand_seed1 and @@rand_seed2 (used by replication)
DROP TEMPORARY TABLE
Warning handling and initial prepared statement handling (last not complete yet)
Changed a lot of functions that returned 0/1 to my_bool type.
GRANT handling now uses read/write locks instead of mutex
Change basic net functions to use THD instead of NET
(needed for 4.1 protocol)
Use my_sprintf instead of sprintf() + strlen()
Added alloc_query() to be able to chare query initialization code with
prepared statements.
Cleanup handling of SHOW COUNT(*) WARNINGS and SELECT LAST_INSERT_ID()
Note that the following test fails (will be fixed ASAP):
sub_select, union, rpl_rotate_logs and rpl_mystery22
Improve mysql-test to be more robust.
Fix that GRANT doesn't delete SSL options
Change innobase_flush_log_at_trx_commit to uint.
Don't rotate logs if we read a rotate log entry from the master.
Changed option variables to my_bool (to avoid bugs in my_getopt())
Added new thread specific mutex LOCK_delete to be able to free LOCK_thread_count early.
Changed usage of LOCK_thread_count -> LOCK_status for statistics variables
More DBUG info for replication
Better error messages from replication
Fixed bug in replication code when connecting to 'localhost' (time was not released properly)
Block ALARM signal on Linux for signal handler thread (Fixes problem with running mysqld with --debug)
Removed warning when setting an AUTO_INCREMENT field to NULL
Fixed configure problems with HPUX and openbsd
SHOW SLAVE STATUS returns empty set if slave is not initialized
SHOW MASTER STATUS returns empty set if binary logging is not enabled.
Fixed shutdown problem on Solaris.
Fixed bugs in my last changeset that made MySQL hard to compile.
Added mutex around some data that could cause table cache corruptions when using OPTIMIZE TABLE / REPAIR TABLE or automatic repair of MyISAM tables.
Added mutex around some data in the slave start/stop code that could cause THD linked list corruptions
Extended my_chsize() to allow one to specify a filler character.
Extend vio_blocking to return the old state (This made some usage of this function much simpler)
Added testing for some functions that they caller have got the required mutexes before calling the function.
Use setrlimit() to ensure that we can write core file if one specifies --core-file.
Added --slave-compressed-protocol
Made 2 the minimum length for ft_min_word_len
Added variables foreign_key_checks & unique_checks.
Less logging from replication code (if not started with --log-warnings)
Changed that SHOW INNODB STATUS requre the SUPER privilege
More DBUG statements and a lot of new code comments
Made a some new buffers thread specific and changeable.
Resize of key_buffer.
AUTO_COMMIT -> AUTOCOMMIT
Fixed mutex bug in DROP DATABASE
Fixed bug when using auto_increment as second part of a key where first part could include NULL.
Split handler->extra() to extra() and extra_opt() to be able to support thread specific buffers.
Don't write message to error log when slave reconnects becasue of timeout.
Fixed possible update problem when using DELETE/UPDATE on small tables
(In some cases we used index even if table scanning would be better)
A lot of minior code cleanups
SHOW DATABASES
CREATE TEMPORARY TABLE
LOCK TABLES
REPLICATION SLAVE & REPLICATION CLIENT
SUPER
EXECUTE
All scripts & documentation is updated for this change.
Added better error messages for global privileges
(All commit emails since 4.0.1 checked)
This had to be done now, before the 4.1 tree changes to much, to make it easy to propagate bug fixes to the 4.1 tree.
DBUG_ macro cleanup
buffer boundary cleanup
This changeset, although not fully tested, works for me better than
anything I've had so far, including what is in the repository. I will
push it unless something crashes while I am writing this :-)
Test of unsigned BIGINT values
Fixes for queries-per-hour
Cleanup of replication code (comments and portability fixes)
Make most of the binary log code 4G clean
Changed syntax for GRANT ... QUERIES PER HOUR
first pull, merge,test, and get it to work.
The main change is the new replication code - now we have two slave threads
SQL thread and I/O thread. I have also re-written a lot of the code to
prepare for multi-master implementation.
I also documented IO_CACHE quite extensively and to some extend, THD class.
Stop event and bugs the test suite could not catch
Did some big restructuring of binlog event classes - most important
change is that now each event class has exec_event method and one does
not need to modify slave core code to add a new event. Slave code is
now much smaller and easier to read
fixed test case to produce same results on txn-enabled binary.
temporary fix - permanent fix is it make mysqld behave the same with
MyISAM on auto_increment insert after delete from tbl_name regardless
of how the binary was compiled
sql/slave.cc
fixed concurrency bug with log_seq
sql/slave.h
last_log_seq needs to be updated atomically with pos
sql/sql_repl.cc
cleared up comment and fixed concurrency bug
Removed new error message
Fixed test case for varbinary
Threads are again killable in "merge_buffers"
Cleanup of sql_repl.cc
SHOW OPEN TABLES now works when no tables are opened
fixed log sequence bugs
fixed bugs in handling Slave event
added test case with SHOW BINLOG EVENTS
have not fixed all the bugs - found some that are also in 3.23,
will fix them there first, then do pull and cleanup
will not push this changeset
updated manual about table lock counter
fixed coredump in DROP DATABASE with long bogus name by non-root user
fixed bug in handling STOP immediately after ROTATE
added test case for buffer overrun on DROP DATABASE by non-root user
added test case for the STOP bug in replication
slave will skip the first SQL_SLAVE_SKIP_COUNTER events on start -
this is needed to recover from broken replication when the next log
event(s) should just be skipped.
Rotate binlogs when max_binlog_size is reached
fixed bug that created empty result files in no-record mode
mysql-test/t/rpl000007.test
testing non-replication of load data infile when the table is excluded from replication
mysql-test/t/rpl000016.test
test purge master logs
sql/slave.cc
fix memory leaks found by purge master logs test
sql/sql_repl.cc
fixed race condition and delete before close bug in reset_slave()
Overall comments - coverage testing is a very good thing!
added system command
myisammrg/myrg_open.c
fixed coredump when opening an empty union table
sql/sql_repl.cc
fixed warnings, remove confusing comment, fixed coredump in
change master to when master.info was corrupted
added new replication test case
send_file() is more stack-friendly - changed large static buffer to alloc_root()
fixed my_b_seek() to work with WRITE_CACHE
test case to make sure the slave starts correctly after being stopped
truncated words file so the replication tests will take less time
cleanup - remove unneeded daeamon restarts
sql/log_event.cc
fixed bug - do not confuse confused slaves even further if they try to
read less than LOG_EVENT_HEADER_LEN before the end of file
sql/mysqlbinlog.cc
give error if constructor fails
sql/mysqld.cc
--disconnect-slave-event-count for replication debugging
sql/slave.cc
--disconnect-slave-event-count
cleanup to remove unneeded sleeps when things go wrong - sleep only
when we have to
sql/slave.h
--disconnect-slave-event-count
test cases for ALTER TABLE ORDER BY and replication offset off by 22
confusion mystery bug (both pass)
fixed coredump in concat_ws
sql/item_strfunc.h
fixed coredump in concat_ws
sql/mini_client.cc
added mc_mysql_error()
sql/mini_client.h
added mc_mysql_errno()
sql/mysqlbinlog.cc
fixed bug in --postion for local files
sql/slave.cc
more verbose message on bad net reads
sql/sql_repl.cc
send errno on error in mysql_binlog_send
mutex now initialized in the constructor
sql/sql_class.h
intialize MASTER_INFO mutex in the constructor
sql/sql_repl.cc
fixed typo in error message
if the user does not specify offset in CHANGE MASTER TO, set it to skip magic number
repl-tests/test-repl-ts/repl-timestamp.master.reject
New BitKeeper file ``repl-tests/test-repl-ts/repl-timestamp.master.reject''
Added magic number to binlog
sql/log_event.cc
distinquish bogus data from truncated logs
sql/log_event.h
added magic number
added LOG_READ_TRUNC error
sql/mysqlbinlog.cc
fixed to handle magic number
added O_BINARY to my_fopen
sql/mysqld.cc
added code for replicate-rewrite-db
sql/slave.cc
replicate-rewrite-db
O_BINARY
handle magic
sql/sql_class.h
added i_string_pair class
sql/sql_repl.cc
added magic
better error messages
support-files/magic
added magic for binlog
Added test case for replication of queries with error
