MDEV-9101 Limit size of created disk temporary files and tables
Two new variables added:
- max_tmp_space_usage : Limits the the temporary space allowance per user
- max_total_tmp_space_usage: Limits the temporary space allowance for
all users.
New status variables: tmp_space_used & max_tmp_space_used
New field in information_schema.process_list: TMP_SPACE_USED
The temporary space is counted for:
- All SQL level temporary files. This includes files for filesort,
transaction temporary space, analyze, binlog_stmt_cache etc.
It does not include engine internal temporary files used for repair,
alter table, index pre sorting etc.
- All internal on disk temporary tables created as part of resolving a
SELECT, multi-source update etc.
Special cases:
- When doing a commit, the last flush of the binlog_stmt_cache
will not cause an error even if the temporary space limit is exceeded.
This is to avoid giving errors on commit. This means that a user
can temporary go over the limit with up to binlog_stmt_cache_size.
Noteworthy issue:
- One has to be careful when using small values for max_tmp_space_limit
together with binary logging and with non transactional tables.
If a the binary log entry for the query is bigger than
binlog_stmt_cache_size and one hits the limit of max_tmp_space_limit
when flushing the entry to disk, the query will abort and the
binary log will not contain the last changes to the table.
This will also stop the slave!
This is also true for all Aria tables as Aria cannot do rollback
(except in case of crashes)!
One way to avoid it is to use @@binlog_format=statement for
queries that updates a lot of rows.
Implementation:
- All writes to temporary files or internal temporary tables, that
increases the file size, are routed through temp_file_size_cb_func()
which updates and checks the temp space usage.
- Most of the temporary file monitoring is done inside IO_CACHE.
Temporary file monitoring is done inside the Aria engine.
- MY_TRACK and MY_TRACK_WITH_LIMIT are new flags for ini_io_cache().
MY_TRACK means that we track the file usage. TRACK_WITH_LIMIT means
that we track the file usage and we give an error if the limit is
breached. This is used to not give an error on commit when
binlog_stmp_cache is flushed.
- global_tmp_space_used contains the total tmp space used so far.
This is needed quickly check against max_total_tmp_space_usage.
- Temporary space errors are using EE_LOCAL_TMP_SPACE_FULL and
handler errors are using HA_ERR_LOCAL_TMP_SPACE_FULL.
This is needed until we move general errors to it's own error space
so that they cannot conflict with system error numbers.
- Return value of my_chsize() and mysql_file_chsize() has changed
so that -1 is returned in the case my_chsize() could not decrease
the file size (very unlikely and will not happen on modern systems).
All calls to _chsize() are updated to check for > 0 as the error
condition.
- At the destruction of THD we check that THD::tmp_file_space == 0
- At server end we check that global_tmp_space_used == 0
- As a precaution against errors in the tmp_space_used code, one can set
max_tmp_space_usage and max_total_tmp_space_usage to 0 to disable
the tmp space quota errors.
- truncate_io_cache() function added.
- Aria tables using static or dynamic row length are registered in 8K
increments to avoid some calls to update_tmp_file_size().
Other things:
- Ensure that all handler errors are registered. Before, some engine
errors could be printed as "Unknown error".
- Fixed bug in filesort() that causes a assert if there was an error
when writing to the temporay file.
- Fixed that compute_window_func() now takes into account write errors.
- In case of parallel replication, rpl_group_info::cleanup_context()
could call trans_rollback() with thd->error set, which would cause
an assert. Fixed by resetting the error before calling trans_rollback().
- Fixed bug in subselect3.inc which caused following test to use
heap tables with low value for max_heap_table_size
- Fixed bug in sql_expression_cache where it did not overflow
heap table to Aria table.
- Added Max_tmp_disk_space_used to slow query log.
- Fixed some bugs in log_slow_innodb.test
2024-03-14 17:59:00 +01:00
|
|
|
--source include/have_sequence.inc
|
|
|
|
--source include/have_innodb.inc
|
|
|
|
--source include/big_test.inc
|
|
|
|
--source include/not_valgrind.inc
|
|
|
|
--source include/have_log_bin.inc
|
|
|
|
|
|
|
|
call mtr.add_suppression("Write to binary log failed: .* temporary space limit reached. An incident event is written to binary log.*");
|
|
|
|
|
|
|
|
#
|
|
|
|
# Test tmp_space_usage
|
|
|
|
#
|
|
|
|
|
|
|
|
# flush global status is needed because of max_tmp_space_used
|
|
|
|
flush status;
|
|
|
|
flush global status;
|
|
|
|
show session status like "max_tmp_space_used";
|
|
|
|
show global status like "max_tmp_space_used";
|
|
|
|
|
|
|
|
--echo #
|
|
|
|
--echo # MDEV-9101 Limit size of total size of created disk temporary files
|
|
|
|
--echo # and tables.
|
|
|
|
|
|
|
|
# Print variables that can affect the test result
|
|
|
|
show session status like "tmp_space_used";
|
|
|
|
show global status like "tmp_space_used";
|
|
|
|
select @@global.max_tmp_session_space_usage, @@global.max_tmp_total_space_usage;
|
|
|
|
select @@binlog_stmt_cache_size,@@binlog_format;
|
|
|
|
|
|
|
|
create table t1 (a int primary key, v varchar(256), c int default(0)) engine=innodb;
|
|
|
|
--error 41
|
|
|
|
insert into t1 (a,v) select seq, repeat(char(64+mod(seq,32)),mod(seq,254)+1) from seq_1_to_65536;
|
|
|
|
set @@max_tmp_session_space_usage=1024*1024*1024;
|
|
|
|
--error 42
|
|
|
|
insert into t1 (a,v) select seq, repeat(char(64+mod(seq,32)),mod(seq,254)+1) from seq_1_to_65536;
|
|
|
|
set @@max_tmp_session_space_usage=default;
|
|
|
|
|
|
|
|
set @@binlog_format="statement";
|
|
|
|
insert into t1 (a,v) select seq, repeat(char(64+mod(seq,32)),mod(seq,254)+1) from seq_1_to_65536;
|
|
|
|
insert into t1 (a,v) select seq, repeat(char(64+mod(seq,32)),mod(seq,254)+1) from seq_65537_to_131072;
|
|
|
|
select count(*) from t1;
|
|
|
|
|
|
|
|
create table t2 (a int, b int) engine=innodb select seq as a, 0 as b from seq_1_to_131072;
|
|
|
|
|
|
|
|
# Force usage of on disk tmp tables
|
|
|
|
set @@binlog_format="row";
|
|
|
|
set @@tmp_memory_table_size=32*1024;
|
|
|
|
|
|
|
|
--echo # The following queries should fail because of tmp_space_usage
|
|
|
|
--error 41
|
|
|
|
select * from t1 order by a,v;
|
|
|
|
--error 200
|
|
|
|
select v,count(*) from t1 group by v limit 2;
|
|
|
|
--error 41
|
|
|
|
update t1 set v=right(v,2);
|
|
|
|
|
|
|
|
set @@binlog_format="statement";
|
|
|
|
set @@max_tmp_session_space_usage=65536;
|
|
|
|
set @@tmp_memory_table_size=0;
|
|
|
|
--error 200
|
|
|
|
update t1,t2 set t1.c=t2.a, t2.b=1 where t1.a=t2.a;
|
|
|
|
set @@binlog_format="row";
|
|
|
|
set @@max_tmp_session_space_usage=default;
|
|
|
|
|
|
|
|
drop table t1,t2;
|
|
|
|
|
|
|
|
--echo #
|
|
|
|
--echo # Check max_tmp_total_space_usage & processlist
|
|
|
|
--echo #
|
|
|
|
|
|
|
|
# We have to set tmp_memory_table_size to ensure we do not use disk for
|
|
|
|
# the following two show commands.
|
|
|
|
set @@tmp_memory_table_size=1024*1024;
|
|
|
|
show session status like "tmp_space_used";
|
|
|
|
# The following is disabled until we can do "show status" without using
|
|
|
|
# temporary files
|
|
|
|
# show global status like "tmp_space_used";
|
|
|
|
set @@tmp_memory_table_size=0;
|
|
|
|
|
|
|
|
let $tmp_usage1=`select variable_value from information_schema.session_status where variable_name="max_tmp_space_used"`;
|
|
|
|
flush status;
|
|
|
|
let $tmp_usage2=`select variable_value from information_schema.global_status where variable_name="max_tmp_space_used"`;
|
|
|
|
|
|
|
|
--disable_query_log
|
|
|
|
if ($tmp_usage1 == $tmp_usage2)
|
|
|
|
{
|
|
|
|
--echo # session.max_tmp_session_space_usage == global.max_tmp_session_space_usage
|
|
|
|
}
|
|
|
|
if ($tmp_usage1 != $tmp_usage2)
|
|
|
|
{
|
|
|
|
--echo session.max_tmp_session_space_usage ($tmp_usage1) != global.max_tmp_session_space_usage ($tmp_usage2)
|
|
|
|
}
|
|
|
|
--enable_query_log
|
|
|
|
|
|
|
|
|
|
|
|
connect(c1, localhost, root,,);
|
|
|
|
connection default;
|
|
|
|
|
|
|
|
create table t1 (a int primary key, v varchar(256), c int default(0)) engine=innodb;
|
|
|
|
create table t2 (a int primary key, v varchar(256), c int default(0)) engine=innodb;
|
|
|
|
|
|
|
|
begin;
|
|
|
|
insert into t1 (a,v) select seq, repeat(char(64+mod(seq,32)),mod(seq,254)+1) from seq_1_to_3000;
|
|
|
|
|
|
|
|
let $id=`select connection_id()`;
|
|
|
|
let $tmp_usage1=`select variable_value from information_schema.session_status where variable_name="tmp_space_used"`;
|
|
|
|
|
|
|
|
connection c1;
|
|
|
|
--disable_query_log
|
|
|
|
let $tmp_usage2=`select tmp_space_used from information_schema.processlist where id=$id`;
|
|
|
|
if ($tmp_usage1 == $tmp_usage2)
|
|
|
|
{
|
|
|
|
--echo # information_schema.process_list.tmp_space_used == status.tmp_space_used
|
|
|
|
}
|
|
|
|
if ($tmp_usage1 != $tmp_usage2)
|
|
|
|
{
|
|
|
|
--echo tmp_space_used difference: $tmp_usage1 != $tmp_usage2
|
|
|
|
}
|
|
|
|
--enable_query_log
|
|
|
|
|
|
|
|
--error 42
|
|
|
|
insert into t2 (a,v) select seq, repeat(char(64+mod(seq,32)),mod(seq,254)+1) from seq_1_to_3000;
|
|
|
|
--echo # Test setting tmp_space_usage to 0 to disable quotas
|
|
|
|
set @save_max_tmp_total_space_usage=@@global.max_tmp_total_space_usage;
|
|
|
|
set @@global.max_tmp_total_space_usage=0;
|
|
|
|
set @@max_tmp_session_space_usage=0;
|
|
|
|
insert into t2 (a,v) select seq, repeat(char(64+mod(seq,32)),mod(seq,254)+1) from seq_1_to_3000;
|
|
|
|
set @@global.max_tmp_total_space_usage=@save_max_tmp_total_space_usage;
|
|
|
|
set @@max_tmp_session_space_usage=0;
|
|
|
|
connection default;
|
|
|
|
insert into t1 (a,v) values(9999990,0);
|
|
|
|
commit;
|
|
|
|
select count(*) from t1;
|
|
|
|
|
|
|
|
disconnect c1;
|
|
|
|
drop table t1,t2;
|
|
|
|
|
|
|
|
--echo #
|
|
|
|
--echo # Test case from Elena
|
|
|
|
--echo #
|
|
|
|
|
|
|
|
SET @@max_tmp_session_space_usage= 64*1024;
|
|
|
|
set @@binlog_format="statement";
|
|
|
|
|
|
|
|
CREATE OR REPLACE TABLE t1 (a INT, b INT);
|
|
|
|
INSERT INTO t1 SELECT seq, seq FROM seq_1_to_100000;
|
|
|
|
--error 41
|
|
|
|
ALTER TABLE t1 ORDER BY a, b;
|
|
|
|
|
|
|
|
# Cleanup
|
|
|
|
DROP TABLE t1;
|
|
|
|
|
|
|
|
--echo #
|
|
|
|
--echo # Show that setting max tmp space too low value can stop binary logging
|
|
|
|
--echo # if non transactional tables are used.
|
|
|
|
--echo #
|
|
|
|
|
|
|
|
set @save_max_tmp_total_space_usage=@@global.max_tmp_total_space_usage;
|
|
|
|
SET @@global.max_tmp_total_space_usage=64*1024;
|
|
|
|
set @@binlog_format="row";
|
|
|
|
|
|
|
|
create table t1 (a int primary key, v varchar(256)) engine=myisam;
|
|
|
|
--error 41,42
|
|
|
|
insert into t1 (a,v) select seq, repeat(char(64+mod(seq,32)),mod(seq,254)+1) from seq_1_to_65536;
|
|
|
|
show warnings;
|
|
|
|
select count(*) <> 0 from t1;
|
|
|
|
|
|
|
|
# Shhow that this problem does not exists with transactional tables
|
|
|
|
truncate table t1;
|
|
|
|
alter table t1 engine=innodb;
|
|
|
|
--error 41,42
|
|
|
|
insert into t1 (a,v) select seq, repeat(char(64+mod(seq,32)),mod(seq,254)+1) from seq_1_to_65536;
|
|
|
|
show warnings;
|
|
|
|
select count(*) <> 0 from t1;
|
|
|
|
drop table t1;
|
|
|
|
|
|
|
|
set @@global.max_tmp_total_space_usage=@save_max_tmp_total_space_usage;
|
|
|
|
|
|
|
|
--echo #
|
|
|
|
--echo # Check updating non transactional table
|
|
|
|
--echo #
|
|
|
|
|
|
|
|
SET max_tmp_session_space_usage= 64*1024;
|
|
|
|
|
|
|
|
CREATE TABLE t1 (
|
|
|
|
a varchar(1024), b varchar(1024), c varchar(1024), d varchar(1024), e varchar(1024), f varchar(1024), g varchar(1024)
|
|
|
|
) ENGINE=MyISAM;
|
|
|
|
|
|
|
|
INSERT INTO t1 VALUES
|
|
|
|
(REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024)),
|
|
|
|
(REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024)),
|
|
|
|
(REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024)),
|
|
|
|
(REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024)),
|
|
|
|
(REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024)),
|
|
|
|
(REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024),REPEAT('x',1024)),
|
|
|
|
('x','x','x','x','x','x','x');
|
|
|
|
|
|
|
|
--error 41
|
|
|
|
UPDATE t1 SET a = '' LIMIT 100;
|
|
|
|
|
|
|
|
DROP TABLE t1;
|
|
|
|
|
2024-04-06 11:07:49 +02:00
|
|
|
--echo #
|
|
|
|
--echo # MDEV-33680 Server hangs or assertion fails upon SELECT with limited
|
|
|
|
--echo # max_tmp_space_usage
|
|
|
|
--echo #
|
|
|
|
|
|
|
|
set max_tmp_session_space_usage = 1024*1024;
|
|
|
|
--error 201
|
|
|
|
select count(distinct concat(seq,repeat('x',1000))) from seq_1_to_1000;
|
|
|
|
|
2024-04-20 13:02:05 +02:00
|
|
|
--echo #
|
|
|
|
--echo # MDEV-33751 Assertion `thd' failed in int
|
|
|
|
--echo # temp_file_size_cb_func(tmp_file_tracking*, int)
|
|
|
|
--echo #
|
|
|
|
|
2024-04-30 13:38:48 +02:00
|
|
|
set @save_max_tmp_total_space_usage=@@global.max_tmp_total_space_usage;
|
|
|
|
set @@global.max_tmp_total_space_usage=64*1024*1024;
|
2024-04-20 13:02:05 +02:00
|
|
|
set @@max_tmp_session_space_usage=1179648;
|
|
|
|
select @@max_tmp_session_space_usage;
|
|
|
|
set @save_aria_repair_threads=@@aria_repair_threads;
|
|
|
|
set @@aria_repair_threads=2;
|
|
|
|
set @save_max_heap_table_size=@@max_heap_table_size;
|
|
|
|
set @@max_heap_table_size=16777216;
|
|
|
|
|
|
|
|
CREATE TABLE t1 (a CHAR(255),b INT,INDEX (b));
|
|
|
|
INSERT INTO t1 SELECT SEQ,SEQ FROM seq_1_to_100000;
|
|
|
|
--error 200
|
|
|
|
SELECT * FROM t1 UNION SELECT * FROM t1;
|
|
|
|
DROP TABLE t1;
|
|
|
|
|
|
|
|
set @@aria_repair_threads=@save_aria_repair_threads;
|
|
|
|
set @@max_heap_table_size=@save_max_heap_table_size;
|
2024-04-30 13:38:48 +02:00
|
|
|
set @@global.max_tmp_total_space_usage=@save_max_tmp_total_space_usage;
|
|
|
|
|
|
|
|
--echo #
|
|
|
|
--echo # MDEV-34016 Assertion `info->key_del_used == 0' failed in maria_close
|
|
|
|
--echo # with limited tmp space
|
|
|
|
--echo #
|
|
|
|
|
|
|
|
set @save_max_tmp_total_space_usage=@@global.max_tmp_total_space_usage;
|
|
|
|
connect(c1, localhost, root,,);
|
|
|
|
CREATE TABLE t1 (a varchar(1024)) engine=aria;
|
|
|
|
INSERT INTO t1 VALUES ('this'),('is'),('just'),('a'),('filling'),('for'),(REPEAT('a',500));
|
|
|
|
set @@global.max_tmp_total_space_usage=2*1024*1024;
|
|
|
|
SET max_tmp_session_space_usage= 1024*1024, max_heap_table_size= 4*1024*1024;
|
|
|
|
--error ER_NOT_KEYFILE
|
|
|
|
SELECT DISTINCT a, seq FROM t1 JOIN seq_1_to_600;
|
|
|
|
DROP TABLE t1;
|
|
|
|
connection default;
|
|
|
|
disconnect c1;
|
|
|
|
set @@global.max_tmp_total_space_usage=@save_max_tmp_total_space_usage;
|
2024-04-20 13:02:05 +02:00
|
|
|
|
MDEV-9101 Limit size of created disk temporary files and tables
Two new variables added:
- max_tmp_space_usage : Limits the the temporary space allowance per user
- max_total_tmp_space_usage: Limits the temporary space allowance for
all users.
New status variables: tmp_space_used & max_tmp_space_used
New field in information_schema.process_list: TMP_SPACE_USED
The temporary space is counted for:
- All SQL level temporary files. This includes files for filesort,
transaction temporary space, analyze, binlog_stmt_cache etc.
It does not include engine internal temporary files used for repair,
alter table, index pre sorting etc.
- All internal on disk temporary tables created as part of resolving a
SELECT, multi-source update etc.
Special cases:
- When doing a commit, the last flush of the binlog_stmt_cache
will not cause an error even if the temporary space limit is exceeded.
This is to avoid giving errors on commit. This means that a user
can temporary go over the limit with up to binlog_stmt_cache_size.
Noteworthy issue:
- One has to be careful when using small values for max_tmp_space_limit
together with binary logging and with non transactional tables.
If a the binary log entry for the query is bigger than
binlog_stmt_cache_size and one hits the limit of max_tmp_space_limit
when flushing the entry to disk, the query will abort and the
binary log will not contain the last changes to the table.
This will also stop the slave!
This is also true for all Aria tables as Aria cannot do rollback
(except in case of crashes)!
One way to avoid it is to use @@binlog_format=statement for
queries that updates a lot of rows.
Implementation:
- All writes to temporary files or internal temporary tables, that
increases the file size, are routed through temp_file_size_cb_func()
which updates and checks the temp space usage.
- Most of the temporary file monitoring is done inside IO_CACHE.
Temporary file monitoring is done inside the Aria engine.
- MY_TRACK and MY_TRACK_WITH_LIMIT are new flags for ini_io_cache().
MY_TRACK means that we track the file usage. TRACK_WITH_LIMIT means
that we track the file usage and we give an error if the limit is
breached. This is used to not give an error on commit when
binlog_stmp_cache is flushed.
- global_tmp_space_used contains the total tmp space used so far.
This is needed quickly check against max_total_tmp_space_usage.
- Temporary space errors are using EE_LOCAL_TMP_SPACE_FULL and
handler errors are using HA_ERR_LOCAL_TMP_SPACE_FULL.
This is needed until we move general errors to it's own error space
so that they cannot conflict with system error numbers.
- Return value of my_chsize() and mysql_file_chsize() has changed
so that -1 is returned in the case my_chsize() could not decrease
the file size (very unlikely and will not happen on modern systems).
All calls to _chsize() are updated to check for > 0 as the error
condition.
- At the destruction of THD we check that THD::tmp_file_space == 0
- At server end we check that global_tmp_space_used == 0
- As a precaution against errors in the tmp_space_used code, one can set
max_tmp_space_usage and max_total_tmp_space_usage to 0 to disable
the tmp space quota errors.
- truncate_io_cache() function added.
- Aria tables using static or dynamic row length are registered in 8K
increments to avoid some calls to update_tmp_file_size().
Other things:
- Ensure that all handler errors are registered. Before, some engine
errors could be printed as "Unknown error".
- Fixed bug in filesort() that causes a assert if there was an error
when writing to the temporay file.
- Fixed that compute_window_func() now takes into account write errors.
- In case of parallel replication, rpl_group_info::cleanup_context()
could call trans_rollback() with thd->error set, which would cause
an assert. Fixed by resetting the error before calling trans_rollback().
- Fixed bug in subselect3.inc which caused following test to use
heap tables with low value for max_heap_table_size
- Fixed bug in sql_expression_cache where it did not overflow
heap table to Aria table.
- Added Max_tmp_disk_space_used to slow query log.
- Fixed some bugs in log_slow_innodb.test
2024-03-14 17:59:00 +01:00
|
|
|
--echo # End of 11.5 tests
|