MDEV-12026: Implement innodb_checksum_algorithm=full_crc32
MariaDB data-at-rest encryption (innodb_encrypt_tables)
had repurposed the same unused data field that was repurposed
in MySQL 5.7 (and MariaDB 10.2) for the Split Sequence Number (SSN)
field of SPATIAL INDEX. Because of this, MariaDB was unable to
support encryption on SPATIAL INDEX pages.
Furthermore, InnoDB page checksums skipped some bytes, and there
are multiple variations and checksum algorithms. By default,
InnoDB accepts all variations of all algorithms that ever existed.
This unnecessarily weakens the page checksums.
We hereby introduce two more innodb_checksum_algorithm variants
(full_crc32, strict_full_crc32) that are special in a way:
When either setting is active, newly created data files will
carry a flag (fil_space_t::full_crc32()) that indicates that
all pages of the file will use a full CRC-32C checksum over the
entire page contents (excluding the bytes where the checksum
is stored, at the very end of the page). Such files will always
use that checksum, no matter what the parameter
innodb_checksum_algorithm is assigned to.
For old files, the old checksum algorithms will continue to be
used. The value strict_full_crc32 will be equivalent to strict_crc32
and the value full_crc32 will be equivalent to crc32.
ROW_FORMAT=COMPRESSED tables will only use the old format.
These tables do not support new features, such as larger
innodb_page_size or instant ADD/DROP COLUMN. They may be
deprecated in the future. We do not want an unnecessary
file format change for them.
The new full_crc32() format also cleans up the MariaDB tablespace
flags. We will reserve flags to store the page_compressed
compression algorithm, and to store the compressed payload length,
so that checksum can be computed over the compressed (and
possibly encrypted) stream and can be validated without
decrypting or decompressing the page.
In the full_crc32 format, there no longer are separate before-encryption
and after-encryption checksums for pages. The single checksum is
computed on the page contents that is written to the file.
We do not make the new algorithm the default for two reasons.
First, MariaDB 10.4.2 was a beta release, and the default values
of parameters should not change after beta. Second, we did not
yet implement the full_crc32 format for page_compressed pages.
This will be fixed in MDEV-18644.
This is joint work with Marko Mäkelä.
2019-02-19 21:00:00 +02:00
|
|
|
-- source include/have_innodb.inc
|
2019-05-29 22:17:00 +03:00
|
|
|
# This test is slow on buildbot.
|
|
|
|
|
--source include/big_test.inc
|
MDEV-12026: Implement innodb_checksum_algorithm=full_crc32
MariaDB data-at-rest encryption (innodb_encrypt_tables)
had repurposed the same unused data field that was repurposed
in MySQL 5.7 (and MariaDB 10.2) for the Split Sequence Number (SSN)
field of SPATIAL INDEX. Because of this, MariaDB was unable to
support encryption on SPATIAL INDEX pages.
Furthermore, InnoDB page checksums skipped some bytes, and there
are multiple variations and checksum algorithms. By default,
InnoDB accepts all variations of all algorithms that ever existed.
This unnecessarily weakens the page checksums.
We hereby introduce two more innodb_checksum_algorithm variants
(full_crc32, strict_full_crc32) that are special in a way:
When either setting is active, newly created data files will
carry a flag (fil_space_t::full_crc32()) that indicates that
all pages of the file will use a full CRC-32C checksum over the
entire page contents (excluding the bytes where the checksum
is stored, at the very end of the page). Such files will always
use that checksum, no matter what the parameter
innodb_checksum_algorithm is assigned to.
For old files, the old checksum algorithms will continue to be
used. The value strict_full_crc32 will be equivalent to strict_crc32
and the value full_crc32 will be equivalent to crc32.
ROW_FORMAT=COMPRESSED tables will only use the old format.
These tables do not support new features, such as larger
innodb_page_size or instant ADD/DROP COLUMN. They may be
deprecated in the future. We do not want an unnecessary
file format change for them.
The new full_crc32() format also cleans up the MariaDB tablespace
flags. We will reserve flags to store the page_compressed
compression algorithm, and to store the compressed payload length,
so that checksum can be computed over the compressed (and
possibly encrypted) stream and can be validated without
decrypting or decompressing the page.
In the full_crc32 format, there no longer are separate before-encryption
and after-encryption checksums for pages. The single checksum is
computed on the page contents that is written to the file.
We do not make the new algorithm the default for two reasons.
First, MariaDB 10.4.2 was a beta release, and the default values
of parameters should not change after beta. Second, we did not
yet implement the full_crc32 format for page_compressed pages.
This will be fixed in MDEV-18644.
This is joint work with Marko Mäkelä.
2019-02-19 21:00:00 +02:00
|
|
|
|
|
|
|
|
FLUSH TABLES;
|
|
|
|
|
|
|
|
|
|
let $MYSQLD_TMPDIR = `SELECT @@tmpdir`;
|
|
|
|
|
let $MYSQLD_DATADIR = `SELECT @@datadir`;
|
|
|
|
|
|
|
|
|
|
--echo # Treating compact format as dynamic format after import stmt
|
|
|
|
|
|
|
|
|
|
CREATE TABLE t1
|
|
|
|
|
(a int AUTO_INCREMENT PRIMARY KEY,
|
|
|
|
|
b blob,
|
|
|
|
|
c blob,
|
|
|
|
|
KEY (b(200))) ENGINE=InnoDB ROW_FORMAT=COMPACT;
|
|
|
|
|
|
2020-02-28 12:59:30 +02:00
|
|
|
BEGIN;
|
MDEV-12026: Implement innodb_checksum_algorithm=full_crc32
MariaDB data-at-rest encryption (innodb_encrypt_tables)
had repurposed the same unused data field that was repurposed
in MySQL 5.7 (and MariaDB 10.2) for the Split Sequence Number (SSN)
field of SPATIAL INDEX. Because of this, MariaDB was unable to
support encryption on SPATIAL INDEX pages.
Furthermore, InnoDB page checksums skipped some bytes, and there
are multiple variations and checksum algorithms. By default,
InnoDB accepts all variations of all algorithms that ever existed.
This unnecessarily weakens the page checksums.
We hereby introduce two more innodb_checksum_algorithm variants
(full_crc32, strict_full_crc32) that are special in a way:
When either setting is active, newly created data files will
carry a flag (fil_space_t::full_crc32()) that indicates that
all pages of the file will use a full CRC-32C checksum over the
entire page contents (excluding the bytes where the checksum
is stored, at the very end of the page). Such files will always
use that checksum, no matter what the parameter
innodb_checksum_algorithm is assigned to.
For old files, the old checksum algorithms will continue to be
used. The value strict_full_crc32 will be equivalent to strict_crc32
and the value full_crc32 will be equivalent to crc32.
ROW_FORMAT=COMPRESSED tables will only use the old format.
These tables do not support new features, such as larger
innodb_page_size or instant ADD/DROP COLUMN. They may be
deprecated in the future. We do not want an unnecessary
file format change for them.
The new full_crc32() format also cleans up the MariaDB tablespace
flags. We will reserve flags to store the page_compressed
compression algorithm, and to store the compressed payload length,
so that checksum can be computed over the compressed (and
possibly encrypted) stream and can be validated without
decrypting or decompressing the page.
In the full_crc32 format, there no longer are separate before-encryption
and after-encryption checksums for pages. The single checksum is
computed on the page contents that is written to the file.
We do not make the new algorithm the default for two reasons.
First, MariaDB 10.4.2 was a beta release, and the default values
of parameters should not change after beta. Second, we did not
yet implement the full_crc32 format for page_compressed pages.
This will be fixed in MDEV-18644.
This is joint work with Marko Mäkelä.
2019-02-19 21:00:00 +02:00
|
|
|
INSERT INTO t1 (b, c) values (repeat("ab", 200), repeat("bc", 200));
|
|
|
|
|
INSERT INTO t1 (b, c) values (repeat("bc", 200), repeat("cd", 200));
|
|
|
|
|
INSERT INTO t1 (b, c) values (repeat("cd", 200), repeat("ef", 200));
|
|
|
|
|
INSERT INTO t1 (b, c) values (repeat("de", 200), repeat("fg", 200));
|
|
|
|
|
INSERT INTO t1 (b, c) values (repeat("ef", 200), repeat("gh", 200));
|
|
|
|
|
INSERT INTO t1 (b, c) values (repeat("fg", 200), repeat("hi", 200));
|
|
|
|
|
INSERT INTO t1 (b, c) values (repeat("gh", 200), repeat("ij", 200));
|
|
|
|
|
INSERT INTO t1 (b, c) values (repeat("hi", 200), repeat("jk", 200));
|
|
|
|
|
INSERT INTO t1 (b, c) values (repeat("ij", 200), repeat("kl", 200));
|
|
|
|
|
INSERT INTO t1 (b, c) values (repeat("jk", 200), repeat("lm", 200));
|
|
|
|
|
INSERT INTO t1 (b, c) SELECT b,c FROM t1 ORDER BY a;
|
|
|
|
|
INSERT INTO t1 (b, c) SELECT b,c FROM t1 ORDER BY a;
|
2020-02-28 12:59:30 +02:00
|
|
|
COMMIT;
|
MDEV-12026: Implement innodb_checksum_algorithm=full_crc32
MariaDB data-at-rest encryption (innodb_encrypt_tables)
had repurposed the same unused data field that was repurposed
in MySQL 5.7 (and MariaDB 10.2) for the Split Sequence Number (SSN)
field of SPATIAL INDEX. Because of this, MariaDB was unable to
support encryption on SPATIAL INDEX pages.
Furthermore, InnoDB page checksums skipped some bytes, and there
are multiple variations and checksum algorithms. By default,
InnoDB accepts all variations of all algorithms that ever existed.
This unnecessarily weakens the page checksums.
We hereby introduce two more innodb_checksum_algorithm variants
(full_crc32, strict_full_crc32) that are special in a way:
When either setting is active, newly created data files will
carry a flag (fil_space_t::full_crc32()) that indicates that
all pages of the file will use a full CRC-32C checksum over the
entire page contents (excluding the bytes where the checksum
is stored, at the very end of the page). Such files will always
use that checksum, no matter what the parameter
innodb_checksum_algorithm is assigned to.
For old files, the old checksum algorithms will continue to be
used. The value strict_full_crc32 will be equivalent to strict_crc32
and the value full_crc32 will be equivalent to crc32.
ROW_FORMAT=COMPRESSED tables will only use the old format.
These tables do not support new features, such as larger
innodb_page_size or instant ADD/DROP COLUMN. They may be
deprecated in the future. We do not want an unnecessary
file format change for them.
The new full_crc32() format also cleans up the MariaDB tablespace
flags. We will reserve flags to store the page_compressed
compression algorithm, and to store the compressed payload length,
so that checksum can be computed over the compressed (and
possibly encrypted) stream and can be validated without
decrypting or decompressing the page.
In the full_crc32 format, there no longer are separate before-encryption
and after-encryption checksums for pages. The single checksum is
computed on the page contents that is written to the file.
We do not make the new algorithm the default for two reasons.
First, MariaDB 10.4.2 was a beta release, and the default values
of parameters should not change after beta. Second, we did not
yet implement the full_crc32 format for page_compressed pages.
This will be fixed in MDEV-18644.
This is joint work with Marko Mäkelä.
2019-02-19 21:00:00 +02:00
|
|
|
SELECT COUNT(*) FROM t1;
|
|
|
|
|
|
|
|
|
|
FLUSH TABLE t1 FOR EXPORT;
|
|
|
|
|
--echo # List before copying files
|
|
|
|
|
let MYSQLD_DATADIR =`SELECT @@datadir`;
|
|
|
|
|
|
|
|
|
|
--list_files $MYSQLD_DATADIR/test
|
|
|
|
|
perl;
|
|
|
|
|
do "$ENV{MTR_SUITE_DIR}/include/innodb-util.pl";
|
|
|
|
|
ib_backup_tablespaces("test", "t1");
|
|
|
|
|
EOF
|
|
|
|
|
|
|
|
|
|
UNLOCK TABLES;
|
|
|
|
|
ALTER TABLE t1 ROW_FORMAT=DYNAMIC;
|
|
|
|
|
ALTER TABLE t1 DISCARD TABLESPACE;
|
|
|
|
|
|
|
|
|
|
--list_files $MYSQLD_DATADIR/test
|
|
|
|
|
perl;
|
|
|
|
|
do "$ENV{MTR_SUITE_DIR}/include/innodb-util.pl";
|
|
|
|
|
ib_discard_tablespaces("test", "t1");
|
|
|
|
|
ib_restore_tablespaces("test", "t1");
|
|
|
|
|
EOF
|
|
|
|
|
|
|
|
|
|
--remove_file $MYSQLD_DATADIR/test/t1.cfg
|
2019-02-21 18:29:17 +01:00
|
|
|
--disable_warnings
|
MDEV-12026: Implement innodb_checksum_algorithm=full_crc32
MariaDB data-at-rest encryption (innodb_encrypt_tables)
had repurposed the same unused data field that was repurposed
in MySQL 5.7 (and MariaDB 10.2) for the Split Sequence Number (SSN)
field of SPATIAL INDEX. Because of this, MariaDB was unable to
support encryption on SPATIAL INDEX pages.
Furthermore, InnoDB page checksums skipped some bytes, and there
are multiple variations and checksum algorithms. By default,
InnoDB accepts all variations of all algorithms that ever existed.
This unnecessarily weakens the page checksums.
We hereby introduce two more innodb_checksum_algorithm variants
(full_crc32, strict_full_crc32) that are special in a way:
When either setting is active, newly created data files will
carry a flag (fil_space_t::full_crc32()) that indicates that
all pages of the file will use a full CRC-32C checksum over the
entire page contents (excluding the bytes where the checksum
is stored, at the very end of the page). Such files will always
use that checksum, no matter what the parameter
innodb_checksum_algorithm is assigned to.
For old files, the old checksum algorithms will continue to be
used. The value strict_full_crc32 will be equivalent to strict_crc32
and the value full_crc32 will be equivalent to crc32.
ROW_FORMAT=COMPRESSED tables will only use the old format.
These tables do not support new features, such as larger
innodb_page_size or instant ADD/DROP COLUMN. They may be
deprecated in the future. We do not want an unnecessary
file format change for them.
The new full_crc32() format also cleans up the MariaDB tablespace
flags. We will reserve flags to store the page_compressed
compression algorithm, and to store the compressed payload length,
so that checksum can be computed over the compressed (and
possibly encrypted) stream and can be validated without
decrypting or decompressing the page.
In the full_crc32 format, there no longer are separate before-encryption
and after-encryption checksums for pages. The single checksum is
computed on the page contents that is written to the file.
We do not make the new algorithm the default for two reasons.
First, MariaDB 10.4.2 was a beta release, and the default values
of parameters should not change after beta. Second, we did not
yet implement the full_crc32 format for page_compressed pages.
This will be fixed in MDEV-18644.
This is joint work with Marko Mäkelä.
2019-02-19 21:00:00 +02:00
|
|
|
ALTER TABLE t1 IMPORT TABLESPACE;
|
2019-02-21 18:29:17 +01:00
|
|
|
--enable_warnings
|
MDEV-12026: Implement innodb_checksum_algorithm=full_crc32
MariaDB data-at-rest encryption (innodb_encrypt_tables)
had repurposed the same unused data field that was repurposed
in MySQL 5.7 (and MariaDB 10.2) for the Split Sequence Number (SSN)
field of SPATIAL INDEX. Because of this, MariaDB was unable to
support encryption on SPATIAL INDEX pages.
Furthermore, InnoDB page checksums skipped some bytes, and there
are multiple variations and checksum algorithms. By default,
InnoDB accepts all variations of all algorithms that ever existed.
This unnecessarily weakens the page checksums.
We hereby introduce two more innodb_checksum_algorithm variants
(full_crc32, strict_full_crc32) that are special in a way:
When either setting is active, newly created data files will
carry a flag (fil_space_t::full_crc32()) that indicates that
all pages of the file will use a full CRC-32C checksum over the
entire page contents (excluding the bytes where the checksum
is stored, at the very end of the page). Such files will always
use that checksum, no matter what the parameter
innodb_checksum_algorithm is assigned to.
For old files, the old checksum algorithms will continue to be
used. The value strict_full_crc32 will be equivalent to strict_crc32
and the value full_crc32 will be equivalent to crc32.
ROW_FORMAT=COMPRESSED tables will only use the old format.
These tables do not support new features, such as larger
innodb_page_size or instant ADD/DROP COLUMN. They may be
deprecated in the future. We do not want an unnecessary
file format change for them.
The new full_crc32() format also cleans up the MariaDB tablespace
flags. We will reserve flags to store the page_compressed
compression algorithm, and to store the compressed payload length,
so that checksum can be computed over the compressed (and
possibly encrypted) stream and can be validated without
decrypting or decompressing the page.
In the full_crc32 format, there no longer are separate before-encryption
and after-encryption checksums for pages. The single checksum is
computed on the page contents that is written to the file.
We do not make the new algorithm the default for two reasons.
First, MariaDB 10.4.2 was a beta release, and the default values
of parameters should not change after beta. Second, we did not
yet implement the full_crc32 format for page_compressed pages.
This will be fixed in MDEV-18644.
This is joint work with Marko Mäkelä.
2019-02-19 21:00:00 +02:00
|
|
|
SHOW CREATE TABLE t1;
|
|
|
|
|
UPDATE t1 set b = repeat("de", 100) where b = repeat("cd", 200);
|
|
|
|
|
--replace_column 9 #
|
|
|
|
|
explain SELECT a FROM t1 where b = repeat("de", 100);
|
|
|
|
|
SELECT a FROM t1 where b = repeat("de", 100);
|
|
|
|
|
SELECT COUNT(*) FROM t1;
|
|
|
|
|
DELETE FROM t1;
|
|
|
|
|
--source include/wait_all_purged.inc
|
|
|
|
|
CHECK TABLE t1;
|
|
|
|
|
DROP TABLE t1;
|
|
|
|
|
|
|
|
|
|
CREATE TABLE t1
|
|
|
|
|
(c1 int AUTO_INCREMENT PRIMARY KEY,
|
|
|
|
|
c2 POINT NOT NULL,
|
|
|
|
|
c3 LINESTRING NOT NULL,
|
|
|
|
|
SPATIAL INDEX idx1(c2)) ENGINE=InnoDB ROW_FORMAT=COMPACT;
|
|
|
|
|
|
|
|
|
|
INSERT INTO t1(c2,c3) VALUES(
|
|
|
|
|
ST_GeomFromText('POINT(10 10)'),
|
|
|
|
|
ST_GeomFromText('LINESTRING(5 5,20 20,30 30)'));
|
|
|
|
|
|
|
|
|
|
INSERT INTO t1(c2,c3) VALUES(
|
|
|
|
|
ST_GeomFromText('POINT(20 20)'),
|
|
|
|
|
ST_GeomFromText('LINESTRING(5 15,20 10,30 20)'));
|
|
|
|
|
|
|
|
|
|
INSERT INTO t1(c2,c3) VALUES(
|
|
|
|
|
ST_GeomFromText('POINT(30 30)'),
|
|
|
|
|
ST_GeomFromText('LINESTRING(10 5,20 24,30 32)'));
|
|
|
|
|
|
|
|
|
|
INSERT INTO t1(c2,c3) VALUES(
|
|
|
|
|
ST_GeomFromText('POINT(40 40)'),
|
|
|
|
|
ST_GeomFromText('LINESTRING(15 5,25 20,35 30)'));
|
|
|
|
|
|
|
|
|
|
INSERT INTO t1(c2,c3) VALUES(
|
|
|
|
|
ST_GeomFromText('POINT(50 10)'),
|
|
|
|
|
ST_GeomFromText('LINESTRING(15 15,24 10,31 20)'));
|
|
|
|
|
|
|
|
|
|
INSERT INTO t1(c2,c3) VALUES(
|
|
|
|
|
ST_GeomFromText('POINT(60 50)'),
|
|
|
|
|
ST_GeomFromText('LINESTRING(10 15,20 44,35 32)'));
|
|
|
|
|
|
2020-02-28 12:59:30 +02:00
|
|
|
BEGIN;
|
MDEV-12026: Implement innodb_checksum_algorithm=full_crc32
MariaDB data-at-rest encryption (innodb_encrypt_tables)
had repurposed the same unused data field that was repurposed
in MySQL 5.7 (and MariaDB 10.2) for the Split Sequence Number (SSN)
field of SPATIAL INDEX. Because of this, MariaDB was unable to
support encryption on SPATIAL INDEX pages.
Furthermore, InnoDB page checksums skipped some bytes, and there
are multiple variations and checksum algorithms. By default,
InnoDB accepts all variations of all algorithms that ever existed.
This unnecessarily weakens the page checksums.
We hereby introduce two more innodb_checksum_algorithm variants
(full_crc32, strict_full_crc32) that are special in a way:
When either setting is active, newly created data files will
carry a flag (fil_space_t::full_crc32()) that indicates that
all pages of the file will use a full CRC-32C checksum over the
entire page contents (excluding the bytes where the checksum
is stored, at the very end of the page). Such files will always
use that checksum, no matter what the parameter
innodb_checksum_algorithm is assigned to.
For old files, the old checksum algorithms will continue to be
used. The value strict_full_crc32 will be equivalent to strict_crc32
and the value full_crc32 will be equivalent to crc32.
ROW_FORMAT=COMPRESSED tables will only use the old format.
These tables do not support new features, such as larger
innodb_page_size or instant ADD/DROP COLUMN. They may be
deprecated in the future. We do not want an unnecessary
file format change for them.
The new full_crc32() format also cleans up the MariaDB tablespace
flags. We will reserve flags to store the page_compressed
compression algorithm, and to store the compressed payload length,
so that checksum can be computed over the compressed (and
possibly encrypted) stream and can be validated without
decrypting or decompressing the page.
In the full_crc32 format, there no longer are separate before-encryption
and after-encryption checksums for pages. The single checksum is
computed on the page contents that is written to the file.
We do not make the new algorithm the default for two reasons.
First, MariaDB 10.4.2 was a beta release, and the default values
of parameters should not change after beta. Second, we did not
yet implement the full_crc32 format for page_compressed pages.
This will be fixed in MDEV-18644.
This is joint work with Marko Mäkelä.
2019-02-19 21:00:00 +02:00
|
|
|
INSERT INTO t1(c2, c3) SELECT c2, c3 FROM t1;
|
|
|
|
|
INSERT INTO t1(c2, c3) SELECT c2, c3 FROM t1;
|
|
|
|
|
INSERT INTO t1(c2, c3) SELECT c2, c3 FROM t1;
|
|
|
|
|
INSERT INTO t1(c2, c3) SELECT c2, c3 FROM t1;
|
|
|
|
|
INSERT INTO t1(c2, c3) SELECT c2, c3 FROM t1;
|
|
|
|
|
INSERT INTO t1(c2, c3) SELECT c2, c3 FROM t1;
|
|
|
|
|
INSERT INTO t1(c2, c3) SELECT c2, c3 FROM t1;
|
|
|
|
|
INSERT INTO t1(c2, c3) SELECT c2, c3 FROM t1;
|
|
|
|
|
INSERT INTO t1(c2, c3) SELECT c2, c3 FROM t1;
|
|
|
|
|
INSERT INTO t1(c2, c3) SELECT c2, c3 FROM t1;
|
|
|
|
|
INSERT INTO t1(c2, c3) SELECT c2, c3 FROM t1;
|
2020-02-28 12:59:30 +02:00
|
|
|
COMMIT;
|
MDEV-12026: Implement innodb_checksum_algorithm=full_crc32
MariaDB data-at-rest encryption (innodb_encrypt_tables)
had repurposed the same unused data field that was repurposed
in MySQL 5.7 (and MariaDB 10.2) for the Split Sequence Number (SSN)
field of SPATIAL INDEX. Because of this, MariaDB was unable to
support encryption on SPATIAL INDEX pages.
Furthermore, InnoDB page checksums skipped some bytes, and there
are multiple variations and checksum algorithms. By default,
InnoDB accepts all variations of all algorithms that ever existed.
This unnecessarily weakens the page checksums.
We hereby introduce two more innodb_checksum_algorithm variants
(full_crc32, strict_full_crc32) that are special in a way:
When either setting is active, newly created data files will
carry a flag (fil_space_t::full_crc32()) that indicates that
all pages of the file will use a full CRC-32C checksum over the
entire page contents (excluding the bytes where the checksum
is stored, at the very end of the page). Such files will always
use that checksum, no matter what the parameter
innodb_checksum_algorithm is assigned to.
For old files, the old checksum algorithms will continue to be
used. The value strict_full_crc32 will be equivalent to strict_crc32
and the value full_crc32 will be equivalent to crc32.
ROW_FORMAT=COMPRESSED tables will only use the old format.
These tables do not support new features, such as larger
innodb_page_size or instant ADD/DROP COLUMN. They may be
deprecated in the future. We do not want an unnecessary
file format change for them.
The new full_crc32() format also cleans up the MariaDB tablespace
flags. We will reserve flags to store the page_compressed
compression algorithm, and to store the compressed payload length,
so that checksum can be computed over the compressed (and
possibly encrypted) stream and can be validated without
decrypting or decompressing the page.
In the full_crc32 format, there no longer are separate before-encryption
and after-encryption checksums for pages. The single checksum is
computed on the page contents that is written to the file.
We do not make the new algorithm the default for two reasons.
First, MariaDB 10.4.2 was a beta release, and the default values
of parameters should not change after beta. Second, we did not
yet implement the full_crc32 format for page_compressed pages.
This will be fixed in MDEV-18644.
This is joint work with Marko Mäkelä.
2019-02-19 21:00:00 +02:00
|
|
|
|
|
|
|
|
FLUSH TABLE t1 FOR EXPORT;
|
|
|
|
|
--echo # List before copying files
|
|
|
|
|
let MYSQLD_DATADIR =`SELECT @@datadir`;
|
|
|
|
|
|
|
|
|
|
--list_files $MYSQLD_DATADIR/test
|
|
|
|
|
perl;
|
|
|
|
|
do "$ENV{MTR_SUITE_DIR}/include/innodb-util.pl";
|
|
|
|
|
ib_backup_tablespaces("test", "t1");
|
|
|
|
|
EOF
|
|
|
|
|
|
|
|
|
|
UNLOCK TABLES;
|
|
|
|
|
ALTER TABLE t1 ROW_FORMAT=DYNAMIC;
|
|
|
|
|
ALTER TABLE t1 DISCARD TABLESPACE;
|
|
|
|
|
perl;
|
|
|
|
|
do "$ENV{MTR_SUITE_DIR}/include/innodb-util.pl";
|
|
|
|
|
ib_discard_tablespaces("test", "t1");
|
|
|
|
|
ib_restore_tablespaces("test", "t1");
|
|
|
|
|
EOF
|
|
|
|
|
|
|
|
|
|
--remove_file $MYSQLD_DATADIR/test/t1.cfg
|
2019-02-21 18:29:17 +01:00
|
|
|
--disable_warnings
|
MDEV-12026: Implement innodb_checksum_algorithm=full_crc32
MariaDB data-at-rest encryption (innodb_encrypt_tables)
had repurposed the same unused data field that was repurposed
in MySQL 5.7 (and MariaDB 10.2) for the Split Sequence Number (SSN)
field of SPATIAL INDEX. Because of this, MariaDB was unable to
support encryption on SPATIAL INDEX pages.
Furthermore, InnoDB page checksums skipped some bytes, and there
are multiple variations and checksum algorithms. By default,
InnoDB accepts all variations of all algorithms that ever existed.
This unnecessarily weakens the page checksums.
We hereby introduce two more innodb_checksum_algorithm variants
(full_crc32, strict_full_crc32) that are special in a way:
When either setting is active, newly created data files will
carry a flag (fil_space_t::full_crc32()) that indicates that
all pages of the file will use a full CRC-32C checksum over the
entire page contents (excluding the bytes where the checksum
is stored, at the very end of the page). Such files will always
use that checksum, no matter what the parameter
innodb_checksum_algorithm is assigned to.
For old files, the old checksum algorithms will continue to be
used. The value strict_full_crc32 will be equivalent to strict_crc32
and the value full_crc32 will be equivalent to crc32.
ROW_FORMAT=COMPRESSED tables will only use the old format.
These tables do not support new features, such as larger
innodb_page_size or instant ADD/DROP COLUMN. They may be
deprecated in the future. We do not want an unnecessary
file format change for them.
The new full_crc32() format also cleans up the MariaDB tablespace
flags. We will reserve flags to store the page_compressed
compression algorithm, and to store the compressed payload length,
so that checksum can be computed over the compressed (and
possibly encrypted) stream and can be validated without
decrypting or decompressing the page.
In the full_crc32 format, there no longer are separate before-encryption
and after-encryption checksums for pages. The single checksum is
computed on the page contents that is written to the file.
We do not make the new algorithm the default for two reasons.
First, MariaDB 10.4.2 was a beta release, and the default values
of parameters should not change after beta. Second, we did not
yet implement the full_crc32 format for page_compressed pages.
This will be fixed in MDEV-18644.
This is joint work with Marko Mäkelä.
2019-02-19 21:00:00 +02:00
|
|
|
ALTER TABLE t1 IMPORT TABLESPACE;
|
2019-02-21 18:29:17 +01:00
|
|
|
--enable_warnings
|
MDEV-12026: Implement innodb_checksum_algorithm=full_crc32
MariaDB data-at-rest encryption (innodb_encrypt_tables)
had repurposed the same unused data field that was repurposed
in MySQL 5.7 (and MariaDB 10.2) for the Split Sequence Number (SSN)
field of SPATIAL INDEX. Because of this, MariaDB was unable to
support encryption on SPATIAL INDEX pages.
Furthermore, InnoDB page checksums skipped some bytes, and there
are multiple variations and checksum algorithms. By default,
InnoDB accepts all variations of all algorithms that ever existed.
This unnecessarily weakens the page checksums.
We hereby introduce two more innodb_checksum_algorithm variants
(full_crc32, strict_full_crc32) that are special in a way:
When either setting is active, newly created data files will
carry a flag (fil_space_t::full_crc32()) that indicates that
all pages of the file will use a full CRC-32C checksum over the
entire page contents (excluding the bytes where the checksum
is stored, at the very end of the page). Such files will always
use that checksum, no matter what the parameter
innodb_checksum_algorithm is assigned to.
For old files, the old checksum algorithms will continue to be
used. The value strict_full_crc32 will be equivalent to strict_crc32
and the value full_crc32 will be equivalent to crc32.
ROW_FORMAT=COMPRESSED tables will only use the old format.
These tables do not support new features, such as larger
innodb_page_size or instant ADD/DROP COLUMN. They may be
deprecated in the future. We do not want an unnecessary
file format change for them.
The new full_crc32() format also cleans up the MariaDB tablespace
flags. We will reserve flags to store the page_compressed
compression algorithm, and to store the compressed payload length,
so that checksum can be computed over the compressed (and
possibly encrypted) stream and can be validated without
decrypting or decompressing the page.
In the full_crc32 format, there no longer are separate before-encryption
and after-encryption checksums for pages. The single checksum is
computed on the page contents that is written to the file.
We do not make the new algorithm the default for two reasons.
First, MariaDB 10.4.2 was a beta release, and the default values
of parameters should not change after beta. Second, we did not
yet implement the full_crc32 format for page_compressed pages.
This will be fixed in MDEV-18644.
This is joint work with Marko Mäkelä.
2019-02-19 21:00:00 +02:00
|
|
|
SHOW CREATE TABLE t1;
|
|
|
|
|
UPDATE t1 SET C2 = ST_GeomFromText('POINT(0 0)');
|
|
|
|
|
SELECT COUNT(*) FROM t1;
|
|
|
|
|
DELETE FROM t1;
|
|
|
|
|
CHECK TABLE t1;
|
|
|
|
|
--source include/wait_all_purged.inc
|
|
|
|
|
DROP TABLE t1;
|
2019-09-10 17:41:26 +05:30
|
|
|
|
|
|
|
|
SET @save_algo = @@GLOBAL.innodb_compression_algorithm;
|
2020-01-07 13:25:36 +02:00
|
|
|
--error 0,ER_WRONG_VALUE_FOR_VAR
|
2019-09-10 17:41:26 +05:30
|
|
|
SET GLOBAL innodb_compression_algorithm=2;
|
|
|
|
|
CREATE TABLE t1(a SERIAL) PAGE_COMPRESSED=1 ENGINE=InnoDB;
|
|
|
|
|
INSERT INTO t1 VALUES(1);
|
|
|
|
|
|
|
|
|
|
FLUSH TABLE t1 FOR EXPORT;
|
|
|
|
|
--echo # List before copying files
|
|
|
|
|
let MYSQLD_DATADIR =`SELECT @@datadir`;
|
|
|
|
|
|
|
|
|
|
--list_files $MYSQLD_DATADIR/test
|
|
|
|
|
perl;
|
|
|
|
|
do "$ENV{MTR_SUITE_DIR}/include/innodb-util.pl";
|
|
|
|
|
ib_backup_tablespaces("test", "t1");
|
|
|
|
|
EOF
|
|
|
|
|
|
|
|
|
|
UNLOCK TABLES;
|
|
|
|
|
SET GLOBAL innodb_compression_algorithm=0;
|
|
|
|
|
ALTER TABLE t1 FORCE;
|
|
|
|
|
ALTER TABLE t1 DISCARD TABLESPACE;
|
|
|
|
|
|
|
|
|
|
--list_files $MYSQLD_DATADIR/test
|
|
|
|
|
perl;
|
|
|
|
|
do "$ENV{MTR_SUITE_DIR}/include/innodb-util.pl";
|
|
|
|
|
ib_discard_tablespaces("test", "t1");
|
|
|
|
|
ib_restore_tablespaces("test", "t1");
|
|
|
|
|
EOF
|
|
|
|
|
|
|
|
|
|
ALTER TABLE t1 IMPORT TABLESPACE;
|
|
|
|
|
INSERT INTO t1 VALUES(2);
|
|
|
|
|
SELECT * FROM t1;
|
|
|
|
|
|
2019-10-20 15:48:20 +03:00
|
|
|
|
2020-01-07 13:25:36 +02:00
|
|
|
--error 0,ER_WRONG_VALUE_FOR_VAR
|
2019-09-10 17:41:26 +05:30
|
|
|
SET GLOBAL innodb_compression_algorithm=3;
|
|
|
|
|
FLUSH TABLE t1 FOR EXPORT;
|
|
|
|
|
--echo # List before copying files
|
|
|
|
|
let MYSQLD_DATADIR =`SELECT @@datadir`;
|
|
|
|
|
|
|
|
|
|
--list_files $MYSQLD_DATADIR/test
|
|
|
|
|
perl;
|
|
|
|
|
do "$ENV{MTR_SUITE_DIR}/include/innodb-util.pl";
|
|
|
|
|
ib_backup_tablespaces("test", "t1");
|
|
|
|
|
EOF
|
|
|
|
|
|
|
|
|
|
UNLOCK TABLES;
|
|
|
|
|
SET GLOBAL innodb_compression_algorithm=0;
|
|
|
|
|
ALTER TABLE t1 FORCE;
|
|
|
|
|
ALTER TABLE t1 DISCARD TABLESPACE;
|
|
|
|
|
|
|
|
|
|
--list_files $MYSQLD_DATADIR/test
|
|
|
|
|
perl;
|
|
|
|
|
do "$ENV{MTR_SUITE_DIR}/include/innodb-util.pl";
|
|
|
|
|
ib_discard_tablespaces("test", "t1");
|
|
|
|
|
ib_restore_tablespaces("test", "t1");
|
|
|
|
|
EOF
|
|
|
|
|
|
|
|
|
|
ALTER TABLE t1 IMPORT TABLESPACE;
|
|
|
|
|
INSERT INTO t1 VALUES(3);
|
|
|
|
|
SELECT * FROM t1;
|
|
|
|
|
DROP TABLE t1;
|
|
|
|
|
|
|
|
|
|
SET GLOBAL innodb_compression_algorithm=@save_algo;
|