This commit fixes a bug in the algorithm for converting hexadecimal
strings to binary key values, which leads to incompatibility with other
plugins and reduces the effective information capacity of the keys.
The new key conversion algorithm is incompatible with tables which
alrady encrypted using a old plugin (plugin version less than or
equalt to the 1.05).
This commit contains changes to refactor the the Hashicorp plugin code
which hides all variables previously declared as "static" and which are
not user-visible parameters into a special class that contains all the
plugin's dynamic data. This was done primarily to significantly simplify
the code of the initialization and deinitialization functions, which
previously contained a large number of gotos and complex branching
conditions to control memory deallocation.
This commit fixes an issue with no visible update in caching
option values after changing them dynamically while the server
is running. This issue was related to forgotten copy operations
of new values into dynamic variables. At the same time, internal
variables (responsible for caching) were always updated correctly.
The commit includes a test that checks that the update is now
reflected in the values of dynamic variables.
For the plugin to work properly, we need support for key versioning,
and for this, the kv storage in Hashicorp Vault must be created with
version 2 or higher. This commit adds such a check performed during
plugin initialization.
Note: checking for kv storage version during plugin initialization
can be disabled via --hashicorp-key-management-check-kv-version=off
command-line option or via the corresponding option in the server
configuration files.
According to the Hashicorp Vault API specifications,
the URL to access the keys must include the "/v1/" prefix
at the beginning of the path. This commit adds this parameter
check, as well as a check for the presence of at least one
letter in the hostname inside the URL and in the secret
store name (after "/v1/").
This commit adds an indication of the ID of the not found key
(and, when appropriate, also an indication of the version number
of the key) in the log file, making it easier to find errors.
- Authentication is done using the Hashicorp Vault's token
authentication method;
- If additional client authentication is required, then the
path to the CA authentication bundle file may be passed
as a plugin parameter;
- The creation of the keys and their management is carried
out using the Hashicorp Vault KMS and their tools;
- Key values stored as hexadecimal strings;
- Key values caching is supported.
- Implemented a time-invalidated cache for key values and
for key version numbers received from the Hashicorp Valult
server;
- The plugin uses libcurl (https) as an interface to
the HashiCorp Vault server;
- JSON parsing is performed through the JSON service
(through the include/mysql/service_json.h);
- HashiCorp Vault 1.2.4 was used for development and testing.
The problem happened because the the new client capability flag
CLIENT_EXTENDED_METADATA was not put into the cache entry key.
So results cached by a new client were sent to the old client (and vica versa)
with a mis-matching metadata, which made the client abort the connection on
an unexpected result set metadata packet format.
The problem was caused by the patch for:
MDEV-17832 Protocol: extensions for Pluggable types and JSON, GEOMETRY
which forgot to adjust the query cache code.
Fix:
- Adding a new member Query_cache_query_flags::client_extended_metadata,
so only clients with equal CLIENT_EXTENDED_METADATA flag values can
reuse results.
- Adding a new column CLIENT_EXTENDED_METADATA into
INFORMATION_SCHEMA.QUERY_CACHE_INFO (privided by the qc_info plugin).
The 10.5 version of the patch.
Removing DEFAULT from INFORMATION_SCHEMA columns.
DEFAULT in read-only tables is rather meaningless.
Upgrade should go smoothly.
Also fixes:
MDEV-20254 Problems with EMPTY_STRING_IS_NULL and I_S tables
Removing DEFAULT from INFORMATION_SCHEMA columns.
DEFAULT in read-only tables is rather meaningless.
Upgrade should go smoothly.
Also fixes:
MDEV-20254 Problems with EMPTY_STRING_IS_NULL and I_S tables
Hybrid functions (IF, COALESCE, etc) did not preserve the JSON property
from their arguments. The same problem was repeatable for single row subselects.
The problem happened because the method Item::is_json_type() was inconsistently
implemented across the Item hierarchy. For example, Item_hybrid_func
and Item_singlerow_subselect did not override is_json_type().
Solution:
- Removing Item::is_json_type()
- Implementing specific JSON type handlers:
Type_handler_string_json
Type_handler_varchar_json
Type_handler_tiny_blob_json
Type_handler_blob_json
Type_handler_medium_blob_json
Type_handler_long_blob_json
- Reusing the existing data type infrastructure to pass JSON
type handlers across all item types, including classes Item_hybrid_func
and Item_singlerow_subselect. Note, these two classes themselves do not
need any changes!
- Extending the data type infrastructure so data types can inherit
their properties (e.g. aggregation rules) from their base data types.
E.g. VARCHAR/JSON acts as VARCHAR, LONGTEXT/JSON acts as LONGTEXT
when mixed to a non-JSON data type. This is done by:
- adding virtual method Type_handler::type_handler_base()
- adding a helper class Type_handler_pair
- refactoring Type_handler_hybrid_field_type methods
aggregate_for_result(), aggregate_for_min_max(),
aggregate_for_num_op() to use Type_handler_pair.
This change also fixes:
MDEV-27361 Hybrid functions with JSON arguments do not send format metadata
Also, adding mtr tests for JSON replication. It was not covered yet.
And the current patch changes the replication code slightly.