mariadb/include
Sachin Setiya 84726906c9 MDEV-10177 Invisible Columns and Invisible Index
Feature Definition:-

This feature adds invisible column functionality to server.
There is 4 level of "invisibility":

1. Not invisible (NOT_INVISIBLE) — Normal columns created by the user

2. A little bit invisible (USER_DEFINED_INVISIBLE) — columns that the
    user has marked invisible. They aren't shown in SELECT * and they
    don't require values in INSERT table VALUE (...). Otherwise
    they behave as normal columns.

3. More invisible (SYSTEM_INVISIBLE) — Can be queried explicitly,
    otherwise invisible from everything. Think ROWID sytem column.
    Because they're invisible from ALTER TABLE and from CREATE TABLE
    they cannot be created or dropped, they're created by the system.
    User cant not create a column name which is same as of
    SYSTEM_INVISIBLE.

4. Very invisible (COMPLETELY_INVISIBLE) — as above, but cannot be
    queried either. They can only show up in EXPLAIN EXTENDED (might
    be possible for a very invisible indexed virtual column) but
    otherwise they don't exist for the user.If user creates a columns
    which has same name as of COMPLETELY_INVISIBLE then
    COMPLETELY_INVISIBLE column is renamed again. So it is completely
    invisible from user.

Invisible Index(HA_INVISIBLE_KEY):-
   Creation of invisible columns require a new type of index which
   will be only visible to system. User cant see/alter/create/delete
   this index. If user creates a index which is same name as of
   invisible index then it will be renamed.

Syntax Details:-

  Only USER_DEFINED_INVISIBLE column can be created by user. This
  can be created by adding INVISIBLE suffix after column definition.

  Create table t1( a int invisible, b int);

Rules:-
  There are some rules/restrictions related to use of invisible columns
  1. All the columns in table cant be invisible.
     Create table t1(a int invisible); \\error
     Create table t1(a int invisible, b int invisble); \\error
  2. If you want invisible column to be NOT NULL then you have to supply
     Default value for the column.
     Create table t1(a int, b int not null); \\error
  3. If you create a view/create table with select * then this wont copy
     invisible fields. So newly created view/table wont have any invisible
     columns.
     Create table t2 as select * from t1;//t2 wont have t1 invisible column
     Create view v1 as select * from t1;//v1 wont have t1 invisible column
  4. Invisibility wont be forwarded to next table in any case of create
     table/view as select */(a,b,c) from table.
     Create table t2 as select a,b,c from t1; // t2 will have t1 invisible
                           // column(b), but this wont be invisible in t2
     Create view v1 as select a,b,c from t1; // v1 will have t1 invisible
                           // column(b), but this wont be invisible in v1

Implementation Details:-
  Parsing:- INVISIBLE_SYM is added into vcol_attribute(so its like unique
      suffix), It is also added into keyword_sp_not_data_type so that table
      can have column with name invisible.
  Implementation detail is given by each modified function/created function.
   (Some function are left as they were self explanatory)
   (m= Modified, n= Newly Created)

  mysql_prepare_create_table(m):- Extra checks for invisible columns are
  added. Also some DEBUG_EXECUTE_IF are also added for test cases.

  mysql_prepare_alter_table(m):- Now this will drop all the
  COMPLETELY_INVISIBLE column and HA_INVISIBLE_KEY index. Further
  Modifications are made to stop drop/change/delete of SYSTEM_INVISIBLE
  column.

  build_frm_image(m):- Now this allows incorporating field_visibility
  status into frm image. To remain compatible with old frms
  field_visibility info will be only written when any of the field is
  not NOT_INVISIBLE.

  extra2_write_additional_field_properties(n):- This will write field
  visibility info into buffer. We first write EXTRA2_FIELD_FLAGS into
  buffer/frm , then each next char will have field_visibility for each
  field.

  init_from_binary_frm_image(m):- Now if we get EXTRA2_FIELD_FLAGS,
  then we will read the next n(n= number of fields) chars and set the
  field_visibility. We also increment
  thd->status_var.feature_invisible_columns. One important thing to
  note if we find out that key contains a field whose visibility is
  > USER_DEFINED_INVISIBLE then , we declare this key as invisible
  key.

  sql_show.cc is changed accordingly to make show table, show keys
  correct.

  mysql_insert(m):- If we get to know that we are doing insert in
  this way insert into t1 values(1,1); without explicitly specifying
  columns, then we check for if we have invisible fields if yes then
  we reset the whole record, Why ? Because first we want hidden columns
  to get default/null value. Second thing auto_increment has property
  no default and no null which voilates invisible key rule 2, And
  because of this it was giving error. Reseting table->record[0]
  eliminates this issue. More info put breakpoint on handler::write_row
  and see auto_increment value.

  fill_record(m):- we continue loop if we find invisible column because
  this is already reseted/will get its value if it is default.

Test cases:- Since we can not directly add > USER_DEFINED_INVISIBLE
  column then I have debug_dbug to create it in mysql_prepare_create_table.

  Patch Credit:- Serg Golubchik
2017-12-15 02:41:52 +05:30
..
atomic Merge bb-10.2-ext into 10.3 2017-12-12 09:57:17 +02:00
mysql Cleanups 2017-12-08 11:38:22 +02:00
big_endian.h Temporary commit of 10.0-merge 2013-03-26 00:03:13 +02:00
byte_order_generic.h Temporary commit of 10.0-merge 2013-03-26 00:03:13 +02:00
byte_order_generic_x86.h MDEV-10951 Field_newdate::cmp access violation 2016-10-25 22:35:35 +00:00
byte_order_generic_x86_64.h Merge branch '10.1' into 10.2 2016-12-29 13:23:18 +01:00
CMakeLists.txt MDEV-13773 client packages need my_global.h and/or my_config.h 2017-09-18 10:12:23 +02:00
decimal.h Adding "const" qualifier to the argument of decimal_actual_fraction(). 2016-01-14 17:23:23 +04:00
dur_prop.h Merge InnoDB 5.7 from mysql-5.7.9. 2016-09-02 13:22:28 +03:00
errmsg.h 5.5.38 merge 2014-06-06 00:07:27 +02:00
ft_global.h mysql-5.1.73 merge 2014-03-15 18:24:15 +01:00
handler_ername.h Fix for MDEV-5547: Bad error message when moving very old .frm files to MariaDB 5.5. 2014-01-22 15:16:57 +02:00
handler_state.h Fix for MDEV-533: Confusing error code when doing auto-increment insert for out-of-range values 2012-09-18 15:14:19 +03:00
hash.h Enusure that my_global.h is included first 2017-08-24 01:05:44 +02:00
heap.h 5.5 merge 2014-03-26 22:25:38 +01:00
json_lib.h MDEV-11856 json_search doesn't search for values with double quotes 2017-03-14 15:25:02 +04:00
keycache.h Fixed problem with very slow shutdown when using 100,000 MyISAM tables with delay_key_write 2014-07-19 17:46:08 +03:00
lf.h Restore LF_BACKOFF 2017-12-08 13:44:45 +02:00
little_endian.h Temporary commit of 10.0-merge 2013-03-26 00:03:13 +02:00
m_ctype.h Enusure that my_global.h is included first 2017-08-24 01:05:44 +02:00
m_string.h Less dependencies in include files 2017-11-23 09:49:45 +02:00
ma_dyncol.h MDEV-8949: COLUMN_CREATE unicode name breakage 2017-11-14 10:49:46 +01:00
maria.h Fix that end_bulk_insert() doesn't write to to-be-deleted files 2017-05-17 00:34:48 +03:00
my_alarm.h MDEV-8450: PATCH] Wrong macro expansion in Query_cache::send_result_to_client() 2015-09-06 22:26:33 +02:00
my_alloc.h Reducing memory when using information schema 2017-11-02 20:37:25 +02:00
my_atomic.h Merge bb-10.2-ext into 10.3 2017-12-12 09:57:17 +02:00
my_attribute.h 5.1 merge 2014-03-16 13:59:44 +01:00
my_base.h MDEV-10177 Invisible Columns and Invisible Index 2017-12-15 02:41:52 +05:30
my_bit.h Enusure that my_global.h is included first 2017-08-24 01:05:44 +02:00
my_bitmap.h Adding multi_range_read support to partitions 2017-12-03 13:58:34 +02:00
my_byteorder.h Temporary commit of 10.0-merge 2013-03-26 00:03:13 +02:00
my_check_opt.h 5.5 merge 2015-01-21 12:03:02 +01:00
my_compare.h Merge branch '10.1' into 10.2 2016-12-29 13:23:18 +01:00
my_compiler.h Merge bb-10.2-ext into 10.3 2017-10-04 08:24:06 +03:00
my_context.h Merge branch '10.1' into 10.2 2016-03-23 22:36:46 +01:00
my_cpu.h Merge bb-10.2-ext into 10.3 2017-12-12 09:57:17 +02:00
my_crypt.h MDEV-10332 support for OpenSSL 1.1 and LibreSSL 2017-05-09 18:53:10 +02:00
my_dbug.h don't flush dbug buffers for every assert 2017-08-28 15:33:03 +02:00
my_decimal_limits.h Merge branch '10.1' into 10.2 2017-03-30 12:48:42 +02:00
my_default.h Correct FSF address 2017-03-10 18:21:29 +01:00
my_dir.h Enusure that my_global.h is included first 2017-08-24 01:05:44 +02:00
my_getopt.h New option for slow logging (log_slow_disable_statements) 2017-08-24 01:05:51 +02:00
my_global.h Merge branch 'bb-10.2-ext' into 10.3 2017-08-26 00:34:43 +02:00
my_handler_errors.h Merge 10.2 into bb-10.2-ext 2017-09-20 17:47:49 +03:00
my_libwrap.h Bug 16395495 - OLD FSF ADDRESS IN GPL HEADER 2013-03-19 13:29:12 +01:00
my_list.h Bug 16395495 - OLD FSF ADDRESS IN GPL HEADER 2013-03-19 13:29:12 +01:00
my_md5.h encryption cleanup: small changes 2015-04-05 13:14:37 +02:00
my_net.h Enusure that my_global.h is included first 2017-08-24 01:05:44 +02:00
my_nosys.h Bug 16395495 - OLD FSF ADDRESS IN GPL HEADER 2013-03-19 13:29:12 +01:00
my_pthread.h Enusure that my_global.h is included first 2017-08-24 01:05:44 +02:00
my_rdtsc.h MDEV-9172 - Analyze patches for IBM System z 2015-12-09 16:52:12 +04:00
my_rnd.h Correct FSF address 2017-03-10 18:21:29 +01:00
my_stacktrace.h Added DBUG_ASSERT_AS_PRINTF compile flag 2017-08-24 01:05:50 +02:00
my_sys.h Added DBUG_ASSERT_AS_PRINTF compile flag 2017-08-24 01:05:50 +02:00
my_systemd.h MDEV-427/MDEV-5713 Add systemd script with notify functionality 2015-10-12 17:51:49 +02:00
my_time.h Enusure that my_global.h is included first 2017-08-24 01:05:44 +02:00
my_tree.h Fix that end_bulk_insert() doesn't write to to-be-deleted files 2017-05-17 00:34:48 +03:00
my_uctype.h
my_user.h Enusure that my_global.h is included first 2017-08-24 01:05:44 +02:00
my_valgrind.h Merge branch '10.1' into 10.2 2017-03-30 12:48:42 +02:00
my_xml.h MDEV-4928 Merge collation customization improvements 2013-10-02 15:04:07 +04:00
myisam.h Fix that end_bulk_insert() doesn't write to to-be-deleted files 2017-05-17 00:34:48 +03:00
myisamchk.h Merge branch '10.1' into 10.2 2017-03-30 12:48:42 +02:00
myisammrg.h Added "const" to new data for handler::update_row() 2017-04-18 12:23:53 +03:00
myisampack.h Optimized intkorr() and intstore(functions) for intel 64 bits. 2016-02-07 10:34:03 +02:00
mysql.h Merge branch '10.1' into 10.2 2017-03-30 12:48:42 +02:00
mysql.h.pp Merge bb-10.2-ext into 10.3 2017-11-10 16:12:45 +02:00
mysql_async.h MDEV-13384 - misc Windows warnings fixed 2017-09-28 17:20:46 +00:00
mysql_com.h MDEV-11371 - column compression 2017-08-31 15:44:17 +04:00
mysql_com_server.h Next part of merge. See TODO for details 2012-08-14 17:23:34 +03:00
mysql_embed.h
mysql_time.h 10.0-base merge 2013-07-18 16:46:57 +02:00
mysql_version.h.in MDEV-12501 -- set --maturity-level by default 2017-12-09 23:34:43 +00:00
mysqld_default_groups.h MDEV-34 delete storage/ndb and sql/*ndb* (and collateral changes) 2014-10-11 18:53:06 +02:00
mysys_err.h
pack.h Merge branch '10.2' into bb-10.2-connector-c-integ-subm 2016-09-21 12:54:56 +02:00
password.h Enusure that my_global.h is included first 2017-08-24 01:05:44 +02:00
probes_mysql.d.base
probes_mysql.h more portable fix for lp:942266 - 5.5 builds fail with systemtap-sdt-dev installed on Ubuntu 2012-05-11 09:18:00 +02:00
probes_mysql_nodtrace.h.in Merge branch '5.5' into 10.0 2016-02-15 22:50:59 +01:00
queues.h Enusure that my_global.h is included first 2017-08-24 01:05:44 +02:00
rijndael.h Bug 16395495 - OLD FSF ADDRESS IN GPL HEADER 2013-03-19 13:29:12 +01:00
service_versions.h MDEV-13384 Fix Windows warnings. thd_alloc functions now use size_t parameters 2017-09-28 17:20:46 +00:00
source_revision.h.in SOURCE_REVISION should always be defined in source_revision.h 2017-11-01 13:20:32 +00:00
sql_common.h BUG#25575605: SETTING --SSL-MODE=REQUIRED SENDS CREDENTIALS BEFORE VERIFYING SSL CONNECTION 2017-03-10 14:11:26 +05:30
ssl_compat.h MDEV-12763 10.2 uses deprecated openssl 1.0 apis even with 1.1 2017-09-18 10:12:23 +02:00
sslopt-case.h BUG#25575605: SETTING --SSL-MODE=REQUIRED SENDS CREDENTIALS BEFORE VERIFYING SSL CONNECTION 2017-03-10 14:11:26 +05:30
sslopt-longopts.h WL#9072: Backport WL#8785 to 5.5 2016-02-19 23:31:10 +04:00
sslopt-vars.h BUG#25575605: SETTING --SSL-MODE=REQUIRED SENDS CREDENTIALS BEFORE VERIFYING SSL CONNECTION 2017-03-10 14:11:26 +05:30
t_ctype.h Updated/added copyright header. Added line "use is subject to license terms" 2014-02-17 18:19:04 +05:30
thr_alarm.h 5.2 merge 2014-03-16 21:03:01 +01:00
thr_lock.h Fix many -Wconversion warnings. 2017-03-07 19:07:27 +02:00
thr_timer.h followup changes to timeout commit 2014-10-10 22:27:36 +02:00
thread_pool_priv.h MDEV-11418 - AliSQL: [Feature] Issue#1 KILL IDLE TRANSACTIONS 2017-03-22 19:08:24 +04:00
typelib.h Fix many -Wconversion warnings. 2017-03-07 19:07:27 +02:00
violite.h MDEV-14412 Support TCP keepalive options 2017-11-17 21:40:20 +00:00
waiting_threads.h Enusure that my_global.h is included first 2017-08-24 01:05:44 +02:00
welcome_copyright_notice.h Merge remote-tracking branch 'mysql/5.5' into 5.5 2017-04-11 10:18:04 -04:00
wqueue.h Enusure that my_global.h is included first 2017-08-24 01:05:44 +02:00
wsrep.h Merge remote-tracking branch 'origin/10.0-galera' into 10.1 2017-08-21 13:35:00 +03:00