(after discussion with SerG)
include/my_base.h:
Handler error code added
myisam/mi_write.c:
New error code handled
myisam/rt_index.c:
check for zero length added
myisam/sp_key.c:
zero length check added
mysql-test/r/gis-rtree.result:
appropriate test result
mysql-test/t/gis-rtree.test:
test case
sql/handler.cc:
new error code handling added