mirror/mariadb
1
0
Fork 0
mirror of https://github.com/MariaDB/server.git synced 2025-01-31 11:01:52 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions

ATRT documentation

Browse source
This commit is contained in:
skozlov@mysql.com 2006-03-27 14:59:19 +04:00
parent ddcfa13d27
commit 8cf57c5520
2 changed files with 516 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

292
storage/ndb/test/run-test/ATRT_SETUP_README.txt Normal file
View file

@ -0,0 +1,292 @@
ATRT_SETUP_README.txt
Author: Serge Kozlov, MySQL
Date: 03/23/2006
Contents
1. Overview
2. Setup
3. Preparing for testing
4. CPCD. Command line and configuration options
5. d.tmp. Examples
6. Test Case File format. Examples
Note: how to run ATRT tests please look ATRT_USAGE_README.txt
1.Overview
==========
Auto Test Run Test (ATRT) is a home made test frame work. This frame work can
start and stop processes on different Linux hosts through another executable
called Cluster Process Control Daemon (CPCD) running on the other hosts.
It uses for testing cluster configurations located on different machines.
ATRT isn't replacement for mysqltest. In fact, ATRT can invoke mysql-test as
samples blow show. This framework has been designed to run most any test
(mysqltest testcases, stress tests, any applications) in cluster and detects
all errors and issues which happened in any node. In addition, ATRT starts
applications that are defined as test with arguments and will analyze log files
from all nodes that are produced.
2.Setup
=======
Following steps described how to setup ATRT framework on a cluster:
* You need to have cloned source tree (e.g mysql-5.1-new) on machine where you
plan to use ATRT.
* Compile and install build (e.g. for Linux/x86 can use
BUILD/compile-pentium-max --prefix=/path/to/installation).
* Copy $MYSQL_DIR on all machines and on same path which you plan to use as
nodes (you can use scp utility or ask JonathanMiller about distribution on
ndbXX servers).
* Compile and install ATRT:
* Go to $TREE_ROOT/storage/ndb/test
* do make
* do make install
* Make sure that $MYSQL_DIR/bin now contains files like that: testBasic,
testBlobs, testDict and so on
* Make sure that $MYSQL_DIR/mysql-test/ndb now contains files like that: atrt,
atrt-analyze-result.sh, atrt-setup.sh, make-config.sh and so on
* Add $MYSQL_DIR/mysql-test/ndb to PATH and make sure that you can call atrt
from any place on disk.
* Configure CPCD processes on all machines. Repeat following steps for each
node:
* Create $VAR_DIR/run/ndb_cpcd directory
* Create /etc/ndb_cpcd.cnf file. It will configuration file ndb_cpcd daemon.
It strongly recommended instead command line options because more simply.
More option for configuration can fe found in this document in section CPCD
* Add to file following text:
[ndb_cpcd]
work-dir= $VAR_DIR/run/ndb_cpcd # e.g. /mysql/builds/5.1/var/run/ndb_cpcd
logfile= $VAR_DIR/run/ndb_cpcd/log.txt # e.g. /mysql/builds/5.1/var/run/ndb_cpcd/log.txt
debug= 1
user= ndbdev
* Register ndb_cpcd in etc/initab. Hint: you can use more simply way if use
Linux - add following line to /etc/rc.d/rc.local file
$MYSQL_DIR/libexec/ndb_cpcd > /dev/null 2>&1 &
* Start ndb_cpcd
$MYSQL_DIR/libexec/ndb_cpcd > /dev/null 2>&1 &
* Open log file and make sure that ndb_cpcd process started properly. Ususal mistakes are: wrong definition of paths and ndb_cpcd process already running. You can test ndb_cpcd from any ndb system by using the command line below. If nothing is returned the process is up and running. Otherwise you will see Failed to connect to node:1234:
$MYSQL_DIR/libexec/ndb_cpcc node
Now ATRT Setup done. After accomplishment all steps above you should be have
following:
* Each node contains fresh build
* Each node has configured and started ndb_cpcd process.
* The node that will used for starting ATRT has installed atrt binaries/scripts
and some tests.
3.Preparing for testing
=======================
* Create new empty directory where you plan to use for testing. Note: if you
plan to test different cluster configuration you need to create own directory
for each cluster configuration. Do not use same directory.
* Create d.tmp file for your cluster configuration. See details in this
document in section d.tmp.
* Run make-config.sh ./d.tmp. It will create necessary directories and files for
ATRT.
* Create ATRT testcase file(s) (e.g. test1.atrt). See details in this document
in section Test Case File format. Examples.
* Put your test application into $MYSQL_DIR/bin directory. If your application
requires another directory (e.g. mysql-test-run.pl) create redirect script
such as example below:
#!/bin/sh
set -x
cd $MYSQL_BASE_DIR/mysql-test
./mysql-test-run.pl --with-ndbcluster --ndb-connectstring=$*
Now preparing for testing done.
4.CPCD.
=======
The CPCD process needs to be running on each host contacts CPCD to tell it
what process to execute. In other word for testing each node should have to
running ndb_cpcd. Usually ndb_cpcd can be found in storage/ndb/src/cw/cpcd for
source tree or in libexec/ndb_cpcd for binary distribution. ndb_cpcd uses
configuration file /etc/ndb_cpcd.cnf but all options from one can be replaced
by command line. Names of options in command line are same as from
configuration file except some some commands have short notation (-X).
ndb_cpcd command line options
-w, --work-dir=name
Work directory. Should be exist before starting ndb_cpcd. Usually it's
$VARDIR/run/ndb_cpcd
-p, --port=#
TCP port to listen on. By default 1234
-S, --syslog
Log events to syslog
-L, --logfile=name
File to log events to. Usually it's $VARDIR/run/ndb_cpcd/logfile.txt
-D, --debug
Enable debug mode.
-u, --user=name
Run as user
5.d.tmp
=======
The d.tmp file is used to create the d.txt file (configuration file for ATRT)
and the config.ini file (configuration file for MySQL Cluster). This file does
not have to be called d.tmp, it can be called by any name. The file will be feed
to make-config.sh. All options in the file separated by two parts:
* Original options. They are located before '-- cluster config'. Description of
all these options are below.
* Options for config.ini. They are located after '-- cluster config'. All
options, sections and values completely coincide with config.ini.
d.tmp options
baseport
Port used for communicating to the cluster on.
basedir
basedir has to point to the root of the mysql install. Note that ATRT will
create a run directory under the base directory. All test directories and
files created will be copied to all hosts in the test under the basedir/run directory.
mgm
Host to NDB Cluster manager on. Put hostnames separated by blanks.
ndb
Host(s) to run NDB Data Nodes on. Put hostnames separated by blanks.
api
Host(s) that NDB API should be ran on. Put hostnames separated by blanks.
mysqld
Host(s) that mysqld processes should be started on. Put hostnames separated by blanks.
mysql
Host(s) that mysql processes should be started on. Put hostnames separated
by blanks.
Example d.tmp for cluster configuration: 1 ndb node, 1 replica, 1 mgm, 1 api,
1 mysql, 1 mysqld. Available hosts for nodes: ndb16, ndb17
baseport: 14000
basedir: /home/ndbdev/skozlov/builds
mgm: ndb16
ndb: ndb17
api: ndb16
mysqld ndb16
mysql ndb16
-- cluster config
[DB DEFAULT]
NoOfReplicas: 1
[MGM DEFAULT]
PortNumber: 14000
ArbitrationRank: 1
Example d.tmp for cluster configuration: 2 ndb nodes, 2 replicas, 1 mgm, 1 api,
1 mysql, 1 mysqld. Available hosts for nodes: ndb16, ndb17, ndb18
baseport: 14000
basedir: /home/ndbdev/skozlov/builds
mgm: ndb16
ndb: ndb17 ndb18
api: ndb16
mysqld ndb16
mysql ndb16
-- cluster config
[DB DEFAULT]
NoOfReplicas: 2
[MGM DEFAULT]
PortNumber: 14000
ArbitrationRank: 1
Example d.tmp for cluster configuration: 4 ndb nodes, 4 replicas, 1 mgm, 3 api,
2 mysql, 1 mysqld. Available hosts for nodes: ndb14, ndb15, ndb16, ndb17, ndb18
baseport: 14000
basedir: /home/ndbdev/skozlov/builds
mgm: ndb16
ndb: ndb17 ndb18 ndb15 ndb14
api: ndb16 ndb17 ndb18
mysqld ndb16
mysql ndb16 ndb17
-- cluster config
[DB DEFAULT]
NoOfReplicas: 4
[MGM DEFAULT]
PortNumber: 14000
ArbitrationRank: 1
6.Test Case File Format. Examples.
==================================
A test file consists of a list of test cases. Each test case is ended by an
empty/blank line. Each test case is described by a set of name value pairs.
ATRT looks for tests executables, shell and Perl scripts in the basedir/bin
directory.
Test Case File options:
max-time
This value is in seconds. Test ATRT how long to allow the test to run before
marking the test a failure and moving on to the next test. If the test
finishes before max-time, it will move on to the next test.
cmd
Tells ATRT which test or script to run from the bin directory
args
These are the command-line arguments to be passed to the test or script
that is being executed.
type
Currently only bench. See ATRT Command-line parameters for details.
run-all
will start the same command for each defined api/mysql (normally it only
started in 1 instance)
Example 1.
ATRT starts test $MYSQL_DIR/bin/testBlobs without arguments and sets time for
execution as 10 min. testBlobs is binary application
max-time: 600
cmd: testBlobs
args:
Example 2.
ATRT starts test $MYSQL_DIR/bin/testRead -n PkRead and sets time for execution
as 20 min. testRead is binary application
max-time: 1200
cmd: testRead
args: -n PkRead
Example 3.
ATRT starts test $MYSQL_DIR/bin/atrt-mysql-test-run -force and sets time for
execution as one hour. atrt-mysql-test-run is bash script and it points to
$MYSQL_DIR/mysql-test/mysql-test-run. In fact this test will start
mysql-test-run --force that means the execution all mysqltest testcases in
mysql-test/t directory.
max-time: 3600
cmd: atrt-mysql-test-run
args: --force
Example 4.
ATRT starts test $MYSQL_DIR/bin/MyTest1 -n and sets time for execution as 2 min.
Then starts $MYSQL_DIR/bin/MyTest2 for each defined mysql/api node and set
timeout 3 min.
max-time: 120
cmd: MyTest1
args: -n
max-time: 180
cmd: MyTest2
run-all: yes

224
storage/ndb/test/run-test/ATRT_USAGE_README.txt Normal file
View file

@ -0,0 +1,224 @@
ATRT_USAGE_README.txt
Author: Serge Kozlov, MySQL
Date: 03/23/2006
Contents
1. How to run
2. Results
3. ATRT Command Line Options
Note: how to setup ATRT please look ATRT_SETUP_README.txt
1.How to run.
=============
Simple way to start atrt:
atrt --testcase-file=atrt_test_case_file
Command line above doesn't produce any log files. Better use following :
atrt --log-file=log.txt --testcase-file=atrt_test_case_file
Now we can look log.txt for investigation any issues. If it is insufficiently
then add one or more -v arguments:
atrt -v -v --log-file=log.txt --testcase-file=atrt_test_case_file
If the test case file contains two or more test we can add -r options for
preventing stopping testing if one test fails (like --force for mysql-test-run)
atrt -v -v -r --log-file=log.txt --testcase-file=atrt_test_case_file
The line below is optimal solution for testing:
atrt -v -v -r -R --log-file=log.txt --testcase-file=atrt_test_case_file
All additional command line arguments and description of used in exampes above
available in this document in section atrt command line options
2.Results.
==========
Unlike mysql-test-run frame work atrt doesn't inform to console passed/failed
status of tests. You need to use --log-file option and look into log file for
getting information about status of tests. When atrt finished you can look into
log file defined --log-file option. It's main source of information about how
were performed atrt tests. Below added the examples of content of log-file for
different failures (except example 1 for passed test). Examples include
probable cases with reasons and recommended solutions and cover not test issues
only but mistakes of atrt configuration or cluster settings. As ATRT testcase
file used simple test included in MySQL installation:
max-time: 600
cmd: testBasic
args: -n PkRead? T1
Of course these examples don't cover all possible failures but at least most
probable and often appearing.
Note: Before start atrt I recommend try to run selected cluster configuration
by manual and make sure that it can work: enough to run mgmd, ndbd, mysqld and
look via mgm to status of these nodes
Example 1. Test passed
2006-03-02 15:36:51 [ndb_atrt] INFO -- Setup path not specified, using /home/ndbdev/skozlov/asetup
2006-03-02 15:36:51 [ndb_atrt] INFO -- Starting...
2006-03-02 15:36:51 [ndb_atrt] INFO -- Connecting to hosts
2006-03-02 15:36:51 [ndb_atrt] DEBUG -- Connected to node1:1234
2006-03-02 15:36:51 [ndb_atrt] DEBUG -- Connected to node2:1234
2006-03-02 15:36:51 [ndb_atrt] DEBUG -- Connected to node3:1234
2006-03-02 15:36:55 [ndb_atrt] INFO -- (Re)starting ndb processes
2006-03-02 15:37:11 [ndb_atrt] INFO -- Ndb start completed
2006-03-02 15:37:11 [ndb_atrt] INFO -- #1 - testBasic -n PkRead? T1
2006-03-02 16:37:16 [ndb_atrt] INFO -- #1 OK (0)
Example 2. Test failed.
Reason: ATRT not started properly. d.txt not found. Probably necessary folders
and d.txt file don't exist.
Solution: run make-config.sh d.tmp and try again.
2006-03-02 18:32:08 [ndb_atrt] INFO -- Setup path not specified, using /home/ndbdev/skozlov/asetup
2006-03-02 18:32:08 [ndb_atrt] INFO -- Starting...
2006-03-02 18:32:08 [ndb_atrt] CRITICAL -- Failed to open process config file: d.txt
Example 3. Test failed.
Reason: ATRT not started properly. Necessary folders were removed but d.txt file
exists.
Solution: run make-config.sh d.tmp and try again.
2006-03-02 18:30:54 [ndb_atrt] INFO -- Setup path not specified, using /home/ndbdev/skozlov/asetup
2006-03-02 18:30:54 [ndb_atrt] INFO -- Starting...
2006-03-02 18:30:54 [ndb_atrt] INFO -- Connecting to hosts
2006-03-02 18:30:54 [ndb_atrt] DEBUG -- Connected to node1:1234
2006-03-02 18:30:54 [ndb_atrt] DEBUG -- Connected to node2:1234
2006-03-02 18:30:54 [ndb_atrt] DEBUG -- Connected to node3:1234
2006-03-02 18:30:58 [ndb_atrt] INFO -- (Re)starting ndb processes
2006-03-02 18:31:00 [ndb_atrt] CRITICAL -- Failed to setup process
Example 4. Test failed.
Reason: ATRT not started properly. node2 hasn't running ndb_cpcd process.
Solution: log into node2 and starts ndb_cpcd process.
2006-03-02 18:15:05 [ndb_atrt] INFO -- Setup path not specified, using /home/ndbdev/skozlov/asetup
2006-03-02 18:15:05 [ndb_atrt] INFO -- Starting...
2006-03-02 18:15:05 [ndb_atrt] INFO -- Connecting to hosts
2006-03-02 18:15:05 [ndb_atrt] DEBUG -- Connected to node1:1234
2006-03-02 18:15:05 [ndb_atrt] ERROR -- Unable to connect to cpc node2:1234
Example 5. Test failed.
Reason: ATRT not started properly. baseport option isn't defined in d.tmp or
baseport and PortNumber are different.
Solution: correct d.tmp, run make-config.sh d.tmp and try again.
2006-03-02 18:25:31 [ndb_atrt] INFO -- Setup path not specified, using /home/ndbdev/skozlov/asetup
2006-03-02 18:25:31 [ndb_atrt] INFO -- Starting...
2006-03-02 18:25:31 [ndb_atrt] INFO -- Connecting to hosts
2006-03-02 18:25:31 [ndb_atrt] DEBUG -- Connected to node1:1234
2006-03-02 18:25:31 [ndb_atrt] DEBUG -- Connected to node2:1234
2006-03-02 18:25:31 [ndb_atrt] DEBUG -- Connected to node3:1234
2006-03-02 18:25:36 [ndb_atrt] INFO -- (Re)starting ndb processes
2006-03-02 18:26:08 [ndb_atrt] CRITICAL -- Unable to connect to ndb mgm node1:0
Example 6. Test failed.
Reason: ATRT not started properly. basedir option points to wrong path.
Solution: correct basedir, run make-config.sh d.tmp and try again.
2006-03-02 18:40:10 [ndb_atrt] INFO -- Setup path not specified, using /home/ndbdev/skozlov/asetup
2006-03-02 18:40:10 [ndb_atrt] INFO -- Starting...
2006-03-02 18:40:10 [ndb_atrt] INFO -- Connecting to hosts
2006-03-02 18:40:10 [ndb_atrt] DEBUG -- Connected to node1:1234
2006-03-02 18:40:10 [ndb_atrt] DEBUG -- Connected to node2:1234
2006-03-02 18:40:10 [ndb_atrt] DEBUG -- Connected to node3:1234
2006-03-02 18:40:14 [ndb_atrt] INFO -- (Re)starting ndb processes
2006-03-02 18:40:16 [ndb_atrt] ERROR -- Unable to start process: Failed to start
Example 7. Test failed.
Reason: ndb nodes have problems on starting.
Solution: Check configuration of ndb nodes.
2006-03-02 18:46:44 [ndb_atrt] INFO -- Setup path not specified, using /home/ndbdev/skozlov/asetup
2006-03-02 18:46:44 [ndb_atrt] INFO -- Starting...
2006-03-02 18:46:44 [ndb_atrt] INFO -- Connecting to hosts
2006-03-02 18:46:44 [ndb_atrt] DEBUG -- Connected to node1:1234
2006-03-02 18:46:44 [ndb_atrt] DEBUG -- Connected to node2:1234
2006-03-02 18:46:44 [ndb_atrt] DEBUG -- Connected to node3:1234
2006-03-02 18:46:50 [ndb_atrt] INFO -- (Re)starting ndb processes
2006-03-02 18:52:59 [ndb_atrt] CRITICAL -- wait ndb timed out 3 3 4
2006-03-02 18:58:59 [ndb_atrt] CRITICAL -- wait ndb timed out 3 3 4
Example 8. Test failed.
Reason: test application not found.
Solution: Correct file name in ATRT test case file and make sure that the file
exists in $MYSQL_DIR/bin directory.
2006-03-02 20:21:54 [ndb_atrt] INFO -- Starting...
2006-03-02 20:21:54 [ndb_atrt] INFO -- Connecting to hosts
2006-03-02 20:21:54 [ndb_atrt] DEBUG -- Connected to node1:1234
2006-03-02 20:21:54 [ndb_atrt] DEBUG -- Connected to node2:1234
2006-03-02 20:21:54 [ndb_atrt] DEBUG -- Connected to node3:1234
2006-03-02 20:21:57 [ndb_atrt] INFO -- (Re)starting ndb processes
2006-03-02 20:22:12 [ndb_atrt] INFO -- Ndb start completed
2006-03-02 20:22:12 [ndb_atrt] INFO -- #1 - testBasic123 -n PkRead? T2
2006-03-02 20:22:13 [ndb_atrt] ERROR -- Unable to start process: Failed to start
Example 9. Test failed.
Probable reasons:
* wrong arguments for test application
* itself test failed
* timeout reached
Solution: Since mgmd/ndbd nodes started properly in such case then try to
investigate log files in result/X.api/, result/X.mysqld, result/X.mysql
directories.
2006-03-02 19:59:35 [ndb_atrt] INFO -- Setup path not specified, using /home/ndbdev/skozlov/asetup
2006-03-02 19:59:35 [ndb_atrt] INFO -- Starting...
2006-03-02 19:59:35 [ndb_atrt] INFO -- Connecting to hosts
2006-03-02 19:59:35 [ndb_atrt] DEBUG -- Connected to node1:1234
2006-03-02 19:59:35 [ndb_atrt] DEBUG -- Connected to node2:1234
2006-03-02 19:59:35 [ndb_atrt] DEBUG -- Connected to node3:1234
2006-03-02 20:00:49 [ndb_atrt] INFO -- (Re)starting ndb processes
2006-03-02 20:01:05 [ndb_atrt] INFO -- Ndb start completed
2006-03-02 20:01:05 [ndb_atrt] INFO -- #1 - testBasic -n PkRead? T1
2006-03-02 20:01:38 [ndb_atrt] INFO -- #1 FAILED(256)
3.ATRT Command Line Options.
============================
--process-config=string
Specify ATRT configuration file. If not specified, ATRT will look in local
directory for the d.txt file.
--setup-path=string
This path points to place where necessary directories (created by
make-config) can be found. Note: d.txt should be in same directory where
you start atrt!
-v (verbose)
* without the option: atrt prints only [CRITICAL] events
* one -v: atrt prints [CRITICAL], [INFO] events
* two -v: atrt prints [CRITICAL], [INFO], [DEBUG] events
--log-file=string
Used to specify file to log ATRT's results on starting application and
running tests.
--testcase-file=string -f
Used to feed ATRT test cases in a text file.
--report-file=string
File to record test results
-i, --interactive
ATRT terminates on first test failure
-r, --regression
Continues even on test failures
-b, --bench
Always produce report