ATRT_SETUP_README.txt Author: Serge Kozlov, MySQL Date: 03/23/2006 Contents 1. Overview 2. Setup 3. Preparing for testing 4. CPCD. Command line and configuration options 5. d.tmp. Examples 6. Test Case File format. Examples Note: how to run ATRT tests please look ATRT_USAGE_README.txt 1.Overview ========== Auto Test Run Test (ATRT) is a home made test frame work. This frame work can start and stop processes on different Linux hosts through another executable called Cluster Process Control Daemon (CPCD) running on the other hosts. It uses for testing cluster configurations located on different machines. ATRT isn't replacement for mysqltest. In fact, ATRT can invoke mysql-test as samples blow show. This framework has been designed to run most any test (mysqltest testcases, stress tests, any applications) in cluster and detects all errors and issues which happened in any node. In addition, ATRT starts applications that are defined as test with arguments and will analyze log files from all nodes that are produced. 2.Setup ======= Following steps described how to setup ATRT framework on a cluster: * You need to have cloned source tree (e.g mysql-5.1-new) on machine where you plan to use ATRT. * Compile and install build (e.g. for Linux/x86 can use BUILD/compile-pentium-max --prefix=/path/to/installation). * Copy $MYSQL_DIR on all machines and on same path which you plan to use as nodes (you can use scp utility or ask JonathanMiller about distribution on ndbXX servers). * Compile and install ATRT: * Go to $TREE_ROOT/storage/ndb/test * do make * do make install * Make sure that $MYSQL_DIR/bin now contains files like that: testBasic, testBlobs, testDict and so on * Make sure that $MYSQL_DIR/mysql-test/ndb now contains files like that: atrt, atrt-analyze-result.sh, atrt-setup.sh, make-config.sh and so on * Add $MYSQL_DIR/mysql-test/ndb to PATH and make sure that you can call atrt from any place on disk. * Configure CPCD processes on all machines. Repeat following steps for each node: * Create $VAR_DIR/run/ndb_cpcd directory * Create /etc/ndb_cpcd.cnf file. It will configuration file ndb_cpcd daemon. It strongly recommended instead command line options because more simply. More option for configuration can fe found in this document in section CPCD * Add to file following text: [ndb_cpcd] work-dir= $VAR_DIR/run/ndb_cpcd # e.g. /mysql/builds/5.1/var/run/ndb_cpcd logfile= $VAR_DIR/run/ndb_cpcd/log.txt # e.g. /mysql/builds/5.1/var/run/ndb_cpcd/log.txt debug= 1 user= ndbdev * Register ndb_cpcd in etc/initab. Hint: you can use more simply way if use Linux - add following line to /etc/rc.d/rc.local file $MYSQL_DIR/libexec/ndb_cpcd > /dev/null 2>&1 & * Start ndb_cpcd $MYSQL_DIR/libexec/ndb_cpcd > /dev/null 2>&1 & * Open log file and make sure that ndb_cpcd process started properly. Ususal mistakes are: wrong definition of paths and ndb_cpcd process already running. You can test ndb_cpcd from any ndb system by using the command line below. If nothing is returned the process is up and running. Otherwise you will see Failed to connect to node:1234: $MYSQL_DIR/libexec/ndb_cpcc node Now ATRT Setup done. After accomplishment all steps above you should be have following: * Each node contains fresh build * Each node has configured and started ndb_cpcd process. * The node that will used for starting ATRT has installed atrt binaries/scripts and some tests. 3.Preparing for testing ======================= * Create new empty directory where you plan to use for testing. Note: if you plan to test different cluster configuration you need to create own directory for each cluster configuration. Do not use same directory. * Create d.tmp file for your cluster configuration. See details in this document in section d.tmp. * Run make-config.sh ./d.tmp. It will create necessary directories and files for ATRT. * Create ATRT testcase file(s) (e.g. test1.atrt). See details in this document in section Test Case File format. Examples. * Put your test application into $MYSQL_DIR/bin directory. If your application requires another directory (e.g. mysql-test-run.pl) create redirect script such as example below: #!/bin/sh set -x cd $MYSQL_BASE_DIR/mysql-test ./mysql-test-run.pl --with-ndbcluster --ndb-connectstring=$* Now preparing for testing done. 4.CPCD. ======= The CPCD process needs to be running on each host contacts CPCD to tell it what process to execute. In other word for testing each node should have to running ndb_cpcd. Usually ndb_cpcd can be found in storage/ndb/src/cw/cpcd for source tree or in libexec/ndb_cpcd for binary distribution. ndb_cpcd uses configuration file /etc/ndb_cpcd.cnf but all options from one can be replaced by command line. Names of options in command line are same as from configuration file except some some commands have short notation (-X). ndb_cpcd command line options -w, --work-dir=name Work directory. Should be exist before starting ndb_cpcd. Usually it's $VARDIR/run/ndb_cpcd -p, --port=# TCP port to listen on. By default 1234 -S, --syslog Log events to syslog -L, --logfile=name File to log events to. Usually it's $VARDIR/run/ndb_cpcd/logfile.txt -D, --debug Enable debug mode. -u, --user=name Run as user 5.d.tmp ======= The d.tmp file is used to create the d.txt file (configuration file for ATRT) and the config.ini file (configuration file for MySQL Cluster). This file does not have to be called d.tmp, it can be called by any name. The file will be feed to make-config.sh. All options in the file separated by two parts: * Original options. They are located before '-- cluster config'. Description of all these options are below. * Options for config.ini. They are located after '-- cluster config'. All options, sections and values completely coincide with config.ini. d.tmp options baseport Port used for communicating to the cluster on. basedir basedir has to point to the root of the mysql install. Note that ATRT will create a run directory under the base directory. All test directories and files created will be copied to all hosts in the test under the basedir/run directory. mgm Host to NDB Cluster manager on. Put hostnames separated by blanks. ndb Host(s) to run NDB Data Nodes on. Put hostnames separated by blanks. api Host(s) that NDB API should be ran on. Put hostnames separated by blanks. mysqld Host(s) that mysqld processes should be started on. Put hostnames separated by blanks. mysql Host(s) that mysql processes should be started on. Put hostnames separated by blanks. Example d.tmp for cluster configuration: 1 ndb node, 1 replica, 1 mgm, 1 api, 1 mysql, 1 mysqld. Available hosts for nodes: ndb16, ndb17 baseport: 14000 basedir: /home/ndbdev/skozlov/builds mgm: ndb16 ndb: ndb17 api: ndb16 mysqld ndb16 mysql ndb16 -- cluster config [DB DEFAULT] NoOfReplicas: 1 [MGM DEFAULT] PortNumber: 14000 ArbitrationRank: 1 Example d.tmp for cluster configuration: 2 ndb nodes, 2 replicas, 1 mgm, 1 api, 1 mysql, 1 mysqld. Available hosts for nodes: ndb16, ndb17, ndb18 baseport: 14000 basedir: /home/ndbdev/skozlov/builds mgm: ndb16 ndb: ndb17 ndb18 api: ndb16 mysqld ndb16 mysql ndb16 -- cluster config [DB DEFAULT] NoOfReplicas: 2 [MGM DEFAULT] PortNumber: 14000 ArbitrationRank: 1 Example d.tmp for cluster configuration: 4 ndb nodes, 4 replicas, 1 mgm, 3 api, 2 mysql, 1 mysqld. Available hosts for nodes: ndb14, ndb15, ndb16, ndb17, ndb18 baseport: 14000 basedir: /home/ndbdev/skozlov/builds mgm: ndb16 ndb: ndb17 ndb18 ndb15 ndb14 api: ndb16 ndb17 ndb18 mysqld ndb16 mysql ndb16 ndb17 -- cluster config [DB DEFAULT] NoOfReplicas: 4 [MGM DEFAULT] PortNumber: 14000 ArbitrationRank: 1 6.Test Case File Format. Examples. ================================== A test file consists of a list of test cases. Each test case is ended by an empty/blank line. Each test case is described by a set of name value pairs. ATRT looks for tests executables, shell and Perl scripts in the basedir/bin directory. Test Case File options: max-time This value is in seconds. Test ATRT how long to allow the test to run before marking the test a failure and moving on to the next test. If the test finishes before max-time, it will move on to the next test. cmd Tells ATRT which test or script to run from the bin directory args These are the command-line arguments to be passed to the test or script that is being executed. type Currently only bench. See ATRT Command-line parameters for details. run-all will start the same command for each defined api/mysql (normally it only started in 1 instance) Example 1. ATRT starts test $MYSQL_DIR/bin/testBlobs without arguments and sets time for execution as 10 min. testBlobs is binary application max-time: 600 cmd: testBlobs args: Example 2. ATRT starts test $MYSQL_DIR/bin/testRead -n PkRead and sets time for execution as 20 min. testRead is binary application max-time: 1200 cmd: testRead args: -n PkRead Example 3. ATRT starts test $MYSQL_DIR/bin/atrt-mysql-test-run -force and sets time for execution as one hour. atrt-mysql-test-run is bash script and it points to $MYSQL_DIR/mysql-test/mysql-test-run. In fact this test will start mysql-test-run --force that means the execution all mysqltest testcases in mysql-test/t directory. max-time: 3600 cmd: atrt-mysql-test-run args: --force Example 4. ATRT starts test $MYSQL_DIR/bin/MyTest1 -n and sets time for execution as 2 min. Then starts $MYSQL_DIR/bin/MyTest2 for each defined mysql/api node and set timeout 3 min. max-time: 120 cmd: MyTest1 args: -n max-time: 180 cmd: MyTest2 run-all: yes