|
TRCBasicInstallation
Tungsten Replicator Cookbook - basic installation
Basic InstallationGet the replicatorAll the recipes about installation require that you get Tungsten Replicator binaries. This recipe explains how to do it once for ever.
Install a master / slave cluster
TUNGSTEN_HOME=$HOME/replication
MASTER=m1.mynetwork.com
SLAVE1=m2.mynetwork.com
SLAVE2=m3.mynetwork.com
./tools/tungsten-installer \
--master-slave \
--master-host=$MASTER \
--datasource-user=MYUSER \
--datasource-password=MYPWD \
--service-name=dragon \
--home-directory=$TUNGSTEN_HOME \
--cluster-hosts=$MASTER,$SLAVE1,$SLAVE2 \
--start-and-reportThis command takes several defaults into account. It assumes that MySQL is installed in /var/lib/mysql/, that the server uses port 3306, and that the binary logs are in the data directory. We use these values because they are the ones used by RPM installers. If you use different values, please see TRCBasicInstallation#Install_a_master_slave_directory_with_customized_parameters. Install a master slave directory with customized parametersIf the defaults used in TRCBasicInstallation#Install_a_master_/_slave_cluster are not convenient, you can instruct the replicator to use paths and ports that describe your server. You should add these options to the installation command. --datasource-port=${MASTER_MYSQL_PORT} \
--datasource-log-directory=/path/to/your/binary/logs \
--datasource-log-pattern=my-binInstall more than one Tungsten Replicator in one hostYou may install more than one Tungsten Replicator in the same host. One good reason for doing it may be because you want to test several servers at once, or because you have more MySQL servers in the same host and they need to replicate from different masters. In the first case, you may want to have a look at Tungsten sandbox. But if you want to do that on your own, you need to make sure that there is no overlapping of three base elements in the replicator:
For example, after installing Tungsten master/slave, I want to use an additional database with port 7102 and data directory somewhere else. With the following command, I install a second replicator, using a different directory for the replicator, and pointing to the existing replicator ./tools/tungsten-installer \ --master-slave \ --cluster-hosts=127.0.0.1 \ --master-host=HOST_NAME_OF_THE_MASTER \ --master-thl-port=2112 \ --datasource-port=7102 \ --datasource-user=msandbox \ --datasource-password=msandbox \ --service-name=SAME_NAME_OF_THE_MASTER_SLAVE_SERVICE \ --home-directory=/tmp/some_place \ --thl-port=4112 \ --rmi-port=12000 \ --start-and-report Your replicator will end up in /tmp/some_place To check the status you may run /tmp/some_place/tungsten/tungsten-replicator/bin/trepctl -port 12000 services Install a direct slave with parallel replication
Direct slave replication is a quick method to set replication in a slave, without installing a master replicator. It works by establishing a connection with the master, fetching the binary logs locally, and creating a THL stream with that. The additional effort of creating relay logs is then compensated by using parallel replication. This installation assumes that the slave is not running native MySQL replication. For that case, see TRCBasicInstallation#Taking_over_replication_from_a_MySQL_slave_in_direct_mode. TUNGSTEN_HOME=$HOME/replication
MASTER=m1.mynetwork.com
SLAVE1=m2.mynetwork.com
./tools/tungsten-installer \
--direct \
--master-host=$MASTER \
--slave-host=$SLAVE \
--master-user=tungsten \
--slave-user=tungsten \
--master-password=secret \
--slave-password=secret \
--service-name=mydirect \
--channels=10 \
--home-directory=$TUNGSTEN_BASE \
--svc-parallelization-type=disk \
--start-and-report Notice that you need to pass connection credentials for both the master and the slave. As in regular master/slave replication, there are defaults, which you can override using the following options (defaults in brackets) --master-host --master-port [3306] --master-user --master-password --master-log-directory [/var/lib/mysql] --master-log-pattern [mysql-bin] For more information, see Adding parallel replication in a hurry. Taking over replication from a MySQL slave in direct modeIn the previous recipe, we have shown how to start replication from a MySQL master, without a replicator on the master side. If the slave is already using MySQL native replication with the intended master, you can take over from the native replication stream simply adding to your installation command --native-slave-takeover With this command, Tungsten stops the MySQL slave, gets the binlog file and position from the slave status, and starts replicating from that point. When the replicator goes offline, it will send a "CHANGE MASTER TO" command to the slave to update log file and position, so that you can continue moving data either with Tungsten or with native replication. BEWARE: If you use this option on a MySQL slave that has invalid slave setting or out-of-date slave position it can cause completely inscrutable problems as the replicator may start at the wrong position in the MySQL binlog. (Ask us how we know...) You can correct by turning off slave replication completely or removing the --native-slave-takeover option from the installation. You can also set to false in the static properties file of your replication service after installation. Install Tungsten master/slave replication in a sandboxTungsten Sandbox is a dedicated script that allows you to install multiple dataabse servers and multiple Tungsten Replicator services in a single host.
Please refer to Tungsten-Sandbox instructions for more info. Chain two replication clustersLet's say that you have two replication clusters, one installed using the master slave recipe and one using Tungsten sandbox. You want the master of the Tungsten sandbox in your local host to become a slave of the master at m1.mynetwork.com.
Go to the home directory of the tungsten sandbox master, and run this command: cd $HOME/tdb2/db1 ./tungsten/tools/configure-service -C \ --local-service-name=tsandbox \ --thl-port=12111 \ --role=slave \ --service-type=remote \ --master-thl-host=m1.mynetwork.com \ --master-thl-port=2112 \ --datasource=127_0_0_1 \ --svc-start \ dragon Notes:
After this command, the two clusters are chained. Whatever is inserted into m1.mynetwork.com will also be inserted into the Tungsten sandbox master, and from there it will go to its slaves. Modify one or more properties with the installerTungsten Replicator gets its configuration from a file called static-SERVICE_NAME.properties, located $TUNGSTEN_HOME/tungsten/tungsten-replicator/conf/. This file can be edited with a regular text editor. If you know what you are doing, you can fine tune the replicator to your will. The procedure is not painless. You need to install the replicator first, then edit the file, then eventually restart the replicator. This procedure is difficult to script, and it is especially inconvenient when the changed property needs to be there right when the replicator starts. To your help, there is an option of tungsten-installer, which allows you to change any property in the properties file. For example, let's suppose that you don;t want the replicator to go ONLINE automatically when it starts. This behavior is controlled by a property called replicator.auto_enable, which is true by default. Of course, if you install the replicator with the --start option, you won't get a chance of modifying the property, because the replicator will be already online. To achieve your purpose, you will add this option to the installation command: --property=replicator.auto_enable=false Restrict replication to specific schemas and tablesTo forward or ignore operations on entire schemas or specific tables, use the 'replicate' filter. The filter works from two comma-separated lists of tables and/or schema names. You can use * and ? as wildcards; these match strings and single characters respectively. The 'do' filter list contains tables that should specifically be replicated. The 'ignore' filter contains tables that should be ignored. Here is an example of how to replicate only tables in schema test or tables named foo in any schema. The --property option is quoted to prevent accidental shell wildcard expansion. ./tools/tungsten-installer --master-slave -a \ ... --svc-extractor-filters=replicate \ "--property=replicator.filter.replicate.do=test,*.foo" \ ... --start-and-report The following example drops tables foo and bar in schema tpc: ./tools/tungsten-installer --master-slave -a \ ... --svc-extractor-filters=replicate \ --property=replicator.filter.replicate.ignore=tpc.foo,tpc.bar \ ... --start-and-report Important note: Due Issue 258 the replicate filter currently only works on statements if it is an extract filter. It works in any stage for row updates. Add one slave to an existing masterThe procedure is almost the same used to create a master-slave cluster. You use a similar command, with --master-slave and --master-host as in the full cluster installation command. The difference is that the --cluster-hosts list will only contain the slave host. Tungsten will do the right thing. Start a master service with a given binlog and positionThe easiest way of starting the master service at a given binlog file and position is by using --master-log-file=mysql-bin.000045 --master-log-pos=10292 (Note: there is a problem with this procedure as documented in Issue#216. If the replicator has already been installed and we want to start replicating from a given binlog file and position, we can do this: trepctl -host host_name -service service_name offline
trepctl -host host_name -service service_name online -from-event 000045:10292Notice that we don't give the complete binlog file name, but only the extension. The number after the colon is the position. If you are installing a new copy of the replicator, run tungsten-installer with --auto-enable=false to keep the replicator from going online the first time. After you have put the replication service online, turn on auto-enable by running tools/configure-service -a -U --auto-enable=true service_name. Modify the configuration template file prior to configurationThere are advanced scenarios where you would like to modify the source template files or even add additional templates to the code. One of the simplest is when you need to add a new filter definition. We recommend that you use diff/patch to modify the release package prior to configuration.
echo "diff -Nru tungsten-replicator-2.0.5/tungsten-replicator/samples/conf/filters/default/sample.tpl tungsten-replicator-2.0.5-diff/tungsten-replicator/samples/conf/filters/default/sample.tpl --- tungsten-replicator-2.0.5/tungsten-replicator/samples/conf/filters/default/sample.tpl 1969-12-31 19:00:00.000000000 -0500 +++ tungsten-replicator-2.0.5-diff/tungsten-replicator/samples/conf/filters/default/sample.tpl 2011-09-16 12:13:01.000000000 -0400 @@ -0,0 +1,2 @@ +# Sample filter +replicator.filter.sample=com.continuent.tungsten.replicator.filter.SampleFilter \ No newline at end of file" | patch -p1 -dtungsten-replicator-2.0.5 |
► Sign in to add a comment