I have a MySQL (actually mariaDB) install that I just started replicating to another server. I want to ensure that I’ve setup SSL correctly. The SSL portion of SHOW SLAVE STATUS looks like:

Master_SSL_Allowed: Yes
Master_SSL_CA_File: /etc/mysql/newcerts/ca-cert.pem
Master_SSL_CA_Path: /etc/mysql/newcerts/
Master_SSL_Cert: /etc/mysql/newcerts/client-cert.pem
Master_SSL_Cipher:
Master_SSL_Key: /etc/mysql/newcerts/client-key.pem
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No

Is this typical or show that replication is using SSL? Is there something else I can check to ensure it’s using SSL? The slave is successfully replicating.

Inspect the relevant traffic with tcpdump/wireshark.

Check more discussion of this question.

Our company is using TokuDB on production and we are having a lot of problems trying to mitigate lag on our slave. Is very strange, because we’re talking about very few rows… but with a few data it gets lagged.

Slave is a read-only DB.

For more information, we are using:

CPU: Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz (4 cores)

RAM: 16Gb

HDD: 2Tb ST2000DM001 (EXT4 filesystem)

Here you can see some I/O performance outputs. I paste it outside of this post, because I think that of this way it will easier for reading.

iostat -x 1 output, when we have a lag situation
http://paste.laravel.com/bjv

fio, for disk I/O:
http://paste.laravel.com/bjG

We did a few disk tweaks, extracted from Steven Corona’s book http://www.scalingphpbook.com:

• Changed I/O scheduler to noop.
• Turned off file system access times, noatime and nodiratime in /etc/fstab
• Increased the number of open files, in /etc/security/limits.conf:

* soft  nofile  999999
* hard  nofile  999999

Some tweaks we have made in config:

MASTER

# * Query Cache Configuration
query_cache_limit               = 0
query_cache_size                = 0
query_cache_type                = 0
innodb_file_per_table           = 1innodb_file_format              = barracuda
innodb_flush_method             = O_DIRECT
innodb_flush_log_at_trx_commit  = 1
innodb_log_file_size            = 128Minnodb_buffer_pool_size         = 13500M
innodb_read_io_threads          = 16
innodb_write_io_threads         = 16
innodb_io_capacity              = 180
innodb_thread_concurrency       = 4

SLAVE

# * Query Cache Configuration
query_cache_limit               = 0
query_cache_size                = 0
query_cache_type                = 0
innodb_file_per_table           = 1innodb_file_format              = barracuda
innodb_flush_method             = O_DIRECT
innodb_flush_log_at_trx_commit  = 2
sync_binlog                     = 1innodb_log_file_size            = 128M
innodb_buffer_pool_size         = 13500M
innodb_read_io_threads          = 16
innodb_write_io_threads         = 16innodb_io_capacity              = 180
innodb_thread_concurrency       = 4

I think this is already solved, due to @symcbean’s help.

I disabled barriers, set data=ordered, used async commit and checksums. Finally, we migrated to MariaDB, but still having TokuDB

I forget to mention that we also set tokudb_cache_size = 8G, as TokuDB recommends, 50% of physical memory

Check more discussion of this question.

I am trying to implement a master-master replication between 3 MySQL Databases running on 3 different servers.

I implemented master-master replication for 2 servers, but can’t figure out how can I implement if for 3 different MySQL Servers.

Does anyone know how to accomplish this?

You have to set it up as a ring — A is the master for B, B is the master for C, and C is the master for A. And don’t forget to set the log_slave_updates option so that they will pass on upstream changes, and set replicate-same-server-id to 0 so that the updates won’t go round and round the ring for ever.

Read this article — the ring configuration is described on page 2.

Check more discussion of this question.

I’m trying to setup a mysql replication over an SSH tunnel. The odd thing about this setup is that I have one master with mysql 5.0.60sp1-enterprise-gpl-log and one slave with mysql 5.0.67-community-log. Could it be so that it’s not possible to replicate between community and enterprise edition?

As you can see in my log below, it’s possible to login on the remote server with the mysql client. But the replication get “Can’t connect to MySQL server on ’127.0.0.1′ (13)”

Is it any log file I have forgotten to look in, to get more info?

[root@mysql1-av ~]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 73
Server version: 5.0.67-community-log MySQL Community Edition (GPL)Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

The version of the slave mysql

[root@mysql1-av ~]# autossh -f -M 20001 -L 3307:10.200.200.200:3306 dali@www.ourbounceserver.com -N[root@mysql1-av ~]# mysql -h127.0.0.1 --port 3307 -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 5189
Server version: 5.0.60sp1-enterprise-gpl-log MySQL Enterprise Server (GPL)Type 'help;' or '\h' for help. Type '\c' to clear the buffer.mysql> Aborted

Login to the master mysql with the mysql client over the ssh tunnel.

[root@mysql1-av ~]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 75
Server version: 5.0.67-community-log MySQL Community Edition (GPL)Type 'help;' or '\h' for help. Type '\c' to clear the buffer.mysql> change master to master_host='127.0.0.1', MASTER_PORT=3307, master_user='xxxx', master_password='xxxx', master_log_file='bin.000001';
Query OK, 0 rows affected (0.00 sec)mysql> start slave;
Query OK, 0 rows affected (0.00 sec)mysql> show slave status \G
*************************** 1. row ***************************
             Slave_IO_State: Connecting to master
                Master_Host: 127.0.0.1
                Master_User: replNSG
                Master_Port: 3307
              Connect_Retry: 60
            Master_Log_File: bin.000001
        Read_Master_Log_Pos: 4
             Relay_Log_File: relay.000001
              Relay_Log_Pos: 98
      Relay_Master_Log_File: bin.000001
           Slave_IO_Running: No
          Slave_SQL_Running: Yes
            Replicate_Do_DB: 
        Replicate_Ignore_DB: 
         Replicate_Do_Table: 
     Replicate_Ignore_Table: 
    Replicate_Wild_Do_Table: 
Replicate_Wild_Ignore_Table: 
                 Last_Errno: 0
                 Last_Error: 
               Skip_Counter: 0
        Exec_Master_Log_Pos: 4
            Relay_Log_Space: 98
            Until_Condition: None
             Until_Log_File: 
              Until_Log_Pos: 0
         Master_SSL_Allowed: No
         Master_SSL_CA_File: 
         Master_SSL_CA_Path: 
            Master_SSL_Cert: 
          Master_SSL_Cipher: 
             Master_SSL_Key: 
      Seconds_Behind_Master: NULL
1 row in set (0.00 sec)

Start the replication, but it breaks on IO.

[root@mysql1-av ~]# tail /var/log/mysqld.log
120921 22:17:59 [Note] Slave I/O thread killed while connecting to master
120921 22:17:59 [Note] Slave I/O thread exiting, read up to log 'bin.000001', position 4
120921 22:17:59 [Note] Error reading relay log event: slave SQL thread was killed
120921 22:29:36 [Note] Slave SQL thread initialized, starting replication in log 'bin.000001' at position 4, relay log '/var/lib/mysql/relay.000001' position: 4
120921 22:29:36 [ERROR] Slave I/O thread: error connecting to master 'xxxx@127.0.0.1:3307': Error: 'Can't connect to MySQL server on '127.0.0.1' (13)'  errno: 2003  retry-time: 60  retries: 86400

Because it can’t connect to the master server.

I have now solved the problem. It’s a bug in the mysql replication command “change master” and the argument “MASTER_PORT=3307″. It doesn’t care about that argument, and always uses port 3306.

When using a MySQL 5.0.18 slave, if the master is listening on any port other than 3306, replication fails even when master_port is set to the correct listening port on the master.

http://bugs.mysql.com/bug.php?id=17105

Check more discussion of this question.

Very trick task to find a good title for this question, but here goes the q:

I have a few development machines, where I develop my PHP applications on, and testing via a local webserver.

This works out pretty well for each machine. However, I would like to replicate the DB from my machines to a central location.

So, to sum up:

DEV1 -> CENTRAL
DEV2 -> CENTRAL
DEV3 -> CENTRAL
CENTRAL -> DEV1
CENTRAL -> DEV2
CENTRAL -> DEV3

I hope this makes sense, as I cannot find an easy way to tell it.

Basically, it is a 2-way replication, where all 4 databases contain the same info, and each of them can be updated locally, to then be pushed out to the others.

Is this actually doable?

All my dev machines are running Windows 7, and my central DB server is running CentOS 6.

You could go with Percona XtraDB Cluster

I am currently evaluating it and I think it is brilliantly designed for MultiMaster writes. It can use mysqldump (default), rsync, and xtrabackup (preferred) for initializing new Cluster node. You have total freedom and power. This may be the greatest cliche of all time but WITH GREAT POWER, THEIR MUST ALSO ALWAYS BE GREAT RESPONSIBILITY (19:16 – 19:25 of the Video).

I discussed this as one of many options for cross-datacenter replication in the DBA StackExchange : http://dba.stackexchange.com/questions/22788/best-solution-for-cross-datacenter-mysql-master-slave-replication/22790#22790

Check more discussion of this question.

I am trying to add a node to a new cluster but am met with the following log entries:

120915 16:47:32 [Warning] WSREP: error executing 'SET GLOBAL innodb_disallow_writes=1': 1193 (Unknown system variable 'innodb_disallow_writes'). Was mysqld built with --with-innodb-disallow-writes ?
120915 16:47:32 [ERROR] WSREP: Failed to disallow InnoDB writes
    rsync: open "mysql/dummy.bak" (in rsync_sst) failed: Permission denied (13)
    rsync: open "test/dummy.bak" (in rsync_sst) failed: Permission denied (13)
    rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.9]
    rsync returned code 23:
120915 16:47:45 [ERROR] WSREP: Failed to read from: wsrep_sst_rsync 'donor' '10.194.110.201:4444/rsync_sst' 'wsrep_sst:wspass' '/usr/local/mysql-5.5/data/' '/etc/my.cnf' 'b334d89c-ff6c-11e1-0800-ba5125ffffff' '2' '0'
120915 16:47:45 [Warning] WSREP: error executing 'SET GLOBAL innodb_disallow_writes=0': 1193 (Unknown system variable 'innodb_disallow_writes'). Was mysqld built with --with-innodb-disallow-writes ?
120915 16:47:45 [ERROR] WSREP: Failed to disallow InnoDB writes
120915 16:47:45 [Note] WSREP: Provider resumed.
120915 16:47:45 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync 'donor' '10.1.1.2:4444/rsync_sst' 'wsrep_sst:wspass' '/usr/local/mysql-5.5/data/' '/etc/my.cnf' 'b334d89c-ff6c-11e1-0800-ba5125ffffff' '2' '0': 255 (Unknown error 255)
120915 16:47:45 [Warning] WSREP: 0 (master): State transfer to 1 (slave1) failed: -1 (Operation not permitted)

My node1 (gcomm://) starts and shows 1 node in cluster. I updated node2 to point to node1, restart, and the following appears consistently, causing MySQL to crash.

It looks like part of the issue may be related to rsync failing allong with Unknown error 255.

My node1 is 10.1.1.1 and listening on 3306 and 4567 (by default).

Update: I changed SST type to mysqldump and it started working. But since then, performance has been very poor:

mysql> show databases;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)
ERROR:
Can't connect to the servermysql> show databases;
No connection. Trying to reconnect...
Connection id:    1
Current database: *** NONE ***ERROR 1317 (70100): Query execution was interrupted
mysql> show databases;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id:    7
Current database: *** NONE ***+--------------------+
| Database           |
+--------------------+
| information_schema |
| Text               |
| mysql              |
| performance_schema |
| test               |
+--------------------+
5 rows in set (0.00 sec)mysql>

Also, this is located in .err

19:41:41 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/key_buffer_size=33554432
read_buffer_size=2097152
max_used_connections=0
max_threads=100
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1058026 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.120919 15:42:33 mysqld_safe Starting mysqld daemon with databases from /usr/local/mysql-5.5/data
120919 15:42:33 [Note] Flashcache bypass: disabled
120919 15:42:33 [Note] Flashcache setup error is : ioctl failed120919 15:42:33 InnoDB: The InnoDB memory heap is disabled
120919 15:42:33 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120919 15:42:33 InnoDB: Compressed tables use zlib 1.2.3.4
120919 15:42:33 InnoDB: Using Linux native AIO
120919 15:42:33 InnoDB: Initializing buffer pool, size = 2.0G
120919 15:42:34 InnoDB: Completed initialization of buffer pool
120919 15:42:34 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
120919 15:42:34  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
120919 15:42:34  InnoDB: Assertion failure in thread 140700858599232 in file fsp0fsp.c line 2938
InnoDB: Failing assertion: space != 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/local/mysql-5.5/bin/mysqld(my_print_stacktrace+0x29)[0x786eb9]
/usr/local/mysql-5.5/bin/mysqld(handle_fatal_signal+0x367)[0x678597]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f97d479ecb0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f97d3e0e445]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f97d3e11bab]
/usr/local/mysql-5.5/bin/mysqld[0x8b5f02]
/usr/local/mysql-5.5/bin/mysqld[0x8b63b5]
/usr/local/mysql-5.5/bin/mysqld[0x84db6f]
/usr/local/mysql-5.5/bin/mysqld[0x884f39]
/usr/local/mysql-5.5/bin/mysqld[0x8857bf]
/usr/local/mysql-5.5/bin/mysqld[0x82a9d5]
/usr/local/mysql-5.5/bin/mysqld[0x7ed3b9]
/usr/local/mysql-5.5/bin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x41)[0x67a721]
/usr/local/mysql-5.5/bin/mysqld[0x593c35]
/usr/local/mysql-5.5/bin/mysqld(_Z11plugin_initPiPPci+0x829)[0x597b69]
/usr/local/mysql-5.5/bin/mysqld[0x51a8d7]
/usr/local/mysql-5.5/bin/mysqld(_Z11mysqld_mainiPPc+0x953)[0x51f1a3]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f97d3df976d]
/usr/local/mysql-5.5/bin/mysqld[0x515e45]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.

120915 16:47:32 [Warning] WSREP: error executing 'SET GLOBAL
innodb_disallow_writes=1': 1193 (Unknown system variable
'innodb_disallow_writes'). Was mysqld built with
--with-innodb-disallow-writes ?

120915 16:47:32 [ERROR] WSREP: Failed to disallow InnoDB writes

Although the mysqlbug is obsolete in 5.5, it still can be used to get the compile options. Make sure that you see -DWITH_INNODB_DISALLOW_WRITES in the Compilation info (used): line:

Compilation info (used): CC='/usr/bin/gcc44' CFLAGS='-fPIC -Wall -O3 -g -static-libgcc -fno-omit-frame-pointer -DPERCONA_INNODB_VERSION=rel28.1 -DWITH_WSREP -DWSREP_PROC_INFO -DMYSQL_MAX_VARIABLE_VALUE_LEN=2048 -DWITH_INNODB_DISALLOW_WRITES...

rsync: open "mysql/dummy.bak" (in rsync_sst) failed: Permission denied (13)
rsync: open "test/dummy.bak" (in rsync_sst) failed: Permission denied (13)

What are permissions of these files?

Update: I changed SST type to mysqldump and it started working.

Quite strange. Don’t you get Unknown system variable 'innodb_disallow_writes'?

Also, this is located in .err

19:41:41 UTC - mysqld got signal 6 ;
This could be because you hit a bug...

Could you please post some lines above this? Have you tried to use xtrabackup method?

Check more discussion of this question.

I’m having a master/slave mysql setup. This night I had some problems with the master. I needed to recreate all databases again because they were corrupt.

Now I stopped my mysql slave but want to start it again and rebuild everything from scratch. Is there a way to do this? Without locking the master…

I also think the binlog file should be reset. So everything is clean again.

Any ideas?

if you use innodb engine on the master – just use mysqldump with –skip-lock-tables, –single-transaction and –master-data=2 switches.

if your slave is only logically corrupted – try pt-table-sync with lock=0 option

Check more discussion of this question.

I’ve created a slave server from live machine which is acting as a master now.
I use the following procedure to create it:

mysqldump --opt -Q -B --master-data=2 --all-databases > dump.sql

• then I imported this dump on the new machine, applied the “CHANGE MASTER TO…” directive with a log file/position from the dump.

Please note that I have around 8000 databases and I didn’t stop the master while the dumps were running.

The replication works fine but is this a properly method for creating a slave server? I’m planning to promote this slave to a master (different location) so I would like to make sure that there is a 100% data consistency between the servers.

I’ve found this article where it says:

The naive approach is just to use mysqldump to export a copy of the
master and load it on the slave server. This works if you only have
one database. With multiple database, you’ll end up with inconsistent
data. Mysqldump will dump data from each database on the server in a
different transaction. That means that your export will have data from
a different point in time for each database.

Thank you

The method you’re using (specifically the --master-data=2 option) should be enough to get you a consistent dump, since --master-data implies --lock-all-tables (and the resulting offences against database performance that are implied). The caveats specified in the mysqldump(1) manpage for the --lock-all-tables option should be observed.

I would suggest that the article you cite was either written prior to the availability of --lock-all-tables (or it’s implicit activation with --master-data) or was written by someone incapable or unwilling to read documentation. You may wish to query the author to determine which is the case.

Check more discussion of this question.

It appears overnight our Master RDS mysql server went offline for 3-4 minutes. When it came back online the replication was broken.

On the slave, issuing SHOW SLAVE STATUS\G;

“Got fatal error 1236 from master when reading data from binary log: ‘Client requested master to start replication from impossible position’”

In AWS monitoring, slave lag shows “-1.0″.

Is there a way to get the slave replication restarted, or should I just create a new slave?

There’s really no harm in just spooling up a new slave, so I’d go that route.

Check more discussion of this question.

I am running with one master and two slaves in a chain. ie

Master-->Primary Slave-->Secondary Slave. 

The relay log of primary slave got corrupted.

How can I recreate the relay log of primary slave without making any changes to the Primary Slave --> Secondary Slave replication?

If it’s the relay log (rather than the master bin log) which got corrupted, then

1. STOP SLAVE (on both slaves)
2. SHOW SLAVE STATUS on the failing machine to get the current log file and position
3. RESET SLAVE on the faling machine
4. CHANGE MASTER TO MASTER_LOG_FILE=’filename from step 2′, MASTER_LOG_POS=’position from step 2′
5. START SLAVE on both slaves

Check more discussion of this question.