Today

• moments ago

  r5437 (Remove last packetno handling. It's not reliable.) committed by dan...@percona.com   -   Remove last packetno handling. It's not reliable.

  Remove last packetno handling. It's not reliable.
• 31 min ago

  r5436 (Handle session creation and deleting better. Catch out of o...) committed by dan...@percona.com   -   Handle session creation and deleting better. Catch out of order mysql packetnos. First 5 bytes of data. (issue 794).

  Handle session creation and deleting better. Catch out of order mysql packetnos. First 5 bytes of data. (issue 794).
• 3 hours ago

  issue 794 (MySQLProtocolParser does not handle client port reuse) commented on by dan...@percona.com   -   A big sample file from the real world shows me that we should handle session creation a little more strictly, else we are tricked by missing packets which become more and more common the large the continue input collected. Right now we create a session simply if it does not exist. This causes a fews problems. E.g., when a session is deleted, it gets re-created when the code hits the TCP FIN packets; I have verified this. And when client port numbers get reused, this causes more problems because there's bleed-over from the previous session (somewhat fixed in r5432). Closing or reseting a session needs better definition, again for cases of port re-use. I've seen where session X disappears and then reappears for a new connection. The code should see the new connection and say, "Oh, the first session didn't terminate properly (we didn't see a COM_QUIT) so let's reset it." I think session creation and/or reset should only happen if either: 1) we catch TCP SYN (full start of new connection) 2) packet from server and packet id==0 (should only happen for server handshake) 3) packet from client and packet id==0 (should only happen for start of new client request) Then we close on COM_QUIT (like we currently do) and the above should keep us from either starting a new session due to the TCP FIN and/or make us restart the session if the port gets reused and somehow the session was still open.

  A big sample file from the real world shows me that we should handle session creation a little more strictly, else we are tricked by missing packets which become more and more common the large the continue input collected. Right now we create a session simply if it does not exist. This causes a fews problems. E.g., when a session is deleted, it gets re-created when the code hits the TCP FIN packets; I have verified this. And when client port numbers get reused, this causes more problems because there's bleed-over from the previous session (somewhat fixed in r5432). Closing or reseting a session needs better definition, again for cases of port re-use. I've seen where session X disappears and then reappears for a new connection. The code should see the new connection and say, "Oh, the first session didn't terminate properly (we didn't see a COM_QUIT) so let's reset it." I think session creation and/or reset should only happen if either: 1) we catch TCP SYN (full start of new connection) 2) packet from server and packet id==0 (should only happen for server handshake) 3) packet from client and packet id==0 (should only happen for start of new client request) Then we close on COM_QUIT (like we currently do) and the above should keep us from either starting a new session due to the TCP FIN and/or make us restart the session if the port gets reused and somehow the session was still open.
• 4 hours ago

  mk_schema_audit (Roadmap and vision for mk-schema-audit) Wiki page edited by baron.schwartz   -   Revision r5435 Edited wiki page through web user interface.

  Revision r5435 Edited wiki page through web user interface.
• 5 hours ago

  issue 795 (mk-query-digest ignores --ask-pass) reported by hobodayve   -   Ex 1: ~: mk-query-digest --ask-pass --processlist localhost DBI connect(';host=localhost;mysql_read_default_group=client','',...) failed: Access denied for user 'dabdemoulaie'@'localhost' (using password: NO) at /usr/bin/mk-query-digest line 261 DBI connect(';host=localhost;mysql_read_default_group=client','',...) failed: Access denied for user 'dabdemoulaie'@'localhost' (using password: NO) at /usr/bin/mk-query-digest line 261 ... Ex 2: ~: mk-query-digest --create-review-history-table --ask-pass --review h=192.168.0.74,D=test,t=query_review /var/log/mysql/slow.log DBI connect('test;host=192.168.0.74;mysql_read_default_group=client','',...) failed: Access denied for user 'dabdemoulaie'@'192.168.0.74' (using password: NO) at /usr/bin/mk-query- digest line 261 Versions: mk-query-digest Ver 0.9.13 Distrib 5427 Changeset 5425 /usr/sbin/mysqld Ver 5.1.36-log for pc-linux-gnu on i686 (Gentoo Linux mysql-community- 5.1.36) This is perl, v5.8.8 built for i686-linux DBI: 1.601 DBD::mysql: 4.005 Linux 2.6.29-gentoo-r5 #4 SMP PREEMPT Wed Jul 15 22:50:18 PDT 2009 i686 Intel(R) Xeon(TM) CPU 2.80GHz GenuineIntel GNU/Linux

  Ex 1: ~: mk-query-digest --ask-pass --processlist localhost DBI connect(';host=localhost;mysql_read_default_group=client','',...) failed: Access denied for user 'dabdemoulaie'@'localhost' (using password: NO) at /usr/bin/mk-query-digest line 261 DBI connect(';host=localhost;mysql_read_default_group=client','',...) failed: Access denied for user 'dabdemoulaie'@'localhost' (using password: NO) at /usr/bin/mk-query-digest line 261 ... Ex 2: ~: mk-query-digest --create-review-history-table --ask-pass --review h=192.168.0.74,D=test,t=query_review /var/log/mysql/slow.log DBI connect('test;host=192.168.0.74;mysql_read_default_group=client','',...) failed: Access denied for user 'dabdemoulaie'@'192.168.0.74' (using password: NO) at /usr/bin/mk-query- digest line 261 Versions: mk-query-digest Ver 0.9.13 Distrib 5427 Changeset 5425 /usr/sbin/mysqld Ver 5.1.36-log for pc-linux-gnu on i686 (Gentoo Linux mysql-community- 5.1.36) This is perl, v5.8.8 built for i686-linux DBI: 1.601 DBD::mysql: 4.005 Linux 2.6.29-gentoo-r5 #4 SMP PREEMPT Wed Jul 15 22:50:18 PDT 2009 i686 Intel(R) Xeon(TM) CPU 2.80GHz GenuineIntel GNU/Linux
• 7 hours ago

  issue 781 (mk-query-digest doesn't distill or extract tables properly) Status changed by baron.schwartz   -   Hi Istvan, I think it needs to handle more complex things, like FROM (tbl1 JOIN tbl2) and UPDATE (tbl1 JOIN tbl2)... and INSERT INTO (tbl1) SELECT .... I am not sure how to do all of these different cases, but I'm thinking that it might be a good idea to delete all matched pairs of ( ) from the SQL and let the current code take care of the rest. The only thing is, you can't delete a matched ( ) pair that is around a subselect, like this, select * from ( select * from tbl1) as x; otherwise you'll get select * from select * from tbl1 as x; and the tables/distill that come out of this will be SELECT SELECT tbl1. So don't remove the '(' or the matching ')' when the '(' is followed by SELECT. Also, there is another function called get_tables() or something, which needs to be able to handle the same cases.

  Status: Started

  Hi Istvan, I think it needs to handle more complex things, like FROM (tbl1 JOIN tbl2) and UPDATE (tbl1 JOIN tbl2)... and INSERT INTO (tbl1) SELECT .... I am not sure how to do all of these different cases, but I'm thinking that it might be a good idea to delete all matched pairs of ( ) from the SQL and let the current code take care of the rest. The only thing is, you can't delete a matched ( ) pair that is around a subselect, like this, select * from ( select * from tbl1) as x; otherwise you'll get select * from select * from tbl1 as x; and the tables/distill that come out of this will be SELECT SELECT tbl1. So don't remove the '(' or the matching ')' when the '(' is followed by SELECT. Also, there is another function called get_tables() or something, which needs to be able to handle the same cases.

  Status: Started
• 13 hours ago

  issue 781 (mk-query-digest doesn't distill or extract tables properly) Status changed by istvan.podor   -   Anyhow its done, please give me some response to see if its a good patch or not :)

  Status: Fixed

  Anyhow its done, please give me some response to see if its a good patch or not :)

  Status: Fixed
• 13 hours ago

  r5434 (issue #781 patch) committed by istvan.podor   -   issue #781 patch

  issue #781 patch
• 14 hours ago

  issue 781 (mk-query-digest doesn't distill or extract tables properly) commented on by istvan.podor   -   Baron you were right, its about the ( ) . To fix it, I implement a $query rewrite when the string is like 'FROM (`TBL`)' its rewrite without () . What do you think?

  Baron you were right, its about the ( ) . To fix it, I implement a $query rewrite when the string is like 'FROM (`TBL`)' its rewrite without () . What do you think?
• 22 hours ago

  r5433 (Update MySQLProtocolParser in mqd.) committed by dan...@percona.com   -   Update MySQLProtocolParser in mqd.

  Update MySQLProtocolParser in mqd.
• 22 hours ago

  issue 794 (MySQLProtocolParser does not handle client port reuse) commented on by dan...@percona.com   -   MySQLProtocolParser fixed in r5432.

  MySQLProtocolParser fixed in r5432.
• 22 hours ago

  r5432 (Handle client port reuse (issue 794).) committed by dan...@percona.com   -   Handle client port reuse (issue 794).

  Handle client port reuse (issue 794).
• 23 hours ago

  issue 794 (MySQLProtocolParser does not handle client port reuse) reported by dan...@percona.com   -   [15:48] <Daniel> the bad event, the admin;connect w/ time Query_time => '131.677970', [15:48] <Daniel> i can repro it: [15:48] <Daniel> perl -MMySQLProtocolParser [15:48] <Daniel> print MySQLProtocolParser::timestamp_diff('2010-01-06 15:14:46.546334', '2010-01-06 15:16:58.234623'); [15:48] <Daniel> 131.688289 [15:49] <Daniel> kind of like we suspected yesterday... [15:49] <Daniel> there's an event "bleed-over" [15:49] <Daniel> that first time is from that sessions first connect, and the second time is from the sessions 2nd connect some 113s later [15:50] <Daniel> so there's two things going on here: [15:50] <Daniel> 1) the server reuses ports, in particular.. [15:50] <Daniel> 1.2.3.4:34777 in this case [15:50] <Daniel> so 1.2.3.4:34777 connects, does some crap, then disconnects... [15:51] <Daniel> 113s later, the very same ip:port (1.2.3.4:34777) re-connects [15:51] <Daniel> now, the code is supposed to handle this but... [15:51] <Daniel> the change in http://code.google.com/p/maatkit/source/diff?path=/trunk/common/MySQLProtocolParser.pm&format=side&r=4134 where the session set gets undef'ed in _make_event() [15:52] <Daniel> coupled with very poor handling of the first timestamp of an event in the code is causing this bug, and probably others.

  [15:48] <Daniel> the bad event, the admin;connect w/ time Query_time => '131.677970', [15:48] <Daniel> i can repro it: [15:48] <Daniel> perl -MMySQLProtocolParser [15:48] <Daniel> print MySQLProtocolParser::timestamp_diff('2010-01-06 15:14:46.546334', '2010-01-06 15:16:58.234623'); [15:48] <Daniel> 131.688289 [15:49] <Daniel> kind of like we suspected yesterday... [15:49] <Daniel> there's an event "bleed-over" [15:49] <Daniel> that first time is from that sessions first connect, and the second time is from the sessions 2nd connect some 113s later [15:50] <Daniel> so there's two things going on here: [15:50] <Daniel> 1) the server reuses ports, in particular.. [15:50] <Daniel> 1.2.3.4:34777 in this case [15:50] <Daniel> so 1.2.3.4:34777 connects, does some crap, then disconnects... [15:51] <Daniel> 113s later, the very same ip:port (1.2.3.4:34777) re-connects [15:51] <Daniel> now, the code is supposed to handle this but... [15:51] <Daniel> the change in http://code.google.com/p/maatkit/source/diff?path=/trunk/common/MySQLProtocolParser.pm&format=side&r=4134 where the session set gets undef'ed in _make_event() [15:52] <Daniel> coupled with very poor handling of the first timestamp of an event in the code is causing this bug, and probably others.

Yesterday

• 26 hours ago

  r5431 (Add mk-test-env reset and don't start slaves at log 1, pos 0...) committed by dan...@percona.com   -   Add mk-test-env reset and don't start slaves at log 1, pos 0 (don't need to replay old stuff).

  Add mk-test-env reset and don't start slaves at log 1, pos 0 (don't need to replay old stuff).
• 26 hours ago

  r5430 (Modularize mk-table-sync.t, path and order independent.) committed by dan...@percona.com   -   Modularize mk-table-sync.t, path and order independent.

  Modularize mk-table-sync.t, path and order independent.
• 41 hours ago

  issue 793 (mk-loadavg doesn't count processlist statuses right) reported by baron.schwartz   -   I ran mk-loadavg for a customer, like this: mk-loadavg --watch "Processlist:state:statistics:count:>:1" --interval 10 --execute monitor.sh It never triggered that script, but the server was freaking out, full of queries in statistics state. I don't think this is working right. We need a test case.

  I ran mk-loadavg for a customer, like this: mk-loadavg --watch "Processlist:state:statistics:count:>:1" --interval 10 --execute monitor.sh It never triggered that script, but the server was freaking out, full of queries in statistics state. I don't think this is working right. We need a test case.
• 42 hours ago

  issue 792 (Maatkit utilities (mk-slave-move in particular) needs quanti...) reported by da...@davidfreeman.net   -   If this is a problem report, what steps will reproduce the problem? 1. Experience an interest in seeing working examples of an mk-slave-move DSN, as described in the documentation --or, rather, a functional command string in general. 2. Use either 'perldoc| man | $utility_name --help' to find an example. 3. What is the expected output? What do you see instead? The absence of three-to-five rich illustrations, which might in every instance make Maatkit utilities more quickly and widely adopted. What information do you get from mk-<toolname>--version? What is your MySQL, Perl, DBI, and DBD::mysql version? What is your operating system/distribution? Maatkit: mk-slave-move Ver 0.9.9 Distrib 5240 Changeset 5237; Centos 5; mysql 5.0.22, 5.0.77 or 5.1.42 Please provide any additional information below. I believe that Maatkit utilities will see a jump in deployment when such examples are generously provided by contributors to the project. This ticket submitted at Baron's behest from 01042009 in #maatkit on Freenode.

  If this is a problem report, what steps will reproduce the problem? 1. Experience an interest in seeing working examples of an mk-slave-move DSN, as described in the documentation --or, rather, a functional command string in general. 2. Use either 'perldoc| man | $utility_name --help' to find an example. 3. What is the expected output? What do you see instead? The absence of three-to-five rich illustrations, which might in every instance make Maatkit utilities more quickly and widely adopted. What information do you get from mk-<toolname>--version? What is your MySQL, Perl, DBI, and DBD::mysql version? What is your operating system/distribution? Maatkit: mk-slave-move Ver 0.9.9 Distrib 5240 Changeset 5237; Centos 5; mysql 5.0.22, 5.0.77 or 5.1.42 Please provide any additional information below. I believe that Maatkit utilities will see a jump in deployment when such examples are generously provided by contributors to the project. This ticket submitted at Baron's behest from 01042009 in #maatkit on Freenode.
• 46 hours ago

  issue 786 (Create mk-error-log) commented on by baron.schwartz   -   Related: http://blog.endpoint.com/2010/01/monitoring-postgres-log-files-with.html Related: http://www.logcheck.org/

  Related: http://blog.endpoint.com/2010/01/monitoring-postgres-log-files-with.html Related: http://www.logcheck.org/
• 46 hours ago

  issue 791 (Add more mysql protocol flags to event attributes for mk-que...) reported by baron.schwartz   -   From an IRC chat, <Baron> Daniel, can you check the mysql protocol and see if there is a this_connection_has_a_txn_open bit? <Daniel> Baron: maybe #define SERVER_STATUS_IN_TRANS 1 /* Transaction has started */ <Baron> Daniel, that's it. Great. Can you add that as an event attribute? Please also add SERVER_STATUS_AUTOCOMMIT These should go into $event->{In_trans} and $event->{Is_autocommit} as booleans. This will help us a lot with things like debugging deadlocks.

  From an IRC chat, <Baron> Daniel, can you check the mysql protocol and see if there is a this_connection_has_a_txn_open bit? <Daniel> Baron: maybe #define SERVER_STATUS_IN_TRANS 1 /* Transaction has started */ <Baron> Daniel, that's it. Great. Can you add that as an event attribute? Please also add SERVER_STATUS_AUTOCOMMIT These should go into $event->{In_trans} and $event->{Is_autocommit} as booleans. This will help us a lot with things like debugging deadlocks.

Last 7 days

• Jan 06, 2010

  r5429 (Make latest-changelog work until next century.) committed by dan...@percona.com   -   Make latest-changelog work until next century.

  Make latest-changelog work until next century.
• Jan 06, 2010

  maatkit-5427-1.noarch.rpm (.rpm release 5427) file uploaded by dan...@percona.com   -  

  Labels: Featured

  Labels: Featured
• Jan 06, 2010

  maatkit_5427-1_all.deb (.deb release 5427) file uploaded by dan...@percona.com   -  

  Labels: Featured

  Labels: Featured
• Jan 06, 2010

  maatkit-5427.zip (zip release 5427) file uploaded by dan...@percona.com   -  

  Labels: Featured

  Labels: Featured
• Jan 06, 2010

  maatkit-5427.tar.gz (tarball release 5427) file uploaded by dan...@percona.com   -  

  Labels: Featured

  Labels: Featured
• Jan 06, 2010

  r5428 (Changelog) committed by dan...@percona.com   -   Changelog

  Changelog
• Jan 06, 2010

  r5427 (Bump version) committed by dan...@percona.com   -   Bump version

  Bump version
• Jan 06, 2010

  r5426 (bump version) committed by dan...@percona.com   -   bump version

  bump version
• Jan 06, 2010

  r5425 (update RISKS) committed by baron.schwartz   -   update RISKS

  update RISKS
• Jan 06, 2010

  r5424 (Make Changelogs proper.) committed by dan...@percona.com   -   Make Changelogs proper.

  Make Changelogs proper.
• Jan 06, 2010

  r5423 (Update RISKS.) committed by dan...@percona.com   -   Update RISKS.

  Update RISKS.
• Jan 06, 2010

  r5422 (Update common modules in all scripts.) committed by dan...@percona.com   -   Update common modules in all scripts.

  Update common modules in all scripts.
• Jan 06, 2010

  issue 382 (Some tests are very flaky) changed by dan...@percona.com   -   Bleh, mk-table-sync is too. One thing I'm going to do after the tests are modularized is run them N times and see how many times they pass/fail. I'm hoping for 100% pass, but at the moment mk-table-sync is very temperamental on my machine. :-( Summary was "mk-table-checksum is very flaky".

  Summary: Some tests are very flaky

  Status: Started

  Owner: dan...@percona.com

  Labels: Priority-High Tool-all −Priority-Medium −Tool-mk_table_checksum

  Bleh, mk-table-sync is too. One thing I'm going to do after the tests are modularized is run them N times and see how many times they pass/fail. I'm hoping for 100% pass, but at the moment mk-table-sync is very temperamental on my machine. :-( Summary was "mk-table-checksum is very flaky".

  Summary: Some tests are very flaky

  Status: Started

  Owner: dan...@percona.com

  Labels: Priority-High Tool-all −Priority-Medium −Tool-mk_table_checksum
• Jan 05, 2010

  r5421 (Alphabetize pod opts.) committed by dan...@percona.com   -   Alphabetize pod opts.

  Alphabetize pod opts.
• Jan 05, 2010

  r5420 (Update check-module-usage for new MaatkitTest::load_file.) committed by dan...@percona.com   -   Update check-module-usage for new MaatkitTest::load_file.

  Update check-module-usage for new MaatkitTest::load_file.
• Jan 05, 2010

  r5419 (mk-upgrade use new parse_event interface. Make semi-path-in...) committed by dan...@percona.com   -   mk-upgrade use new parse_event interface. Make semi-path-independent.

  mk-upgrade use new parse_event interface. Make semi-path-independent.
• Jan 05, 2010

  r5418 (mk-table-sync semi-path-independent.) committed by dan...@percona.com   -   mk-table-sync semi-path-independent.

  mk-table-sync semi-path-independent.
• Jan 05, 2010

  issue 790 (mk-table-sync --lock-and-rename doesn't work on 5.1) reported by dan...@percona.com   -   The mk-table-sync --lock-and-rename test causes strange behavior on 5.1. After it runs, doing DROP DATABASE IF EXITS db1 on the master (which a subsequent test does), the slave dies with the error: Last_SQL_Errno: 1192 Last_SQL_Error: Error 'Can't execute the given command because you have active locked tables or an active transaction' on query. Default database: 'db1'. Query: 'DROP DATABASE IF EXISTS db1' This doesn't happen with 5.0.

  The mk-table-sync --lock-and-rename test causes strange behavior on 5.1. After it runs, doing DROP DATABASE IF EXITS db1 on the master (which a subsequent test does), the slave dies with the error: Last_SQL_Errno: 1192 Last_SQL_Error: Error 'Can't execute the given command because you have active locked tables or an active transaction' on query. Default database: 'db1'. Query: 'DROP DATABASE IF EXISTS db1' This doesn't happen with 5.0.
• Jan 05, 2010

  r5417 (mk-table-checksum semi-path-independent.) committed by dan...@percona.com   -   mk-table-checksum semi-path-independent.

  mk-table-checksum semi-path-independent.
• Jan 05, 2010

  r5416 (mk-slave-prefetch use new parse_event interface. Make tests...) committed by dan...@percona.com   -   mk-slave-prefetch use new parse_event interface. Make tests semi-path-independent.

  mk-slave-prefetch use new parse_event interface. Make tests semi-path-independent.
• Jan 05, 2010

  r5415 (mk-query-digest.t semi-path-independent.) committed by dan...@percona.com   -   mk-query-digest.t semi-path-independent.

  mk-query-digest.t semi-path-independent.
• Jan 05, 2010

  r5414 (mk-parallel-restore semi-path-indepent.) committed by dan...@percona.com   -   mk-parallel-restore semi-path-indepent.

  mk-parallel-restore semi-path-indepent.
• Jan 05, 2010

  r5413 (mk-parallel-dump semi-path-indepent.) committed by dan...@percona.com   -   mk-parallel-dump semi-path-indepent.

  mk-parallel-dump semi-path-indepent.
• Jan 05, 2010

  r5412 (mk-log-player.t semi-path-independent.) committed by dan...@percona.com   -   mk-log-player.t semi-path-independent.

  mk-log-player.t semi-path-independent.
• Jan 05, 2010

  r5411 (mk-fk-error-log semi-path-independent.) committed by dan...@percona.com   -   mk-fk-error-log semi-path-independent.

  mk-fk-error-log semi-path-independent.
• Jan 05, 2010

  r5410 (mk-duplicate-key-checker.t path-independent.) committed by dan...@percona.com   -   mk-duplicate-key-checker.t path-independent.

  mk-duplicate-key-checker.t path-independent.
• Jan 05, 2010

  r5409 (Get idb status for 5.0 or 5.1.) committed by dan...@percona.com   -   Get idb status for 5.0 or 5.1.

  Get idb status for 5.0 or 5.1.
• Jan 05, 2010

  issue 789 (SHOW INNODB STATUS changed from 5.0 to 5.1) commented on by dan...@percona.com   -   Aye, getting hahsref->{Status} is better, like: --- mk-deadlock-logger.t (revision 5406) +++ mk-deadlock-logger.t (working copy) -my ($stat) = $dbh1->selectrow_array('show innodb status'); +my $stat = $dbh1->selectrow_hashref('show innodb status')->{Status};

  Aye, getting hahsref->{Status} is better, like: --- mk-deadlock-logger.t (revision 5406) +++ mk-deadlock-logger.t (working copy) -my ($stat) = $dbh1->selectrow_array('show innodb status'); +my $stat = $dbh1->selectrow_hashref('show innodb status')->{Status};
• Jan 05, 2010

  r5408 (mk-archiver.t path-independent.) committed by dan...@percona.com   -   mk-archiver.t path-independent.

  mk-archiver.t path-independent.
• Jan 05, 2010

  ReleaseInstructions (How to create a new Maatkit release) Wiki page edited by dan...@percona.com   -   Revision r5407 Edited wiki page through web user interface.

  Revision r5407 Edited wiki page through web user interface.
• Jan 05, 2010

  r5406 (Make Daemon.t path-independent. Use system() instead of to...) committed by dan...@percona.com   -   Make Daemon.t path-independent. Use system() instead of to daemonize.

  Make Daemon.t path-independent. Use system() instead of to daemonize.