ppss - Project Hosting on Google Code

Last 30 days Dec 17, 2009 r354 (backup) committed by Louwrentius - backup backup Dec 17, 2009 r353 (Reworked the ppss-test script to work with the new ppss exec...) committed by Louwrentius - Reworked the ppss-test script to work with the new ppss executable and changed the default workig dir of ppss Reworked the ppss-test script to work with the new ppss executable and changed the default workig dir of ppss Dec 17, 2009 r352 (Renamed ppss.sh to ppss, to make it more like a regular unix...) committed by Louwrentius - Renamed ppss.sh to ppss, to make it more like a regular unix program. Renamed ppss.sh to ppss, to make it more like a regular unix program. Dec 17, 2009 DistributedPPSS (How distributed PPSS works) Wiki page edited by Louwrentius - Revision r351 Edited wiki page through web user interface. Revision r351 Edited wiki page through web user interface. Dec 17, 2009 DistributedPPSS (How distributed PPSS works) Wiki page edited by Louwrentius - Revision r350 Edited wiki page through web user interface. Revision r350 Edited wiki page through web user interface. Dec 17, 2009 DistributedPPSS (How distributed PPSS works) Wiki page edited by Louwrentius - Revision r349 Edited wiki page through web user interface. Revision r349 Edited wiki page through web user interface. Dec 17, 2009 Changelog (Changelog of distributed PPSS) Wiki page edited by Louwrentius - Revision r348 Edited wiki page through web user interface. Revision r348 Edited wiki page through web user interface. Dec 17, 2009 Changelog (Changelog of distributed PPSS) Wiki page edited by Louwrentius - Revision r347 Edited wiki page through web user interface. Revision r347 Edited wiki page through web user interface. Dec 17, 2009 ppss-2.50.tgz (Distributed Parallell Processing Shell Script - Current docu...) file uploaded by Louwrentius - Labels: Featured Type-Archive Labels: Featured Type-Archive Dec 17, 2009 r346 (Fixed issue with master SSH connection on node not being set...) committed by Louwrentius - Fixed issue with master SSH connection on node not being setup. Fixed issue with master SSH connection on node not being setup. Dec 17, 2009 Changelog (Changelog of distributed PPSS) Wiki page edited by Louwrentius - Revision r345 Edited wiki page through web user interface. Revision r345 Edited wiki page through web user interface. Dec 17, 2009 r344 (Some 'gui' bugfixes.) committed by Louwrentius - Some 'gui' bugfixes. Some 'gui' bugfixes. Dec 16, 2009 r343 (workdir must be relative to current path.) committed by Louwrentius - workdir must be relative to current path. workdir must be relative to current path. Dec 16, 2009 r342 (Reworked ppss regarding process management..) committed by Louwrentius - Reworked ppss regarding process management.. Reworked ppss regarding process management.. Dec 16, 2009 Changelog (Changelog of distributed PPSS) Wiki page edited by Louwrentius - Revision r341 Edited wiki page through web user interface. Revision r341 Edited wiki page through web user interface. Dec 16, 2009 issue 19 (Processes launched are not killed when I kill ppss.sh) Status changed by Louwrentius - In the upcoming version of PPSS this will be fixed. After some work I found a solution to make sure that all children are killed when PPSS itself is killed using ctrl-c. The downside is some cosmetic issues, however, everything is killed properly. Status: Fixed In the upcoming version of PPSS this will be fixed. After some work I found a solution to make sure that all children are killed when PPSS itself is killed using ctrl-c. The downside is some cosmetic issues, however, everything is killed properly. Status: Fixed Dec 16, 2009 r340 (More small fixes. ) committed by Louwrentius - More small fixes. More small fixes. Dec 16, 2009 issue 21 (Bug or feature - $ITEM or "$ITEM") Status changed by Louwrentius - Status: Fixed Status: Fixed Dec 16, 2009 r339 (some rework) committed by Louwrentius - some rework some rework Dec 16, 2009 Overview (Introduction about PPSS and its features.) Wiki page edited by Louwrentius - Revision r338 Edited wiki page through web user interface. Revision r338 Edited wiki page through web user interface. Dec 16, 2009 Overview (Introduction about PPSS and its features.) Wiki page edited by Louwrentius - Revision r337 Edited wiki page through web user interface. Revision r337 Edited wiki page through web user interface. Dec 16, 2009 Overview (Introduction about PPSS and its features.) Wiki page edited by Louwrentius - Revision r336 Edited wiki page through web user interface. Revision r336 Edited wiki page through web user interface. Dec 16, 2009 Manual2 (PPSS Manual (Distributed)) Wiki page edited by Louwrentius - Revision r335 Edited wiki page through web user interface. Revision r335 Edited wiki page through web user interface. Dec 16, 2009 Manual1 (PPSS Manual (Standalone) read me first.) Wiki page edited by Louwrentius - Revision r334 Edited wiki page through web user interface. Revision r334 Edited wiki page through web user interface. Dec 16, 2009 Manual1 (PPSS Manual (Standalone) read me first.) Wiki page edited by Louwrentius - Revision r333 Edited wiki page through web user interface. Revision r333 Edited wiki page through web user interface. Dec 16, 2009 Manual1 (PPSS Manual (Standalone) read me first.) Wiki page edited by Louwrentius - Revision r332 Edited wiki page through web user interface. Revision r332 Edited wiki page through web user interface. Dec 16, 2009 issue 21 (Bug or feature - $ITEM or "$ITEM") Status changed by Louwrentius - I now see the issue. In order to run this command properly, you must use $ITEM instead of "$ITEM" as how I documented it. This depends on the situation. I cannot fix this, I will update the documentation. Status: Started I now see the issue. In order to run this command properly, you must use $ITEM instead of "$ITEM" as how I documented it. This depends on the situation. I cannot fix this, I will update the documentation. Status: Started Dec 16, 2009 issue 18 (If multiple tasks are the same it runs only once) Status changed by Louwrentius - This issue is expected behaviour. Status: WontFix This issue is expected behaviour. Status: WontFix Dec 16, 2009 issue 21 (Bug or feature - $ITEM or "$ITEM") Status changed by Louwrentius - I could not get your script to work with 'sh ', although 'bash ' worked withouth a hitch. You need to 'escape' the $ITEM substitute if it contains spaces or it wil not be parsed properly. If I do not understand this issue properly, please let me know. Status: WontFix I could not get your script to work with 'sh ', although 'bash ' worked withouth a hitch. You need to 'escape' the $ITEM substitute if it contains spaces or it wil not be parsed properly. If I do not understand this issue properly, please let me know. Status: WontFix Dec 16, 2009 issue 20 (Intermittent ppss.sh failures) Status changed by Louwrentius - It has been fixed in version 2.45 which will be released shortly. Status: Fixed It has been fixed in version 2.45 which will be released shortly. Status: Fixed Dec 16, 2009 issue 16 (ppss.sh does not work in a FAT32 filesystem) Status changed by Louwrentius - This is fixed since 2.41 Status: Fixed This is fixed since 2.41 Status: Fixed Dec 16, 2009 r331 (Significant bug fixing and testing. Updated to 2.45) committed by Louwrentius - Significant bug fixing and testing. Updated to 2.45 Significant bug fixing and testing. Updated to 2.45 Dec 15, 2009 issue 17 (Arguments cannot contain ":" or directory and log file creat...) Status changed by Louwrentius - {{{ nan03@Core7i:~/ppss$ echo "http://hello.wordpress.com" > temp.txt nan03@Core7i:~/ppss$ ./ppss.sh -c 'echo ' -f temp.txt Dec 16 02:45:46: ========================================================= Dec 16 02:45:46: \|P\|P\|S\|S\| Dec 16 02:45:46: Distributed Parallel Processing Shell Script version 2.45 Dec 16 02:45:46: ========================================================= Dec 16 02:45:46: Hostname: Core7i Dec 16 02:45:46: --------------------------------------------------------- Dec 16 02:45:46: CPU: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz Dec 16 02:45:46: Found 8 logic processors. Dec 16 02:45:46: Starting 8 parallel workers. Dec 16 02:45:46: --------------------------------------------------------- Dec 16 02:45:46: Currently 100 percent complete. Processed 1 of 1 items. Dec 16 02:45:52: 1 job is remaining. Dec 16 02:45:52: Finished. Consult ./ppss/job_log for job output. nan03@Core7i:~/ppss$ cat ppss/job_log/http\:hello.wordpress.com ===== PPSS Item Log File ===== Host: Core7i Process:9073 Item: http://hello.wordpress.com Start date: Dec 16 02:44:47 http://hello.wordpress.com Status: SUCCESS Elapsed time (h:m:s): 0:0:0 nan03@Core7i:~/ppss$ cat /etc/u ucf.conf ufw/ update-manager/ update-notifier/ udev/ updatedb.conf update-motd.d/ usplash.conf nan03@Core7i:~/ppss$ cat /etc/deb debconf.conf debian_version nan03@Core7i:~/ppss$ cat /etc/debian_version squeeze/sid }}} Solved. Status: Fixed {{{ nan03@Core7i:~/ppss$ echo "http://hello.wordpress.com" > temp.txt nan03@Core7i:~/ppss$ ./ppss.sh -c 'echo ' -f temp.txt Dec 16 02:45:46: ========================================================= Dec 16 02:45:46: \|P\|P\|S\|S\| Dec 16 02:45:46: Distributed Parallel Processing Shell Script version 2.45 Dec 16 02:45:46: ========================================================= Dec 16 02:45:46: Hostname: Core7i Dec 16 02:45:46: --------------------------------------------------------- Dec 16 02:45:46: CPU: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz Dec 16 02:45:46: Found 8 logic processors. Dec 16 02:45:46: Starting 8 parallel workers. Dec 16 02:45:46: --------------------------------------------------------- Dec 16 02:45:46: Currently 100 percent complete. Processed 1 of 1 items. Dec 16 02:45:52: 1 job is remaining. Dec 16 02:45:52: Finished. Consult ./ppss/job_log for job output. nan03@Core7i:~/ppss$ cat ppss/job_log/http\:hello.wordpress.com ===== PPSS Item Log File ===== Host: Core7i Process:9073 Item: http://hello.wordpress.com Start date: Dec 16 02:44:47 http://hello.wordpress.com Status: SUCCESS Elapsed time (h:m:s): 0:0:0 nan03@Core7i:~/ppss$ cat /etc/u ucf.conf ufw/ update-manager/ update-notifier/ udev/ updatedb.conf update-motd.d/ usplash.conf nan03@Core7i:~/ppss$ cat /etc/deb debconf.conf debian_version nan03@Core7i:~/ppss$ cat /etc/debian_version squeeze/sid }}} Solved. Status: Fixed Dec 15, 2009 r330 (Seems to fix some file name issues...) committed by Louwrentius - Seems to fix some file name issues... Seems to fix some file name issues... Dec 13, 2009 issue 21 (Bug or feature - $ITEM or "$ITEM") commented on by Louwrentius - I think I understand, It must be better documented that if you use the $ITEM placeholder, you need to use " ". I think I understand, It must be better documented that if you use the $ITEM placeholder, you need to use " ". Dec 13, 2009 issue 19 (Processes launched are not killed when I kill ppss.sh) commented on by Louwrentius - Thanx for your input, this must be fixed. Thanx for your input, this must be fixed. Dec 13, 2009 issue 20 (Intermittent ppss.sh failures) commented on by Louwrentius - Thanx for the report, this should be fixed. Thanx for the report, this should be fixed. Dec 13, 2009 issue 18 (If multiple tasks are the same it runs only once) commented on by Louwrentius - Thank you for your feedback. I understand the issue. However, this is expected behavior. If you do not want this behavior, I think PPSS is not the answer to your problem in the first place, or you should make the items unique. PPSS is designed with the assumption that items are unique. PPSS detects if an item already has been processed based on the existence of a file name and will not process it again. So that is why identical items will only be processed once. Thank you for your feedback. I understand the issue. However, this is expected behavior. If you do not want this behavior, I think PPSS is not the answer to your problem in the first place, or you should make the items unique. PPSS is designed with the assumption that items are unique. PPSS detects if an item already has been processed based on the existence of a file name and will not process it again. So that is why identical items will only be processed once. Dec 09, 2009 issue 21 (Bug or feature - $ITEM or "$ITEM") reported by sarkarprateek - What steps will reproduce the problem? [psarkar@teentaal ppssTrial]$ cat largeLoop.sh #!/bin/bash N=0 PN=$BASHPID while (( $N < 1000 )); do N=$(( $N + 1 )); echo ${PN}_${N}; sleep 1; done [psarkar@teentaal ppssTrial]$ head -5 joblistNumbered.txt /tmp/ppssTrial/largeLoop.sh 1 /tmp/ppssTrial/largeLoop.sh 2 /tmp/ppssTrial/largeLoop.sh 3 /tmp/ppssTrial/largeLoop.sh 4 /tmp/ppssTrial/largeLoop.sh 5 [psarkar@teentaal ppssTrial]$ wc -l joblistNumbered.txt 170 joblistNumbered.txt [psarkar@teentaal ppssTrial]$ What is the expected output? What do you see instead? I expect ppss.sh -f aaa -c 'sh ' and ppss.sh -f aaa -c 'sh $ITEM' to behave the same. instead former behaves like ppss.sh -f aaa -c 'sh "$ITEM"' See below: [psarkar@teentaal ppssTrial]$ ppss.sh -f joblistNumbered.txt -c 'sh ' Dec 09 10:27:38: ========================================================= Dec 09 10:27:38: \|P\|P\|S\|S\| Dec 09 10:27:38: Distributed Parallel Processing Shell Script version 2.41 Dec 09 10:27:38: ========================================================= Dec 09 10:27:38: Hostname: teentaal Dec 09 10:27:38: --------------------------------------------------------- Dec 09 10:27:38: CPU: Intel(R) Xeon(TM) CPU 3.20GHz Dec 09 10:27:38: Found 8 logic processors. Dec 09 10:27:38: Starting 8 parallel workers. Dec 09 10:27:38: --------------------------------------------------------- Dec 09 10:27:45: Currently 100 percent complete. Processed 170 of 170 items. Dec 09 10:27:46: 1 job is remaining. Dec 09 10:27:46: Finished. Consult ./ppss/job_log for job output. [psarkar@teentaal ppssTrial]$ more ppss/job_log/tmpppssTriallargeLoop.sh\ 1 ===== PPSS Item Log File ===== Host: teentaal Process:12686 Item: /tmp/ppssTrial/largeLoop.sh 1 Start date: Dec 09 10:27:39 sh: /tmp/ppssTrial/largeLoop.sh 1: No such file or directory Status: FAILURE Elapsed time (h:m:s): 0:0:0 [psarkar@teentaal ppssTrial]$ \rm -r ppss/ [psarkar@teentaal ppssTrial]$ ppss.sh -f joblistNumbered.txt -c 'sh $ITEM' Dec 09 10:28:55: ========================================================= Dec 09 10:28:55: \|P\|P\|S\|S\| Dec 09 10:28:55: Distributed Parallel Processing Shell Script version 2.41 Dec 09 10:28:55: ========================================================= Dec 09 10:28:55: Hostname: teentaal Dec 09 10:28:55: --------------------------------------------------------- Dec 09 10:28:55: CPU: Intel(R) Xeon(TM) CPU 3.20GHz Dec 09 10:28:55: Found 8 logic processors. Dec 09 10:28:55: Starting 8 parallel workers. Dec 09 10:28:55: --------------------------------------------------------- Dec 09 10:28:55: Currently 4 percent complete. Processed 8 of 170 items. What version of the product are you using? On what operating system? Please provide any additional information below. What steps will reproduce the problem? [psarkar@teentaal ppssTrial]$ cat largeLoop.sh #!/bin/bash N=0 PN=$BASHPID while (( $N < 1000 )); do N=$(( $N + 1 )); echo ${PN}_${N}; sleep 1; done [psarkar@teentaal ppssTrial]$ head -5 joblistNumbered.txt /tmp/ppssTrial/largeLoop.sh 1 /tmp/ppssTrial/largeLoop.sh 2 /tmp/ppssTrial/largeLoop.sh 3 /tmp/ppssTrial/largeLoop.sh 4 /tmp/ppssTrial/largeLoop.sh 5 [psarkar@teentaal ppssTrial]$ wc -l joblistNumbered.txt 170 joblistNumbered.txt [psarkar@teentaal ppssTrial]$ What is the expected output? What do you see instead? I expect ppss.sh -f aaa -c 'sh ' and ppss.sh -f aaa -c 'sh $ITEM' to behave the same. instead former behaves like ppss.sh -f aaa -c 'sh "$ITEM"' See below: [psarkar@teentaal ppssTrial]$ ppss.sh -f joblistNumbered.txt -c 'sh ' Dec 09 10:27:38: ========================================================= Dec 09 10:27:38: \|P\|P\|S\|S\| Dec 09 10:27:38: Distributed Parallel Processing Shell Script version 2.41 Dec 09 10:27:38: ========================================================= Dec 09 10:27:38: Hostname: teentaal Dec 09 10:27:38: --------------------------------------------------------- Dec 09 10:27:38: CPU: Intel(R) Xeon(TM) CPU 3.20GHz Dec 09 10:27:38: Found 8 logic processors. Dec 09 10:27:38: Starting 8 parallel workers. Dec 09 10:27:38: --------------------------------------------------------- Dec 09 10:27:45: Currently 100 percent complete. Processed 170 of 170 items. Dec 09 10:27:46: 1 job is remaining. Dec 09 10:27:46: Finished. Consult ./ppss/job_log for job output. [psarkar@teentaal ppssTrial]$ more ppss/job_log/tmpppssTriallargeLoop.sh\ 1 ===== PPSS Item Log File ===== Host: teentaal Process:12686 Item: /tmp/ppssTrial/largeLoop.sh 1 Start date: Dec 09 10:27:39 sh: /tmp/ppssTrial/largeLoop.sh 1: No such file or directory Status: FAILURE Elapsed time (h:m:s): 0:0:0 [psarkar@teentaal ppssTrial]$ \rm -r ppss/ [psarkar@teentaal ppssTrial]$ ppss.sh -f joblistNumbered.txt -c 'sh $ITEM' Dec 09 10:28:55: ========================================================= Dec 09 10:28:55: \|P\|P\|S\|S\| Dec 09 10:28:55: Distributed Parallel Processing Shell Script version 2.41 Dec 09 10:28:55: ========================================================= Dec 09 10:28:55: Hostname: teentaal Dec 09 10:28:55: --------------------------------------------------------- Dec 09 10:28:55: CPU: Intel(R) Xeon(TM) CPU 3.20GHz Dec 09 10:28:55: Found 8 logic processors. Dec 09 10:28:55: Starting 8 parallel workers. Dec 09 10:28:55: --------------------------------------------------------- Dec 09 10:28:55: Currently 4 percent complete. Processed 8 of 170 items. What version of the product are you using? On what operating system? Please provide any additional information below. Dec 09, 2009 issue 20 (Intermittent ppss.sh failures) reported by sarkarprateek - What steps will reproduce the problem? [psarkar@teentaal ppssTrial]$ cat largeLoop.sh #!/bin/bash N=0 PN=$BASHPID while (( $N < 1000 )); do N=$(( $N + 1 )); echo ${PN}_${N}; sleep 1; done [psarkar@teentaal ppssTrial]$ head -5 joblistNumbered.txt /tmp/ppssTrial/largeLoop.sh 1 /tmp/ppssTrial/largeLoop.sh 2 /tmp/ppssTrial/largeLoop.sh 3 /tmp/ppssTrial/largeLoop.sh 4 /tmp/ppssTrial/largeLoop.sh 5 [psarkar@teentaal ppssTrial]$ wc -l joblistNumbered.txt 170 joblistNumbered.txt [psarkar@teentaal ppssTrial]$ What is the expected output? What do you see instead? [psarkar@teentaal ppssTrial]$ ppss.sh -f joblistNumbered.txt -c 'sh ' Dec 09 10:11:40: ========================================================= Dec 09 10:11:40: \|P\|P\|S\|S\| Dec 09 10:11:40: Distributed Parallel Processing Shell Script version 2.41 Dec 09 10:11:40: ========================================================= Dec 09 10:11:40: Hostname: teentaal Dec 09 10:11:40: --------------------------------------------------------- Dec 09 10:11:40: CPU: Intel(R) Xeon(TM) CPU 3.20GHz Dec 09 10:11:40: Found 8 logic processors. Dec 09 10:11:40: Starting 8 parallel workers. Dec 09 10:11:40: --------------------------------------------------------- /tilde/psarkar/UTILS/bin/ppss.sh: line 1514: 100 * / 170 : syntax error: operand expected (error token is "/ 170 ") ^^^^^^ I have seen this failure a few times, but it happens intermittently I have not figured out how to reproduce it. What version of the product are you using? On what operating system? ppss 2.4.1 on Fedora 11, Intel Xeon Quad w/hyperthreading. Please provide any additional information below. What steps will reproduce the problem? [psarkar@teentaal ppssTrial]$ cat largeLoop.sh #!/bin/bash N=0 PN=$BASHPID while (( $N < 1000 )); do N=$(( $N + 1 )); echo ${PN}_${N}; sleep 1; done [psarkar@teentaal ppssTrial]$ head -5 joblistNumbered.txt /tmp/ppssTrial/largeLoop.sh 1 /tmp/ppssTrial/largeLoop.sh 2 /tmp/ppssTrial/largeLoop.sh 3 /tmp/ppssTrial/largeLoop.sh 4 /tmp/ppssTrial/largeLoop.sh 5 [psarkar@teentaal ppssTrial]$ wc -l joblistNumbered.txt 170 joblistNumbered.txt [psarkar@teentaal ppssTrial]$ What is the expected output? What do you see instead? [psarkar@teentaal ppssTrial]$ ppss.sh -f joblistNumbered.txt -c 'sh ' Dec 09 10:11:40: ========================================================= Dec 09 10:11:40: \|P\|P\|S\|S\| Dec 09 10:11:40: Distributed Parallel Processing Shell Script version 2.41 Dec 09 10:11:40: ========================================================= Dec 09 10:11:40: Hostname: teentaal Dec 09 10:11:40: --------------------------------------------------------- Dec 09 10:11:40: CPU: Intel(R) Xeon(TM) CPU 3.20GHz Dec 09 10:11:40: Found 8 logic processors. Dec 09 10:11:40: Starting 8 parallel workers. Dec 09 10:11:40: --------------------------------------------------------- /tilde/psarkar/UTILS/bin/ppss.sh: line 1514: 100 * / 170 : syntax error: operand expected (error token is "/ 170 ") ^^^^^^ I have seen this failure a few times, but it happens intermittently I have not figured out how to reproduce it. What version of the product are you using? On what operating system? ppss 2.4.1 on Fedora 11, Intel Xeon Quad w/hyperthreading. Please provide any additional information below. Dec 09, 2009 issue 19 (Processes launched are not killed when I kill ppss.sh) reported by sarkarprateek - What steps will reproduce the problem? [psarkar@teentaal ppssTrial]$ cat largeLoop.sh #!/bin/bash N=0 PN=$BASHPID while (( $N < 1000 )); do N=$(( $N + 1 )); echo ${PN}_${N}; sleep 1; done [psarkar@teentaal ppssTrial]$ head -5 joblistNumbered.txt /tmp/ppssTrial/largeLoop.sh 1 /tmp/ppssTrial/largeLoop.sh 2 /tmp/ppssTrial/largeLoop.sh 3 /tmp/ppssTrial/largeLoop.sh 4 /tmp/ppssTrial/largeLoop.sh 5 [psarkar@teentaal ppssTrial]$ wc -l joblistNumbered.txt 170 joblistNumbered.txt [psarkar@teentaal ppssTrial]$ \rm -r ppss/ [psarkar@teentaal ppssTrial]$ ps \| grep largeLoop [psarkar@teentaal ppssTrial]$ [psarkar@teentaal ppssTrial]$ ppss.sh -f joblistNumbered.txt -c 'sh $ITEM' Dec 09 10:14:20: ========================================================= Dec 09 10:14:20: \|P\|P\|S\|S\| Dec 09 10:14:20: Distributed Parallel Processing Shell Script version 2.41 Dec 09 10:14:20: ========================================================= Dec 09 10:14:20: Hostname: teentaal Dec 09 10:14:20: --------------------------------------------------------- Dec 09 10:14:20: CPU: Intel(R) Xeon(TM) CPU 3.20GHz Dec 09 10:14:20: Found 8 logic processors. Dec 09 10:14:20: Starting 8 parallel workers. Dec 09 10:14:20: --------------------------------------------------------- ^C[psarkar@teentaal ppssTrial]$ Dec 09 10:14:34: Finished. Consult ./ppss/job_log for job output. [psarkar@teentaal ppssTrial]$ ps a \| grep largeLoop 5766 pts/2 S 0:00 sh /tmp/ppssTrial/largeLoop.sh 1 5821 pts/2 S 0:00 sh /tmp/ppssTrial/largeLoop.sh 2 5852 pts/2 S 0:00 sh /tmp/ppssTrial/largeLoop.sh 3 5887 pts/2 S 0:00 sh /tmp/ppssTrial/largeLoop.sh 4 5922 pts/2 S 0:00 sh /tmp/ppssTrial/largeLoop.sh 5 5967 pts/2 S 0:00 sh /tmp/ppssTrial/largeLoop.sh 6 6005 pts/2 S 0:00 sh /tmp/ppssTrial/largeLoop.sh 7 6027 pts/2 S 0:00 sh /tmp/ppssTrial/largeLoop.sh 8 6417 pts/2 S+ 0:00 grep largeLoop [psarkar@teentaal ppssTrial]$ What is the expected output? What do you see instead? The above shows the bash-shell interactions. As you can see, even after the ppss.sh process has stopped with a Ctrl-C, the 8 processes launched are still running. What version of the product are you using? On what operating system? . ppss version 2.4.1 on Fedora 11, intel Xeon quad core w/ hyperthreading Please provide any additional information below. What steps will reproduce the problem? [psarkar@teentaal ppssTrial]$ cat largeLoop.sh #!/bin/bash N=0 PN=$BASHPID while (( $N < 1000 )); do N=$(( $N + 1 )); echo ${PN}_${N}; sleep 1; done [psarkar@teentaal ppssTrial]$ head -5 joblistNumbered.txt /tmp/ppssTrial/largeLoop.sh 1 /tmp/ppssTrial/largeLoop.sh 2 /tmp/ppssTrial/largeLoop.sh 3 /tmp/ppssTrial/largeLoop.sh 4 /tmp/ppssTrial/largeLoop.sh 5 [psarkar@teentaal ppssTrial]$ wc -l joblistNumbered.txt 170 joblistNumbered.txt [psarkar@teentaal ppssTrial]$ \rm -r ppss/ [psarkar@teentaal ppssTrial]$ ps \| grep largeLoop [psarkar@teentaal ppssTrial]$ [psarkar@teentaal ppssTrial]$ ppss.sh -f joblistNumbered.txt -c 'sh $ITEM' Dec 09 10:14:20: ========================================================= Dec 09 10:14:20: \|P\|P\|S\|S\| Dec 09 10:14:20: Distributed Parallel Processing Shell Script version 2.41 Dec 09 10:14:20: ========================================================= Dec 09 10:14:20: Hostname: teentaal Dec 09 10:14:20: --------------------------------------------------------- Dec 09 10:14:20: CPU: Intel(R) Xeon(TM) CPU 3.20GHz Dec 09 10:14:20: Found 8 logic processors. Dec 09 10:14:20: Starting 8 parallel workers. Dec 09 10:14:20: --------------------------------------------------------- ^C[psarkar@teentaal ppssTrial]$ Dec 09 10:14:34: Finished. Consult ./ppss/job_log for job output. [psarkar@teentaal ppssTrial]$ ps a \| grep largeLoop 5766 pts/2 S 0:00 sh /tmp/ppssTrial/largeLoop.sh 1 5821 pts/2 S 0:00 sh /tmp/ppssTrial/largeLoop.sh 2 5852 pts/2 S 0:00 sh /tmp/ppssTrial/largeLoop.sh 3 5887 pts/2 S 0:00 sh /tmp/ppssTrial/largeLoop.sh 4 5922 pts/2 S 0:00 sh /tmp/ppssTrial/largeLoop.sh 5 5967 pts/2 S 0:00 sh /tmp/ppssTrial/largeLoop.sh 6 6005 pts/2 S 0:00 sh /tmp/ppssTrial/largeLoop.sh 7 6027 pts/2 S 0:00 sh /tmp/ppssTrial/largeLoop.sh 8 6417 pts/2 S+ 0:00 grep largeLoop [psarkar@teentaal ppssTrial]$ What is the expected output? What do you see instead? The above shows the bash-shell interactions. As you can see, even after the ppss.sh process has stopped with a Ctrl-C, the 8 processes launched are still running. What version of the product are you using? On what operating system? . ppss version 2.4.1 on Fedora 11, intel Xeon quad core w/ hyperthreading Please provide any additional information below. Dec 09, 2009 issue 18 (If multiple tasks are the same it runs only once) reported by sarkarprateek - What steps will reproduce the problem? 1. Create a joblist file, by repeating a command many times. [psarkar@teentaal ppssTrial]$ head -5 joblist.txt /tmp/ppssTrial/largeLoop.sh /tmp/ppssTrial/largeLoop.sh /tmp/ppssTrial/largeLoop.sh /tmp/ppssTrial/largeLoop.sh /tmp/ppssTrial/largeLoop.sh [psarkar@teentaal ppssTrial]$ wc -l joblist.txt 170 joblist.txt [psarkar@teentaal ppssTrial]$ 2. Here is what my largeLoop.sh looks like. The thing to observe is that every time it runs, it should produce a different output. [psarkar@teentaal ppssTrial]$ cat largeLoop.sh #!/bin/bash N=0 PN=$BASHPID while (( $N < 1000 )); do N=$(( $N + 1 )); echo ${PN}_${N}; sleep 1; done [psarkar@teentaal ppssTrial]$ What is the expected output? What do you see instead? I would have expected that all processes would run (i.e., the program would run once for each time it is listed in the joblist file). Instead it runs only once. [psarkar@teentaal ppssTrial]$ \rm -r ppss/ [psarkar@teentaal ppssTrial]$ ppss.sh -f joblist.txt -c 'sh ' Dec 09 09:55:14: ========================================================= Dec 09 09:55:14: \|P\|P\|S\|S\| Dec 09 09:55:14: Distributed Parallel Processing Shell Script version 2.41 Dec 09 09:55:14: ========================================================= Dec 09 09:55:14: Hostname: teentaal Dec 09 09:55:14: --------------------------------------------------------- Dec 09 09:55:14: CPU: Intel(R) Xeon(TM) CPU 3.20GHz Dec 09 09:55:14: Found 8 logic processors. Dec 09 09:55:14: Starting 8 parallel workers. Dec 09 09:55:14: --------------------------------------------------------- Dec 09 09:55:21: Currently 100 percent complete. Processed 170 of 170 items. Dec 09 09:55:24: 1 job is remaining. ^C[psarkar@teentaal ppssTrial]$ Dec 09 09:55:34: Finished. Consult ./ppss/job_log for job output. [psarkar@teentaal ppssTrial]$ more ppss/job_log/tmpppssTriallargeLoop.sh ===== PPSS Item Log File ===== Host: teentaal Process:14109 Item: /tmp/ppssTrial/largeLoop.sh Start date: Dec 09 09:55:15 14225_1 14225_2 14225_3 14225_4 14225_5 14225_6 14225_7 14225_8 14225_9 14225_10 14225_11 14225_12 14225_13 14225_14 14225_15 14225_16 14225_17 14225_18 14225_19 14225_20 14225_21 14225_22 14225_23 14225_24 14225_25 14225_26 14225_27 14225_28 14225_29 What version of the product are you using? On what operating system? \|P\|P\|S\|S\| Distributed Parallel Processing Shell Script 2.41 Please provide any additional information below. There are a number of different programs that will produce different output each time it is run. E.g., optimizers with random initialization. It would be good if these can be handled. Perhaps the process ID or the command sequence number can serve as a hash/descriptor -- rather than the altered command line. This may also alleviate some of the other issues I see (e.g. regarding ':' or special chars in commands.) What steps will reproduce the problem? 1. Create a joblist file, by repeating a command many times. [psarkar@teentaal ppssTrial]$ head -5 joblist.txt /tmp/ppssTrial/largeLoop.sh /tmp/ppssTrial/largeLoop.sh /tmp/ppssTrial/largeLoop.sh /tmp/ppssTrial/largeLoop.sh /tmp/ppssTrial/largeLoop.sh [psarkar@teentaal ppssTrial]$ wc -l joblist.txt 170 joblist.txt [psarkar@teentaal ppssTrial]$ 2. Here is what my largeLoop.sh looks like. The thing to observe is that every time it runs, it should produce a different output. [psarkar@teentaal ppssTrial]$ cat largeLoop.sh #!/bin/bash N=0 PN=$BASHPID while (( $N < 1000 )); do N=$(( $N + 1 )); echo ${PN}_${N}; sleep 1; done [psarkar@teentaal ppssTrial]$ What is the expected output? What do you see instead? I would have expected that all processes would run (i.e., the program would run once for each time it is listed in the joblist file). Instead it runs only once. [psarkar@teentaal ppssTrial]$ \rm -r ppss/ [psarkar@teentaal ppssTrial]$ ppss.sh -f joblist.txt -c 'sh ' Dec 09 09:55:14: ========================================================= Dec 09 09:55:14: \|P\|P\|S\|S\| Dec 09 09:55:14: Distributed Parallel Processing Shell Script version 2.41 Dec 09 09:55:14: ========================================================= Dec 09 09:55:14: Hostname: teentaal Dec 09 09:55:14: --------------------------------------------------------- Dec 09 09:55:14: CPU: Intel(R) Xeon(TM) CPU 3.20GHz Dec 09 09:55:14: Found 8 logic processors. Dec 09 09:55:14: Starting 8 parallel workers. Dec 09 09:55:14: --------------------------------------------------------- Dec 09 09:55:21: Currently 100 percent complete. Processed 170 of 170 items. Dec 09 09:55:24: 1 job is remaining. ^C[psarkar@teentaal ppssTrial]$ Dec 09 09:55:34: Finished. Consult ./ppss/job_log for job output. [psarkar@teentaal ppssTrial]$ more ppss/job_log/tmpppssTriallargeLoop.sh ===== PPSS Item Log File ===== Host: teentaal Process:14109 Item: /tmp/ppssTrial/largeLoop.sh Start date: Dec 09 09:55:15 14225_1 14225_2 14225_3 14225_4 14225_5 14225_6 14225_7 14225_8 14225_9 14225_10 14225_11 14225_12 14225_13 14225_14 14225_15 14225_16 14225_17 14225_18 14225_19 14225_20 14225_21 14225_22 14225_23 14225_24 14225_25 14225_26 14225_27 14225_28 14225_29 What version of the product are you using? On what operating system? \|P\|P\|S\|S\| Distributed Parallel Processing Shell Script 2.41 Please provide any additional information below. There are a number of different programs that will produce different output each time it is run. E.g., optimizers with random initialization. It would be good if these can be handled. Perhaps the process ID or the command sequence number can serve as a hash/descriptor -- rather than the altered command line. This may also alleviate some of the other issues I see (e.g. regarding ':' or special chars in commands.) Older Nov 17, 2009 issue 17 (Arguments cannot contain ":" or directory and log file creat...) commented on by Louwrentius - A solution will be found for the make dir issue and incorporated. A solution will be found for the make dir issue and incorporated. Nov 17, 2009 issue 16 (ppss.sh does not work in a FAT32 filesystem) Owner changed by Louwrentius - Owner: Louwrentius Owner: Louwrentius Nov 17, 2009 issue 17 (Arguments cannot contain ":" or directory and log file creat...) changed by Louwrentius - This is some Ubuntu specific issue: nan03@Core7i:~$ ./ppss.sh -f temp.txt -c 'echo ' Nov 17 23:50:23: ========================================================= Nov 17 23:50:23: \|P\|P\|S\|S\| Nov 17 23:50:23: Distributed Parallel Processing Shell Script version 2.41 Nov 17 23:50:23: ========================================================= Nov 17 23:50:23: Hostname: Core7i Nov 17 23:50:23: --------------------------------------------------------- Nov 17 23:50:23: CPU: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz Nov 17 23:50:23: Found 8 logic processors. Nov 17 23:50:23: Starting 8 parallel workers. Nov 17 23:50:23: --------------------------------------------------------- Nov 17 23:50:23: Currently 100 percent complete. Processed 1 of 1 items. mkdir: cannot create directory `/http:': Permission denied Nov 17 23:50:23: 1 job is remaining. Nov 17 23:50:23: Finished. Consult ./ppss/job_log for job output. Status: Accepted Owner: Louwrentius This is some Ubuntu specific issue: nan03@Core7i:~$ ./ppss.sh -f temp.txt -c 'echo ' Nov 17 23:50:23: ========================================================= Nov 17 23:50:23: \|P\|P\|S\|S\| Nov 17 23:50:23: Distributed Parallel Processing Shell Script version 2.41 Nov 17 23:50:23: ========================================================= Nov 17 23:50:23: Hostname: Core7i Nov 17 23:50:23: --------------------------------------------------------- Nov 17 23:50:23: CPU: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz Nov 17 23:50:23: Found 8 logic processors. Nov 17 23:50:23: Starting 8 parallel workers. Nov 17 23:50:23: --------------------------------------------------------- Nov 17 23:50:23: Currently 100 percent complete. Processed 1 of 1 items. mkdir: cannot create directory `/http:': Permission denied Nov 17 23:50:23: 1 job is remaining. Nov 17 23:50:23: Finished. Consult ./ppss/job_log for job output. Status: Accepted Owner: Louwrentius Nov 17, 2009 issue 17 (Arguments cannot contain ":" or directory and log file creat...) commented on by Louwrentius - No problem on Debian Lenny: Mini:~# echo "http://hello.wordpress.com" > temp.txt Mini:~# ./ppss.sh -c 'echo ' -f temp.txt Nov 17 23:45:41: ========================================================= Nov 17 23:45:41: \|P\|P\|S\|S\| Nov 17 23:45:41: Distributed Parallel Processing Shell Script version 2.41 Nov 17 23:45:41: ========================================================= Nov 17 23:45:41: Hostname: Mini Nov 17 23:45:41: --------------------------------------------------------- Nov 17 23:45:41: CPU: Genuine Intel(R) CPU 1300 @ 1.66GHz Nov 17 23:45:41: Found 2 logic processors. Nov 17 23:45:41: Starting 2 parallel workers. Nov 17 23:45:41: --------------------------------------------------------- Nov 17 23:45:41: Currently 100 percent complete. Processed 1 of 1 items. Nov 17 23:45:41: 1 job is remaining. Nov 17 23:45:41: Finished. Consult ./ppss/job_log for job output. Mini:~# cat ppss/job_log/http\:hello.wordpress.com ===== PPSS Item Log File ===== Host: Mini Process:31253 Item: http://hello.wordpress.com Start date: Nov 17 23:45:41 http://hello.wordpress.com Status: SUCCESS Elapsed time (h:m:s): 0:0:0 Mini:~# cat /etc/de debconf.conf debian_version default/ defoma/ deluser.conf Mini:~# cat /etc/de debconf.conf debian_version default/ defoma/ deluser.conf Mini:~# cat /etc/debian_version 5.0.3 No problem on Debian Lenny: Mini:~# echo "http://hello.wordpress.com" > temp.txt Mini:~# ./ppss.sh -c 'echo ' -f temp.txt Nov 17 23:45:41: ========================================================= Nov 17 23:45:41: \|P\|P\|S\|S\| Nov 17 23:45:41: Distributed Parallel Processing Shell Script version 2.41 Nov 17 23:45:41: ========================================================= Nov 17 23:45:41: Hostname: Mini Nov 17 23:45:41: --------------------------------------------------------- Nov 17 23:45:41: CPU: Genuine Intel(R) CPU 1300 @ 1.66GHz Nov 17 23:45:41: Found 2 logic processors. Nov 17 23:45:41: Starting 2 parallel workers. Nov 17 23:45:41: --------------------------------------------------------- Nov 17 23:45:41: Currently 100 percent complete. Processed 1 of 1 items. Nov 17 23:45:41: 1 job is remaining. Nov 17 23:45:41: Finished. Consult ./ppss/job_log for job output. Mini:~# cat ppss/job_log/http\:hello.wordpress.com ===== PPSS Item Log File ===== Host: Mini Process:31253 Item: http://hello.wordpress.com Start date: Nov 17 23:45:41 http://hello.wordpress.com Status: SUCCESS Elapsed time (h:m:s): 0:0:0 Mini:~# cat /etc/de debconf.conf debian_version default/ defoma/ deluser.conf Mini:~# cat /etc/de debconf.conf debian_version default/ defoma/ deluser.conf Mini:~# cat /etc/debian_version 5.0.3 Nov 17, 2009 issue 17 (Arguments cannot contain ":" or directory and log file creat...) commented on by Louwrentius - Strange. No proglems on Mac OS X Snow Leopard: bash-3.2$ echo "http://hello.wordpress.com" > temp.txt bash-3.2$ ./ppss.sh -c 'echo ' -f temp.txt nov 17 23:43:44: ========================================================= nov 17 23:43:44: \|P\|P\|S\|S\| nov 17 23:43:44: Distributed Parallel Processing Shell Script version 2.41 nov 17 23:43:44: ========================================================= nov 17 23:43:44: Hostname: MacBoek.local nov 17 23:43:44: --------------------------------------------------------- nov 17 23:43:44: CPU: Intel Core 2 Duo 2,16 GHz nov 17 23:43:44: Found 2 logic processors. nov 17 23:43:44: Starting 2 parallel workers. nov 17 23:43:44: --------------------------------------------------------- nov 17 23:43:45: Currently 100 percent complete. Processed 1 of 1 items. nov 17 23:43:45: 1 job is remaining. nov 17 23:43:45: Finished. Consult ./ppss/job_log for job output. bash-3.2$ tail ppss/ppss-log-43925.txt tail: ppss/ppss-log-43925.txt: No such file or directory bash-3.2$ tail ppss/ppss-log-44035.txt nov 17 23:43:44: INFO --------------------------------------------------------- nov 17 23:43:45: DEBUG Got lock on http://hello.wordpress.com, processing. nov 17 23:43:45: DEBUG No transfer of item hello.wordpress.com to local workpath. nov 17 23:43:45: INFO Currently 100 percent complete. Processed 1 of 1 items. nov 17 23:43:45: DEBUG Processing item http://hello.wordpress.com nov 17 23:43:45: DEBUG Using item straight from INPUT FILE nov 17 23:43:45: INFO 1 job is remaining. nov 17 23:43:45: DEBUG File transfer is disabled. nov 17 23:43:45: INFO Finished. Consult ./ppss/job_log for job output. nov 17 23:43:45: DEBUG Listener stopped. bash-3.2$ cat ppss/job_log/* ===== PPSS Item Log File ===== Host: MacBoek.local Process:44035 Item: http://hello.wordpress.com Start date: nov 17 23:43:45 http://hello.wordpress.com Status: SUCCESS Elapsed time (h:m:s): 0:0:0 Strange. No proglems on Mac OS X Snow Leopard: bash-3.2$ echo "http://hello.wordpress.com" > temp.txt bash-3.2$ ./ppss.sh -c 'echo ' -f temp.txt nov 17 23:43:44: ========================================================= nov 17 23:43:44: \|P\|P\|S\|S\| nov 17 23:43:44: Distributed Parallel Processing Shell Script version 2.41 nov 17 23:43:44: ========================================================= nov 17 23:43:44: Hostname: MacBoek.local nov 17 23:43:44: --------------------------------------------------------- nov 17 23:43:44: CPU: Intel Core 2 Duo 2,16 GHz nov 17 23:43:44: Found 2 logic processors. nov 17 23:43:44: Starting 2 parallel workers. nov 17 23:43:44: --------------------------------------------------------- nov 17 23:43:45: Currently 100 percent complete. Processed 1 of 1 items. nov 17 23:43:45: 1 job is remaining. nov 17 23:43:45: Finished. Consult ./ppss/job_log for job output. bash-3.2$ tail ppss/ppss-log-43925.txt tail: ppss/ppss-log-43925.txt: No such file or directory bash-3.2$ tail ppss/ppss-log-44035.txt nov 17 23:43:44: INFO --------------------------------------------------------- nov 17 23:43:45: DEBUG Got lock on http://hello.wordpress.com, processing. nov 17 23:43:45: DEBUG No transfer of item hello.wordpress.com to local workpath. nov 17 23:43:45: INFO Currently 100 percent complete. Processed 1 of 1 items. nov 17 23:43:45: DEBUG Processing item http://hello.wordpress.com nov 17 23:43:45: DEBUG Using item straight from INPUT FILE nov 17 23:43:45: INFO 1 job is remaining. nov 17 23:43:45: DEBUG File transfer is disabled. nov 17 23:43:45: INFO Finished. Consult ./ppss/job_log for job output. nov 17 23:43:45: DEBUG Listener stopped. bash-3.2$ cat ppss/job_log/* ===== PPSS Item Log File ===== Host: MacBoek.local Process:44035 Item: http://hello.wordpress.com Start date: nov 17 23:43:45 http://hello.wordpress.com Status: SUCCESS Elapsed time (h:m:s): 0:0:0 Nov 17, 2009 issue 16 (ppss.sh does not work in a FAT32 filesystem) changed by Louwrentius - Thank you for your reply. I changed the FIFO path exactly according to your suggestion. This will be incorporated in the next release of PPSS. Status: Accepted Labels: OpSys-All Thank you for your reply. I changed the FIFO path exactly according to your suggestion. This will be incorporated in the next release of PPSS. Status: Accepted Labels: OpSys-All Nov 17, 2009 issue 17 (Arguments cannot contain ":" or directory and log file creat...) reported by Roger.Sindreu - What steps will reproduce the problem? >echo "http://hello.wordpress.com" > temp.txt >ppss.sh -c 'echo ' -f temp.txt What is the expected output? What do you see instead? This is what I see: nov 17 11:56:31: ========================================================= nov 17 11:56:31: \|P\|P\|S\|S\| nov 17 11:56:31: Distributed Parallel Processing Shell Script version 2.41 nov 17 11:56:31: ========================================================= nov 17 11:56:31: Hostname: i7-desktop nov 17 11:56:31: --------------------------------------------------------- nov 17 11:56:31: CPU: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz nov 17 11:56:31: Found 8 logic processors. nov 17 11:56:31: Starting 8 parallel workers. nov 17 11:56:31: --------------------------------------------------------- nov 17 11:56:31: Currently 100 percent complete. Processed 1 of 1 items. /usr/bin/ppss.sh: line 1387: ppss/job_log/http:hello.wordpress.com: Invalid argument /usr/bin/ppss.sh: line 1388: ppss/job_log/http:hello.wordpress.com: Invalid argument /usr/bin/ppss.sh: line 1389: ppss/job_log/http:hello.wordpress.com: Invalid argument /usr/bin/ppss.sh: line 1390: ppss/job_log/http:hello.wordpress.com: Invalid argument /usr/bin/ppss.sh: line 1391: ppss/job_log/http:hello.wordpress.com: Invalid argument /usr/bin/ppss.sh: line 1392: ppss/job_log/http:hello.wordpress.com: Invalid argument /usr/bin/ppss.sh: line 1405: ppss/job_log/http:hello.wordpress.com: Invalid argument /usr/bin/ppss.sh: line 1410: ppss/job_log/http:hello.wordpress.com: Invalid argument /usr/bin/ppss.sh: line 1415: ppss/job_log/http:hello.wordpress.com: Invalid argument mkdir: no s’ha pogut crear el directori «/http:»: Permission denied /usr/bin/ppss.sh: line 1443: ppss/job_log/http:hello.wordpress.com: Invalid argument /usr/bin/ppss.sh: line 1444: ppss/job_log/http:hello.wordpress.com: Invalid argument nov 17 11:56:32: 1 job is remaining. nov 17 11:56:32: Finished. Consult ./ppss/job_log for job output. What version of the product are you using? On what operating system? Ubuntu 9.10, last version of ppss.sh Please provide any additional information below. The problem is that ppss.sh removes / and \ but does not replaces : which are not valid filenames. I could fix most of the errors, but I could not fix the mkdir issue: 1372c1372 < LOG_FILE_NAME=`echo "$ITEM" \| sed s/^\\\.//g \| sed s/^\\\.\\\.//g \| sed s/\\\///g \| sed s@:@@g` --- > LOG_FILE_NAME=`echo "$ITEM" \| sed s/^\\\.//g \| sed s/^\\\.\\\.//g \| sed s/\\\///g` This removes ":" from the name, but it will still fail while creating the directory (mkdir) What steps will reproduce the problem? >echo "http://hello.wordpress.com" > temp.txt >ppss.sh -c 'echo ' -f temp.txt What is the expected output? What do you see instead? This is what I see: nov 17 11:56:31: ========================================================= nov 17 11:56:31: \|P\|P\|S\|S\| nov 17 11:56:31: Distributed Parallel Processing Shell Script version 2.41 nov 17 11:56:31: ========================================================= nov 17 11:56:31: Hostname: i7-desktop nov 17 11:56:31: --------------------------------------------------------- nov 17 11:56:31: CPU: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz nov 17 11:56:31: Found 8 logic processors. nov 17 11:56:31: Starting 8 parallel workers. nov 17 11:56:31: --------------------------------------------------------- nov 17 11:56:31: Currently 100 percent complete. Processed 1 of 1 items. /usr/bin/ppss.sh: line 1387: ppss/job_log/http:hello.wordpress.com: Invalid argument /usr/bin/ppss.sh: line 1388: ppss/job_log/http:hello.wordpress.com: Invalid argument /usr/bin/ppss.sh: line 1389: ppss/job_log/http:hello.wordpress.com: Invalid argument /usr/bin/ppss.sh: line 1390: ppss/job_log/http:hello.wordpress.com: Invalid argument /usr/bin/ppss.sh: line 1391: ppss/job_log/http:hello.wordpress.com: Invalid argument /usr/bin/ppss.sh: line 1392: ppss/job_log/http:hello.wordpress.com: Invalid argument /usr/bin/ppss.sh: line 1405: ppss/job_log/http:hello.wordpress.com: Invalid argument /usr/bin/ppss.sh: line 1410: ppss/job_log/http:hello.wordpress.com: Invalid argument /usr/bin/ppss.sh: line 1415: ppss/job_log/http:hello.wordpress.com: Invalid argument mkdir: no s’ha pogut crear el directori «/http:»: Permission denied /usr/bin/ppss.sh: line 1443: ppss/job_log/http:hello.wordpress.com: Invalid argument /usr/bin/ppss.sh: line 1444: ppss/job_log/http:hello.wordpress.com: Invalid argument nov 17 11:56:32: 1 job is remaining. nov 17 11:56:32: Finished. Consult ./ppss/job_log for job output. What version of the product are you using? On what operating system? Ubuntu 9.10, last version of ppss.sh Please provide any additional information below. The problem is that ppss.sh removes / and \ but does not replaces : which are not valid filenames. I could fix most of the errors, but I could not fix the mkdir issue: 1372c1372 < LOG_FILE_NAME=`echo "$ITEM" \| sed s/^\\\.//g \| sed s/^\\\.\\\.//g \| sed s/\\\///g \| sed s@:@@g` --- > LOG_FILE_NAME=`echo "$ITEM" \| sed s/^\\\.//g \| sed s/^\\\.\\\.//g \| sed s/\\\///g` This removes ":" from the name, but it will still fail while creating the directory (mkdir) Nov 14, 2009 issue 16 (ppss.sh does not work in a FAT32 filesystem) reported by Roger.Sindreu - Hi, I was trying to run ppss.sh from a FAT32 filesystem, but since it uses mkfifo on the local directory it does not work. ras@desktop:/media/fat32_disk$ mkfifo jsdf mkfifo: no s’ha pogut crear la cua FIFO «jsdf»: Operation not permitted That was causing ppss.sh to not work, since it tries to do mkfifo in the local directory (which was FAT32). I made this change to make it work, since /tmp will probably not be in a FAT32 (although not always) 591c591 < FIFO=$PPSS_DIR/fifo-$RANDOM-$RANDOM --- > FIFO=/tmp/fifo-$RANDOM-$RANDOM What steps will reproduce the problem? 1. 2. 3. What is the expected output? What do you see instead? What version of the product are you using? On what operating system? Please provide any additional information below. Hi, I was trying to run ppss.sh from a FAT32 filesystem, but since it uses mkfifo on the local directory it does not work. ras@desktop:/media/fat32_disk$ mkfifo jsdf mkfifo: no s’ha pogut crear la cua FIFO «jsdf»: Operation not permitted That was causing ppss.sh to not work, since it tries to do mkfifo in the local directory (which was FAT32). I made this change to make it work, since /tmp will probably not be in a FAT32 (although not always) 591c591 < FIFO=$PPSS_DIR/fifo-$RANDOM-$RANDOM --- > FIFO=/tmp/fifo-$RANDOM-$RANDOM What steps will reproduce the problem? 1. 2. 3. What is the expected output? What do you see instead? What version of the product are you using? On what operating system? Please provide any additional information below.

issue 18 (If multiple tasks are the same it runs only once) reported by sarkarprateek - What steps will reproduce the problem? 1. Create a joblist file, by repeating a command many times. [psarkar@teentaal ppssTrial]$ head -5 joblist.txt /tmp/ppssTrial/largeLoop.sh /tmp/ppssTrial/largeLoop.sh /tmp/ppssTrial/largeLoop.sh /tmp/ppssTrial/largeLoop.sh /tmp/ppssTrial/largeLoop.sh [psarkar@teentaal ppssTrial]$ wc -l joblist.txt 170 joblist.txt [psarkar@teentaal ppssTrial]$ 2. Here is what my largeLoop.sh looks like. The thing to observe is that every time it runs, it should produce a different output. [psarkar@teentaal ppssTrial]$ cat largeLoop.sh #!/bin/bash N=0 PN=$BASHPID while (( $N < 1000 )); do N=$(( $N + 1 )); echo ${PN}_${N}; sleep 1; done [psarkar@teentaal ppssTrial]$ What is the expected output? What do you see instead? I would have expected that all processes would run (i.e., the program would run once for each time it is listed in the joblist file). Instead it runs only once. [psarkar@teentaal ppssTrial]$ \rm -r ppss/ [psarkar@teentaal ppssTrial]$ ppss.sh -f joblist.txt -c 'sh ' Dec 09 09:55:14: ========================================================= Dec 09 09:55:14: |P|P|S|S| Dec 09 09:55:14: Distributed Parallel Processing Shell Script version 2.41 Dec 09 09:55:14: ========================================================= Dec 09 09:55:14: Hostname: teentaal Dec 09 09:55:14: --------------------------------------------------------- Dec 09 09:55:14: CPU: Intel(R) Xeon(TM) CPU 3.20GHz Dec 09 09:55:14: Found 8 logic processors. Dec 09 09:55:14: Starting 8 parallel workers. Dec 09 09:55:14: --------------------------------------------------------- Dec 09 09:55:21: Currently 100 percent complete. Processed 170 of 170 items. Dec 09 09:55:24: 1 job is remaining. ^C[psarkar@teentaal ppssTrial]$ Dec 09 09:55:34: Finished. Consult ./ppss/job_log for job output. [psarkar@teentaal ppssTrial]$ more ppss/job_log/tmpppssTriallargeLoop.sh ===== PPSS Item Log File ===== Host: teentaal Process:14109 Item: /tmp/ppssTrial/largeLoop.sh Start date: Dec 09 09:55:15 14225_1 14225_2 14225_3 14225_4 14225_5 14225_6 14225_7 14225_8 14225_9 14225_10 14225_11 14225_12 14225_13 14225_14 14225_15 14225_16 14225_17 14225_18 14225_19 14225_20 14225_21 14225_22 14225_23 14225_24 14225_25 14225_26 14225_27 14225_28 14225_29 What version of the product are you using? On what operating system? |P|P|S|S| Distributed Parallel Processing Shell Script 2.41 Please provide any additional information below. There are a number of different programs that will produce different output each time it is run. E.g., optimizers with random initialization. It would be good if these can be handled. Perhaps the process ID or the command sequence number can serve as a hash/descriptor -- rather than the altered command line. This may also alleviate some of the other issues I see (e.g. regarding ':' or special chars in commands.)

What steps will reproduce the problem? 1. Create a joblist file, by repeating a command many times. [psarkar@teentaal ppssTrial]$ head -5 joblist.txt /tmp/ppssTrial/largeLoop.sh /tmp/ppssTrial/largeLoop.sh /tmp/ppssTrial/largeLoop.sh /tmp/ppssTrial/largeLoop.sh /tmp/ppssTrial/largeLoop.sh [psarkar@teentaal ppssTrial]$ wc -l joblist.txt 170 joblist.txt [psarkar@teentaal ppssTrial]$ 2. Here is what my largeLoop.sh looks like. The thing to observe is that every time it runs, it should produce a different output. [psarkar@teentaal ppssTrial]$ cat largeLoop.sh #!/bin/bash N=0 PN=$BASHPID while (( $N < 1000 )); do N=$(( $N + 1 )); echo ${PN}_${N}; sleep 1; done [psarkar@teentaal ppssTrial]$ What is the expected output? What do you see instead? I would have expected that all processes would run (i.e., the program would run once for each time it is listed in the joblist file). Instead it runs only once. [psarkar@teentaal ppssTrial]$ \rm -r ppss/ [psarkar@teentaal ppssTrial]$ ppss.sh -f joblist.txt -c 'sh ' Dec 09 09:55:14: ========================================================= Dec 09 09:55:14: |P|P|S|S| Dec 09 09:55:14: Distributed Parallel Processing Shell Script version 2.41 Dec 09 09:55:14: ========================================================= Dec 09 09:55:14: Hostname: teentaal Dec 09 09:55:14: --------------------------------------------------------- Dec 09 09:55:14: CPU: Intel(R) Xeon(TM) CPU 3.20GHz Dec 09 09:55:14: Found 8 logic processors. Dec 09 09:55:14: Starting 8 parallel workers. Dec 09 09:55:14: --------------------------------------------------------- Dec 09 09:55:21: Currently 100 percent complete. Processed 170 of 170 items. Dec 09 09:55:24: 1 job is remaining. ^C[psarkar@teentaal ppssTrial]$ Dec 09 09:55:34: Finished. Consult ./ppss/job_log for job output. [psarkar@teentaal ppssTrial]$ more ppss/job_log/tmpppssTriallargeLoop.sh ===== PPSS Item Log File ===== Host: teentaal Process:14109 Item: /tmp/ppssTrial/largeLoop.sh Start date: Dec 09 09:55:15 14225_1 14225_2 14225_3 14225_4 14225_5 14225_6 14225_7 14225_8 14225_9 14225_10 14225_11 14225_12 14225_13 14225_14 14225_15 14225_16 14225_17 14225_18 14225_19 14225_20 14225_21 14225_22 14225_23 14225_24 14225_25 14225_26 14225_27 14225_28 14225_29 What version of the product are you using? On what operating system? |P|P|S|S| Distributed Parallel Processing Shell Script 2.41 Please provide any additional information below. There are a number of different programs that will produce different output each time it is run. E.g., optimizers with random initialization. It would be good if these can be handled. Perhaps the process ID or the command sequence number can serve as a hash/descriptor -- rather than the altered command line. This may also alleviate some of the other issues I see (e.g. regarding ':' or special chars in commands.)

Last 30 days

Older