Last 30 days

• Dec 08, 2009

  issue 11 (Add Mock objects to unit tests to test individual test metho...) reported by ryan.justin.smith   -   Recently, a bug was found b/c the writer failed to close the output stream properly. More unit tests are needed and mock objects should be used.

  Recently, a bug was found b/c the writer failed to close the output stream properly. More unit tests are needed and mock objects should be used.
• Nov 23, 2009

  issue 10 (HBase Writer isnt closing Reply InputStream objects after th...) reported by ryan.justin.smith   -   Running jmap on the heritrix crawler using hbase-writer will show a growing number of ReplayInputStream objects over time. Manually forcing the close of this connection fixes the problem. This problem was discovered and patched by Joost Ouwerkerk.

  Running jmap on the heritrix crawler using hbase-writer will show a growing number of ReplayInputStream objects over time. Manually forcing the close of this connection fixes the problem. This problem was discovered and patched by Joost Ouwerkerk.
• Nov 21, 2009

  r161 (updating comments) committed by ryan.justin.smith   -   updating comments

  updating comments

Earlier this year

• Oct 25, 2009

  r160 (removing "ping back" from the README since donated time has ...) committed by ryan.justin.smith   -   removing "ping back" from the README since donated time has been unpaid for by Questio. Adding bug fix to remove trailing spaces from zkQuorum, which was effecting name lookups.

  removing "ping back" from the README since donated time has been unpaid for by Questio. Adding bug fix to remove trailing spaces from zkQuorum, which was effecting name lookups.
• Oct 25, 2009

  README Wiki page edited by ryan.justin.smith   -   Revision r159 Edited wiki page through web user interface. Removed "ping back" until questio pays for the time I have used to create and maintain this project. So far, all work done on this project has been done for free by time donated by myself, Ryan Smith.

  Revision r159 Edited wiki page through web user interface. Removed "ping back" until questio pays for the time I have used to create and maintain this project. So far, all work done on this project has been done for free by time donated by myself, Ryan Smith.
• Oct 22, 2009

  README Wiki page commented on by bizicafe   -   =Hello world!=

  =Hello world!=
• Oct 19, 2009

  r158 (adding comments) committed by ryan.justin.smith   -   adding comments

  adding comments
• Oct 18, 2009

  hbase-writer-0.20.3.jar (hbase-writer-0.20.3-RELEASE) file uploaded by ryan.justin.smith   -  

  Labels: Featured Type-Executable OpSys-All

  Labels: Featured Type-Executable OpSys-All
• Oct 18, 2009

  r157 (moving trunk to 0.20.4-SNAPSHOT after release) committed by ryan.justin.smith   -   moving trunk to 0.20.4-SNAPSHOT after release

  moving trunk to 0.20.4-SNAPSHOT after release
• Oct 18, 2009

  r156 (committing 0.20.3) committed by ryan.justin.smith   -   committing 0.20.3

  committing 0.20.3
• Oct 18, 2009

  r155 (committing 0.20.3 - adding bug fix for issue where replayInp...) committed by ryan.justin.smith   -   committing 0.20.3 - adding bug fix for issue where replayInputStream was not being closed properly resulting in a steady grow of open file handles.

  committing 0.20.3 - adding bug fix for issue where replayInputStream was not being closed properly resulting in a steady grow of open file handles.
• Oct 16, 2009

  r154 (updating comments with jautodoc.) committed by ryan.justin.smith   -   updating comments with jautodoc.

  updating comments with jautodoc.
• Oct 16, 2009

  r153 (fetching the replayInputStream from the recording object for...) committed by ryan.justin.smith   -   fetching the replayInputStream from the recording object for optional processor content usage.

  fetching the replayInputStream from the recording object for optional processor content usage.
• Oct 16, 2009

  r152 (adding patch submitted by Joost Ouwerkerk to fix the replyIn...) committed by ryan.justin.smith   -   adding patch submitted by Joost Ouwerkerk to fix the replyInputStream object not closing for the content processor. The stream is now wrapped in a try/finally where the stream is manually closed.

  adding patch submitted by Joost Ouwerkerk to fix the replyInputStream object not closing for the content processor. The stream is now wrapped in a try/finally where the stream is manually closed.
• Oct 15, 2009

  r151 (improving test comments.) committed by ryan.justin.smith   -   improving test comments.

  improving test comments.
• Oct 15, 2009

  r150 (correcting errors found by the 'find bugs' maven reporting p...) committed by ryan.justin.smith   -   correcting errors found by the 'find bugs' maven reporting plugin.

  correcting errors found by the 'find bugs' maven reporting plugin.
• Oct 15, 2009

  r149 (adding comments for zkClientPort) committed by ryan.justin.smith   -   adding comments for zkClientPort

  adding comments for zkClientPort
• Oct 15, 2009

  r148 (clearer variable definitions, fixed redundant logging statem...) committed by ryan.justin.smith   -   clearer variable definitions, fixed redundant logging statement, changed email to project email.

  clearer variable definitions, fixed redundant logging statement, changed email to project email.
• Oct 14, 2009

  r147 (moving trunk to 0.20.3-SNAPSHOT version) committed by ryan.justin.smith   -   moving trunk to 0.20.3-SNAPSHOT version

  moving trunk to 0.20.3-SNAPSHOT version
• Oct 14, 2009

  hbase-writer-0.20.2.jar (hbase-writer-0.20.2-RELEASE) file uploaded by ryan.justin.smith   -  

  Labels: Featured Type-Executable OpSys-All

  Labels: Featured Type-Executable OpSys-All
• Oct 14, 2009

  r146 (tagging version 0.20.2 for release) committed by ryan.justin.smith   -   tagging version 0.20.2 for release

  tagging version 0.20.2 for release
• Oct 14, 2009

  r145 (,pvomg trunk to 0.20.2 after adding support for zkClientPort) committed by ryan.justin.smith   -   ,pvomg trunk to 0.20.2 after adding support for zkClientPort

  ,pvomg trunk to 0.20.2 after adding support for zkClientPort
• Oct 14, 2009

  r144 (adding zk client port to processor description file.) committed by ryan.justin.smith   -   adding zk client port to processor description file.

  adding zk client port to processor description file.
• Oct 14, 2009

  r143 (adding zk client port to README) committed by ryan.justin.smith   -   adding zk client port to README

  adding zk client port to README
• Oct 14, 2009

  r142 (fixing support for alternate zk client port.) committed by ryan.justin.smith   -   fixing support for alternate zk client port.

  fixing support for alternate zk client port.
• Oct 14, 2009

  README Wiki page edited by ryan.justin.smith   -   Revision r141 Edited wiki page through web user interface.

  Revision r141 Edited wiki page through web user interface.
• Oct 14, 2009

  r140 (adding support for the zk client port to be set to something...) committed by ryan.justin.smith   -   adding support for the zk client port to be set to something other than 2181

  adding support for the zk client port to be set to something other than 2181
• Oct 14, 2009

  hbase-writer-0.20.1.jar (hbase-writer-0.20.1-RELEASE) file uploaded by ryan.justin.smith   -  

  Labels: Featured OpSys-All

  Labels: Featured OpSys-All
• Oct 14, 2009

  r139 (committing trunk to 0.20.2-SNAPSHOT) committed by ryan.justin.smith   -   committing trunk to 0.20.2-SNAPSHOT

  committing trunk to 0.20.2-SNAPSHOT
• Oct 14, 2009

  r138 (tagging 0.20.1 release) committed by ryan.justin.smith   -   tagging 0.20.1 release

  tagging 0.20.1 release
• Oct 14, 2009

  r137 (committing version 0.20.1 for release) committed by ryan.justin.smith   -   committing version 0.20.1 for release

  committing version 0.20.1 for release
• Oct 14, 2009

  r136 (adding better comments, finalizing release for 0.20.1) committed by ryan.justin.smith   -   adding better comments, finalizing release for 0.20.1

  adding better comments, finalizing release for 0.20.1
• Oct 13, 2009

  r135 (underscores dont work as characters in variable names for he...) committed by ryan.justin.smith   -   underscores dont work as characters in variable names for heritrix.

  underscores dont work as characters in variable names for heritrix.
• Oct 13, 2009

  r134 (configuring a zk quorum comma seperated string instead of hb...) committed by ryan.justin.smith   -   configuring a zk quorum comma seperated string instead of hbase master host.

  configuring a zk quorum comma seperated string instead of hbase master host.
• Oct 09, 2009

  r133 (removing duplicate key addition which was preventing the Wri...) committed by ryan.justin.smith   -   removing duplicate key addition which was preventing the Writer Processor class to load in heritrix.

  removing duplicate key addition which was preventing the Writer Processor class to load in heritrix.
• Oct 07, 2009

  r132 (better comments for dependencies in pom.xml) committed by ryan.justin.smith   -   better comments for dependencies in pom.xml

  better comments for dependencies in pom.xml
• Oct 07, 2009

  r131 (using Put object instead of BatchUpdate for writing crawled ...) committed by ryan.justin.smith   -   using Put object instead of BatchUpdate for writing crawled records to an HBase table.

  using Put object instead of BatchUpdate for writing crawled records to an HBase table.
• Oct 07, 2009

  r130 (using htable.get(Get(rowKey)) instead of htable.getRow(rowKe...) committed by ryan.justin.smith   -   using htable.get(Get(rowKey)) instead of htable.getRow(rowKey)

  using htable.get(Get(rowKey)) instead of htable.getRow(rowKey)
• Oct 07, 2009

  r129 (changing project version of trunk to 0.20.0-SNAPSHOT) committed by ryan.justin.smith   -   changing project version of trunk to 0.20.0-SNAPSHOT

  changing project version of trunk to 0.20.0-SNAPSHOT
• Oct 07, 2009

  r128 (updating trunk to be compiled up 0.20.0 - batchUpdate object...) committed by ryan.justin.smith   -   updating trunk to be compiled up 0.20.0 - batchUpdate object is deprecated and needs to be replaced by using the new data objects.

  updating trunk to be compiled up 0.20.0 - batchUpdate object is deprecated and needs to be replaced by using the new data objects.
• Oct 07, 2009

  r127 (branching trunk to 19 branch) committed by ryan.justin.smith   -   branching trunk to 19 branch

  branching trunk to 19 branch
• Oct 07, 2009

  r126 (getting ready for branch.) committed by ryan.justin.smith   -   getting ready for branch.

  getting ready for branch.
• Oct 07, 2009

  r125 (Adding more comments and adding logic to modify existing tab...) committed by ryan.justin.smith   -   Adding more comments and adding logic to modify existing tables if they are missing the necessary column families when starting a crawl.

  Adding more comments and adding logic to modify existing tables if they are missing the necessary column families when starting a crawl.
• Oct 07, 2009

  r124 (additional comments and re-organizing code to be easier to r...) committed by ryan.justin.smith   -   additional comments and re-organizing code to be easier to read.

  additional comments and re-organizing code to be easier to read.
• Oct 07, 2009

  r123 (additional comments and re-organizing code to be easier to r...) committed by ryan.justin.smith   -   additional comments and re-organizing code to be easier to read.

  additional comments and re-organizing code to be easier to read.
• Oct 06, 2009

  r122 (Improved comments, changed back the location of setTimestamp...) committed by ryan.justin.smith   -   Improved comments, changed back the location of setTimestamp() to record a more accurate time.

  Improved comments, changed back the location of setTimestamp() to record a more accurate time.
• Feb 17, 2009

  r121 (ran jautodoc on the project to flesh out missing method comm...) committed by ryan.justin.smith   -   ran jautodoc on the project to flesh out missing method comments

  ran jautodoc on the project to flesh out missing method comments
• Feb 17, 2009

  r120 (adding new feature, process_only_new_records.) committed by ryan.justin.smith   -   adding new feature, process_only_new_records.

  adding new feature, process_only_new_records.
• Feb 16, 2009

  README Wiki page edited by ryan.justin.smith
• Feb 16, 2009

  issue 9 ('only_new_records' doesnt write any new records to HBase whe...) Status changed by ryan.justin.smith   -   This has been tested with the logic now residing in shouldWrite() in HbaseWriterProcessor.java If you crawl a brand new site with "only_new_records" set to "true" , it downloads all urls configured to get by heritrix. If you run this exact same heritrix jo configuration a 2nd time, no new records will be downloaded or written to hbase.

  Status: Fixed

  This has been tested with the logic now residing in shouldWrite() in HbaseWriterProcessor.java If you crawl a brand new site with "only_new_records" set to "true" , it downloads all urls configured to get by heritrix. If you run this exact same heritrix jo configuration a 2nd time, no new records will be downloaded or written to hbase.

  Status: Fixed