warc-tools - Project Hosting on Google Code

Earlier this year Oct 20, 2009 warc-tools_phase_III_frs_v8.pdf (WARC Tools Phase III FRS) file uploaded by mark.middleton - Labels: Featured Type-Docs Labels: Featured Type-Docs Jun 23, 2009 r245 (fixed annoying bug that created tmp files in / causing permi...) committed by junklight - fixed annoying bug that created tmp files in / causing permissions issues fixed annoying bug that created tmp files in / causing permissions issues Jun 16, 2009 r244 (Add more "fflush" after "ftruncate") committed by voidptrptr - Add more "fflush" after "ftruncate" Add more "fflush" after "ftruncate" Jun 16, 2009 r243 (Fix fftruncate bug which appears on Linux machine only.) committed by voidptrptr - Fix fftruncate bug which appears on Linux machine only. Fix fftruncate bug which appears on Linux machine only. Apr 29, 2009 issue 102 (How to checkout the source code?) reported by yangqing...@bit.edu.cn - Thanks the great work. I need to index the warc dataset of the trec2009. How can I checkout the source code to index the dataset?It seems that subverson can not find the archives. Who can help me? Thanks the great work. I need to index the warc dataset of the trec2009. How can I checkout the source code to index the dataset?It seems that subverson can not find the archives. Who can help me? Apr 23, 2009 issue 91 (SRS 84 — Libwarc shall be made available as a binary for at ...) Owner changed by gordon.paynter - Not done, see SRS 82 — Libwarc source code shall be released in the following archives... Owner: voidptrptr Not done, see SRS 82 — Libwarc source code shall be released in the following archives... Owner: voidptrptr Apr 23, 2009 issue 90 (SRS 83 — Libwarc binary modules, ready to use binary command...) Owner changed by gordon.paynter - Not done, see SRS 82 — Libwarc source code shall be released in the following archives... Owner: voidptrptr Not done, see SRS 82 — Libwarc source code shall be released in the following archives... Owner: voidptrptr Apr 23, 2009 issue 89 (SRS 82 — Libwarc source code shall be released in the follow...) Owner changed by gordon.paynter - Not done, and IMO the main outstanding deliverable. Will Hanzo be delivering this feature? When? Owner: voidptrptr Not done, and IMO the main outstanding deliverable. Will Hanzo be delivering this feature? When? Owner: voidptrptr Apr 23, 2009 issue 87 (SRS 80 — To ensure code portability on older computer archit...) Status changed by gordon.paynter - Presumed done. Status: Done Presumed done. Status: Done Apr 23, 2009 issue 71 (SRS 64 — Libwarc and the bindings to its functionality shall...) Status changed by gordon.paynter - Unconfirmed. At the end of the day this is a (good) style issue though, so I am going to close it. Status: WontFix Unconfirmed. At the end of the day this is a (good) style issue though, so I am going to close it. Status: WontFix Apr 23, 2009 issue 63 (SRS 57 — WARC files in various test-states shall be provided...) Status changed by gordon.paynter - I used the files supplied with WACRC tools to test jhove, but then since all the code is provided together, I think I can confirm this one in good faith. Status: Verified I used the files supplied with WACRC tools to test jhove, but then since all the code is provided together, I think I can confirm this one in good faith. Status: Verified Apr 23, 2009 issue 62 (SRS 56 — WarcMdoule and WarcHandler plugin modules shall be ...) Status changed by gordon.paynter - Presumed implemented -- I have compiled and run the JHOVE validation code and verified it works. Status: Verified Presumed implemented -- I have compiled and run the JHOVE validation code and verified it works. Status: Verified Apr 23, 2009 issue 49 (SRS 43 — "arc2warc" shall have make a default operation in c...) commented on by gordon.paynter - You can declare an arbitrary field type ("future-type") for both arc AND warc. So it could be invalid on the grounds that you could just copy it over, but this doesn't make much sense. So I think this is a valid requirement. Younes -- has this requirement been implemented? You can declare an arbitrary field type ("future-type") for both arc AND warc. So it could be invalid on the grounds that you could just copy it over, but this doesn't make much sense. So I think this is a valid requirement. Younes -- has this requirement been implemented? Apr 23, 2009 issue 39 (SRS 33 — Libwarc shall provide a set of classes to enable re...) Status changed by gordon.paynter - Unconfirmed. Status: WontFix Unconfirmed. Status: WontFix Apr 23, 2009 issue 35 (SRS 30 — It shall not be possible to use more than one compr...) Status changed by gordon.paynter - Apparently done (but not confirmed). Status: Done Apparently done (but not confirmed). Status: Done Apr 23, 2009 issue 34 (SRS 29 — Libwarc shall support multiple compression schemas,...) Status changed by gordon.paynter - Unconfirmed. Status: WontFix Unconfirmed. Status: WontFix Apr 23, 2009 issue 25 (SRS 20 — Libwarc’s iterators may be combined into composite ...) Status changed by gordon.paynter - Unconfirmed. Status: WontFix Unconfirmed. Status: WontFix Apr 23, 2009 issue 24 (SRS 19 — Libwarc’s generic iterators may be customised for d...) Status changed by gordon.paynter - Unconfirmed. Status: WontFix Unconfirmed. Status: WontFix Apr 23, 2009 issue 23 (SRS 18 — Libwarc shall provide a WARC-record-type iterator) Status changed by gordon.paynter - Apparently done (but not confirmed). Status: Done Apparently done (but not confirmed). Status: Done Apr 23, 2009 issue 22 (SRS 17 — Libwarc shall provide a WARC-record MIME-type itera...) Status changed by gordon.paynter - Apparentyl done. Status: Done Apparentyl done. Status: Done Apr 23, 2009 issue 21 (SRS 16 — Libwarc shall provide a generic iterator, to enable...) Status changed by gordon.paynter - Apparently done (based on other implementations). Status: Done Apparently done (based on other implementations). Status: Done Apr 23, 2009 issue 20 (SRS 15 —Libwarc shall include ADT objects to handle read ope...) Status changed by gordon.paynter - Presumed implemented. Status: Done Presumed implemented. Status: Done Apr 20, 2009 warc-tools_phase_III_frs_v5.pdf (WARC Tools Phase III FRS) file uploaded by mark.middleton - Labels: frs warc warc-tools hanzo Labels: frs warc warc-tools hanzo Mar 04, 2009 issue 101 (The C enumeration types should be available when using the j...) reported by SoerenVCarlsen - The enumeration types in the C header files (e.g. wrectype.h) should be available in the java interface. Many of the java wrapper methods have int arguments, that relate to a certain C enum, e.g. the WRecord_setRecordType method in the wrecord class. The enumeration types in the C header files (e.g. wrectype.h) should be available in the java interface. Many of the java wrapper methods have int arguments, that relate to a certain C enum, e.g. the WRecord_setRecordType method in the wrecord class. Mar 04, 2009 issue 100 (Missing documentation on how to use java wrapper) reported by SoerenVCarlsen - There is not much documentation on how to use the Java wrapper. At the moment, you can only use the wrapper, if you know how it is implemented. What version of the product are you using? revision 142 Javadoc for the java wrapper classes would be sufficient There is not much documentation on how to use the Java wrapper. At the moment, you can only use the wrapper, if you know how it is implemented. What version of the product are you using? revision 142 Javadoc for the java wrapper classes would be sufficient Jan 12, 2009 issue 85 (SRS 78 — Libwarc and associated tools shall be shipped with ...) Status changed by bjarne.andersen - Verified only with Apache. But documentation for Lighttp is available and building the Lighttp module works for me. Closing this Status: Verified Verified only with Apache. But documentation for Lighttp is available and building the Lighttp module works for me. Closing this Status: Verified Jan 12, 2009 issue 63 (SRS 57 — WARC files in various test-states shall be provided...) Owner changed by bjarne.andersen - Gordon. Did you use any supplied WARC files so that we can close this one ? Owner: gordon.paynter Gordon. Did you use any supplied WARC files so that we can close this one ? Owner: gordon.paynter Jan 12, 2009 issue 61 (SRS 55 — The WARC validator tool specified in SRS 31-32 shal...) Status changed by bjarne.andersen - Agree - marking as Done Status: Verified Agree - marking as Done Status: Verified Jan 12, 2009 issue 62 (SRS 56 — WarcMdoule and WarcHandler plugin modules shall be ...) Owner changed by bjarne.andersen - Owner: gordon.paynter Owner: gordon.paynter Jan 12, 2009 issue 46 (SRS 40 — Libwarc shall be incorporated within a Lighttp modu...) Status changed by bjarne.andersen - Only tested with Apache - but bulilding the Lighttp module seems to work and the documentation says how to install - so I'm convinced that things work with Lighttp as well as with Apache Status: Verified Only tested with Apache - but bulilding the Lighttp module seems to work and the documentation says how to install - so I'm convinced that things work with Lighttp as well as with Apache Status: Verified Jan 12, 2009 issue 44 (SRS 39 — Libwarc shall be incorporated within an Apache modu...) Status changed by bjarne.andersen - Status: Verified Status: Verified Jan 12, 2009 issue 40 (SRS 34 — It shall be possible to perform read operations (re...) Status changed by bjarne.andersen - Status: Verified Status: Verified Jan 11, 2009 issue 69 (SRS 62 — An independent Java implementation of libwarc may b...) Status changed by gordon.paynter - I assume we are not doing this, it is only a "may" requirement anyway. Status: WontFix I assume we are not doing this, it is only a "may" requirement anyway. Status: WontFix Jan 11, 2009 issue 61 (SRS 55 — The WARC validator tool specified in SRS 31-32 shal...) Owner changed by gordon.paynter - WARC Tools now includes contrib/java and contrib/jhove. Today I compiled both of these up, then followed the instructions in contrib/jhove/install for downloading jhove and patching from the warc-tools directory to build a version of Jhove with WARC support. I used the command-line tool to validate a WARC file (and to show several other files were or were ot WARC files). Anyway, while we do not have a WARC module "in jhove" we do have a way to compile WARC support into Jhove 1.1, which is probably as good as we can hope to get at this point. I suggest we declare this done. Gordon PS: I am not a java developer, so had a few problems getting this all to run. For example, jhove would not compile out of the box due to a non-warc-tools related compile error, which I solved by commenting out the "pop" method that caused the error. Owner: bjarne.andersen WARC Tools now includes contrib/java and contrib/jhove. Today I compiled both of these up, then followed the instructions in contrib/jhove/install for downloading jhove and patching from the warc-tools directory to build a version of Jhove with WARC support. I used the command-line tool to validate a WARC file (and to show several other files were or were ot WARC files). Anyway, while we do not have a WARC module "in jhove" we do have a way to compile WARC support into Jhove 1.1, which is probably as good as we can hope to get at this point. I suggest we declare this done. Gordon PS: I am not a java developer, so had a few problems getting this all to run. For example, jhove would not compile out of the box due to a non-warc-tools related compile error, which I solved by commenting out the "pop" method that caused the error. Owner: bjarne.andersen Jan 11, 2009 issue 67 (SRS 61 — A Java interface to libwarc shall be implemented us...) Status changed by gordon.paynter - I have successfully built warctools.jar and run the warctools.Main java file on Ubuntu. Only question is that the install file refers to "contrib/java/jar/warctools-with-jna.jar" which perhaps should read "contrib/java/jar/warctools-with-main.jar" No further testing attempted (or planned) at this point. Status: Done I have successfully built warctools.jar and run the warctools.Main java file on Ubuntu. Only question is that the install file refers to "contrib/java/jar/warctools-with-jna.jar" which perhaps should read "contrib/java/jar/warctools-with-main.jar" No further testing attempted (or planned) at this point. Status: Done Jan 11, 2009 issue 16 (SRS 11 — Libwarc shall provide an API describing (1) the set...) changed by gordon.paynter - Status: Done Owner: gordon.paynter Status: Done Owner: gordon.paynter Jan 08, 2009 issue 67 (SRS 61 — A Java interface to libwarc shall be implemented us...) commented on by gordon.paynter - Yes, it should! This issue (and all the other ones in the tracker that are flagged as Type-SRS) is a requirement from the original Hanzo WARC-Tools software requirements specification. I (and a few colleagues) am supposed to be marking the SRS requirements as closedas we confirm each feature has been completed. Unfortunately, I am a bit slow, so I have not yet confirmed that the Java implementation works. If you have used it, and can confirm it works for me, then I'll be very happy to close the issue. Thanks, Gordon Yes, it should! This issue (and all the other ones in the tracker that are flagged as Type-SRS) is a requirement from the original Hanzo WARC-Tools software requirements specification. I (and a few colleagues) am supposed to be marking the SRS requirements as closedas we confirm each feature has been completed. Unfortunately, I am a bit slow, so I have not yet confirmed that the Java implementation works. If you have used it, and can confirm it works for me, then I'll be very happy to close the issue. Thanks, Gordon Jan 07, 2009 r242 (Fix Ruby shared library name detection. Thanks to Erik Hetzn...) committed by voidptrptr - Fix Ruby shared library name detection. Thanks to Erik Hetzner Fix Ruby shared library name detection. Thanks to Erik Hetzner Jan 07, 2009 issue 67 (SRS 61 — A Java interface to libwarc shall be implemented us...) commented on by SoerenVCarlsen - I have just noticed, that a Java wrapper interface for warctools Java wrapper using JNA https://jna.dev.java.net/ now exists in the contrib directory. Doesn't this close this issue? I have just noticed, that a Java wrapper interface for warctools Java wrapper using JNA https://jna.dev.java.net/ now exists in the contrib directory. Doesn't this close this issue? Older Dec 05, 2008 r241 (Dix typo in misc/afsmhdl.c) committed by voidptrptr - Dix typo in misc/afsmhdl.c Dix typo in misc/afsmhdl.c Dec 05, 2008 r240 (Remove some useless debug information) committed by voidptrptr - Remove some useless debug information Remove some useless debug information Dec 05, 2008 r239 (Remove some useless debug information) committed by voidptrptr - Remove some useless debug information Remove some useless debug information Dec 05, 2008 r238 (Improve misc/afsmhdl.c to accept malformed ARC file with mis...) committed by voidptrptr - Improve misc/afsmhdl.c to accept malformed ARC file with missing MIME TYPE Improve misc/afsmhdl.c to accept malformed ARC file with missing MIME TYPE Dec 04, 2008 r237 (Add python/arc2warc.sh to wrap the python/arc2warc.py comman...) committed by voidptrptr - Add python/arc2warc.sh to wrap the python/arc2warc.py command Add python/arc2warc.sh to wrap the python/arc2warc.py command Nov 30, 2008 r236 (Add afsmhdl.c in misc directory to tolerate ARC file with no...) committed by voidptrptr - Add afsmhdl.c in misc directory to tolerate ARC file with non conforming digits Add afsmhdl.c in misc directory to tolerate ARC file with non conforming digits Nov 30, 2008 r235 (Add wrecord.c in misc directory to tolerate ARC file with no...) committed by voidptrptr - Add wrecord.c in misc directory to tolerate ARC file with non conforming digits Add wrecord.c in misc directory to tolerate ARC file with non conforming digits Nov 21, 2008 r234 (Add Licenses) committed by voidptrptr - Add Licenses Add Licenses Nov 21, 2008 r233 (Remove unused files in Java wrapper) committed by voidptrptr - Remove unused files in Java wrapper Remove unused files in Java wrapper Nov 17, 2008 issue 92 (SRS 85 — Libwarc shall compile and run under Solaris, AIX, M...) Status changed by gordon.paynter - Yopunes writes: The only exception is AIX which isn't available at Hanzo. If you have an AIX machine, we can check the portability ASAP. But as we're using GCC, so we can be confident about AIX. Status: Done Yopunes writes: The only exception is AIX which isn't available at Hanzo. If you have an AIX machine, we can check the portability ASAP. But as we're using GCC, so we can be confident about AIX. Status: Done Nov 17, 2008 issue 72 (SRS 65 — It shall be possible for libwarc to be able to hand...) Status changed by gordon.paynter - Repeated test on 5.5GB ARC: ../warc-tools-read-only/app/arc2warc -a six.arc.gz -f six.warc This works fine, output as follows: -rw-r--r-- 1 paynter paynter 6495613350 2008-11-18 12:14 six.warc Status: Done Repeated test on 5.5GB ARC: ../warc-tools-read-only/app/arc2warc -a six.arc.gz -f six.warc This works fine, output as follows: -rw-r--r-- 1 paynter paynter 6495613350 2008-11-18 12:14 six.warc Status: Done

issue 61 (SRS 55 — The WARC validator tool specified in SRS 31-32 shal...) Owner changed by gordon.paynter - WARC Tools now includes contrib/java and contrib/jhove. Today I compiled both of these up, then followed the instructions in contrib/jhove/install for downloading jhove and patching from the warc-tools directory to build a version of Jhove with WARC support. I used the command-line tool to validate a WARC file (and to show several other files were or were ot WARC files). Anyway, while we do not have a WARC module "in jhove" we do have a way to compile WARC support into Jhove 1.1, which is probably as good as we can hope to get at this point. I suggest we declare this done. Gordon PS: I am not a java developer, so had a few problems getting this all to run. For example, jhove would not compile out of the box due to a non-warc-tools related compile error, which I solved by commenting out the "pop" method that caused the error.

Owner: bjarne.andersen

WARC Tools now includes contrib/java and contrib/jhove. Today I compiled both of these up, then followed the instructions in contrib/jhove/install for downloading jhove and patching from the warc-tools directory to build a version of Jhove with WARC support. I used the command-line tool to validate a WARC file (and to show several other files were or were ot WARC files). Anyway, while we do not have a WARC module "in jhove" we do have a way to compile WARC support into Jhove 1.1, which is probably as good as we can hope to get at this point. I suggest we declare this done. Gordon PS: I am not a java developer, so had a few problems getting this all to run. For example, jhove would not compile out of the box due to a non-warc-tools related compile error, which I solved by commenting out the "pop" method that caused the error.

Owner: bjarne.andersen

Earlier this year

Older