My favorites | Sign in
Project Home Downloads Wiki Issues Source
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 47: are there any plans to move to maven?
5 people starred this issue and may be notified of changes. Back to list
Status:  Done
Owner:  ----
Closed:  Jan 2012


Sign in to add a comment
 
Reported by mark.c.e...@gmail.com, May 4, 2011
I was just wondering: have you thought about porting crawler4j to Maven?
I think that using maven would make it easier for other projects to include crawler4j as a dependency
May 4, 2011
Project Member #1 ganjisaffar@gmail.com
I assume you're talking about adding it to a central Maven repository. Do you know the process?

-Yasser
May 5, 2011
#2 mark.c.e...@gmail.com
Hello Yasser,

I created a maven pom file which defines the projects dependencies.
I put it on github (synced the svn history with git).
you can view it here https://github.com/mren/crawler4j

One dependency (dsiutils) is from a non-standard repository,
unfortunately the maintainers did not put a recent version to central
maven.
May 5, 2011
#4 mark.c.e...@gmail.com
awesome
Sep 26, 2011
#5 ndjockab...@yahoo.fr
hi mark , i tried to run the pom file as an aggregated project of parent pom:

<groupId>net.she.swengineering.diplomarbeit</groupId>
  <artifactId>Diff2W3C</artifactId>
  <packaging>war</packaging>
  <version>1.0</version>
  <name>Diff2W3C Maven Webapp</name>
  <url>http://maven.apache.org</url>
  
  
  <modules>
  	<module>crawler4j</module>
  </modules>
  
the module "crawler4j" specifies you described the directory you put to your  github svn account.in this directory there are the pom file and the source of crawler4j.
 

it doesn't work because work here is my trace:

[INFO] ------------------------------------------------------------------------
[INFO] Building Unnamed - crawler4j:crawler4j:jar:2.61
[INFO]    task-segment: [eclipse:eclipse]
[INFO] ------------------------------------------------------------------------
[INFO] Preparing eclipse:eclipse
[INFO] No goals needed for project - skipping
[INFO] [eclipse:eclipse {execution: default-cli}]
[INFO] Using Eclipse Workspace: C:\Dokumente und Einstellungen\andjock\workspace

[WARNING] Workspace defines a VM that does not contain a valid jre/lib/rt.jar: C
:\Programme\Java\jre6
[INFO] no substring wtp server match.
[INFO] Using as WTP server : Apache Tomcat v6.0
[INFO] Adding default classpath container: org.eclipse.jdt.launching.JRE_CONTAIN
ER
Downloading: http://loki.she.de:8080/nexus/content/groups/public/it/unimi/dsi/ds
iutils/1.0.10/dsiutils-1.0.10.pom
[INFO] Unable to find resource 'it.unimi.dsi:dsiutils:pom:1.0.10' in repository
central (http://central
Downloading: http://loki.she.de:8080/nexus/content/groups/public/it/unimi/dsi/ds
iutils/1.0.10/dsiutils-1.0.10.pom
[INFO] Unable to find resource 'it.unimi.dsi:dsiutils:pom:1.0.10' in repository
repo-for-dsiutils (http://ir.dcs.gla.ac.uk/~bpiwowar/maven/)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/it/unimi/dsi/ds
iutils/1.0.10/dsiutils-1.0.10.pom
[INFO] Unable to find resource 'it.unimi.dsi:dsiutils:pom:1.0.10' in repository
oracle (http://download.oracle.com/maven)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/com/sleepycat/j
e/4.0.71/je-4.0.71.pom
[INFO] Unable to find resource 'com.sleepycat:je:pom:4.0.71' in repository centr
al (http://central
Downloading: http://loki.she.de:8080/nexus/content/groups/public/com/sleepycat/j
e/4.0.71/je-4.0.71.pom
[INFO] Unable to find resource 'com.sleepycat:je:pom:4.0.71' in repository repo-
for-dsiutils (http://ir.dcs.gla.ac.uk/~bpiwowar/maven/)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/com/sleepycat/j
e/4.0.71/je-4.0.71.pom
[INFO] Unable to find resource 'com.sleepycat:je:pom:4.0.71' in repository oracl
e (http://download.oracle.com/maven)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/it/unimi/dsi/ds
iutils/1.0.10/dsiutils-1.0.10.jar
[INFO] Unable to find resource 'it.unimi.dsi:dsiutils:jar:1.0.10' in repository
central (http://central
Downloading: http://loki.she.de:8080/nexus/content/groups/public/it/unimi/dsi/ds
iutils/1.0.10/dsiutils-1.0.10.jar
[INFO] Unable to find resource 'it.unimi.dsi:dsiutils:jar:1.0.10' in repository
repo-for-dsiutils (http://ir.dcs.gla.ac.uk/~bpiwowar/maven/)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/it/unimi/dsi/ds
iutils/1.0.10/dsiutils-1.0.10.jar
[INFO] Unable to find resource 'it.unimi.dsi:dsiutils:jar:1.0.10' in repository
oracle (http://download.oracle.com/maven)
[WARNING] An error occurred during dependency resolution.
    Failed to retrieve it.unimi.dsi:dsiutils-1.0.10
Caused by: Unable to download the artifact from any repository

Try downloading the file manually from the project website.

Then, install it using the command:
    mvn install:install-file -DgroupId=it.unimi.dsi -DartifactId=dsiutils -Dvers
ion=1.0.10 -Dpackaging=jar -Dfile=/path/to/file

Alternatively, if you host your own repository you can deploy the file there:
    mvn deploy:deploy-file -DgroupId=it.unimi.dsi -DartifactId=dsiutils -Dversio
n=1.0.10 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]

Path to dependency:
        1) crawler4j:crawler4j:jar:2.61
        2) it.unimi.dsi:dsiutils:jar:1.0.10


  it.unimi.dsi:dsiutils:jar:1.0.10

from the specified remote repositories:
  nexus (http://loki.she.de:8080/nexus/content/groups/public)


[INFO] Unable to read jar manifest from C:\Dokumente und Einstellungen\andjock\.
m2\repository\it\unimi\dsi\dsiutils\1.0.10\dsiutils-1.0.10.jar
Downloading: http://loki.she.de:8080/nexus/content/groups/public/com/sleepycat/j
e/4.0.71/je-4.0.71.jar
[INFO] Unable to find resource 'com.sleepycat:je:jar:4.0.71' in repository centr
al (http://central
Downloading: http://loki.she.de:8080/nexus/content/groups/public/com/sleepycat/j
e/4.0.71/je-4.0.71.jar
[INFO] Unable to find resource 'com.sleepycat:je:jar:4.0.71' in repository repo-
for-dsiutils (http://ir.dcs.gla.ac.uk/~bpiwowar/maven/)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/com/sleepycat/j
e/4.0.71/je-4.0.71.jar
[INFO] Unable to find resource 'com.sleepycat:je:jar:4.0.71' in repository oracl
e (http://download.oracle.com/maven)
[WARNING] An error occurred during dependency resolution.
    Failed to retrieve com.sleepycat:je-4.0.71
Caused by: Unable to download the artifact from any repository

Try downloading the file manually from the project website.

Then, install it using the command:
    mvn install:install-file -DgroupId=com.sleepycat -DartifactId=je -Dversion=4
.0.71 -Dpackaging=jar -Dfile=/path/to/file

Alternatively, if you host your own repository you can deploy the file there:
    mvn deploy:deploy-file -DgroupId=com.sleepycat -DartifactId=je -Dversion=4.0
.71 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]

Path to dependency:
        1) crawler4j:crawler4j:jar:2.61
        2) com.sleepycat:je:jar:4.0.71


  com.sleepycat:je:jar:4.0.71

from the specified remote repositories:
  nexus (http://loki.she.de:8080/nexus/content/groups/public)


[INFO] Unable to read jar manifest from C:\Dokumente und Einstellungen\andjock\.
m2\repository\com\sleepycat\je\4.0.71\je-4.0.71.jar
[INFO] Not writing settings - defaults suffice
[INFO] Wrote Eclipse project for "crawler4j" to C:\Dokumente und Einstellungen\a
ndjock\workspace\Diff2W3C\crawler4j.
Dec 21, 2011
#7 mon...@gmail.com
Hi,

I done working setup for netbeans project which is using crawler4j, jsoup and JPA,

here's my pom.xml



<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.gmail.krzysztof7kowalski</groupId>
    <artifactId>scrap-maven</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>medscrap-maven</name>
    <url>http://maven.apache.org</url>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.3.2</version>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
    <repositories>
        <repository>
            <id>oracleReleases</id>
            <name>Oracle Released Java Packages</name>
            <url>http://download.oracle.com/maven</url>
            <layout>default</layout>
        </repository>
        <repository>
            <id>piwowar</id>
            <url>http://ir.dcs.gla.ac.uk/~bpiwowar/maven</url>
        </repository>
    </repositories>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.1</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.1.2</version>
        </dependency>

        <dependency>
            <groupId>it.unimi.dsi</groupId>
            <artifactId>dsiutils</artifactId>
            <version>1.0.10</version>
        </dependency>

        <dependency>
            <groupId>com.sleepycat</groupId>
            <artifactId>je</artifactId>
            <version>3.3.75</version>
        </dependency>

        <dependency>
            <groupId>commons-logging</groupId>
            <artifactId>commons-logging</artifactId>
            <version>1.1.1</version>
        </dependency>

        <dependency>
            <groupId>commons-codec</groupId>
            <artifactId>commons-codec</artifactId>
            <version>1.6</version>
        </dependency>

        <dependency>
        <!-- jsoup HTML parser library @ http://jsoup.org/ -->
            <groupId>org.jsoup</groupId>
            <artifactId>jsoup</artifactId>
            <version>1.6.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpcore</artifactId>
            <version>4.1.3</version>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.14</version>
        </dependency>
        <dependency>
            <groupId>org.eclipse.persistence</groupId>
            <artifactId>javax.persistence</artifactId>
            <version>2.0.0</version>
            <type>jar</type>
        </dependency>
        <dependency>
            <groupId>com.mysql</groupId>
            <artifactId>management</artifactId>
            <version>5-0-2-beta</version>
            <exclusions>
                <exclusion>
                    <artifactId>mysql-connector-java</artifactId>
                    <groupId>mysql</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>it.unimi.dsi</groupId>
            <artifactId>fastutil</artifactId>
            <version>6.4.1</version>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.18</version>
        </dependency>
    </dependencies>
    
</project>


Jan 1, 2012
Project Member #8 ganjisaffar@gmail.com
As of version 3.0 support for Maven is added. See https://code.google.com/p/crawler4j/source/browse/pom.xml for the pom.xml file. crawler4j will also soon be available through a public Maven repository.

-Yasser
Status: Started
Labels: -Type-Defect Type-Enhancement
Jan 5, 2012
Project Member #9 ganjisaffar@gmail.com
crawler4j is now available from central Maven repository: https://code.google.com/p/crawler4j/wiki/MavenConfig

-Yasser
Status: Done
Feb 1, 2012
#10 vundic...@gmail.com
Great job, ganjisaffar !!!
Sign in to add a comment

Powered by Google Project Hosting