My favorites | Sign in
Project Home Downloads Wiki Issues Source
Search
for
SikuliWebDriver  

Featured
Updated Nov 19, 2012 by doubles...@gmail.com

SikuliFirefoxDriver

SikuliFirefoxDriver extends Selenium's FirefoxDriver by adding Sikuli's image search capability. It is useful for automating interactions with highly visual interfaces such as Google Map.

Use

findImageElement() is the method that allows one to find an element using an image. For example, the code below opens Firefox, opens Google Code's homepage, and locates the logo image of Google Code.

SikuliFirefoxDriver driver = new SikuliFirefoxDriver();		
driver.get("http://code.google.com");
ImageElement image = driver.findImageElement(new URL("http://code.google.com/images/code_logo.gif"));

Since SikuliFirefoxDriver directly inherits from Selenium's FirefoxDriver class, you can use all existing methods provided by FirefoxDriver such as the example below that uses findElement(By ...) to locate Google Code's search box and enter "Sikuli" as a search string:

driver.findElement(By.id("gsearchInput"));
driver.sendKeys("Sikuli");

Setup

If you use Maven, add the dependency statements below to your project's pom.xml:

<dependency>
	<groupId>org.sikuli</groupId>
	<artifactId>sikuli-webdriver</artifactId>
	<version>1.0.1</version>
</dependency>

Google Map example

Here is an complete example that demonstrates how one can use SikuliFirefoxDriver to drive interactions with a web page using both image recognition (Sikuli) and DOM (Selenium). This example performs the sequence of the operations below:

  1. Selenium: Enter "Denver, CO" as search terms, finding the search box using element ID gbqfq
  2. Sikuli: Click on to zoom in to the Lakewood region.
  3. Sikuli: Click on to zoom in to the Kendrick Lake region.
  4. Sikuli: Click on to switch to the satellite view.
  5. Sikuli: Click on to zoom in within the current view.
  6. Selenium: Click on the LINK button, finding the button using element ID link
package org.sikuli.webdriver.examples;

import java.io.IOException;
import java.net.URL;

import org.openqa.selenium.By;
import org.openqa.selenium.Keys;
import org.openqa.selenium.WebElement;
import org.sikuli.webdriver.ImageElement;
import org.sikuli.webdriver.SikuliFirefoxDriver;

public class GoogleMapExample {
	
	public static void main(String[] args) throws IOException {
		
		SikuliFirefoxDriver driver = new SikuliFirefoxDriver();		
		
		// visit Google Map
		driver.get("https://maps.google.com/");
		
		// enter "Denver, CO" as search terms
		WebElement input = driver.findElement(By.id("gbqfq"));
		input.sendKeys("Denver, CO");		
		input.sendKeys(Keys.ENTER);

		ImageElement image;
		
		// find and click on the image of the lakewood area
		image = driver.findImageElement(new URL("https://dl.dropbox.com/u/5104407/lakewood.png"));
		image.doubleClick()	;		
		
		// find and click on the image of the kendrick lake area
		image = driver.findImageElement(new URL("https://dl.dropbox.com/u/5104407/kendrick_lake.png"));
		image.doubleClick();

		// find and click the Satellite icon to switch to the satellite view
		image = driver.findImageElement(new URL("https://dl.dropbox.com/u/5104407/satellite.png"));		
		image.click();

		// find and click the plus button to zoom in
		image = driver.findImageElement(new URL("https://dl.dropbox.com/u/5104407/plus.png"));
		image.click();
		
		// find and click the link button
		WebElement linkButton = driver.findElement(By.id("link"));
		linkButton.click();
				
	}

}

Page Object

You can define page objects with image elements using the @FindByImage annotation. It can be used together with the @FindBy annotation provided by Selenium.

public class GoogleMapPage {

	@FindBy(how = How.ID, using = "gbqfq")
	private WebElement searchInput;

	@FindByImage(url = "https://dl.dropbox.com/u/5104407/plus.png")
	private ImageElement plus;
	
	public void zoomIn(){
		plus.click();
	}

	public void searchFor(String text) {
		searchInput.clear();
		searchInput.sendKeys(text);
		searchInput.submit();
	}
}

Below is a more complete example:

import java.io.IOException;

import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.FindBy;
import org.openqa.selenium.support.How;
import org.sikuli.webdriver.ImageElement;
import org.sikuli.webdriver.SikuliFirefoxDriver;
import org.sikuli.webdriver.support.FindByImage;
import org.sikuli.webdriver.support.SikuliPageFactory;

public class PageObjectsExample {
	public static void main(String[] args) throws IOException {

		SikuliFirefoxDriver driver = new SikuliFirefoxDriver();
		driver.get("http://map.google.com/");

		GoogleMapPage page = SikuliPageFactory.initElements(driver, GoogleMapPage.class);
		page.searchFor("Denver, CO");		
		
		DenverArea denverMap = SikuliPageFactory.initElements(driver, DenverArea.class);
		denverMap.lakewood.doubleClick();
		
		denverMap.searchFor("hotel");
		
		denverMap.zoomIn();

	}
	
	public static class DenverArea extends GoogleMapPage {
		@FindByImage(url = "https://dl.dropbox.com/u/5104407/lakewood.png")
		public ImageElement lakewood;				
	} 
	
	public static class GoogleMapPage {

		@FindBy(how = How.ID, using = "gbqfq")
		private WebElement searchInput;

		@FindByImage(url = "https://dl.dropbox.com/u/5104407/plus.png")
		private ImageElement plus;
		
		public void zoomIn(){
			plus.click();
		}

		public void searchFor(String text) {
			searchInput.clear();
			searchInput.sendKeys(text);
			searchInput.submit();
		}
	} 

}
Comment by butada...@gmail.com, Nov 9, 2012

Really cool guys! Thank you!

Comment by project member doubles...@gmail.com, Nov 9, 2012

Thanks! Glad you discovered it. Please report any issue you encounter.

Comment by butada...@gmail.com, Nov 9, 2012

Sure! Will do that!

I would be really cool if it were possible to do this:

@FindBy?(image = "path/to/image") private ImageElement? imageElement;

Then we can leverage the power of page object pattern. :)

Regards, Dan

Comment by butada...@gmail.com, Nov 9, 2012

Can I get a read-only copy of the source code for SikuliFirefoxDriver??

I would like to look into doing the same for the Chrome/Safari, IE? (mybe).

It's a very useful feature!

Thank you, Regards, Dan

Comment by project member doubles...@gmail.com, Nov 9, 2012

The public repo is at: https://bitbucket.org/doubleshow/sikuli-webdriver. If you can get it to work for other browsers, please share with your solution.

Comment by project member doubles...@gmail.com, Nov 9, 2012

Dan, Page object pattern support is a good idea. We will be working on it. Can you create an issue and mark it as an enhancement? This helps us track things.

Comment by sureneth...@gmail.com, Dec 6, 2012

Hi this is really cool stuff!!!! Great job.

I have a query, does findImageElement() method works when we we run the script in t he background? Sikuli requires the page to be active so that it could perform the corresponding operation on the page where the image is present. Does findImageElement() going to work on the same lines?

Thanks and regards, Surendran Ethiraj

Comment by hmano...@gmail.com, Dec 14, 2012

Tom, It is working with other browsers as well.Nice Integration!

Comment by project member doubles...@gmail.com, Dec 14, 2012

thanks for letting us know. can you please share with us your experience? specifically, what browsers can you get it to work, and whether there are any modification to the code you need to make?

Comment by hmano...@gmail.com, Dec 14, 2012

i just make another class like SikuliFirefoxDriver? and the same class extends from chromedriver after that i run the script and it get to work. But with internet explorer i faced some problem while script running like IE getting max n minimise continuously. It is just caused of takescreenshot class. Window xp sp3 browser IE8

Comment by hmano...@gmail.com, Dec 16, 2012

hey tom...did u try??

Comment by vigneshb...@gmail.com, Dec 19, 2012

Same on IE9 too.

Comment by manga...@gmail.com, Jan 2, 2013

This is nice. Now, the next evolution up would be supporting such SikuliFirefoxDriver? to work under Grid or RemoteWebDriver?, such that both FirefoxDriver? part and the Sikuli part both work in Grid (or remotely). It will certainly be tricky but would be interesting and nice when it can be done. Here’s some resources for trying to support it under remote/Grid mode:

http://autumnator.wordpress.com/2011/12/22/autoit-sikuli-and-other-tools-with-selenium-grid/

FYI, by remotely, I mean Sikuli executes on the correct node, not the local machine that you run code from.

Comment by Peter.Ob...@gmail.com, Jan 3, 2013

Hey all,

I originally started using Sikuli in order to test a browser add-on (which webdriver is not aware of). However, we use webdriver for all of our other automation and I'd like to keep everything under 1 framework if I can. However, does this Sikuli integration allow you to detect regions outside of the browser window, such as the add-on icons in the browser header (which holds the browser action buttons, etc.)? My usecase is that I need to install an add-on, enable it, then do GUI testing on it.

Thanks

Comment by midnight...@gmail.com, Jan 9, 2013

Hey, Its great. But can anybody tell that, is it possible to use SikuliFirefoxDriver? using Python?? DD

Comment by qinjie1...@gmail.com, Jan 10, 2013

Really Amazing!

Comment by feel2w...@gmail.com, Feb 16, 2013

Hi

Just now I downloaded sikuli-api-1.0.2-standalone.jar and tried the above example but I am getting the following error "The import org.sikuli.webdriver cannot be resolved".

Can any one help.

Regards,

Narasimha Raju

Comment by nimesh2...@gmail.com, Mar 12, 2013

I think one has to follow: Supported Platforms Windows 7 (64/32 bits) Mac OS X (64 bits) Linux (64 bits) See special instructions

Comment by matthewd...@gmail.com, Apr 24, 2013

Would be great to know if getting a version of this working on selenium grid is on the roadmap?

Comment by abhijeet...@gmail.com, May 16, 2013

Hats Off!!! This is amazing.

Comment by tech01h...@gmail.com, May 22, 2013

I am trying to use sikuli. Tried the above example. But getting the error: Am I missing something??

Exception in thread "main" java.lang.NoClassDefFoundError?: org/sikuli/api/Screen

at GoogleMapExample?.main(GoogleMapExample?.java:15)
Caused by: java.lang.ClassNotFoundException?: org.sikuli.api.Screen
at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController?.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader?.loadClass(ClassLoader?.java:306) at sun.misc.Launcher$AppClassLoader?.loadClass(Launcher.java:301) at java.lang.ClassLoader?.loadClass(ClassLoader?.java:247) ... 1 more

Comment by PoojaSha...@gmail.com, May 23, 2013

Nice.. but I'm getting exception while trying to find "", How do I resolve it? I have all the required jars properly set. (sikuli-api-1.0.1.jar,sikuli-webdriver-1.0.1.jar, slf4j-api-1.7.5.jar, slf4j-log4j12-1.7.5, log4j.jar and selenium specific ) Here is the stackTrace:-

Exception in thread "main" java.lang.NoClassDefFoundError?: org/sikuli/core/search/algorithm/TemplateMatcher?

at org.sikuli.api.ImageTarget.getUnordredMatches(ImageTarget.java:116) at org.sikuli.api.DefaultTarget?.doFindAll(DefaultTarget?.java:93) at org.sikuli.api.DefaultScreenRegion?.find(DefaultScreenRegion?.java:151) at org.sikuli.api.DefaultScreenRegion?.access$100(DefaultScreenRegion?.java:29) at org.sikuli.api.DefaultScreenRegion?$RepeatFind?.run(DefaultScreenRegion?.java:188) at org.sikuli.api.DefaultScreenRegion?.wait(DefaultScreenRegion?.java:200) at org.sikuli.webdriver.SikuliFirefoxDriver?.findImageElement(SikuliFirefoxDriver?.java:40) at com.jenesys.test.Help.GoogleMap?.main(GoogleMap?.java:29)

Comment by PoojaSha...@gmail.com, May 23, 2013

smartly, this forum is recognising the image, but via code, on line image = driver.findImageElement(new URL("")); it just fails with

Exception in thread "main" java.lang.NoClassDefFoundError?: org/sikuli/core/search/algorithm/TemplateMatcher?

Comment by rdekleij...@gmail.com, May 26, 2013

Indeed would be nice to know if an integration with Selenium Grid is on the roadmap...

Comment by gpa...@gmail.com, May 27, 2013

Where can i download org.sikuli.webdriver package?

Comment by chrish...@gmail.com, Jul 14, 2013

Does firefox have to be the default? my code is returning an error that it cannot find firefox binary

run: Exception in thread "main" org.openqa.selenium.WebDriverException?: Cannot find firefox binary in PATH. Make sure firefox is installed. OS appears to be: VISTA Build info: version: '2.33.0', revision: '4e90c97', time: '2013-05-22 15:33:32' System info: os.name: 'Windows 7', os.arch: 'amd64', os.version: '6.1', java.version: '1.7.0_07' Driver info: driver.version: SikuliFirefoxDriver?

at org.openqa.selenium.firefox.internal.Executable.<init>(Executable.java:72) at org.openqa.selenium.firefox.FirefoxBinary?.<init>(FirefoxBinary?.java:59) at org.openqa.selenium.firefox.FirefoxBinary?.<init>(FirefoxBinary?.java:55) at org.openqa.selenium.firefox.FirefoxDriver?.<init>(FirefoxDriver?.java:92) at org.sikuli.webdriver.SikuliFirefoxDriver?.<init>(SikuliFirefoxDriver?.java:24) at Testing1.main(Testing1.java:23)
Java Result: 1 BUILD SUCCESSFUL (total time: 0 seconds)

Comment by mukilan...@gmail.com, Sep 25, 2013

I dont think this will work in a machine which is locked

Comment by madiredd...@gmail.com, Sep 29, 2013

Hi, I am pavan,i am trying to execute above Google map example in eclipse,i am getting the below error.Can any one find solution for this error.

Exception in thread "main" java.lang.UnsatisfiedLinkError?: C:\Users\PAVAN\AppData?\Local\Temp\javacpp2504108273440\jniopencv_core.dll: Can't find dependent libraries

at java.lang.ClassLoader?$NativeLibrary?.load(Native Method) at java.lang.ClassLoader?.loadLibrary1(Unknown Source) at java.lang.ClassLoader?.loadLibrary0(Unknown Source) at java.lang.ClassLoader?.loadLibrary(Unknown Source) at java.lang.Runtime.load0(Unknown Source) at java.lang.System.load(Unknown Source) at com.googlecode.javacpp.Loader.loadLibrary(Loader.java:418) at com.googlecode.javacpp.Loader.load(Loader.java:368) at com.googlecode.javacpp.Loader.load(Loader.java:315) at com.googlecode.javacv.cpp.opencv_core.<clinit>(opencv_core.java:131) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Unknown Source) at com.googlecode.javacpp.Loader.load(Loader.java:334) at com.googlecode.javacpp.Loader.load(Loader.java:315) at com.googlecode.javacv.cpp.opencv_core$CvArr?.<clinit>(opencv_core.java:158) at org.sikuli.core.cv.ImagePreprocessor?.createGrayscale(ImagePreprocessor?.java:52) at org.sikuli.core.search.algorithm.TemplateMatcher?.findMatchesByGrayscaleAtOriginalResolution(TemplateMatcher?.java:18) at org.sikuli.api.ImageTarget.getUnordredMatches(ImageTarget.java:122) at org.sikuli.api.DefaultTarget?.doFindAll(DefaultTarget?.java:93) at org.sikuli.api.DefaultScreenRegion?.find(DefaultScreenRegion?.java:73) at org.sikuli.api.DefaultScreenRegion?.access$100(DefaultScreenRegion?.java:29) at org.sikuli.api.DefaultScreenRegion?$RepeatFind?.run(DefaultScreenRegion?.java:110) at org.sikuli.api.DefaultScreenRegion?.wait(DefaultScreenRegion?.java:119) at org.sikuli.webdriver.SikuliFirefoxDriver?.findImageElement(SikuliFirefoxDriver?.java:40) at com.sai.Test.main(Test.java:29)

Comment by harihara...@gmail.com, Nov 5, 2013

Is there is any function like sendKeys which can be used to type text on the image?

Comment by pushirat...@gmail.com, Jan 11, 2014

Can we handle a Security alert pop up in Safari using Sikulli Webdriver?

Comment by edwardch...@gmail.com, Jan 11, 2014

My thought to run Sikuli script in background.

Minimize and maximize browser window().setPosition(new Point(-3000, 0)) to default screen coordinates. Use java capture the image of screen/browser, send to sikuli for detect and return coordinate of detected image, and send to Selenium to click on element on specific coordinates or using Selenium Action Class to click on specific coordinates by offset for flash.

I still new on Selenium, I hope if someone figure out can contribute part by part of making sikuli run background seamlessly as Selenium

Comment by Eileen.Z...@vitals.com, Feb 4, 2014

Is there a way to use an image from workspace/local file system instead of using URL? I'm referring to these 2 lines:

@FindByImage?(url = "")
image = driver.findImageElement(new URL(""));

Comment by mkhb...@gmail.com, Mar 13, 2014

I would also like to know the answer as well

Is there a way to use an image from workspace/local file system instead of using URL? I'm referring to these 2 lines:

@FindByImage??(url = "") image = driver.findImageElement(new URL(""));

Comment by randomke...@gmail.com, Mar 21, 2014

I would also like to know if we can specify a path to a file on the filesystem instead of URL. Thanks.

Comment by vitalii....@gmail.com, Apr 7, 2014

 If I am not mistaken, version 1.0.1 uses webdriver-java 2.26.0 which supports Firefox 22 only. Is it possible to update and rebuild sikuli-driver library and put it in maven repo? I tried to update the dependency for sikuli-driver library to use 2.41.0 version of selenium but I still have the same stack trace. Thanks!

Comment by manga...@gmail.com, Apr 30, 2014

FYI, if anyone forking/extending SikuliFirefoxDriver? for other browsers, it is best to do it for RemoteWebDriver?, so that it works in connecting to a remote/grid node as well as be agnostic to any browser driver.

Comment by scorpion...@gmail.com, Jun 13, 2014

How do I change the default timeout for findImageElement() to be more than 10 seconds?

Comment by vijaykm...@gmail.com, Jun 17, 2014

any particular usage apart from this?

Coz we can do same using @FindBy?(css="img[src=\"/logos/doodles/2014/world-cup-2014-15-5971113733521408-hp.gif\"]")

Comment by kucheren...@gmail.com, Aug 9, 2014

I used the code from your example and it didn't work. It stopped working at this point: // find and click on the image of the lakewood area

image = driver.findImageElement(new URL("")); image.doubleClick() ;
It just hangs after searching the "Denver, CO" and can not recognize any image then. Can anyone say what it can be wrong? My Sikuli versions in the pom.xml: <dependency>
<groupId>org.sikuli</groupId> <artifactId>sikuli-webdriver</artifactId> <version>1.0.1</version>
</dependency>

<dependency>
<groupId>org.slf4j</groupId> <artifactId>slf4j-simple</artifactId> <version>1.7.5</version>
</dependency>

and Firefox 18.0 browser

Comment by shahmaul...@gmail.com, Aug 11, 2014

Can you share run-able code with details of dependencies to be added for beginner.


Sign in to add a comment
Powered by Google Project Hosting