Sikuli Tool

Sikuli is a visual technology to search and automate graphical user interfaces (GUI) using images (screenshots). The first release of Sikuli contains Sikuli Script, a visual scripting API for Jython, and Sikuli IDE, an integrated development environment for writing visual scripts with screenshots easily. Sikuli Script automates anything you see on the screen without internal APIs support. You can programmatically control a web page, a desktop application running on Windows/Linux/Mac OS X

URL Home Site: http://www.sikuli.org/

URL for Selenium Site:

https://code.google.com/p/sikuli-api/downloads/list

 

• How to use Sikuli when automating tests with Selenium Webdriver

  Selenium Webdriver is a great tool for test automation. But what if somewhere along the way you need to test some file upload (or other action) which doesn’t have any visible input field where you could send the file path and get it over with?
  Well, I have also looked for solution, like many others, on forums and I found Sikuli. Sikuli provides image-based GUI automation functionalities to Java programmers.
  Let’s assume we have a web page where we need to upload files, but the upload is made using a Flash component. So the steps would be:

  • click on a button (the Flash component)
  • a system window is opened where you can search for your file
  • select the file
  • click the system’s Open button

  In the above case the problem with Selenium Webdriver is that as soon as the system window appears, you can no longer control the events. So something else must come in and take over. This something is Sikuli.
  The solution for people who write Selenium tests using Java is:

  • go to the Sikuli API page https://code.google.com/p/sikuli-api/downloads/list
  • download the API standalone jar and add it to your Build Path at Libraries
  • make screenshots of the buttons or text boxes you use in the process of uploading (for example the system’s Open button)
  • write some code to control events using Sikuli methods

  Below there is an example of how you could handle a file upload:

  ScreenRegion screen = new DesktopScreenRegion();
  Mouse mouse = new DesktopMouse();
  Keyboard kb = new DesktopKeyboard();
  
  //tell Sikuli which part of the desktop it should target - in my case click on My Documents icon, the folder where my upload image is
  Target target = new ImageTarget(new File(Helper.getSikuliImagesPath() + “myDocuments.jpg”));
  //target the region
  ScreenRegion region = screen.find(target);
  //click on the center of it
  mouse.click(region.getCenter());
  //now we are in My Documents folder
  
  //target the text box where we will write the image’s name
  target = new ImageTarget(new File(Helper.getSikuliImagesPath() + “textBox.jpg”));
  region = screen.find(target);
  //click on it
  mouse.click(region.getCenter());
  //and write the image to upload
  kb.type(“images700.jpg”);
  
  //target the system’s Open button
  target = new ImageTarget(new File(Helper.getSikuliImagesPath() + “open.jpg”));
  region = screen.find(target);
  //and click it
  mouse.click(region.getCenter());
  //the upload is now complete
  
  
  Please remember that myDocuments.jpg , textBox.jpg and open.jpg are SCREENSHOTS of actual parts of the environment, which are used by Sikuli in order to recognize what to click or where to type.