What is Sikuli?

Sikuli Script automate flash websites

Sikuli is a technology that uses a “Visual Image Match” to automate and test the creation of a graphical user interfaces (GUI) using images (screenshots of elements). All the web elements in Sikuli are taken as an image and stored inside the project.

Sikuli is an alternative technology for test automation to Selenium or Watir. Whilst Sikuli builds up a repository of graphic UI elements, Selenium and Watir have built up a repository of web UI locators. The way you use Sikuli is to define “atomic” UI elements (i.e. a button on the page, a link, some text) typically without any dependency on layout of other elements and then save the cropped screenshots of these UI elements as reference images. These can then be used to compare them against what you see on screen at runtime.

Sikuli is comprised of:

  • Sikuli Script
  • Visual Scripting API for Jython
  • Sikuli IDE

The practical uses of Sikuli are:

  • It can be used to automate flash websites or objects
  • It can automate windows based applications and anything you see on screen without using internal API support
  • It provides a simple API
  • It can easily be linked with tools like Selenium
  • It can automate Desktop applications
  • Sikuli offers extensive support to automate flash objects
  • To automate desktop it uses powerful “Visual Match” and Flash objects
  • It can work on any technology .NET, Java, Python

Additional positives of Sikuli are:

  • It provides extensive support to automate flash objects
  • It has simple API
  • It uses visual match to find elements on the screen. So we can automate anything we see on the screen
  • It can automate web as well as windows application

A Sikuli Script is a combination of Jython and Java library elements that automate GUI interaction using image patterns to direct mouse or keyboard events.

The main segment of a Sikuli Script is a Java library which contains two parts:

awt.Robot: This delivers mouse and keyboard events to appropriate locations
C++ engine based on OpenCV: It searches given pattern on the screen

To run a Sikuli Script, Sikuli IDE creates an org.python.util.PythonInterpreter and passes a few lines of headers to the interpreter automatically. Once these headers are defined or set, the .py script is simply executed by PythonIntepreter.execfile().

How To Retrieve the name of the Image on pop-up?

You can retrieve the name of the image on pop-up in two ways

Method: 1

query = capture()

gotit = Filename [query] [0]


Method: 2

popup(“select a icon in the”)

query = capture()

Result = Filename [query]

If result:



popup (“unknown image”)

How to Create an Object for Screen Class

In Sikuli, screen is a base class, for this screen class we have to create an object first. This allows us to then access all the methods provided by Sikuli.

Syntax for creating an object in Sikuli: Screen s=new Screen()

To kill the Sikuli Script, you have to use the short cut Alt+Shift+C.

To find a specific element on the screen you use the syntax


In Sikuli, Region class provides an abstraction for the screen regions returned by the find() function matching a given visual pattern. Its attributes include X and Y co-ordinates, height, width, and similarity score.

Sikuli slides can be used to test and automate anything you see on the screen using your presentation slides. For automating repetitive tasks and to crate PowerPoint presentation.

For Sikuli, it maintains a list of location to locate images when they are not found in the Sikuli folder. Sikuli will use following the command to look, add or remove images from the folder.

getImagePath(): # getImagePath() returns a Java array of Unicode strings
addImagePath: It adds a new path to the list of image search path
removeImagePath : It removes a path from the list of image search paths

The setAutoWaitTimeout command is used to determine the maximum waiting time for all subsequent find operations. While the getAutoWaitTimeout is used to determine the current value of the maximum waiting time for find operations.

In Sikuli, Match is a class available in Sikuli Package. Class match extends class Region, all methods of class region can be availed with a match object. An object of class Match displays the result of a successful find operation.

The file extension-name.py contains the classes and methods, which represents the API; that might be used in a Sikuli Script.

To enter particular text on the text box this method is used and command is

s.type(“<>” , “String to be typed”);

Yes, it is possible to work with a particular region in Sikuli. Based on the location of a unique pattern (image) on your display, you can also increase the size of the region.

How to Capture an Image using Sikuli

To capture an image using Sikuli you should take the following steps:

  • Open your Sikuli IDE on your screen
  • Tap on the button “Take Screenshot”. After selecting the target area where you want to take a screen shot by dragging the mouse over it.
  • After selecting the area for image click, now select the image displayed on your Sikuli IDE
  • This image will be saved as the default in your temp folder “%tmp%” and tap on the enter button. Now you can see your image will be displayed in the temp


How to Clear Field using Sikuli Java?

To clear input field Sikuli does not provide any function; we can use Java Robot class and Sikuli. First you have to click on the input field using function click() and then use Java Robot class to clear input field.


Sikuli logo jython

Be the first to comment

Leave a Reply

Your email address will not be published.