Sikuli Tutorial: Automate Desktop & Windows Application with Selenium Integration

We have discussed different aspects and uses of Selenium WebDriver for our Test automation projects so far. There are various automation testing tools which fall under Licensed as well as Open source categories. Sikuli is one of the open source test automation tool. Today we are going to discuss all the basic guide of GUI (Graphical User Interface) automation using Sikuli. This Sikuli tutorial will answer all your basic questions. Additionally, we are going to discuss Sikuli integration with Selenium, which ultimately solves the OS or Windows-based pop-up handling problems.

Suggested Readings: How to parse XML file in Selenium?

Let’s begin!

What is Sikuli?

Sikuli is an open source automation testing tool which is used for test automation of web application as well as a desktop application.

How does Sikuli work?

Skiuli works in a different and easy way. You don’t need to bother about locators, XPath or any other techniques. Here you just need to capture the image of the element at which you are going to perform UI action. But, you must sure that you don’t change the resolution of the environment. Hence, automation testing with Sikuli is only feasible on static UI objects.

Significances of using Sikuli

This section of the Sikuli tutorial will help you understand some of the true behavior of Sikuli automation tool. It will help you in deciding the correct tool for your project.

Here we go!

• Sikuli is an open source tool
• Convenient for Flash-based testing
• Easy for video app testing
• Able to automate windows based application
• Simple to implement
• Able to automate desktop application

Pre-Requisites

We need to follow some pre-requisites before using Sikuli in our test automation project. Some of the pre-requisites are as follows:

• You must have- Image capturing tool (Like Snipping tool in Windows OS)
• Java Development Kit (JDK)
• Any IDE like Eclipse

How to Integrate Sikuli with Maven project?

As we all know, we need to define the dependencies in pom.xml, if we are going to use the Maven project. You can use below dependencies structure in pom.xml or click here to get it from Maven site.

<dependency>
    <groupId>com.sikulix</groupId>
    <artifactId>sikulixapi</artifactId>
    <version>1.1.2</version>
</dependency>

How to configure the Selenium project with SikuliX API Jar file?

If you are not using Maven or not able to resolve the dependencies written inside the pom.xml, then you can directly download the jar file of SikuliX API and can add them in your project. Here I assume you know the way to add external jar file in a project! If you still don’t know then you can read our first step to Selenium blog post.

Click here to download the SikuliX API jar file.

How to interact with UI objects using Sikuli?

We mainly use Screen Class and Pattern Class of Sikuli to perform basic UI actions. Here Screen class acts as the base class mainly. Let’s have a look over some of the important methods used for UI interaction with Sikuli.

Screen Class Instantiation

Screen sc = new Screen();

Note: Sikuli uses images for object identification, hence, each and every method will accept at least one parameter as the path to the image file.

How to uniquely identify web elements with Sikuli?

This segment of Sikuli tutorial will help you identify the web elements in a unique way. There might be chances that Sikuli identifies different objects so in such case we can use Pattern class to uniquely identify the objects.

Pattern pat = new Pattern(“path of the image”);

Some of the methods written inside Pattern class is as follows:

This was all about basic Sikuli Tutorial which we should know before implementing in any real-time test automation project.

Let’s look at the implementation part of this Sikuli Tutorial so that we could understand its use during the scripting of the automation project.

Sikuli Program: Selenium integration with Sikuli

In this program we simply open the Google.com using Selenium and the rest of the search operation is handled by Sikuli alone. We basically first captured the image of the elements like the search bar and search button using the Snipping tool.

In below code, we just gave the image path wherever we are calling Sikuli methods.

Let’s have a look at the below program.

package Test;

import java.util.concurrent.TimeUnit;

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.sikuli.script.FindFailed;
import org.sikuli.script.Pattern;
import org.sikuli.script.Screen;

public class SikuliDemo {

	public static void main(String[] args) throws InterruptedException, FindFailed {
		System.setProperty("webdriver.chrome.driver", "C:\\Selenium\\chromedriver.exe");
	  	 
		WebDriver driver  = new ChromeDriver();
	  	  
	  	  driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
		  
	  	  driver.get("https://www.google.com");
		  
	  	  driver.manage().window().maximize();
	  	  
	  	  Screen sc = new Screen();
	  	  
	  	  sc.type("C:\\Users\\blogg\\]Desktop\\Sikuli\\search bar.PNG", "Inviul");
	  	  
	  	  sc.wait(3);
	  	  
	  	  sc.click("C:\\Users\\blogg\\Desktop\\Sikuli\\search btn.PNG");
	  	  
	  	  driver.close();
	  	  
	  	  driver.quit();

	}

}

How to automate desktop application with Sikuli?

Similarly, we can automate desktop application as well. We just need to capture the image of the object and then we have to use the methods written inside sikuli. That’s it.

This was all about Sikuli tutorial, Hope, it will solve your problem. Click here to read more from the Sikuli official documentation.