Image Automation

1. Overview #

Clicknium automation is integrated with many image automation functions. Image detection is the best choice when it is hard to locate UI elements with attributes.

2. Capture image locator #

Open a project in Visual Studio Code by pressing Ctrl + F10 or clicking the button Capture to invoke Clicknium recorder.

  • Hover the mouse over the element to highlight it.
  • Press “Ctrl” and click the element to capture its locator.
  • Press” Shift” and select one area by the mouse to capture the image. The locator will be stored as an image locator.

For example :

Sample1: Choose the target element
Sample2: Press Shift and drag to select an area
Result : the locator you captured

The image locator contains two parts.
① The first part is the anchor element. In the running process, first, find the anchor element, then capture the anchor image depending on the image method attributes, and find the target image in the anchor image by image matching algorithm.

② The second part is image itself with the following attributes:

accuracyThe minimum similarity between the target image and the one in image locator. It is useful when the image to be found is slightly different from the one in the locator. Measurement unit is between 0 and 1 with default value 0.75.YesNoNoNo
dpiwe recommend the DPI settings of Windows OS in the process is the same as the ones when recordingYesNoNoNo
methodenum values define the algorithm of image matching. The value is autoset when recordingYesNoNoNo
filePaththe selection area of image recording, used in the process of image matchingYesNoNoNo
matchIndexwhich will be selected when more than one area is found during image matching, default is 0, it means the first matching areaYesNoNoNo
elementRectstore the anchor element area during recording. No need to modify the valueYesNoNoNo
selectionRectstore the selection area during recording. No need to modify the valueYesNoNoNo
timeoutSpecify the maximum time interval in image matching, default is 5000 millisecondsYesNoNoNo

The definition of methods:

  • HighestAccuracy: compare the images found in whole Windows screen.
  • InRegionHighestAccuracy: compare the images found in anchor element area.
  • OutRegionHighestAccuracy: comapre the images found out of the anchor element area.

Method attributes are automatically set based on the selection area and corresponding anchor area in the process of recording If the selection area is inside the anchor element area, set to ‘InRegionHighestAccuracy’;

If the selection area intersects with the anchor element area, set to ‘HighestAccuracy’; If the selection area is outside the anchor element area, set to ‘OutRegionHighestAccuracy’;

3. Use image locator in project #

Image locator can be used in the same way as other locators, for example

from clicknium import clicknium as cc, locator, ui

#open new browser window
driver ="")


The following functions support image locator:

  • click
  • double_click
  • mouse_up
  • mouse_down
  • drag_drop
  • get_position
  • get_size
  • highlight
  • hover
  • send_hotkey
  • set_text (parameter by should be sendkey-after-click)
What are your feelings
Updated on 30 November 2023