November 2, 2022 · 4 min read
Will
Background
We started a project called Clicknium this year, a Python automation library for browser and desktop applications. It depends on the Clicknium browser extension to control the browser, and simulate mouse and keyboard actions.
Since January 2022, adding new extensions based on Manifest V2 to the Chrome Web Store has been forbidden. All extensions on Manifest V2 will stop working in early 2023, even those that were added to Chrome Web Store earlier.
We have no choice but to migrate to V3. After we review our code and migration checklist, the most significant impact on Clicknium is that running remote code will be forbidden in Manifest V3. It asked to include all logic in the extension’s package to review the behavior of the extension better.
How does Clicknium run remote code in Manifest V3? Here is a brief introduction.
How Clicknium runs remote code in V2
This is a simple Clicknium automation case, which executes a JS script on the Bing page to set the value of the search box to “Clicknium”, and return “success”.
from clicknium import clicknium as cc, locator
import os,sys
def main():
tab = cc.chrome.open("bing.com")
ele = tab.find_element(locator.bing.cn.search_sb_form_q)
result = ele.execute_js("function SetText(st){_context$.currentElement.value = st; return \"success\"}", "SetText(\"Clicknium\")")
print(result)
if __name__ == "__main__":
main()In Manifest V2:
- In the background: Through chrome.runtime.sendMessage sends the received string to the content.
- In the content: Using “new Function(jsCode)” to convert the received string to Javascript’s Function,
const innerFunction = new Function('_context$', jsCode);
const ret = innerFunction({
currentElement: targetElement,
tabId: msg.tabId
});In Manifest V3, Function and eval is not available. So the main problem is that how to convert string to Javascript’s Function.
How Clicknium run remote code in V3
There are two approaches to achieving it.
Solution 1: Use pop up window to imitate the background scripts/pages.
In Manifest V3, Using Service worker replace background scripts/pages. The window object can’t be used in the Service worker. But we can open a popup window in the Service worker to imitate background scripts/pages.
- Step 1 : Open a popup window in the Service worker.
// Get the URL of the popup window.
const backgroundUrl = chrome.runtime.getURL("backgroundPage.html");
await chrome.windows.create({
type: "popup",
state: "minimized",
url: backgroundUrl,
});
- Step 2 : In the popup window, when received a message, using chrome.debugger.sendCommand sends to the target of the popup window. And then, get Function from the window object.
// Get the target of the popup window
const backgroundUrl = chrome.runtime.getURL("backgroundPage.html");
const targets = await chrome.debugger.getTargets();
let target;
for (const t of targets) {
if (target.url === backgroundUrl) {
target = t;
}
}
// Wrap up the Javascript's string, register to the window object
const jsCodeWrapper = `window["evalFunction"] = function () {
${jsCode};
};`;
// Attach to the popup window. And send expression.
await chrome.debugger.attach(target, "1.2");
await chrome.debugger.sendCommand(target, "Runtime.evaluate", {
expression: jsCodeWrapper,
});
// Get the function just registered in the window
const wrapperFunc = window["evalFunction"];Step 3: Using chrome.scripting.executeScript to run Function in content.
await chrome.scripting.executeScript({
target: { tabId: msg.tabId, frameIds: [msg.frameId] },
func:wrapperFunc,
});Advantage:
- Remote code works in content and background.
- Support to cross-domain iFrame.
Disadvantage:
- Need to open a popup window.
Solution 2: In Service worker, using chrome.debugger.sendCommand sends to content.
interface CDPRuntimeEvaluateResult {
exceptionDetails?: { exception?: { description?: string } };
result: { value: unknown };
}
await chrome.debugger.attach({tabId: msg.tabId}, "1.2");
const result = (await chrome.debugger.sendCommand(
{ tabId: msg.tabId },
"Runtime.evaluate",
{
expression: js,
}
)) as CDPRuntimeEvaluateResult;Advantage:
- No need to open a popup window.
Disadvantage:
- No support for cross-domain iFrame. Cross-domain iFrame’s target isn’t in the list of available debug targets, which uses the chrome.debugger.getTargets to get.
- Remote code can’t work in the background.
Summary
As cross-domain iFrame is a critical feature we need, Clicknium chooses Solution 1: use popup window to call chrome.debugger.sendCommand.
Manifest V3 has been supported by Clicknium since version 0.1.10. You can try it in pypi website and Clicknium VS Code Extension.
One thing that should be noticed is that the Clicknium browser extension has to work with the Clicknium Python SDK. If we publish the browser extension to Chrome Web Store, all the existing extensions will auto-upgrade to the latest version, and it can not work with the SDK before V0.1.0. So we have to pull it off shelves and wait for a period. If you want to try the M3 version, please upgrade to the latest Python SDK.
Tags:
Using Clicknium project packaging
October 14, 2022 · 2 min read
Will
Quick and easy project packaging with Clicknium
Tags:
Locator Store vs POM
September 29, 2022 · 3 min read
Justin
Page Object Model is a design pattern. It is popular in test automation for enhancing test maintenance and reducing code duplication.
Similar to POM in design, Clicknium locator store enables developers to manage the locator independently of the code (test case code or other automation project). While Clicknium locator store also enables locators to be shared among people and projects.
Tags:
Generate reliable locators for web UI testing or data scraping
September 28, 2022 · 4 min read
Justin
If you are spending a large chunk of development time finding elements on a web page, such as input, checkbox, submit button, table, and divs. It can be challenging to locate the correct elements, especially when they lack unique attributes, such as id, name, and class names; it is even worse if the attributes are dynamic.
This blog shares ways to generate reliable locators with features provided by Clicknium.
Tags:
Java application automation by Clicknium
September 28, 2022 · 2 min read
Will
Background
Java is an extremely popular programming language that is used to create many desktop GUI applications. It has become a typical demand to automate these Java GUI applications. Clicknium is perfectly suited to the task.
This blog takes SwingSet.jar as an example to introduce how to use Clicknium to automate Java application.
Tags:
3 Ways of Attaching to Existing Browsers in Web Automation
September 27, 2022 · 5 min read
Robin
For partial web automation that part of the work is prepared by human, and part of the work is automated. Attaching to existing browser is then required in this case.
This blog introduces 3 ways of attaching to browser by different automation libraries.
Tags:
Scrape images from the website
August 6, 2022 · 5 min read
Robin
This article describes how to extract numerous related images from a website. It is a basic necessity to download all images from a web page and store them to a local folder. It is also a necessary to preserve the titles of the images so that they may be managed or processed more easily in the future.
Tags:
Clicknium vs RPA
July 26, 2022 · 12 min read
Robin
RPA is in full boom these days, more and more individuals and organizations start adopting RPA in different areas to replace repeatitive tasks by leveraging different automation technologies. Based on this, we are comparing Clicknium to RPA, especially from development perspective, to understand what could be a better option in accomplishing automation tasks in different use scenarios.
Tags:
Clicknium vs AutoIt
July 26, 2022 · 4 min read
Dicey
There are several frequently used tools on the market for Windows GUI automation, such as AutoIt, AutoHotKey, Squish, and among others.
Let me introduce a powerful new tool Clicknium that is great for GUI automation for all types of applications, including Windows GUI, Web, SAP, JAVA as well as image recognition.
Tags:
Clicknium vs Pywinauto
July 26, 2022 · 2 min read
Will
There are much clerical works in desktop client side which are repeatitive tasks, like filling forms, sending messages to multiple groups, manipulating ERP system etc. These operations cost much of employees’ time, and automating these client side software to reduce human time in these operations is a good alternative.
There are some options to do the software automation, and I will introduce Pywinauto and Clicknium as below.
Tags:
