Run AskUI Workflows with Pipedream for Smoke Testing

Dylan Pierce shared an interesting approach to smoke testing at our meetup: Fill out a contact form with the help of Computer Vision & Large Language Models. Especially the vision part was impressive but it lacked one specific feature: Interaction with the form. How can you be sure your form works if you do not try to fill it out and send it?

This is where I thought an AskUI integration into the Pipedream workflow could be useful. AskUI uses visual selectors instead of code selectors and can interact like a human with a form like that. Giving us arealsmoke test!

In this blog I will describe how I integrated AskUI into a Pipedream workflow, so we get the benefit of visual selection and user interaction.

Prerequisites

• AskUI Controller installed and configured on a remote accessible system like Gitpod or a cloud machine. You can use our Try-Out-Repository or install on your own system (Windows, Linux, macOS).
• Pipedream account and a workflow. Follow their Introduction

What Will We Build? Aka the Use Case

Dylan Pierce showed an awesome use case with Pipedream and Puppeteer, where he implemented a smoke test with the help of AI without writing any selectors by himself. I highly recommend to watch the recording: https://youtu.be/o6hej9Ip2vs

Our Use Case: Smoke Test with Visual Selectors

Dylans use case involved querying a Large Language/Multimodal Model to implement the smoke test. We will modify this a little bit to use the visual selectors from AskUI, which do not rely on the specific UI-technology but identify elements through their appearance with an AI vision model.

Here are the steps we will implement:

• Trigger the smoke test once per day
• Perform the smoke test with AskUI on a remote system
• Send an email if the smoke test was successful or not

1. Add a Trigger

The first thing Pipedream wants us to add is a trigger. We add aScheduletrigger that will run our Workflow everyday at 9:00 AM.

2. Prepare a Custom Code Action

AskUI does not have an action in Pipedream. So we will make use of the AskUI node package and run a custom code action. For this we a a _Codeaction. In this action we will fill out the simple authentication from https://authenticationtest.com/simpleFormAuth/. If we see the success page we will send

We have to do the following steps:

• Add a propertyuiControllerUrl, so we do not have to hardcode it into the code
• Store our credentials in environment variables
• Import UiControlClient from the askui node package
• Configure the AskUIUiControlClientto use the credentials and theuiControllerUrl

The first thing we want to add to our custom code is theuiControllerUrl. ClickRefresh fieldsso theConfiguretab shows up at the start of the action. Here is the relevant code snippet.

... export default defineComponent({ props: { uiControllerUrl: { type: "string" } }, async run({ steps, $ }) { ...

Then head over to environment variables and add yourworkspaceIdandaccessTokenthere. You obtained those by following the prerequisites instructions above. You can now setup the connection to the AskUI Controller over theUiControlClientlike this. Notice how easy it is to use arbitrary node-packages in Pipedream? I only needed to importUiControlClientand it just worked ?.

Note: The following code also contains the teardown.

import { UiControlClient } from 'askui'; ... async run({ steps, $ }) { const result = {}; const aui = await UiControlClient.build({ credentials: { workspaceId: process.env.workspaceId, token: process.env.token, }, uiControllerUrl: this.uiControllerUrl }); await aui.connect(); // AskUI Workflow will be added here aui.disconnect(); return result; }, })

3. Write the AskUI Workflow

When you look at our example form, we want to fill out, you notice that we have to do the following things:

• Write simpleForm@authenticationtest.com into the textfieldE-Mail Address
• Write pa$$w0rd into the next textfield
• Click the button Log In
• Validate the success

The less inference we invoke the faster the AskUI workflow will execute. Everything that prompts AskUI to search for an element on the screen invokes inference. So let us try to invoke the inference only once by finding the first textfield for the E-Mail Address. Then we will use keypresses to navigate the form. This is the code to achieve this:

// This line only works with the Gitpod setup shown in the next section // Select the browser url textfield for your use case with the appropriate instruction! await aui.typeIn('https://authenticationtest.com/simpleFormAuth/') .textfield() .contains() .text() .containsText('docs.askui') .exec(); await aui.pressKey('enter').exec(); // Fill out the form await aui.typeIn('simpleForm@authenticationtest.com') .textfield() .contains() .text('E-Mail Address') .exec(); await aui.pressKey('tab').exec(); await aui.type('pa$$w0rd').exec(); await aui.pressKey('tab').exec(); await aui.pressKey('enter').exec(); // Check if the the login succeeded: Login Success is shown on the page // Pass result to next step try { await aui.expect().text('Login Success').exists().exec(); result.success = "Smoke Test successful!"; } catch(error) { result.success = "Smoke Test failed!"; } aui.disconnect(); return result;

4. Send an Email

Doing a smoke test without reporting about its success state would not help us. So we will just send us an email with theSend Yourself an Emailaction.

As subject we chooseSmoke Test Stateand for the text we reference our success variable we returned in our action above with {{steps.code.$return_value.success}}.

Gitpod As Remote Machine

If you do not have a remote machine ready-to-go you can use a service like Gitpod. We have a prepared Try-Out-Repository which comes with AskUI already setup and a VNC to observe the AskUI workflow. Start the repository in Gitpod over theOpen in Gitpod-button and let it finish the predefined AskUI workflow. When it reached the AskUI Docs (docs.askui.com) maximize the browser window in theSimple Browsertab.

Switch to theTERMINALtab and start the AskUI-Controller with the following command:

./node_modules/askui/dist/release/latest/linux/askui-ui-controller.AppImage

Also make sure that you expose the port to the AskUI Controller (open lock icon). Head to thePORTStab and make the port 6769 public. Then copy the URL and add it as the propertyuiControllerUrlin the Pipedream action.

Conclusion

Building a smoke test with Pipedream and AskUI was a practical use case to see how both tools integrate. The simplicity of Pipedream and its ability to integrate JavaScript code and Node packages was helpful. With that AskUI could be setup seamlessly inside an action and connected to an external AskUI Controller.

The above is the detailed content of Run AskUI Workflows with Pipedream for Smoke Testing. For more information, please follow other related articles on the PHP Chinese website!