Defined path desktop actions in AI Desktop Actions
Desktop actions enable you to automate repetitive tasks on your desktop and web applications. This capability helps you streamline repetitive tasks, improve efficiency, and integrate desktop application workflows into your ServiceNow processes.
Desktop actions are tools that AI agents use to interact with web and desktop applications. When you configure an AI agent and select desktop action as a tool, you define whether the AI agent follows a defined path (fixed steps designed in AI Desktop Actions) or an adaptive path (high-level goal described in the tool configuration). Unlike adaptive path desktop actions that dynamically plan execution, defined path desktop actions enable AI agents to execute preconfigured workflows through a fixed sequence of steps.
Types of defined desktop actions
AI agents use these desktop actions to interact with desktop applications, perform UI-based tasks, and automate end-to-end workflows. There are two types of desktop actions that can be added as tools to AI agents:
On-screen task
These desktop actions help you simulate humans interacting with UI elements on your thick client applications, legacy systems, or SaaS applications without APIs. These actions include clicking buttons, typing into text boxes, selecting from dropdown menus, and more. These desktop actions make it easier to automate repetitive tasks. They encapsulate repeatable UI interactions, such as screens, anchors, and steps. You can create, manage, and test your desktop actions using the no-code Design workspace.
Example: Filling in fields on a payroll application, updating inventory in a point-of-sale system, or submitting a request through a desktop insurance claims app.
You can access and view the existing desktop actions in the ServiceNow instance by navigating to All > AI Desktop Actions > Desktop Actions. You can see the details of related desktop action applications, AI agents, and desktop action execution records for on-screen task, and AI agents and desktop action execution records for background desktop actions.
Background task
These desktop actions include prebuilt connectors that enable your AI agents to interact with various applications and system components in the background. These connectors streamline automation by offering actions for common tasks, reducing the need for complex scripting. Each connector focuses on a specific application or system area, providing a collection of related methods. You can't create background task desktop actions.
Example: Reading data from Microsoft Excel, extracting emails from Microsoft Outlook, generating a PDF, compressing files into a ZIP, fetching records from a database, or sending a system notification.
Background task desktop actions are supported for the following applications:
Note: For the AI agents to perform Background task automations, the following applications must already be installed on the end-user's system.
| Application | Description |
|---|---|
| Microsoft Excel | Enables AI agents to perform standard actions on the Microsoft Excel documents. For example, data manipulation, content modification, and information retrieval from spreadsheets.The following methods are supported:
|
| Microsoft Outlook | Enables AI agents to perform standard actions on the Microsoft Outlook application. For example, you can automate sending or replying to emails.The following methods are supported:
|
| Microsoft Word | Enables AI agents to perform standard actions on the Microsoft Word documents. For example, you can replace text in a document.The following methods are supported:
|
Enables AI agents to perform standard actions on the PDF documents. For example, extracting text, converting to Excel or Images, and merging and spiting files.The following methods are supported:
| |
| PowerShell | Enables AI agents to execute PowerShell commands and scripts and returns the results.The following methods are supported:
|
| SQL | Enables AI agents to execute custom SQL queries against supported databases \(MSSQL, ORACLE, MYSQL, OLEDB, ODBC\) and returns results as JSON.The following method is supported: ExecuteQuery |
| Secure Shell \(SSH\) | Enables AI agents to securely connect to remote servers to run SSH commands and scripts; and returns the execution results. This connector supports non-interactive commands.The following method is supported: RunCommand Note: Each SSH step runs in a new session. Output from one step isn’t carried over to the next step. |
| System Actions | Enables AI agents to perform standard Windows system operations. For example, starting an app, creating a ZIP file, or deleting any file or folder.The following methods are supported: - StartApp - Terminate - DateTimeNow - SetEnvironmentVariable - GetEnvironmentVariable - CopyFileOrFolder - DeleteFileOrFolder - WriteToFile - ReadFromFile - GetFilesFromFolder - ExtractFile - CreateZip |
Ways to create desktop actions
Note: If your automation requires manual inputs, such as entering an OTP or CAPTCHA, you must provide instructions to the AI Agent to wait for user input during execution. Otherwise, the automation can't proceed.
You can create on-screen desktop actions in the following ways:
Auto-capture screens and steps
With Action recorder, you can record your interactions with desktop applications to create automated workflows. It records every step you take, including clicks, keystrokes, and data entry, along with visual and contextual information. By recording steps, you can automate tasks that replicate your interactions. You can save the recorded screens and steps as a reusable desktop action. For more information, see Action recorder in AI Desktop Actions.
Manually capture screens and steps
You can manually take a screen capture, add anchors and steps to automate a series of steps you perform on your computer, such as clicking buttons, typing text, or interacting with different applications, and then save this sequence as a reusable desktop action.
Benefits of desktop actions
- Eliminate manual effort by automating repetitive tasks for frequent desktop activities.
- Execute complex sequences of steps quickly and consistently.
- Connect desktop application processes with your broader ServiceNow automations.
- Standardize task execution to minimize human error.
- You can record steps directly on your desktop applications.
- Automate tasks that span across multiple desktop applications.
- Save recorded sequences as desktop actions to use them in various automations.
- Review, rearrange, edit, or delete individual steps within a recorded action.
How the Design workspace works
Record steps
Capture your interactions with desktop applications, including clicks, text input, and screen navigation.
Define steps
Specify individual steps like Click, Set Text, or Get Text to interact with elements on your screen.
Application details
Select and store the desktop applications involved in your automations in the Details tab.
Organize screens
Structure your recorded steps by application screens, making it easy to review and edit.
Save and publish
Store your desktop actions in your ServiceNow instance for reuse and deployment.
What you can automate
Desktop actions support a wide range of desktop application interactions, including:
- Entering data into forms.
- Extracting information from applications.
- Navigating through application menus and screens.
- Performing steps like saving files or generating reports.
- Transferring data between different desktop applications.
Related topics
AI Desktop Actions Design workspace
Action recorder in AI Desktop Actions
Automate repetitive tasks by auto-capturing steps in AI Desktop Actions
Extend a desktop action by manually capturing steps in AI Desktop Actions
Add details to desktop actions in AI Desktop Actions
Test and activate a desktop action in AI Desktop Actions
Add a defined desktop action tool to an AI agent for desktop and web-based task