Create an Azure Data Factory metadata collector
Create a collector to import metadata from Azure Data Factory.
Before you begin
Before you begin, verify the following:
- A MID Server is setup for the collectors. For more information, see MID Server for metadata collectors.
- All per-requisite tasks are completed. For more information, see Prepare to run the Azure Data Factory collector.
- Role required: connection-admin
Procedure
Navigate to All > Workflow Data Fabric > Workflow Data Fabric Home.
Select the Connect Hub
Connect Hub icon icon in the left sidebar.
Select Create > Metadata collector.
From the System list, select Azure Data Factory.
From the Connection type list, select one of the following:
Select New connection to configure a new connection.
Select Existing connection to reuse an existing connection and select an existing connection from the Connections list.
The configuration form is filled with details from the existing connection. The name is appended with the word Copy and sensitive details like password aren't copied.
On the form, fill in the fields.
Field Description Connection name Unique identifier for the connection. This field can't be modified once the connection is established. Short description Purpose and details of the connection. Enter the Azure Data Factory authentication details.
Field Description Microsoft Entra client ID Azure Active Directory application client ID for the Azure Data Factory app. Microsoft Entra client secret Azure Active Directory application client secret for the Azure Data Factory app. Azure subscription ID The subscription ID in which the data factories are available. Microsoft Entra Tenant ID Azure Active Directory application tenant ID for the Azure Data Factory app. Enter the Azure Data Factory configuration details.
Field Description Include Data Factory name(s) Names of data factories to catalog. Enter either the exact name or a regular expression to match. List one data factory name per line. Exclude Data Factory name(s) Names of data factories to exclude from cataloging. Enter either the exact name or a regular expression to match. List one data factory name per line. Configure the advanced options.
| Field | Description |
|---|---|
| Max retries | The number of times the system retries a failed API call.Default: 5 |
| Retry delay | The number of seconds to wait between retry attempts for a failed API call.Default: 2 seconds |
- Select Save.
Result
The metadata collector is created and appears on the Connectors page with a Configured status. It is now ready to connect to the source system and harvest metadata.
What to do next
After creating the collector, you can perform any of the following tasks:
- Run the collector manually to harvest metadata immediately. See Run metadata collectors manually.
- Automate metadata collection by scheduling regular collector runs. See Schedule metadata collector runs.
- Monitor execution status and troubleshoot issues by viewing the runtime logs. See View runtime logs for collector runs.
- Discover and evaluate the harvested data assets in the Data Catalog. See Governing the Data Catalog.
Parent Topic:Azure Data Factory metadata collector