Back to interview questions
By
Team Sherlock.sh
Hiring an Azure Data Factory engineer requires more than recognizing pipeline activities. Recruiters need to distinguish candidates who can copy a table from engineers who can explain a missed update, an integration runtime bottleneck, a failed deployment, or a replay that duplicates production data. Practical discussions should connect orchestration choices to correctness, security, and operational evidence.
This guide to 100+ Azure Data Factory Interview Questions and Answers contains 120 questions across beginner, intermediate, and experienced levels. It covers pipelines, activities, datasets, linked services, integration runtimes, Copy activity, Mapping Data Flows, expressions, triggers, incremental ingestion, recovery, networking, CI/CD, monitoring, performance, and cost. Azure Data Factory, Azure Synapse pipelines, and Data Factory in Microsoft Fabric are treated as related but distinct services.
How can recruiters detect potential AI cheating, deepfakes, impersonation, and unauthorized assistance during Azure Data Factory interviews?
A candidate may describe a watermark pipeline fluently yet struggle to explain what happens when a sink commits and the checkpoint update fails. Ask for a concrete ADF expression or recovery walkthrough, then change a network, concurrency, or late-arrival constraint to see whether the reasoning remains consistent.
Sherlock AI helps recruiters detect potential AI-assisted cheating, deepfakes, candidate impersonation, and unauthorized real-time assistance during Azure Data Factory interviews.
Combine integrity signals with a focused pipeline or code review and ask which claims require testing against the actual connector and runtime. Review signals in context rather than treating any single indicator as proof of misconduct.
Azure Data Factory Interview Questions
Choose questions for the role and workload. State the service, connector version, integration runtime, source system, and target consistency requirements before evaluating implementation-dependent answers.
Azure Data Factory Interview Questions for Beginners
Establish core orchestration concepts, object relationships, runtime selection, expressions, control flow, security, and monitoring.
What is Azure Data Factory, and what problem does it solve?
Is Azure Data Factory a database or a transformation engine?
How does Azure Data Factory differ from Azure Synapse pipelines and Microsoft Fabric Data Factory?
What does ADF v2 mean, and how should you discuss versions?
What is a pipeline in Azure Data Factory?
What is an activity, and what categories are available?
What is a linked service?
What is a dataset, and how does it differ from a linked service?
How do a pipeline, activity, dataset, linked service, and integration runtime work together?
What is an integration runtime, and which types does ADF offer?
When would you choose Azure integration runtime?
When would you choose a self-hosted integration runtime?
What is Azure-SSIS integration runtime used for?
What does the Copy activity do?
Does Copy activity support every connector in both directions?
How do file formats and compression affect copying data?
What is the difference between explicit column mapping and automatic mapping?
What are Mapping Data Flows?
How do Mapping Data Flow expressions differ from pipeline expressions?
What is the difference between ETL and ELT in an ADF solution?
What is a pipeline parameter, and how do you reference it?
How are pipeline variables different from parameters?
How do ADF expressions and string interpolation differ?
How do you reference the output of a preceding activity?
How do you construct a date-based folder path?
What does the Lookup activity return?
What is Get Metadata used for?
How do If Condition and Switch activities differ?
How do ForEach and Until activities differ?
How do activity dependency conditions control execution?
What is a trigger, and which common trigger types are available?
How does a schedule trigger differ from a tumbling window trigger?
What should you know before using a storage event trigger?
What is the difference between debugging and publishing a pipeline?
What does Execute Pipeline do?
How should a pipeline handle timeouts and transient failures?
How do managed identities and Azure Key Vault improve security?
Why does a successful linked-service connection test not prove a pipeline is production-ready?
How do you monitor and troubleshoot a failed pipeline run?
What are the main cost drivers in Azure Data Factory?
Azure Data Factory Interview Questions for Intermediate
Explore parameterized ingestion, incremental state, data-flow transformations, connector constraints, failure recovery, and promotion.
How do you parameterize one dataset for multiple folders or tables?
How do you build a metadata-driven ingestion pipeline?
How do you consume Lookup output without confusing its two shapes?
When should you use Get Metadata rather than Lookup?
How do you filter a directory listing before copying files?
How do you choose safe ForEach parallelism?
How do you handle nested iteration without shared-variable races?
How do you debug an expression that works in the editor but fails at runtime?
How do you preserve types when using dynamic expressions?
How do you construct dynamic SQL without introducing injection risks?
How do you implement a bounded watermark load?
How do you handle equal timestamps and late-arriving records?
When is a watermark insufficient for change capture?
How do you recover when data lands but the watermark update fails?
How do you prevent overlapping incremental runs from corrupting progress?
How do you make a file-ingestion retry idempotent?
How do you parameterize a Mapping Data Flow safely?
How do you handle schema drift without silently accepting bad data?
How do you deduplicate records deterministically in a data flow?
How do you choose between Join, Lookup, and Exists transformations?
How do you flatten nested JSON without accidentally multiplying rows?
How do you implement update, upsert, and delete behavior with Alter Row?
How do you separate malformed records from valid records?
How do you implement a slowly changing dimension Type 2 load?
How do you control multiple sink execution and consistency?
How do you tune a skewed Mapping Data Flow join?
How do you distinguish Copy mapping from data transformation?
How do you troubleshoot Copy throughput before increasing resources?
How do you choose an integration runtime for a mixed-network copy?
How do you paginate a REST source reliably?
How do you use activity retry policies safely?
How do dependency conditions affect the final pipeline status?
How do you poll a long-running operation without an unbounded loop?
How do you pass values between parent and child pipelines?
How do you design a replayable tumbling-window pipeline?
How do you process duplicate or early storage events?
How do you validate a new pipeline without assuming debug is harmless?
How do you promote parameterized artifacts across environments?
How do you diagnose a pipeline that succeeds with the wrong row count?
How do you implement reconciliation before marking a batch complete?
Azure Data Factory Interview Questions for Experienced
Use architecture, reliability, deployment, networking, performance, and cost scenarios to assess engineering judgment.
How would you design an enterprise ADF platform without creating one unmanageable factory?
How do you choose between Azure IR, managed virtual network IR, and self-hosted IR?
How would you design a highly available self-hosted integration runtime?
What does an ADF private endpoint protect, and what does it not protect?
How do managed identities and least privilege work in a production ADF design?
How would you manage and rotate secrets without exposing them in pipeline logs?
How do you troubleshoot a private-network pipeline that works in development but fails in production?
How do you separate developer, operator, and deployment privileges?
What should a production ADF Git and CI/CD workflow look like?
How do you parameterize deployments without turning every property into a parameter?
How do you deploy safely when triggers and running pipelines are active?
What does a safe rollback strategy look like for ADF?
How would you test ADF changes before they reach production?
How do you manage linked-service and connector upgrades?
How do you define an ADF service-level objective around data freshness?
How do you build actionable pipeline observability rather than noisy failure alerts?
How do you investigate a Copy activity whose runtime has doubled?
How do you tune Mapping Data Flows without simply buying larger compute?
How would you reduce ADF cost while preserving the delivery SLA?
How do you prevent parallel pipelines from overwhelming a source system?
Can ADF guarantee exactly-once delivery across arbitrary systems?
How do you prevent two runs from committing the same incremental load?
How would you recover from a partially committed batch across several target tables?
How would you process late-arriving or corrected records safely?
What do you do when a CDC consumer falls behind the source retention window?
How do you design a consistent snapshot plus change-stream bootstrap?
How do you handle deletes and slowly changing dimensions in a production ingestion design?
How do you design a replayable data lake ingestion pipeline?
How do you make a metadata-driven framework safe for hundreds of tables?
How would you enforce schema contracts without rejecting every additive change?
How do you quarantine invalid records without silently declaring success?
How do you protect personal data across ingestion, logs, and replay storage?
How do you establish useful lineage when ADF orchestrates external transformations?
How would you plan regional disaster recovery for ADF?
How do you run historical backfills without disrupting current ingestion?
How do you orchestrate a multi-step load that spans systems without a shared transaction?
When should ADF hand work to another compute or orchestration service?
How do you evaluate moving ADF pipelines into Azure Synapse pipelines?
How would you assess migration from ADF to Data Factory in Microsoft Fabric?
What evidence would convince you that an ADF platform is production-ready?
Azure Data Factory Interview Questions and Answers
These answers follow the question bank exactly and lead with direct responses candidates can adapt. Examples are illustrative fragments or local fixtures, not deployed pipelines. Confirm connector schemas, permissions, source behavior, and runtime capabilities before using them in an authorized environment.
Azure Data Factory Interview Questions and Answers for Beginners
These answers explain the ADF foundations while separating orchestration, data movement, transformation, and external compute.
1. What is Azure Data Factory, and what problem does it solve?
Azure Data Factory is Azure's managed data-integration service. I use it to coordinate ingestion, movement, and processing across data stores rather than hand-scheduling unrelated scripts. A typical pipeline copies operational data into a landing area, invokes a transformation, checks the result, and records completion.
Its value is the coordination around the data: connections, dependencies, parameters, triggers, and run history. For example, a retailer might ingest daily orders from a SQL database and supplier files from storage before preparing an analytics table. ADF does not make inconsistent source data correct automatically; the team still defines validation, ownership, and recovery rules. I would describe both the successful path and what should happen when one supplier file is missing.
2. Is Azure Data Factory a database or a transformation engine?
ADF is not the database that permanently holds my business records. It orchestrates work against storage systems and processing services. Some operations execute on its integration infrastructure, including Copy activity and Spark-based Mapping Data Flows; other activities submit work to external engines such as SQL or Azure Databricks.
That distinction matters when diagnosing a slow job. If ADF invokes a SQL statement, the database query plan and available database resources can determine the execution time; changing an orchestration setting does not automatically fix them. I would keep raw and curated data in an appropriate store, put transformations in the engine best suited to them, and use ADF to sequence and observe the overall workflow. Calling ADF only a scheduler would also understate its native movement and transformation capabilities.
3. How does Azure Data Factory differ from Azure Synapse pipelines and Microsoft Fabric Data Factory?
ADF is a standalone Azure integration service; Synapse pipelines provide related orchestration capabilities inside an Azure Synapse Analytics workspace. They share many concepts, but I do not assume complete feature parity: for example, Synapse pipelines support Azure and self-hosted integration runtimes, not Azure-SSIS IR.
Microsoft Fabric Data Factory belongs to the Fabric platform and integrates with its workspace and OneLake model. It uses connections rather than ADF's separate linked-service and dataset design, and its deployment and connectivity model differs. As of the October 2026 review, Microsoft documents Mapping Data Flow transforms in Dataflow Gen2 as a public preview; saying Fabric can never offer that authoring experience would be outdated.
In an interview, I would first identify which product the employer uses. Similar activity names do not establish identical JSON, connector support, security configuration, or migration behavior. A migration needs a feature-by-feature compatibility assessment rather than a service-name replacement.
4. What does ADF v2 mean, and how should you discuss versions?
ADF v2 refers to the Azure Data Factory service generation and resource/API model used for modern pipeline development. I would not describe Fabric Data Factory as simply “ADF v3,” or use a connector's version number as the version of the entire service.
There are several independent version concerns: the REST API version in deployment tooling, the connector version configured by a linked service, the installed self-hosted IR software, and any external compute runtime. Those can change independently. For example, upgrading a database connector can change authentication or type behavior while the factory still uses the same ADF v2 pipeline model.
My answer would identify the component being versioned, check its release and lifecycle documentation, and test the affected behavior before upgrading. A fixed statement that every ADF connector has the same latest version is not credible.
5. What is a pipeline in Azure Data Factory?
A pipeline is a named workflow that groups activities into one unit of work. Its definition describes what should happen; a pipeline run is one execution of that definition with a particular run ID, parameters, and outcome.
For example, an IngestOrders pipeline could copy a file, validate its record count, and call a child pipeline to load a reporting table. Independent branches can run in parallel, while dependency conditions order activities that need earlier results. I would choose pipeline boundaries around an operational responsibility, such as one ingestion process, rather than putting the entire platform in a single diagram.
Parameters make the same definition reusable for different dates or entities. A run's success should correspond to a meaningful completion criterion, not merely the fact that the first copy activity finished.
6. What is an activity, and what categories are available?
An activity is one executable step in a pipeline. ADF groups activities broadly into data movement, data transformation, and control flow. Copy moves data; a Data Flow activity executes a Mapping Data Flow; If Condition, ForEach, and Execute Pipeline control which work runs and how it is organized.
The activity type determines its configuration and where work happens. For example, a Stored Procedure activity invokes database-side logic; it does not execute that procedure inside the orchestration service. Some activities reference datasets, some reference compute connections, and some need neither.
I select an activity by its contract: inputs, outputs, failure behavior, and side effects. That prevents treating every box as equivalent or assuming every activity supports the same connectors and execution environment.
7. What is a linked service?
A linked service describes a connection to a data store or compute service. It holds connector-specific details such as an endpoint, database name, authentication configuration, and, where applicable, the integration runtime used for access. It is reusable across datasets and activities.
For example, an Azure SQL linked service identifies the database connection, while separate datasets can identify the Orders and Customers tables. I prefer supported managed-identity authentication where appropriate and use a Key Vault reference when a secret is required, rather than embedding a password in version-controlled JSON.
A linked service is not the data itself and does not automatically grant permissions. The selected identity still needs authorization at the destination, and the runtime needs a valid network path. A connection definition can therefore be syntactically valid but unusable at runtime.
8. What is a dataset, and how does it differ from a linked service?
A dataset describes the data an activity should access, such as a table or a file location and format. A linked service describes how to connect to the containing system. I think of the linked service as the connection and the dataset as the address and structure used through that connection.
For an ADLS Gen2 CSV source, the linked service might identify the storage account and authentication method. The dataset can specify the file system, folder, file name, delimiter, and schema-related settings. A parameterized folder lets one dataset represent daily partitions instead of creating hundreds of almost identical datasets.
A dataset is metadata, not an in-memory collection of rows or a copied file. Its declared schema also does not replace data-quality validation; actual incoming files can still violate the expected contract.
9. How do a pipeline, activity, dataset, linked service, and integration runtime work together?
The pipeline coordinates the workflow, an activity performs a step, datasets identify the relevant data, linked services supply connections, and the integration runtime supplies or dispatches the execution work. I would explain the relationship with one concrete transfer rather than memorized definitions alone.
Suppose LoadOrders copies a SQL table into a lake. Its Copy activity references a source table dataset and a destination file dataset. Each dataset points to a linked service. The runtime selected through the relevant configuration must reach both systems and support those connectors. A later SQL activity could then invoke a transformation in the warehouse.
This model helps isolate failures: a wrong table name is a different problem from an invalid identity, a blocked route, or incorrect orchestration dependencies. I would inspect the corresponding layer instead of recreating the whole factory.
10. What is an integration runtime, and which types does ADF offer?
An integration runtime, or IR, provides the execution infrastructure between ADF activities and the systems they access. ADF offers Azure IR, self-hosted IR, and Azure-SSIS IR. The choice depends on the operation, connector, network access, and whether existing SSIS packages must run.
Azure IR supplies managed cloud execution. A self-hosted IR runs on infrastructure the organization manages and can access systems reachable from that host. Azure-SSIS IR runs SSIS packages in Azure. These are not interchangeable worker sizes: a self-hosted IR does not execute Mapping Data Flow Spark transformations, and Azure-SSIS IR is not the standard runtime for ordinary Copy activities.
I would select the runtime after drawing the source-to-destination network path and checking connector support. The factory's metadata location alone does not explain where every data-processing operation executes.
11. When would you choose Azure integration runtime?
I choose Azure IR when managed Azure execution supports the activity, connectors, and network path. It avoids installing and maintaining my own runtime machines. Typical uses include cloud-to-cloud Copy activities and the Azure-managed compute associated with Mapping Data Flows.
For private access to supported services, I would evaluate Azure IR in an ADF managed virtual network with managed private endpoints rather than assume that a public endpoint must be opened. Private endpoints require the correct approval and target configuration; selecting a networking checkbox does not make access work automatically.
I also consider region placement, source and sink capacity, and the actual operation. Increasing Copy resources cannot overcome an exhausted source database or a throttled destination. The managed runtime removes host administration, not the need to design and measure the data path.
12. When would you choose a self-hosted integration runtime?
I choose a self-hosted IR when a supported activity needs connectivity or dependencies available on organization-managed infrastructure, such as an on-premises database reachable from a Windows host. “Self-hosted” describes who manages the runtime, not a requirement that the machine be physically on-premises; it can also run on an appropriately connected Azure VM.
The host needs compatible software, supported drivers where required, network access to the data systems, and the required outbound communication with Azure. My team owns patching, availability, capacity, and operational monitoring. For resilience, I would evaluate multiple nodes rather than a single employee's workstation.
It is not a way to run Mapping Data Flow Spark jobs locally. I would verify the connector and activity support before choosing it, and keep credentials and runtime registration material out of code repositories.
13. What is Azure-SSIS integration runtime used for?
Azure-SSIS IR provides an Azure execution environment for existing SQL Server Integration Services packages. It is useful when an organization wants to move an established SSIS workload without immediately rewriting every package as ADF-native activities or data flows.
I would assess package dependencies first: connection managers, custom components, drivers, package protection, access to source systems, and deployment model. Depending on the chosen model, packages may use an SSIS catalog or other supported package storage. A successful migration requires validating behavior and performance, not merely importing the package.
Azure-SSIS IR has its own sizing, startup, networking, and cost considerations. It does not automatically convert SSIS logic into Mapping Data Flows, and it is not the runtime used simply because the source happens to be SQL Server.
14. What does the Copy activity do?
Copy activity transfers data from a supported source to a supported destination. Depending on the connectors and formats, it can deserialize, map columns, convert supported types, and serialize the destination representation. For instance, I can read a relational table and write Parquet files without writing a separate export script.
I would not use Copy as a general replacement for multi-source joins, complex business rules, or arbitrary code execution. Those belong in an appropriate transformation engine or Mapping Data Flow. I also define what happens on repeat execution: appending to a table twice can duplicate data even if both runs report success.
Before considering the transfer correct, I check source and sink settings, mappings, rejected or skipped rows, output counts, and the agreed reconciliation rule. A green activity alone is not proof of business completeness.
15. Does Copy activity support every connector in both directions?
No. Connector support is specific to the operation, direction, integration runtime, and connector version. A connector might support reading with Copy but not writing, or support Copy without supporting Mapping Data Flow. Authentication options can also differ by activity and runtime.
I check the supported-capabilities matrix and the connector's current documentation before proposing a design. For a SaaS export, for example, I would verify whether the connector offers incremental extraction, pagination, query filtering, and the required authentication method. Seeing the service's name in a connector list is not enough.
I also distinguish preview, generally available, end-of-support, and removed versions. A previously working linked service is not evidence of indefinite support. Connector upgrades need representative tests for schema, type conversion, permissions, and expected output, not only a successful connection test.
16. How do file formats and compression affect copying data?
A file format defines how records and fields are represented, while compression changes how those bytes are stored. I configure both explicitly because a .csv name alone does not establish the delimiter, quote rules, encoding, header behavior, or compression codec.
For example, copying CSV to Parquet requires interpreting the source records and writing a different representation. A Binary dataset instead supports copying file content without interpreting individual columns; it is appropriate when I want to move a supported file as bytes rather than transform its records.
I verify the format-and-connector combination and test real edge cases, including quoted delimiters, nulls, embedded newlines, and timestamp precision. Compression can reduce storage and transfer volume, but its CPU and parallel-read characteristics affect performance. I would not claim every codec is supported for every source and sink.
17. What is the difference between explicit column mapping and automatic mapping?
Automatic mapping uses the supported default matching behavior, commonly matching source and sink column names for tabular data. Explicit mapping specifies which source field feeds which destination field. I prefer explicit mappings when names differ, the destination contract is controlled, or only selected columns should be loaded.
For example, a source customer_id can map to destination CustomerId; that is a structural mapping, not a business transformation that validates the customer's identity. Hierarchical sources require attention to paths and collections rather than assuming every JSON object is a flat table.
I test data types, nullability, precision, and truncation behavior as well as names. An apparently successful transfer can still be wrong if identifiers lose leading zeroes or decimal values are rounded unexpectedly. I document whether new source columns should be rejected, ignored, or explicitly incorporated.
18. What are Mapping Data Flows?
Mapping Data Flows are visually authored transformation graphs that ADF executes on managed Spark infrastructure. I use them for operations such as deriving columns, joining streams, filtering, aggregating, and writing transformed data to supported sinks.
The pipeline schedules and coordinates the Data Flow activity; the data-flow graph defines the row-level transformations. Those are separate responsibilities. A simple ingestion might need only Copy, while a customer-standardization process could justify a data flow that normalizes values and separates invalid records.
I still need to understand schemas, null semantics, partitioning, and sink behavior. Visual authoring does not remove distributed-processing concerns. Startup and compute costs also matter, so I would not choose a Spark-backed flow for every tiny transformation automatically. Debug previews help develop logic but do not prove production-scale correctness or throughput.
19. How do Mapping Data Flow expressions differ from pipeline expressions?
Pipeline expressions evaluate orchestration values, such as a file path or an activity setting. Mapping Data Flow expressions operate within the transformation engine, often against columns in each row. They are different expression contexts, even when function names look similar.
For example, in a Derived Column transformation, this data-flow expression converts a string column named amount to a decimal:
Plain-text example
For an input string 12.50, the intended result is a decimal value with two fractional digits. I would separately validate malformed or out-of-range input rather than assume conversion establishes data quality.
I do not paste @pipeline().parameters.x directly into a data-flow row expression and expect it to behave as a pipeline expression. Pipeline values must be passed through the activity's data-flow parameter configuration, then referenced using the data-flow parameter syntax.
20. What is the difference between ETL and ELT in an ADF solution?
ETL transforms data before loading the final target; ELT loads data first and transforms it in the destination processing environment. ADF can orchestrate either pattern. The difference is where the important transformation happens, not whether the pipeline uses a visual designer.
For ETL, I might clean records with a Mapping Data Flow before writing a curated table. For ELT, I might use Copy to load a staging table and then execute SQL inside the warehouse. I choose based on data size, available compute, governance, existing skills, and the need to retain raw history.
Both patterns still require validation, access control, lineage, and recovery design. Loading first is not permission to expose unrestricted raw data, and transforming first does not guarantee that invalid records or duplicate deliveries have been handled correctly.
21. What is a pipeline parameter, and how do you reference it?
A pipeline parameter is a named input whose value is fixed for that run. It lets the caller select a date, entity, or environment-specific value without modifying the workflow definition. I reference it through pipeline().parameters.
This is a pipeline-properties fragment, not a complete deployable factory:
JSON example
The following pipeline expression returns orders when the default is used:
Plain-text example
A trigger, manual run, or parent pipeline can supply a different value. I validate acceptable inputs before using them in a file path or object selection. Parameterization improves reuse, but it is not a substitute for authorization or a safe way to concatenate untrusted SQL.
22. How are pipeline variables different from parameters?
Parameters are inputs that remain fixed during a run; variables hold mutable pipeline state. I use a parameter for the requested business date and a variable for a small intermediate value needed by later control-flow steps.
For example, after defining a string variable named status, a Set Variable activity can assign ready, and @variables('status') reads it. Variables are scoped to the pipeline run, not independently to each parallel ForEach iteration. Concurrent writes to shared state can therefore produce races.
My preference is to pass immutable parameters into child pipelines when iterations need isolated state. Variables also should not become an improvised database: they do not provide a durable transaction log or survive as reliable business state across unrelated runs. Persistent checkpoints belong in a suitable external store.
23. How do ADF expressions and string interpolation differ?
An expression occupying the whole value can preserve its result type; string interpolation produces text. That distinction matters when an activity property expects a number, Boolean, array, or object rather than a string.
For a pipeline integer parameter batchSize set to 10, compare:
Plain-text example
The first evaluates to the numeric value 10; the second evaluates to the string "10". I use interpolation when building text, such as batch-@{pipeline().parameters.batchSize}, whose result is batch-10.
A literal leading at-sign is escaped with @@ where ADF expression parsing applies. When debugging a dynamic setting, I inspect both the value and its type rather than assuming that identical-looking output means identical JSON. The expression editor helps author values but does not validate every downstream service's expectations.
24. How do you reference the output of a preceding activity?
I reference a previous activity by its name and then navigate its documented output shape. For example, if a Lookup named ReadConfig uses first-row mode and returns an entity field, this expression reads that field:
Plain-text example
For a returned row { "entity": "orders" }, the result is orders. That result is not available merely because both activities appear on the canvas: I add an appropriate dependency so the consuming activity runs after the producing activity succeeds.
I also handle the absence of a row or field intentionally. The output contract differs by activity and configuration, so I inspect a representative output rather than guess property names. Secrets and sensitive business values should not be casually copied into logs just to simplify troubleshooting.
25. How do you construct a date-based folder path?
I construct the folder from an explicit business-date parameter rather than repeatedly evaluating the current clock. That makes a rerun for an older day target the same logical partition instead of today's folder.
For a string parameter businessDate containing 2026-10-01T00:00:00Z, this pipeline expression produces orders/2026/10/01:
Plain-text example
The caller should provide a validated timestamp with an agreed timezone. I use UTC for technical boundaries unless the business requires a named local-calendar definition, and I make daylight-saving behavior explicit when local dates matter.
This expression only creates a path; it does not guarantee that a partition is complete or prevent duplicate writes. I keep naming, completeness checks, and rerun behavior as separate design decisions.
26. What does the Lookup activity return?
Lookup retrieves a small result used to drive orchestration, such as one configuration row or a list of tables. With firstRowOnly enabled, consumers use the firstRow object. When disabled, they consume the value array, commonly as the input to ForEach.
I would use Lookup to read a control table containing a few ingestion definitions, not to transfer millions of business records between services. Its documented limits include 5,000 returned rows and a 4 MB output limit; a query or stored procedure must return exactly one result set.
For a control table, I select only the fields needed by the workflow, define behavior for an empty result, and check that the list has not been truncated. Copy or a transformation activity is the appropriate tool for the actual bulk data movement.
27. What is Get Metadata used for?
Get Metadata returns supported properties about a dataset or object rather than reading its business rows. Depending on the connector, it can inspect fields such as existence, size, last-modified time, structure, or a folder's child items.
For a supported file dataset, I can request the exists field and branch on:
Plain-text example
If the file is missing, requesting exists allows the activity to return false instead of treating that absence as an activity failure. I still define whether absence means “skip,” “wait,” or “fail the business process.”
The supported metadata differs across connectors, and output is limited to 4 MB. A directory listing is not an unlimited recursive inventory. File existence or a nonzero size also does not prove the producer has finished delivering all expected records.
28. How do If Condition and Switch activities differ?
If Condition chooses between two branches based on a Boolean expression. Switch chooses a matching named case from a string-valued expression and can use a default branch when no case matches. I choose the construct that makes the business decision easiest to understand.
An If Condition might test whether an inspected file exists. A Switch might route a sourceType parameter to sql, file, or a default rejection branch. For a Boolean parameter named fullLoad, the If expression can be:
Plain-text example
A value of true selects the true activities; false selects the false activities. I do not use the literal string "false" where a Boolean is required. I also make unexpected inputs visible rather than silently sending them through a convenient default ingestion path.
29. How do ForEach and Until activities differ?
ForEach iterates over an array of known items; Until repeats its body until a Boolean condition becomes true or its configured timeout is reached. I use ForEach for a list of tables and Until for bounded polling of an asynchronous job.
Inside ForEach, @item() identifies the current element. If the array contains orders and customers, each iteration can pass one of those values into a reusable child pipeline. Parallel iterations require independent destinations or properly controlled shared resources.
Until has do-until behavior: the body runs before the termination condition is evaluated. I include a Wait when polling, a realistic timeout, and explicit treatment of failure states. A failed inner activity does not automatically end an Until loop, so I must design the condition and failure handling instead of assuming a failed request stops repeated work.
30. How do activity dependency conditions control execution?
Dependencies tell ADF which upstream outcome permits a downstream activity to run. The standard conditions are Succeeded, Failed, Skipped, and Completed; Completed covers success or failure, not a skipped activity. I use them to separate normal processing, failure handling, and cleanup paths.
For example, a transformation should depend on a successful copy, while a diagnostic step may depend on copy failure. This fragment belongs inside a downstream activity:
JSON example
It means the downstream activity is eligible after CopyOrders succeeds. I also inspect the overall pipeline outcome: adding a successful error-handling branch can change whether the pipeline reports failure. If the business operation failed, I preserve that outcome deliberately rather than letting a notification activity make the run look healthy.
31. What is a trigger, and which common trigger types are available?
A trigger determines when a pipeline run starts and can supply its parameter values. Common ADF trigger types include schedule, tumbling window, storage event, and custom event triggers. Pipelines can also be run manually or invoked by another pipeline without a recurring trigger.
I choose the trigger based on the business event. A daily reporting refresh may suit a schedule; processing hourly time intervals with state and dependencies may suit tumbling windows; reacting to an arriving storage object may suit an event trigger. Starting a run does not establish that all required input data is complete.
I also distinguish the trigger run from the pipeline run when troubleshooting. An event-subscription or scheduling problem may prevent a pipeline from starting at all, while a successful trigger can launch a pipeline that subsequently fails. Both layers need operational visibility.
32. How does a schedule trigger differ from a tumbling window trigger?
A schedule trigger starts work at configured wall-clock times. A tumbling window trigger tracks a sequence of fixed, nonoverlapping, contiguous time windows and maintains state for those windows. Tumbling windows support features such as window dependencies, retries, and backfill scenarios.
For example, an hourly aggregation can use explicit start and end boundaries for each hour instead of assuming the actual run-start time represents the data interval. If a run is delayed, it should still process its assigned interval. A tumbling window trigger references one pipeline; it is not simply a general schedule with a different name.
I choose a schedule for straightforward recurring work and tumbling windows when the interval itself is an important processing unit. Neither choice automatically makes sink writes exactly-once: rerunning a window still requires safe, repeatable loading behavior.
33. What should you know before using a storage event trigger?
A storage event trigger reacts to supported Azure Storage events through Event Grid. It is useful for file-arrival workflows, but it is not a universal watcher for every filesystem or storage service. I verify the supported account type, event filters, subscription permissions, and network requirements first.
The event identifies an object; the pipeline's identity still needs permission to read its content. I pass the relevant event path into parameters and validate that it belongs to the expected landing area. Producer conventions also matter: a collection of files may require a separate completion marker rather than processing the first arriving file as the whole batch.
I design repeat delivery and reruns to be harmless and avoid assuming an event establishes completeness or ordering. Event-driven orchestration reduces polling, but the data contract remains the team's responsibility.
34. What is the difference between debugging and publishing a pipeline?
Debug runs let me exercise current authoring changes without first publishing them as the live pipeline definition. Publishing makes the intended definitions available in the live service; saving a Git branch is not the same as publishing that content.
Debug is not a dry-run guarantee. Activities can contact real systems, copy or delete data, invoke external jobs, and incur charges. I therefore use isolated test inputs, restricted identities, and nonproduction destinations. For a destructive step, I validate the intended target before allowing execution rather than trusting the word “debug.”
I also distinguish pipeline debugging from a Mapping Data Flow debug session, which can start compute for interactive development. Before release, I validate the deployed definition and trigger configuration through an authorized test process. A successful editor preview alone does not establish production readiness.
35. What does Execute Pipeline do?
Execute Pipeline invokes another pipeline and passes parameter values to it. I use it to separate reusable units of work from their orchestration, such as a parent that iterates over table definitions and a child that ingests one table.
When the parent must depend on completion, I use the activity's waitOnCompletion behavior rather than treating submission of a child run as proof that its data is ready. The default is to wait, but I verify the actual definition. If the parent does not wait, downstream steps need another valid completion mechanism.
A child pipeline is also a useful boundary for per-invocation state, operational ownership, and testing. I avoid passing large datasets through parameters: parameters carry configuration and small control values, while bulk records should remain in a suitable data store.
36. How should a pipeline handle timeouts and transient failures?
I set bounded timeouts and retry only failures that might succeed without changing the request, such as a temporary service interruption. Authentication errors, invalid queries, and malformed data normally need a fix, not repeated execution.
For an activity type supporting execution policy, this fragment allows two retries after the initial attempt, with a 30-second retry interval:
JSON example
The timeout shown is one hour; this is an activity-policy fragment, not a complete pipeline. I still account for the overall workflow deadline and external service behavior.
Retries are safe only when repeated side effects are controlled. If a request commits data but its response is lost, another attempt can duplicate writes. I design idempotent loading and retain enough error context to distinguish a transient failure from partial completion.
37. How do managed identities and Azure Key Vault improve security?
A managed identity lets supported connections authenticate without my team storing a long-lived password in the pipeline. Azure Key Vault provides controlled storage for secrets that are still needed, such as credentials for a system without a suitable identity-based option.
I grant the identity only the required data permissions and scope. The factory's management-plane permissions are not automatically permission to read every storage container or database table. For a Key Vault reference, the relevant identity needs authorization to retrieve the required secret, and the selected connector must support the intended authentication pattern.
Neither mechanism bypasses firewalls, private-endpoint approval, or DNS requirements. I keep sensitive activity inputs and outputs out of ordinary operational logs where appropriate, while remembering that masking in ADF does not prevent an external service from logging the same secret.
38. Why does a successful linked-service connection test not prove a pipeline is production-ready?
A connection test establishes only a limited connectivity and authentication result for the tested configuration. It does not prove that the actual run can read every required partition, write the target schema, handle production volume, or recover after a partial failure.
I would next test the real operation with representative nonproduction data and the intended runtime and identity. For a database load, I check table-level permissions, write behavior, schema compatibility, transaction expectations, and the repeat-run result. For a private endpoint, I verify the runtime's actual route and name resolution rather than relying on my workstation's access.
Production readiness also includes monitoring, alerts, ownership, secret rotation, deployment controls, and recovery evidence. I would describe a green connection test as one useful signal, not an end-to-end acceptance test or a guarantee of network reachability in every execution context.
39. How do you monitor and troubleshoot a failed pipeline run?
I start with the pipeline run ID, its parameters, time range, and failed activity. Then I inspect the activity error, relevant input and output, duration, and upstream dependencies to determine whether the issue is orchestration, connectivity, authentication, data, or the external processing engine.
For a Copy failure, I would inspect source and sink details and the runtime involved. For a SQL transformation, I would correlate the ADF run with database-side diagnostics rather than assuming ADF contains the complete cause. I avoid dumping sensitive payloads into a ticket while collecting evidence.
After fixing the cause, I evaluate partial side effects before rerunning. For sustained operations, I configure appropriate Azure Monitor diagnostics and alerts instead of relying only on someone opening the Monitor interface. A useful incident record includes the failed business interval and the recovery outcome, not just a screenshot of a red activity.
40. What are the main cost drivers in Azure Data Factory?
ADF cost comes from the work performed, not just the existence of a factory. Important drivers include orchestration and activity runs, data movement, transformation compute, and development or monitoring operations. Mapping Data Flow compute and Azure-SSIS runtime usage have different billing characteristics from a small control-flow activity.
The complete solution can also incur storage transactions, network transfer, database or external compute charges, and infrastructure costs for self-hosted IR machines. I would estimate a representative workload using the current regional pricing and validate the result against observed billing rather than quote one universal price per pipeline.
To reduce waste, I remove unnecessary polling, avoid repeated full loads where correct incremental processing is possible, stop unneeded debug sessions, and measure resource changes against elapsed time and throughput. The cheapest-looking configuration is not cheaper if it repeatedly fails or doubles processing time.
Azure Data Factory Interview Questions and Answers for Intermediate
These answers emphasize implementation details, bounded workloads, explicit state, and the failure cases hidden by a successful activity status.
41. How do you parameterize one dataset for multiple folders or tables?
I declare dataset parameters for the parts that vary and pass their values from the consuming activity. A dataset parameter describes an input to that dataset; it does not automatically inherit a pipeline parameter with the same name. The linked service still defines the connection, while the dataset defines the selected location or object.
For example, the following is the expression for a parameterized folder property, followed by the expression used when passing a pipeline parameter into the dataset:
Plain-text example
If sourceFolder is orders/2026/10, the activity binds that folder to the dataset. I validate permitted paths or table names against configuration rather than accepting arbitrary user input. I also keep source and sink parameters separate so a mistaken binding cannot overwrite the input. This makes reuse explicit without hiding environment-specific connections inside string-building logic.
42. How do you build a metadata-driven ingestion pipeline?
I separate the ingestion engine from its configuration. A control table identifies approved source objects, destination locations, load modes, keys, and checkpoint state. A Lookup retrieves a bounded set of enabled entries, a ForEach iterates over those entries, and a child pipeline processes one object with explicit parameters.
The orchestration should record both a logical batch identifier and the ADF run identifier. The logical identifier remains stable across retries; the run identifier distinguishes execution attempts. I validate configuration before starting a copy, limit concurrency to protect the source, and record per-object outcomes rather than treating one successful activity as proof that every table loaded.
I do not put millions of work items into one Lookup. Large inventories need partitioned configuration reads or another durable work-distribution mechanism. Configuration also needs ownership and versioning: a metadata-driven design should reduce duplication, not turn unreviewed table rows into unrestricted executable SQL.
43. How do you consume Lookup output without confusing its two shapes?
I check whether firstRowOnly is enabled before referencing Lookup output. With its default single-row behavior, values are under firstRow. With firstRowOnly set to false, the returned rows are in the value array, which is suitable for a ForEach input.
Plain-text example
For an all-rows result containing [{"TableName":"Orders"},{"TableName":"Customers"}], the second expression supplies two work items. I define explicit handling for an empty result rather than indexing the first element blindly.
Lookup is for small control results, not bulk extraction: it returns at most 5,000 rows and supports an output payload up to 4 MB. A result above the row limit is truncated to the first 5,000 rows; an oversized payload fails. Queries or stored procedures must produce exactly one result set. I therefore reconcile the selected work-item count instead of assuming a successful Lookup returned the complete inventory.
44. When should you use Get Metadata rather than Lookup?
I use Get Metadata for properties such as existence, file size, or child names, and Lookup when I need actual control values contained in a file or query result. Listing a folder does not read each file's business records.
For an existence check, I explicitly request exists and branch on:
Plain-text example
A missing object then returns false instead of failing merely because it is absent. That distinction lets me implement an expected no-data case separately from an authentication or network failure.
The returned metadata is limited to 4 MB, and available fields vary by connector and format. A huge flat directory is therefore a poor control queue. I prefer predictable partition paths or a manifest, and I do not assume childItems recursively enumerates every descendant. An existence check also does not prove that an upload is complete; readiness needs a producer contract, such as a completion manifest.
45. How do you filter a directory listing before copying files?
I retrieve a bounded directory listing, pass its childItems array to a Filter activity, and iterate only over the returned value array. I filter both item type and naming convention so subfolders or temporary files do not accidentally become copy inputs.
A Filter condition for CSV files is:
Plain-text example
For orders.csv, notes.txt, and a folder named archive.csv, only the first item qualifies. The subsequent ForEach uses @activity('KeepCsv').output.value, and its child activity builds a path from the original directory plus item().name.
This is a control-flow filter over metadata, not a row filter inside CSV content. For very large directories I narrow the listing by partition or use a supported Copy source file-selection feature. I also agree on completion and naming rules with the producer rather than treating a matching extension as evidence that a file is ready.
46. How do you choose safe ForEach parallelism?
I choose concurrency from measured source, sink, and integration-runtime capacity, not from the largest supported number. ForEach supports up to 100,000 items and a parallel batchCount up to 50. That count is an upper bound, not a promise that exactly that many iterations always run simultaneously.
For example, eight table copies with batchCount four can have at most four iterations active together, but each Copy may itself use parallel connections. I consider the combined pressure before increasing either setting. Parallel writes must target independent partitions or use sink-side concurrency controls.
Pipeline variables are shared across iterations, so I do not use Set Variable as an iteration-local accumulator in a parallel loop. I pass item() values directly into activities or into an isolated child pipeline. If ordered processing or a shared mutable accumulator is genuinely necessary, I use sequential execution and accept the throughput trade-off.
47. How do you handle nested iteration without shared-variable races?
I put the inner iteration in a child pipeline and invoke it from the outer ForEach with explicit parameters. ADF does not support directly nesting ForEach inside another ForEach or an Until container. The child-pipeline boundary also provides independent variables and a clearer per-unit execution history.
For example, the parent iterates over customer accounts and passes accountId plus a bounded array of partitions to a child. The child iterates over its own partitions without writing a parent-level variable. I normally wait for the child to complete when the parent must aggregate success or failure.
Isolation does not remove capacity limits: ten parallel parent items each starting five parallel child copies can create substantial pressure. I budget concurrency across both levels. I also avoid using a shared output filename or global database staging table unless every row is namespaced by a stable batch key and cleanup is scoped to that key.
48. How do you debug an expression that works in the editor but fails at runtime?
I reproduce the failing activity's actual input shape and inspect which upstream branches executed. An editor can accept syntactically valid expressions even when a runtime property is missing, a value is null, or a skipped activity never produced the expected output. I first distinguish those cases from a genuine syntax error.
For a Lookup, I verify whether the output contains firstRow or a value array before accessing a field. I branch on supported existence or status checks before referencing optional results; coalesce() handles a null value but is not a general guarantee that an invalid property access will become safe. I also check whether interpolation changed an integer or array into a string.
Finally, I verify the expression language: pipeline expressions use orchestration syntax, while Mapping Data Flow expressions evaluate row columns and data-flow parameters. I test with an empty result, null field, quoted text, and a normal value. Secrets stay out of diagnostic logs, and the repaired expression is verified against the specific activity property receiving it.
49. How do you preserve types when using dynamic expressions?
I use a whole-value expression when the receiving property expects a number, Boolean, object, or array. String interpolation is appropriate for text construction, but it converts the result to a string.
With an integer pipeline parameter limit equal to 25:
Plain-text example
The first evaluates to numeric 25; the second evaluates to the string 25. A field expecting an integer may reject or misinterpret the interpolated form. Similarly, I pass an array expression directly to ForEach instead of serializing the array into text.
I validate values at boundaries and convert only when the target schema requires it. For a JSON configuration string, parsing with json() is different from surrounding that string with another layer of quotes. For literal text starting with an at-sign, I account for ADF's escaping rules. Most importantly, I inspect the effective activity inputs rather than judging correctness only from how the dynamic-content editor displays them.
50. How do you construct dynamic SQL without introducing injection risks?
I separate SQL values from SQL identifiers. Values such as timestamps or customer IDs should be bound through supported stored-procedure or query parameter mechanisms. Table and column names cannot generally be supplied as ordinary SQL value parameters, so I resolve them through an allowlisted configuration and quote identifiers correctly on the database side.
For example, a procedure can accept @Low and @High as typed timestamps and execute a fixed query against an approved table. That is safer than concatenating arbitrary incoming text into a WHERE clause. If the table must vary, I validate its schema and name against a controlled registry before constructing server-side dynamic SQL.
ADF string expressions are not SQL parameterization, and replacing a single quote is not a comprehensive authorization boundary. I also restrict the connection identity to the intended objects. This protects against both malicious input and accidental configuration mistakes while preserving a reusable ingestion pattern.
51. How do you implement a bounded watermark load?
I read the last committed watermark, capture a new upper bound, and process a fixed interval. I advance the checkpoint only after the target batch has been validated and committed. Capturing the upper bound once prevents a moving source from continually extending the current run.
For a SQL Server datetime2 last-modified watermark, the extraction predicate can be:
SQL example
The @Low and @High parameters are datetime2 values supplied by the calling mechanism, not literal ADF substitutions. With @Low at 10:00 and @High at 11:00, a 10:30 row is selected, an 11:00 row is selected, and a 10:00 row is excluded.
That predicate alone does not guarantee correctness under arbitrary concurrent commits. I need a reliable modification column and a source consistency contract, or an overlap-and-deduplicate strategy. A watermark also does not reveal physical deletes; those require a separate deletion signal or reconciliation.
52. How do you handle equal timestamps and late-arriving records?
I distinguish timestamp ties from late visibility. If I paginate rows that share a timestamp, I use a deterministic composite cursor such as (LastModified, OrderId) so the next page does not skip the remaining tied rows. The comparison is lexicographic: later timestamps qualify, or the same timestamp with a larger key qualifies.
That does not solve a transaction that becomes visible later with an older timestamp. For that case I use a source-supported change sequence or reread an overlap window and merge by business key and version. The overlap duration is based on an explicit lateness assumption; it is not an unlimited guarantee.
For example, if the checkpoint is 11:00 and the overlap is ten minutes, I reread from 10:50, deduplicate, and apply only newer versions. A record arriving with 10:30 remains outside that contract and needs reconciliation or a different capture mechanism. I document the behavior instead of claiming timestamp filtering is equivalent to database CDC.
53. When is a watermark insufficient for change capture?
A watermark is insufficient when I must capture physical deletes, every intermediate change, or changes that do not reliably advance the chosen column. An increasing insert ID handles new rows but does not discover updates to older rows. A last-modified timestamp can identify updates only if the source maintains it consistently.
I choose among source-native CDC, change tracking, application tombstones, or periodic comparison according to the required semantics and retention period. Native change logs can expose ordered changes and delete operations, but the pipeline still needs correct checkpointing and a recovery plan when retention is exceeded.
In ADF I verify the specific connector's supported capture mode rather than assuming every connector supports native CDC. The factory-level CDC resource is a separate feature from a hand-built watermark pipeline and is documented as public preview in the reviewed material. Its limits, including lack of self-hosted IR support, matter before choosing it for a private-network workload.
54. How do you recover when data lands but the watermark update fails?
I treat the outcome as an incomplete commit protocol, not as a reason to blindly advance the watermark. The target may already contain some or all of the batch, while the control table still points to the previous interval. On retry, the same interval will be read again, so the sink operation must be idempotent.
I persist a stable batch key, the exact interval, and a staged-data manifest. Recovery checks whether that batch was fully committed at the target. If it was, I finalize the checkpoint through a controlled operation; if not, I resume or replace the batch safely. A unique business key plus version-aware upsert, or replacement of an isolated partition, prevents duplicates.
Where target data and checkpoint live in the same transactional database, a stored procedure can atomically apply staged rows and update progress. Across independent systems I use explicit states and reconciliation because ADF does not provide a universal distributed transaction over Copy plus a separate checkpoint activity.
55. How do you prevent overlapping incremental runs from corrupting progress?
I serialize work for each logical source partition or enforce a database-side lease and compare-and-swap checkpoint update. Pipeline concurrency can reduce overlap within one pipeline, but it does not protect against a second pipeline, manual repair job, or another factory processing the same object.
A checkpoint update should verify the expected previous value:
SQL example
The calling procedure checks that exactly one row changed and otherwise rolls back or reports a concurrency conflict. This assumes a unique control row per object and appropriately typed, non-null checkpoint values.
The compare-and-swap prevents lost progress updates; it does not by itself prevent conflicting target writes. I combine it with an exclusive lease or independently safe version-aware writes. Lease ownership, expiry, and fencing matter if a stalled worker resumes after another worker has acquired the partition.
56. How do you make a file-ingestion retry idempotent?
I give each logical input a stable identity and make repeated processing converge on one committed result. A new ADF run ID is useful for tracing attempts, but it is not a stable deduplication key because a retry or manual rerun can receive a different run ID.
For immutable files, a useful identity combines source path with a producer version, checksum, or storage object version. I land data in a batch-scoped staging location, validate it, and publish through a sink-supported atomic operation or a committed manifest. Readers consume only committed batches. If the sink cannot atomically rename or replace a directory, I do not pretend a multi-file copy is atomic.
I record the source identity and completion status in a durable ledger with uniqueness enforcement. On a duplicate delivery, I verify the existing committed result rather than appending it again. For mutable files, path alone is insufficient: the new content must either become a new version or explicitly replace the previous logical batch.
57. How do you parameterize a Mapping Data Flow safely?
I declare typed parameters in the Mapping Data Flow and bind them from the Data Flow activity. Inside a data-flow expression, parameters use a dollar-sign prefix, while source columns are row values. I distinguish a literal string from an expression supplied as a parameter so user text cannot accidentally become transformation logic.
For example, a string parameter pPrefix and integer column orderId can be used in a Derived Column expression:
Plain-text example
With pPrefix equal to ORD- and orderId equal to 17, the result is ORD-17. This is Mapping Data Flow syntax, not a pipeline expression with a leading at-sign.
I use parameters for controlled variation such as an effective date, permitted path, or business-rule threshold. I do not overload them with arbitrary SQL or unrestricted expressions. Tests include nulls, empty strings, and boundary values, and I ensure debug parameter values match the intended pipeline bindings rather than silently using a convenient default.
58. How do you handle schema drift without silently accepting bad data?
I decide which changes are acceptable and enforce that contract separately from schema drift. Allowing drift can carry previously unknown columns through a Mapping Data Flow, but it does not prove that required fields, types, or business meanings remain valid.
For example, I may permit a new optional campaignCode while rejecting removal of OrderId or a nonnumeric Amount. Drifted fields can be accessed with byName() and explicitly converted when needed. I route invalid records to quarantine with the source file and batch identifier, and I fail the batch if a required-schema violation exceeds the agreed policy.
I also review sink mappings: automatic propagation of every new column can expose unexpected personal data or break a curated table. The raw landing layer can be permissive while the serving layer remains explicit. This balances resilience to harmless additions with observability and governance instead of using schema drift as a blanket instruction to accept any input.
59. How do you deduplicate records deterministically in a data flow?
I define the business key and an unambiguous winner before implementing deduplication. In a Window transformation, I partition by that key, sort by descending source version or modification time, add a stable tie-breaker, and compute rowNumber(). A downstream Filter retains rows where the assigned number is one.
For example, two records for Order 7 with versions 12 and 13 produce one retained record: version 13. If both versions are 13 but the payload differs, I do not silently claim timestamp sorting resolves the conflict. I need a documented secondary ordering rule or an error route.
An Aggregate that takes an arbitrary first value is not a deterministic latest-record strategy. Likewise, independently taking the maximum of every column can construct a row that never existed. I retain the complete winning row, handle null keys explicitly, and test that repartitioning or replay does not change which record survives.
60. How do you choose between Join, Lookup, and Exists transformations?
I use Join when I need relational combinations of columns from two streams, Lookup when enriching the main stream with matching reference information and controlled match behavior, and Exists when I only need to retain or reject rows based on whether a match is present. Exists avoids carrying unnecessary right-side columns.
Before selecting a transformation, I establish key cardinality. A left join against a reference table with two matches can duplicate the left row; that is not a performance bug but a consequence of the data relationship. If the reference should be unique, I validate or deduplicate it with an explicit rule before enrichment.
For example, an anti-existence check can isolate incoming customer keys missing from a master dataset. A Lookup intended to choose one match needs a deterministic ordering when multiple matches are possible. I also consider null-key semantics and data types on both sides, then inspect row counts before and after the operation to detect unexpected fan-out.
61. How do you flatten nested JSON without accidentally multiplying rows?
I choose the array that defines the target row grain, preserve its parent keys, and flatten only relationships that belong at that grain. Unrolling independent arrays together can create a Cartesian product rather than a meaningful relationship.
For example, one order with two line items and three unrelated tags can become six rows if both arrays are expanded together. If the target is an order-line table, I flatten the two items and retain tags as an array or write them to a separate order-tag table. The expected line count remains two, not six.
I explicitly test empty arrays, null arrays, missing nested objects, and optional children because the unroll-root choice changes which parents survive. After flattening, I reconcile parent and child counts and check key uniqueness. This is especially important before summing order-level amounts: repeating an order total on every line and then summing it will overstate revenue even when the transformation executes successfully.
62. How do you implement update, upsert, and delete behavior with Alter Row?
Alter Row assigns row-operation policies; the sink must also support and enable the corresponding operations. I configure keys for updates, deletes, or upserts, then verify that each incoming row has the necessary identity. Merely adding Alter Row does not turn an arbitrary file sink into a transactional database merge.
Conditions are evaluated in priority order and the first matching policy wins. For example, I place a deletion condition before a broad upsert condition so a tombstone is not accidentally reinserted. I make the fallback policy explicit and test one row for each branch.
I also distinguish a data preview from execution: previewing policy markers does not demonstrate that the target accepted the DML. Connector-specific constraints, identity columns, distribution keys, and permissions can change what is valid. Native CDC sources can already carry change-operation markers, so an extra Alter Row is not always required. The final design must preserve the source change semantics and prevent an older replay from overwriting a newer target version.
63. How do you separate malformed records from valid records?
I separate structural parsing errors from business-rule failures. A file that cannot be parsed may fail before row-level transformations run, while a parsed row with a missing key can be routed using Assert, Conditional Split, or supported sink error handling.
I define quality rules such as required key, valid date, and nonnegative quantity, then write rejected records to a restricted quarantine location with a reason, source identity, and batch ID. Sensitive values are minimized or masked. A batch policy decides whether any rejection fails the run or whether a bounded threshold is acceptable.
For example, 998 accepted rows and two quarantined rows should reconcile to 1,000 parsed input rows. If the accepted count is 998 but no reject count exists, the load is not adequately explained. I avoid silently enabling skip-incompatible-row options as a universal repair: tolerated data loss must be explicit, measurable, and recoverable through a corrected replay.
64. How do you implement a slowly changing dimension Type 2 load?
A Type 2 dimension preserves history by closing the current version and inserting a new version when tracked attributes change. I identify rows by business key, compare the incoming attributes with the current dimension row, and assign effective-start, effective-end, and current-status values using a consistent interval convention.
For example, with half-open validity intervals, a change effective at 2026-10-01 closes the old row at that instant and opens the replacement at the same instant. An unchanged record creates no new version. The surrogate key distinguishes historical versions even when the business key stays the same.
ADF can orchestrate staging and transformations, but expiring and inserting rows must not leave two current versions after a retry. I prefer an atomic sink-side apply procedure where feasible, with uniqueness and replay checks. I also define policies for late-arriving historical changes and multiple changes for one key in a batch; simply joining to the current row and updating once may lose intermediate history.
65. How do you control multiple sink execution and consistency?
I make sink dependencies explicit when one output must exist before another, using custom sink ordering where appropriate. Sinks with the same ordering group may be eligible for parallel execution according to the activity configuration. I do not rely on the visual left-to-right position of sinks as an execution contract.
Ordering is not atomicity. If the first sink commits and the second fails, the first write is not automatically rolled back. For a curated table plus an audit table, I therefore design each write to be repeatable and use a durable commit state that readers can interpret. If both writes must be one transaction, I move that commit into a system that can actually enforce the transaction.
Cached sink dependencies also need correct ordering before downstream lookups consume them. I test partial failures, not just the happy path, and document whether a retry replaces, merges, or appends each output. The expected outcome is consistent recovery, not a claim that one Data Flow activity makes all external sinks atomic.
66. How do you tune a skewed Mapping Data Flow join?
I inspect stage timing and partition behavior before adding cores. A join can be slow because most rows share one key, because the build side is much larger than expected, or because unnecessary columns and rows are being shuffled. More compute does not automatically fix one oversized partition.
I first filter early where semantics permit, project only needed columns, and verify key distribution. Broadcasting a genuinely small reference side can avoid a large shuffle, but forcing a large side to broadcast can cause memory failures. If one key dominates, I may isolate that key for separate processing or use a carefully designed salting strategy that preserves join semantics.
I compare representative runs using the same input volume and check both duration and output correctness. Repartitioning has its own shuffle cost, so I do not add it after every transformation. The target is a measured reduction in the bottleneck without introducing duplicate matches, missing rows, or excessive small output files.
67. How do you distinguish Copy mapping from data transformation?
Copy mapping aligns source fields with sink fields and handles supported type conversion during movement. It is appropriate for tasks such as mapping customer_id to CustomerId or selecting a defined subset of fields. It is not a general substitute for row-level business transformations.
If I need joins, grouped aggregates, complex derivations, or conditional routing, I choose Mapping Data Flows, source-side SQL, or another compute engine that supports those operations. Source SQL pushdown can be efficient, but it also consumes source database capacity and changes where business logic is maintained.
For example, renaming amount to Amount belongs naturally in mapping. Calculating a customer's lifetime spend from many orders requires aggregation, not a simple rename. I define mappings explicitly for curated tables to avoid accidental column drift, and test precision, null handling, date interpretation, and truncation behavior instead of assuming compatible-looking types preserve every value.
68. How do you troubleshoot Copy throughput before increasing resources?
I break the Copy duration into queueing, source read, transfer, and sink write behavior, then compare those observations with source and destination metrics. A slow query, API throttle, saturated network link, or limited sink transaction capacity needs a different fix from insufficient copy compute.
I establish a representative baseline and change one major factor at a time. For a partitionable relational source, parallel reads may help, but they can overload the database or produce inconsistent results if the extraction contract is weak. For many small files, listing and per-file overhead may dominate; for a large database load, sink indexing or batch behavior may dominate.
Data Integration Units apply to supported Copy execution on Azure IR, not as a universal tuning knob for self-hosted IR or Mapping Data Flow Spark compute. I inspect the effective runtime and connector support before changing settings. Finally, I compare cost per successfully reconciled batch, not merely the shortest activity duration.
69. How do you choose an integration runtime for a mixed-network copy?
I choose the runtime that can reach both endpoints and supports the required connectors. When a Copy uses a self-hosted IR for either side, that runtime performs the transfer and must have network access, credentials, and required drivers for both source and sink. It is not a two-agent tunnel between independent self-hosted runtimes.
For an on-premises SQL source and cloud storage sink, I verify outbound connectivity from the self-hosted node to storage as well as connectivity to SQL. For two private stores, I use one appropriate IR instance able to reach both. If that network path is impossible, I split the movement through an approved staging location and treat the two legs as separate recoverable operations.
Mapping Data Flows execute on Azure IR rather than self-hosted IR. ADF managed virtual network and managed private endpoints offer other supported private-access patterns, but they require the correct network setup; simply selecting managed VNet does not create arbitrary connectivity to every on-premises service.
70. How do you paginate a REST source reliably?
I map the API's documented next-page mechanism into the REST connector's supported pagination rules and define an explicit termination condition. Tokens, next links, page numbers, and offset/limit APIs are not interchangeable. I also account for throttling, token expiry, and whether the API provides a stable snapshot during pagination.
For a Copy REST source whose response includes a top-level next property containing the next absolute URL, the relevant pagination fragment can be:
JSON example
This is a source-settings fragment, not a complete Copy activity. The endpoint must return the expected response shape and a valid termination value; I test the last page and repeated-token cases. Connector pagination does not support every response layout, including top-level JSON-array responses for the documented pagination feature.
I reconcile unique record keys across pages because a mutable endpoint can return duplicates or omit records while the request sequence is in progress. Pagination success alone is not proof of a complete snapshot.
71. How do you use activity retry policies safely?
I retry operations that are likely to succeed after a transient failure and make their side effects safe to repeat. A retry is not a repair for a missing table, invalid mapping, expired permission assignment, or deterministic data-quality error.
For an activity supporting the standard execution policy, this fragment allows two retries after the initial attempt:
JSON example
The timeout shown is one hour. Secure-input/output settings must be changed when sensitive payloads require suppression; they are not general-purpose encryption controls.
Before enabling retries, I ask what happens if the remote operation committed but its response was lost. Appending the same rows or charging an external service twice is not safe. I use stable request or batch keys, inspect partial-write behavior, and bound retries so an outage does not become a sustained load spike. Alerts report the final failed outcome and preserve enough attempt context for diagnosis.
72. How do dependency conditions affect the final pipeline status?
I design the dependency graph to represent the business outcome, not just the desire to execute a logging activity. ADF supports success, failure, completion, and skipped dependency conditions. An error-handling branch that succeeds can cause a handled failure pattern to finish successfully, so a failed core operation does not always imply a failed pipeline.
For a mandatory load, I log diagnostic context on the failure path and then use a Fail activity when the business operation must remain failed. For an optional notification, I may deliberately allow best-effort completion without blocking the load. Those are different contracts and should be visible in the design.
Multiple incoming dependencies generally form an AND requirement; they are not an automatic OR trigger. I test success, failure, and skipped paths with small fixtures and inspect final pipeline status. A dashboard that monitors only pipeline failures will otherwise miss a copy failure that was accidentally swallowed by a successful handler.
73. How do you poll a long-running operation without an unbounded loop?
I submit the operation once, persist its operation identifier, and poll its status in an Until loop with a Wait between requests. The exit condition includes all terminal states, not only success. After exiting, I branch on the final status and explicitly fail the pipeline for a remote failure or cancellation when required.
An Until loop executes its body before checking the stopping condition. I configure a finite timeout and usually an attempt budget as well. A failed inner activity does not automatically stop the Until container, so polling errors need an explicit policy rather than an assumption that the loop will abort.
For example, I may poll every 30 seconds for an operation expected to finish within 20 minutes, with a timeout that reflects the service contract. I avoid re-submitting the original operation on every poll. If the process must survive a long interruption, the operation ID and progress belong in durable storage rather than only in a pipeline variable.
74. How do you pass values between parent and child pipelines?
I define child-pipeline parameters as its input contract and pass them through Execute Pipeline. The child should not depend on a parent's variable names or internal activity layout. I wait for completion when downstream parent work requires the child's result.
A child can expose named Pipeline Return Value entries, and the parent can read the Execute Pipeline output. For a return key named processedRows, the parent expression is:
Plain-text example
The child must actually set that key on the executed path, and the parent must wait for the child. I keep return schemas consistent across branches, including no-data outcomes, so missing-key failures do not appear only in production.
Return values are for small control summaries such as a count or manifest path, not bulk datasets. Large results belong in storage with a reference returned to the parent. I also distinguish the child's execution run ID from the stable business batch ID passed through both pipelines for replay and reconciliation.
75. How do you design a replayable tumbling-window pipeline?
I make the processing interval an explicit input and derive extraction boundaries from the trigger window rather than from utcNow(). A replay should process the same logical window even when executed days later. Tumbling-window triggers support stateful window scheduling and dependencies, but the pipeline still owns data correctness and idempotency.
For example, a pipeline receiving start 10:00 and end 11:00 can consistently apply a half-open interval: timestamps greater than or equal to 10:00 and strictly less than 11:00. Adjacent windows then have no boundary overlap. This convention differs from a (low, high] watermark predicate; either can work, but I do not mix conventions accidentally.
I isolate output by window or use a version-aware merge, define lateness and backfill rules, and ensure dependent windows reference the correct upstream completion. Replaying a window must not append duplicates or silently use today's configuration when historical reproducibility is required. I retain the relevant parameter and configuration version with the batch record.
76. How do you process duplicate or early storage events?
I treat storage events as notifications to evaluate an input, not as an exactly-once processing guarantee. The pipeline validates the event path against an allowed prefix, checks a stable file or batch identity, and consults a durable processing ledger before writing results.
I agree on a readiness protocol with the producer. A completion manifest or controlled final-path handoff is safer than assuming every notification means all related files are available. For multi-file deliveries, the manifest can list the expected members and checksums; processing starts only when that set is complete.
A repeated event for an already committed immutable input becomes a verified no-op. An event for a new version of the same path is handled according to the versioning policy, not discarded solely because the name is familiar. I also provide a reconciliation scan or inventory process for recovery, since an event-driven design still needs a way to discover unprocessed inputs and resolve uncertain outcomes without manually guessing what was delivered.
77. How do you validate a new pipeline without assuming debug is harmless?
I separate static validation from execution. ADF validation checks definitions and references, but it does not prove the data, permissions, connector behavior, or business logic will work. Debugging a pipeline executes activities and can write to real sinks, invoke external services, or incur compute charges.
My test design uses dedicated nonproduction connections, a tiny fixture dataset, restricted permissions, and clearly isolated output paths. I inspect linked-service and dataset bindings before any execution. Breakpoints and sampled Data Flow previews are useful, but they are not a universal dry-run mode and do not establish full-volume performance.
For logic that can be checked locally, I test interval boundaries, deduplication rules, JSON validity, and expected counts without contacting Azure. An authorized integration test then verifies the service-specific behavior. I report those layers separately: local fixture success is not evidence that a pipeline was deployed or that a production connector was exercised.
78. How do you promote parameterized artifacts across environments?
I promote a reviewed artifact set and supply environment-specific configuration through the deployment mechanism rather than hand-editing production definitions. Linked-service endpoints, resource identifiers, and approved parameters change by environment; pipeline logic should remain traceable to the same source revision.
I validate dependencies before deployment and account for triggers explicitly so a deployment does not start partially configured work. Integration runtimes need compatible names and types across stages, and self-hosted runtime infrastructure is an environment prerequisite rather than something a pipeline JSON file magically provisions and configures completely.
Secrets stay outside source control, using supported secure references and identities. After promotion, I verify resolved connections, trigger state, permissions, and a bounded smoke test under the appropriate authorization. Publishing from the authoring experience and deploying to a different factory are related but distinct operations. I also keep a rollback plan for artifacts while recognizing that rolling back a definition does not reverse data already written by the newer version.
79. How do you diagnose a pipeline that succeeds with the wrong row count?
I trace the expected population through each stage rather than treating activity success as a row-count assertion. I first verify source filters, time zones, interval boundaries, parameter values, and whether a Lookup inventory was truncated. Then I inspect copy statistics, transformation counts, rejected rows, join fan-out, and sink write semantics.
For example, 1,000 source rows can legitimately produce 900 target keys after deduplication, but the removed 100 must match a documented rule. Conversely, a one-to-many reference join can turn 1,000 input rows into 1,200 outputs without any runtime error. A successful upsert can also update existing records rather than increasing the table count.
I compare counts at the same grain and under the same source snapshot or interval. I include aggregate measures and key-level samples because equal counts can hide substituted or missing records. The investigation ends when each difference is explained and the reconciliation rule passes, not merely when rerunning the pipeline happens to produce a green status.
80. How do you implement reconciliation before marking a batch complete?
I define reconciliation as a commit gate with measurable invariants. The batch records its source identity or interval, expected population, accepted and rejected counts, duplicate handling, and target commit evidence. Only after the required checks pass do I mark it complete and advance downstream progress.
For a simple no-filter ingestion fixture:
Plain-text example
If deduplication removes rows, I add a separately explained duplicate category rather than forcing the original equation. For updates and deletes, I reconcile operations and business keys, not just the net target row count.
I supplement counts with suitable aggregates, key coverage, or checksums using consistent encoding and null rules. These checks are evidence, not interchangeable guarantees: a sum can collide, and a checksum comparison needs matching canonicalization. Failed reconciliation quarantines the batch and preserves diagnostics for replay. It must not quietly advance a watermark simply because the Copy activity itself succeeded.
Azure Data Factory Interview Questions and Answers for Experienced
These answers connect senior design decisions to measurable service objectives, data correctness, security boundaries, and controlled recovery.
81. How would you design an enterprise ADF platform without creating one unmanageable factory?
I divide factories along operational boundaries such as environment, ownership, data sensitivity, and regional requirements, rather than placing every pipeline in one global factory. Development, test, and production need separate identities and destinations. Shared conventions can include reusable deployment templates, observability fields, and parameter contracts without forcing unrelated teams to share production privileges.
For example, finance and marketing can use the same ingestion template while retaining separate factory identities, release approvals, and storage access. I keep an inventory of owners, dependencies, criticality, source limits, and recovery requirements. Cross-factory dependencies need an explicit completion contract rather than an undocumented schedule offset.
The trade-off is additional infrastructure and deployment coordination. I would not create one factory per table either: that multiplies administration without providing meaningful isolation. The architectural test is whether a failed release, compromised author identity, or overloaded integration runtime can affect workloads outside its intended boundary.
82. How do you choose between Azure IR, managed virtual network IR, and self-hosted IR?
I choose an integration runtime from the activity, connector, network path, and operational constraints together. Azure IR is suitable for supported cloud data movement and activity dispatch. An Azure IR with managed virtual network supports managed private endpoints to supported resources. Self-hosted IR provides connectivity from a customer-managed Windows host that can reach private or on-premises sources.
I draw the actual source-to-sink data path before choosing. A private SQL Server reachable only from an enterprise network may require SHIR, whereas supported private Azure endpoints can fit managed virtual network IR. SHIR is not a customer-hosted Spark engine for Mapping Data Flows; those run on Azure IR compute. Azure-SSIS IR serves a different purpose: executing SSIS packages.
I also verify the selected connector's supported IRs and authentication methods. Choosing a factory region or adding a private endpoint does not, by itself, prove that every activity will execute in the desired network or that every connector has the same capabilities.
83. How would you design a highly available self-hosted integration runtime?
I run multiple SHIR nodes on separate failure domains and treat them as production infrastructure, with patching, capacity monitoring, certificate management, and a tested replacement procedure. Nodes need equivalent drivers, network access, and configuration; a second node that cannot reach the source is not useful redundancy. I avoid installing unrelated workloads that can consume its memory or CPU unexpectedly.
I size the cluster from measured workload concurrency, source limits, and transfer characteristics. If one node is unavailable, remaining capacity must still meet the recovery target, not merely accept connections. I monitor queue time, host resource pressure, connectivity, and version consistency.
For an interruption during a copy, I assume the activity may fail or need retry rather than promising transparent transfer of every in-flight operation. Recovery therefore includes idempotent sink writes and batch reconciliation. I also protect registration credentials and document re-registration or recovery steps; access to a SHIR host is part of the data-platform trust boundary.
84. What does an ADF private endpoint protect, and what does it not protect?
An ADF private endpoint provides a private route to supported Data Factory service endpoints; it does not automatically make every linked service or data movement path private. I distinguish communication with the orchestration service from communication between an integration runtime and a data store.
For example, SHIR can communicate with ADF through a private endpoint while its SQL connection still follows the network configuration of the SHIR machine. Conversely, an Azure IR in an ADF managed virtual network can use an approved managed private endpoint to a storage account. These are different endpoints serving different directions of connectivity.
My review covers DNS resolution, routing, endpoint approval, public network settings, and the runtime actually selected by the linked services. I verify supporting services such as Key Vault as well as the main source and sink. A successful private connection to the factory is evidence for that route only, not proof of end-to-end network isolation.
85. How do managed identities and least privilege work in a production ADF design?
I prefer managed identity where the connector and target service support it, then grant only the required data-plane operations at the narrowest practical scope. Authentication answers who the pipeline is; authorization determines what it can read or change. An Azure resource-management Reader role is not a substitute for SQL database permissions or storage data access.
A system-assigned identity follows the factory lifecycle. A user-assigned identity has an independent lifecycle and can support controlled identity reuse, subject to connector support. I avoid sharing one highly privileged identity across unrelated factories because that expands the blast radius.
For an ingestion pipeline, I might grant source SELECT access and write access only to its landing area, leaving curated-table publication to another identity. I also review who can author and run pipelines: an author may use the factory's attached identities through pipeline execution. Therefore least privilege includes authoring rights, deployment rights, and target permissions, not simply removing passwords.
86. How would you manage and rotate secrets without exposing them in pipeline logs?
I keep secret values out of Git and pipeline parameters wherever possible, use managed identity for supported connections, and reference Azure Key Vault when a password or key is unavoidable. The factory identity needs permission to retrieve the intended secret, while secret administration belongs to a separate controlled role.
Rotation is a compatibility change. I identify whether the linked service pins a secret version, arrange overlap when the target supports it, validate a replacement credential in the appropriate environment, and retire the previous credential only after consumers have moved. A versionless reference and a version-pinned reference have different operational behavior.
I also suppress sensitive activity inputs and outputs where appropriate and ensure downstream activities do not log the same value again. For example, hiding a token-returning Web activity's output does not protect a later request that exposes the token in its logged input. Monitoring, exception messages, query strings, and exported artifacts all need review; secure logging is broader than a single checkbox.
87. How do you troubleshoot a private-network pipeline that works in development but fails in production?
I compare the failed runtime path rather than assuming identical pipeline JSON means identical connectivity. First I identify the activity, linked service, effective integration runtime, target hostname, authentication method, and exact failure stage. DNS failure, TCP timeout, TLS error, and authorization denial require different investigations.
For SHIR, I inspect the node's DNS, route, firewall, proxy, driver, and certificate configuration. For a managed virtual network IR, I check the managed private endpoint's target and approval state. I also inspect production Key Vault reachability and the production identity's target permissions.
I compare environment parameters against the approved deployment artifact and look for accidental references to development endpoints. A connection test is useful but insufficient: the failing operation may require additional permissions or access a different endpoint. I preserve error codes and correlation identifiers without copying credentials into incident tickets. Once the cause is understood, I fix the environment or template rather than weakening firewall rules globally to make the error disappear.
88. How do you separate developer, operator, and deployment privileges?
I separate the ability to edit pipeline definitions, deploy approved artifacts, run operational recovery actions, and grant access to external resources. Production operators normally need monitoring and narrowly scoped execution capabilities, not unrestricted authoring. The deployment identity should promote reviewed artifacts rather than act as a personal administrator account.
The crucial risk is indirect data access: someone able to create and run arbitrary pipelines can potentially use the factory's linked services and identities. I therefore treat production authoring as a privileged capability even when the person cannot read a password directly.
I use separate environments and factories where security boundaries require them, with Azure RBAC and source-control protections aligned to those boundaries. I do not assume every pipeline is an independently secure tenancy boundary inside a shared factory. Break-glass access needs time limits, audit evidence, and reconciliation back into source control. Permissions to change Key Vault policies or target database grants should remain separately governed.
89. What should a production ADF Git and CI/CD workflow look like?
I use a development factory connected to Git, reviewed feature branches, automated validation and artifact generation, and controlled promotion of the same versioned artifact through test and production. The collaboration branch is the source definition; a generated deployment artifact is what I release. I record the commit, artifact hash, environment parameters, and approval together.
I distinguish saving to Git from publishing changes to a live factory. Debugging unpublished work in development is also not evidence that production has the same definition. Automated publishing utilities can validate and export templates without requiring a person to click Publish for every release.
My release gates check resource references, environment isolation, secret handling, trigger changes, and representative integration tests. Production changes should not be made ad hoc in the UI and forgotten. If an emergency edit is necessary, I reconcile it into the governed source and artifact history before subsequent deployment. This prevents the next routine release from silently overwriting the fix.
90. How do you parameterize deployments without turning every property into a parameter?
I parameterize values that genuinely vary by environment, such as endpoints, database names, Key Vault locations, runtime references, and approved environment flags. I keep stable business logic and transformation contracts versioned as code. Parameterizing every property obscures review and can encounter deployment-template limits without adding useful flexibility.
I keep a typed configuration inventory with required fields and validation rules. For example, production must not resolve to a development storage account, and a workload requiring private connectivity must reference its approved runtime. Secrets are references or securely injected deployment inputs, never ordinary values committed to a parameter file.
ADF supports custom ARM parameterization through its parameter-definition configuration. I review generated templates as artifacts rather than assuming source JSON and deployment JSON are interchangeable. A useful release check compares resolved non-secret configuration against an allowlist. It should fail on an unknown production endpoint instead of accepting any syntactically valid string.
91. How do you deploy safely when triggers and running pipelines are active?
I treat trigger state, in-flight work, and downstream compatibility as explicit release concerns. Stopping a trigger prevents new invocations from that trigger; it does not cancel or drain existing runs. Before deploying a breaking change, I determine whether active runs can finish safely or require a planned maintenance window.
The release captures the intended trigger states, applies the appropriate pre-deployment stop logic, deploys the versioned artifact, handles removed resources deliberately, and restores only the approved triggers. Microsoft's deployment scripts provide useful starting points, but I review their behavior for the factory and release method rather than running an unexamined script.
For example, a dataset-path change may be backward compatible, while dropping a column used by active jobs is not. I use expand-and-contract changes when possible. After deployment, I verify the live definitions, environment bindings, trigger states, and a controlled smoke test. A successful ARM deployment alone does not establish that the next scheduled load will produce correct data.
92. What does a safe rollback strategy look like for ADF?
I separate code rollback from data recovery. Redeploying a previously approved template can restore pipeline definitions, but it does not undo rows written, files deleted, checkpoint advances, or external side effects. The release plan therefore identifies which changes are reversible and how to reconcile work executed by the faulty version.
I retain immutable artifacts, their environment configurations, and a deployment audit trail. A rollback decision includes compatibility with current database schemas, connectors, secrets, and runtime versions. If the old pipeline expects a column already removed by a migration, restoring the old JSON is not a viable recovery.
For a bad transformation, I would halt affected publication, identify impacted batch IDs, restore or rebuild the relevant partitions from preserved input, validate the repair, and then reopen downstream consumption. For a configuration-only error, a targeted roll-forward may be safer than a broad rollback. Either way, I confirm trigger states and reconcile all in-flight and uncertain runs before resuming normal scheduling.
93. How would you test ADF changes before they reach production?
I use several layers because static validation cannot prove connectivity or data correctness. Local checks can parse JSON, inspect references, validate parameter contracts, test SQL or transformation logic against fixtures, and detect accidental secrets. ADF validation and template export add service-specific definition checks. Authorized integration tests in a nonproduction environment establish connector, identity, network, and activity behavior.
The important cases are not only happy paths. I test empty input, duplicate input, late updates, invalid records, schema changes, throttling, partial failure, and rerunning the same batch. Expected results include sink contents and checkpoint state, not merely a green pipeline status.
For example, a fixture containing two versions of one business key should leave the intended latest version after the initial run and again after a retry. I distinguish tests actually executed from planned checks. Local parsing of an ADF expression string does not evaluate it in the ADF runtime, and this article's illustrative examples are not evidence of deployed or executed Azure pipelines.
94. How do you manage linked-service and connector upgrades?
I manage a connector upgrade as a change to a data contract and execution dependency, not just a new version label. I inventory connector type and version, supported authentication, runtime requirements, source and sink options, data-type mapping, and any retirement deadlines. SHIR driver or host dependencies may also need coordinated updates.
I compare the old and new paths with representative data, including nulls, high-precision decimals, Unicode, timestamps, and large values. I measure throughput and source load as well as row counts. A compatible schema does not guarantee identical timestamp interpretation or write semantics.
I release through nonproduction first, retain a documented fallback where supported, and check whether reverting is still possible after target-side changes. There is no single universal 'ADF connector version' that establishes all connectors' behavior. For interview answers and production runbooks, I name the actual connector and consult its current version-specific guidance rather than borrowing options from a similarly named Synapse or Fabric connector.
95. How do you define an ADF service-level objective around data freshness?
I define the objective in terms of usable data, such as '99% of hourly sales partitions are validated and available within twenty minutes of the agreed source cutoff.' Pipeline duration alone is insufficient: a fast run can load stale data, and a successful run can publish incomplete results.
I record source cutoff, ingestion start, commit time, quality-check completion, and publication time. Those timestamps distinguish source lateness from orchestration delay, processing delay, and downstream delay. I also define which events count as failures and how planned maintenance or unavailable source data is handled.
For a hypothetical day with 24 required partitions and one missed deadline, on-time delivery is 23 / 24, or approximately 95.83%. That business measure is separate from Microsoft's service availability SLA. I alert on approaching freshness deadlines and missing expected batches, not only failed activity runs, and I keep recovery playbooks aligned to the remaining time budget.
96. How do you build actionable pipeline observability rather than noisy failure alerts?
I combine technical telemetry with a durable business-level run ledger. Every run should be traceable to a pipeline run ID, logical batch or window, source, destination, artifact version, owner, and final publication state. Activity failures are useful evidence, but they do not establish whether the business dataset is current or complete.
I route diagnostic logs and metrics to the approved monitoring destination with retention appropriate to incident analysis and audit requirements. Dashboards distinguish queueing, execution, retry, rejected rows, and final data freshness. Alerts should identify the affected dataset and recovery action instead of sending one message for every failed retry.
I also detect absence: a disabled trigger can produce no failure event at all. A separate expectation check can flag a missing scheduled partition. Sensitive input and output logging is suppressed, with non-secret identifiers retained for correlation. The operating model includes an owner and escalation path; a dashboard nobody watches is not an operational control.
97. How do you investigate a Copy activity whose runtime has doubled?
I compare equivalent workloads and break the elapsed time into queueing, source reading, transfer, and sink writing. I check whether bytes, file counts, row widths, or partition distribution changed before blaming the integration runtime. A twofold runtime increase may simply reflect twice the data, but unchanged throughput can still violate the delivery target.
Copy monitoring exposes useful execution details; I correlate them with source query plans, throttling, sink utilization, network behavior, and SHIR host pressure where relevant. More DIUs or parallel copies will not fix a serialized source query or a saturated destination.
For example, millions of tiny files can make listing and per-file overhead dominate even when total bytes are modest. A large relational extract may instead need a selective query or supported partitioned reads. I change one meaningful variable at a time, compare measured results, and cap source load. The acceptance criterion includes cost and correctness, not simply the shortest successful run.
98. How do you tune Mapping Data Flows without simply buying larger compute?
I locate the expensive stage first: startup, source scan, shuffle, skewed join, transformation, or sink write. I reduce unnecessary rows and columns early where semantics permit, avoid repeated reshuffles, and inspect whether one partition is doing disproportionate work. Increasing cores cannot compensate indefinitely for skew or a slow external sink.
For joins, I consider broadcasting a genuinely small side, but do not force broadcast merely because the current development fixture is small. Production cardinality and memory requirements matter. Repartitioning can help when the existing distribution is poor, but it also costs a shuffle and should be justified by measurements.
I examine file layout, sink partitioning, and connection pressure alongside compute settings. Cluster startup can dominate small workloads; reuse through an appropriate time-to-live setting can help sequential work but incurs an idle-cost trade-off. My benchmark compares end-to-end runtime, output equivalence, resource consumption, and cost using representative data rather than preview timings alone.
99. How would you reduce ADF cost while preserving the delivery SLA?
I attribute cost to business workloads before optimizing. I separate orchestration activity runs, data movement, Mapping Data Flow compute, external compute, storage transactions, and network transfer where applicable. A lower ADF bill can hide increased database or Spark costs, so I compare the end-to-end cost of delivering a validated dataset.
Common opportunities include replacing unnecessary full extracts with correct incremental processing, consolidating excessively granular orchestration, avoiding idle debug sessions, right-sizing data flow compute, and tuning time to live to the actual execution pattern. I preserve boundaries needed for isolation and recovery rather than merging everything solely to reduce activity counts.
For a hypothetical compute setting billed at rate R per hour, ten minutes of otherwise unnecessary idle time costs R / 6; R must come from the relevant current pricing model. I do not quote a universal dollar saving. I validate changes against freshness, source impact, and rerun cost, since a cheap but unreliable pipeline can be more expensive operationally.
100. How do you prevent parallel pipelines from overwhelming a source system?
I enforce concurrency at the shared bottleneck, not only inside one ForEach activity. Ten parent runs with five simultaneous children each can create fifty active loads before each Copy activity opens its own source connections. Pipeline concurrency, trigger concurrency, loop batch counts, connector parallelism, and source-side limits interact.
I assign capacity budgets per source and workload class. For a constrained operational database, current ingestion may get reserved capacity while backfills use a smaller queue. A durable admission mechanism or orchestrated scheduler can coordinate across factories when local pipeline limits are insufficient.
I monitor source latency, locks, connections, and throttling, then tune to an agreed operating envelope. A single global maximum is often inefficient because a tiny reference-table query and a large partitioned export have different costs. I also add jitter or controlled staggering to avoid every hourly schedule hitting the source simultaneously. Concurrency is a reliability control as well as a performance setting.
101. Can ADF guarantee exactly-once delivery across arbitrary systems?
No. ADF coordinates activities, but it cannot impose one atomic transaction across arbitrary sources, sinks, and external APIs. A failed or timed-out activity can have completed some side effects, and retries may repeat them. I design for duplicate attempts and uncertain outcomes rather than treating a green orchestration state as an exactly-once guarantee.
I build an effectively-once business result through stable batch IDs, idempotent writes, unique constraints or deduplication, and checkpoint commits tied to durable target outcomes. Where the sink can atomically commit both data and a processed-batch record, that transaction becomes the authoritative commit point.
For example, retrying batch sales-2026-10-01-10 should detect an already committed ledger entry instead of appending the same sales again. An ADF run ID is useful for tracing attempts, but a new run ID must not turn a retry into a new business batch. Any exactly-once claim must identify its scope, transaction boundary, and assumptions about source ordering and sink behavior.
102. How do you prevent two runs from committing the same incremental load?
I use durable concurrency control around the logical dataset or batch. Pipeline concurrency can reduce overlap, but it does not protect against every alternative entry point, another factory, or a manually initiated repair. The commit protocol must reject a second owner or stale writer at the authoritative state store.
A common design uses a transactional lease or optimistic version check, plus a unique batch key. The following T-SQL fragment illustrates a compare-and-swap checkpoint update inside a controlled commit transaction; it assumes the data commit and checkpoint can share that transaction and the table already exists.
SQL example
If another writer has advanced the revision, zero rows match and the transaction must roll back. This is not a complete locking framework. External sinks may require a batch ledger, fencing token, and reconciliation because their writes cannot be rolled back by this SQL transaction.
103. How would you recover from a partially committed batch across several target tables?
I reconcile each target table against a durable batch manifest before deciding what to rerun. The manifest records the logical batch ID, expected entities, staging locations, per-table commit state, validation results, and publication status. An orchestration timeout is an uncertain outcome, not proof that every target rolled back.
If all tables share a transactional database and the volume permits it, I can stage first and publish the related changes in one target transaction. If separate target systems or independently committed tables are involved, I preserve per-table idempotency and hold the batch's consumer-visible completion marker until all required targets are valid.
For example, if customers and orders committed but order lines did not, recovery verifies the first two commits and reapplies only the missing or invalid component using the same batch identity. It must not blindly append all three again. I check referential consistency before publication and define a repair path for irreconcilable partial results. Watermarks advance only according to the documented multi-entity completion policy.
104. How would you process late-arriving or corrected records safely?
I distinguish event time, ingestion time, and source change order. A record describing yesterday's sale may arrive today, and a correction can modify a partition already published. Therefore a simple filter on business date is not enough to identify every change.
For timestamp-based ingestion, I can use a justified overlap window with idempotent upserts and periodic reconciliation, but that provides only bounded-lateness protection. For stronger change tracking, I use a supported ordered change feed and preserve its position, delete semantics, and retention assumptions. I define what happens when an update is older than the supported correction horizon.
Suppose a late correction changes a customer's region. Recomputing only today's fact partition may leave historical aggregates wrong. The recovery design needs affected-key or affected-partition propagation, not just ingestion deduplication. I agree on finalization rules with consumers, measure lateness distributions, and retain enough raw history to repair accepted corrections without pretending that any finite overlap catches arbitrarily late data.
105. What do you do when a CDC consumer falls behind the source retention window?
I stop normal incremental advancement and declare a data gap rather than silently moving the checkpoint to the earliest available change. Once required change records have been removed, the missing interval cannot be reconstructed from that feed alone. The recovery choice depends on retained snapshots, archives, source history, and acceptable downtime.
For SQL Server CDC, I validate the saved position against the capture instance's current minimum valid LSN and respect its inclusive interval semantics. I do not treat LSNs as timestamps or invent missing transactions. A supported rebaseline procedure captures a consistent state and restarts change consumption without losing changes during the handoff.
Operationally, I alert on lag relative to retention before the gap occurs and reserve catch-up capacity. A seven-day retention setting is not a seven-day guaranteed recovery window if cleanup, workload, or an outage changes the effective available range. After reinitialization I reconcile counts and business totals, record the discontinuity, and communicate which downstream datasets needed correction.
106. How do you design a consistent snapshot plus change-stream bootstrap?
I need a source-supported consistency boundary linking the initial snapshot to subsequent changes. Independently copying the current tables and then starting a change feed at 'now' can lose changes made during the snapshot. Starting too early without reconciliation can replay changes already reflected in the snapshot.
The exact protocol depends on the database or connector. I establish the required change capture before the snapshot, obtain the appropriate consistent boundary, retain changes while the snapshot is extracted, and apply the post-boundary changes with ordered, idempotent semantics. If the source offers a native initial-load-and-CDC mechanism, I verify its guarantees rather than recreating them casually.
I size retention and temporary storage for the snapshot plus catch-up period. A multi-table snapshot also needs an explicit cross-table consistency requirement; separate table reads may observe different transaction states. My acceptance tests include concurrent inserts, updates, and deletes during bootstrap. I do not label a generic ADF Copy followed by a timestamp filter as an atomic snapshot-plus-stream solution.
107. How do you handle deletes and slowly changing dimensions in a production ingestion design?
I make delete handling explicit in the ingestion contract. A high-watermark query over surviving rows cannot discover rows that were physically deleted. I need a supported CDC delete event, source tombstone, soft-delete marker, or a justified snapshot-comparison process. The target policy then decides whether to remove, deactivate, or retain the record for history.
For a Type 2 dimension, I preserve the business key, surrogate key, validity interval, and current-row indicator. A change should close the prior version and insert the new version consistently, with deterministic handling of duplicates and multiple changes for the same key. A retry must not create another identical history row.
I distinguish the time the business change became effective from the time ingestion observed it. Late corrections can require interval repair rather than simply appending a latest row. Mapping Data Flow or target SQL can implement parts of the logic, but selecting an upsert option alone does not establish a correct Type 2 history or erase sensitive data from retained raw layers.
108. How do you design a replayable data lake ingestion pipeline?
I separate immutable landing data from validated and published data. Each logical batch has an explicit input manifest, source position or extraction window, schema version, and processing version. I avoid making consumers infer completion from whichever files happen to be present in a directory while a copy is still running.
A publication manifest can be an application-level convention such as this JSON, not an ADF pipeline definition:
JSON example
Consumers process only batches published under the agreed commit protocol. Replaying those files with the same logical batch ID should reproduce or safely replace the intended target state, not append duplicates. I retain raw inputs according to governance requirements and record transformations separately. ADF orchestrates these operations, but storage and table-format semantics determine whether rename, overwrite, or manifest updates are atomic for the chosen system.
109. How do you make a metadata-driven framework safe for hundreds of tables?
I treat metadata as executable configuration and give it a schema, versioning, approval workflow, and validation. Each entity needs an owner, source and destination contract, extraction strategy, business keys, concurrency class, quality thresholds, and recovery policy. An arbitrary SQL string in an editable table is not automatically a safe metadata-driven design.
I whitelist supported connectors and operations, validate identifiers, and pass values through supported parameters rather than concatenating untrusted metadata into SQL. I keep credentials in governed connection resources, not the control table. I also separate configuration from mutable run state so a checkpoint update cannot accidentally alter the next job's specification.
The framework should support bounded batches of entities, per-entity failures, durable status, and selective replay. Hundreds of tables should not require one enormous activity output or a single all-or-nothing parent run. I start with a small set of explicit templates and extension points; a universal framework that hides all connector differences is often harder to operate than several well-governed patterns.
110. How would you enforce schema contracts without rejecting every additive change?
I classify changes by their effect on consumers instead of treating every schema difference equally. Adding a nullable column may be acceptable in a raw landing layer, while removing a key, changing decimal precision, or altering timestamp semantics can break curated contracts. Schema drift support is a mechanism for handling unknown columns, not a substitute for governance.
I validate required columns, types, nullability, keys, and semantic rules at the appropriate boundary. Accepted additive fields can remain in raw data while a reviewed mapping controls curated publication. Breaking changes enter a quarantine or compatibility workflow with an owner and an explicit resolution.
For example, a new optional campaign_code can be retained without modifying a stable sales contract, but changing amount from decimal to free-form text must not silently coerce invalid values to null. I version transformations and capture the observed schema with each batch so replay uses the intended contract. Consumers receive a migration plan for deliberate breaking changes.
111. How do you quarantine invalid records without silently declaring success?
I define the rejection policy before choosing an implementation. Invalid rows should carry a reason code, source locator, logical batch ID, and sufficient protected context for repair. Valid records can continue only when that behavior is acceptable to the dataset's contract; some financial loads must fail as a whole rather than publish a partial result.
I separate technical success from quality acceptance. A Copy activity configured to skip incompatible rows can succeed while omitting data, and an assertion strategy needs explicit routing and failure behavior. The run ledger records input, accepted, rejected, and published counts, with threshold checks and an accountable owner.
For example, if a batch contains 10,000 rows and 25 are quarantined, the rejection rate is 0.25%. Whether that is acceptable depends on the contract and severity, not a universal threshold. I test that replaying repaired rows does not duplicate already accepted data and that quarantine storage has appropriate access controls and retention. Silent discard is not a recovery strategy.
112. How do you protect personal data across ingestion, logs, and replay storage?
I minimize personal data at each layer and treat logs, temporary staging, rejected records, and backups as part of the data inventory. Encrypting the main destination does not address sensitive values copied into activity outputs or long-lived replay files. Access controls, retention, and deletion workflows must cover those secondary copies.
I use narrow identities and separate restricted landing areas from broadly consumed curated data. Masking or tokenization is applied according to the purpose of the workload, with key management and reidentification risks reviewed separately. Pseudonymized identifiers may still be sensitive; replacing a name with a hash does not automatically make a dataset anonymous.
Operationally, I record which pipelines and retained batches can reproduce a person's data. A downstream deletion can be undone by a later replay unless the replay process also applies the governing deletion policy. I coordinate those requirements with privacy and security owners rather than claiming ADF settings alone establish regulatory compliance. Tests and incident evidence should use synthetic or properly approved data.
113. How do you establish useful lineage when ADF orchestrates external transformations?
I distinguish orchestration lineage from transformation lineage. ADF can show that a pipeline invoked a notebook or stored procedure, but that does not automatically reveal every table and column touched inside arbitrary external code. I combine supported catalog integration with metadata emitted by the transformation itself.
A useful lineage event identifies the logical input and output datasets, batch or snapshot versions, transformation version, execution ID, and owner. It should connect an affected downstream table back to the precise source interval and released code, not merely to a generic pipeline name.
I verify the supported activities, connectors, and lineage granularity of Microsoft Purview integration rather than assuming universal column-level coverage. For unsupported paths, I document gaps and use governed supplemental metadata where appropriate. A reconciliation check can flag outputs with no registered producer or changed dependencies. The goal is practical impact analysis and auditability: when an upstream column changes, I need to know which published datasets and consumers may be affected.
114. How would you plan regional disaster recovery for ADF?
I define recovery time and recovery point objectives for the whole data path, not only the factory definition. A second-region factory without reachable data stores, identities, secrets, runtimes, checkpoints, and network approvals is not an operational recovery environment. I retain versioned infrastructure and pipeline artifacts and document region-specific configuration.
The recovery plan covers source availability, replicated or recoverable sinks, SHIR placement, Key Vault dependencies, monitoring, and the authoritative batch ledger. I prevent both regions from processing the same work concurrently unless the design explicitly supports active-active operation. Trigger activation and checkpoint ownership need a controlled handover.
I rehearse recovery with representative failures and measure time to validated data, not just time to deploy the factory. After an interruption, some batches may have uncertain commit states and need reconciliation before replay. Azure's service resilience and recovery capabilities are valuable, but they do not automatically satisfy a workload's business RTO or guarantee zero data loss across every external dependency.
115. How do you run historical backfills without disrupting current ingestion?
I treat backfill as a separate workload class with explicit source windows, resource budgets, and publication rules. It should not overwrite the live ingestion checkpoint or compete without limits for the same source connections. I use stable logical batch identities so each historical window can be validated, retried, and audited independently.
For a multi-year repair, I split work into bounded partitions and prioritize windows according to business impact. I cap parallelism, monitor source pressure, and pause backfill when current ingestion approaches its freshness deadline. Historical source queries must still respect the source's consistency and retention constraints.
I also decide how backfill interacts with newer corrections. Replaying an old snapshot must not overwrite a newer business version just because it was processed later. Target writes can compare source sequence or version information where available. Before publication I reconcile counts and business aggregates and verify that partition replacement or merge semantics are correct. Completion means repaired, validated consumer data, not simply that every historical pipeline run turned green.
116. How do you orchestrate a multi-step load that spans systems without a shared transaction?
I model the workflow as a durable state machine with explicit commit points and compensating or repair actions. ADF activity dependencies sequence work, but they do not create a distributed transaction between a database, a lake, and an external API. Each step needs an idempotency contract and a way to determine its outcome after a timeout.
For example, publishing a dataset and notifying an external catalog can be decoupled using a durable outbox or pending-notification record committed with publication. A retry can then deliver the notification using a stable event ID without reloading the data. Where the external service supports idempotency keys, I use them; otherwise I define deduplication or reconciliation behavior.
Compensation is not always a true undo. An email cannot be unsent, and an already consumed dataset may require a correction notice. I design irreversible actions late in the workflow where practical and retain enough state to resume safely. The runbook explains which states can be retried automatically and which require an authorized repair decision.
117. When should ADF hand work to another compute or orchestration service?
I use ADF for data movement and coordination when its managed connectors and activity model fit the workload, and delegate computation to the engine best suited to the transformation. Complex distributed algorithms, highly customized libraries, or existing governed Spark jobs may belong in external compute; relational transformations may be better executed close to the database.
I avoid passing large datasets through pipeline variables or activity outputs. ADF should pass references, parameters, and status while the selected compute reads and writes the data. The interface needs an input contract, job identifier, completion semantics, timeout, retry policy, and useful failure evidence.
I also reconsider the orchestration layer when requirements involve durable application workflows, very fine-grained low-latency coordination, or continuous event processing rather than scheduled integration. That is a fit assessment, not a claim that one product replaces every other service. The final choice accounts for operations, security, team skills, cost, and how the entire workflow recovers after partial failure.
118. How do you evaluate moving ADF pipelines into Azure Synapse pipelines?
I begin with an inventory of pipelines, activities, linked services, runtimes, triggers, security controls, deployment tooling, and dependencies. Azure Synapse pipelines share many ADF concepts, but they live in a Synapse workspace with different surrounding security, authoring, and analytics integrations. I do not treat a similar visual designer as proof of complete feature parity.
I compare the current documented differences and test each required connector and activity in the intended workspace configuration. Workspace-managed identities, private connectivity, permissions, resource references, and CI/CD must be mapped deliberately. Existing deployment automation for an ADF factory is not automatically a complete Synapse workspace deployment strategy.
I would migrate a representative vertical slice first, including failure recovery and monitoring, then reconcile its results against the existing workload. The business justification matters: consolidating with Synapse compute can simplify operations, while moving a stable integration estate solely for interface consistency may add risk. Azure Synapse pipelines are also distinct from Data Factory in Microsoft Fabric; migration guidance for one is not interchangeable with the other.
119. How would you assess migration from ADF to Data Factory in Microsoft Fabric?
I assess the migration workload by workload, starting with connector and authentication coverage, private connectivity, trigger semantics, deployment, monitoring, and cost. Fabric Data Factory is a separate SaaS experience with workspace and capacity considerations; it is not simply a new version number for an existing Azure factory.
I map ADF datasets and linked services to Fabric's connection and activity configuration model and verify current migration-tool eligibility. For private access, I evaluate the relevant Fabric gateway options rather than assuming an ADF SHIR can be reused unchanged. Mapping Data Flow support also needs current documentation: Fabric's migration guidance describes MDF transforms in Dataflow Gen2 with Spark-based execution in preview, so a blanket claim that Fabric has no mapping-flow path is outdated, while assuming every ADF flow migrates unchanged is equally unsafe.
I pilot a representative pipeline and compare output, failure recovery, security, operational tooling, and capacity consumption. The cutover requires checkpoint continuity and a rollback plan. A successful import is only a starting point; preview status and unsupported workload details can determine whether production migration is appropriate.
120. What evidence would convince you that an ADF platform is production-ready?
I want evidence that the platform delivers correct, timely data and recovers safely, not merely a collection of successful screenshots. That evidence includes approved architecture and permissions, versioned deployment artifacts, representative performance measurements, data-quality results, and operational ownership.
I expect demonstrations of duplicate-safe replay, partial-failure recovery, checkpoint correctness, late-arriving data handling, and source-retention monitoring where CDC is used. Security verification covers identities, secrets, network paths, and sensitive logging. Release evidence covers environment isolation, trigger handling, rollback or roll-forward, and reconciliation after uncertain outcomes.
I also require measurable freshness objectives, actionable alerts for failures and missing runs, cost attribution, and rehearsed recovery procedures. Connector and runtime assumptions must be documented with the actual deployed versions or settings rather than vague claims about ADF as a whole. Finally, I distinguish completed tests from planned ones. For this interview resource, examples and local fixture checks illustrate reasoning; they do not establish that an Azure environment has been deployed, load-tested, secured, or certified.
Azure Data Factory Documentation and Further Reading
This article targets Azure Data Factory v2 concepts, not the retired v1 model. ADF is a managed service rather than a single installable product version; connector versions, self-hosted integration runtime builds, APIs, and feature availability must be checked independently. Synapse pipelines and Fabric Data Factory have separate capabilities, deployment models, and limits. Fabric documentation evolves, including Dataflow Gen2 and Mapping Data Flow transforms; migration is not assumed to be a JSON-only lift-and-shift.
These curated primary references support further study. Documentation review and safe local parsing, arithmetic, or fixture checks do not establish deployed pipeline behavior, live connector compatibility, Azure permissions, private-network connectivity, source isolation, runtime throughput, billing, or production readiness.
Similar Posts