Carry forward the business rules
Begin with the source stage graph and column derivations. Identify joins, lookups, filters and calculations that define the result. Capture expected behaviour for unmatched records, duplicate keys and rejected rows before rewriting it.
Choose an execution shape
Pyspark transformations and delta tables provide a starting point for the target design. Databricks provides reference documentation for PySpark and Delta Lake APIs. Choose and test against the runtime your workload will actually use.
PipelineX generates target code for engineering review. Inspect the stage mappings, function translations and conversion findings for each job, then configure its connections and runtime for the intended environment.
Make it operational
Agree catalog and schema names, select an appropriate runtime and establish how jobs receive parameters, credentials and compute resources.
Keep environment-specific settings separate from transformation logic. Check permissions, failure handling, rerun behaviour and monitoring with the team that will operate the workload.
Compare the result
Run source and target against controlled inputs. Compare schemas, business keys, record counts, nulls, duplicate behaviour and important aggregates. Investigate record-level differences before accepting the migration.
Test edge cases deliberately: an unmatched lookup, an empty feed, a duplicate join key and a value near a decimal or date boundary.