DataStage migration, down to the detail.

Explore the stages, expressions and delivery capabilities behind PipelineX. See how your DataStage designs become inspectable code and a managed migration to Microsoft Fabric, Databricks or Snowflake.

What travels with your job
Source designIBM DataStageJobs, stages, links & sequences
Business rulesData typesExecution flow
Target artifactsCode you can inspect.
Context you can follow.
Microsoft FabricDatabricksSnowflake
Source understanding

Inventory, search, job graphs and dependencies.

Detailed conversion

Stage logic, expressions, types and output behavior.

Target artifacts

Transformation code, table definitions and orchestration.

Managed delivery

Reviews, validation, Fabric CI/CD and acceptance.

Source stages captured for inspection.

Capture 70 DataStage stage types across database connectors, cloud storage, files, processing and diagnostics. Inspect stage settings, field definitions and link relationships in their original job context.

Databases and enterprise connectors

Inspect declared connection references, table and query definitions, connector settings, column interfaces, and input/output links.

  • Oracle ConnectorOracleConnectorPX
  • IBM Db2 ConnectorDB2ConnectorPX
  • ODBC ConnectorODBCConnectorPX
  • JDBC ConnectorJDBCConnectorPX
  • Dynamic RDBMS ConnectorDRSConnectorPX
  • Teradata ConnectorTeradataConnectorPX
  • Netezza ConnectorNetezzaConnectorPX
  • SybasePxSybase
  • Informix XPSPxInformixXPS
  • Google BigQuerybigqueryPX
  • Amazon RedshiftRedshiftPX
  • IBM Cognos TM1 ConnectorCognosTM1ConnectorPX
  • Stored ProcedureSTPPX
  • SAP ABAP ExtractionABAP_EXT_for_R3_PX

Files, cloud storage and mainframe data

Retain declared paths, file and cloud settings, format metadata, and column definitions. Complex Flat File layouts stay connected to their owning stage through the original record-set declarations.

  • Sequential FilePxSequentialFile
  • Data SetPxDataSet
  • File SetPxFileSet
  • Lookup File SetPxLookupFileSet
  • File ConnectorFileConnectorPX
  • Complex Flat FilePxCFF
  • z/OS FilePxzOSFile
  • Big Data FilePxBDFS
  • SASPxSAS
  • Unstructured Data ConnectorUnstructuredDataConnectorPX
  • External TargetPxExternalTarget
  • Amazon S3AmazonS3PX
  • Azure Storage ConnectorAzureStorageConnectorPX
  • Azure Data Lake StorageAzureDatalakePX
  • Google Cloud StorageGoogleCloudStoragePX
  • IBM Cloud Object StoragecloudobjectstoragePX

Processing and transformation stages

Inspect the source design through declared keys, expressions, operator properties, link order, and output definitions. Stage-specific settings remain available alongside the job's connected structure.

  • TransformerCTransformerStage
  • AggregatorPxAggregator
  • Bloom FilterPxBLM
  • CopyPxCopy
  • FilterPxFilter
  • JoinPxJoin
  • LookupPxLookup
  • MergePxMerge
  • ModifyPxModify
  • FunnelPxFunnel
  • SortPxSort
  • Remove DuplicatesPxRemDup
  • SwitchPxSwitch
  • Pivot and Pivot EnterprisePivotPX / PxPivot
  • Slowly Changing DimensionPxSCD
  • Surrogate Key GeneratorPxSurrogateKeyGeneratorN / PxSurrogateKeyGenerator
  • Change CapturePxChangeCapture
  • Change ApplyPxChangeApply
  • ComparePxCompare
  • DifferencePxDifference
  • ChecksumPxChecksum
  • Data MaskingDMConnectorPX
  • CompressPxCompress
  • ExpandPxExpand
  • EncodePxEncode
  • DecodePxDecode
  • External FilterPxExternalFilter
  • FTPPxFTP
  • Generic OperatorPxGeneric
  • Wave GeneratorPxWaveGenerator

Generation, sampling and diagnostics

Capture generation, sampling, record-selection and range-map settings, together with the declared link and column definitions used to inspect a job.

  • Column GeneratorPxColumnGenerator
  • Row GeneratorPxRowGenerator
  • HeadPxHead
  • TailPxTail
  • SamplePxSample
  • PeekPxPeek
  • Write Range MapPxWriteRangeMap

Container structure

Retain container definitions, boundary links and nested job structure so engineers can inspect reusable components in their surrounding dataflow.

  • ContainerContainerStage
From captured design to conversion behavior.

Source capture gives your team an inventory to inspect and assess. The conversion catalog below describes the specific transformations, source adapters and orchestration behaviors PipelineX generates.

Conversion behavior, stage by stage.

Explore stage conversions for Spark, source access and Fabric sequence activities. Open an entry for supported behavior. The target outputs section shows how each destination is packaged.

29 capability entries

Stage or activity Supported behavior

Transform & derive

Carry business rules into target expressions and typed outputs.

TransformerDerivations, stage variables and output constraints.

Translate nested expressions, evaluate stage variables in dependency order, apply output constraints and assign values to declared output types. Generate separate accepted and reject streams for configured routing.

  • Nested expressions
  • Stage variables
  • Reject streams
CopyIndependent column selection for each output link.

Build a distinct projection for each outgoing link, preserving its selected columns, column order and declared output assignments.

  • Column projection
  • Multiple outputs
ModifyColumn renaming, removal and type conversion.

Translate captured field transformations into named target projections and casts so the output schema follows the source design.

  • Rename
  • Drop
  • Cast
AggregatorGrouped COUNT, SUM, MIN and MAX calculations.

Use captured grouping keys and aggregate expressions, including declared decimal and double calculation types. Carry nullable grouping keys and output assignments into the generated calculation.

  • Grouping keys
  • Decimal precision
  • Typed aggregates
Row GeneratorSequential integer values and a declared row count.

Generate rows from captured integer sequence settings and the requested record count. Carry declared output columns into downstream transformations.

  • Integer sequences
  • Record count
Surrogate Key GeneratorGenerated row keys with an explicit starting offset.

Generate surrogate keys with an explicit starting offset available for configuration and review.

  • Generated keys
  • Starting offset

Join & enrich

Make matching rules and reference behavior visible in the conversion.

JoinKey-based inner, left, right and full joins.

Build target joins from explicit input roles and key mappings. Carry selected output columns into the resulting projection.

  • Inner / outer joins
  • Input roles
  • Join keys
LookupEqui-joins, range matching and configured lookup outcomes.

Translate ordered reference lookups, equi and range conditions, and configured multirow matching. Model Continue, Drop, Fail and Reject outcomes with accepted and reject output handling.

  • Range conditions
  • Multiple references
  • Outcome routing
MergeMaster and update inputs with key-based enrichment.

Use captured master and update input order, merge keys and output mappings to build the enrichment logic and retain unmatched master rows.

  • Master / update roles
  • Key mappings
Slowly Changing DimensionType 1 updates and source-declared Type 2 history transitions.

Translate captured dimension keys, change columns and state assignments. Generate insert-only, Type 1 and source-declared Type 2 or mixed transitions with expiry expressions, new dimension versions and downstream resolved keys.

  • Business keys
  • Type 1 / Type 2
  • History lifecycle
Change CaptureInsert-only change comparison by key.

Generate the insert-only comparison path from before and after inputs and captured comparison keys.

  • Before / after inputs
  • Insert detection

Route & order

Retain the rules that decide where a row goes and which row survives.

FilterFirst-match and multiple-match predicate routing.

Translate typed predicates with explicit null handling. Route rows according to the captured first-match or multiple-match setting, including the configured reject link.

  • Typed predicates
  • Match mode
  • Reject output
SwitchSelector-based routing to multiple output links.

Use the captured selector expression and case definitions to generate the corresponding output branches.

  • Selector expression
  • Output branches
FunnelContinuous, sequence and sorted input modes.

Combine named input branches using the captured mode. Apply branch projections and declared output types, and use captured ordering requirements for sequence and sorted delivery.

  • Three funnel modes
  • Branch projection
  • Output types
SortDeclared sort keys, direction and null placement.

Read source sort requirements and carry ordering metadata into target sorting and downstream operations that depend on row order.

  • Ascending / descending
  • Null placement
  • Order metadata
Remove DuplicatesFirst or last survivor using declared keys and ordering.

Use duplicate keys, sort keys and partition information to express which row survives. Carry that selection into the downstream output.

  • Duplicate keys
  • First / last
  • Partition information
PeekSelected-field row logging in generated execution.

Emit row logging for the configured fields, keeping the diagnostic step connected to the generated data flow.

  • Selected columns
  • Row logging

Files & source access

Bring file structure, source SQL and connection context into target preparation.

Sequential FileDelimited file reads and writes with captured format settings.

Read comma, tab, pipe and semicolon delimiters with captured quoting, header, null-marker, encoding and file-selection settings. Generate comma-delimited writes with column order, LF or CRLF record endings and the configured header and quote policy.

  • Read / write
  • File patterns
  • Encoding & framing
Data SetTyped Delta exchanges between producer and consumer jobs.

Translate paired Data Set producers and readers into versioned Delta exchanges. Retain the logical source identity and declared schema across the job boundary.

  • Producer / consumer
  • Versioned Delta
  • Schema continuity
Oracle ConnectorSource SQL, connection identity and parameterized reads.

Preserve captured OracleConnector and OracleConnectorPX queries and parameters. Prepare source reads with explicit parameter binding and typed date and timestamp transport.

  • Original SQL
  • Parameter binding
  • Typed source reads
SAP ABAP source extractionA source-side RFC extraction adapter for captured SELECT logic.

Prepare source-side ABAP extraction using the original SELECT and declared field mappings for data transport into the target flow.

  • ABAP SELECT
  • RFC adapter
  • Source extraction

Job sequences

Rebuild execution relationships and control flow in Fabric orchestration.

Job ActivityChild-job execution, parameters and recorded outcomes.

Invoke child jobs with captured parameter handoff. Use recorded success, warning and failure outcomes to drive the sequence.

  • Child runs
  • Parameters
  • Status branches
Execute CommandCaptured UNIX commands through a configured execution adapter.

Carry the source command into adapter-based execution and return its outcome to the sequence control flow.

  • Command adapter
  • Execution outcome
User VariablesOrdered variable expressions in sequence state.

Evaluate captured variable expressions in order and make their values available to subsequent sequence activities and conditions.

  • Expression order
  • Sequence state
SequencerAll-input and any-input execution joins.

Coordinate incoming activities using the captured all or any setting before continuing to the next activity.

  • All / any
  • Activity dependencies
Nested ConditionCustom trigger expressions and conditional branches.

Translate captured trigger expressions into sequence decisions using activity outcomes and the available sequence context.

  • Custom triggers
  • Conditional branches
Start Loop / End LoopNumeric and delimited-list iteration in a linear loop.

Carry captured numeric bounds or delimited-list values into loop execution, with the iteration value available to the loop body.

  • Numeric bounds
  • List values
  • Iteration state
Exception HandlerException paths connected to sequence outcomes.

Route execution into the captured exception-handling path and retain the activity outcome in the sequence record.

  • Exception paths
  • Outcome context
TerminatorStop the sequence and wait for active child runs.

Apply captured termination behavior, including child-run stop and wait handling. Track execution progress for sequence restart and recovery.

  • Stop / wait
  • Child-run handling
  • Restart & recovery

Start with the stages in your estate. Bring a representative XML export to see the corresponding target artifacts.

Walk through your jobs with us

The details that define the result.

A business rule includes its types, intermediate values and output behavior. PipelineX gives those details a place in the conversion.

Nested logic retains its structure.

Translate nested function calls, conditional logic and concatenation while respecting expression structure, quoted values, identifiers and parameters. Inspect the resulting expression in its source-job context.

If / Then / Else, IIF, DECODE, nested calls

Null is part of the business rule.

Null-aware comparisons, typed predicates and output assignments carry null behavior through filters, Transformers, lookups and aggregates. Runtime comparison distinguishes null values from empty strings.

IsNull, NullToValue, nullable keys, empty strings

Precision follows the value.

Declared precision and scale inform decimal calculations and assignments. Ceiling, floor, nearest with ties away from zero, and truncation toward zero receive distinct rounding handling. Conversion, arithmetic and assignment keep their own numeric boundaries.

Four rounding modes, precision / scale, display scale

Dates carry a format and a calendar.

Date and timestamp conversion uses captured types, format expressions and calendar rules. Microsecond timestamp capacity follows typed values through source reads, stage variables and history assignments.

Date formats, microsecond precision, calendar arithmetic

Order is carried where logic depends on it.

Sort keys, direction, null placement and partition information inform order-sensitive operations. Duplicate removal, sorted funnels and sequential calculations use the captured ordering requirements.

Sort direction, first / last survivor, ordered inputs

A lookup includes what happens next.

Matching keys are one part of lookup behavior. Conversion also considers reference order, range conditions, multiple matches and the configured action when a lookup succeeds or fails.

Continue, Drop, Fail, Reject, accepted / reject streams

Stage variables have an evaluation order.

Variable dependencies are resolved before output expressions are assigned. Declared string widths, numeric types and timestamp types inform the intermediate values used by downstream expressions.

Dependency order, typed intermediate values, output constraints

Source context travels with the code.

Schemas, derivations, connection references and job parameters remain available for inspection. Generated mappings connect source fields and stage names to target expressions and artifacts.

Column mappings, connection bindings, parameters, source stages

A field’s journey matters.

When a Lookup assigns a 30-byte field before a later 50-byte output, the target receives the value assigned at that intermediate step. PipelineX accounts for the field width where the assignment happens.

Source field50 bytes
Lookup output30 bytes
Target field50 bytes
Illustrative field-width assignment

170+ registered function names, including aliases.

String, date and time, numeric, casting, null, bit and row expressions. Browse the published function reference and target mappings.

Explore function mappings

Know what your team receives.

Each destination has its own artifacts. Inspect the transformation logic, table structure and orchestration in the format your engineers will work with.

Microsoft Fabric

Spark notebooks & Data Factory

A migration package that connects transformation code, Lakehouse setup and Fabric orchestration.

Transformation
PySpark notebook source and Fabric notebook definitions.
Data foundation
Lakehouse DDL and setup notebooks with column definitions.
Orchestration
Data Factory pipeline JSON and mapped job-sequence dependencies.
Delivery package
Parameter mappings, validation notebooks, deployment scripts and review artifacts.
Explore the Fabric migration path

Databricks

PySpark, Delta Lake & Workflows

Spark transformation code with Delta table setup and workflow artifacts for the destination environment.

Transformation
PySpark notebooks with Spark SQL expressions.
Data foundation
Delta Lake table DDL and column mappings.
Orchestration
Databricks Jobs workflow JSON generated from job sequences.
Engineering handover
Parameters, source context and generated conversion reports.
Explore the Databricks migration path

Snowflake

SQL, Snowpark & Tasks

Snowflake-oriented code artifacts that make translated logic, table structure and sequence dependencies available to engineers.

Transformation
Snowflake SQL mappings and Snowpark Python scripts.
Data foundation
Table DDL with target column types.
Orchestration
Snowflake Task SQL generated from job-sequence dependencies.
Engineering handover
Parameters, source context and generated conversion reports.
Explore the Snowflake migration path
Source context stays close.

Generated packages include supporting mappings and review artifacts so engineers can connect the delivered files to the captured job design.

Understand the estate.
Organize the work.

Give architects a view of the source logic and delivery leads a way to organize the migration around jobs, people and evidence.

Explore projects & teams

An inventory from your XML exports

Import IBM DataStage XML and inspect the captured jobs, stages, links, schemas and parameters. Import diagnostics record the source capture and give each job a place in the project.

Search that opens the source context

Find jobs, stages, columns, SQL, expressions and parameters across the imported projects you can access. Open the matching job and investigate the design behind the result.

Job graphs and dependencies

Follow stage-to-stage data flow, inspect column mappings and connect jobs through shared source and target tables. Use dependencies to plan the order of migration work.

Complexity for migration planning

Use stage-based complexity indicators to organize the estate. Build migration groups and waves, assign owners and track work against acceptance criteria.

An AI assistant with job context

Start a conversation from the job you are inspecting. Preview and enable selected source context, then use your configured AI provider to explore the logic and migration questions.

Client, project and team organization

Organize customer work in client accounts and projects with membership and access permissions. Keep assignments, review work and delivery evidence connected to the responsible team.

Evidence through to delivery.

Connect the code under review, the results of a controlled pilot and the decisions behind acceptance. Keep the record available as the migration progresses.

  1. Review the generated package

    Inspect source logic alongside generated code. Package checks cover artifact integrity, Fabric item structure, conversion checks and notebook syntax. Code revisions keep the reviewed version identifiable.

    The record

    Code revision, package files, check results

  2. Compare controlled runtime results

    Use a configured Fabric execution adapter and customer-supplied DataStage baseline results for pilot verification. Compare a complete pilot output using schema, row-count, key, value and aggregate rules. Record elapsed time against the agreed batch window.

    The record

    Baseline, comparison rules, output results, run timing

  3. Connect Fabric delivery to CI/CD

    With configured GitHub or Azure DevOps connections, save a prepared Fabric package to a review branch and open a draft pull request. Follow checks for the recorded commit through GitHub Actions or Azure Pipelines.

    The record

    Review branch, pull request, commit, provider checks

  4. Keep acceptance tied to the work

    Track ownership, review decisions, recorded effort and handover. Evidence and acceptance stay tied to the source and artifact revision; changes surface the work that needs renewed review.

    The record

    Owners, acceptance criteria, decisions, migration reports

Put your own jobs
under the lens.

Bring a representative DataStage XML export. We’ll walk through the source design, generated artifacts and the evidence your team wants to see.

Arrange a technical walkthroughRead the migration guide