Cloudera, Hortonworks, MapR
A Hadoop to Databricks migration moves HDFS data, Hive tables, Spark and MapReduce jobs, Oozie workflows and Ranger or Sentry policies to a governed cloud lakehouse. Spark jobs usually move with small changes; MapReduce and Oozie logic is rebuilt as declarative pipelines and Lakeflow Jobs.
HDFS data, Hive tables, Spark and MapReduce jobs, Oozie workflows, Ranger and Sentry policies
Each component has a clear target on the Databricks Data Intelligence Platform.
| Legacy Hadoop | Databricks |
|---|---|
| HDFS | Cloud object storage with Delta Lake tables |
| Hive tables and Hive Metastore | Delta tables in Unity Catalog |
| Spark on YARN | Serverless or job compute |
| MapReduce jobs | Spark code or Lakeflow Declarative Pipelines |
| Oozie workflows | Lakeflow Jobs |
| Impala queries | Databricks SQL warehouses |
| Ranger or Sentry policies | Unity Catalog permissions and audit |
Inventory every table, job, report and user. Map dependencies, find what is unused, and estimate effort and cost.
Output: Migration assessment and business caseDesign the target lakehouse, governance and security. Group workloads into waves by value and risk.
Output: Target architecture and wave planConvert SQL and ETL with LakeBridge and our accelerators, then review and test the output.
Output: Converted, tested code per waveRun old and new side by side and compare results table by table until they match.
Output: Signed-off reconciliation reportsMove users in waves, retire the old system, then tune cost and performance.
Output: Decommission plan and FinOps baselineWe use Databricks LakeBridge for profiling, code conversion and reconciliation, alongside our own accelerators and engineer review of every critical path.
An on-premise Hadoop cluster near end of support, with a backlog of regulatory and risk reports.
Read the journey: Hadoop to Databricks for a retail bank →Hadoop clusters are expensive to keep alive and hard to staff. Here is how to move without breaking the reports that depend on them.
Read the article: Leaving Hadoop: a step-by-step path to Databricks →It depends on the number of tables, jobs and users. We start with a short, fixed-scope assessment that inventories the cluster and produces a wave plan with effort and cost estimates.
Usually not. Most Spark jobs move with small changes. MapReduce jobs, Oozie workflows and custom scripts are the parts that need rewriting.
We map them to Unity Catalog permissions, then add lineage and audit that Hadoop did not provide.
