Plan a pod
Menu
Platform

Real-time on the lakehouse: Real-Time Mode and Lakehouse//RT

Spark Declarative Pipelines gained a real-time mode, and Lakehouse//RT entered beta as a real-time SQL warehouse.

By NeuralShore EngineeringPublished 1 min read
Cover illustration: Real-time on the lakehouse: Real-Time Mode and Lakehouse//RT
Short answer

Real-time mode for declarative pipelines and the Lakehouse//RT warehouse let fraud checks, payments monitoring and network analytics run on the lakehouse with very low latency, instead of a separate streaming stack. Check each feature’s availability for your cloud region, because some capabilities start in beta or in US regions first.

Key takeaways

  • One platform for batch and real-time.
  • Strong fits: fraud, payments and network monitoring.
  • Check feature status: some capabilities are in beta.

Fraud checks, payment monitoring and network analytics used to need a separate streaming stack. That meant another set of tools, another team and another security model.

What is new

Spark Declarative Pipelines now has a real-time mode for continuous streaming with very low latency. It is generally available. Lakehouse//RT is a new real-time SQL warehouse type for low-latency queries on Delta and Iceberg tables. It is in beta. ZeroBus Ingest adds managed, Kafka-compatible push ingestion for high-volume events.

Why it matters

When streaming lives on the lakehouse, it uses the same governance, lineage and access rules as everything else. One team can own the full path from event to decision.

How to pilot it

  1. Pick one use case where seconds matter, such as card fraud or payment failures.
  2. Ingest events with ZeroBus or an existing Kafka feed.
  3. Build the scoring logic in a declarative pipeline with real-time mode.
  4. Serve results to the team that acts on them, with clear alert rules.
  5. Measure latency, cost and false positives for a month before expanding.

Because Lakehouse//RT is still in beta, we keep production serving on proven warehouses and test the new type side by side.

About the author

NeuralShore Engineering, Databricks delivery team

Databricks architects, data engineers and AI engineers at NeuralShore, DLytica’s Databricks delivery company. Leadership comes from IBM data and AI architecture.

FAQ

Frequently asked questions

Do we still need a separate streaming platform?

Often not. Real-time pipelines and a real-time warehouse on the lakehouse cover many fraud, payments and monitoring use cases.

Are these real-time features generally available?

Real-time mode for declarative pipelines is generally available; Lakehouse//RT was in beta at announcement. Check current status for your region.

Keep reading

Related articles