Data lake - Revolutionizing Travel Data Management

A Netherlands-based company operating in the global online travel sector, offering complex multi-leg flight connections across more than 50 countries. By combining routes from multiple carriers, including smaller airlines, the company delivers flexible travel options supported by a broad partner ecosystem.

Rapid growth, combined with acquisitions and expanding partnerships, led to a fragmented data environment where multiple systems operated independently, making it difficult to maintain consistency and extract value from data. As data volumes increased, the organization struggled to process information efficiently and turn it into actionable insights.

At the same time, the business needed to better understand customer behavior and respond quickly to changing market conditions, which required faster access to reliable data and the ability to analyze it in real time. Without a unified approach, decision-making was slower, personalization was limited, and operational complexity continued to grow.

To support further expansion, the platform needed to:

• integrate data from multiple distributed systems into a single environment
• support structured, semi-structured, and unstructured data
• enable real-time data processing and updates
• improve data quality through cleaning and aggregation
• support advanced analytics and personalization
• scale with growing data volumes and business needs

The goal was to create a centralized data foundation that could support both operational efficiency and strategic growth.

The project combined data lake and platform development to implement a scalable architecture that centralizes information across multiple sources and supports consistent processing, analysis, and reporting. Data ingestion, transformation, and storage were integrated into a unified pipeline, preparing information for downstream analytical applications.

A key element of the solution was the introduction of Change Data Capture (CDC), which enables real-time tracking of data changes and supports immediate processing rather than delayed reporting. This significantly improves the timeliness and relevance of insights.

The platform also supports advanced analytics use cases, including predictive and prescriptive models, enabling the organization to better understand customer behavior and optimize offerings. By creating a flexible and scalable data environment, the system supports both current operations and future expansion.

The centralized data lake provides a foundation for analyzing customer behavior, identifying travel patterns, and supporting personalized offers. Our guide to advanced data analytics explains the methods organizations can use to extract meaningful insights from large and diverse datasets.

Technology stack:

• Kafka for data streaming
• Debezium for Change Data Capture
• Java (Spring Boot) for backend services
• Talend / ETL tools for data processing
• PostgreSQL and CockroachDB for storage
• PHP for supporting services

Data is collected from multiple systems and ingested into the data lake, where it is cleaned, aggregated, and enriched to ensure quality and consistency. CDC mechanisms capture changes in real time, allowing the system to process updates continuously rather than in batches.

Once processed, the data is used for reporting, analytics, and business intelligence, enabling teams to identify patterns, relationships, and opportunities. The architecture supports integration with modern technologies, making it possible to extend the platform with advanced analytics and machine learning capabilities.

Key capabilities:

• Centralized data ingestion from multiple systems
• Support for structured, semi-structured, and unstructured data
• Real-time data processing using CDC mechanisms
• Data cleaning, aggregation, and enrichment
• Advanced analytics and reporting
• Scalable architecture supporting business growth

The introduction of a centralized data platform significantly improved how data is accessed and processed across the organization, reducing fragmentation and enabling faster, more reliable insights. Teams can now work with consistent data, improving collaboration and reducing the time required to analyze information.

Real-time data processing also allows the organization to respond more quickly to changes, supporting more dynamic and efficient operations.

The platform delivered measurable improvements across key areas:

• Unified data environment, reducing system fragmentation
• Faster decision-making, through improved data access and quality
• Real-time insights, enabled by continuous data processing
• Improved personalization, through better understanding of customer behavior
• Scalable architecture, supporting rapid business growth
• Cost optimization, through centralized and automated data processing

The platform continues to evolve as new data sources, analytics capabilities, and technologies are introduced. It provides a strong foundation for expanding into new markets, improving customer experience, and leveraging data as a strategic asset across the organization.

How does Change Data Capture keep a travel data lake updated in real time?

Change Data Capture (CDC) tracks changes in source systems as they happen, so the data lake processes updates continuously rather than in batches. In this project, InTechHouse used Debezium for CDC and Kafka for data streaming, feeding a pipeline where data is cleaned, aggregated and enriched before it is used for reporting, analytics and business intelligence.

Can a data lake combine structured, semi-structured and unstructured travel data from acquired systems?

Yes, a data lake can centralize structured, semi-structured and unstructured data from multiple distributed systems in one environment. In this project, InTechHouse integrated data ingestion, processing and storage into a single pipeline, with PostgreSQL and CockroachDB for storage, Java (Spring Boot) for backend services and Talend or other ETL tools for data processing.

What analytics use cases does a centralized travel data lake support?

A centralized travel data lake supports reporting, business intelligence and advanced analytics, including predictive and prescriptive models. In this project, InTechHouse built the platform as a foundation for analyzing customer behavior, identifying travel patterns and supporting personalized offers, with an architecture that can be extended with machine learning capabilities as new data sources are introduced.