Senior Data Engineer (Cloud Data Platforms)

About the role

Valtech is looking for a Senior Data Engineer to design, build and optimise modern cloud‑based data platforms that power analytics, AI and data products across the organisation. The role combines hands‑on engineering with a strong curiosity about the business problems behind the data, ensuring that solutions are both technically robust and aligned with business needs.

Key responsibilities

• Design and implement scalable batch, streaming and near‑real‑time pipelines on AWS (Databricks) or Azure/Fabric, using Spark and Delta Lake.
• Develop lakehouse architectures with medallion layering (bronze, silver, gold) and ensure data governance, lineage and security.
• Deliver high‑quality datasets for analytics, machine learning, causal modelling and GenAI use cases, including LLM‑powered pipelines.
• Orchestrate workflows with Airflow, dbt, Databricks Workflows or Azure Data Factory, focusing on automation and reliability.
• Collaborate with data scientists, ML engineers and platform teams to support production‑grade AI systems.
• Monitor and optimise pipeline performance, cost and scalability, and establish observability metrics.

Required profile

• Strong hands‑on experience with Apache Spark, Delta Lake, Python and SQL.
• Deep expertise in either AWS data services (S3, Glue, EMR, Redshift, Kinesis, Lambda) or Azure/Fabric services (Data Factory, Synapse, Fabric Lakehouse, Azure Databricks).
• Experience with lakehouse architectures, data modelling and governance best practices.
• Curious, analytical mindset with the ability to translate business questions into data solutions.

Required skills

• Apache Spark
• Delta Lake
• Python
• SQL
• AWS (S3, Glue, EMR, Redshift, Kinesis, Lambda)
• Microsoft Azure / Fabric (Data Factory, Synapse, Fabric Lakehouse, Azure Databricks)
• Databricks Lakehouse Platform
• Airflow
• dbt
• Unity Catalog / AWS Lake Formation / Microsoft Purview
• Terraform or CloudFormation
• Kafka (optional)