Platform Data Engineer - Contractor

About the Role:

We are looking for a talented Platform & Data Engineer - Contractor to join our greenfield project, where you'll play a critical role in building and deploying a cutting-edge data platform in Azure Databricks. As part of a highly technical team, you’ll be responsible for designing, implementing, and managing the entire infrastructure pipeline, from provisioning cloud resources in Azure to deploying scalable data solutions on Databricks.

Your expertise will be pivotal in ensuring the seamless operation of the platform, from cloud infrastructure provisioning to continuous deployment and monitoring on Databricks.

What Your Day Will Look Like:

• Designing, deploying, and managing cloud infrastructure on Azure, ensuring it supports the data platform and applications at scale.
• Implementing and managing Infrastructure as Code (IAC) using Terraform to automate provisioning and maintain cloud resources.
• Supporting Databricks deployment, including the integration of data pipelines, notebooks, and cluster management.
• Working with the broader data engineering team to optimise the data platform for scalability, performance, and security.
• Collaborating with other engineering teams to ensure a smooth deployment process and end-to-end delivery of the solution.
• Automating continuous integration and continuous deployment (CI/CD) pipelines for the data platform, ensuring efficient and reliable delivery of updates.
• Ensuring platform security, monitoring, and alerting are in place to maintain a high level of uptime and performance.
• Assisting with troubleshooting and optimising infrastructure and data pipelines to meet operational and performance goals.

Requirements:

• 5+ years of experience in platform and data engineering, with hands-on experience building and managing cloud-based platforms.
• Strong experience with Azure cloud infrastructure, including provisioning resources, managing services, and configuring network security.
• Expertise in Infrastructure as Code (IaC) using Terraform or Bicep for automating cloud resource provisioning and infrastructure management.
• Experience deploying, managing, and optimising Databricks environments, including clusters, notebooks, jobs, and integrations with data pipelines.
• A solid understanding of CI/CD pipelines, automation, and deployment processes for cloud-based data platforms.
• Strong technical background in data engineering, including building data pipelines, managing large-scale data systems, and working with cloud-native tools.
• Familiarity with dbt and data modelling principles to work closely with the data engineering team in deploying solutions.
• Excellent communication and problem-solving skills, with the ability to collaborate effectively across multiple teams.
• Ability to quickly adapt to new technologies and solve complex infrastructure and deployment challenges.

Bonus Points:

• Experience with machine learning pipelines or MLflow on Databricks.
• Familiarity with containerisation technologies like Docker or Kubernetes for managing data workflows.
• Experience with monitoring tools, FinOps and cost management for cloud infrastructure and Databricks environments.

If you are passionate about building robust platforms and data solutions at scale, automating infrastructure, and ensuring smooth deployment workflows in Azure Databricks, we’d love to hear from you!