Data Engineer Lead/Architect

• NO C2C allowed - W2 only

10+ years of experience as a Data Engineer and 5+ years as lead

• Hands-on experience with Azure Databricks, Spark, and Python
• Experience with Delta Live Tables (DLT) or Databricks SQL
• Strong SQL and database background
• Experience with Azure Functions, messaging services, or orchestration tools
• Familiarity with data governance, lineage, or cataloging tools (e.g., Purview, Unity Catalog)
• Experience monitoring and optimizing Databricks clusters or workflows
• Experience working with Azure cloud data services and understanding how they integrate with Databricks and enterprise data platforms
• Experience with Terraform for cloud infrastructure provisioning
• Experience with GitHub and GitHub Actions for version control and CI/CD automation
• Strong understanding of distributed computing concepts (partitions, joins, shuffles, cluster behavior)
• Familiarity with SDLC and modern engineering practices
• Ability to balance multiple priorities, work independently, and stay organized

Key Responsibilities

• Analyze, design, and develop enterprise data solutions with a focus on Azure, Databricks, Spark, Python, and SQL
• Develop, optimize, and maintain Spark/PySpark data pipelines, including managing performance issues such as data skew, partitioning, caching, and shuffle optimization
• Build and support Delta Lake tables and data models for analytical and operational use cases
• Apply reusable design patterns, data standards, and architecture guidelines across the enterprise, including collaboration with 84.51° when needed
• Use Terraform to provision and manage cloud and Databricks resources, supporting Infrastructure as Code (IaC) practices
• Implement and maintain CI/CD workflows using GitHub and GitHub Actions for source control, testing, and pipeline deployment
• Manage Git-based workflows for Databricks notebooks, jobs, and data engineering artifacts
• Troubleshoot failures and improve reliability across Databricks jobs, clusters, and data pipelines
• Apply cloud computing skills to deploy fixes, upgrades, and enhancements in Azure environments
• Work closely with engineering teams to enhance tools, systems, development processes, and data security
• Participate in the development and communication of data strategy, standards, and roadmaps
• Draft architectural diagrams, interface specifications, and other design documents
• Promote the reuse of data assets and contribute to enterprise data catalog practices
• Deliver timely and effective support and communication to stakeholders and end users
• Mentor team members on data engineering principles, best practices, and emerging technologies