Senior DataOps Engineer

Job Title : Senior DataOps Engineer
Duration: 12+ Months
Rate: $80-85/hr on Vendor W2- MAX
Location: Remote or Hybrid (Charlotte, NC)
Department: Integration, Data & AI Engineering

Position Overview: Seeking a Senior DataOps Engineer to design, build, automate, and support enterprise-scale data platforms leveraging Databricks and AWS. This role is responsible for developing scalable, high-performing, and governed data pipelines while driving platform automation, Infrastructure as Code (IaC), and DataOps best practices.

The ideal candidate will have extensive experience administering Databricks environments, optimizing platform performance, implementing CI/CD pipelines, and supporting enterprise analytics and AI/ML initiatives.

Key Responsibilities

Databricks Platform Engineering & DataOps
• Design, build, optimize, and support enterprise-scale ETL/ELT pipelines within Databricks.
• Develop scalable batch and streaming data pipelines using PySpark, Spark SQL, Delta Lake, and Databricks Workflows.
• Configure and optimize Databricks clusters for performance, scalability, reliability, and cost efficiency.
• Implement Delta Lake best practices, including partitioning, schema evolution, compaction, optimization, and performance tuning.
• Administer Unity Catalog, including governance, access controls, auditing, lineage, and security.
• Design and support Medallion (Bronze, Silver, Gold) Lakehouse architectures.
• Monitor, troubleshoot, and optimize Databricks jobs, workflows, and platform services.
• Support enterprise AI/ML and analytics workloads running within Databricks.
Cloud Data Engineering
• Build and maintain scalable data ingestion and transformation pipelines using Python, PySpark, SQL, AWS Glue, and other AWS services.
• Integrate structured, semi-structured, unstructured, and streaming data from enterprise systems.
• Develop real-time data processing solutions using AWS Kinesis and Firehose.
• Partner with architecture, analytics, AI/ML, and engineering teams to deliver enterprise data solutions.
Infrastructure Automation & DevOps
• Implement Infrastructure as Code (IaC) using Terraform to provision and manage Databricks environments and AWS infrastructure.
• Automate deployments, environment provisioning, configuration management, and operational workflows.
• Design and maintain CI/CD pipelines supporting Databricks deployments and infrastructure automation.
• Manage version control repositories and DataOps best practices.
• Drive platform standardization and deployment consistency across development, test, and production environments.
Governance & Operational Excellence
• Ensure compliance with enterprise security, governance, privacy, and regulatory standards.
• Implement data quality controls, lineage tracking, auditing, and operational monitoring.
• Develop operational standards, monitoring frameworks, and support procedures.
• Provide technical leadership and mentor engineers on DataOps and Databricks best practices.

Required Qualifications
• 8+ years of experience in Data Engineering, Platform Engineering, or DataOps.
• 5+ years of hands-on experience with Databricks in enterprise environments.
• Strong experience with Python, PySpark, Spark SQL, and SQL.
• Deep expertise with Delta Lake, Databricks Workflows, Unity Catalog, cluster administration, and performance optimization.
• Experience designing and supporting Lakehouse/Medallion architectures.
• Proven experience with Terraform and Infrastructure as Code (IaC).
• Strong knowledge of CI/CD pipelines and DevOps/DataOps methodologies.
• Experience with AWS services including:
o AWS Glue
o Kinesis
o Firehose
o S3
o IAM
• Strong understanding of data governance, security, observability, and monitoring.
• Excellent communication, leadership, problem-solving, and collaboration skills.

Deliverables
• Production-ready Databricks ETL/ELT pipelines and workflows.
• Optimized and governed Databricks platform environments.
• Terraform modules and Infrastructure as Code automation.
• Monitoring and observability dashboards for Databricks workloads.
• Enterprise data models, lineage documentation, and operational runbooks.
• CI/CD pipelines and deployment automation.
• Weekly status updates and participation in Agile ceremonies.

Interested candidates send your resume to alex@amtexenterprises.com