About this role
Role Overview
We are seeking hands-on Databricks Platform & Data Engineer to operate and continually develop its Databricks data platform. This role will bring new data sources onto the platform, maintain dependable and cost-aware data operations, and establish a well-governed data catalog that enables business and executive users to explore trusted information through Databricks Genie. The successful candidate will work across platform operations, data engineering, governance, and business enablement. They will partner with data owners and business stakeholders to turn source systems and business questions into secure, reliable data products and useful Genie experiences. Candidates with strong hands-on Databricks experience are highly preferred. Applicants should ideally have completed at least three Databricks projects, with relevant Databricks certification considered a strong advantage.
Key Responsibilities
1. Data-source onboarding and engineering
- Assess new source systems and design secure, maintainable ingestion patterns for databases, business applications, files, APIs, and other approved sources.
- Build and maintain batch and, where needed, streaming data pipelines using Databricks and suitable data-engineering tools.
- Develop curated, documented datasets that are accurate, timely, and fit for analytics and executive reporting.
- Implement data-quality checks, reconciliation, error handling, and observability across ingestion and transformation workflows.
- Coordinate with source-system owners and business teams on access, definitions, refresh requirements, and data issues.
2. Governance, catalog, and Genie enablement
- Develop and maintain a governed data catalog using Unity Catalog, including catalogs, schemas, ownership, descriptions, tags, lineage, and access controls.
- Work with data owners and subject-matter experts to define business-friendly names, terms, measures, relationships, and definitions for priority datasets.
- Ensure that executive-facing data is appropriately permissioned, documented, and traceable to trusted sources.
- Configure and improve Databricks Genie spaces for executive users, including selecting and preparing relevant data, adding clear business instructions, and validating representative questions and answers.
- Gather feedback from users, monitor quality, and refine datasets and Genie configurations to improve clarity, trust, and usefulness.
- Promote responsible self-service analytics while maintaining privacy, security, and least-privilege access.
3. Collaboration and continuous improvement
- Translate business reporting needs into technical requirements, delivery plans, and measurable outcomes.
- Partner with analytics, IT, security, and business teams to prioritize platform improvements and data-source onboarding.
- Maintain concise technical documentation, data definitions, support procedures, and knowledge-transfer materials.
- Identify opportunities to simplify operations, improve adoption, and increase confidence in the organization’s data.
Required Qualifications
- Demonstrated experience building and operating data platforms or production data pipelines, ideally with Databricks.
- Strong SQL skills and practical experience with Python and/or PySpark for data engineering.
- Experience ingesting data from multiple source types and implementing transformation and data-quality practices.
- Understanding of data modeling, analytical datasets, metadata management, and access governance.
- Experience troubleshooting production workloads and communicating technical issues and resolutions clearly.
- Ability to work effectively with both technical teams and business stakeholders, including non-technical users.
- Databricks is highly recommended, and candidates should have hands-on experience with the platform.
- Candidates with at least 3 completed Databricks projects will be strongly preferred.
Preferred Qualifications
- Hands-on experience with Databricks Unity Catalog, workflow/job orchestration, and platform administration.
- Experience preparing governed datasets and configuring or supporting Databricks Genie for natural-language analytics.
- Familiarity with cloud data platforms, identity and access management, secrets management, and secure connectivity to source systems.
- Experience with CI/CD, Git, infrastructure as code, automated testing, and deployment practices for data workloads.
- Knowledge of data lineage, privacy controls, auditability, and enterprise data-governance practices.
- Relevant Databricks certification is highly recommended.
What success looks like
- Priority data sources are onboarded through repeatable, documented, and supportable pipelines.
- Production jobs meet agreed reliability and freshness expectations, with clear monitoring and incident procedures.
- Curated datasets are discoverable, documented, governed, and trusted by their intended users.
- Executive users can answer agreed priority business questions through Genie using appropriately governed data.
- Data-quality issues, platform incidents, and unnecessary consumption are identified and addressed proactively.
- Business and technical stakeholders understand who owns key data, how it is defined, and how it can be accessed safely.
Candidate profile
You are an accountable, practical engineer who enjoys owning a platform beyond initial implementation. You balance reliable operations with thoughtful data engineering, can explain technical choices in business terms, and care about making trusted data easy for people to find and use.
