Senior Data Engineer

Senior Data EngineerHillwood, a Perot Company, is a premier real estate investment and development company founded on a culture of integrity, respect, excellence and teamwork. The company is a full-service real estate developer, investor and advisor focused on developing opportunities for investors, partners and communities around the world.Hillwood is seeking a Senior Data Engineer to join its Corporate IT team in Dallas, Texas. This role will play a key part in the development and ongoing evolution of Hillwood's enterprise-wide Data Lakehouse on the Databricks platform, which serves as the foundation for analytics, artificial intelligence, and data-driven decision-making across the organization.The Senior Data Engineer is responsible for designing, developing, and maintaining scalable data pipelines and integration solutions that deliver reliable, trusted, and AI-ready data across all business units. This role partners closely with business stakeholders and technology teams to understand data requirements, ensure data quality, and deliver trusted data solutions that support meaningful business insights. The ideal candidate combines strong technical expertise with a customer-focused mindset and a commitment to delivering secure, scalable, and high-quality data solutions that support business growth and innovation.Responsibilities:Data Engineering:Design, develop, and maintain scalable data pipelines within the Databricks platform to integrate data from Hillwood's core business systems.Develop and maintain data models across Bronze, Silver, and Gold layers in accordance with Hillwood's enterprise data architecture and medallion framework.Implement incremental processing and change data capture (CDC) methodologies to ensure data remains current, accurate, and cost-efficient.Administer and maintain Unity Catalog, including schema management, data governance, permissions, and lineage tracking.Platform Reliability and Engineering Excellence:Design and support production-ready data pipelines with automated scheduling, monitoring, alerting, and recovery capabilities.Optimize data processing workflows, jobs, and compute resources to improve performance, scalability, and cost efficiency.Establish and maintain engineering best practices, including version control, code reviews, automated testing, and continuous integration/continuous deployment (CI/CD) processes.Create and maintain technical documentation, standards, and operational procedures to support long-term platform sustainability and knowledge transfer.Data Quality and Governance:Develop and implement data quality frameworks, validation rules, reconciliation processes, and monitoring capabilities to ensure data accuracy and reliability.Investigate and resolve data quality issues by identifying and addressing root causes.Partner with business stakeholders and data owners to define data standards, ownership, business definitions, and quality expectations.Ensure certified and business-critical datasets align with Hillwood's data governance framework and established quality standards.AI-Enabled Data Solutions:Leverage AI-assisted development tools, including coding assistants and platform-native AI capabilities, to accelerate development, improve code quality, and increase team productivity.Support the establishment of standards and best practices for the responsible use of AI-assisted software development.Continuously evaluate emerging AI technologies and identify opportunities to improve engineering processes, platform operations, and solution delivery.Prepare and optimize lakehouse data assets for AI, machine learning, and natural language access use cases.Advanced Analytics and AI Platform Enablement:Design and build data foundations that support advanced analytics and AI initiatives, including feature stores, vector search, retrieval architectures, and document-based data pipelines.Partner with AI engineers, developers, and business stakeholders to deliver solutions that integrate structured and unstructured data within the Databricks ecosystem.Support AI workloads and platform capabilities, including model serving, agent frameworks, and emerging GenAI use cases.Identify and onboard new structured, unstructured, and third-party data sources that expand the value of the enterprise data platform.Business Partnership and Knowledge Ownership:Collaborate directly with finance and business unit stakeholders to understand business processes, data requirements, and reporting needs.Validate data outputs and business metrics with the individuals who own the data before datasets are certified for business use.Communicate technical concepts, risks, and trade-offs effectively to both technical and non-technical audiences.Follow Hillwood's principle that technology solutions are business-led, partnering closely with stakeholders to align data capabilities with operational objectives and business priorities.Partner with external engineering resources, review solution quality, and assume long-term ownership of delivered data pipelines and platform components.Serve as an internal subject matter expert on the architecture, operation, and ongoing evolution of Hillwood's Enterprise Data Lakehouse platform.Required Skills and Abilities:Strong proficiency in SQL, Python, PySpark, and modern data engineering development practices.Strong understanding of lakehouse architectures, data modeling principles, and enterprise data management concepts.Ability to design scalable, reliable, and maintainable data solutions that support reporting, analytics, and AI use cases.Knowledge of data governance, data quality, metadata management, and data lineage best practices.Familiarity with source control, testing methodologies, code reviews, and CI/CD processes.Strong analytical and problem-solving skills, with the ability to identify root causes and implement effective solutions.Ability to balance performance, scalability, maintainability, and cost considerations when designing data solutions.Effective written and verbal communication skills, including the ability to explain technical concepts to non-technical audiences.Strong organizational skills and attention to detail, with the ability to manage multiple priorities in a fast-paced environment.Ability to work independently while collaborating effectively with business stakeholders, technical teams, and external partners.Curiosity and adaptability to evaluate emerging technologies, AI capabilities, and evolving data engineering best practices.Customer-focused mindset with a commitment to delivering high-quality, business-driven solutions.Education and Experience:Bachelor's degree in Computer Science, Data Engineering, Information Systems, Software Engineering, or a related field required.Minimum of five years of data engineering experience, including at least two years developing and supporting production solutions on the Databricks platform utilizing PySpark, Spark SQL, Delta Lake, and Databricks Workflows.Experience designing, building, and maintaining enterprise data pipelines, data integrations, and lakehouse architectures in a cloud environment.Hands-on experience administering and utilizing Unity Catalog, including metadata management, data lineage, security, and access controls.Experience operating production data platforms, including monitoring, alerting, incident response, troubleshooting, and performance optimization.Experience implementing data quality controls, reconciliation processes, and data governance standards.Experience integrating data from ERP, financial, operational, or other enterprise business systems.Experience working with Microsoft Azure and Git-based development and deployment tools, including Azure DevOps and/or GitHub.Experience leveraging AI-assisted development tools as part of a professional software engineering workflow.Experience supporting AI and machine learning initiatives through data engineering, feature engineering, vector search, retrieval-augmented generation (RAG), model-serving pipelines, or related technologies.Demonstrated success partnering directly with business stakeholders to understand requirements, validate data, and deliver business-focused solutions.Preferred Experience:Databricks Data Engineer certification or equivalent Databricks credentials.Experience supporting JD Edwards, OneStream, Yardi, Entrata, UKG, Medius, or similar enterprise platforms.Experience working with data in the real estate, construction, energy, financial services, or related industries.Experience transitioning, stabilizing, and optimizing data solutions developed by third-party consultants or implementation partners.Experience supporting large language model (LLM), generative AI, or agent-based solution development.Experience designing solutions that combine structured and unstructured data to support AI-enabled business applications.Benefit Highlights:Medical InsuranceDental InsuranceVision InsuranceFlexible Spending Accounts (FSA)Health Savings Accounts (HSA)Vacation timeSick timeHolidays