Join a Rapidly Growing AI Consumer Intelligence Platform Delivering Insights for the World's Leading BrandsCompany BackgroundOur client's AI-powered consumer intelligence platform synthesizes massive-scale data (billions of Google searches, social conversations, product reviews, and videos) to deliver actionable insights for the world’s leading brands, such as Coca-Cola, Dove, Colgate, Clorox, and Vaseline, among others.With a newly appointed CTO building the engineering team, this is your opportunity to shape how production AI is built at a high-growth company. Your impact will be immediate and visible. No legacy code politics, no entrenched hierarchies, and no inherited technical debt.The OpportunityOur client is seeking a Senior Data Architect to lead the modernization of its data platform. This is a high-impact role where you will redesign a Databricks-based data architecture, establishing patterns that will scale with the company's growth.You will work directly with the CTO and collaborate with our data engineering team to transform a functional but organically grown platform into a production-grade, automated, and maintainable global system. This role requires someone who has successfully completed similar modernization projects and can mentor junior engineers while implementing best practices.Success Criteria• Modular, reusable Databricks notebooks with configuration-driven pipelines replacing hardcoded customer logic• Canonical data model with clear schema contracts enabling easy onboarding of new customers, segments, and data providers• End-to-end automation from data ingestion through to dashboard refresh with minimal human intervention• Production-grade error handling with comprehensive logging, alerting, and recovery mechanisms• Clear environment separation (dev/staging/prod) with CI/CD pipelines for notebook deployment• Improved data science workflows making it easier for ML engineers to build and deploy models• Documented architecture standards and patterns adopted by the broader team Key ResponsibilitiesArchitecture & Design• Redesign and implement a Medallion Architecture (Bronze/Silver/Gold) with clear data contracts at each layer• Create a canonical data model that accommodates multi-tenant, multi-segment data (product line + country combinations)• Establish schema evolution and deprecation strategies using Unity Catalog• Design integration patterns for multiple data providers (social media, e-commerce, searchdata)Implementation & Migration• Refactor existing Databricks notebooks into modular, parameterized, reusable components• Implement comprehensive error handling, logging, and alerting frameworks• Migrate legacy tables to new canonical schemas with data validation• Set up CI/CD pipelines for Databricks using GitHub integrationAutomation & Operations• Design orchestration workflows using Databricks Workflows or Azure Data Factory• Implement monitoring dashboards for pipeline health and data quality• Create runbooks and documentation for operational proceduresLeadership & Mentorship• Mentor and upskill data engineers on Databricks best practices and modern data architecture patterns• Conduct code reviews and establish coding standards for the data engineering team• Lead architectural discussions and drive consensus on technical decisions• Collaborate with data scientists to optimize ML workflows and model deploymentRequired QualificationsExperience• 8-12+ years of experience in data engineering, data architecture, or related roles• 4+ years of hands-on experience with Databricks (notebooks, Delta Lake, Unity Catalog,Workflows)• 3+ years designing and implementing data architectures for multi-tenant SaaS platforms• 2+ successful data platform modernization or migration projects (legacy to modernarchitecture)• 2+ years of experience mentoring or leading data engineersTechnical Skills• Azure Cloud Platform: Deep expertise with Azure Data Services (Storage, Data Factory,Synapse, Key Vault, Monitor)• Databricks: Advanced proficiency with PySpark, Spark SQL, Delta Lake, Unity Catalog,DLT (Delta Live Tables),• Databases: Strong SQL skills with SQL Server; schema design and optimization• Python: Production-quality Python for data engineering (error handling, logging, testing)• CI/CD: Experience with GitHub Actions, Azure DevOps, or similar for Databricksdeployment• BI Tools: Familiarity with Power BI and/or Apache Superset data modeling requirements• Data Modeling: Expertise in dimensional modeling, Medallion architecture, and schemaevolution strategiesSoft Skills• Strong communication skills; ability to explain technical concepts to non-technical stakeholders• Proactive problem-solver who can work independently while keeping leadership informed• Patient mentor who enjoys helping junior engineers grow Nice to Have• Experience with LLM/GenAI integration in data pipelines (OpenAI, Azure OpenAI, Gemini)• Background in consumer insights, CPG, or market research data• Experience with social media data APIs (TikTok, Instagram, Facebook) or e-commercedata (Amazon, Walmart)• Databricks certifications (Data Engineer Associate/Professional)• Experience with NLP pipelines and text analytics at scale• PostgreSQL