2 days ago Be among the first 25 applicants
Direct message the job poster from Xoriant
We are seeking a highly skilled Databricks Architect with deep expertise in designing and building real-time, high-throughput data pipelines in Databricks across Azure, AWS, and GCP. The ideal candidate will have strong hands-on experience in Scala, Apache Spark, and deep familiarity with modern data lake architectures. Experience working with data requirements from financial institutions—especially regarding security, governance, and performance—is highly preferred.
Key Responsibilities
* Design and implement real-time streaming and batch data pipelines using Databricks + Spark (Scala preferred)
* Architect and optimize data lakehouse platforms on Azure (primary), AWS, and GCP
* Collaborate with data engineering, analytics, DevOps, and governance teams to ensure secure and performant data architectures
* Lead end-to-end architecture for ingestion, transformation, enrichment, and publishing of large volumes of structured and unstructured data
* Design for low-latency processing, auto-scaling, and fault tolerance for high-throughput pipelines
* Drive infrastructure-as-code adoption (Terraform, CI/CD) to automate deployments and platform provisioning
* Apply best practices for data quality, lineage, governance (Unity Catalog / Purview), and security (RBAC, encryption, private networking)
* Interface with financial clients to understand regulatory constraints, auditability, and secure access needs
* Mentor and guide engineering teams on best practices for Databricks, Spark, and real-time data solutions
Required Skills and Experience
Core Technical Expertise:
* Databricks (Azure, AWS, GCP) – deep understanding of cluster management, workspace security, Delta Lake, Unity Catalog
* Scala (Strong proficiency) – for high-performance data processing jobs
* Apache Spark (Structured Streaming, Core APIs)
* Real-time Data Pipelines – using Kafka, Delta Live Tables, or Apache Flink
* AWS: S3, Glue, Kinesis, Redshift, IAM, CloudWatch
* GCP: GCS, BigQuery, Dataflow, Pub/Sub
* Delta Lake architecture (ACID transactions, schema evolution, time travel)
* Data modeling (Star, Snowflake, Data Vault)
* Orchestration with Azure Data Factory, Airflow, dbt, or Dagster
* Proficiency in SQL, Python for scripting and notebooks
Security and Governance:
* Role-based access control (RBAC), token authentication
* Data encryption (in transit and at rest), network security (Private Link, VNet/Security Groups)
* Unity Catalog, Azure Purview, or Alation for data governance
DevOps & Infra:
* Infrastructure-as-Code (IaC) using Terraform, Pulumi, or ARM templates
* Monitoring with Datadog, Azure Monitor, or Prometheus/Grafana
Preferred Qualifications
* Experience working with financial institutions or in regulated environments
* Strong understanding of data compliance (GDPR, CCPA, SOX, PCI, etc.)
* Familiarity with machine learning pipelines on Databricks
* Knowledge of event-driven architectures and microservices
Soft Skills
* Excellent communication and stakeholder management skills
* Leadership experience in mentoring and guiding engineering teams
* Ability to translate complex requirements into scalable designs
* Strong documentation and solutioning mindset
Education
Bachelor's or Master’s degree in Computer Science, Engineering, or related field
Certifications (Preferred)
* Azure Solutions Architect / AWS Certified Solutions Architect / GCP Cloud Architect
* Terraform Associate / HashiCorp Certified
Seniority level
* Seniority level
Mid-Senior level
Employment type
* Employment type
Full-time
Job function
* Job function
Information Technology and Engineering
* Industries
IT Services and IT Consulting, Financial Services, and Banking
Referrals increase your chances of interviewing at Xoriant by 2x
Get notified about new Architect jobs in Greater Boston.
Greater Boston $86,100.00-$144,700.00 23 hours ago
Mansfield, MA $100,000.00-$110,000.00 1 day ago
Beverly, MA $70,000.00-$90,000.00 3 days ago
Boston, MA $55,552.00-$65,000.00 1 month ago
Architect Co-op Fall 2025 (Foxborough, MA)
Registered Architect and Designer-High End Residential
Boston, MA $75,000.00-$100,000.00 1 month ago
Cambridge, MA $177,000.00-$263,000.00 1 week ago
Solutions Architect, Retail, Google Cloud
Cambridge, MA $147,000.00-$216,000.00 2 days ago
Cambridge, MA $147,000.00-$216,000.00 1 week ago
Winchester, MA $70,000.00-$120,000.00 1 day ago
We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.
#J-18808-Ljbffr