RetailGrid is an AI-powered retail analytics platform (grid-style UI) running on Google Cloud.
Current stack includes:
Cloud Run (microservices)
Cloud SQL (PostgreSQL)
Cloud Storage (GCS)
Pub/Sub
Python services for forecasting, elasticity, optimization
React + AG Grid frontend
We are now implementing ClickHouse as a dedicated analytics layer to offload heavy aggregation queries from Cloud SQL and support large-scale retail datasets (millions to billions of rows).
.
Objective
Design and implement a production-grade ClickHouse analytics layer integrated into our existing GCP-based microservice architecture.
This is an architectural + hands-on implementation role.
Scope of Work
1. Architecture & Design
Design ClickHouse deployment (ClickHouse Cloud preferred; self-managed as alternative).
Define data modeling strategy:
Fact tables (e.g., sales_transactions)
Dimension tables (product_dim, store_dim, calendar_dim)
Multi-tenant design (workspace isolation)
Partitioning, sorting keys, TTL, compression strategy.
Versioning / dedup strategy (ReplacingMergeTree / CollapsingMergeTree, etc.).
2. Data Ingestion
Design and implement:
Initial historical backfill from Cloud SQL + GCS
Incremental ingestion pipeline (Pub/Sub → Cloud Run → ClickHouse)
Ensure:
Idempotency
Late-arriving data handling
Schema evolution support
Monitoring of ingestion lag
3. Query Layer Integration
Implement backend integration (Python / Go services).
Migrate selected heavy analytics queries to ClickHouse.
Optimize:
Aggregations
Window functions
SKU/store time-series queries
Ensure performance benchmarking (before vs after).
4. Performance & Cost Optimization
Benchmark:
Query latency
Cost per query
Optimize storage and query efficiency.
Provide scaling strategy (horizontal / vertical).
5. Documentation
Data model documentation
Ingestion architecture diagram
Operational guide (monitoring, backups, scaling)
Expected Outcomes
ClickHouse deployed and production-ready.
At least 3–5 major analytics queries fully migrated.
5x improvement in heavy aggregation query latency.
Clear ingestion and monitoring pipeline.
Documented scalable architecture.
Required Experience
Deep expertise in ClickHouse (production systems)
Strong data modeling for analytical workloads
Experience with:
GCP (Cloud Run, Pub/Sub, Cloud SQL, GCS)
Event-driven ingestion pipelines
Large-scale time-series / retail / e-commerce datasets
Strong Python (required), Go (nice to have)
Experience with multi-tenant SaaS data architectures
Nice to Have
Experience migrating from Postgres → ClickHouse
Retail analytics or pricing systems background
Experience with columnar storage optimization
Engagement Details
Type: Contract (initial 4–8 weeks, potential long-term)
Start: ASAP
Timezone: Europe preferred (Finland-based team)
Deliverable-based milestones possible
Contract duration of 1 to 3 months. with 30 hours per week.
Mandatory skills: Google Cloud Platform, Big Data, ClickHouse, Data Engineering