Snowflake Cloud Migration
Migrating an enterprise on-prem data warehouse to Snowflake and AWS
Overview
The organization's analytics ran on legacy on-premises warehousing: SQL Server pipelines, Teradata sources, and manually managed scripts. As Lead BI Developer, I'm driving the modernization of that platform, architecting cloud-native solutions on Snowflake and AWS and establishing the migration patterns the rest of the organization follows.
The Challenge
The platform had to keep serving enterprise analytics while being rebuilt underneath. Refresh cycles were daily, transformations lived inside Tableau extracts straining production servers, critical HR data sat in Teradata, and the team managed scripts by hand with no version control, so every change was risky and slow.
The Solution
Architected hybrid pipelines in Python and SQL Server as the foundation, then built cloud-native solutions on Snowflake and AWS with repeatable migration patterns so each workload move got cheaper than the last. Re-engineered ETL/ELT workflows to cut ingestion from daily to hourly, pushed transformations out of Tableau extracts into backend stored procedures, and moved the team from manual script management onto Git with a structured CI/CD workflow.
System Architecture
Ingest
Python and SQL Server pipelines integrating processing across hybrid on-prem and cloud environments, including automated Teradata-to-warehouse migration of critical HR data
Transform
ETL/ELT workflows re-engineered for hourly refresh, with heavy transformations moved from Tableau extracts into backend stored procedures and temporary tables
Serve & Deploy
Cloud-native warehousing on Snowflake and AWS serving enterprise analytics, shipped through Git-based collaborative workflows and CI/CD
Key Features
Repeatable Migration Patterns
Designed scalable cloud architectures and documented migration patterns so each on-prem workload moves to Snowflake the same proven way.
Daily-to-Hourly Ingestion
Re-engineered ETL/ELT workflows to cut refresh latency from daily to hourly, enabling near real-time decisions for business stakeholders.
Transformation Pushdown
Moved heavy transformations out of Tableau extracts into backend stored procedures, significantly reducing load on production servers.
Team Engineering Standards
Introduced Git version control and CI/CD to a team previously managing scripts by hand, and mentored junior developers on SQL optimization and data modeling.
Results & Impact
Ingestion latency cut by re-engineering ETL/ELT workflows
Solution Delivery metrics automated with SQL, replacing manual Excel processes
Tech Stack Deep Dive
Cloud & Warehouse
Pipeline & Tooling
Lessons Learned
Migration patterns matter more than individual migrations; the second workload should be cheaper than the first.
Moving transformations off the BI layer and into the warehouse pays for itself in server load alone.
Version control adoption is a culture change, not a tooling change; mentoring is part of the rollout.