首页 » Architecture » 正文

The Best Practices and Strategy for Migrating Data Warehouse to the Cloud

| Architecture,Cloud | | 暂无评论 | 1703 views

Analyst: Sting Gao

Cloud-based data warehouses are providing elastic, robust and agile architecture for enterprise data empowerment, but migrating data warehouses to the cloud going to be an undertaking. Enterprises should follow best practices and adopt strategic planning to successfully implement cloud migration.

Key Findings

  • Enterprises have concern about migrating data warehouses to the cloud, they belief that such migration is going to be long, arduous, and costly, also worry about data security and compliance risks, possible business availability and interruption during the migration process, and uncertain due to lack of experience and best practice.
  • The enterprise data warehouse migration to the cloud is often full of twists and turns or not easy to succeed without complete and specific execution strategy and plan.
  • Enterprises have uncertain about the lack of a more comprehensive assessment based on a global perspective to reduce the risk of data warehouse migration to the cloud.

Recommendations

For a successful migrating Data Warehouse from On-Premises to Cloud, tech CEOs must:

  • Follow and adopt best methods and practices to plan cloud migration of enterprise data warehouses, to overcome unknown challenges and uncertainties.
  • Building a complete migration strategy and implementing a migration project in accordance with the established plan, will help landing enterprise data warehouse on the cloud successfully.
  • Avoid and reduce the migration risk by considering the enterprise overall cloud strategy, the data analytic ecosystems, security, compliance, regulations, and laws. The related local and global laws like the Cyber Security Law of China (CSL), Data Security Law of China (DSL) and General Data Protection Regulation (GDPR).

Strategic Planning Assumptions

By 2022, 75% of databases will be deployed in or migrated to the cloud.

According to Gartner, “83% of data migration projects either fail or exceed their budgets and schedules”. More than 80% of data migration projects run overtime and/or over budget. Cost overruns average 30%. Time overruns average 41%.

 

Analysis

Introduction

A robust, agile, and scalable Cloud-based Data Architecture will be a key driver of enterprise success in the coming years. A new report by Gartner predicts that by 2022, 75% of databases will be deployed in or migrated to the cloud. but the cloud migration process can be complicated especially for data warehouse.  For those making the initial foray to the cloud, companies often lack experience and want some best practice advice.

A data warehouse migration is a challenge for any company, there is a lot to consider before and during migration. There are some challenges may face include security, technical skills and cost, may also need to optimize your data architecture, data storage, data model and applications according to actual data application scenarios. Therefore, the company must take a strategic approach to streamline the process.

To avoid any unwelcome surprises and unplanned costs, company need to thoroughly research the challenge, mitigate risk, and plan the migration strategy in advance to ensure as ready as possible. At a high level, the plan should cover the core data warehouse migration process steps and any tasks within them.

This article presents a common practice and the step-by-step approach for migrating a data warehouse to the cloud.

The Best Approach and Practices for Migrating Data Warehouse

The tenet of the best practice of enterprise data warehouse cloud migration includes 6 phases in an orderly manner, they are including complete preparation, comprehensive evaluation, design goals, architecture optimization and design, strategic execution and ensuring continuous optimization. (see Figure 1).

Figure 1. The 6 phases practice for migrating data warehouse

 

 

The 6 steps of the best implementation are described in detail below:

1. Preparation

Cloud readiness check, use Strategic Migration Assessment and Readiness Tool to prepare a scale migration is critical to ensure your project is executed smoothly.

Companies should assess their requirements in terms of analytics, performance, and cloud services. Use cases can vary from data warehousing, analytic sandboxes to testing and development. Choosing the right platform will depend on the service models, availability levels and performance services (this article will consider choosing one of AWS Redshift and Azure Synapse based on the current local cloud in China):

  • Strategy alignment, ensure to follow the enterprise IT cloud strategy if there it is
  • Local cloud vendor vs. Global cloud vendor, ensure the cloud capability can meet business requirement in data analysis, and evaluate future business overseas expansion and global data integration.
  • Local Landing, Enterprises usually choose cloud providers that land on local to avoid cross-border data transmission, while also considering the convenience of data integration, because most of the data sources will be local.

One important that enterprise should also preparation the resource for Roles, Skills and Budget.

2. Accessing

Assessing existing On-Prem Data Warehouse and Cloud Data Warehouse. In this stage it is important to assess your environment and determine the factors that will govern the migration, such as:

  • Critical application data, legacy data, and application interoperability.
  • DB schemas and tables/data migration.
  • Feature gaps and compatibility.
  • Workload discovery and assessment.

3. Goal and Plan

There needs to be a set of objectives or goals that should be defined to indicate success. Targets can then be set to achieve these goals and people given responsibility for reaching them. Types of goal and metric include:

  • Migration scope by align with business goal.
  • Goal of capability, productively, performance KPIs.
  • Availability and service levels.
  • Schedule and timeline plan.

4. Solution and Architecture

Before data migration commences, team will need to convert the current data warehouse to new cloud warehouse include the big data platform process capability start from data ETL, data storage, data mart and analytics. (See Figure 2 for the sample solution architecture design)

The detail solutions should also include:

  • Architecture optimization according to business: As-Is and To-Be architecture.
  • Solution design including security and compliance policy.
  • Tools for copy/transfer/file export and schema migration.
  • Cloud monitoring, backup, disaster recovery solutions.

Figure 2. Cloud-based modern Data Warehouse

5. Strategic Execution

Enterprise needs to consider the type and source of data to determine the migration method to use and develop a detailed execution strategy and execute according to strategy.

6. Optimize

Enterprise needs to build an operating model & a service management framework and keep the continues optimization including cloud configuration, security, cost and performance.

 

The Strategic Execution for Migrating Data Warehouse

In response to the strategic execution in the best practices mentioned above, we have given more specific execution steps and the key points of execution in each step for help company avoid common mistakes.

Figure 3. The 6 specific strategic exaction steps for cloud data warehouse migration

The migrate execution start with the infrastructure including cloud configuration and network plan. See the sample of infrastructure design through build the VPN network between On-Premises and Cloud for secure the data transformation and plan the network segmentation in cloud according to modern data warehouse and data analytic application. (See Figure 4).

Figure 4. Architecture for On-Premises Data Warehouse to Cloud Migration

 

De-risking for Migrating Data Warehouse

There are 4 possible risks and pitfalls that companies need to pay attention to and avoid when implementing cloud migration:

1. Cloud based infrastructure and operations teams are ready

The migration team should ensure your cloud-based infrastructure and operations capability are ready.

2. Including the data surrounding ecosystems in migration scope

The migration needs to include data surrounding ecosystem as needed, such as data ingestion tools, BI or analysis tools, and other upstream and downstream systems.

3. Introducing a data virtualization layer for business service availability

The introduction of a data virtualization layer helps to achieve business service availability during the migration process.

4. Optimization and security re-check

  • Re-check security and cost, like security groups, encryption, reserved resource for cost optimization.
  • Re-check local regulations and laws compliance: CSL, DSL, PIPL, GDPR, data privacy and cross-board transformation etc.

Conclusion

By adopt the best practices to well preparation and define the specific strategy and execution to avoid common mistakes and risk for cloud data warehouse migration, through comprehensive risk assessment and cloud-based solution architecture design optimization, successful cloud data warehouse migration can bring leap-forward improvements to enterprise data applications and bring convenience to operation and maintenance in service availability and cost-efficiency.

 

Reference

  1. https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/plan/data-warehouse-migration
  2. https://www.dataversity.net/migrating-big-data-to-the-cloud-challenges-and-benefits/
  3. https://datatonic.com/insights/data-warehouse-migration-google-cloud/
  4. https://www.gartner.com/en/documents/3671418
  5. https://aws.amazon.com/cn/redshift/data-warehouse-migration/