AWS DataSync Process- Migrating Data from GCP to AWS

Step 1: Generate GCP HMAC key

The DataSync agent uses the HMAC credential for authentication with the Google Cloud Platform and for the purpose of handling objects within the Cloud Storage bucket

Step 2: Setup S3

Setup an Amazon S3 destination bucket for the DataSync transfer

Step 3: Setup S3 access

In order for AWS DataSync to transfer data to the destination S3 bucket, it is necessary for DataSync to have access for S3 bucket

Step 4: Setup Network

Create the VPC as per the network requirements. This will help to establish the connection between an agent and the DataSync privately

Step 5: Deploy Agent

Launch Amazon EC2 instance using the latest DataSync Amazon Machine Image (AMI). The agent acts as a bridge between your GCP environment and AWS

Step 6: Configurations

Configure Datasync agent to establish a connection with your GCP storage and AWS services. 1. Create Amazon S3 location 2. Create GCP storage location 3. Create DataSync task

Step 7: Run Datasync

Start the DataSync task, monitor & verify the progress

As businesses advance and their needs evolve, the smooth and efficient transfer of data between different cloud providers becomes increasingly critical. Migrating data from a Google Cloud Platform (GCP) bucket to Amazon S3 (Simple Storage Service) involves a set of crucial steps that organizations must adhere to in order to ensure a seamless and successful data transfer. For a detailed introduction to the basic steps involved in this process, please refer to the article titled Migrate Data from GCP Bucket to AWS.

In this article, we will explore the process of migrating data from Google Cloud Platform (GCP) to Amazon Web Services (AWS) using AWS DataSync. AWS DataSync is a fully managed service that simplifies and accelerates data transfer between on-premises storage systems and cloud services.

Prerequisites:

  • An active AWS account with access to AWS DataSync.
  • AWS Command Line Interface (CLI).
  • A Google Cloud Storage (GCS) bucket containing the data you wish to migrate.
  • Network connectivity between the GCP environment and your AWS environment.

Step 1: Generate GCP HMAC Key

The DataSync agent uses the HMAC credential for authentication with Google Cloud Platform and for handling objects within the Cloud Storage bucket.

Follow the steps at Manage HMAC keys for service accounts to generate an access ID and a secret. Ensure the service account has the “Storage Object Viewer” role to grant the necessary permissions.

Step 2: Set Up Amazon S3 Destination Bucket

Set up an Amazon S3 destination bucket for the DataSync transfer. After successfully creating the destination bucket, navigate to the bucket’s Properties tab to retrieve the Amazon Resource Name (ARN) associated with the bucket.

Step 3: Set Up Access for S3 Bucket

For AWS DataSync to transfer data to the destination S3 bucket, it needs access to the bucket. This involves DataSync assuming an IAM role with the required permissions and trust relationship. Create a new role and associate a policy that grants DataSync the ability to read from and write to your Amazon S3 bucket.

Step 4: Set Up Network

Create the VPC endpoints as per the network requirements. These VPC endpoints let you privately connect the DataSync agent and AWS.

Step 5: Deploy DataSync Agent

To begin the migration process, set up an AWS DataSync agent. The agent acts as a bridge between your GCP environment and AWS. Launch an Amazon EC2 instance using the latest DataSync Amazon Machine Image (AMI) into the subnet from the previous step with the security group for agents. Follow the AWS Documentation for the detailed process to deploy the agent on EC2. Once the Amazon EC2 instance is running, create a DataSync agent component using the VPC endpoint. Finally, activate your agent to associate it with your account.

Step 6: Configure DataSync Agent

Once the DataSync agent is set up, configure it to establish a connection with your GCP storage and AWS services.

1.Create an Amazon S3 Location:

  • In the AWS Management Console, navigate to the DataSync service.
  • Create a new DataSync location for Amazon S3 and use the IAM role created in Step 3.

2.Create a GCP Storage Location:

  • Similarly, create a GCP storage location in the DataSync console.
  • Provide the necessary credentials and access details for your GCP bucket.

3.Create a DataSync Task:

  • Set up a DataSync task to define the migration process.
  • Specify the source (GCP storage location) and the destination (AWS S3 storage location) for the task.
  • Configure additional options like file filters, scheduling, and error handling according to your requirements.

Step 7: Initiate the DataSync Migration

Once the DataSync agent and task are configured, you can initiate the migration process.

1.Start the DataSync Task:

  • Go to the DataSync console and select the desired task.
  • Click on the “Start” button to begin the migration process.

2.Monitor the Migration Progress:

  • AWS DataSync provides real-time progress updates during the migration.
  • Monitor the status, track the transferred data, and view any encountered errors in the DataSync console.

3.Verify the Migrated Data:

  • Once the migration is completed, verify the data in your AWS storage destination to ensure a successful transfer.
  • Perform any necessary validation or testing to confirm the integrity of the migrated data.

By following these steps, you can ensure a smooth and secure migration of your data from Google Cloud Platform to Amazon S3 using AWS DataSync.

Recommended Reads