Overview

ENGINYRING is proud to introduce our enhanced Automated HDD Migration Tool 2.0 – a powerful shell script that simplifies the process of migrating hard disk data over a network. If you need to transfer an entire hard drive's contents from one system to another, our tool uses SSH and NetCat to perform a raw, byte-for‑byte disk clone with numerous options for safety, speed, and security. In this tutorial, we will explain what the tool does, how it works, and guide you through using it safely, even if you've never heard of it before.

The tool is designed to run on rescue systems, whether Debian‑based or RHEL‑based, and leverages standard utilities such as lsblk, dd, and blockdev. It also manages its dependencies by installing NetCat (using netcat‑openbsd on Debian‑based systems and nmap‑ncat on RHEL‑based systems) and uses SSH (with optional sshpass support) for remote command execution. With interactive prompts throughout the process, this tool ensures that you select the correct source and destination disks while performing disk size validation for a safe migration.

Important: Since this tool performs low‑level disk cloning that overwrites data irreversibly, it should only be used on non‑production systems and after making complete backups of your data.

Section 1: Introduction to Disk Cloning and Migration

Disk cloning refers to copying every bit of data from one hard disk to another, creating an exact replica. This technique is vital for data backup, system rescue, and migration. Whether you are upgrading hardware, recovering from a system failure, or setting up a new server environment, disk cloning ensures that your entire system—operating system, applications, settings, and user data—is transferred accurately.

In our increasingly interconnected world, managing data efficiently is essential. ENGINYRING's enhanced tool automates the cloning process, reducing manual errors and simplifying a complex task. This tutorial will walk you through each step of the process, ensuring that even if you are unfamiliar with low-level disk operations, you can successfully migrate your HDD using SSH and NetCat.

Warning: This tool clones disks at a raw, byte-for‑byte level. Any data on the destination disk will be permanently overwritten. Always use the tool on non‑production systems and ensure that full backups are in place before proceeding.

Section 2: Key Features of the Enhanced HDD Migration Tool

Our Automated HDD Migration Tool 2.0 offers several enhanced features designed to make disk migration both safe, efficient, and flexible:

Core Functionality

  • Automated Dependency Management: The script automatically checks for required utilities like NetCat and installs them if missing.
  • Disk Discovery and Validation: It uses lsblk to list disks and blockdev to check disk sizes, ensuring you know exactly what is available on your system.
  • Interactive Prompts: The tool guides you through the process by asking for the source disk, remote host details, and the destination disk, reducing the risk of human error.
  • Secure Remote Execution: Leveraging SSH (and optionally sshpass), the tool securely connects to the destination system to verify configurations before data transfer.
  • Data Transfer Using dd and NetCat: The tool uses dd to read data from the source disk and pipes it through NetCat to write directly to the destination disk, ensuring a complete clone.
  • Real-Time Progress Reporting: You receive continuous updates during the transfer, along with detailed logging for troubleshooting.

Enhanced Safety Features

  • Metadata Backup: Automatically creates backups of MBR/partition tables before overwriting.
  • Mounted Filesystem Detection: Warns and offers safety options when disks are mounted.
  • LVM Snapshot Support: For safely migrating live systems without downtime.
  • Transfer Validation: Optional checksum verification of transferred data.
  • Structured Logging: Comprehensive timestamped logs with different severity levels.
  • Resume Capability: Can continue interrupted transfers without starting over.

Performance Enhancements

  • Configurable Block Size: Tune performance based on network and disk characteristics.
  • Compression: Optional on-the-fly compression to speed up transfers over slow links.
  • Bandwidth Throttling: Optional bandwidth limiting to prevent network saturation.
  • Progress Monitoring: Real-time transfer status with speed and time calculations.

Security Features

  • Transfer Encryption: Optional AES-256 encryption for sensitive data.
  • SSH Security Options: Configurable timeouts and connection settings.
  • Secure Password Handling: Better management of SSH credentials.

Usability Improvements

  • Command-Line Interface: Scriptable with full command-line options.
  • Configuration Files: Store common settings in config files.
  • Dry Run Mode: Preview operations without making changes.
  • Notification System: Optional email alerts on completion or failure.
  • Non-Interactive Mode: Run without prompts for scripting and automation.

Section 3: Requirements

Before you begin, ensure your system meets the following prerequisites:

  • You must run the script as root.
  • Reliable network connectivity is required, and the transfer port must be open on both the source and destination systems.
  • Essential utilities: lsblk, dd, and blockdev must be installed.
  • A package manager is needed – use apt-get on Debian‑based systems or yum/dnf on RHEL‑based systems.
  • NetCat is necessary; the script will install netcat‑openbsd on Debian‑based systems or nmap‑ncat on RHEL‑based systems if missing.
  • SSH must be available on both the source and destination systems, with optional sshpass support for password-based, non‑interactive authentication.
  • Optional: pv, gzip, openssl, mail (for advanced features).

Verifying these requirements is crucial to ensure a smooth and successful disk migration.

Section 4: Installation Instructions

Downloading and preparing the tool is straightforward. You can use either wget or curl to fetch the script directly from our GitHub repository.

Using wget

Open your terminal and run:

wget -O migration.sh https://raw.githubusercontent.com/ENGINYRING/Automated-HDD-Migration-Tool/refs/heads/main/migration.sh

Then, make the script executable:

chmod +x migration.sh

Using curl

Alternatively, run:

curl -o migration.sh https://raw.githubusercontent.com/ENGINYRING/Automated-HDD-Migration-Tool/refs/heads/main/migration.sh

And then set the proper permissions:

chmod +x migration.sh

This downloads the script directly, so you do not need to clone the entire repository.

Section 5: Basic Usage Guide

Our tool provides both an interactive mode for ease of use and a command-line interface for automation. Here's how to use it:

Interactive Mode

For beginners or one-time operations, simply run the script with no arguments:

sudo ./migration.sh

The script will guide you through each step of the process:

  1. Local Preparation: The script checks for required tools and displays available disks.
  2. Source Disk Selection: You'll be prompted to select the source disk.
  3. Remote Connection Setup: Enter the destination host details and SSH credentials.
  4. Destination Disk Selection: Choose the target disk on the remote system.
  5. Validation and Confirmation: The script validates disk sizes and asks for confirmation before proceeding.
  6. Data Transfer: The cloning process begins with real-time progress updates.
  7. Completion: The script notifies you when the migration is complete.

Command-Line Interface

For scripting or automation, you can specify all parameters on the command line:

sudo ./migration.sh -s sda -d sdb -H remote.host -u admin

This performs the same operation as interactive mode but without requiring user input during the process.

Section 6: Advanced Features and Usage Examples

The enhanced tool includes several advanced features for specific use cases. Here are examples of how to use them:

With Data Validation

Verify the integrity of the transferred data using checksums:

sudo ./migration.sh -s sda -d sdb -H remote.host -u admin --validate

With Compression

Enable compression to speed up transfers over slow networks:

sudo ./migration.sh -s sda -d sdb -H remote.host -u admin --compress

For Live Systems (using LVM)

Safely migrate a running system using LVM snapshots:

sudo ./migration.sh -s sda -d sdb -H remote.host -u admin --snapshot

With Transfer Encryption

Encrypt data during transfer for sensitive information:

sudo ./migration.sh -s sda -d sdb -H remote.host -u admin --encrypt

Limiting Bandwidth

Prevent the transfer from saturating your network:

sudo ./migration.sh -s sda -d sdb -H remote.host -u admin -l 10M

Resuming an Interrupted Transfer

Continue a transfer that was previously interrupted:

sudo ./migration.sh -s sda -d sdb -H remote.host -u admin --continue

Note: Always run this command on the same source server where the original transfer was initiated.

Complete Options List

Usage: ./migration.sh [OPTIONS]

Disk Migration Tool - Safely clone disks over SSH

OPTIONS:
  -h, --help                Show this help message
  -c, --config FILE         Use specific config file
  -s, --source DISK         Source disk (e.g., sda)
  -d, --dest DISK           Destination disk
  -H, --host HOST           Destination host
  -u, --user USER           SSH username
  -p, --port PORT           SSH port (default: 22)
  -t, --transfer-port PORT  Data transfer port (default: 9000)
  -b, --block-size SIZE     Block size for dd (default: 64K)
  -l, --limit RATE          Bandwidth limit (e.g., 10M)
  -v, --verbose             Enable verbose output
  --compress                Enable compression during transfer
  --validate                Verify transfer with checksums
  --encrypt                 Encrypt data during transfer
  --dry-run                Show commands without executing
  --snapshot                Use LVM snapshot for live migration
  --continue                Continue interrupted transfer
  --offset BYTES            Starting offset for continued transfer
  --notify EMAIL            Email to notify on completion

Section 7: Using Configuration Files

For recurring transfers or to standardize settings across multiple operations, you can use configuration files:

# Use default configuration
sudo ./migration.sh

# Use specific configuration file
sudo ./migration.sh -c /path/to/my-config.conf

A default configuration file is created in $HOME/.config/disk-migration/config.conf the first time you run the script. You can customize this file with your preferred settings.

Example Configuration File

# Disk Migration Tool Configuration

# Transfer Settings
BLOCK_SIZE="64K"      # Block size for dd
DEFAULT_PORT=9000     # Default netcat port
COMPRESSION=0         # Enable compression (0=off, 1=on)
VALIDATE=0            # Validate transfer with checksums
ENCRYPT_TRANSFER=0    # Encrypt data during transfer
#BANDWIDTH_LIMIT="10M" # Limit bandwidth (e.g., 10M, 1G)

# SSH Settings
dest_host="server.example.com"  # Destination host
dest_user="admin"               # SSH username
ssh_port="22"                   # SSH port

# LVM Settings
USE_LVM_SNAPSHOT=0    # Use LVM snapshot for live migration
SNAPSHOT_NAME="disk_migration_snapshot"

# Notification Settings
NOTIFICATION_EMAIL="admin@example.com"  # Email for notifications

Section 8: Resuming Interrupted Transfers

One of the most powerful features of our enhanced tool is the ability to resume interrupted transfers, which is especially valuable when migrating large disks over unreliable networks.

How Resume Works

During transfer, the script periodically saves transfer progress to a state file. This file tracks source/destination disks, hostname, and exact byte offset. If a transfer is interrupted, you can resume it with:

sudo ./migration.sh --continue

The script will load the saved state and resume from where it left off, saving potentially hours or days of retransfer time.

For example, if you're transferring a 4TB disk and after 2TB your network connection drops, simply run the same command with --continue added, and the transfer resumes from approximately the 2TB mark.

Manual Offset

If you need to specify a particular starting point manually:

sudo ./migration.sh -s sda -d sdb -H remote.host -u admin --offset 1073741824

This starts the transfer from the 1GB mark. This can be useful if you know exactly where a transfer failed or if you want to skip certain portions of the disk.

Section 9: Best Practices and Safety Tips

To ensure a safe and successful HDD migration, follow these best practices:

  • Back Up Your Data: Always create a full backup of any critical data before running the tool. Disk cloning will overwrite all data on the destination disk.
  • Test in a Controlled Environment: Use non‑production systems or virtual machines to test the tool before deploying it in a live environment.
  • Use Dry Run Mode: When unsure, use the --dry-run flag to preview what commands would be executed without making any changes.
  • Verify Disk Identifiers: Double-check the disk names and sizes provided by lsblk and blockdev to ensure you are cloning the correct disks.
  • Use LVM Snapshots for Live Systems: When migrating disks with mounted partitions, use the --snapshot option to create a consistent point-in-time copy.
  • Enable Validation for Critical Data: Use the --validate option to verify data integrity for important transfers.
  • Monitor the Process: Keep an eye on progress reports and log files generated by the script to quickly detect and address any errors.

These precautions are critical to avoid accidental data loss and to ensure that the migration process proceeds smoothly.

Section 10: Advanced Use Cases

Scheduled Migrations

Use cron to schedule regular migrations:

# Example cron entry for daily migration at 2 AM
0 2 * * * /path/to/migration.sh -c /path/to/config.conf > /var/log/scheduled-migration.log 2>&1

Creating Disk Images

To create a disk image file instead of directly cloning to another disk:

# On the destination machine, create a file of appropriate size
dd if=/dev/zero of=/path/to/disk.img bs=1M count=<size_in_MB>

# Set up a loop device on the destination
losetup /dev/loop0 /path/to/disk.img

# Then use the script with loop0 as destination
./migration.sh -s sda -d loop0 -H destination-host -u admin

Pre/Post Migration Scripts

For complex migrations, create wrapper scripts that run preparation or finalization tasks:

#!/bin/bash
# Pre-migration tasks
umount /dev/sda1  # Unmount filesystems
systemctl stop some-service  # Stop services

# Run migration
./migration.sh -s sda -d sdb -H remote.host -u admin

# Post-migration tasks
ssh user@remote.host "mount /dev/sdb1 /mnt && chroot /mnt grub-install /dev/sdb"

Section 11: Conclusion and Further Resources

The Enhanced Automated HDD Migration Tool is a versatile and efficient solution for performing raw, byte-for‑byte disk cloning over a network. By automating dependency management, disk discovery, and data transfer through the use of SSH and NetCat, this tool dramatically simplifies a complex process.

In this tutorial, we introduced you to the tool, explained its key features, and provided a detailed, step-by-step guide on how to use it safely. We emphasized the importance of backups and testing in controlled environments due to the irreversible nature of disk cloning.

We encourage you to review the source code, experiment in a safe setting, and contribute to ongoing improvements. For further learning and technical insights, consider exploring the following resources:

  • GNU Bash Manual – Detailed documentation on shell scripting.
  • GitHub Repository – Access the source code, report issues, and contribute to further development.

By following this tutorial and adhering to best practices, you can confidently migrate HDD data over a network, ensuring a reliable and secure disk cloning process. Remember, proper preparation and caution are key to a successful migration.

At ENGINYRING, we are committed to Engineering the interconnected world by providing innovative, secure, and scalable hosting solutions. We hope this guide has equipped you with the knowledge to use our enhanced tool effectively.

For additional information on advanced disk cloning, system rescue, and backup strategies, visit our Web Hosting Services page or contact us for expert support.

Additional Resources

  • GNU Bash Manual – A must-read for understanding shell scripting basics and advanced techniques.
  • GitHub Repository – Explore the source code, contribute, and stay updated on the tool's development.

ENGINYRING – Engineering the interconnected world