Category: Blog

Your blog category

  • Moving 1 TB Between Two Cloud Drives: What Worked, What Didn’t, and Why

    Moving 1 TB Between Two Cloud Drives: What Worked, What Didn’t, and Why

    Over a weekend I set out to merge two personal cloud drive accounts into one. On paper it sounds simple: copy everything from the old account into the new one, skip what’s already there, and close the old account. In practice it turned into a small lesson in how the internet actually moves data. Here is what I tried, what I learned, and a side-by-side comparison of the three approaches.

    The starting point

    I had two accounts with the same cloud storage provider: an older one with about 685 GB and a newer one with about 820 GB. Over the years both had collected photos, documents, courses, books and family and friends and travel videos, and a lot of it existed in both places, sometimes under a different name or in a different folder.

    The first useful step was not copying anything, but comparing. I used a free, open-source command-line tool for cloud storage to list both accounts and compare every file by its content fingerprint (a hash that the cloud provider already stores for each file). Comparing by content instead of by name matters: it found about 138 GB of duplicates that only needed to be skipped, and narrowed the real job down to roughly 160–180 GB that was missing from the new account.

    Option 1 – Using my laptop as the middle point

    The obvious approach: run the copy tool on my laptop, which downloads each file from Cloud Drive 1 and immediately uploads it to Cloud Drive 2. Nothing is stored on the laptop’s disk; the files just stream through.

    It worked, but painfully slowly. Overnight it averaged about 227 KB/s, roughly 0.8 GB per hour. After about eleven hours only around 3,600 files had been copied. At that pace the remaining data would have taken eight to nine days of the laptop running day and night. To make it worse, my internet connection dropped for about an hour around midnight, which produced several hundred failed files that had to be retried.

    The key realisation: every single file was crossing my home connection twice – once down and once up – and the slow direction set the pace.

    Option 2 – Download everything to an external hard disk, then upload

    My next idea was to buy a large external disk (around €100), download everything to it, and upload from there. Downloads at home are fast, so the first half would be quick, and as a bonus I would end up with an offline backup.

    The problem is the second half. Uploading from a disk still goes through the same home connection, and home connections are asymmetrical: upload is usually several times slower than download. So this option doesn’t remove the bottleneck, it just moves it to a different day.

    Why is home upload so slow? A note on asymmetric connections

    This was the part I found most interesting. From what I read, it isn’t that the cable to your home contains separate wires reserved for download and for upload. Whether it’s a copper phone line, a coaxial TV cable or a fiber, the provider divides the cable’s available range of frequencies into channels, and decides how many of those channels are used for each direction.

    • On DSL over phone lines, the frequency band is split into a small part for upload and a much larger part for download. The “A” in ADSL literally stands for Asymmetric.
    • On cable (coax) networks, most of the spectrum is allocated to download channels and only a narrow, low-frequency band to upload.
    • On fiber, the technology can be close to symmetric, but many home plans still cap upload well below download.

    The reasons are partly technical and partly commercial. Most households consume far more than they send – streaming, browsing, downloading – so providers allocate the capacity where most customers use it. That’s perfectly reasonable for everyday use, but it’s exactly the wrong shape for anyone trying to move hundreds of gigabytes into the cloud.

    Option 3 – Renting a server in a data center

    At work, when we move a whole project folder between company servers, it’s fast. Not because “networks are magic”, but because the data never leaves the data center: the servers are connected to each other by very fast links, and your own PC only gives the order.

    I tried to replicate that idea for personal files. I rented the smallest cloud server I could find from Hetzner, a German hosting company (several other providers offer similar servers), billed by the hour. I copied my existing settings for the copy tool onto it, gave it the list of missing files, and started the copy there. From that moment the data travelled from Cloud Drive 1 to the server to Cloud Drive 2 entirely over data-center networks. My home connection was no longer involved, and I could close the laptop.

    The difference was dramatic. In about 17 hours the server copied around 34,000 files, more than nine times what the laptop managed overnight. The total cost for a couple of days of server time was under one euro.

    It wasn’t completely smooth, which is worth knowing if you try it:

    • Scanning thousands of folders before copying is slow even from a server (1–1.5 hours in my case).
    • The cheapest server has less than 1 GB of memory. Running two copy jobs at once made it run out of memory and kill one of them. Adding a 4 GB swap file solved it.
    • I had renamed a top-level folder in the old account in the meantime, so the file list no longer matched and that job had to be re-mapped.
    • Because the copy was set to never overwrite and always skip existing files, every restart was safe and nothing was lost or duplicated.

    Two shortcuts that didn’t work

    Before renting the server I also tried two “free” shortcuts. The cloud provider can sometimes copy files directly between two accounts on its own servers, which would be the fastest possible route – but the request between my two personal accounts was simply rejected as unauthenticated. The provider’s website did let me copy a shared file from one account into the other in seconds, but only individual files, not folders. With the missing data spread across about 2,000 folders, that wasn’t realistic.

    Side-by-side comparison

     1. Laptop as middle point2. External hard disk3. Rented cloud server
    Data passes throughHome internet, both waysHome internet, both waysData-center network only
    Main bottleneckHome upload speedHome upload speedCloud provider limits, folder scanning
    Laptop must stay onYesYes (twice)No
    Local disk spaceNot neededFull size of the dataNot needed
    ~160 GB would take≈ 8–9 days≈ 1 week (upload)≈ 1 day
    Cost€0≈ €100 disk< €1 (hourly)
    VerdictToo slowGood backup, slow transferBest for large transfers

    The full comparison, a time estimate you can adjust to your own internet speed, and typical speeds by connection type are in the accompanying spreadsheet.

    What I’d tell a friend

    • Compare before you copy. Matching by content fingerprint saved me copying well over 100 GB.
    • Check your upload speed first. If any method passes through your home internet, upload speed decides how many days it takes.
    • Move the work to where the data lives. For large transfers, a rented server in a data center turned a week into about a day, for less than the price of a coffee.
    • Keep it safe. Copy without overwriting, keep the old account until you’ve verified everything, and delete the rented server when you’re done – it holds your login tokens.
    • An external disk is still worth it, as a backup, not as the transfer method.