Mission Impossible: Upload Faster Than Your Upload Limit

Warning: this is a very technical post. Read at your own risk...

The Problem
There's a certain US show that airs on Wednesday nights. When it's Wednesday night in the US, it's actually early morning in Israel, where I live. By Thursday morning Israel time, there's already a torrent for downloading a high quality, commercial free, video of that certain show. The episode takes 350MB, and usually takes 4-7 hours to download. That's great, since I usually watch the episode on Thursday night Israel Time, which is less than a day after it originally aired.

But what about my fellow workers at the office? Usually they only get to see the episode on Sunday, since the weekend in Israel starts on Friday. But for the new season that started in October I decided I would try to "provide" the episodes on Thursdays. That's not that easy: let's say the download (which I'm doing at my home) finishes at 2 PM. Transferring 350MB would take around 5.5 hours (My actual upload speed from home is 18KB/Sec). This brings us to 7:30 PM on Thursday - most of my co-workers try to leave earlier on Thursday, and start the weekend "early". If I could only download the episode a little faster, and it would be at the office by 6:30 PM, it would be just in time.

The Solution
I usually start the download (using Azureus) at 7 AM when I wake up. Then I need to wake up the kids, dress them up, take them to kindergarten, take a shower, pack up the laptop and go to work. It's probably around 9 AM by the time I leave my home for work. Since I take my laptop from home to work, it can be used for "file transferring". By 9 AM 20% to 30% of the download already completed, so I perform the following steps:

• I make a copy of the video file on my home computer, and then copy that copy to my laptop, which I take to work.
• I go to work and wait for the download to complete. I use the Azureus HTML WebUI plugin to monitor the download progress from work.
• When the download completes, I create a binary diff file between the copy I took at 9 AM and the complete file. The diff file is smaller than 350MB. For example, if by 9 AM I already downloaded 100 MB, the diff file would be around 250 MB.
• I then stop Azureus (to get maximum upload speed to the office), and transfer the diff file to work. A 250MB file transfer takes "only" 4 hours.
• At work I apply the diff file as a patch to the copy of the 9 AM file and get the finalized file.
• I never forget to start Azureus once again, and make sure I seed the file for a long time.
  

The Technical Details
The first time I tried this "trick". I wasn't sure there was a program that could take two binary files, and produce a difference file that could later be used to patch one into the other. But since I use Ubuntu at home, and I have a friend (and co-worker) who's a Linux expert. I was sure that if I take a copy of the video download before it's completed, I would find a way (asking my friend) to patch it to be complete.

I was right. My friend told my about a tools called bsdiff. But after looking at its man page, I saw I wouldn't be able to use it. To do its magic it would require 5.8GB of memory (17 times the size of the original file, aparently it's optimized for executables, which as usually small). I only have 320MB. Still, the bsdiff documentation mentioned another tool - xdelta, which was supposed to produce larger diff files, but doesn't require so much memory.

Indeed it worked, and here's what I did:

• 9 AM at home:
  copy episode.avi ep-9am.avi
• I then copy ep-9am.avi to my laptop computer, so I have it at work later.
  
• 2 PM at home (using Putty to access my home computer from work):
  xdelta delta ep-9am.avi episode.avi episode.patch
  
• I then copy the file from home to work using WinSCP.
• 6 PM at work:
  xdelta patch episode.patch ep-9am.avi episode.avi

That's it - I get the episode fast enough for weekend viewing by my co-workers. If you need xdelta on Ubuntu/Debian you can simply use:
sudo apt-get install xdelta

If you're using Windows, you'll need to get a Windows port of this tool, which is supposed to exist.

Disclaimer
You could watch the unmentioned TV show on one of the cable channels in Israel, but you'll see episodes that were broadcast in the US last year. Since we're in Israel, we're blocked (based on Geolocation information) from downloading the episodes from legitimate places (like the Apple Store). For some of us that's not acceptable, so we need to download the episodes and over the internet using BitTorrent. Hopefully, this phenomenon would be widespread enough to force the content providers to allow for international, legal distribution of their content. I would be willing to pay for such a service.