Friday, November 24, 2006

Mission Impossible: Upload Faster Than Your Upload Limit

Warning: this is a very technical post. Read at your own risk...

The Problem
There's a certain US show that airs on Wednesday nights. When it's Wednesday night in the US, it's actually early morning in Israel, where I live. By Thursday morning Israel time, there's already a torrent for downloading a high quality, commercial free, video of that certain show. The episode takes 350MB, and usually takes 4-7 hours to download. That's great, since I usually watch the episode on Thursday night Israel Time, which is less than a day after it originally aired.

But what about my fellow workers at the office? Usually they only get to see the episode on Sunday, since the weekend in Israel starts on Friday. But for the new season that started in October I decided I would try to "provide" the episodes on Thursdays. That's not that easy: let's say the download (which I'm doing at my home) finishes at 2 PM. Transferring 350MB would take around 5.5 hours (My actual upload speed from home is 18KB/Sec). This brings us to 7:30 PM on Thursday - most of my co-workers try to leave earlier on Thursday, and start the weekend "early". If I could only download the episode a little faster, and it would be at the office by 6:30 PM, it would be just in time.

The Solution
I usually start the download (using Azureus) at 7 AM when I wake up. Then I need to wake up the kids, dress them up, take them to kindergarten, take a shower, pack up the laptop and go to work. It's probably around 9 AM by the time I leave my home for work. Since I take my laptop from home to work, it can be used for "file transferring". By 9 AM 20% to 30% of the download already completed, so I perform the following steps:
  • I make a copy of the video file on my home computer, and then copy that copy to my laptop, which I take to work.
  • I go to work and wait for the download to complete. I use the Azureus HTML WebUI plugin to monitor the download progress from work.
  • When the download completes, I create a binary diff file between the copy I took at 9 AM and the complete file. The diff file is smaller than 350MB. For example, if by 9 AM I already downloaded 100 MB, the diff file would be around 250 MB.
  • I then stop Azureus (to get maximum upload speed to the office), and transfer the diff file to work. A 250MB file transfer takes "only" 4 hours.
  • At work I apply the diff file as a patch to the copy of the 9 AM file and get the finalized file.
  • I never forget to start Azureus once again, and make sure I seed the file for a long time.
The Technical Details
The first time I tried this "trick". I wasn't sure there was a program that could take two binary files, and produce a difference file that could later be used to patch one into the other. But since I use Ubuntu at home, and I have a friend (and co-worker) who's a Linux expert. I was sure that if I take a copy of the video download before it's completed, I would find a way (asking my friend) to patch it to be complete.

I was right. My friend told my about a tools called bsdiff. But after looking at its man page, I saw I wouldn't be able to use it. To do its magic it would require 5.8GB of memory (17 times the size of the original file, aparently it's optimized for executables, which as usually small). I only have 320MB. Still, the bsdiff documentation mentioned another tool - xdelta, which was supposed to produce larger diff files, but doesn't require so much memory.

Indeed it worked, and here's what I did:
  • 9 AM at home:
    copy episode.avi ep-9am.avi
  • I then copy ep-9am.avi to my laptop computer, so I have it at work later.
  • 2 PM at home (using Putty to access my home computer from work):
    xdelta delta ep-9am.avi episode.avi episode.patch
  • I then copy the file from home to work using WinSCP.
  • 6 PM at work:
    xdelta patch episode.patch ep-9am.avi episode.avi
That's it - I get the episode fast enough for weekend viewing by my co-workers. If you need xdelta on Ubuntu/Debian you can simply use:
sudo apt-get install xdelta

If you're using Windows, you'll need to get a Windows port of this tool, which is supposed to exist.

You could watch the unmentioned TV show on one of the cable channels in Israel, but you'll see episodes that were broadcast in the US last year. Since we're in Israel, we're blocked (based on Geolocation information) from downloading the episodes from legitimate places (like the Apple Store). For some of us that's not acceptable, so we need to download the episodes and over the internet using BitTorrent. Hopefully, this phenomenon would be widespread enough to force the content providers to allow for international, legal distribution of their content. I would be willing to pay for such a service.


Jerry said...

cheap wedding gowns,
discount bridal gowns,
China wedding dresses,
discount designer wedding dresses,
China wedding online store,
plus size wedding dresses,
cheap informal wedding dresses,
junior bridesmaid dresses,
cheap bridesmaid dresses,
maternity bridesmaid dresses,
discount flower girl gowns,
cheap prom dresses,
party dresses,
evening dresses,
mother of the bride dresses,
special occasion dresses,
cheap quinceanera dresses,
hot red wedding dresses

ally said...

Fashion is all about lookingmoncler jackets good and being comfortable at the same time. monclermoncler coatsEvery woman loves being complimentedmoncler t-shirtmoncler vestmoncler outlet for her sense of style and elegance. ugg bootscheap ugg bootsdiscount ugg bootsNo woman wants to be labeled as outdated and old fashioned.ugg classic tall bootsclassic ugg bootsWomen are expected to be in constant touch with the latest trends in ordermoncler jacketsmonclernew moncler coats to adopt those trends and look fashionable and stylish. moncler vestmoncler outletmoncler polo t-shirtWomen need to make sure that whatever they wear, coach outletcoach handbagscoach bagsit should compliment their personality and they do not feel awkward while wearing it. coach totescoach outletlouis vuitton handbagsAn article about Gothic costume clothing and the importance of choosing Gothic choker. LV handbags 2010Louis Vuitton bagsLouis Vuitton totesThis article explores the wide variety of Gothicrain boots
rainweardiesel jeans chokers available from everyday to special occasions. Ture Religion Jeanslevis jeansabercrombie and fitch outletIt should inspire people to go shopping for their own Gothic chokers. ed hardy wholesaleed hardy outletcheap ed hardy wholesaleThe origins of the Flat cap span discount ed hardy wholesalewholesale ed hardy

ally said...

Fashion trends change on daily basis, like Gold GHD. Following the latest in designer shades has become a passion of everyone, now Burberry Sunglasses. If you are the type of a woman who loves to explore in fashion, our ED Hardy Sunglasses will definitely satisfy your taste. Cheap Ed Hardy Sunglasses is also OK. Ed hardy streak of clothing is expanded into its wholesale ED Hardy T-shirt chain so that a large number of fans and users can enjoy the cheap ED Hardy Clothing range easily with the help of numerous secured websites, actually, our ED Hardy Outlet. As we all know, in fact discount ED Hardy, is based on the creations of the world renowned tattoo artist Don Ed Hardy. Well, this question is bound to strike the minds of all individuals. Many people may say Prada shoes is a joke, but we can give you Prada Sunglasses, because we have Prada handbags. Almost everyone will agree that Prada Purses are some of the most beautiful designer handbags marketed today. Now we have one new product: Prada totes. The reason is simple: fashion prohibited by ugg boots, in other words, we can say it as Cheap ugg boots. Would you like to wear Discount ugg boots. We have two kinds of fashionable boots: classic ugg boots and ugg classic tall boots. Ankh Royalty--the Cultural Revolution. Straightens out the collar, the epaulette epaulet, the Ankh Royalty Clothing two-row buckle. Now welcome to our Ankh Royalty Outlet. And these are different products that bear the most famous names in the world of fashion, like Ankh Royalty T shirt by the way -Prada, Spyder, Moncler(Moncler jackets,or you can say Moncler coats, Moncler T-shirt, Moncler vest,and you can buy them from our discount Moncler outlet), GHD, ED Hardy, Ankh Royalty, Twisted Heart.

janewangleilei said...

You can have a look at it.
jordan shoes
jordan ajf shoes
There are cheap shoes to choose
jordan 6
jordan 7
Good quality with low price.
air jordan 2010
Air Jordan 2009
If you like,you can contact us.
jordan 3
jordan 4
We offer different styles.
jordan 1
jordan 2
jordan 5
I can't believe it.
nike outlet
You can have a look at it.
adidas outlet
puma outlet
north face
ak jackets
spyder jackets
Columbia Sportswear
quiksilver jackets
Enjoy it.
burton jacket
powder room jackets
karbon jacket
goldwin jackets
I can't believe it.
eider jackets
You can have a look at it.
sportalm jackets
west scout

janewangleilei said...

You can have a look at it.
jordan shoes
newest jordans
jordan 7
air yeezy
jordan true flight
If you like,you can contact us.
jordan 3
jordan 4
We offer different styles.
jordan 1
jordan 2
jordan 5
jordan ajf shoes
There are cheap shoes to choose
jordan 6
jordan flight 45
Good quality with low price.
air jordan 2010
Air Jordan 2009
Enjoy it!
jordan 28

Pink Love said...

5 Nike Dunks
Nike Dunk High
Nike Dunk Low
SB Nike dunk

Nike Dunk High Premium

6 Abercrombie and fitch
cheap hollister
Abercrombie & fitch
7 Abercrombie & Fitch

Cheap Abercrombie

ruehl no.925

Abercrombie and Fitch Hoodies

Abercrombie and Fitch bikini

8 ED Hardy

ED Hardy swimwear

ED Hardy bikinis

ED Hardy shoes

ED Hardy Sunglasses

dfadf said...

The Tax Return Crack-Up<2>
I was not shocked because this was old news -- practically ancient, in fact. In R. Microsoft Office Emmett Tyrrell, Office 2010 Jr.'s most recent book The Clinton Microsoft Office 2010 Crack-
Up, page fiv Office 2007 e, paragraph two, we learn that in Bill Clinton's "first four years out of the White H Microsoft Office 2007 ouse, he ea Office 2010 key rned over Office 2010 download $43 million Office 2010 Professional after
expenses... Microsoft outlook "
The next Outlook 2010 page directs Windows 7 us to Appendix Microsoft outlook 2010 I, a list of the conniving couple's fees for speeches and book royalties and other income. The first

elizabeth bernot said...

Costa Rica Fishing

Fishing Costa Rica

Betting Online said...

i'm mario : Thanks for sharing. Very impressive

Bella Tran said...

thank you so much for the information you present very useful,
play game juegos de terror and play game jogos online gratis online and play game unblocked games