Mission Impossible: Upload Faster Than Your Upload Limit

Warning: this is a very technical post. Read at your own risk...

The Problem
There's a certain US show that airs on Wednesday nights. When it's Wednesday night in the US, it's actually early morning in Israel, where I live. By Thursday morning Israel time, there's already a torrent for downloading a high quality, commercial free, video of that certain show. The episode takes 350MB, and usually takes 4-7 hours to download. That's great, since I usually watch the episode on Thursday night Israel Time, which is less than a day after it originally aired.

But what about my fellow workers at the office? Usually they only get to see the episode on Sunday, since the weekend in Israel starts on Friday. But for the new season that started in October I decided I would try to "provide" the episodes on Thursdays. That's not that easy: let's say the download (which I'm doing at my home) finishes at 2 PM. Transferring 350MB would take around 5.5 hours (My actual upload speed from home is 18KB/Sec). This brings us to 7:30 PM on Thursday - most of my co-workers try to leave earlier on Thursday, and start the weekend "early". If I could only download the episode a little faster, and it would be at the office by 6:30 PM, it would be just in time.

The Solution
I usually start the download (using Azureus) at 7 AM when I wake up. Then I need to wake up the kids, dress them up, take them to kindergarten, take a shower, pack up the laptop and go to work. It's probably around 9 AM by the time I leave my home for work. Since I take my laptop from home to work, it can be used for "file transferring". By 9 AM 20% to 30% of the download already completed, so I perform the following steps:

• I make a copy of the video file on my home computer, and then copy that copy to my laptop, which I take to work.
• I go to work and wait for the download to complete. I use the Azureus HTML WebUI plugin to monitor the download progress from work.
• When the download completes, I create a binary diff file between the copy I took at 9 AM and the complete file. The diff file is smaller than 350MB. For example, if by 9 AM I already downloaded 100 MB, the diff file would be around 250 MB.
• I then stop Azureus (to get maximum upload speed to the office), and transfer the diff file to work. A 250MB file transfer takes "only" 4 hours.
• At work I apply the diff file as a patch to the copy of the 9 AM file and get the finalized file.
• I never forget to start Azureus once again, and make sure I seed the file for a long time.
  

The Technical Details
The first time I tried this "trick". I wasn't sure there was a program that could take two binary files, and produce a difference file that could later be used to patch one into the other. But since I use Ubuntu at home, and I have a friend (and co-worker) who's a Linux expert. I was sure that if I take a copy of the video download before it's completed, I would find a way (asking my friend) to patch it to be complete.

I was right. My friend told my about a tools called bsdiff. But after looking at its man page, I saw I wouldn't be able to use it. To do its magic it would require 5.8GB of memory (17 times the size of the original file, aparently it's optimized for executables, which as usually small). I only have 320MB. Still, the bsdiff documentation mentioned another tool - xdelta, which was supposed to produce larger diff files, but doesn't require so much memory.

Indeed it worked, and here's what I did:

• 9 AM at home:
  copy episode.avi ep-9am.avi
• I then copy ep-9am.avi to my laptop computer, so I have it at work later.
  
• 2 PM at home (using Putty to access my home computer from work):
  xdelta delta ep-9am.avi episode.avi episode.patch
  
• I then copy the file from home to work using WinSCP.
• 6 PM at work:
  xdelta patch episode.patch ep-9am.avi episode.avi

That's it - I get the episode fast enough for weekend viewing by my co-workers. If you need xdelta on Ubuntu/Debian you can simply use:
sudo apt-get install xdelta

If you're using Windows, you'll need to get a Windows port of this tool, which is supposed to exist.

Disclaimer
You could watch the unmentioned TV show on one of the cable channels in Israel, but you'll see episodes that were broadcast in the US last year. Since we're in Israel, we're blocked (based on Geolocation information) from downloading the episodes from legitimate places (like the Apple Store). For some of us that's not acceptable, so we need to download the episodes and over the internet using BitTorrent. Hopefully, this phenomenon would be widespread enough to force the content providers to allow for international, legal distribution of their content. I would be willing to pay for such a service.

I'm Too Old For My Job

Yesterday I saw this piece of code in a unit test (We're using JUnit like everyone else) we have for one of our components:

Timestamp dawnOfTheAges = new Timestamp(0);

I immediately e-mailed the person who wrote the code complaining about the fact that constructing a Timestamp like this creates something that's much more recent than the dawn of the ages. It's midnight as it was celebrated at any London pub on January 1st, 1970. Hell, how could this be the dawn of time if I was born over a year before that?

I'm too old for my job as a programmer...

Linux, Microsoft and XNA

About a year ago I decided to give Linux a chance (I chose Ubuntu). I assume I'm not an average PC user since I've been a programmer since 1986, so if someone could use Linux as the operating system for a desktop computer (my home computer), I should be able to do it as well.

In the last year I used Linux (I'm definitely not an expert, and I don't have the time to become one), and found many great things it can do, but also a lot of shortcomings. During this time I found myself wondering a lot about what's missing (or wrong) in Linux that prevents me from recommending it to my relatives (for whom I provide regular technical support).

At this point, there are quite a few reasons why non-technical people should not switch to Linux. One that's been there for quite a while is that modern games are not ported to Linux. A PC is still used a lot for games, and that's where Linux fails. Don't get me wrong - after working a bit with Linux, it looks like it's inherently better than Windows when it comes to many aspects like handling files, security, and networking. Still, I assume many people won't switch to Linux because there are no games.

Enters Microsoft XNA. It's main purpose is to let hobbyists develop games more easily - for Windows and for the Xbox 360. I gave it a try and it looks like an excellent framework - I only tried 2D stuff and still managed to see its power and simplicity. I think XNA will be a success.

The real question is if we're going to stop playing games on PCs, and start playing them mostly on game consoles.

• If the answer is yes, then Microsoft just made it easy for small publishers to create small games for the Xbox 360, but it will loose one of the main reasons not to switch to Linux. In this case, it's also important for Sony to come up with something similar to XNA really fast...
• If the answer is no, then XNA is a really nice framework for building PC games, and the Linux community should start working on an alternative. Microsoft was smart in investing in DirectX a decade ago, and if PCs are going to be used a lot for playing games in the future, this is another smart move from Microsoft.
  

The open source community can definitely build an XNA alternative for Linux. Something that would make developing Linux games a breeze. I'll join such a project, but I'm too new to Linux to be able to contribute significantly. Any volunteers?

Why HD 1080i Sucks (and 1080p Doesn't)

CNET published a recent article saying on a visual comparison, they can't see substantial difference between 1080i sets and 1080p sets (The case against 1080p). I find such articles alarming. I think 1080i sucks, and here's why.

Simply put, 1080i sucks because interlaced broadcasting sucks.

The real issue with interlaced content is not the TV set. I know CNET is all about telling people what equipment to buy, but the real issue is the content itself. When your content is interlaced you don't have frames, you have half-frames that are interleaved. It's not that terrible when all the TVs are also interlaced, and all have the same resolution.

But this is not the case anymore. Now you can watch HDTV on displayes with various resolutions such as 1920x1080 (full HD), 1280x720 (the lower resolution HD), or even 1366x768 (which is a wide version of the popular 1024x768 computer resolution, useful when you need to use your display for movies as well as a computer monitor).

Now here's the tricky part - if you don't have complete frames, you can't resize the image. What you need to do is break the 1920x1080 resolution into two images, each 1920x540, scale each image, and then try to combine the result into a complete frame once again (or maybe keep it interlaced). The results of this process are not excellent. Another option is to try to recreate the full frames from the half frames you have, but doing this properly is the kind of magic that consts a lot of money, I mentioned that ealier in my post "HDTV is cheaper than your old TV".

You might think your safe if you bought a 1920x1080 set, but I've already heard of plans to manufacture 2560x1536 sets. If your content is 1080i, it will be hard to scale properly for the TV of the future (or next year's projectors?!?).

Here's another good reason why interlaced sucks - because display technologies today are progressive. LCD, Plasma, and DLP are all progressive technologies. When you feed them interlace content, they have to process it somehow (deinterlace) to fit their progressive nature. Sure, if your original material is progressive (film) and you "convert" it to interlaced, it will look well when you play it back on progressive devices, but if your material was shot by an 1080i camcorder, this conversion will either have interlacing artifact, or you'll loose some sharpness.

When modern HDTV was concieved in the US in the mid 1990s, CRT was the dominating technology, and the people who concieved HDTV were visionairs, but still had a limited vision. They thought the amount of information in a 1080p (at 60 full frames per second) would be too much. Well, it's not too much now, even before 1080i is popular, and just to make sure what I need when I say 1080p, I want a set that supports 1080p at 24 full frames per second, so that movie transfers would be as accurate as possible. 1080 @ 24p contains less information than 1080 @ 60i (If someone starts shooting movies at 60 frames per second, I'll be happy to see it on a 1080 @ 60p set).

I don't really care if CNET recommends 1080i sets over 1080p sets because they're cheaper, but if this means it will take more time until we have enough content in 1080p, then that's a shame. And CNET is right about one thing - 1080p content looks good even on a 1080i set (assuming it can accept it).

Converting DVD content to DV

Why should you convert DVDs to DV?

If you’d like to backup complete DVD movies, there’s little sense in converting them to DV. A 2-hour movie, converted to DV would consume a lot of your HD space - something like 26GB. DV is used for video editing, so if you’d like to edit your own videos, and insert clips from DVDs into them, this tutorial is for you.

What you need

You'll need the following:

• DVD content - these are VOB files that contain the video materials. We’ll talk more about these files later.

• VirtualDubMpeg2 - a variant of the popular VirtualDub program that supports the MPEG2 format (the format used in DVDs). Get version 1.6 or newer, otherwise you might have synchronization issues with NTSC video. You'll also need a an AC-3 codec, see information in the above link.

• Cedocida DV codec - a software codec for converting video to the DV format.

Obtaining the DVD content

Open the DVD using Explorer. You should see 2 folders in it: AUDIO_TS and VIDEO_TS. Open the VIDEO_TS folder. You’ll see something like this:




The audio/video content is kept in the VOB files. These are definitely the larger files on the DVD. Usually, the main movie is stored in VTS_01_1.VOB, VTS_01_2.VOB, etc. Other VOB files contain extras and menus.

The DVD standard dictates that VOB files will be at most 1GB in size. Each such files stores 20-25 minutes of video. You’ll have to find the VOB with the movie segment you’re interested in (again, I assume you’re not planning on converting a whole movie to DV, it just doesn’t make sense). You’ll do that by opening each VOB in VirtualDubMpeg2 and checking the content in it. Since you’ll be working a lot with the VOB files, I suggest you copy them to your HD, and remove them after you’re done.

One last note: An original DVD is encrypted and cannot be read by VirtualDubMpeg2. You’ll have to decrypt the DVD using a software like DVD Decrypter, DVDFAB Decrypter or DVD Shrink. In some places around the world this decryption is considered illegal, while in other, it’s allowed as long as it’s used for creating backup of content you legally own. In any case, I’m sure doing this to add a few 5 second clips into your son’s Bar-Mitzvah’s movie won’t result in an FBI team knoking your door. Still, you’ll have to find another tutorial to see how to work with these decryption software programs.

Loading the DVD video into VirtualDubMpeg2

1. Run VirtualDubMpeg2 (the executable is called VirtualDub, so I’ll be using this name from this point forward).
2. To load a VOB file, either drag-n-drop the file into VirtualDub or use File » Open video file…
3. In many cases, the video within the DVD contains multiple audio streams (usually for multiple languages or director’s commentary). If that’s the case with your DVD, you’ll see the following dialog:
   
   You’ll have to choose the audio stream to be used in your DV avi file. a DV avi file can only have a single audio stream. If you don’t know which audio stream to choose, pick the top one. You can later play the video, and if it’s the wrong audio track, reload the VOB and choose another…
4. VirtualDub will now scan the VOB file, this should take a few seconds. If you load the VOB directly from the DVD disc (you didn’t bother to copy the VOB to your HD), it might take a few minutes. You’ll see the following progress dialog:
   
   If you only used VirtualDub for manipulating AVI files, you never saw this progress dialog. The AVI file format requires no parsing, since it contains a chunk with this information (towards the end of the movie). A VOB file requires such parsing.

Converting the video to DV

1. First, you might want to select the range you want to convert. Remember, DV takes 3.6MBps (that’s Bytes, not bits per second). There’s a chance the range you’re intereseted in is actually between VOBs. We’ll talk about this later.
2. We’ll now set the audio conversion options:
   
• Select the Audio » Full processing mode option.
• Now select the Audio » Compression… option. The audio compression dialog will open:
  
• Make sure the option is selected. DV audio is stored as uncompressed 16 bit stereo samples at 48Khz. Fortunately, DVD audio is also stored at 48Khz, which means no sample rate conversion is required. Click the OK button to return to the main screen.
  
3. Next, we’ll set the video conversion options:
   
• Make sure the Video » Full processing mode option is selected.
• Select the Video » Compression… option. The video compression dialog will open:
  
• Select the Cedocida DV codec compression (selected above). Click the OK button.
4. Now, to convert to DV avi, choose File » Save as AVI… (F7). Choose a name for your file and click the Save button. Now sit back and wait while the conversion (or transcoding) takes place. Either that, or you can read on, for a few “advanced” topics…
   

Advanced: Handling Wide Anamorphic DVDs

Many DVDs are encoded in anamorphic widescreen format. If that’s the case, looking at the video in VirtualDub, everything should look vertically streched - all people will look tall and narrow. For example, see the following frame from the TV show ‘Joey’:



This will also happen when you combine this video with your content, which you probably didn’t shoot in anamorphic widescreen. You’ll have to resize the video content to fit your video.

The reason for this is very simple. Your video is shot in “normal” 4:3 mode (if it doesn’t, you can safely skip this part). The video on the DVD is stored in widescreen format known as 16:9. The two formats (wide and normal), can’t co-exist in a single movie. You’ll need to modify the DVD video to fit your video. You have two options:

• Add black bars at the bottom and top of the DVD video. The added bars increase the height of the video image so that it’s 4:3. You’ll see the whole content of the original DVD video, but it will look smaller. This method is called “Letterbox”. The above frame should looke like this in letterbox format:
  
• Crop the DVD videe, removing some content, usually from the sides of the image. You remove just enough content so that the video image is 4:3. You’ll loose some of the content of the original video, but it will occupy the whole screen. This method is called “Pan & Scan”, although we won’t do any panning when we implement this method. The above frame should look like this in cropped format (notice the missing plant on the left hand side):
  

To letterbox the video, perform these steps just before saving the AVI:

1. Choose Video » Filters…. Click the Add… button, choose the resize filter and click the OK button. You’ll see the following dialog:
   
2. Check the Expand frame and letterbox image
3. Based on your TV system (Europen PAL or American NTSC), fill the following values in the dialog fields:
   

   Field	PAL value	NTSC value
   New width	720	720
   New height	432	360
   Frame width	720	720
   Frame height	576	480

   
   
4. Click on the OK button. You’ll return to the filter list dialog, which should show the resize filter. Click OK to exit from the filter list. You can now save your letterboxed AVI.
   

To crop the video, perform these steps just before saving the AVI:

1. Add a resize filter as described above. Don’t check the Expand frame and letterbox image, and fill in the following values, based on your TV system (PAL or NTSC):
   

   Field	PAL value	NTSC value
   New width	720	720
   New height	576	480

   
   
2. Click on the OK button. You’ll return to the filter list dialog, which should show the resize filter. It should be selected. Now click the “Cropping…” button.
3. For fields “X1 offset” and “X2 offset”, fill the value 90. Leave “Y1 offset” and “Y2 offset” as is (zero) and click the OK button. Click OK again to exit the filter list. You can now save your cropped AVI.
   

Advanced: Joining VOBs

There’s a chance the video fragment you’re interested in starts in one VOB and ends in another (probably the next one). You could convert each part separately, but there’s a slight risk of a sound click between segments. To do it the right way, you’ll need to combine the two VOBs into a single file, which you’ll then open in VirtualDub.

If you’re not afraid of using the command line, it’s quite simple to combine two VOBs into once. Just use the COPY command in the following way (assuming the current directory is where you want to store the result VOB):

copy d:VIDEO_TSVTS_01_1.VOB /b + d:VIDEO_TSVTS_01_2.VOB /b COMBINED.VOB

This command says to copy the binary (/b) file VTS_01_1.VOB and the binary file VTS_01_2.VOB into a file called COMBINED.VOB.

HDTV is cheaper than your old TV

Okay, not for everyone, but if you want to really enjoy seeing movies at your home, you should be happy that HDTV is getting closer and closer, because it's going to be much cheaper to enjoy a cinema-like experience at your home.

How come? Everyone is talking about how expensive high-def TVs cost and the premium you have to pay for high-def players that support BluRay or HD DVD. While this is true, providing content in HD is essential for lowering the cost of cinema-like experience at your home.

To recreate the cinema experience at your home you'll have to use a projector (maybe a 65" Plasma/LCD display can do the job as well, it's unlikely I'll be ever rich enough to know). Show the average guy a wide 100" diagonal screen, project any decent DVD content on that screen, and he'll tell you it's like going to the cinema, but it's not.

The fact of the matter is that a DVD simply doesn't hold enough data so that a 100" diagonal display would look good enough on it. The image is not sharp enough and you begin to see the compression artifacts. You now have two options:

1. You can watch DVDs on a smaller screen. 32" to 42" is fine. The DVDs would look great, but you'll definitely loose the cinema-like feeling.
2. You can start investing in equipment that will try to digitally enhance the DVD image so that it will look better on a large screen. I did a quick search on such a device - the Faroudja DVP-1510. It costs $7,299.99 (that's web price as of October 2006, not MSRP). You can try the cheaper DVP-1010, which will "only" cost $3,299.99.

I know there are such video processors for around $1000 (for example DVDO Iscan HD), but one has to wonder why these devices are so expensive. The simple answer is that these devices are supposed to perform the impossible, and that's quite hard - they try to create an image that's better than the source. They try to add details to an image (well, to a video - a sequence of images) so that it would be good enough to show on a large screen. Since the original image doesn't have enough detail, they have to perform really complex tricks, and billions of calculations per second to give us the impression of images with more information.

The funny thing is that these devices can perform this magic, but the price is very high.

Enters HDTV. At 1280x720 pixels, a HD image has 2.6 more data than an NTSC DVD frame (with 720x480 pixels) and 2.2 more data than a PAL DVD frame (with 720x576 pixels). No magic from Faroudja (and at no cost) can add so much information to an image. Now that's only the "low-resolution" HD. At the higher HD resolution, a 1920x1280 image fits exacly 6 DVD images. That's really a lot of pixels for our eyes.

So what's the bottom line? If the source is high-def enough, you won't have to invest in technology to enhance the image so that it's decent on a large enough screen. Trying to recreate the added data is a very complex and expensive task (and the results are never as good as HD). If you buy an HD player instead of a good scaler, you'll have enough spare money for a good projector.

Only with HDTV you could really have a home theater (that gives you a cinema-like experience) on a budget.