Since YouTube is going to recompress whatever you give it, your goal should be to get the file into the highest res format that they will accept, so that they do not compress an already overly compressed video. Youtube will now accept up to 1GB, but I would shoot for about 20MB or so per minute - 100MB for 5 minute video. Larger than this may also work fine but your upload will be longer with no noticeable difference. You can try this out yourself.
Here are some guidelines:
1. Resolution: Use 640×480 source as that gives youtube more resolution to work with when they re-encode down to 320×240. (or do the resize yourself if you think you have better software to do it, but they’ll just re-encode your 320×240 source back to 320×240 again, so why not give them more to work with?). In any case, keep an aspect ratio of 4:3, which is what Youtube uses. Also, the higher res will allow the new high res versions which Youtube will offer if your uploaded video is higher res.
2. Bit Rate: Use the HIGHEST bit rate combination for audio and video that you can and still stay under the 1GB upload per video limit. Again, this allows you to give YouTube the best possible source quality that they will accept, so when they run it through their transcoder, they’re starting with a nice clean source. Some say to aim for 1500kbps and this should be fine. Some encoders will let you set a target file size, so if yours does so, shoot for 100 MB, and you’re golden.
3. Format: MPEG4 (DivX, Xvid) is recommended. You can output from most encoders to mpeg4 and I’ve also had good results using Quicktime Pro saving to .mov format.
4. Audio: 64k Mono or 128k Stereo MP3 audio. Again, youtube will downsize your audio so start with at least this level of quality.
5. Frame rate: No matter what you upload, YouTube is going to encode it down to 15 fps, so you might be better off getting the cleanest 15 fps source you can in a good editor/encoder before uploading. On the other hand, if their encoder is good, you might want to feed it 30 fps so it can interpolate between frames itself with that higher bit rate source and 640×480 resolution. I recommend 30fps at 640 x 480 resolution at high bit rate.
Anyway, you should simply give Youtube the highest bit rate source to start with that still meets their requirements. This one thing alone will do more for end-result quality than the complicated technical tips offered on most how to optimize for Youtube video pages.