If I understand you correctly you have already created an .AVI file that has an alpha channel. I do that often as well. I then use Adobe Flash CS3 and Apple QuickTime Pro to then create .FLV’s and .MOV’s Both of those products have an OPTIONAL SETTING to Include the alpha channel. But I should also say I don’t believe you can currently encode an .FLV to H264 For that I use Quicktime Pro.
One other thing to consider is the player you use to display the content. Not all players have a transparency mode. In fact most do not. Again as I understand it using a flash player in the transparent mode disables the full screen option of the flash plugin. That may have changed in the newer versions of the Flash Plugin but I haven’t seen any mention of it yet.
In my case I have modified the JW Player to permit transparency. Here’s a quick example. Demo In the example the transparent file is an .FLV with alpha channel.
Hope this offers you some help. - Jimb