i am not sure that there is a performance issue but it sounds logical to me that there will be some performance issue there. anyway i like order (like all programmers) and i do not like having a 1.1. assembly on a 2.0 project
i really think that 2.5 years after VS 2005 was released you should have released the .net 2 version of the assembly. in my project i have about 10 diffrent 3rd party assemblies and your's is the only one who wasn't compiled for .NET 2.
P.S VS vNext is just around the corner and i think you should be ready for it too and release a build immidietly when it's launched (even with no modifications).