Yes, ASI affects the speed of attack animations, so it should have an effect on your switch-shooting rate.
That said, the point of switch shooting is to cut the animations short. So it might be true that ASI affects non-switch-shooting rates more than switch-shooting rates. I haven't done detailed timing tests, and I'm not a great switch-shooter either.