I hope you didn't do it exactly like you've drawn up there ^. If you did, you limited yourself to the current that poor little light bulb can handle when the switch is on for manual operation... and that probably isn't enough to run the fan. Also, if the light bulb burns out your fan will not be working when the switch is thrown.
Probably best to continue the conversation.
You are indeed right.. i had a switch with a built in light.. my pal used my the above drawing with an led light.. his fan went full speed.