There are no silly fears. If you're afraid of something, it's real, and it affects your thinking and performance no matter how invalid or valid someone else thinks it is. It's like there's no minor pain when it's yours. If it hurts, it hurts. What you've got to do is figure out how to deal with it, not use up your energies trying to decide if it's real or not.
Join other gay dads at gayfatherhood.com, a parenting site with resources and support by and for fathers.