It depends on your session location. Parks and shaded areas near buildings etc. are more forgiving at different times of the day. Mostly I suggest sunrise and sunset. Here's why...
[Writing this from the car as I wait on my sunrise session client to arrive!]
Now for the sunrise pros and cons...
Sunrise Pros: less tourists in the background, less bugs biting, less humidity and much cooler temps. There's nothing like the feeling of having the early morning all to yourself! Sunrise sessions are unique and in my experience, you tend to be more relaxed at the start of the day. Maybe it has something to do with the fact that you're still waking up!
Sunrise Cons: you have to wake up early! Trust me when I say--- it's always worth it!
Sunset Pros: beautiful rays. Sunsets are romantic and sunset sessions are no exception!
Sunset Cons: depending on the time of year, it can be really hot and humid. No bueno! I've noticed kids are prone to meltdowns later in the day versus early morning sessions.
(By the way: anytime in between sunrise and sunset is going to be risky. Once the sun is up it doesn't matter if we meet at 8am, 9am, 10am, 2pm or 3pm etc. Its going to be bright, squinty and usually HOT session!)
My final answer... [drumroll]...SUNRISE! It takes a sacrifice on us all to choose sunrise but I think its 100% worth it and I haven't been disappointed yet!
p.s. Ideal session location: the beach. Always the beach!