My biggest frustration is that I let people down. Currently, I work with teams of 50+ people, which means I need to be available all the time. The more senior as a PM you become the more folks you need to work with. Time becomes the most valuable thing and I'm still learning how to manage it effectively.
Being patient when I would like to be impatient. Product management is deeply humbling in that good things often take time. I like to say that I am "tactically impatient, strategically patient" - as in, obsess over small details and steps every day and treat them with urgency, but know that things take time to build, and big goals take time to achieve. But, being honest, this comes with a hidden frustration for me, an inherently impatient person :-)