I had been an Actor in Australia for around 20 years - you are so right. Turning up and being ready to work while, all the while, you're surrounded by a cast and crew who are very familiar and friendly with one another is a huge test of one's self belief. It can be quite distracting and, for many, an impediment to being anywhere near as good as you can be. It's a massive skill to master.
My honest belief is that there are many great actors but far fewer that are able to handle the pressures involved in being 'on set'.....when they're ready you had better be :-)