We have a family of Western bluebirds nesting in our front yard. We moved 2-1/2 years ago and had the birds where we used to live but they were living in holes made by woodpeckers in large manzanita bushes. They were impossible to see from the house and we could not coach them into bluebird nesting boxes.
My wife and grand-kids put a nesting box straight out from the front windows where we are currently living. Now, a year later, the bluebirds have moved in.
What I am learning is that western grey squirrels and woodpeckers are both considered pests by the bluebirds. The nesting box is a few feet from a yellow pine that seems to be the home of a squirrel. The bluebird's home is directly between the yellow pine and the general mixed-seed feeder for the seed-eating birds (and the squirrels). The first time I noticed a problem was when I saw the squirrel climbing the pole to the bluebird box. He had just started up the pole when he was attacked by the BB. From then on, whenever the squirrel would pass by on the way to the seed feeder, he would be attacked. Once the squirrel ran down the road in front of our house with a BB attacking his tail. A passing jeep locked his brakes in an attempt to avoid hitting them. I saw the BB get away but I had to walk out into the road to verify that the squirrel also escaped.
There is also a flowering pear in between the nesting box and the feeder. I once saw an acorn woodpecker try working on that tree when he was driven off by the BBs.

Of course, tree squirrels are not carnivores but their attempts to rifle through the vegetative matter collected by the BBs would be hard on eggs or baby birds. Some people put metal around the opening to prevent squirrels (or woodpeckers) from widening the small opening. I think that woodpeckers would be a threat to take over the nest would explain the animosity.

Squirrels are my nemesis. They have destroyed so much stuff at my house.

I finally started relocating them by various ways.