The olive tree (Olea europaea) is a native of the Mediterranean region and, as such, it requires very specific soil conditions that are seldom reproduced in New Zealand.
Attempts made to cultivate the trees in New Zealand have met with varying success and a lot of trees planted have reverted to their wild state. Trees that revert to the wild often have very small fruit with very little flesh and large stones, (pits). Therefore they are not viable as a food source. The tree in the front garden, (and a younger tree in the rear garden, between the sheoak and totara trees), both exhibit that tendency and can be considered to have reverted.
Even though the Williams House trees are not producing viable fruit, they are nonetheless excellent specimens as ornamental trees. The down side of this is that the smaller fruit and seeds are easily spread by native birds attracted to the fruit.
So far this has not caused major problems in New Zealand. However, in some parts of Australia such trees threaten the stability of the native fauna, both from competition for sapce and nutrients as well as the high oil content of the foliage posing a danger during bushfires.