The less revealing answer is because most agents are licensed real estate salespersons who work for a broker that takes a 50% commission split. Therefore, if an agent gives away 1% commission to the home buyer, he or she might be left with only half a percent in commission after the commission split with his broker.
The more controversial and accurate answer is that there is an act of collective collusion among the large brokerages in NYC. While it may be difficult to imagine 50,000 agents in a competitive market all independently deciding to collude and keep commission rates high, it is not so difficult once you realize that Principal Brokers at brokerages set firm policy on commission rates.
And because most agents in the city work for the biggest brokerages, collusion is only needed by a handful of Principal Brokers at the biggest firms. This is not so hard to imagine in a city with a tight-knit real estate broker community like New York City.