Nelson Mandela was held prisoner on the island for most of the 27 years he was in jail, and it is now a World Heritage site and major tourist attraction.
But reports say thousands of feral rabbits are devastating local wildlife and undermining historic buildings.
The premier of the Western Cape has called for urgent action on the site.
Helen Zille has demanded a meeting with South African Minister of Culture Lulu Xingwana to discuss how to halt its decline.
The island sits in the bay just seven kilometres from Cape Town’s beaches, and is visited by more than 1,800 tourists a day at peak times.
They come to see the grim conditions under which political prisoners, including Mr Mandela, were held.
But in a statement on Friday the City of Cape Town and the Western Cape provincial government said that if poor management of the island continued, it risked losing its World Heritage status.
Helen Bamford, a journalist with the Cape Argus Newspaper, told the BBC: “The rabbits are the main problem. I mean there are estimates of up to 25,000 of them, you know, roaming around the island and they are burrowing under historic buildings and eating all the vegetation.
“And there are also fallow deer. Last year they were in fact starving to death because of the lack of vegetation and supplementary food was brought onto the island for them and that caused the population to double and there are now about 500.
“At the moment they are being shot by a team of 10 hunters and the rabbits are also being culled, but that’s going to take years.”
Concern over the island comes just as the city prepares for the World Cup – and the hundreds of thousands for foreign visitors this will bring.