The Famous Five, Julian, Dick, Ann, George and Tim, the dog, go to stay at a cottage near the sea. Here they meet a strange boy named Wilfred who has an uncanny power over animals and can summon them to him when he plays his pipe. They learn about the legend of buried treasure on the nearby Whispering Island. They also hear rumours of traps and snares on the island and of gamekeepers patrolling the grounds. Wilfred, with Tim, goes to the island to see for himself what is going on. They are chased into hiding by the gamekeepers.
The other four children go to the island to find what has happened to Wilfred and Tim. While they are rowing across, a high wind whips up the sea and their boat is nearly flung against the rock. Meanwhile Wilfred and Tim discover the traps and snares on the island. Wilfred reports this to Sir Hugo Blaize, the owner, who disbelieves him and has him locked up for the night. The other children, whose boat has finally capsized, are left floundering in the water.
The children swim ashore through the rough sea. Wilfred manages to escape from the castle and in the process discovers crates of rare striped rabbits waiting to be sent away. He overhears the gamekeepers say that they are to be sold to fur dealers in Paris. The children search the island in their effort to locate Blaize Castle and find Wilfred. As night descends Wilfred is sighted and chased by the gamekeeper. He hides on top of an old well-shaft whose rotten wood suddenly gives way, dropping him to the bottom of the well.
Julian, Dick, Anne and George, still searching the island for Wilfred, hear his cries and go to the rescue, but as he is hauled to safety the gamekeepers surprised them all and takes them to Sir Hugo to be dealt with. Eventually the children manage to persuade Sir Hugo that the gamekeepers are trapping the animals and are planning to sell them for personal gain. Realising they have been found out, the gamekeepers lock up Sir Hugo and the children in the kitchen, but they are ingeniously out-witted by their prisoners, who escape. In the old cellars and passages beneath the castle they search for the legendary crusader's treasure.
As they follow Sir Hugo's clues the children discover a secret passage hidden in the wall shaft which leads them to the much sought after crusader's treasure. They accidentally discover a secret door in the treasure chamber which leads into Blaize Castle but they are again surprised by the gamekeeper, who lock them and Sir Hugo in a dungeon while they themselves start to take the treasure down to a boat. There is a passage from the dungeon to the sea and the tide is rising dangerously.
It seems that Sir Hugo and the children are trapped but as Wilfred plays the pipe which is so irresistible to animal a badger appears and shows them a way out through his ears. They escape, turn the tables on the gamekeepers, who are trying to ship the treasure away, and hand them over to the Police, who by now have been called to the island. By this time Sir Hugo has become very fond of the children and decides to adopt orphan Wilfred and take him to live on the island with him and his beloved animals.
