About a mile from Banstead stands the old ivy-covered brick mansion known as The Oaks, surrounded by magnificent trees. In 1788 the fee simple of the estate was purchased by the Earl of Derby, who greatly improved the building, adding the circular towers. This hospitable nobleman kept open house at The Oaks, from which is derived the name of the famous race for 4,500 sovereigns, established in 1779, for three-year-old fillies, and run at the Epsom Summer Meeting on the Friday following the Derby After Lord Derby died in 1834, Sir Charles Grey was one of the tenants of The Oaks and in recent years it has been occupied by the widow of Mr. H. Berkeley James. About two acres of common land were enclosed when the park was laid out.
source: The Queen's London : a Pictorial and Descriptive Record of the Streets, Buildings, Parks and Scenery of the Great Metropolis, 1896