Born  in 1835, Arthur Hayter was the only son of Sir William Hayter, 1st Baronet.  He was educated at Eton and Brasenose College, Oxford and later joined the Grenadier Guards.  Hayter sat as Member of Parliament for 30 years for the Liberal Party, and after succeeding his father in the baronetcy in 1878, he served under William Gladstone as Lord of the Treasury and as Financial Secretary to the War Office from 1882 to 1885.  Hayter chaired the Public Accounts Committee from 1901 to 1905 and was sworn to the Privy Council in 1901.  In 1906, he was raised to the peerage as Baron Haversham of Bracknell.

Lord Haversham married Henrietta Hope and they lived at South Hill Park.  They did not have any children and the baronetcy and barony became extinct on Lord Haversham's death in May 1917, aged 81.

After the death of Lady Haversham in 1929, South Hill Park passed to Major Rickman, OBE, Lady Haversham's nephew who was the last person to both own and live in South Hill Park as one house.