Sir Thomas Gascoigne left a vast fortune to his adopted daughter Mary, and her husband Richard Oliver- on condition that Richard change his name to ‘Oliver-Gascoigne’. Richard and Mary had two daughters, Elizabeth and Isabella. Elizabeth married Frederick Trench (2nd Lord Ashtown). Frederick’s grandson William Cosby Trench inherited Castle Oliver from Elizabeth- his step-grandmother. William’s son Walter and his wife Norah inherited the castle, which then passed to Walter’s 2nd wife Lynn Trench (nee Turner). Lynn Trench was the last family member to live at the castle and moved away in 1978.