Because naturally one needs an entire mini castle to take tea in! Anyway, after Victoria was done with it, the fort opened to the public—before going private again in 1910 when it was used by Sir Malcolm Murray as a "grace and favor residence" (the term for homes owned by the monarch and lent to their friends rent-free).