I spent several weeks visiting England in late August to mid September each year from 1992 to 1996. I'd start with 3 days at the Reading Festival, then go to London for the Notting Hill Carnival. I'd spend a week or so in London before heading to Amsterdam for a few days. While I was in London, I'd occasionally take day trips by train to visit various castles, palaces, historic sites and other places of interest. Many of my day trips were to various places in the south east. Not sure if any of these might interest you as much as they interested me, but I'll throw them out as possible suggestions.
Dover Castle | English Heritage
Walmer Castle and Gardens | English Heritage
Deal Castle | English Heritage
1066 Battle of Hastings, Abbey and Battlefield | English Heritage - There were also a lot of interesting antiques/junk shops in town to browse in when I was there.
Royal Pavilion - The Brighton Marine Palace and Pier is nearby and there's an aquarium next to the pier (which I thought was much more interesting) billed as the world's oldest operating aquarium.
Southsea Castle - Castle / Fort in Portsmouth, Portsmouth - South East England - Portsmouth Historic Dockyard is nearby and has the Royal Navy Submarine Museum and HMS Alliance, The Museum of Naval Firepower, the Royal Marines Museum, the new Mary Rose Museum and world famous ships HMS Victory and HMS Warrior 1860.
Leeds Castle - Castle / Fort in Maidstone, Maidstone - South East England - Urban already mentioned this one, but I'll second the recommendation.
Arundel Castle & Gardens - Castle / Fort in Arundel, Arun - South East England