When I left my hedge fund job a few years ago, I was sad to leave my colleagues and my plush office in Mayfair.