Prince Philip kept a folder of affair rumors and weighed legal action ...