The character Jubal Harshaw says this line in Stranger. Interestingly, in the "Uncut" version of the book, the character does not speak this line. Many argue that Harshaw is the true main character of the book, as he consistently proffers the philosophies another character must deal with.
Letter to Virginia Woolf (21 January 1926), quoted in Love Letters : A Romantic Treasury (1996) by Rick Smith, p. 78
Sackville-West wrote this letter to Virginia Woolf, with whom she had an affair. Woolf returned her affection in the form of the novel Orlando, which many recognize as a direct outpouring of Woolf's love. The letter also contains the line, "...you have broken down my defences. And I don't really resent it."