The character Jubal Harshaw says this line in Stranger. Interestingly, in the "Uncut" version of the book, the character does not speak this line. Many argue that Harshaw is the true main character of the book, as he consistently proffers the philosophies another character must deal with.
The author's Journal, excerpted from The Thoreau you don't Know... (2009) by Robert Sullivan, page 87
Deeply in love with a woman named Ellen, whom his own brother John also loved, Henry David Thoreau penned this line in his journal in 1839. He later proposed to her in 1841, as did his brother, and both were rejected.