A partial archive of meta.discourse.org as of Tuesday July 18, 2017.

# “Broken” quoting of/into email

hudrob

Hi all,

I wasn’t sure which category to put this in - I’m happy to have it moved.

We have a community of around 400 users at present. We migrated from a purely mailing-list format (an old version of mailman, hacked together with OpenLDAP for user authentication, etc), and presented the result back to our membership as “mail-enabled web forums” rather than a straight-up mailing list replacement.

Many of our users still use the mail interface as their primary interaction with Discourse, and a vocal minority are quite, well, vocal, about some issues they see with how Discourse treats email.

The primary concerns they seem to have is that they believe that Discourse breaks quoting and threading of replies. I believe “mailing list mode” largely addresses the latter - so it’s quoting I am focusing on here.

I believe it is possible to “prompt” Discourse to correctly quote emails (or parts thereof) by introducing double carriage returns (I assume this is the trick) between quoted text and comments made inline (seems to only matter after the quoted text, not after the comment before the next quoted section). However, even in my own experience, this seems a bit hit and miss - when doing it myself, and doing it in what I believe is a consistent fashion, sometimes my quotations and comments are nicely separated, at other times they seem to just run into each other (be that within quotes our outside them)

What’s the cause? I’m not sure. I saw this thread:

Which mentions that support for “>” characters indicating quoting was added to Discourse.

I’ve tested doing that, and it seems to work consistently.

What doesn’t seem to work consistently is other characters that get used to indicate quoting - Gmail in particular in my case, which seems to use a vertical line in front of the quoted text.

So, the question is - is there a “secret sauce” to getting Discourse to recognise the quoted text? Can characters other than > be used to indicate quoting - and how many email clients (be it gmail web client or “thick” clients" do we think it’s reasonable for Discourse to be able to “recognise” quoting behaviour from?

codinghorror

Why is recognizing quoted text important? In terms of email threading, that is determined entirely by mail headers set when someone presses “reply” on a specific post, or replies to an email (same thing, that email represents one post, replying to the email is mechanically the same as replying to the post).

Can you provide some specific examples?

hudrob

It is important when people reply inline - which our membership is particularly fond of doing, particularly in email. I’ve replied to your post in a similar fashion to what our members want to be able to do reliably (I’ve used the web interface to do so - it works consistently here, they wish to do it by email).

Threading, as I mentioned, I believe is largely dealt with by mailing list mode, which I believe we’ve worked to make the default for all of our users.

I believe that getting quoting “fixed” will help us a lot in terms of member satisfaction (and potentially retention). It will also let those of us running the org to focus on other things.

codinghorror

I am still unclear what exactly is broken about it – I am afraid that reply does not help me at all.

mpalmer

People want to be able to inline-quote via e-mail, with a similar end result as via the web interface (highlight, click “quote”). It’s really unlikely we’d be able to get the results quite the same (extracting the [quote] metadata would be fiendishly tricky), but we could at least recognise common HTML quoting styles, and transmogrify them into the equivalent of how > gets handled in markdown:

Why is recognizing quoted text important?

hudrob

Hi Jeff,

The issue is that when replying via email, the Discourse system seems to merge in-line comments with the quoted text. If you provide me with an email, I’m sure I can provide you with some examples (what was sent verses what Discourse recognised)

That’s exactly what we’re looking for. This would make for some very happy users within my organisation.

codinghorror

This is actually what I was asking you for – I think you would need to run a mailing list mode only site to really see this and have many real world examples of it?

hudrob

Provide me with an email address, and I shall forward you some examples.

And yes, the majority of our users interact via email.

hudrob

What got sent to Discourse. Note the last few lines in particular.

> | Yet more of what Karl didn't say
> | Yep, you guessed it, more of what Karl didn't say
> | More of what Karl didn't say
>
Single CR before, single CR after.

What Discourse displayed:

As you can see, the “Single CR before, single CR after.” part is contained within a quote, when it should be stand-alone.

LeoMcA

Yes, turn on the incoming email prefer html setting.

hudrob

Thanks. Checking that now. From what I can see, it means that Discourse obeys the HMTL formatting that the email client sends. I guess it requires a level of trust that your users don’t do dumb things…

fefrei

As far as I understand, the setting is for more innocent:

Discourse can use both incoming text-only and HTML-only mails. Text-only mails are simply used as raw Markdown (which does explain the quoting behavior). HTML-only mails are sent through a HTML-to-Markdown-converter, and the result is once again used as the raw Markdown of the user’s post.

When receiving mails which have both HTML and text (which is quite common), Discourse normally uses the text version and ignores the HTML part. This setting changes this behavior to instead ignore the text part and treat the mail like a HTML-only mail

codinghorror

The guideline there is that users must put a blank line between the thing they are quoting and the response. That seems simple enough.

hudrob

H Jeff,

[codinghorror](https://meta.discourse.org/u/codinghorror) [Jeff Atwood](https://meta.discourse.org/u/codinghorror) co-founder July 6 > hudrob: > Note the last few lines in particular.

The guideline there is that users must put a blank line between the thing they are quoting and the response. That seems simple enough.

It would seems so, yes - that would be the double carriage return I mentioned earlier. But it doesn’t seem to work reliably.

Sometimes we see it work. Other times, we see the comment merged into the quote. A few times, we’ve seen an entire message after “Hi guys” vanish entirely.

Let me find those example emails…

hudrob
[fefrei](https://meta.discourse.org/u/fefrei) [Felix Freiberger](https://meta.discourse.org/u/fefrei) Regular July 6 As far as I understand, the setting is for more innocent:

Surely it’s all innocent.

Discourse can use both incoming text-only and HTML-only mails. Text-only mails are simply used as raw Markdown (which does explain the quoting behavior). HTML-only mails are sent through a HTML-to-Markdown-converter, and the result is once again used as the raw Markdown of the user’s post.

That is useful info, thank you.

When receiving mails which have both HTML and text (which is quite common), Discourse normally uses the text version and ignores the HTML part. This setting changes this behavior to instead ignore the text part and treat the mail like a HTML-only mail

That would explain my testing.

When I sent something which had some text bolded when the setting was off, Discourse converted my bold to italics.

When I turned the setting on, the bolding was honoured. The issue, unfortunately, is that it also stripped out the quoted text entirely.

Thus, I have to assume not a solution in this case.

LeoMcA

Do you have that raw email? In my own testing I never saw this happen, unless the quote wasn’t followed by any reply, but I would consider that a feature, not a bug.

hudrob

Here is an example.

Email sent:

Discourse displayed:

As you can see, a blank line was inserted between the quoted text and the reply - yet Discourse picked both up as the same quote.

LeoMcA

Often it’ll look like a blank line has been inserted between the quoted text and the reply, but in the plaintext email your client produces there isn’t one - this is why the raw email is useful in debugging these problems because it’ll show exactly what Discourse received and processed.

I know for sure Gmail has this behaviour, and the solution (if you don’t turn on incoming email prefer html) is to educate your users hit return twice a second time after a quote. That screenshot doesn’t look like Gmail to me, so I’m not sure if that’ll work.