HyperNews: Source: Installation: Email Gateway

This page is for discussing gateways between HyperNews and Email, although most of this is about how to configure HyperNews to enable incoming email. Also see the HyperNews Usenet Gateway page.

HyperNews can already send out email notification to subscribers when something has been posted. See the Instructions for Subscribing. The 'sendmail' program is used for this purpose since it takes care of alot of concerns we would have, such as handling rejection, queuing for retries, etc.

Enabling Outgoing Email

(Need more about configuring outgoing email...)

To enable outgoing email, you might have to configure sendmail to allow your server process to be a trusted user (See the sendmail.cf T option). This is because HyperNews wants to use the sendmail -f command line option to say that it is Owner-HyperNews sending a message on behalf of the author. Any mailing list software would want to do the same.

HyperNews is configured by default to use a set of options with sendmail that should probably work on your system too. But every sendmail configuration tends to be different, and different sendmail versions have different restrictions and/or bugs.

In some sendmail versions/configurations, if there is a failure to deliver a message to any local user, mail to all recipients fails. With the -oem option, at least the error is mailed to the 'sender'. Perhaps immediate queueing of the message with -odq will avoid this problem. Hints from sendmail experts appreciated here.

Mailing to HyperNews

If HyperNews only mailed out messages that were posted, readers could not reply directly to the email they received, and instead, they would have to use a web browser to visit the HyperNews page and add their message that way. But HyperNews supports both outgoing and incoming email. That is, HyperNews has the ability to receive an incoming mail message and post it to the appropriate place, in addition to sending out postings in email to subscribers. This incoming email capability is disabled by default, but it seems to be safe to use. See instructions below for enabling it.

There are two ways to reply to a message in most mail programs. Either reply to just the sender of the message or reply to the sender and all recipients of the message. First a little background on how email works with HyperNews.

When HyperNews sends out email, the sender is always the author, and is identified by the "From:" header line. One exception is when the author is unknown; in that case, "Owner-HyperNews" is used since mail programs require that there be some sender.

[BTW, the real recipients of the message are those who subscribed to receive the notification - they are not listed visibly in the outgoing email because they are given in the Bcc field. The only visible recipient is 'HyperNews' (specified by the $mail_to variable). When HyperNews is configured to receive incoming email, this special 'HyperNews' address is an email alias that has a program (mail-incoming.pl) attached to it that posts the message to a HyperNews forum.]

So if you receive email from HyperNews you can reply directly to the author in which case no one else will see it. Or you can reply to the author and all the recipients (i.e. just 'HyperNews'), and all the subscribers will see it. This is done typically with a command called "Reply to All" in your mail program, but check that it does the right thing. Of course, you can mail your reply just to HyperNews too; the author will probably also get the email indirectly via HyperNews.

There is a way to configure HyperNews so it will include a "Reply-to" line in outgoing messages. If you (the administrator) do this, then all email replies will typically be sent to whatever is specified in the Reply-to line, such as "HyperNews". Some mailing lists like do this to encourage all replies to be sent to the list, but I find it very annoying since sometimes I know I just want to send my reply to the author and there is no way to do so without manually copying and pasting the author's address. Furthermore, people accidently post to the list when they intended to reply only to the sender.

How does a message get posted to the right forum?

There are a few other things you should know about replying via email. HyperNews has to somehow figure out what you are replying to so your message can be added to the right place in the right forum in an appropriately threaded way. To tell HyperNews what you are replying to, your mail program must include an "In-Reply-To:" header line that specifies the message ID of the message you are replying to. Some mail programs (e.g. Eudora Pro 3.x) will automatically create an In-Reply-To header when you reply (though this may not be visible in your message editor). In this case, you won't have to do anything special to ensure that your reply is posted to the right place.

However, other mail programs (e.g. older versions of Eudora) do not include an In-Reply-To header. If your reply does not get posted properly, the cause might be that your mail program has not created this header. You can try to post a message on the HyperNews Test Forum and reply to it by email to see if your mail program does the right thing.

Outgoing mail from HyperNews includes, at the top of the body, exactly the In-Reply-To header line that should appear in the header of your reply. If you cannot add it to the header section (some mail programs don't let you add header lines) then you have one more option: you can include it as the very first line of your reply where HyperNews will look for it. HyperNews version 1.9.7 will also ignore leading spaces and '>' chars on that line, but for older versions, be sure to start the "In-Reply-To" at the beginning of the line.

Another issue is the title of your message, which is specified by the "Subject" header line. The default Subject line that many mail programs produce when you reply to a message is not terribly informative (e.g. "Re: What he said"), especially when viewed in a HyperNews forum, so you are encouraged to change the Subject line to something specific to your message.

The In-Reply-To header works fine if you are replying to a message, but what if you want to post a new message to a forum? In this case, it is not a reply to anything. But HyperNews lets you post a new message to a forum by including a "Newsgroups:" header line that specifies the URL of the forum you are posting to. E.g.

Newsgroups: http://www.hypernews.org/HyperNews/get/hypernews/test.html

This line, either in the header section or in the first line of the body, will cause HyperNews to post the message to that forum. If there is an "In-Reply-To:" header as well, then that is used first and the Newsgroups line is basically ignored.

For each mail message received by the HyperNews email gateway, if it is not a reply to a preexisting thread and does not have a "Newsgroups" header, HyperNews returns a "HyperNews Posting Error" message indicating that it doesn't know where to post the new message.

There is one other way to post a message to a forum that works for both replies and new messages. If you configure your forum to use an email address that is specific to the forum, then you can also mail directly to that address. See below for how to do this.

If the HyperNews site you want to post a message (or reply) to is restricted to members-only, then you must make sure that the email address used to identify you as the author of your message is also a member of the HyperNews site. You can check this by using the Inquire About Member option on the membership form for the site. The upper and lower case characters in the email addresses are significant. (This will be fixed in a future version.)

Why do I still get email?

If you are a normal member or user, you are probably subscribed to the forum or to one of the messages in the forum. You may be subscribed automatically to get replies to your own messages, depending on the HyperNews configuration.

If you are the "owner" of a forum you will be subscribed by default. This means you get email notification for every posting. If you don't want that, you can explicitly unsubscribe.

If you are an administrator, you might be getting all email notifications because you have been named in the "HyperNews" email alias. Your version of sendmail might require that email be sent to some explicit address and that address might have to be for a user who exists (such as you) in order for email to be sent to anyone. If you want to disable all email, that would be one rather drastic solution to the problem. Another solution is to make the $mail_to address be an alias that sends the mail to /dev/null. The following works with sendmail V8.7.5.

  HyperNews: "| cat > /dev/null"

or just try:

  HyperNews: /dev/null

Another way to avoid getting email from the $mail_to address is to enable incoming email, as described in the next section.

Don't swap the $mail_to and $mail_from addresses in hnrc. This may appear to work, but you will be setting yourself up for more confusion.

Enabling Incoming Email

The mail-incoming feature is out of beta testing - but please be cautious in its use anyway. We do attempt to avoid mail loops and spurious administrative postings, but there are probably more ways these could occur.

To use this feature, you need to do a few things in addition to the normal installation of HyperNews.

Do setup via the web using setup-form.pl or setup-short-form.pl, or if you run setup.pl via the command line, manually set the values of $ENV{SERVER_HOST} and $ENV{SERVER_PORT} in the hnrc file. Also, make sure $hnprefix is correct.
Enable incoming email in hnrc by setting $allowEmailIn to 1. It is 0 by default.
Create an administrator for use only by the mail-incoming program. This admin will only authenticate for the user, but nothing else. Set values of $mail_admin and $mail_admin_password in hnrc file. (Also edit the member info for the $mail_admin and set its password to the same thing in $mail_admin_password.) Add the $mail_admin to the @hnAdmin list. Run setup.pl after changing the @hnAdmin list.
Add an alias to your /usr/lib/aliases file (or whatever it is called) that looks like:
```
  HyperNews: "| /full/path/to/HyperNews/.scripts/mail-incoming.pl"
```
The "HyperNews" alias is whatever is in your $mail_to variable. You should probably also have an Owner-HyperNews alias that points to a real user such as you. Make sure mail-incoming.pl is executable by the user:group running sendmail or procmail, typically root:bin.
If you have multiple installations, you will want to have one HyperNews alias for each, but they could share a common Owner-HyperNews.
If you do not have access to your system's alias file, perhaps you can pipe mail to the mail-incoming.pl program by some other means.
- If you are using procmail, the following lines in your .procmailrc file will copy all incoming mail that is to your "HyperNews" address to the mail-incoming.pl program.
```
# All hypernews mail
:0 c
* ^TO hypernews
| /full/path/to/HyperNews/.scripts/mail-incoming.pl
```
- If you get mail via a POP server, try the batch-fetch-pop-mail.pl program. Variants of this for imap and nntp are still needed.
You can also set up a list per forum (or base article), but you don't have to. See the next section for details.

The astute reader might notice that the HyperNews address is both the sender and the receiver. This is so users can reply easily to the sender. But this also sets us up for a potential mail loop. We stop the loop by checking if the message ID has already been seen. (But there are other kinds of loops that have new message IDs. Maybe we need to use Majordomo's resend to change the address on the way out or in.)

A loop can occur if the .midDB directory is not correct for some reason, because then HyperNews can't tell that you have already posted a message. It checks whether the message ID has already been seen, and if not, then it assumes this is a new message and proceeds to post it. Make sure you have no filter on your mail that changes the message ID before mail-incoming.pl sees it. That would cause a loop.

We also add a special X-Hn-Loop header to outgoing email, and check for that header in incoming email. But some gateways may remove unknown headers, so the message ID check is also useful.

Be sure to test your incoming email gateway, and be around when you do test it so you can quickly disable a mail loop. You can disable a mail loop by turning off all handling of incoming email: move mail-incoming.pl to somewhere else, or edit mail-incoming.pl to add "exit 0;", or edit your hnrc to set "$allowEmailIn = 0;", or change the mail alias that forwards mail to mail-incoming.pl. Take your pick, but be prepared to do it quick.

Setting up a Mailing List per Forum

In order to allow users to add new base messages to a forum without the cumbersome "Newsgroups" syntax described above, a mail alias can be setup associated with a specific forum.

To create a forum specific email gateway:

Edit the <forum>.html,urc file manually or use the SECURED/edit-article.pl/<forumname>.html web form to set the Email List Address to the forum-specific alias you are going to use.
For example, if the base article / forum were called "books" and you wanted any messages sent to "books@domain.com" to be added directly to the forum, you would specify "books@domain.com" in the 'books.html,urc' file (or use the web form '../edit-article.pl/books.html').
Create a mail alias in your aliases file ('/usr/lib/aliases' or '.qmail-' or procmail configuration, or similar) which runs the mail-incoming.pl script with a '-f /<forum>' argument.
Using the example ("books") above, the mail alias line in /etc/aliases would be:
```
 books: "| /full/path/to/HyperNews/.scripts/mail-incoming.pl -f /books"
```
If you are using qmail, the alias might appear in the file .qmail-domain-books or .qmail-books and contain only:
```
 | /full/path/to/HyperNews/.scripts/mail-incoming.pl -f /books
```
So you could add similar lines for specific forums just as above.
Note that the <forum> must have a leading '/' and not a trailing '.html'. And there is a space after the '-f'.

Troubleshooting the email gateway for HyperNews

"HyperNews Posting Error"

If messages are being returned to the sender with this message, check the <forum>.html,urc file for the forum/article and verify that the mail alias is functioning and contains the proper -f /<forum> command.

Verify that the file permissions on the mail-incoming script allow the mail proc to run it.

Others...

Still To Do

If you start work on any of these, let us know.

Read and process the date header. - the email date format is slightly different from HTTP format.
Handle To and Cc lines to add them as public subscribers. (store in Cc field in urc file?)
Support MIME and attachments. Put attachments in subdirectory of message, accessible via HN script.
Refuse to post or delay posting a reply within a short time (1 minute?) of the date the original was posted. This would be another loop stopper. But some loops can take longer. Also, some non-loops might take shorter time and still be OK. Better to be safe than sorry.
Need to better support multiple email aliases for members, and finish support for case-insensitive email addresses and userids.

References

Project Cyrus Home page has lots of references to mail related resources.
Hypermail converts email to HTML with indexes.
MHonArc is one of the better MIME mail handlers, written in Perl.
CoCoBoard allows a simple and inituitive email to html interface.
MHTML group - an IETF effort to develop a standard for sending HTML-formatted e-mail. I believe the group is misnamed because it is really about URLs in MIME packages.

Daniel LaLiberte (liberte@hypernews.org)

Last modified: Fri Feb 19 23:23:01 EST 1999