PHP-list

Posted on 8 June 2012

0


I am using PHP list for a few years now, but it was time for an update. Unfortunately, I needed to send out an email to a bunch of people quickly so I decided to use it once more. I had the address list ready in php list so this would be easy.

Installing

But that was without counting in Murphy’s law. I was busy on maintaining another website too and found an invalid PHP-list installation in the installatron section of DirectAdmin. I deleted it but wish I have never done, as it turned out to be my php-list. Gone… I found the table back in mySQL so I made a new php-list installation, opened the config.php and pointed it to the old database. Fortunately, this did work.

Importing

So next up was re-importing my contacts list. I decided to add surnames and gender with the attribute fields of php-list. I made a csv table with all the information in it and imported it:

1. export csv list from CRM system
2. import in open office: West-Europa (windows-1252/WinLatin1 / puntkomma)
3. replace m by (Sir[spatie]) and v by (Miss[spatie]d
4. save as csv (windows-1252/WinLatin1 / all text with quotes)
5. import to phplist (overwrite existing, make confirmed immediately, receive html)

Works like a charm: now I could use the  [NAME] placeholder to call the data from the name attribute. The only thing I could not figure out is if it is possible to use a if[GENDER]=m then (sir) elseif[GENDER]=f then (miss) else (sir/madam) kind of system. So I just did a find and replace in the original table and imported it again. Unfortunately, until I find out, I can’t use a general salutation for if there was no value in the attribute field. I couldn’t find how to fill all empty fields either.

The nice thing is that you can also use this csv list to invite your contacts on linked-in. Just click add-connections->import and select the list. People that are already invited from your previous import only get an invite-email once (“your contact(s) have already been invited. Please invite different contacts.”) so you’re not bothering anyone.

Sending

The next problem was a server overload.  I found that my server was almost unreachable for 21 minutes when I was sending the batch of 836 (the report states 2322 msgs/hr). So it was taking about 1,5 seconds per message. This is with the MAILQUEUE_BATCH_SIZE set to 0, which means batch processing is disabled and messages are sent out as fast as possible.

I remember I had to set a batch limit of 2500 mails per day, which take one hour for each batch. To do this I should set  MAILQUEUE_BATCH_SIZE to 100 and MAILQUEUE_BATCH_PERIOD to 3600 seconds. This would send a maximum of 2400 messages a day. But, the 100 messages would be fired of in one go every hour, leaving the server unreachable for about 3 minutes every hour. I’m not sure, but it seems like it wants me to keep the ‘process queue’ window open to reload every hour, otherwise no messages will be sent.

So now I’ve set MAILQUEUE_BATCH_SIZE to 300 and MAILQUEUE_BATCH_PERIOD to 3600 seconds, and enabled the experimental MAILQUEUE_AUTOTHROTTLE spread the 300 messages evenly over the 3600 seconds. As far as I can see this also needs me to keep the ‘process queue’ window open, when I close this window it will stop sending messages out. Also in this way, I found that the server was unreachable for a long period of time. So this is not really a solution either…

Why is it that sending a few simple messages is overloading this server? I’ve checked the Resource Usage stats but it shows a CPU usage and memory usage that are way below the limit, you don’t even notice the difference with other days. Also the entry processes only gets up to 12/30. So really, what is going on there? Here and especially here is some more information about sending messages from php-list.

So I found the right way to do this from my shared server: set the MAILQUEUE_THROTTLE to send out one e-mail every 12 seconds and make sure you set the MAILQUEUE_BATCH_SIZE to 0 (off). Now it sends out 286 messages per hour, you don’t have to keep phplist open while sending, you get a nice summary e-mail when it’s done and last but not least: the website is still responsive!

Replies

Now I’ve got a few new messages in my inbox. These are the reactions, autoresponders and bounces.

The replies are sent to the From address that you can manually set in the From Line when you make a message. They require a reaction by a real person so I think it is good to use your personal e-mail address. By the way, I found that you can use the format My Name my@email.com in this From Line. I couldn’t find how to set it as a default.

Autoresponders are the “I am on holidays” type of messages and should also be sent to the From address. They are annoying but hey: you are the one sending out a lot of emails so the least you can do is delete these autoresponses from your mailbox.

The bounces, or  Mail delivery failed messages are of another kind. These are the “550: unknown user” kind of messages which should be send to your php-list address. It shows in the header as return-path, (envelope-from) and errors-to address. This address could be set in the config.php under “Settings for handling bounces” and should NOT be the same as the From address. You do not want people to get unsubscribed from your list, just because they send you a nice reply!

Autoresponse to bounce address

The stupid thing here is that 1 out of five autoresponders seem to send their mail to this address. Is this because it is in the header as return-path (which is different from but might get confused with the reply-to header) or is it the errors-to line? So I went to admin/class.phplistmailer.php (it’s also in lib.php but I left it there) and disabled the errors-to line (make sure you leave the $this->Sender = $GLOBALS[“message_envelope”]; line intact, otherwise the system will use the user@localhost address. Unfortunately: I still got autoresponses (1:15) on the bounce-address, so it must be sent to the return-path or envelope-from address. 

Fake 550 and 554 bounces

So amongst a few true bounces I also got two fake SMTP errors:

554 5.6.0 Invalid content
550 authentication required

I resent the message from my personal email account and now I didn’t get a bounce: which means that somehow the recipients SMTP server recognised the mail the be spam. This can’t be the message itself, as it was the exact same message, so this could be due to a number of factors:

  1. PHP-lists uses my .com address instead of my .nl address;
  2. The return-path and the From address are different;
  3. The PHPlist footer with embedded image gives it all away;
  4. It could be one of these headers:
X-Mailer: phplist v2.10.19
X-Priority: 3
X-MessageID: 33
X-ListMember: name@mail.com
Message-ID: <0b2e53e6d9af5447c3f96868ad3376f5@www.mail.com>

I deleted the X-headers. Then I corrected the Message-ID in admin/phpmailer/class.phpmailer.php line 812 by replacing $this->ServerHostname() by mail.com so it doesn’t show www. It’s a bit of a dirty fix but hey: it works. And I added a line to admin/class.phplistmailer.php with a “User-Agent: Webmail”.

Let’s send it from the .nl address and see what happens next time.

DNSBL

I got one reply that states that the message was found to be spam: the URL is in DNSBL. I resent it from my .nl account and now it’s fine. So let’s find out which DNSBL (DNS blacklist) this is.

Cron-Job

You can have PHP-list to automatically import the bounces from this dedicated mailbox by using a cron-job and setting define (“MANUALLY_PROCESS_BOUNCES”,1) to 0. I did have to set $bounce_mailbox_port = “110/pop3/notls” for it to work with my server. This is my cronjob that runs every sunday at 03.00 (0,3,*,*,0) and sends me the output of the job (if I don’t want that i should add > /dev/null to the end of the job):

/usr/local/bin/php -q /home/<MYDOMAIN>/public_html/mailinglist/admin/index.php -pprocessbounces -c/home/<MYDOMAIN>/public_html/mailinglist/config/config.php.

Bounce count

The standard way PHP-list handles bounces is by marking a user to be unconfirmed after a certain number of bounces. You can set the set the $bounce_unsubscribe_threshold to 0 to have them unconfirmed immediately, but in my opinion a bounce should lead to immediate deletion of the email address from our mailing-list. Otherwise, the unconfirmed email address will still sit in the mailinglist, waiting for you to hit the ‘activate all’ button. This because having a lot of these kind of bounces are making sure you get blacklisted by other mail systems.

Advanced bounce processing

To get PHP-list to really delete users and bounces, I needed the experimental feature from the config.php: advanced bounce processing. I added few bounce-rules, like: 550 -> delete user and bounce. Unfortunately, if the bounce does not fit with this bounce rule the user will still get unconfirmed and will need to be manually deleted.

unconfirmed users

So I would need a ‘all bounces’ bounce rule, but when looking for this I found that from v. 2.11.4 there also is the default blacklist option, which might be a pain in the ass with an all bounces rule. And the other problem with an all bounces rule is that there is a chance that is a “mailbox full” message. That’s the other reason why the few unconfirmed users are not so bad: I get a nice summary e-mail that looks like this:

22 bounces to process:
Report of advanced bounce processing (10 users):
User user@domain.com deleted by bounce rule 2
User: <link to the user detail page>
Rule: <link to the bounce rule page>
// and so on for 9 other users

Below are users who have been marked unconfirmed. The number in [] is their userid (5 users):
otheruser@domain.com [110] (1)

From this e-mail it is easy to get the e-mail addresses of the deleted users and check or delete them in my personal address book. The bounces of unsubscribed users can be read by clicking ‘view bounces’, if they are mailbox full or autoreply messages they can be subscribed again, if they are true ‘mailbox doesn’t exist’ messages they can be completely deleted from the system.

unidentified bounces

The missing seven bounces (22-10-5=7) of this example could not be linked to a user by the system and thus are marked as ‘unidentified’. The problem with unidentified bounces is that there is no way to see them, apart from by accident when choosing ‘delete this bounce and go to the next’ when you are viewing the bounces that are identified through the ‘view bounces’ screen of PHPlist. I was lucky enough to see them once and they all were auto responses that were sent from a general address to envelope-from address (see above). It’s possible to delete them with one touch on the button, and it is possible to config phplist to leave those emails in the imap inbox. So that could be an alternative.

Double X-header

From one server I received this e-mail:

    SMTP error from remote mail server after end of data:
    host mail.ish.nl [195.8.208.126]: 554 5.7.7 too many equal X-headers (X-Mailer:)

This is correct: there are two x-headers:

X-Mailer: PHPMailer [version 1.73]
X-Mailer: phplist v2.10.18

So the interesting question: can we fix this? The answer might be to update PHPMailer. But there also is a quick fix: The first X-mailer is defined in admin/phpmailer/class.phpmailer.php, so I deleted this line:

$result .= $this->HeaderLine(“X-Mailer”, “PHPMailer [version ” . $this->Version . “]”);

Now the e-mail only shows the X-Mailer line from admin/class.phpmailer.php.

Alternatives

There are a few great alternatives, like Benchmark e-mail and iContact (about $20 per month for 1000 subscribers). They even have a social media department added to the e-mailing options. I’ve tested both Benchmark and iContact, and had to find that the free Benchmark trial is limited to only 250 subscribers, so I could not really test it. The iContact trial is limited to 500 users, so still not very useful, but I got to test it by importing my .csv file. I found that the first iContact mail was not trusted by hotmail, while the Benchmark email was. This seems to be a problem with one of the template I used, I did not get this problem with my own template. Also, unsubscribe and sharing options of the Benchmark system are better. It all works very nice, with big buttons and design help, which is a breeze compared to PHP-list. After sending the mail, also feedback is great! You get nice statistics on the dashboard about the amount of emails that have been opened, bounced, marked as spam and unsubscribed (in this example, there was no link to click in the e-mail):

In detail view you can even see what type of bounces etc:

All in all, this is pretty great! Note that the ‘opened’ stats come from clicked html links or loaded images. If someone just displays the e-mail without loading the image, it is not registered as an ‘open’.

Advertisements
Posted in: CMS