Build advanced email capabilities with IMAP and PHP-PHP Tutorial-php.cn

What you will create

In this tutorial, I'll show you some real-world examples of how to use PHP and IMAP to build new functionality for managing email that the big email providers haven't built for us yet.

My interest in this started in 2010, when I wrote Twelve Gmail Ideas to Revolutionize Email (Again) , but most of the ideas I wanted were still out of reach. Despite its importance, email as an application has been fairly slow to innovate.

We are being inundated with email, and managing your inbox can still be a heavy burden. Mail services and customers have done little to help us in this regard. Most of the emails we receive are sent by machines rather than people, yet we have to deal with them all individually.

Analysis of my own email shows that I receive emails from over 230 automated senders, with the number of actual senders being much lower. I'm tired of building filters and filling out countless unsubscribe forms in Gmail. I want to have more control over my email management and simplify my life.

Finally, over the past year, I decided to build the functionality I needed. The result is Simplify Email (SE), a small web app you can host yourself that offers a variety of cool new email features, all of which you can check out on the project website.

The cool thing about SE is that it's a platform for reading, analyzing, routing and managing email - the possibilities abound. Simplified Email is essentially a programmable playground for "hacking" your own email.

I'll walk you through the code for three examples of email processing in SE using PHP, IMAP, and MySQL:

Check your inbox and filter emails
Implement whitelist challenge for unknown senders
Report Unanswered Email

This tutorial will definitely give you a head start on writing IMAP code in PHP. But you can also use the Simplify Email codebase directly. You can buy the code for as little as $10, and there is an older open source version (which lacks some of the features we describe below). Installation instructions for typical Linux configurations are provided. I also offer pre-installed images at Digital Ocean for $25 and offer handheld valet installation. SE is written in PHP within the Yii framework.

Please note that unless you compile a secure IMAP library for PHP, you will not be able to access most email servers from your local development machine. This is one of the reasons I encourage people to run Simplify Email as a droplet in Digital Ocean. There are also some tips for keeping your Google account secure when you get in via IMAP.

Use IMAP

How to simplify how email works

With SE, you can continue to use the email client of your choice on the web and mobile devices. You don't need to change any applications or personal habits. SE accesses your email account behind the scenes via IMAP; as an intelligent personal assistant, SE preprocesses your email, moving messages to the appropriate location based on everything you tell it.

When a message arrives from a familiar sender, SE moves it to the folder you specify. When an unknown sender receives a message for the first time, it is moved to a review folder.

Every few hours (or as often as you choose), SE will send you a summary of where it moved messages and which messages are under review. Note that the audit folder contains a link to the training sender, making it very easy to train the SE over time.

Build advanced email capabilities with IMAP and PHP

You can browse your review folder at any time - no need to wait for summaries to arrive. But the advantage of SE is that you no longer need to browse your folders; you can just read the summary to view received emails and train new senders.

1. Check your inbox and filter messages

SE Use multiple cron tasks to run in the background of the server. Each is called from DaemonController.php.

The first one, processInbox, is called frequently and needs to be done quickly - its job is to sift through emails and move them out of the inbox and into categorized folders as quickly as possible, called filtering folder.

The second one, processFiltering, is more processing intensive and performs more in-depth operations on the email, ultimately moving the message to its final destination.

ProcessInbox method

cron tasks call processInbox regularly:

public function actionInbox() {
    
    // moves inbox messages to @filtering
    // runs frequently 
    $r = new Remote();
    $r->processInbox();	  
    
}

Copy after login

Build advanced email capabilities with IMAP and PHP

For each account, we decrypt your email credentials and then use imap_open to create an IMAP stream to your inbox folder:

public function open($account_id, $mailbox='',$options=NULL) {
  // opens folder in an IMAP account
  $account = Account::model()->findByPk($account_id);
  $this->hostname = $account->address;
  if (!stristr($this->hostname,'{'))
    $this->hostname = '{'.$this->hostname.'}';
  $cred = Account::model()->getCredentials($account->cred);
  if ($account->provider == Account::PROVIDER_ICLOUD) {
    // icloud accepts only name part of mailbox e.g. stevejobs vs. stevejobs@icloud.com
    $temp = explode('@',$cred[0]);
    $cred[0]=$temp[0];
  }
  $this->stream = imap_open($this->hostname.$mailbox,$cred[0],$cred[1],$options,1) or die('Cannot connect to mail server - account_id:'.$account_id .' '.print_r(imap_errors()));
}

Copy after login

In processInbox we use the PHP library functions imap_search and imap_fetch_overview to retrieve the message array:

// lookup folder_id of this account's INBOX
$folder_id = Folder::model()->lookup($account_id,$this->path_inbox);
$this->open($account_id,$this->path_inbox);
$cnt=0;
$message_limit= 50; // break after n messages to prevent timeout
echo 'Sort since: '.date("j F Y",$tstamp);           
// imap_search date format 30 November 2013
 $recent_messages = @imap_search($this->stream, 'SINCE "'.date("j F Y",$tstamp).'"',SE_UID); 
 if ($recent_messages===false) continue; // to do - continue into next account
 $result = imap_fetch_overview($this->stream, implode(',',array_slice($recent_messages,0,$message_limit)),FT_UID);

Copy after login

Then we process the array of messages in the inbox:

foreach ($result as $item) {         
   if (!$this->checkExecutionTime($time_start)) break;
   // get msg header and stream uid
   $msg = $this->parseHeader($item);

Copy after login

This is an adapted version of the publicly available IMAP header parsing code that collects additional information needed by the SE to complete various tasks. Basically, it uses imap_rfc822_parse_adrlist to determine the recipient information, message ID, subject, and timestamp (or sender information when scanning the Sent folder):

  public function parseHeader($header) {
    // parses header object returned from imap_fetch_overview    
    if (!isset($header->from)) {
      return false;
    } else {
      $from_arr = imap_rfc822_parse_adrlist($header->from,'gmail.com');
      $fi = $from_arr[0];
      $msg = array(
        "uid" => (isset($header->uid))
              ? $header->uid : 0,
         "personal" => (isset($fi->personal))
            ? @imap_utf8($fi->personal) : "",
          "email" => (isset($fi->mailbox) && isset($fi->host))
              ? $fi->mailbox . "@" . $fi->host : "",
          "mailbox" => (isset($fi->mailbox))
            ? $fi->mailbox : "",
          "host" => (isset($fi->host))
            ? $fi->host : "",
          "subject" => (isset($header->subject))
              ? @imap_utf8($header->subject) : "",
          "message_id" => (isset($header->message_id))
                ? $header->message_id : "",
          "in_reply_to" => (isset($header->in_reply_to))
                    ? $header->in_reply_to : "",
          "udate" => (isset($header->udate))
              ? $header->udate : 0,
          "date_str" => (isset($header->date))
              ? $header->date : ""
      );    
      // handles fetch with uid and rfc header parsing
      if ($msg['udate']==0 && isset($header->date)) {
          $msg['udate']=strtotime($header->date);
      }
      $msg['rx_email']='';        
      $msg['rx_personal']='';
        $msg['rx_mailbox']='';
        $msg['rx_host']='';
      if (isset($header->to)) {        
        $to_arr = imap_rfc822_parse_adrlist($header->to,'gmail.com');
        $to_info = $to_arr[0];
        if (isset($to_info->mailbox) && isset($to_info->host)) {
          $msg['rx_email']=$to_info->mailbox.'@'.$to_info->host;
        }
        if (isset($to_info->personal))
          $msg['rx_personal']=$to_info->personal;
        if (isset($to_info->mailbox))
          $msg['rx_mailbox']=$to_info->mailbox;
        if (isset($to_info->host))
          $msg['rx_host']=$to_info->host;
      }
      return $msg;
    }
  }

Copy after login

We create records in the database for the sender and mail envelope:

   // skip any system messages
   if ($msg['email']==$system_email) continue;
    // if udate is too old, skip msg
    if (time()-$msg['udate']>$this->scan_seconds) continue; // skip msg
    // default action
     $action = self::ACTION_MOVE_FILTERED;
     $isNew = $s->isNew($account_id,$msg["email"]);
    // look up sender, if new, create them
    $sender_id = $s->add($user_id,$account_id,$msg["personal"], $msg["mailbox"], $msg["host"],0);                       
    $sender = Sender::model()->findByPk($sender_id);
     // create a message in db if needed
     $message_id = $m->add($user_id,$account_id,0,$sender_id,$msg['message_id'],$msg['subject'],$msg['udate'],$msg['in_reply_to']);        
        $message = Message::model()->findByPk($message_id);

Copy after login

If the sender is new (unknown) to us, we will send a whitelist challenge email (we discuss whitelist challenges in detail in the next section below):

if ($isNew) {
        $this->challengeSender($user_id,$account_id,$sender,$message);
      }

Copy after login

Next, we determine whether the user may have dragged a message from another folder back to the inbox - intending to train it on drag and drop. If so, we'll set this sender's training to the inbox. In other words, next time we only want to route mail from this sender to the inbox:

         if ($message['status'] == Message::STATUS_FILTERED ||
   	    $message['status'] == Message::STATUS_REVIEW ||
   	    ($message['status'] == Message::STATUS_TRAINED && $message['folder_id'] <> $folder_id) ||
   	    ($message['status'] == Message::STATUS_ROUTED && $message['folder_id'] <> $folder_id))
   	    {
     	  // then it's a training
   	    $action = self::ACTION_TRAIN_INBOX;     	    
   	  } else if (($message['status'] == Message::STATUS_TRAINED || $message['status'] == Message::STATUS_ROUTED) && $message['folder_id'] == $folder_id) {
   	    // if trained already or routed to inbox already, skip it
   	    $action = self::ACTION_SKIP;  
   	    echo 'Trained previously, skip ';lb();   	  
   	    continue;  
   	  }

Copy after login

If not, we will prepare the message to be moved to the Filter folder for further processing. First, we may send a notification to the user's phone if the notification's sender matches or a keyword matches (and it's not a quiet time):

   if ($action == self::ACTION_MOVE_FILTERED) {
     $cnt+=1;         
     if ($sender->exclude_quiet_hours == Sender::EQH_YES or !$this->isQuietHours($user_id)) {
       // send smartphone notifications based on sender
       if ($sender->alert==Sender::ALERT_YES) {
         $this->notify($sender,$message,Monitor::NOTIFY_SENDER);
       }
       // send notifications based on keywords
       if (AlertKeyword::model()->scan($msg)) {
         $this->notify($sender,$message,Monitor::NOTIFY_KEYWORD);
       }               
     }               
     // move imap msg to +Filtering
     echo 'Moving to +Filtering';lb();
     //$result = @imap_mail_move($this->stream,$msg['uid'],$this->path_filtering,CP_UID);
     $result = $this->messageMoveHandler($msg['uid'],$this->path_filtering,false);             
     if ($result) {
       echo 'moved<br />';
       $m->setStatus($message_id,Message::STATUS_FILTERED);
     }
   }

Copy after login

If a message is dragged to the inbox, then we will update our training settings:

else if ($action == self::ACTION_TRAIN_INBOX) {
     // set sender folder_id to inbox
     echo 'Train to Inbox';lb();
     $m->setStatus($message_id,Message::STATUS_TRAINED);
     // only train sender when message is newer than last setting
     if ($msg['udate']>=$sender['last_trained']) {
       $s->setFolder($sender_id,$folder_id);
     } 
   }

Copy after login

ProcessFiltering method

The secondary processing method is called processFiltering and is also in DaemonController.php. It does the more time-consuming job of moving messages to the appropriate folders:

public function actionIndex()
{
  // processes messages in @Filtering to appropriate folders
  $r = new Remote();
  $r->processFiltering();
  // Record timestamp of cronjob for monitoring
  $file = file_put_contents('./protected/runtime/cronstamp.txt',time(),FILE_USE_INCLUDE_PATH);	  
}

Copy after login

This method opens your email account to search for recent messages and collect data about them. It also uses imap_search, imap_fetch_overview and parseHeader:

$tstamp = time()-(7*24*60*60); // 7 days ago
$recent_messages = @imap_search($this->stream, 'SINCE "'.date("j F Y",$tstamp).'"',SE_UID);
if ($recent_messages===false) continue; // to do - continue into next account
$result = imap_fetch_overview($this->stream, implode(',',array_slice($recent_messages,0,$message_limit)),FT_UID);
foreach ($result as $item) {         
      $cnt+=1;
      if (!$this->checkExecutionTime($time_start)) break;
      // get msg header and stream uid
      $msg = $this->parseHeader($item);

Copy after login

The main processing loop for each message in the filter folder is very detailed. First we look at the recipient address, since SE allows people to train folders by recipient address, for example a message sent to the happyvegetarian.com domain will go to the veggie folder:

 // Set the default action to move to the review folder
 $action = self::ACTION_MOVE_REVIEW;
 $destination_folder =0;
 // look up & create recipient
 $recipient_id = $r->add($user_id,$account_id,$msg['rx_email'],0);
 $routeByRx = $this->routeByRecipient($recipient_id);
 if ($routeByRx!==false) {
  $action = $routeByRx->action;
  $destination_folder = $routeByRx->destination_folder;
 }

Copy after login

We then look up the sender and create a new record in the database (if necessary). If training exists for the sender, we can set the target folder:

 // look up sender, if new, create them
 $sender_id = $s->add($user_id,$account_id,$msg["personal"], $msg["mailbox"], $msg["host"],0);                       
 $sender = Sender::model()->findByPk($sender_id);
 // if sender destination known, route to folder
 if ($destination_folder ==0 && $sender['folder_id'] > 0) {
   $action = self::ACTION_ROUTE_FOLDER;  
   $destination_folder = $sender['folder_id'];      
 }

Copy after login

If the untrained (new) sender has authenticated themselves via a whitelist challenge (which we discuss in the next section below), then we will route this message to the inbox:

// whitelist verified senders go to inbox
 if ($sender->is_verified==1 && $sender['folder_id'] ==0 && UserSetting::model()->useWhitelisting($user_id)) {
   // place message in inbox
   $action = self::ACTION_ROUTE_FOLDER;  
   $destination_folder = Folder::model()->lookup($account_id,$this->path_inbox);             
 }

Copy after login

We then create a message entry in the database that contains the envelope information about this message:

  // create a message in db
      $message = Message::model()->findByAttributes(array('message_id'=>$msg['message_id']));
  if (!empty($message)) {
    // message exists already, 
	  $message_id = $message->id;    	  
  } else {
    $message_id = $m->add($user_id,$account_id,0,$sender_id,$msg['message_id'],$msg['subject'],$msg['udate'],$msg['in_reply_to']);         
  }

Copy after login

If the message comes from an unknown, unverified sender, we can move the message to a review folder. The review folder contains all messages from senders we do not recognize.

If the message is from a known sender and we have determined the destination, we can move it as long as it is not a quiet time (and Do Not Disturb is turned off):

  if ($recipient_id!==false) $m->setRecipient($message_id,$recipient_id);
  if ($action == self::ACTION_MOVE_REVIEW) {
    echo 'Moving to +Filtering/Review';lb();
    //$result = @imap_mail_move($this->stream,$msg['uid'],$this->path_review,CP_UID);
    $result = $this->messageMoveHandler($msg['uid'],$this->path_review,false);               
    if ($result) {
      echo 'moved<br />';
      $m->setStatus($message_id,Message::STATUS_REVIEW);
    }      
 } else if ($action == self::ACTION_ROUTE_FOLDER || $action == self::ACTION_ROUTE_FOLDER_BY_RX) {
  // lookup folder name by folder_id
  $folder = Folder::model()->findByPk($destination_folder);       
  // if inbox & quiet hours, don't route right now
  if (strtolower($folder['name'])=='inbox' and $sender->exclude_quiet_hours == Sender::EQH_NO and $this->isQuietHours($user_id)) continue;
  echo 'Moving to '.$folder['name'];lb();
  $mark_read = Folder::model()->isMarkRead($folder['mark_read']) || Sender::model()->isMarkRead($sender['mark_read']);
  //$result = @imap_mail_move($this->stream,$msg['uid'],$folder['name'],CP_UID);
  $result = $this->messageMoveHandler($msg['uid'],$folder['name'],$mark_read);
  if ($result) {
    echo 'moved<br />';
    $m->setStatus($message_id,Message::STATUS_ROUTED);         
    $m->setFolder($message_id,$destination_folder);
  }
}

Copy after login

During quiet hours, emails are primarily saved in filtered folders.

Every few hours, a different process builds a message digest using message table records to determine recently received and filtered emails and how they were routed.

2. Implement whitelist challenge for unknown senders

The goal of the whitelist challenge is to keep any messages from unknown senders, such as marketing bots or spammers that might be in your inbox. SE places messages from unknown senders into a review folder. However, if you turn on whitelisting, we will send a challenge email to give the sender a chance to verify that they are human. If they reply, we'll move the message to your inbox. If the email turns out to be unwanted, you can delete the message from the digest or drag it to any folder you want to train it to.

Users can turn whitelisting on and off in settings:

Build advanced email capabilities with IMAP and PHP

To implement whitelisting, every time a new sender receives a message, we will send an email challenge:

if ($isNew) {
        $this->challengeSender($user_id,$account_id,$sender,$message);
      }

Copy after login

ChallengeSender Sends an encoded link to the user for them to click. We've also put some safeguards in place to ensure we don't get stuck in an email loop with out-of-office messages:

 public function challengeSender($user_id,$account_id,$sender,$message) {
   // whitelist email challenge
   $yg = new Yiigun();
   $ac = Account::model()->findByPk($account_id);
   if (!empty($ac['challenge_name']))
     $from = $ac['challenge_name'].' <no-reply@'.$yg->mg_domain.'>';
    else
      $from = 'Filter <no-reply@'.$yg->mg_domain.'>';      
   $cred = Account::model()->getCredentials($ac->cred);
   $account_email = $cred[0];
   unset($cred); 
   // safety: checks no recent email
   if ($sender->last_emailed>(time()-(48*60*60))) return false;   
   if ($sender->isBot($sender['email'])) {
     // to do - can also set this person to bulk by default
     return false;
   }
$link=Yii::app()->getBaseUrl(true)."/sender/verify/s/".$sender->id."/m/".$message->id.'/u/'.$message->udate;
    $subject = 'Please verify the message you sent to '.$account_email;
    $body="<p>Hi,<br /><br /> I'm trying to reduce unsolicited email. Could you please verify your email address by clicking the link below:<br /><a href=\"".$link.'">'.$link.'</a><br /><br />Verifying your email address will help speed your message into my inbox. Thanks for your assistance!</p>';
    $yg->send_html_message($from, $sender['email'], $subject,$body);
    // update last_emailed
    $sender->touchLastEmailed($sender->id);
 }

Copy after login

Then, if the recipient clicks on the encoded link, we validate them in the database. The sender controller handles these requests and checks their validity:

  public function actionVerify($s = 0, $m=0,$u=0) {
    // verify that secure msg url from digest is valid, log in user, show msg
    $sender_id = $s;
    $message_id = $m;
    $udate = $u;
    $msg = Message::model()->findByPk($message_id);
    if (!empty($msg) && $msg->sender_id == $sender_id && $msg->udate == $udate) {
      $result = 'Thank you for your assistance. I\'ll respond to your email as soon as possible.';
      $a = new Advanced();
      $a->verifySender($msg->account_id,$sender_id);
    } else {
      $result = 'Sorry, we could not verify your email address.';
    }
    	$this->render('verify',array(
			'result'=>$result,
		));
  }

Copy after login

This tells our processing loop to move this message and future messages from this sender to the inbox.

3。报告未回复的电子邮件

有时，查看您已发送但未收到回复的消息摘要会有所帮助。为了识别这些邮件，Simplify Email 会监视已发送但尚未收到回复的邮件。

我们收到的每条消息都包含一个唯一的 ID，称为 message_id（IMAP 规范的一部分）。它通常看起来像这样：

Message-Id: <CALe0OAaF3fb3d=gCq2Fs=Ex61Qp6FdbiA4Mvs6kTQ@mail.gmail.com>

Copy after login

此外，当发送消息以回复其他消息时，它们有一个 in_reply_to 字段，该字段链接回原始 message_id。

因此，我们使用 SQL 查询来查找所有收到的消息，这些消息没有引用其 message_id 的相应回复消息。为此，我们在没有 in_reply_to id 的情况下使用 LEFT OUTER JOIN：

public function getUnanswered($account_id,$mode=0, $range_days = 7) {
    if ($mode==0)
      $subject_compare = 'not';
    else
      $subject_compare = '';
    $query = Yii::app()->db->createCommand("SELECT fi_sent_message.id, fi_sent_message.recipient_id as sender_id,fi_sent_message.subject,fi_sent_message.udate,fi_message.in_reply_to,fi_sent_message.message_id  FROM fi_sent_message LEFT OUTER JOIN fi_message ON fi_message.in_reply_to = fi_sent_message.message_id WHERE fi_sent_message.account_id = ".$account_id." AND fi_message.in_reply_to is null and fi_sent_message.udate > ".(time()-(3600*24*$range_days))." and fi_sent_message.subject ".$subject_compare." like 'Re: %' ORDER BY fi_sent_message.udate  DESC")->queryAll();
    return $query;
  }

Copy after login

我们使用 $subject_compare 模式来区分我们发送的尚未回复的消息和我们发送给尚未回复的线程的回复。以下是您帐户中的未回复消息报告：

Build advanced email capabilities with IMAP and PHP

SE 还将此信息作为可选摘要提供，称为未回复电子邮件摘要。您可以每天、每隔几天或每周收到它。

我们还使用类似的 SQL 表格和 Google Charts 来提供有关某些人向您发送电子邮件的频率的报告：

Build advanced email capabilities with IMAP and PHP

  public function reportInbound($account_id,$range=30,$limit = 100) {
    $result= Yii::app()->db->createCommand('SELECT fi_sender.personal, fi_sender.email,count(sender_id) as cnt
      FROM fi_message LEFT JOIN fi_sender ON fi_sender.id =fi_message.sender_id WHERE fi_sender.account_id = :account_id AND fi_message.created_at > DATE_SUB( NOW() , INTERVAL :range DAY )  
      GROUP BY sender_id ORDER BY cnt desc LIMIT :limit ')->bindValue('range',$range)->bindValue('account_id',$account_id)->bindValue('limit',$limit)->queryAll();
    return $result;
  }

Copy after login

我很快就会撰写更多有关 Tuts+ 的 Google Charts 的文章。