Announcing vSMTP AI

Two years ago, when we started thinking about version 3 of vSMTP, one of our main areas of thought was to reduce manual maintenance operations to favor automated actions as much as possible.

The first step was to enable users to interact with vSMTP behavior on the fly and be able to write complex scenarios to interface with their systems.

I am proud to announce that the second stage is underway: gradually delegating basic functions to artificial intelligence: vSMTP now includes a library made up of AI routines capable of interacting with email flows on the fly.

An example: bounces and transients

Categorizing bounced and transient emails is crucial for your business to maintain a positive sender reputation with service providers and thus meet your customer service level agreements.

For years, the solutions used were limited to an obscure jumble of lists of regular expressions, allowing empirical categorization of DSNs trying to catch sentences like:

*) 550 5.0.0 zzmail services - this mailbox is full. Please retry later.
*) 552 5.2.2 foomail - the end user's mailbox quota has been exceeded. Please retryCode language: Markdown (markdown)

I’m not even talking about the case where the service provider decides to change a word like “mailbox” to “mail box”…

Even now, our competitors are announcing new features like “custom categories” or “arbitrary bounce classification labels”. We do know that users must have the flexibility to define their own categories to refine their processing and meet their own requirements.

However, heavy use of REGEX lists is definitely an old-fashioned processing method.

This is why we have developed an AI routine specifically for bounces categories.

Natural Language Processing

vSMTP automatically detect and categorize bounced or transient emails using a neural network specialized in natural language recognition (NLP).

Extracting meaning, sentiment and context… That’s exactly what we need. You should consider the DSN reasons for a bounce message as a review of a movie and the category searched for the sentiment/review of the movie, rated from 1 to 5.

Benefits

After an initial learning period, the SMTP engine no longer works with regex or patterns but instead analyzes and infers the meaning of the DSN status. Periodically, the neural network updates itself. This contributes to smoother and more reliable email handling. Notable benefits are:

Reduce the work devoted to maintenance: instead of dealing with endless lists of REGEX, the user can focus on much more valuable work.

Simplify the classification process: instead of managing endless lists of REGEX, the AI can intelligently categorize reasons based on natural language understanding, which is particularly valuable when dealing with variations in language use.

Eliminate emergency fixes: updates from service providers are no longer considered as breaking changes. By leveraging AI, vSMTP adapts to changes in message language or structure without requiring manual intervention.

The better the tools, the better the business

This is an example among others of the power of vSMTP artificial intelligence subroutines.

If you would like to learn more about the multiple capabilities of vSMTP and how it addresses other challenges in email processing, please do not hesitate to contact us.

Stay tuned.

Email authentication : SPF

In the previous post we described DNS lookup and the IPREV mechanisms. They were great SPAM killers but the rise of the Internet in the 2000s saw the emergence of increasingly complex email architectures. A new protocol became popular through an experimental RFC : the Send Policy Framework (SPF)

In 2014 the adoption of the RFC7208 definitely anchors SPF as a standard.

This is the second stage of your journey in the email authentication landscape.

What is SPF ?

Basically, the SPF framework specifies which servers are allowed to send mail from a specific domain name. If a spoofed server attempts to send an email, the receiving MTA checks the spoofer’s IP. As it will not be declared in the SPF framework of the original domain, the connection will be rejected.

That means that the SPF framework allows the Administrative Management Domains (ADMDs) to explicitly authorize hosts to use their domain in the “MAIL FROM” or “HELO” identities. The authorization list is published in the DNS records of the sender’s domain.

DNS Records

The type of a SPF record is TXT. There should be only one SPF record per domain.

Here is a basic SPF record example: “only servers in the range 123.123.123.0/24 and MTA (MX) are authorized to send emails from my domain mydomain.com. All other senders are considered unauthorized.”

mydomain.com.          TXT "v=spf1 +mx ip4:123.123.123.0/24 -all"Code language: JavaScript (javascript)

There is also a tilde version: ~all. It warns that other senders are not allowed, but must still be accepted. This “Soft Fail” statement was first introduced for testing purposes, but is now used by various hosting providers.

As a domain must not have multiple SPF records, multiple include must be declared into a unique DNS records.

mydomain.com	      TXT "v=spf1 include:domain.com include:otherdomain.com -all"Code language: JavaScript (javascript)

Please also notice that the total number of DNS lookups can’t exceed 10.

Great tools and articles about SPF compliancy can be found on various websites such as PowerDMARC or EasyDMARC.

SPF evaluation

HELO/EHLO versus MAIL FROM identity

The RFC7208 does not enforce a HELO/EHLO verification.

“It is RECOMMENDED that SPF verifiers not only check the “MAIL FROM” identity but also separately check the “HELO” identity […] Additionally, since SPF records published for “HELO” identities refer to a single host, when available, they are a very reliable source of host authorization status. Checking “HELO” before “MAIL FROM” is the RECOMMENDED sequence if both are checked.“

The RFC 5321 tends to normalize the HELO/EHLO arguments to represent the fully qualified domain name of the SMTP client. However the vSMTP SPF verifier is prepared for the identity to be an IP address literal or simply be malformed. In this case the “MAIL FROM” check occurs.

“SPF check can only be performed when the “HELO” string is a valid, multi-label domain name […] SPF verifiers must check the “MAIL FROM” identity if a “HELO” check either has not been performed or has not reached a definitive policy result.“

Note that RFC5321 allows the reverse-path to be null. In this case, the RFC7208 defines the “MAIL FROM” identity to be the mailbox composed of the local-part “postmaster” and the “HELO” identity.

Results

The vSMTP SPF verifier implements results semantically equivalent to the RFC.

Result	     Description
--------------------------------------------------------------------------------------------------------------------
none	     (a) no syntactically valid DNS domain name was extracted from the SMTP session that could be used as
             the one to be or (b) no SPF records were retrieved from the DNS.
neutral	     The ADMD has explicitly stated that it is not asserting whether the IP address is authorized.
pass	     The client is authorized to inject mail with the given identity.
fail	     The client is not authorized to use the domain in the given identity.
softfail     The host is probably not authorized but the ADMD has not published a stronger policy.
temperror    A transient (generally DNS) error while performing the check.
permerror    The domain’s published records (DNS) could not be correctly interpreted.
policy	     (NOT IMPLEMENTED) Code language: JavaScript (javascript)

None vs PermError

When a receiving MTA begins to perform SPF authentication on an email, it fetches all the DNS TXT records that begin with “v=spf1”. In case SPF is not configured for the sending domain, and no SPF record is found in the DNS, a None result is returned.

On the contrary, if multiple SPF records beginning with “v=spf1” are found to exist for the same domain, an SPF PermError result is returned.

Result headers

The RFC7208 recommends to store the SPF evaluation in a message header. Two options are available.

The “Received-SPF” header

This is the legacy header. It includes enough information to enable reconstruction of the SPF evaluation of the message.

Received-SPF: pass (mybox.example.org: domain of myname@example.com
  designates 192.0.2.1 as permitted sender)
  receiver=mybox.example.org; client-ip=192.0.2.1;
  envelope-from="myname@example.com"; helo=foo.example.com;Code language: JavaScript (javascript)

Please notice that the Received-SPF header field should be considered as trace field and therefore :

• It must appear above all other Received-SPF fields in the message.
• It should be prepended to the existing header, above the Received: field that is generated by the SMTP receiver.

The “Authentication-Results” header

This header is described in RFC8601 : Message Header Field for Indicating Message Authentication Status. It is designed to relay the result itself and related output details of likely use to end users.

Authentication-Results: example.com; spf=pass smtp.mailfrom=example.netCode language: HTTP (http)

Both headers are in common use. However, it is important to note that they were designed to serve slightly different purposes and therefore they should both be added.

vSMTP implementation

The vSMTP spf module describe all the required functions to handle SPF queries. To query a DNS server we have to build a DNS resolver. Let’s use the previous DNS resolvers we used in the lastest post.

const google_dns = dns::resolver(#{
    config: "google_tls",
});Code language: PHP (php)

A basic configuration.

fn on_helo(ctx) {

  // EHLO/HELO SPF check
  let helo_identity = spf::check_host(#{
    ip: ctx.client_ip,
    helo: ctx.helo,
    dns_resolver: global::dns_resolver
  });
  
  // Deny and close the connection if SPF result is not "pass"    
  if helo_identity != "pass" {
    return status::deny(`550 5.7.23 ${ctx.sender} is not allowed to send mail from ${ctx.client_ip}`)
  }

  ctx.store("helo", helo_identity);
  status::next()
}


fn on_mail_from(ctx) {

  // MAIL FROM SPF check
  let mail_from_identity = spf::check_host(#{
    ip: ctx.client_ip,
    helo: ctx.helo,
    mail_from: ctx.sender,
    dns_resolver: global::dns_resolver
  });

  if mail_from_identity != "pass" {
    status::deny(`550 5.7.23 ${ctx.sender} is not allowed to send mail from ${ctx.client_ip}`)
  } else {
    ctx.store("mail_from", mail_from_identity);
    status::next()
  }
}


fn on_pre_queue(ctx) {

  // Add authentication header
  auth::add_header(ctx, #{
    auth_serv_id: "mydomain.tld" // The domain name of the authentication server
  });
  status::next()
}Code language: JavaScript (javascript)

What’s next ?

With IPREV and SPF, we focused on sender verification. Now it’s time to ensure and verify email integrity. This is done via the DKIM protocol…

Stay tuned.

Email authentication #1 : DNS and IPREV

Email authentication protocols play crucial roles in securing email communication by verifying the authenticity of the sender and preventing email spoofing and phishing attacks. During this journey through the landscape of email authentication protocols, we will overfly protocols like IPREV or SPF but also DKIM, ARC and DMARC and the way to operate them using vSMTP.

Let’s take off.

Domain Name System : the common base

The first stage brings us in the 80s when the IETF (Internet Engineering Task Force) released the RFC 1034 (Request For Comment) : “DOMAIN NAMES – CONCEPTS AND FACILITIES”. This RFC aims to:

“[…]standardize the domain style names, their use for Internet mail and host address support, and the protocols and servers used to implement domain name facilities[…]“

Whatever the email authentication mechanism, it relies on the DNS (Domain Name System). If you have any doubts about how DNS works, there is an excellent article on the CloudFlare website.

IPREV : back to the 90s

In the 1990s, hackers quickly noticed the lack of SMTP sending server authentication, allowing them to send emails using a spoofed Internet domain name.

The first countermeasure deployed against Domain Name Spoofing was the Reverse DNS Lookup or PTR (Pointer) Record method. It was a simple but effective email authentication method.

By associating an IP address with a domain name by creating a reverse DNS record, when an email server receives an email, the server can perform a reverse DNS lookup to check if the sending IP address matches the domain’s DNS records.

Mail servers use reverse DNS to perform simple anti-spoofing checks but this method can be refined. The method is called “Forward-confirmed reverse DNS” or IPREV. This forward DNS lookup check that the reverse lookup matches the FQDN of the email header (aka seeing if it resolved back to the original IP number).

This mechanism has some limits. In some scenario i.e. when many MTAs are masked behind a unique IP address, reverse DNS query may return in a round-robin manner one of the server hostname.

vSMTP implementation

As always, we tried to make the work of administrators easier by adding functions that can be used directly in the configuration scripts of vSMTP.

The vSMTP dns and the iprev classes describe all the required functions to handle IPREV and DNS queries. To query a DNS server we have to build a DNS resolver. Let’s create a resolver that connects to Google DNS servers.

const google_dns = dns::resolver(#{
    config: "google_tls",
});Code language: PHP (php)

And the reverse lookup :

fn on_mail_from(ctx) {
  let result = dns::rlookup(#{
     dns_resolver: global::google_dns,
     ip: ctx.client_ip
  }); 

  // rlookup function returns an array of FQDN
  if result.is_empty() {
    return status::deny("550 5.7.25 Reverse DNS validation failed.");
  }

  // The first record must match the MAIL FROM domain
  if result[0] != ctx.sender.domain {
    return status::deny("550 5.7.25 Reverse DNS validation failed."); 
  }
 
  return status::next();
}
Code language: JavaScript (javascript)

This simple implementation should block most basic SPAM.

You should also use the iprev::check fonction to perform a forward-confirmed reverse DNS and, in case of failure, log the IP and close the connection.

fn on_connect(ctx) {
  let iprev_res = iprev::check(#{
     ip: ctx.client_ip,
     dns_resolver: global::google_dns
  });

  if iprev_res.value != "pass" {
    // Send a custom message to the log dispatcher using RFC 5424 "SD" style.
    log("info", `[netsec@vsmtp mechanism="iprev" ip="${ctx.client_ip}" domain="${ctx.sender.domain}"]`); 

    // Send an error code and close the connection
    return status::deny("550 5.7.25 Reverse DNS validation failed.");
  }

  return status::next();
}Code language: PHP (php)

Perhaps you may want to refine this implementation by fetching the log and updating a firewall rule to block this IP address or you may wish to add this pair (IP, domain) to an ephemeral cache to avoid saturating your DNS servers with requests… we will explain how to use the log dispatcher service and the native vSMTP plugins in upcoming posts.

Otherwise you can only store the iprev result in the context, so it can be used later to generate the “Authentication-Results” header field and continue processing.

fn on_connect(ctx) {
  let iprev_res = iprev::check(#{
     ip: ctx.client_ip,
     dns_resolver: global::google_dns
  });

  ctx.store(iprev_res);
  return status::next();
}Code language: PHP (php)

Imagination is the only limit to the power of vSMTP scripting.

What’s next ?

IPREV and reverse DNS lookup were great SPAM killers but the rise of the Internet in the 2000s saw the emergence of increasingly complex email architectures. A new protocol became popular through an experimental RFC : the Send Policy Framework (SPF)

In 2014 the adoption of the RFC 7208 definitely anchors SPF as a standard.

This will be the second stage of your journey.

Stay tuned.

Designed for cloud computing

There was not much movement happening in vSMTP for a few months, as we were working behind the scenes on the third version of vSMTP with a completely new architecture.

We rebuilt the server from the ground up, moving from a monolithic architecture to a micro-service architecture, using AMQP as the main way of communication between services.

A new architecture

Since then we did great optimizations, the transition to a micro-service architecture was the major shift in how vSMTP version 3 is structured. This change involved breaking down the server into smaller, more manageable services, each responsible for specific functions.

The micro-service architecture offers benefits such as scalability, flexibility, and easier maintenance.

RabbitMQ and AMQP

Using AMQP (Advanced Message Queuing Protocol) and RabbitMQ as vSMTP messaging infrastructure was an obvious choice.

RabbitMQ acts as a message broker, facilitating the exchange of messages between different vSMTP services. It provides features such as message queuing, routing, and load balancing, which improve the reliability and scalability of vSMTP.

Microservices, AMQP and cloud computing

Microservices architecture and the use of Advanced Message Queuing Protocol are well suited to cloud computing for several reasons, as they align with the key principles and benefits of cloud-based applications:

• Scalability and elasticity
• Resource allocation and efficiency
• High availability and fault tolerance
• Global reach

In summary, microservices architecture and cloud computing naturally complement each other due to the alignment of their fundamental principles and capabilities.

The combination of vSMTP’s new internal architecture with the scalability, flexibility, and managed services of cloud platforms enables messaging service providers to build and operate modern, resilient, and efficient infrastructures that can meet the demands today’s dynamic and evolving business environments.

You can right now download the release candidate on GitHub. Our teams will be happy to guide you during your installation.

Stay tuned. Join us on Discord, X and LinkedIn.

A rocket is launched

350,000,000,000

An astronomical number – 350 billion emails per day routed and analyzed by software built in the 90s.

The numerous energy and environmental crises have made us aware that any new IT product must be designed with the major objective of reducing the environmental footprint. It is on this premise that the vSMTP project was born in 2020.

Since its first releases in 2021, vSMTP is certainly one of the fastest MTAs in the world.

Since then we did great optimizations, but, when considering an MTA in 2023, its intrinsic performance is not the only quality to take into account. It must be easily usable and configurable. The vSMTP version 2 specifications met these criteria by providing an advanced scripting language as well as a very easy to use rules system.

Version 3

We wanted to go further, and design a new version capable of integrating into a world where cloud computing reigns supreme and where companies’ commitment to CSR has become mainstream.

The viridIT teams have worked hard to answer them and I am proud to announce the release of version 3 of vSMTP at the beginning of October. This major release brings a completely new architecture based on microservices.

You can right now download the release candidate on GitHub. Our teams will be happy to guide you during your installation.

Stay tuned. Join us on Discord, X and LinkedIn.