Good practice in minimising e-mail abuse

Version 2.0, 17th August 2004
Malcolm Hutty, Richard Clayton

DRAFT update Rodney Tillotson September 2005

Material changed from RIPE-206 has a pink background like this.

Material added to RIPE-206 has a yellow background like this.

Top

Introduction

Unsolicited Bulk Email (UBE) is a widespread problem on the Internet. It is sometimes called "junk email" or "spam". Because of the volumes involved and the indiscriminate nature of its sending, there can be few email users who do not have first hand experience of receiving UBE, often in significant quantity.

The sending of UBE is considered to be unacceptable behaviour because:

Furthermore:

It is resource intensive and to a large extent ineffective for ISPs to try to block UBE once it has been sent, so this BCP does not describe the limited manner in which this may be attempted. In the fight against UBE the ISP's most practical contribution is to minimise or eliminate the sending (or other use) of UBE by its customers or from its systems. The purpose of this BCP is to describe the industry’s current collective opinion of the Best Practice in achieving this.

Besides being in the general interest for ISPs to adopt Best Practice, many ISPs will wish to be publicly seen to be doing what they can to combat UBE. To that end, it is expected that ISPs will wish to state formally that they have adopted the recommendations of this BCP. To assist in this, the document has been written as a "standard", using the terms MUST, SHOULD, MAY and MUST NOT as defined in RFC 2119 (see Appendix D for a summary of this).

For an ISP to be effective in combating UBE, Best Practice is as follows.

  1. The ISP MUST ensure that their email systems will not relay email for unauthorised third parties.
  2. The ISP MUST ensure that all email generated within their network can be traced to its source; and MUST ensure that the immediate source of email which arrives from other networks can be determined.
  3. The ISP MUST ensure that all email generated within their own networks can be attributed to a particular customer or system.
  4. The ISP MUST operate appropriate arrangements for the handling of reports of abuse by their customers. They MUST also ensure that IP allocation entries in regional registries such as RIPE contain appropriate abuse team email addresses.
  5. Where abuse is proved, the ISP MUST take effective action to prevent the customer from continuing that abuse. The legal basis on which services are provided to customers MUST allow such action to be taken.
  6. The ISP MUST treat use of UBE to promote secondary services as an abuse of the provision of that secondary service.
  7. The ISP MUST NOT permit customers to distribute tools, or lists of email addresses, whose purpose is the sending of UBE.
  8. The ISP SHOULD disseminate information on the action taken in regard to customers who have sent UBE.
  9. The ISP MUST educate their customers on the nature of UBE, and MUST ensure that their customers have been made aware that sending UBE will be treated as unacceptable behaviour. The ISP MUST inform their customers about any automated anti-spam mechanisms in operation, and MUST educate their customers about any potential harmful side-effects.
These nine points are expanded below.

Along with the extended explanations, this BCP lists a number of conditions that ISPs MUST impose upon their customers. It will be necessary to ensure that the contract made between ISP and customer gives the ISP the legal right to make these impositions and to withdraw services when unacceptable behaviour occurs.

To ensure fair competition between ISPs, so that no marketing advantage can be gained by failing to spell out these obligations properly, the ISP MAY use the standard clauses set out in Appendix C and MUST use these clauses or others which are at least as effective. The ISP MAY place these clauses into a more general Acceptable Use Policy (AUP) that covers other abuse issues.

The provisions of this BCP document are to be applied to all customers. However, some customers will have customers of their own. The ISP will conform to Best Practice by ensuring that such customers adopt this BCP themselves, and thereby apply Best Practice procedures in turn to their own customers.

Appendix A provides a glossary of terms, but in particular, throughout this document the term "ISP" should be understood to apply not only to "top level" providers of Internet connectivity, but also to customers of such ISPs who are "recursively" applying the BCP to their own customers. Also, the term "customer" should be understood to apply not only where there is a formal contractual relationship, but also to other cases where someone may be a "user" of the ISP's facilities.

Contents

1. No email relaying

Discussion

Historically, email systems using the SMTP protocol have been prepared to accept email from anyone and then deliver it to, or towards, its true destination. This willingness to "relay" made Internet email extremely robust, since minor configuration errors on one machine could be overcome by another machine with more accurate knowledge of how to deliver the email. Furthermore, the spirit of co-operation that pervades the Internet has meant that machine owners tended not to log, let alone block, such relaying.

With the advent of the Domain Name System (DNS) and far better connectivity for all machines, this need for relaying passed away long ago. However, the functionality continues to be provided within email programs.

Unfortunately, in recent times, the unscrupulous have been abusing the "relay" function by sending a single piece of email with a long list of destinations. This can cause someone else’s system to generate multiple copies of the email for delivery to many different addresses. By "amplifying" email in this way, the sender of UBE is exploiting the resources of others to do most of the work of generating the UBE. Furthermore, it is possible for the sender to use a poorly configured system to hide the true source of the email or at least to ensure that the less skilled misidentify its source.

As it is no longer required and because it is open to abuse, it is now considered quite improper for systems to be configured in such a way that they will relay email for unauthorised people.

There are several ongoing projects on the wider Internet to identify systems that are still prepared to relay email. Typically, such systems are added to blocking lists that affect the propagation of email. Even if one wished to run an "open relay" the time is approaching when few will be prepared to interwork with such a system.

It is common for ISPs to run "smarthosts", which provide SMTP email delivery for their customers, especially those on dialup connections or local networks. This avoids the necessity for these customer machines to have fully fledged delivery systems of their own. This "smarthosting" is just a form of relaying, but is of course a completely acceptable practice, provided that the smarthost is configured to refuse to relay any email sent to it by unauthorised machines.

Requirements

Appendix B contains pointers to technical information about how to ensure that email relaying does not occur.
Appendix C contains specimen contractual clauses to allow these, and other, requirements to be implemented.

Contents

2. Traceability of email passing through the system

Discussion

Tracing the source of email requires that all systems comply with the email standards and add a "Received" header line as the email passes through them. This serves to identify the machine that is adding the header and the machine from which the email arrived. In principle, the oldest such line indicates the source of the email. In practice, this is sometimes forged, and to trace the true sender it is necessary to work through the Received lines in time order until a discontinuity is found.

The senders of email will sometimes try to obscure the true origin of email by forging the name of the source machine in the "HELO" protocol command. This type of forgery is made easy to detect by ensuring that the Received line contains not only the name, but also the IP address of the sending system, since the latter cannot be disguised.

Requirements

Contents

3. Identification of the sender of email

Discussion

Section 2 has the effect of ensuring that email can be traced back to an originating IP address.

With dialup access, it is common to use "dynamic IP", so that the same address will be reused for other customers. ISDN connections take only a few seconds, so in principle the same IP address can almost immediately be in use by another person entirely.

However, the combination of IP address and time of connection will uniquely identify where the email came from. So an accurate time must be recorded into the email header Received line. The combination of this time with other access logs, held by the originating ISP, will serve to identify the sender.

The above description has only skimmed the surface of a complex topic. The LINX Best Current Practice document on "Traceability" (see Appendix B) can be consulted for further information and advice.

Requirements

Exception

An exception to sections (2) and (3) arises in the case of a system run to deliberately hide the source of email - often called an "anon server". "Anon servers" are used to preserve anonymity where, for example, someone seeks help from a group supporting victims of abuse or wishes to express political views in a country that may punish dissent.

Contents

4. Handle abuse reports

Discussion

ISPs are required to accept and process emailed reports of abuse by their customers.

If a customer posts UBE then complaints are likely to be made to the ISP. These complaints have in the past, by convention, generally been sent to the "postmaster" mailbox. More recently it has become desirable to direct such email to a specialist "abuse" mailbox. This practice was first fully documented in RFC2142.

Some ISPs are developing specialised reporting systems that, for example, allow complaints to be entered into a form on a website. There are many advantages to such systems in that they ensure that reports are complete and they can boost productivity, allowing prompt and efficient handling of the reports. However, they have disadvantages in that they can only be used online and at present there are no standard conventions for their layout or their location. Therefore, although ISPs may wish to encourage their use and to develop other automated submission systems for third-party sites that collate reports from many people, it is not appropriate, at present, to see them as entirely replacing email reports.

It is often "obvious" which ISP is responsible for particular IP addresses and hence which "abuse" mailbox to use. However, in some cases it may be necessaty to consult the appropriate Regional Registry (such as RIPE) in order to determine IP address ownership. It has therefore become standard practice to document the explicit abuse@isp email address to be used within registry entries. It is important that complaints continue to be accepted at the "obvious" address even though the registry entry may indicate that another address is to be preferred. At present, registry entries can only record abuse mailbox details by means of comment fields, which inhibits automatic processing, but a formally specified system may be introduced in the future.

When a complaint is received, it is wise to promptly acknowledge it, perhaps merely with a standard message that describes the local policies and procedures.

It is desirable to run a "ticketing" system that allows incident reports to be tracked. This will assist in combining reports and in collating further correspondence that may arrive from the original complainant.

It is also desirable to reply to people who submit complaints to explain what action is eventually decided upon. Sometimes, especially when a large number of reports are being received, this is not very practical. The standard message described above can usefully explain that this may happen, and it may be possible to direct people to a web site where any action taken by the ISP will be recorded (see section 6 below).

Requirements

Contents

5. Act upon reports of abuse

Discussion

There is no acceptable excuse for the sending of unsolicited bulk email.

Apart from people pleading ignorance of the unacceptable nature of UBE, which is covered in the Requirements section below, the most likely explanation will be a claim that the email was in fact solicited.

In determining whether to accept this explanation the ISP must look at how the email addresses were acquired. Data Protection legislation will normally require that information is processed "fairly and lawfully". In particular, the ISP should look for positive answers to all the following questions:

All EU countries have legislation implementing EC Directive 2002/58/EC and its forerunners 95/46/EC and 97/66/EC; in most cases the questions above will reflect the primary concerns of the legislation.

The effect of these tests is that posting articles to Usenet or the mere visiting of a web site does NOT make the subsequent sending of bulk email "solicited". Nor does it make it likely that acquiring lists of email addresses from a third party will mean that a customer has acquired any entitlement to send solicited email to those addresses.

Clearly, where someone has explicitly signed up for a mailing list the email that arrives is solicited. However, in the real world some mailing lists are dormant for long periods and the people who join them can have poor memories. When email does arrive it may be reported to the mailing list owner’s ISP as being unsolicited. Since the same software can be used to send genuine requested mailing list email and UBE, the ISP will have to apply the tests given above to distinguish the two cases.

Mailing list owners can demonstrate that they are behaving responsibly by keeping good records. Ideally they would be able to produce a copy of the "subscribe" email for the list and would have checked it out at the time by "mailback" confirmation techniques to ensure that a third party had not maliciously requested the subscription. It is of course vital that the recipient of the unwanted email can unsubscribe from the list. Modern mailing list software packages automate all these procedures. There is a great deal more about this topic in the LINX Best Current Practice document on "Opt-in Mailing Lists" (see Appendix B).

As discussed at the start of this document, ISPs may have customers large enough to apply this BCP on their own account, and manage their own customers or users. In these cases the ISP may depend on their customer to deal with the sender of UBE, and need not apply the sanctions discussed below, such as disconnecting these large customers from the Internet. However, the ISP remains accountable to the wider community, which will expect the ISP to be reasonably assured that their customer will indeed take suitable action in the ISP's stead.

Requirements

Contents

6. Deny use of UBE for promotion

Discussion

Improvements in filtering technology have led many senders of UBE to move much of the content of their message from the email to a web site or other medium, and to direct their recipients towards that secondary source. Traffic coming to such web sites provides the incentive for senders to keep sending UBE, and much UBE would not exist or would be more readily controlled but for the existence of these web sites.

It is not acceptable to use UBE to promote web sites or other secondary services, nor is it acceptable to use such services to promote or reap the benefits of sending UBE. Accordingly, use of UBE to promote a web site or other service must be treated as an abuse not only of the email service used to send the UBE, but also as infringing the conditions of use of the web site or other service promoted by the UBE. The expectation should be that promoting web sites via UBE will result in them being shut down.

The unacceptability of using UBE for promotion and the necessity of taking action against web sites is not affected by there being more than one ISP involved. Each ISP is expected to take effective action against their particular customer.

In some cases a franchise system is in operation and a central, legitimately operated, web site is promoted by UBE sent by a franchisee without the knowledge or permission of the central web site owner. In such circumstance UBE will only be eliminated if the web site owner takes firm action to disenfranchise the UBE sender and to ensure that they do not profit from their abuse. ISPs providing services to such web sites must satisfy themselves that appropriate control mechanisms are in place before concluding it would be unfair to suspend the web site and letting it remain operational.

In some cases web sites are promoted by third parties who misrepresent the nature of the email they will send, so that UBE is sent on behalf of the web site owner. In such circumstances the web site owner will look to their service contract with the third party for recompense for the significant damage that will have been done to their reputation. Provided that the ISP is satisfied that the problem will not recur it would clearly be unreasonable to suspend the web site.

Requirements

Contents

7. Prohibit the distribution of UBE tools and address lists by customers

Discussion

Some businesses promote the sending of UBE by making available programs for bulk email sending, email address harvesting and may also sell their own lists of email addresses. Since using these products is unacceptable, the community considers the promotion of these products, usually on the web, as also being unacceptable. Although the major league senders of UBE use their own systems, the ability to obtain "kits" for sending UBE encourages others to attempt to use them and so there is a real benefit in suppressing these kits.

Of course many products have entirely legitimate uses in handling mailing lists run on an opt-in basis and there is no question of preventing these products being promoted. However, legitimate products do not provide methods for hiding the source of email or for seeking out and using third party machines.

Similarly, there are a few legitimate sellers of address lists, although such lists are unusual because of the necessity of complying with Data Protection principles. It is regrettable to note that many alleged "opt-in" lists turn out to be incorrectly described.

Requirements

Contents

8. Disseminate information on action taken against customers

Discussion

There are a number of advantages to making public any action taken against customers who have sent UBE:

However, when publishing information about the action that has been taken it is vital to be accurate and matter of fact, for otherwise there is a risk of an action for defamation.

It is also necessary to comply with Data Protection legislation. This may not apply to companies - so their full name and address can be published; but with individuals it would almost certainly be necessary to avoid exact identification unless contractual steps had been taken to allow this information to be released when abuse had occurred.

The sort of report which would cause no problems would be along the lines of "On <date> we terminated the account known as <username@isp.com> because of its use in sending Unsolicited Bulk Email. Further reports of abuse by this account are unnecessary."

In addition to any public reporting, an ISP will wish to take such steps as are possible to disseminate information about abuse within its own organisation. It is not good practice to allow terminated accounts to be reopened, or the same individual, detectable by name, address or perhaps credit card, to immediately open a new account to replace the previous one.

Requirements

Contents

9. Education

Discussion

ISPs need to take steps to educate their customers in acceptable email behaviour. It is recognised that ISPs may have difficulty in doing this because their marketing departments wish to play up the advantages of the Internet and downplay negative issues.

Many reports of abuse that are received by ISPs do not contain vital information that will allow action to be taken. Customers forget, for example, to include full header information, which is needed to properly identify the sender. Customers can also let their feelings run away with them and heap abuse on the abuse handling personnel.

It is the responsibility of everyone to try and improve this situation so that fewer inadequate or objectionable reports are sent, and less time is wasted dealing with such reports and less frustration is experienced by all concerned.

Many ISPs now operate email filtering systems that attempt to distinguish UBE from legitimate email and block or redirect the UBE. Systems may also attempt to detect mass-mailing email "worms" or "viruses". These systems are not perfect and will let through some UBE and some worms and can, on occasion, also disrupt the flow of items of legitimate email. It is important that ISP customers are aware of whether filtering is occuring, the type of system that is deployed, and hence the likely risk of email disruption.

Because reports sent to "abuse@" mailboxes are highly likely to contain copies of UBE or viruses, it is most important that this email does not pass through filtering systems that discard or reject this type of email.

Requirements

Contents

Appendix A: Glossary

AUP, Acceptable Use Policy
An extension to the contract between ISP and customer that sets out what the customer may and (mainly) may not do whilst using the ISP’s services.
BCP, Best Current Practice
A description of the best practice presently known to the industry.
DNS, Domain Name System
The distributed system that provides a translation service between names and IP addresses. It is described in RFC1035.
HELO, Hello
A command within the SMTP email protocol, used to announce the name of a remote machine.
IP, Internet Protocol
A basic protocol for exchanging packets between machines on the Internet. Other protocols are layered upon this to provide services for users. It is described in RFC791 and RFC1122.
ISP, Internet Service Provider
ISP is used in this document as a generic term to describe companies and organisations that provide Internet access to others. It is also used to describe customers of ISPs who have adopted this BCP and are applying it to their own customers in the ISPs stead.
LINX, London Internet Exchange
The LINX (http://www.linx.net/) is a totally neutral, not for profit partnership between ISPs. It operates the major UK Internet exchange point. As well as its core activity of facilitating the efficient movement of Internet traffic it is involved in non-core activities of general interest to its members. One such activity on "content regulation" has, as part of its work, generated this document.
NTP, Network Time Protocol
A protocol for obtaining an accurate measurement of the current time described in RFC1119 and RFC1305.
RFC, Request for Comments
The RFCs are a series of notes, started in 1969, about the Internet (originally the ARPANET). The notes discuss many aspects of computing and computer communication focusing in networking protocols, procedures, programs, and concepts, but also including meeting notes, opinion, and sometimes humour. The Internet standards are documented within the RFC documents.
See http://www.rfc-editor.org/.
RIPE, Réseaux IP Européens
RIPE (http://www.ripe.net/) is the Regional Internet Registry that handles IP address allocations in Europe, the Middle East, Central Asia and African countries north of the equator.
SMTP, Simple Mail Transfer Protocol
The email transfer protocol. It is currently documented in RFC2821.
UBE, Unsolicited Bulk Email
UBE is email that has been sent in large amounts without any explicit requests for it being made. It is sometimes called "junk email" or "spam". At present it usually contains advertising material for commercial ventures of dubious propriety.
UCE, Unsolicited Commercial Email
Some discussion of UBE distinguishes unsolicited email that is commercial in nature from non-commercial material. This document treats UBE as unacceptable per se, avoiding the need for value judgements on what may or may not be "commercial".

Contents

Appendix B: References and Resources

[Notes:
1. RIPE NCC is not responsible for the content of third party sites, and does not necessarily endorse their contents.
2. It is recognised that the links referred to here may not be available or current at any time in the future.]

There are many sites on the Internet that discuss unsolicited email in general. Some of the more interesting ones are:

There is almost certainly a discussion of the prevention of unauthorised email relaying on the home site of all mail handling software. For example:

For a comprehensive survey of pointers to information about email server software see the MAPS Transport Security Initiative (http://www.mail-abuse.com/support/an_sec3rdparty.html)

There are also generic products that can be used with many systems to control relaying. Mailshield (http://www.mailshield.com) is a commercial example.

You can test (http://www.abuse.net/relay.html) if your system allows unauthorised relaying.

LINX Best Current Practice documents:

All published RFCs are available from: http://www.ietf.org/rfc/

Contents

Appendix C: Specimen clauses

The following are clauses that ISPs may use in their Terms and Conditions and elsewhere to support the enforcement of sanctions against senders and promoters of UBE, as required to conform to this BCP. In these model clauses the ISP is referred to as "We"/"Us" and the customer as "You"/"Your". ISPs may wish to replace these by other defined terms from their own paperwork.

General clause to allow action to be taken

From time to time We publish Acceptable Use Policies ("AUPs") for the various services We provide. As a condition of Your use of a service, You are required to abide by the then current AUP for that service. If You do not do so, then We have the right at our sole discretion to suspend or terminate your account without notice or refund, to make an additional charge for the misuse, to block access to the relevant part of the service, or to apply a combination of these measures.

An AUP clause banning unauthorised mail relaying

You must ensure that You do not further the sending of Unsolicited Bulk Email by others. This applies to both material that originates on Your system and also third party material that might pass through it.

This includes but is not limited to a prohibition on running an "open mail relay", viz a machine which accepts mail from unauthorised or unknown senders and forwards it onward to a destination outside of Your machine or network. If Your machine does relay mail, on an authorised basis, then it must record its passing through Your system by means of an appropriate "Received" line.

As an exception to the ban on relaying and the necessity for a "Received" line, You may run an "anonymous" relay service provided that You monitor it in such a way as to detect unauthorised or excessive use.

General clause to permit scanning

We may, at our discretion, run manual or automatic systems to determine Your compliance with our AUPs (e.g. scanning for "open mail relays"). You are deemed to have granted permission for this limited intrusion onto Your network or machine.

An AUP clause to disallow sending of UBE

You may not use your account to send Unsolicited Bulk Email. You must have explicit permission from all destination addresses before you send an email to multiple recipients.

You may not assume that you have been granted permission by passive actions such as the posting of an article to Usenet or a visit made to Your web site.

Where You have acquired explicit permission, either on a web site or through some other relationship You should keep a record of this permission and must cease sending email when requested to stop.

An AUP clause to prohibit promotion of web sites using UBE

Web sites must not be advertised by You, or by another person, using techniques that would be classified as "abuse" if they were carried out using a service provided by Us. This includes, but is not limited to, the sending of bulk unsolicited email. Such action will be treated under this AUP as if it had been done using Your account.

An AUP clause to prohibit promotion of UBE tools and address lists

You must not offer or distribute any of the following products or services:

Contents

Appendix D: Definition of normative terms

This is a summary of the contents of RFC2119 "Key words for use in RFCs to Indicate Requirement Levels". Readers are encouraged to consult the full document for guidance.

MUST
This word means that the definition is an absolute requirement.
MUST NOT
This phrase means that the definition is an absolute prohibition.
SHOULD
This word means that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.
SHOULD NOT
This phrase means that there may exist valid reasons in particular circumstances when the particular behaviour is acceptable or even useful, but the full implications should be understood and the case carefully weighed before implementing any behaviour described with this label.
MAY
This word means that an item is truly optional.

This version prepared by Malcolm Hutty and Richard Clayton and approved by LINX Members as an authoritative statement of Best Current Practice on 17th August 2004. Version 1.0 of this document was prepared by Richard Clayton and approved by LINX Members on 18th May 1999.

Contents


Copyright © LINX 2004
Copyright © RIPE NCC 2006

The original LINX version of this document has certain references specific to the UK, and is available at http://www.linx.net/www_public/community_involvement/bcp/ubebcp_v2