Write Your Own POP3 Service

So, you want to write a POP3 service? That’s great. In this post, we’ll walk through building a simple POP3 service that uses a folder full of EML files as a mailbox and serves them to anyone logging in.

Getting Started

I’m assuming you are already set-up to be writing and building C# code. If you have Windows, the free version of Visual Studio 2019 is great. (Or use a more recent version if one exists.) Visual Studio Code is great on Linux too.

Download and build billpg industries POP3 Listener. Open up a new console app project and include the billpg,POP3Listener.dll file as a reference. You’ll find the code for this project on the same github in its own folder.

using System;
using System.IO;
using System.Collections.Generic;
using System.Net;
using System.Linq;
using billpg.pop3;

namespace BuildYourOwnPop3Service
{
    class Program
    {
        static void Main()
        {
            /* Launch POP3. */
            var pop3 = new POP3Listener();
            pop3.ListenOn(IPAddress.Loopback, 110, false);

            /* Keep running until the process is killed. */
            while (true) System.Threading.Thread.Sleep(10000);
        }
    }
}

This is the bare minimum to run a POP3 service. It’ll only accept local connections. If you’re running on Linux, you may need to change the port you’re listening on to 1100. Either way, try connecting to it. You can set up your mail reader or use telnet to connect in and type commands.

Accepting log-in requests.

You’ll notice that any username and password combination fails. This is because you’ve not set up your Provider object yet. If you don’t set one up, the default null-provider just rejects all attempts to log in. Let’s write one.

/* Add just before the ListenOn call. */
pop3.Provider = new MyProvider();

/* New class separate from the Program class. */
class MyProvider : IPOP3MailboxProvider
{
}

This won’t compile because MyProvider doesn’t meet the requirements of the interface. Let’s add those.

/* Inside the MyProvider class. */
public string Name => "My Provider";

public IPOP3Mailbox Authenticate(
    IPOP3ConnectionInfo info, 
    string username, 
    string password)
{
    return null;
}

Now, the service is just as unyielding to attempts to log-in, but we can confirm our provider code is running by adding a breakpoint to the Authenticate function. Now, when we attempt to log-in, we can see that the service has collected a username and password and is asking us if these are correct credentials or not. Returning a NULL means they’re not.

This might be a good opportunity to take a look at the info parameter. All of the functions where the listener calls to the provider will include this object, providing you with the client’s IP address, IDs, user names, etc. You don’t have to make use of them but your code may find the information useful.

A basic mailbox with no messages.

We can change our Authenticate function to actually test credentials. For our play project we’ll just accept one combination of user-name and password.

if (username == "me" && password == "passw0rd")
    return new MyMailbox();
else
    return null;

This will fail compilation because we’ve not written MyMailbox yet. Let’s go ahead and do that.

class MyMailbox : IPOP3Mailbox
{
}

Again, we’ll need to write all the requirements of the interface before we can run. So we can move on quickly, let’s provide just the minimum.

The first thing we’ll need is a list of the available messages. We’ll return an empty collection for now.

public IList<string> ListMessageUniqueIDs(
    IPOP3ConnectionInfo info)
    => new List<string>();

The service needs to know if a mailbox is read-only or not. Let’s say it isn’t.

public bool MailboxIsReadOnly(
    IPOP3ConnectionInfo info)
    => false;

The service might sometimes need to know is a message exists or not. For now, it doesn’t.

public bool MessageExists(
    IPOP3ConnectionInfo info,
    string uniqueID)
    => false;

The client might request the size of a message before it downloads it and the service will pass the request along to the provider. I’ve often suspected that clients don’t really need this so let’s just return your favorite positive integer.

public long MessageSize(
   IPOP3ConnectionInfo info, 
   string uniqueID)
   => 58;

The client will, in due course, request the contents of a message, but won’t because both the list-messages and message-exists will deny the existence of any messages, so for now, we can just return null.

public IMessageContent MessageContents(
    IPOP3ConnectionInfo info, 
    string uniqueID)
    => null;

Finally, we need to handle message deletion. Again, we don’t need to do anything just yet.

public void MessageDelete(
    IPOP3ConnectionInfo info, 
    IList<string> uniqueIDs)
{}

And we’re done. Run the code and log-in. Your mailbox will be perpetually empty but you can add breakpoints and confirm everything is running.

List the messages.

Now, let’s actually start with something useful. Let’s change our ListMessageUniqueIDs to return a list of filenames from a folder. You’ll want to replace the value of FOLDER with something that works for you.

const string FOLDER = @"C:\MyMailbox\";

public IList<string> ListMessageUniqueIDs(
    IPOP3ConnectionInfo info)
    => Directory.GetFiles(FOLDER)
           .Select(Path.GetFileName)
           .ToList();

public bool MessageExists(
    IPOP3ConnectionInfo info, 
    string uniqueID)
    => ListMessageUniqueIDs(info)
           .Contains(uniqueID);

Let’s also place an EML file into our mailbox folder. If you don’t have an EML file to hand, you can write your own using notepad. (It doesn’t care if the file has a “.txt” extension.)

Subject: I'm a very simple EML file.
From: me@example.com
To: you@example.com

Message body goes after a blank line.

If we save that into our mailbox folder and run up the POP3 service, we’ll see there’s a message available. It won’t be able to download it though.

Download the message,

The MessageContents function expects an new object that implements the IMessageContent interface.

/* Replace the MessageContents function. */
public IMessageContent MessageContents(
    IPOP3ConnectionInfo info, 
    string uniqueID)
{
    if (MessageExists(info, uniqueID))
        return new MyMessageContents(
                       Path.Combine(FOLDER, uniqueID));
    else
        return null;
}

/* New class. */
class MyMessageContents : IMessageContent
{
    List<string> lines;
    int index;

    public MyMessageContents(string path)
    {
        lines = File.ReadAllLines(path).ToList();
        index = 0;
    }

    public string NextLine()
        => (index < lines.Count) ? lines[index++] : null;

    public void Close()
    {
    }
}

This shows the requirements of the object that regurgitates a single message’s contents. A function that returns the next line, one-by-one, and another that’s called to close down the stream. The Close function could close opened file streams or delete temporary files, but we don’t need it to do anything in our play project.

Note that the command handling code inside this library has an extension that allows the client to ask for a message by an arbitrary unique ID. Make sure your code doesn’t allow, for example, “../../../../my-secret-file.txt”. Observe the code above checks that the requested unique ID is in the list of acceptable message IDs by going through MessageExists.

Delete messages.

The interface to delete messages passes along a collection of string IDs. This is necessary because the protocol requires that a set of messages are deleted in an atomic manner. Either all of them are deleted or none of them are deleted. We can’t have a situation where some of messages are deleted but some are still there.

But since this is just a play project, we can play fast and loose with such things.

public void MessageDelete(
     IPOP3ConnectionInfo info, 
     IList<string> uniqueIDs)
{
    foreach (var toDelete in uniqueIDs)
        if (MessageExists(info, toDelete))
            File.Delete(Path.Combine(FOLDER, toDelete));
}

What now?

I hope you enjoyed building your very own POP3 service using the POP3 Listener component. The above was a simple project to get you going.

billpg.com

Maybe think about your service could handle multiple users and how you’d check their passwords. What would be a good way to achieve atomic transactions on delete? What happens if someone deletes the file in a mailbox folder just as they’re about to download it?

If you do encounter an issue or you have a question, please open an issue on the project’s github page.

reddit’d (Followup to ‘Construct Something Else’)

Fame at last! Fame at last!

My last piece, “Construct something else!” got a bit of attention when someone posted it on reddit.That was unexpected.

Remember the rule; If you publish something that’s a bad idea in hindsight, post a “clarification” article claiming you’ve been misunderstood and that you never thought it was a good idea in the first place. Then hide in the shower.

You see, I think I’ve been misunderstood. I was reading stackoverflow and I found the question asking about c# constructors. There was the comment from Eric Lippert, talking about the possibility of implementing this feature, but they were lacking a good reason to undertake the effort. Then I remembered I had exactly what he was looking for, a real-world use case! So I wrote up my experiences in a blog post and left a comment on the stack overflow question.

I thought I was rather clear that I was just providing Mr Lippert with a use case, rather than actually advocating it. Nonetheless, some people mistakenly took my post as advocacy and responded as such. Now if you’ll excuse me, I’m going to go have a shower.

šŸ™‚

But seriously, I remain of the opinion that implementing pseudo-constructors would be a good thing, but probably not worth the time for Microsoft to implement. But first, a quick aside to clarify (there’s that word again) how it could work. Just so we’re all clear (!) on what it is I’m advocating.

A pseudo-constructor would essentially be a static factory function. Call it, and it returns an instance of the class, perhaps using a private real-constructor inside. The only difference being that it can be called using the new operator. The compiler sees that the parameters match the pseudo-constructor signature and it generates code to call that static function instead. From a MSIL/CIL view, it’s just like a normal static function.

So why would this be a good thing?

Changing the interface without changing caller code.

This is the reason I raised in my original post. If version 1 of a DLL has a real-constructor, version 2 can use a pseudo-constructor in it’s place. The caller code would have to be recompiled, but the C# code would not need to be modified.

Intellisenseā„¢ simplification.

How many times have you needed an instance of particular class, typed new ClassName, only for Intellisenseā„¢ to show that no constructors are available. You slap your forehead and remember that this class uses static factory functions instead. If these could be called with a new operator, they would all appear in the same list.

(I suspect this was the original motivation for considering the feature in the first place.)

That’s it?

There’s a few good reasons not to make this change, which I’ll briefly discuss. Enjoy.

They won’t be real constructors.
(Thanks to reddit user “grauenwolf”.)

Sometimes, only a real constructor will do. When writing a subclass constructor, you can call the base class’s constructor just before the first opening brace using the base keyword. This would have to be a real constructor call, as you can’t just decide which base class to use at run-time.

Adding pseudo constructors doesn’t take away real constructors, but it might lead to confusion when people see that a base classes constructors have gone missing.

You don’t need it.
(Thanks to “Anthony” for commenting on the original post.)

You can do all this by making a class full of delegate instances. The constructor can select what functions to fill into those delegates at run-time. Add some [Obsolete] attributes so anyone writing new code will code against the new preferred objects.


So I don’t think this new feature would break anything, except it would be taking up the time of the clever people at Microsoft. Nice to have, but we don’t need it.

If you’re in the mood for discussing future directions of the C# language, please take at look at my earlier piece on destructors for structs. I’m interested in any thoughts on the subject or any reasons why it wouldn’t work.

I hope I’ve gained a little bit of an readership from this experience. If you’re reading this, please leave a comment. Without comments, we’re just bumping around in a closed system and tending towards entropy. Here’s some nice charts for a bit of insight on the reddit people.


Picture credit:
“The Walk of Fame” by flickr user Storm Crypt.
Readership charts by blogger.

Construct something else! (C#)

Please read my follow-up post after reading this one.

Quoth rjw on stackoverflow

Given the following client code:

    var obj = new Class1();

Is there any way to modify the constructor of Class1 so that it will actually return a subclass (or some other alternate implementation) instead?

C# compiler guru, Eric Lippert commented…

We are considering adding a feature “extension new” which would essentially allow you to make a static factory method that would be called when the “new” operator is used, much as extension methods are called when the “.” operator is used. It would be a nice syntactic sugar for the factory. If you have a really awesome scenario where this sort of pattern would be useful, I’d love to see an example.

I have one!

Version one of our DLL had a class that wrapped a connection to a remote server.

    using (var connect = new ExampleConnection("service.example.com"))
    {
        connect.DoStuff(42);
    }

It worked great. Our customers were very happy with it and developed lots of code to use our little DLL. Life was good.

Time passes and our customers ask us to add support for a different type of server that does a similar job but with a very different protocol. No problem, we develop a new class called DifferentConnection and just to be helpful, both ExampleConnection and DifferentConnection implement a common interface object.

We’re about to release version two to our customers, but a common response comes back;

“This is good, but we were hoping your library would automatically detect which variety of server it’s talking to. Also, we really don’t want to change our code. We want to just drop your updated DLL into the install folder, but we’ll recompile our EXE if we really have to.”

With these new requirements, ExampleConnection had to become a class that supported both varieties of remote server. The constructor has to perform the auto-detect, and all of the public functions now all begin with an if statement, selecting for which variety of remote server is in use.

If we had a bit more foresight, we should have supplied a static Connect function that wrapped a private constructor. That way, version two of this function could have returned a subclass object instead. But we didn’t. There are costs to writing code that way, so you wouldn’t do it unless there was a clear point to it. If a normal constructor could return a subclass instead, there would be no problem.

Mr Lippert, I hope this provides the justification you need to add this to .NET 5, but I’d much rather have destructors on structs instead. I also want a pony.

Picture credit: ‘LEGO Mini Construction Site’ by flickr user ‘bucklava’.
(I don’t really want a pony.)

UPDATE: Someone submitted this to reddit. Lots of discussion there.
UPDATE(2): Follow-up post.

Wishing for a destructor (C#)

 

I like the C# programming language. It feels like C++ done right, divesting itself of much of the C legacy that complicates matters so much. When I do programming, I prefer to use this language. Having to go back and deal with C++ just doesn’t give me that warm feeling like it used to.

But, I have a pet peeve that I miss from C++.

Choose… Choose the form of the Destructor!

Quick recap. Here’s a brief C++ function…

 void MyFunction()
{
string x;
MyOtherFunction(x); /* Pass by value. */
}

Doesn’t look like much is going on, but there’s five function calls in there.

  1. A constructor function is called to build x.
  2. A copy constructor function is called to copy x for the function call.
  3. ‘MyOtherFunction’ is called.
  4. A destructor function is called to tidy up the copy of x.
  5. The same destructor function is called to tidy up x itself.

The clever bit is that the compiler has worked out when objects go “out of scope” and inserted calls to that’s objects destructor function in exactly the right place. Even “anonymous” objects are tidied up. Say a function is called that returns an object, but the caller just ignores the return value. The compiler spots it and inserts the destructor call in just the right place.

C# doesn’t do this. Instead, from the very beginning of the language, unused objects are “garbage collected”. Every so often, some code will run that goes over everything built by the program and sees if its being used anywhere. Anything that can’t be traced to running code is removed. Doing it this way allows the programmer to share objects between two different areas of code, without having to worry about which one has responsibility for tidying up.

I imagine that when the very clever people at Microsoft designed the C# language, they had already decided to use garbage collection, and so concluded that this messing about with destructors was no longer needed. No need to insert a function call into code, just leave the object lying around and the garbage collector will deal with it.

This would all great if memory was the only resource we have to keep track of. Open file handlers, database connections, etc. All must closed in a deterministic manner, instead of at some unknown time in the future when memory is about to run out.

Microsoft didn’t leave us completely out on the branch, classes that need to be tidied up can be written to implement the IDisposable interface. This allows the using block to work.

 using (SqlConnection con = new SqlConnection(db))
{
/* Use con. */
} /* Dispose con. */

With the using block, just like with the C++ destructors shown above, the compiler inserts a call to the tidy-up function at the end of the block. Even if there’s a return or throw statement in the middle, it’ll make sure everything is tidied up when the code leaves the using block.

But why have the using block at all? If you forget to include the using block, the tidy-up code won’t be called (unless you invoke it manually) and you won’t even get a compiler warning. (You don’t get the warning for very good reasons which I won’t go into right now.)

Even when you use using correctly, adding a using block to a function means introducing an additional block, with all the block-visibility issues and additional indenting that implies.

Structs to the rescue

Fortunately, C# and .NET come with a type of object called structs. These are similar to classes except they are solid value types rather than references to data floating in the ether. The practical difference is that when a struct value is copied (such as when passed into a function as a parameter) and you change the value of the one of the copies, the other copy stays the same.

In contrast, when you copy a class value, you’re instead just making a copy of the reference, so both point to same data. Change the contents of one, and the other changes value too, because there is no “other”.

So what if, when a struct appears in code, it came with an automatic using block attached? That way, we could open files or database connections just by introducing one in code and it would be tidied up in a deterministic way.

To complete the job, we would need mechanisms to support copy constructors and assignment as well as the final destructor call, just like the C++ people are used to.

I’ve been nursing this peeve and whining about it for so long that I’m even boring myself. I plan this to be my last word on the topic and in future I’ll just post links to this article. Enjoy.

Picture credits
“staypuft_3feb2009_0621” by patrick h. lauke on flickr
“choose determinism” by alyceobvious on flickr
“John E. Cox Memorial Bridge” by Elizabeth Thomsen on flickr

is, then as (C#)

The C# language has a pair of related operators, is and as. Say you have an object, x, and you want to know if its really type Form underneath.

(x is Form) will be true if it is, or false if it isn’t. The as operator will actually perform the conversion, returning null if it can’t be converted to that type.

One common piece of advice to C# programmers is to avoid using the is operator. Here’s how both of these operators are typically used together.

    if (x is Form)
{
Form frm = x as Form;
/* Use frm. */
}

While the code works, all the is operator is doing is performing an as operation, and checking if the result is null or not. In other words, (x is Form) is equivalent to (x as Form != null). This means the code above is really saying…

    if (x as Form != null)
{
Form frm = x as Form;
/* Use frm. */
}

See that. The same as operation was done twice. What a waste! So, the standard advice is to re-write the code to use just one as operation.

    Form frm = x as Form;
if (frm != null)
{    
/* Use frm. */
}

Simple. So what’s the point of this article? Why use is at all?

The problem for me is that is just looks so darn nice. Beginners very quickly understand what’s going on when they see it. The re-written code that checks for null just doesn’t look as pretty and isn’t as obvious what’s going on.

You may be wondering what the cost of this additional operation is. I knocked together a quick test which looped is-then-as and then looped the optimized version the same number of times and reported the difference in time. When I increased the count to 100,000,000, I saw approx one second difference build up. That’s a lot of looping.

But hey, that cost is non-zero, and I’m wondering why. Compilers are pretty good these days. I recall advice from long ago that its better to use an XOR operation to check if two values are equal, or to use shift operations instead of multiplying. These days, best practice to write what you mean and leave micro-optimizations to the compiler. So why doesn’t Microsoft’s C# compiler optimize is-then-as into a single operation?

I don’t know. It could be that its not worth the development team’s time, or maybe there’s a subtle reason that escapes me right now. I wish I knew. But if we could eliminate the repeated operation and use is freely, code would be lot nicer to look at.

Picture credit: Fruit balance by Pickersgill Reef.
Many thanks to Eric Lippert and the commentators to his article that inspired this piece.
Update 18/Jan/2010: Introduction modified to be better understood by non C# programmers. (Thanks Andrew.)