Comfortably Coding

The Analysis & Fix of an annoying MythTV Bug

admin — Sat, 16 Jan 2010 01:07:21 GMT

A couple of days ago, I upgraded my MythTV frontend to use an SSD drive. It’s been great, it’s very quiet and prodeces a low amount of heat. I also used this as an opportunity to refresh the system by rebuilding it anew. I backed up /etc/ and ~’s and went about rebuilding the system.

Unfortunatly, once I finished and tried the new system, things were not quite smooth and I was hitting a really annoying bug with my infrared remote used to control the system. The remote would only work in MythTV 2 minutes after startup. Wow, how annoying… A few quick commands showed me that this was nothing wrong with the drivers (which I had to patch myself), or lirc as irw reported the button presses coming through just fine. So my conclusion was that MythFrontend was doing something odd.

Beginning the Investigation

I was running mythtv–0-22-fixes, so I fired up SVN and checked out the source for that release

svn co http://svn.mythtv.org/svn/branches/release-0-22-fixes/ mythtv-0-22

Armed with the fantastic power of x-ray-googles, I began looking at how MythFrontend picks up LIRC events. A lot of this magic is in the sources in mythtv/libs/libmythui/*.
In mythmainwindow.cpp, Myth initialises a thread which reads from LIRCD on whichever UNIX socket or tcp/ip socket you have it setup from (this is in MythMainWindow::StartLIRC())

This begins a thread which has a its runloop defined as LIRC::run() in lirc.cpp. However, before this is called, LIRC is initialised by LIRC::Init(). Needless to say, all of this was performing successfully and Myth, for all intents and purposes was reading my keypresses from /dev/lircd which were picked up from my remote by LIRCD just fine (irw prooved this).

At this point I moved further down the stack. When a code is received from LIRCD, the LIRC thread calls LIRC::Process(data), passing it that keypress information. This method will do some logic and then send a KeyPress and a KeyRelease to the MythFrontend main window


            QApplication::postEvent(
                m_mainWindow, new LircKeycodeEvent(
                    QEvent::KeyPress, keycode, mod, text, lirctext));
            QApplication::postEvent(
                m_mainWindow, new LircKeycodeEvent(
                    QEvent::KeyPress, keycode, mod, text, lirctext));

and later

        for (int i = (int)keyReleases.size() - 1; i>=0; i--) {
            QApplication::postEvent(m_mainWindow, keyReleases[i]);
        }

Some quick working with a debugger told me that these functions were being called. At this point I began to wonder if the mainWindow was receiving these events. I tracked the callback handler for these events to libs/libmythui/mythmainwindow.cpp, to the customEvent method.

void MythMainWindow::customEvent(QEvent *ce)

Now, this is where it gets interesting. This customEvent method was being called for all Keypresses, HOWEVER it was ignoring some! - Aha, my problem was here somewhere (or so I thought…).

Inside this customEvent method, there is a large conditional to deal with a number of cases, one of which deals with LIRC keycode events:

    else if (ce->type() ==
             (QEvent::Type) LircKeycodeEvent::kLIRCKeycodeEventType &&
             !d->ignore_lirc_keys)
    {

It turns out that this was not being called, which was why I was experiencing my problem (but not the root cause). Now, the smart readers here will notice the !d->ignore_lirc_keys condition. This was causing the problem. For some reason ignoe_lirc_keys was true for the first 2 minutes after starting up… What… why would this be? Well, to find out I had to figure out how ignore_lirc_keys could be set to true.

Setting ignore_lirc_keys to true

Some plodding around in the source code revealed that there is an LircEventLock::Lock() method in lircevent.cpp. This was being called by someone, causing mythfrontend to ignore LIRC keypresses. To get a simple backtrace at this point, I modified the code to include

#include <execinfo.h>

so that I can call backtrace() and backtrace_symbols(). This allowed me to get a stack trace at the point the Lock was set, and see who might be calling Lock, and annoying the hell out of me. Here is the result of that stack trace:

Obtained 10 stack frames.
/home/user/sandbox/my-myth-build/lib/libmythui-0.22.so.0(_Z11print_tracev+0x26) [0x5f7796]
/home/user/sandbox/my-myth-build/lib/libmythui-0.22.so.0(_ZN13LircEventLock4lockEv+0x1f3) [0x5f7a03]
/home/user/sandbox/my-myth-build/lib/libmythui-0.22.so.0(_ZN13LircEventLockC1Eb+0x25) [0x5f7bf5]
/home/user/sandbox/my-myth-build/lib/libmythui-0.22.so.0(_Z11myth_systemRK7QStringi+0x48) [0x5a9198]
/home/user/sandbox/my-myth-build/lib/libmyth-0.22.so.0(_ZN15MythMediaDevice15performMountCmdEb+0x58e) [0x669807e]
/home/user/sandbox/my-myth-build/lib/libmyth-0.22.so.0(_ZN7MythHDD10checkMediaEv+0x98) [0x67623e8]
/home/user/sandbox/my-myth-build/lib/libmyth-0.22.so.0(_ZN12MediaMonitor12CheckDevicesEv+0x5e) [0x669a66e]
/home/user/sandbox/my-myth-build/lib/libmyth-0.22.so.0(_ZN13MonitorThread3runEv+0x38) [0x669a708]
/usr/lib/libQtCore.so.4 [0x59cce32]
/lib/tls/i686/cmov/libpthread.so.0 [0x64e80e]

As you can see, Lock() is called by myth_system, which is called by MythMediaDevice::performMountCmd, called by MythHDD::checkMedia etc…

BINGO

Armed with this knowledge, I knew exactly what was happening. The myth_system() is used to execute commands, however when these commands execute, sometimes we want mythfrontend to ignore LIRC commands sent to it. For example, if myth_system executes mplayer, which is now in-front of mythfrontend, we should ignore LIRC keypresses on Myth. However, the mount command also caused mythfrontend to Ignore Keypresses while mount was being run.

But why two minutes? Because I have a cardreader which registers 4 SCSI devices which cannot be mounted without cards in them. if you try and mount them, the mount command waits an amount of time (sometimes up to 30 seconds, it seems for me) then gives up.

The sdb, sdc, sdd and sde devices in /dev

user@kmedia2:~/sandbox/my-myth-build$ ls /dev | grep -e "^sd"
sda
sda1
sda2
sda5
sdb
sdc
sdd
sde
user@kmedia2:~/sandbox/my-myth-build$

dmesg

[    7.064559] usb-storage: device scan complete
[    7.101340] scsi 8:0:0:0: Direct-Access     Generic  STORAGE DEVICE   9602 PQ: 0 ANSI: 0
[    7.107230] scsi 8:0:0:1: Direct-Access     Generic  STORAGE DEVICE   9602 PQ: 0 ANSI: 0
[    7.120038] scsi 8:0:0:2: Direct-Access     Generic  STORAGE DEVICE   9602 PQ: 0 ANSI: 0
[    7.148029] scsi 8:0:0:3: Direct-Access     Generic  STORAGE DEVICE   9602 PQ: 0 ANSI: 0
[    7.148391] sd 8:0:0:0: Attached scsi generic sg2 type 0
[    7.148466] sd 8:0:0:1: Attached scsi generic sg3 type 0
[    7.148544] sd 8:0:0:2: Attached scsi generic sg4 type 0
[    7.148619] sd 8:0:0:3: Attached scsi generic sg5 type 0
[    7.447250] sd 8:0:0:1: [sdc] Attached SCSI removable disk
[    7.458246] sd 8:0:0:3: [sde] Attached SCSI removable disk
[    7.469466] sd 8:0:0:0: [sdb] Attached SCSI removable disk
[    7.491250] sd 8:0:0:2: [sdd] Attached SCSI removable disk

the mount command

root@kmedia2:/mnt# time mount /dev/sdb trash
mount: /dev/sdb: unknown device

real    0m18.158s
user    0m0.004s
sys     0m0.000s

By the way, if I rmmod -f usb-storage, which removes these devices in /dev, mythtv starts fine.

The Simple Fix

So, armed with the known cause of the issue, and the fact that myth_system accepts a flag called MYTH_SYSTEM_DONT_BLOCK_LIRC, which causes the system() command not to block LIRC keypresses on the front end, I added a small patch.

I changed line 124 of libs/libmyth/mythmedia.cpp from

        if (0 == myth_system(MountCommand))

        if (0 == myth_system(MountCommand, MYTH_SYSTEM_DONT_BLOCK_LIRC))

I compiled the libmyth library, and everything was OK… the bug was gone!

Today… I played with Codility

admin — Wed, 13 Jan 2010 11:55:47 GMT

Well this seems pretty cool,�http://codility.com

Although, it seems to just quiz you about basic�asymptotic�growth and does not seem to look at things like programming style etc. Still, I’m please my Ruby answer for the demo test got 100%, which, as the histogram shows… Not many people get. I suspect that the reason is, many people probably submit an O(n^2) answer, which fails for very large data sets. Here is what I submitted

And here is what I submitted:

def equi(arr)
  lhs = 0
  tmp = 0
  rhs = arr.inject(0) { |a,b| a + b }
  arr.each_index do |i|
    lhs += tmp
    rhs -= arr[i]
    tmp = arr[i]
    return i if lhs == rhs
  end
  return -1
end

And Here’s the analysis it gave back to me:

test	time	result
example Test from the task description	0.012 s.	OK
extreme_empty Empty array	0.020 s.	OK
extreme_first	0.008 s.	OK
extreme_large_numbers Sequence with extremly large numbers testing arithmetic overflow.	0.020 s.	OK
extreme_last	0.012 s.	OK
extreme_single_zero	0.020 s.	OK
extreme_sum_0 sequence with sum=0	0.020 s.	OK
simple	0.016 s.	OK
single_non_zero	0.012 s.	OK
combinations_of_two multiple runs, all combinations of {-1,0,1}^2	0.008 s.	OK
combinations_of_three multiple runs, all combinations of {-1,0,1}^3	0.012 s.	OK
small_pyramid	0.072 s.	OK
large_long_sequence_of_ones	13.349 s.	OK
large_long_sequence_of_minus_ones	10.781 s.	OK
medium_pyramid	4.336 s.	OK
large_pyramid Large performance test, O(n^2) solutions should fail.	21.369 s.	OK

Dynamic Image Reflection

admin — Sun, 08 Feb 2009 07:26:00 GMT

Todays post is about creating a mirrored effect on images appearing on a website. Looking around, there are already a few people who have attempted to do this, however, I would like to tackle the problem myself and provide an in depth post about how to actually do it. At the end of this post we should have a fully functioning jQuery plugin which allows us to mirror images. We can approach this two ways, using Canvas or SVG. I have opted to use Canvas, however I did create a neat SVG file which is able to mirror arbitrary images, passed in by a query parameter. The first part if this post will address creating a reflection of an image. Once we have the ability to do this, we can roll it into a jQuery plugin to provide reflections for images which we specify.

Creating a Reflection
The approach I have taken for reflection uses the HTML canvas element. The Canvas element was originally introduced by Apple into their WebKit component and allows for a drawable region in HTML. The Canvas element exposes a full set of drawing functions to the developer to draw inside of it. We can utilise these functions to mirror an image, complete with a ‘fading’ effect. Assuming we already have a canvas DOM element, we can retrieve the 2d drawing context with a call to getContext(’2d’) on the DOM element. This can then be used to draw to the canvas via a published API. If you wish to know more the Mozilla Developer Center has published some tutorials on using Canvas.

In order to achieve our mirror effect, we must ‘flip’ the vertical of the image. We can achieve this using the scale function exposed to us in the canvas API, context.scale(1, -1). This will effectively ‘flip’ what we draw, or, more technically add a ‘flip’ transform to the translation matrix of our canvas element. The Mozilla Developer Center again provides some useful resources regarding the use of scale. However, this documentation states that scale() may only take positive arguments.

With a little experimentation however, I revealed that it actually works as expected for negative augments. Quite naturally I followed this up with some source code digging.
For WebKit, in CanvasRenderingContext2D.cpp line 326 we see the definition for the scale function, as provided to us by the 2d drawing context in WebKit. All this does is add a scale transformation to the current transformation matrix of the canvas. If this is negative, it will cause a ‘flip’ of the image.
In Firefox we find in nsCanvasRenderingContext2D.cpp on line 877 the definition for scale. This simply calls cairo_scale, with the supplied parameters.�Cairo is the Firefox graphics backend, so it makes quite a bit of sense that this actually happens and cairo_scale accepts negative parameters as well.

But I digress. We may achieve a reflected image with the following code (assuming we have the 2d context from the canvas (canvas) and an image we want to mirror (image)):

canvas.translate(0, image_height);canvas.scale(1, -1);
canvas.drawImage(image, 0, 0, image_width, image_height);
canvas.restore();

This will render a mirrored form of the image, flipped on the horizontal axis. However, many mirror effects also feature a gradual fading of the mirrored image. In order to add this we add a mask on top of our flipped image, using the ‘destination-in’ globalCompositeOperation:

canvas.globalCompositeOperation = "destination-in";
var gradient = canvas.createLinearGradient(0, 0, 0, height);
gradient.addColorStop(1, "rgba(255, 255, 255, 1.0)");
gradient.addColorStop(0, "rgba(255, 255, 255, 0.0)");
canvas.fillStyle = gradient;
canvas.rect(0, 0, width, height);
canvas.fill();

This now completes our mirror image effect, fading out to black as the distance from the horizontal flip axis increases. In order to make this functionality, cleaner, more modular & reusable and thus more useful to us, we make this functionality available as a jQuery plugin. The goal should be that one only needs to call $(’.mirror’).mirror() to mirror all images selected by the selector (in this case, all images with a class of ‘mirror’).��In order to�achieve�this, we turn our approach into a jQuery extension, and add the function mirror(). As I have explained the general method, I will provide a simple example for a jQuery plugin:

jQuery.fn.mirror = function() {
  /*
  * jQuery functions should return 'this', to conform to the fluent interface.
  * We also iterate through all img elements.
  */
  return this.find("img").each(function(i, e) {

    /* Our jQuery Element */
    var je = jQuery(e);

    /* For a number of calculations, we require the width and the height of the image */
    var width = je.attr('width');
    var height = je.attr('height');

    /*
    * Create a div to contain our original image
    * and its reflection.
    */
    var div = jQuery("<div></div>")
    .css('position', 'relative')
    .css('width', width)
    .css('height', height * 2);

    /* create a canvas for our relection */
    var canvas = jQuery("<canvas></canvas>")
    .attr('width', width)
    .attr('height', height)
    .css('width',  width)
    .css('height', height)
    .css('position', 'absolute')
    .css('left', 0)
    .css('top', height);

    /*
    * Replace the image with our div.
    * This line is magic for taking the image out
    * putting the image & canvas in the div,
    * then putting the div in the images place.
    */
    div.append(je.replaceWith(div)).append(canvas);

    /*
    * Now we must do our voodo with canvas to make a
    * reflection of the image
    */
    canvas = canvas.get(0).getContext('2d');
    je.load(function() {
      canvas.translate(0, height);
      canvas.scale(1, -1);
      canvas.drawImage(e, 0, 0, width, height);
      canvas.restore();
      canvas.globalCompositeOperation = "destination-in";
      var gradient = canvas.createLinearGradient(0, 0, 0, height);
      gradient.addColorStop(1, "rgba(255, 255, 255, 1.0)");
      gradient.addColorStop(0, "rgba(255, 255, 255, 0.0)");
      canvas.fillStyle = gradient;
      canvas.rect(0, 0, width, height);
    canvas.fill();
    });
  });
}

Ruby inside .NET and Java

admin — Tue, 27 Jan 2009 03:07:57 GMT

In the past I have played around with JRuby and accessing some of the Java framework from Ruby. Its fantastic and I have raved about it infront of my friends in the past.� Microsoft as well now have a product called IronRuby which runs Ruby inside Microsofts Dynamic Language Runtime (DLR). IronRuby is a Ruby interpereter which runs in the DLR and as a byproduct gives Ruby access to the .NET framework (yay, power to the programmer!).

I just thought I would blog about this little code snippet I put together on the weekend for some fun. It retrives the IP addresses for the interfaces on the machine it is run on.� If it is run in JRuby (inside the JVM) it uses the Java framework to retrieve the interfaces and print them. If it is running in IronRuby (inside the DLR) it uses the .NET framework to retreive the IP addresses. Finally, if it is not running in either of those it reports that we are not running in either the JVM or the DLR and exits.

I suspect this may be a handy thing to do if you have common tasks which you perform accross the two different environments (Java and .NET) as you dont have to expose functionality from inside your environment, you can intergrate directly into it using the same language (with a bit of added abstraction on your part)

path = {
    :DotNet => lambda {
        require 'System.Management, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
        mc = System::Management::ManagementClass.new("Win32_NetworkAdapterConfiguration")
        moc = mc.GetInstances()
        moc.each { |v| p v.GetPropertyValue("ipAddress") if v.GetPropertyValue("ipEnabled") }
    },
    :Java => lambda {
        java.net.NetworkInterface.networkInterfaces.each { |v| puts v }
    },
    :None => lambda {
        puts "Sorry, you are not running in the JVM or DLR"
    }
}

begin
    # If we are in IronRuby we have access to the .NET classes
    require 'mscorlib'
    call = :DotNet
rescue Exception => e
    begin
        include Java
        call = :Java
    rescue Exception => e
        call = :None
    end
end
path[call].call()

Mail Server with Dovecot and Exim4

admin — Mon, 19 Jan 2009 06:43:52 GMT

I am changing my hosting around at the moment and have a new server, that is completely managed by myself. One of my first requirements is to setup up mail hosting for my domains. Since I am not using any sort of control panel or packaged software, I have to configure this by hand. My needs are simple: virtual users, TLS/SSL security and to run with a small footprint. Further down the track, I also plan on adding virus and spam filtering.

Software
First off, I had to choose the software I would use. I elected t use Dovecot for my IMAP server and Exim4 for my MTA. Dovecot is an excellent choice for an IMAP server as it is designed with security in mind, is simple to configure has a low memory footprint, performs excellently and supports the Maildir format which I intended to use. Exim4 is a good MTA, with a simple and clean configuration (relative to Sendmail!) and is designed with consideration to security.

My server is running Debian Etch, so installation of these packages through apt is easy. However, I had to install Exim4 from Backports as version 6.64 and later include an authentication driver for dovecot that allowing Exim4 to authenticate SMTP sessions against the Dovecot user database. The version of Exim4 that ships with Etch is 6.63, which means I either had to compile a new version from source or use Exim4 from Backports. I elected to use Backports. Installing the required software is easy, once you have the etch-backports repository (note, exim4-daemon-heavy has the extra Dovecot authentication driver):

sudo apt-get -t etch-backports install exim4-daemon-heavy
sudo apt-get install dovecot-imapd

Configuration
I started by configuring Dovecot so I could access a users Maildir through IMAPS. Because all users would be virtual, the Maildir’s would be stored in /home/vmail/$user@$domain/Maildir/ and owned by vmail:vmail. It should be noted here that the Maildir’s have the executable permission set on them so that Exim4 can validate that the user exists. Failure to do so will cause Exim4 to report an Unroutable Address error.

The contents of my /etc/dovecot/dovecot.conf contains a line by line description in comments, and is more or less pretty painless:

# Protocols (others available imap, pop3, pop3s)
protocols = imaps

# Log files for Dovecot
log_path = /var/log/dovecot/main.log
info_log_path = /var/log/dovecot/info.log

# SSL settings (I already generated my certificate, if you don't know how visit http://letmegooglethatforyou.com/?q=openssl%20generate%20certificate).
ssl_listen = *
ssl_disable = no
ssl_cert_file = /etc/ssl/certs/example.crt
ssl_key_file = /etc/ssl/private/exmaple.key

# Auth Settings (do not allow plaintext auth (except when under SSL/TLS)
disable_plaintext_auth = yes

# We are not really a very busy imap server, so we dont need a large number of processes waiting for logon
login_processes_count = 1

# So we get information about processes (in ps).
verbose_proctitle = yes

# Login process can auth multiple connections (less secure then yes, but I'm aiming for less overall processes)
login_process_per_connection = no

# We're using Maildir format
mail_location = maildir:~/Maildir

# we want group xwrx--x-- so exim can validate the addresses (it does a stat on /home/vmail/local_part@domain_part/
umask = 0066

# Not required for me, but in there in case I ever want to support pop3.
# Outlook 2003 has problems with the default, while %08Xu%08Xv is regarded as more or less safe.
pop3_uidl_format = %08Xu%08Xv

# 108 is my vmail user id. If this is too high, Dovecot will not allow the user to read/write the Maildir's
first_valid_uid = 108

# Optimizations
maildir_copy_with_hardlinks=yes

# Authentication Config
auth_verbose = yes

auth default {
    mechanisms = plain login
    passdb passwd-file {
        args = /etc/dovecot/passwd
    }
    userdb static {
        args = uid=vmail gid=vmail home=/home/vmail/%u
    }

    socket listen {
        # This allows authentication to take place between lda (the dovecot deliver) and dovecot
        master {
            path = /var/run/dovecot/auth-master
            mode = 0600
            user = vmail
            group = vmail
        }

        # This allows Exim4 to authenticate against Dovecot's user database
        client {
            path = /var/run/dovecot/auth-client
            mode = 0666
            user = vmail
            group = vmail
        }
    }
}

# deliver Configuration
protocol lda {
    postmaster_address = postmaster@example.org
    log_path = /var/log/dovecot/deliver.log
    info_log_path = /var/log/dovecot/deliver.log
    auth_socket_path = /var/run/dovecot/auth-master
}

This then leaves my user configuration in /etc/dovecot/passwd formatted as follows:

#
# Example.org
#
user1@example.org:{PLAIN}plaintxtpass
user2@example.org:{SSHA}mp9sL0Xl1kqi2N/3g/E6qU/PTtV/+qEx

To generate passwords other then plain, use the dovecotpw program, e.g. dovecotpw -s SSHA

Configuring Exim4 is slightly tricker, also note that I am using the Debian split configuration here, so if you are using the monolithic single file configuration, you will have to figure out where things go yourself.

First either update your update-exim4.conf.conf by hand and issue the `update-exim4.conf` command or use `dpkg-reconfigure exim4-config` to configure your install as a Internet mail server listening on a public interface and to accept email for the domains you plan to host email for. Once this was done I began to configure Exim4 to work the way I wanted. In order to do this I had to do a few things.

Configuring TLS
If you do not already have a certificate you may use `/usr/share/doc/exim4-base/examples/exim-gencert` to generate the exim.crt and exim.key files which are placed in /etc/exim4. Once I had my certificates I set the MAIN_TLS_ENABLE macro to true. I did this by creating the file /etc/exim4/conf.3/main/00_local containing:

MAIN_TLS_ENABLE = true

Configuring Authentication with Dovecot
As I said before, Exim4 can be used to authenticate against Dovecot. In order do do this I created the file /etc/exim4/conf.d/auth/20_dovecot and inside placed:

dovecot_login:
driver = dovecot
public_name = LOGIN
server_socket = /var/run/dovecot/auth-client
server_set_id = $auth1
dovecot_plain:
driver = dovecot
public_name = PLAIN
server_socket = /var/run/dovecot/auth-client
server_set_id = $auth1

You should notice that the server_socket corresponds to my configuration in the Dovecot config file.

Alias & Virtual Emails Routing
My actual email accounts do not correspond to system users and as such Exim4 needs a mechanism to know if an email is valid (and how to deliver it, however I deal with that in the next section). As well as this, I wanted to allow aliases to be created, for example alias@example.org is an alias for user1@example.org. In order to do this I created both an alias router and a router to my virtual mailbox’s (my Dovecot configuration expects the mailboxes to be in /home/vmail/user@domain/Mailbox).

I first created a router for Dovecot in /etc/exim4/conf.d/router/880_dovecot_router. It must be noted that this sends it to a transport called dovecot_transport. This is not yet created, however I will create one shortly. Also, we allow the local part to have a suffix, this allows us to route emails such as user1+emailsuffix@example.org which you may wish to mark certain subscriptions with.

dovecot_router:
driver = accept
local_part_suffix = +*
local_part_suffix_optional
require_files = +/home/vmail/${local_part}@${domain}/Maildir
transport = dovecot_transport

I also created a virtual alias router to allow me to specify mappings, such as a catchall for a domain *@example.org -> user1@example.org or aliases such as alias@example.org -> user1@example.org. To do this I created the file /etc/exim4/conf.d/router/870_virtual_aliases

virtual:
driver = redirect
allow_defer
allow_fail
data	= ${lookup{$local_part@$domain}lsearch*@{/etc/exim4/virtual.users}}
domains	= partial-lsearch;/etc/exim4/virtual.domains
retry_use_local_part

Along with this, I also maintain two files /etc/exim4/virtual.users that specifies my aliases:

#
# Example.org
#
alias@example.org    : user1@example.org
*@example2.org       : user2@example.org

And /etc/exim4/virtual.domains which specifies any domain with a virtual alias:

example.org
example2.org

Virtual Email Delivery
In order to deliver mail to my virtual user’s Maildir, which will allow Dovecot to read them I had to create my own transport. There are two ways to accomplish this. I could deliver to the Maildir directly or use Dovecot’s deliver binary. Dovecot’s deliver will do magical things such as update Dovecot’s transparent indexes, so this is the approach I took, and would recommend. I will however also provide the Maildir transport but commented out. I defined my dovecot_transport in /etc/exim4/conf.d/transport/36_dovecot_transport:

#dovecot_transport:
#  driver = appendfile
#  user = vmail
#  group = vmail
#  mode = 0600
#  directory=/home/vmail/${lc:$local_part}@${lc:$domain}/Maildir/
#  maildir_format = true
#  mode_fail_narrower = false
#  envelope_to_add = true
#  return_path_add = true

dovecot_transport:
driver = pipe
command = /usr/lib/dovecot/deliver -d $local_part@$domain
message_prefix =
message_suffix =
log_output =
delivery_date_add
envelope_to_add
return_path_add
group = vmail
user = vmail

Rebuilding Configuration& Restarting
With all this done, I rebuilt the exim4 configuration file with `update-exim4.conf.template` and restarted both Dovecot and Exim4. They should be now configured to process and serve Email together:

sudo update-exim4.conf.template -r
sudo /etc/init.d/dovecot restart
sudo /etc/init.d/exim4 restart

Testing
You may also test your configuration with a few commands in order to work out how and where email is being delivered to, I recommend you try this on an alias. You should see it map to a virtual address.

To run exim in address testing mode (pass the -d flag for debugging information)

exim -bt 'user1@example.org'

To run exim in address verification mode (pass the -d flag for debugging information)

exim -bv 'user1@example.org'

And lastly, you are able to use SMTP directly by starting a telnet session to port 25 and testing things out. It should appear something like this (the lines without the prefixed numbers are what I type):

khaless@galactica:~$ telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.localdomain.
Escape character is '^]'.
220 galactica ESMTP Exim 4.69 Mon, 19 Jan 2009 01:38:29 -0500
EHLO example.org
250-galactica Hello localhost.localdomain [127.0.0.1]
250-SIZE 52428800
250-PIPELINING
250-AUTH LOGIN PLAIN
250-STARTTLS
250 HELP
MAIL FROM: user1@example.org
250 OK
RCPT TO: user2@example.org
250 Accepted
DATA
354 Enter message, ending with "." on a line by itself
Subject: My Test Email

Hello, I am a test email.

.
250 OK id=1LOnmh-0006Hx-Vt
quit
221 galactica closing connection
Connection closed by foreign host.
khaless@galactica:~$

Tsukasa’s PyGolf Challenge

admin — Sat, 06 Dec 2008 00:53:16 GMT

Greg recently offered a challenge: Python Golf. The idea, to create a parser for Apache’s Common Log Format, to aggregate the amount of data sent to each IP address. The challenge being to do it with the least amount of characters. Soon after this challenge was issued, Sam and Greg were both convinced (before even talking to me) that I would attempt a Ruby solution. How could I dissapoint them? :).

I tried a number of things, however I eventually settled on this solution (63 chars):

m=Hash.new(0);STDIN.map{|l|m[l[/^\S+/]]+=l[/\d+\s+$/].to_i};p m

A slightly longer version, with a slightly prettier printing can be accomplished with inject (78 chars):

STDIN.inject(Hash.new(0)){|m,l|m[l[/^\S+/]]+=l[/\d+\s+$/].to_i;m}.each{|v|p v}

Not to brag, but the winning Python entry was 121 characters. So the Ruby solution managed to be almost half this count!

Some things for the debugging toolbox

admin — Fri, 10 Oct 2008 04:39:56 GMT

Reading Steve’s blog today I saw his post on accessing the current functions name in python. This could end up being a useful thing to know if you are� playing with some unfamiliar code and need this sort of information, or are doing something nasty. So without anymore fuss here’s how to do the same thing in Ruby and JavaScript!

JavaScript

We can access the current functions name through arguments.callee.name, in fact to take this a bit further, arguments.callee is the function actually being called (so we can recurse in anonymous functions!

function NamedFunction() { console.log(arguments.callee.name); }
var fn = NamedFunction;
fn();

Outputs: “NamedFunction”

Ruby

In Ruby you can use the caller method of the Kernel module (all objects have this). caller returns an array of strings representing the stack and we can use this (as Ruby Facets does) to find out the current functions name. Ruby Facets extend the Kernel module with `called`:

module Kernel
def called
/`([^']+)'/.match(caller(1).first)[1].to_sym
end
end

and one can now write

def namedFunction
puts called
end
namedFunction()

Outputs “namedFunction”

JavaScript Stack trace

Getting� a stack trace in JavaScript is a fun task, because browsers are not the same, but here are some quick tips. When Firefox throws an exception, that exception has a ’stack’ member which contains a stack trace. For example:

try { var x = nil; x.x; } catch(e) {console.log(e.stack); }

Opera’s exceptions contain the ‘message’ member which contains stack information. And finally IE and Safari don’t provide anything, but we can roll our own using arguments.callee.caller, which is as you guessed it the function which has called the current function. So we can roll our own!

function stack() {
var fn = arguments.callee;
while(fn = fn.caller) {
console.log(fn.name || "Anonymous");
}
}
function a() { stack() }
function b() { a() }
function c() { b() }
c();

Outputs: “a b c”

JavaScript & recursing anonymous functions

And just a little icing on the cake for the end of this post. Take some time and run this in a JavaScript Shell

var x = 0;
(function() {
console.log(x++);
if(x < 10) arguments.callee();
})();

Avoiding Bad Email Communication

admin — Fri, 03 Oct 2008 05:25:12 GMT

It often occurs that once I subscribe to a website, they wants to communicate with me via email. This may be information about my account, confirmation of some process or an occasional� email promoting some new feature or deal. In many of these cases websites may choose to send an email in which the content is presented in HTML. HTML may appear like a good choice to someone who only thinks about bells and whistles, however in reality there are a number of issues. Off the top of my head, here is a list:

Problem 1 - Not everyone can read HTML

There is a portion of your subscribers who will receive emails that are unable to render the HTML. In this case, if all they see is HTML it is probably safe to assume they cannot read it, and to them this email is worthless.

Problem 2 - HTML typically uses more bandwidth & CPU

Remember that users are becoming increasingly mobile. I really don’t want to be waiting 30 seconds to download the larger email, possibly with images, and wasting my battery life on my mobile device just because you want HTML, consider this growing portion of users and provide an alternative for them.

Problem 3 -� It is hard to create good HTML Emails

Most email clients use old rendering engines which don’t support many new features. Adding to this the variation in so many of these clients, to write a HTML email that will even display the same in most mail clients is a challenge. Restrict yourself mostly to older flavours of HTML, if you do need to use CSS, use inline styles and keep everything simple.

Problem 4 - Developers struggle with Mime Types

All to often I have received a simple communication, provided to me only in HTML. Please, if you insist on a HTML communication� provide a text only alternative for people who cannot read HTML emails. Its not hard, emails can be sent containing two forms, HTML and Plain Text and the reader can choose which one they want to view.

Problem 5 - Sloppy HTML & big images appears like spam

Many email communications are put together by someone with little to no experience in HTML. They often use a WYSIWYG editor to generate formatted content, which will be placed in a template and then sent. However some of these WYSIWYG editors are known for producing sloppy HTML which can be a flag to a spam filter. Compounding this issue further is that HTML email’s with many images, or a single large image are bound to be classed as spam!

Mathews Rule of Thumb

I understand that companies like the idea of a HTML email as it allows them to effectively brand their communication by including elements such as their colours, styles and logo. But it may also exclude a large number of recipients from reading their email. So with this in mind, the general advice I give is:

If the email is a result of the user interacting with the website i.e. signing up, receiving a message, submitting a form, use Plain Text as this is most likely an important email which must be readable by the recipient. However, if the email is a form of passive communication, where the user may have opted in, but it is not the direct result of some action or important process then you may choose to use HTML to present your email. However please, in all cases provide a text only alternative. If you choose to have a HTML communication design simple (and do not forget about mobile readers) and test, test, test as Email clients are all different. Make� sure you test on all the majority desktop clients as well as Yahoo, Gmail, Hotmail� etc.

Remember nothing looks more unprofessional then a sloppy communication, and when it comes down to it many companies choose to spend $1000’s of dollars on pamphlets while paying next to nothing for an email communication that ends up looking unprofessional and amateurish.

Ruby & the Hex value of a number

admin — Wed, 01 Oct 2008 10:28:05 GMT

Sure you could use

sprintf "0x%X" % v

but isnt this snippet I put together more fun?

(-7..0).inject("0x"){|h,i|h<<"0123456789ABCDEF"[(v>>(i.abs*4))&0x0F].chr}

Something to note here is that Ruby’s range Object does not support a range going backwards i.e. from 7 to 0, so I use a range from -7 to 0 and take the abs value of those numbers.

Modular Exponentiation and a lesson in Ruby

admin — Wed, 01 Oct 2008 10:24:44 GMT

Today someone asked me if I knew of a large number class in C++. Specifically they wanted to be able to hold the value 5⁵⁸ in a variable. I was suspicious of why they would want to do this and asked what their ultimate goal was. It turned out they wanted to calculate n^x mod p for arbitrary values of n,x and p, their ultimate goal being an implementation of Diffie Hellman key exchange.

Now, Calculating n^x then taking the modulus p of it is a valid way of solving the problem, but it has a few nagging problems. The first is that you must be able to handle large numbers in whatever environment you are doing these calculations in, and the second is that you must be able to calculate these large numbers and take them modulo some value. To begin with,� I remembered back to my early days at University when we covered something called Modular Arithmetic. Consider this example:

We have a clock face which has 12 hours on it and we want to find where the clock hand will be pointing to in 7 hours (If we are already at 12 o’clock). Easy, it will be pointing to 7. Now extending this, where will it be pointing in 14 hours? Well the answer is 14 mod 12 or 2.

Further extending this, if we want to know where the hand will be in 2 * 14 hours = 28 hours we can do 28mod12 = 4 or (2mod12 * 14)mod12 = 4.

Now, what if we want to know 14⁴ mod 12? Well we know 14⁴ is 14 * 14 * 14 * 14 so we can apply what we have just realised above and

14⁴mod12 = ((((14mod12) * 14)mod12 * 14)mod12 * 14)mod12

And we can do the same with 5⁵⁸ mod 97 and we can do all this without requiring� a large exponent and modular operation.

A quick implementation in Ruby

Taking what we have learn, we know that we can now efficiently calculate a modular exponentiation without using large numbers. Implementing this in Ruby works as follows:

>> def modpow(base, pow, mod)

>>� � (1..pow).inject(1) { |mult,x| (mult * base) % mod }

>> end

(1..pow) is shorthand for creating a Range object (Range.new(1, pow)) and it represents the set of whole integers between 1 and pow as an enumerable object. Please note, it does not create an array of numbers [1,2,3,…,10].

inject is a method which can be called on an enumerable object which is similar to reduce. Here is how it works for summing a collection:

[1,2,3,4,5].inject(0) { | a, b| a + b }

inject() operates over the collection and we pass the block a + b into it, which take two parameters a and b. a is the value last returned by the last call to the block and b is the current value we are enumerating over. In this example, the first call to block provided to inject will yield a = 0 and b = 1. it will return a + b = 1.

The second call will then yield a = 1 and b = 2 and will return a + b = 3 and so on.

This gives us a mechanism to iterate over everything in a collection and reduce the result to one variable, in this case it will be the sum of all integers in the collection.

Looking back at our original code, our block { (mult * base) % mod } will be called pow times by the inject function, and mult will be the result of all previous calls. This is exactly what we want so that line of code effectively does

(((((1*5)%97)*5)%97)*5)%97 … 55 more times.