love is all around

I recently got myself a phone with a data plan (a Treo 700p to be exact), so that I can check my email when I am laptop-less and/or wifi-less.

Living in Canada, I’m subjected to absurdly expensive mobile data rates, so every byte is precious. My solution to this is to limit the amount of email data that goes to my phone as much as possible.

Here’s what I decided my email-to-phone system needed to do:

Filter out the bacn and spam

I only want to receive important personal mail on my phone - no bulk mail, no low-signal-to-noise mailing lists.

Remove HTML, attachments, images

Modern mail user agents make it possible to increase email size by a factor of 10 without adding any more relevant information. I want to be able to slim my email down to just the facts.

How it works

At the end of my Procmail script, I have a recipe that forwards any mail that has gotten by all the filters to a different account from which my phone is set to get mail.

:0c
| awk 'NR > 2' | ~/bin/plaintextforward.pl keith.grennan@example.com

That awk bit there strips out the extra header (From line) and sends the rest to my email-slimming script, ‘plaintextforward.pl’, which looks like this:

#!/usr/bin/perl
use strict;
use warnings;
use Mail::Message;
use IO::File;

die "usage: $0 to-address < msg" unless @ARGV;
my $to = shift @ARGV;

# flatten, and create alternative text/plain parts for any HTML parts
my $msg = Mail::Message->read(\*STDIN)->rebuild(
  extra_rules => ['flattenNesting', 'removeEmptyMultiparts','textAlternativeForHtml']);

# join all text/plain parts into one string, and discard the rest
my $body = join("\n", map($_->decoded, 
  grep(lc $_->get('Content-Type') eq 'text/plain', $msg->parts('RECURSE'))));

# make this string the body of the message
$body  = Mail::Message::Body->new(mime_type => 'text/plain', data => $body);
$msg->body($body);

# forward the message
$msg->head->set(To => $to);
my $out = IO::File->new("$ENV{HOME}/last-msg", 'w');
my $bounce = $msg->bounce(To => $to);
$bounce->print($out);
$bounce->send;

I use the Perl module Mail::Message to do all the processing. It creates plain text alternates to all HTML parts, if they don’t already exist, and then I gather up all the plain text parts and join them together, discarding everything else - this becomes the new message. Then it gets forwarded on to the phone account.

So that works pretty well, but I still have a problem - the backlog of mail building up in the phone mailbox. If I don’t check my phone mail for a while, I don’t want to get a flood of already-read messages. And I don’t want to have to log in to that other account and clean it up all the time. Solution - make a mail box emptying script that uses the POP3 protocol, ‘clearmail.pl’:

#!/usr/bin/perl
use strict;
use Mail::Box::POP3;
my $p = Mail::Box::POP3->new(username=>"keith.grennan", password=>"XXXXX", server_name=>"pop.example.com"); 
$_->delete for $p->messages;
$p->close(write => 'MODIFIED', force => 1);

I hook this script up to a shell script called ‘m’ that i use to invoke my mail reader so it gets run in the background whenever I open my mail client:

#!/bin/bash
clearmail.pl &
mutt

So I know that when I check my main mail account, I am clearing out my phone mail account.

Hope this helps my fellow Canadians, until we see some actually competitive mobile data rates.

Update (2007-10-02): Net::OAuth now supports the RSA-SHA1 signing method!

Update (2008-06-04): Net::OAuth 0.11 released, with many new fixes and features Changelog

Quick links:

• Net::OAuth documentation and downloads on CPAN
• Live OAuth demo powered by Net::OAuth

Today I noticed the link ‘OAuth 1.0 Draft’ appear in the del.icio.us popular feed. I followed it, and to my great delight found a spec for a protocol that is long overdue.

OAuth is

An open protocol to allow secure API authentication in a simple and standard method from desktop and web applications.

This abstract definition can be explained by a simple example:

I keep my photos on a photo-sharing site. I want to print some photos on a photo printing site, and have them shipped to me. How does the photo printing site get access to my photos (say they are my private photos, only visible to me)? One way would be to give the printing site (call this site the Consumer) my username and password at the sharing site where my photos are stored (call this the Service Provider). Problem: Then the Consumer has my credentials - this gives them total access - they could, in effect ‘be me’ on that other site. This obviously isn’t what I want. What I want to do is just give them the right to use my private data at the Service Provider, without giving away my password.

OAuth allows that to happen, in a simple standardized way.

OAuth is OpenID-like (not in its purpose, but in the way it is architected), but simpler. From reading the spec I can see that learnings from the OpenID process have been applied here - this spec has a clean, mature feel, despite being 1.0. It was also nice seeing some familiar names listed as authors on the document, like Blaine Cook and Andy Smith.

After all that I decided to help out with some Perl code. After a few hours of hacking, Net::OAuth was born! You can learn more about it on the Net::OAuth page on CPAN.

It’ll be fun to watch this protocol as it spreads. It is, like I said before, long overdue, and is immediately useful to many sites.

The latest Text::Microformat is available on CPAN. The biggest change is the addition of hCalendar support, thanks to Franck Cuny. Also there is changed handling (i.e. ignoring) of namespaces. The parser now strips off the namespace prefixes and matches on the local names. This means that the tags <a> and <html:a> and <monkey:a> would be treated the same way. This may seem backwards to XML-heads, but with microformats namespaces are considered harmful.

Text::Microformat on CPAN

Project Homepage

Enjoy!

DropUnrar is a handy tool I made for recursively extracting RAR archives.

I use it to extract movies that I’ve downloaded via Bittorrent, which are often RAR-compressed and sometimes split into multiple directories.

Just drag the top-level movie directory onto DropUnrar, and it will scan through it and all its subfolders looking for RAR archives and extracting all it finds.

Also included is a utility called “DropUnrar and delete RAR files”, which does just that. It is identical to DropUnrar except that it deletes the RAR files after it’s done expanding them.

DropUnrar is written in Perl, and wrapped using Platypus, an awesome script-wrapping tool for OS X.

Download DropUnrar v0.1

Click here to download unrar.pl, the Perl script that DropUnrar is based on. It can be used by itself as a CLI version of DropUnrar (requires you to have the unrar binary installed - I use the one from Fink).

Text::Microformat is a Microformat parser for Perl.

More specifically, Text::Microformat is an extensible Microformat-parsing framework, which allows not only new kinds of Microformats to be added, but also extension of the parser itself, so new parsing metaphors and source document encodings could be added.

Features:

• Extracting Microformats from HTML, XHTML and XML
• Extracting Microformats from entity-encoded or CDATA sections in RSS feeds.
• The include pattern
• Microformats built from other Microformats

Supported formats:

• hCard

Other supported semantic markup:

• hGrant

Links:

• The project page
• Text::Microformat on CPAN (includes some documentation)

Wanna help?

• Submit bug and feature requests
• Contribute code
  • You can create your own plugins and release them independently of Text::Microformat
  • You can contribute code, docs or tests to Text::Microformat (Submit a feature request, then attach a patch to it)
• Blog about it (tag blog posts with ‘ufperl’)

Updated 2007-04-26: hGrant is not a microformat.