Categories
Tech Blog

Tech Environment

I’m sometimes asked which tools I use for a particular job, so here goes:

  • At home I work on a Dell XPS Laptop with a dual boot setup of Ubuntu 12.04 and Windows 8.1. When developing I usually hook-up a small external screen.
  • I use Eclipse, with the Aptana extension installed for Java and PHP programming. I like the fact that I can use the same editor on both Linux and Windows, and for C, Java, Python and Web development. I also use Microsoft Visual Studio for C#, and PyCharm for Python development. For smaller editing tasks I use Notepad++ on Windows and Vim on Ubuntu.
  • Chrome is my default browser (although I did use Firefox for a while). I especially like the developer tools.
  • I use Mercurial and Git, combined with Bitbucket and Github respectively, for version control. Currently I am most comfortable with Mercurial.
  • Just about everything that isn’t code related goes in my Dropbox folder, of which I’m a big fan.
  • I use cloud based tools, especially Google’s, for as much as possible. I like the portability they provide and in general trust ‘them’ to look after my data more than I trust myself!

Top 10 consumer software tools:

  1. Gmail
  2. Google Drive
  3. Dropbox
  4. LastPass
  5. Pocket (saving articles I come across on the web for reading on my iPad later)
  6. Evernote (longer notes, recipes etc.)
  7. Google Maps
  8. Feedly (RSS feeds)
  9. Goodreads
  10. Rotten Tomatoes

Other gear:

  • Google Nexus 4
  • Kindle Paperwhite
  • Panasonic Lumix FT4
Categories
Tech Blog

Abstraction

It struck me, sitting in the Eng library, studiously studying (really, I was!) for my upcoming embedded systems exam that our generation has it harder. That probably sounds exactly the sort of pretentious statement you might expect from a college student of today; so allow me to explain a little further.

Computing knowledge and education is dependent on abstraction.  As one of my lecturers likes to say (on repeat), computers are very simple and also very complicated. What he means by this paradox is that any layer of a computing system looked at in isolation can be understood without too much trouble: arrange four MOSFETS correctly and you get a NAND gate, hook up a few ANDs, ORs and XORs and you’ve got a full-adder – easy. Similarly, for any budding programmer writing a script to pull some records from a database and display them to a user might seem a logical enough exercise.

The complexity is in trying to mentally link these two ends of the computing spectrum: the high level and low level. After three years of Computer Engineering courses I’m only just able to go from thinking about some high level bit of functionality and mentally traverse the various layers of abstractions and have some understanding of the machine level functions invoked.

One of the books I was studying from, Mano’s, Computer System Architecture, was published in 1983. It continues to remain relevant and useful in 2012. The topics of RISC architectures, control paths and pipelining have evolved only gradually in the intervening 30 odd years. What has evolved is layer upon layer of high level functionality. We’ve had the OO revolution, with many Computer Science students graduating with their lowest level understanding being how to implement quicksort in C#, Java or Python. There are now countless networking protocols, web frameworks, mobile devices, and embedded systems that are starting to look less and less like embedded systems.

A consequence of all of these extra layers of abstraction is that students can no longer be expected to learn and understand them all! In recognition of this, there has been a proliferation of new, more specific qualifications. Where we once had Electrical Engineering, we now have Computer Science (the first course was taught in 1953 at the University of Cambridge), Computer Engineering (1971), Software Engineering and Information Technology. Is there any other area of professional education that has seen such an expansion?

As a student studying in this new world, I’ve found myself somewhat frustrated that I’m largely expected to learn in detail some narrow area of knowledge. I’ve spent a large amount of my own time trying to understand as much as possible from the other areas of qualification. I’m very aware that to remain current with ever evolving levels of knowledge I will need to continue self-learning throughout my career.

So if my generation has it hard trying to make sense of the layers of computing abstractions, it’s both exciting and ominous to think of the complexity that will be encountered by the next generation of computing students.

XKCD – Abstraction
Categories
Tutorial

Integrating Google Forms with Mailman

Recently I was researching new email list solutions for the University of Canterbury Canoe Club. Our existing setup involves a Google Form which new members fill out, with data accessible in a Google Spreadsheet. Mailman is used for our mailing list. Currently the Webmaster is notified when a new form is submitted and manually puts the corresponding email address into the mailing system. This seemed cumbersome to me, and set me searching for a better approach.

The most promising alternative I found was MailChimp, a very popular email marketing tool. MailChimp has many options, takes steps to combat emails being wrongly detected as spam and can host a built-in signup form. However the flashy HTML emails it generates seemed overkill for our needs, and as it is designed for marketing campaigns, does not support the concept of any person on the mailing list being able to send email to it – a feature of Mailman that is utilized by the club (with emails coming to the Webmaster for moderation). Additionally, committee members wanting to send emails would have had to log into the MailChimp web application with a shared password, rather than using their personal email address and mail client. This presented two problems: non-technical committee members would have had to learn how to use a new tool, and there would not be a reply address matched to the actual person sending the email.

In light of these findings, I set about finding a way to integrate Google Forms with Mailman. I came across the answer in the form of Google Apps Scripts, a service provided by Google which allows chunks of Javascript code to be written against an API and integrate with a number of Google services – in my case Google Forms/Spreadsheets. Here I present the steps for taking the email address out of the form submit and submitting it to your Mailman server as a new subscriber.

I presume you have already created a Google Form which asks users to enter an email address. In the corresponding spread sheet, choose Tools -> Scripts Editor.

Paste in the following code, substituting the following strings with those of your setup:

  • SheetName eg. Sheet1
  • cellnumber eg. 5
  • yourdomain eg. http://wikipedia.org
  • yourmailinglist eg. allsubscribers
  • yourpassword eg. password
  • webmaster@yourdomain eg. webmaster@wikipedia.org
function SubscribeEmailOfNewMemberToMailman() {
  var spreadsheet = SpreadsheetApp.getActiveSpreadsheet();
  var sheet = spreadsheet.getSheetByName("SheetName");
  var row = sheet.getLastRow();
  var range = sheet.getRange(row,cellnumber);
  var email = range.getValue().trim();
  range.setValue(email); //Write the trimed email back to the spreadsheet.

  if (validateEmail(email)) {

UrlFetchApp.fetch(yourdomain/mailman/admin/yourmailinglist/members/add?subscribe_or_invite=0&' +
                                   'send_welcome_msg_to_this_batch=1&send_notifications_to_list_owner=0&subscribees_upload='+email+'&adminpw=yourpassword);

    	Logger.log("Subscribed {8b0b96060afa46d1ca7c90aeb24f8da085e11471ba2cd3d6bf9cf57471789b98}s with email {8b0b96060afa46d1ca7c90aeb24f8da085e11471ba2cd3d6bf9cf57471789b98}s.", name, email);

  } else {
    MailApp.sendEmail(
      "webmaster@yourdomain",
      "Failed to auto-subscribe new member",
      "Email regex failed for new member '" + name + "' with email '" + email + "'.");
  }
}

function validateEmail(email) {
    var re = /^(([^<>()[\]\\.,;:\s@\"]+(\.[^<>()[\]\\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
    return re.test(email);
}

The behaviour of this script is as follows:

  1. The email address is pulled from the email column in the last row of the spreadsheet.
  2. The email string is trimmed in case the user accidently left some whitespace at the start or end.
  3. The email is feed into the validateEmail function which uses a regular expression to check if the email is of a valid form. It returns a Boolean.
    1. If it returns True. The email is past to the Mailman server in the URL. (Note: the mailing list password is sent in plain text in the URL which is probably not very secure). A log is made of this subscription attempt to help with any troubleshooting in the future.
    2. If it returns False an email is sent to the Webmaster with the problematic email for them to follow up on manually.

The final step is to tell the Google Spreadsheet to call the SubscribeEmailOfNewMemberToMailman function every time the form is submitted. To do this, from the scripts editor select Resources->Current script’s triggers. In that dialogue, select the function to invoke – in this case ‘SubscribeEmailOfNewMemberToMailman’, In the next two boxes, select ‘From spreadsheet’ and ‘On form submit’ respectively.

With this integration with Mailman and the club’s bank account and payment information supplied in the confirmation page of the form our signup process is now completely automated, making for a happy Webmaster!

The Google Apps Scripts platform is very well documented, and I’d encourage you to modify this script to suit your needs. One addition I made is to call a PHP function on the club website, passing through the user’s name and email address. The PHP function then creates a new username and password for the new member, and notifies them by email.

I hope you find this useful!