Blue Cog Blog Rotating Header Image

Python

COHPy Meeting – October 2010

Here are some links from last night’s meeting of the Central Ohio Python Users Group.

Austin Godber talked about virtualenv. Materials from Austin’s presentation are on GitHub.

Eric Floehr, of Intellovations, presented Building a Small Business/Personal Website With Django. He discussed some Pythonic choices for building web sites such as Blogofile for generating sites that are static content, and Plone for enterprise-scale content management. Django falls somewhere in the middle as a good choice for small business or personal blogging sites.

Other links from Eric’s talk:

Also (FWIW), here’s a bit of .bash_history from my following along with part of Eric’s presentation on a VM running Ubuntu 10.10:

sudo apt-get install python-virtualenv python-pip
mkdir dev
cd dev
mkdir oct
cd oct
virtualenv --no-site-packages pyenv
source pyenv/bin/activate
sudo apt-get install mercurial
pip install -e hg+http://bitbucket.org/stephenmcd/mezzanine#egg=mezzanine
mezzanine-project sample
cd sample
python manage.py syncdb
python manage.py runserver
pip install django-debug-toolbar
python manage.py runserver
pip install django-extensions
python manage.py graph_models blog>blog.dot
sudo apt-get install graphviz
dotty blog.dot

I’m not presenting this as a how-to or a tutorial, just some notes. If you don’t know what the above commands will do then I’d recommend not running them.

COHPy Meeting – September 2010

Here are some links from the September 2010 meeting of the Central Ohio Python Users Group:

Scott Scites gave a talk on Pyjamas, a "Python Javascript Compiler, Desktop Widget Set and RIA Web Framework."

The following are among items discussed during Scott’s talk:

PureMVC

JSON-RPC

Flask (A Python Microframework)

Raphaël JavaScript Library

gRaphaël Charting JavaScript Library

Minesweeper written in Python with Pyjamas

357 Guts – One of the guys at the meeting built this online card game using Pyjamas (and if someone tells me his name I’ll update this post, unless he wishes to remain anonymous).

Eric also mentioned GeoDjango.

I thought this was a good meeting and I certainly came away with a list of some pretty cool Pythonic stuff to check out.

PyOhio Attendee Wannabe

PyOhio badge

Yeah, the badge says “attendee” but it turns out I have a conflict this year and won’t be attending. It’s a family matter that could not be rescheduled, and it’s important to me, so I will have to miss out on a terrific event around the Python programming language. I made it to one day of the two day event last year and was really hoping to make it for both in 2010.

So if for some reason you are reading this blog, and you have even the slightest interest in Python, and you will be in the Columbus area on July 31st and/or August 1st, you really should check out PyOhio. I would. Alas, maybe next year.

COhPy Meeting – December 2009

Here is my link dump from last night’s meeting of the Central Ohio Python Users Group:

The scheduled presenter, Brian Costlow, didn’t make it. Something about work being more important than a Python meeting. Priorities?

To fill the void, Eric Floehr showed a weather-related web application he has been working on that is built with Django. The app uses HTMLCalendar (Django, calendar – Stack Overflow).

Mark Erbaugh showed the web application he built using web.py. He also uses ReportLab.org to generate PDF files.

I had not run across this before: 29.2. zipimport – Import modules from Zip archives.

Catherine Devlin presented reStructuredText, S5, and Sphinx.

A few related links:
reStructuredText on Wikipedia
Quick reStructuredText
Easy Slide Shows With reST & S5
reStructuredText Primer — Sphinx v0.6.3 documentation

Catherine also mentioned:
PyCon 2010 Atlanta – A Conference for the Python Community
Python Package Index : PyPI, AKA the Cheese Shop

Also discussed was the construction of the COhPy web site:
Code at cohpy — bitbucket.org.
Using Google App Engine.

Finally, I haven’t used decorators in Python (nor in my house) but I’d like to read up on that:
PEP 318 — Decorators for Functions and Methods
Dr. Dobb's – Python 2.4 Decorators

Pair Networks Database Backup Automation

I have a couple WordPress blogs, this being one of them, hosted at Pair Networks. I also have another non-blog site that uses a MySQL database. I have been doing backups of the databases manually through Pair’s Account Control Center (ACC) web interface on a somewhat regular basis, but it was bugging me that I hadn’t automated it. I finally got around to doing so.

A search led to this blog post by Brad Trupp. He describes how to set up an automated database backup on a Pair Networks host. I used “technique 2” from his post as the basis for the script I wrote.

Automating the Backup on the Pair Networks Host

First I connected to my assigned server at Pair Networks using SSH (I use PuTTY for that). There was already a directory named backup in my home directory where the backups done through the ACC were written. I decided to use that directory for the scripted backups as well.

In my home directory I created a shell script named dbbak.sh.

touch dbbak.sh

The script should have permissions set to make it private (it will contain database passwords) and executable.

chmod 700 dbbak.sh

I used the nano editor to write the script.

nano -w dbbak.sh

The script stores the current date and time (formatted as YYYYmmdd_HHMM) in a variable and then runs the mysqldump utility that creates the database backups. The resulting backup files are simply SQL text that will recreate the objects in a MySQL database and insert the data. The shell script I use backs up three different MySQL databases so the following example shows the same.

#!/bin/sh

dt=`/bin/date +%Y%m%d_%H%M`

/usr/local/bin/mysqldump -hDBHOST1 -uDBUSERNAME1 -pDBPASSWORD1 USERNAME_DBNAME1 > /usr/home/USERNAME/backup/dbbak_${dt}_DBNAME1.sql

/usr/local/bin/mysqldump -hDBHOST2 -uDBUSERNAME2 -pDBPASSWORD2 USERNAME_DBNAME2 > /usr/home/USERNAME/backup/dbbak_${dt}_DBNAME2.sql

/usr/local/bin/mysqldump -hDBHOST3 -uDBUSERNAME3 -pDBPASSWORD3 USERNAME_DBNAME3 > /usr/home/USERNAME/backup/dbbak_${dt}_DBNAME3.sql

Substitute these tags in the above example with your database and account details:

  • DBHOST is the database server, such as db24.pair.com.
  • DBUSERNAMEn is the full access username for the database.
  • DBPASSWORDn is the password for that database user.
  • USERNAME_DBNAMEn is the full database name that has the account user name as the prefix.
  • USERNAME is the Pair Networks account user name.
  • DBNAMEn is the database name without the account user name prefix.

Once the script was written and tested manually on the host, I used the ACC (Advanced Features / Manage Cron jobs) to set up a cron job to run the script daily at 4:01 AM.

Automating Retrieval of the Backup Files

It was nice having the backups running daily without any further work on my part but, if I wanted a local copy of the backups, I still had to download them manually. Though FileZilla is easy to use, downloading files via FTP seemed like a prime candidate for automation as well. I turned to Python for that. Actually I turned to an excellent book that has been on my shelf for a few years now, Foundations of Python Network Programming by John Goerzen. Using the ftplib examples in the book as a foundation, I created a Python script named getdbbak.py to download the backup files automatically.

#!/usr/bin/env python
# getdbbak.py

from ftplib import FTP
from datetime import datetime
from DeleteList import GetDeleteList
import os, sys
import getdbbak_email

logfilename = 'getdbbak-log.txt'
msglist = []

def writelog(msg):
    scriptdir = os.path.dirname(sys.argv[0])
    filename = os.path.join(scriptdir, logfilename)
    logfile = open(filename, 'a')
    logfile.writelines("%s\n" % msg)
    logfile.close()

def say(what):
    print what
    msglist.append(what)
    writelog(what)

def retrieve_db_backups():
    host = sys.argv[1]
    username = sys.argv[2]
    password = sys.argv[3]
    local_backup_dir = sys.argv[4]
    
    say("START %s" % datetime.now().strftime('%Y-%m-%d %H:%M'))
    say("Connect to %s as %s" % (host, username))

    f = FTP(host)
    f.login(username, password)

    ls = f.nlst("dbbak_*.sql")
    ls.sort()
    say("items = %d" % len(ls))
    for filename in ls:
        local_filename = os.path.join(local_backup_dir, filename)
        if os.path.exists(local_filename):
            say("(skip) %s" % local_filename)
        else:
            say("(RETR) %s" % local_filename)
            local_file = open(local_filename, 'wb')
            f.retrbinary("RETR %s" % filename, local_file.write)
            local_file.close()
            
    date_pos = 6
    keep_days = 5
    keep_weeks = 6
    keep_months = 4    
    del_list = GetDeleteList(ls, date_pos, keep_days, keep_weeks, keep_months)
    if len(del_list) > 0:
        if len(ls) - len(del_list) >= keep_days:
            for del_filename in del_list:
                say("DELETE %s" % del_filename)
                f.delete(del_filename)
        else:
            say("WARNING: GetDeleteList failed sanity check. No files deleted.")
    
    f.quit()
    say("FINISH %s" % datetime.now().strftime('%Y-%m-%d %H:%M'))
    getdbbak_email.SendLogMessage(msglist)


if len(sys.argv) == 5:
    retrieve_db_backups()
else:
    print 'USAGE: getdbbak.py Host User Password LocalBackupDirectory'

This script runs via cron on a PC running Ubuntu 8.04 LTS that I use as a local file/subversion/trac server. The script does a bit more than just download the files. It deletes older files from the host based on rules for number of days, weeks, and months to keep. It also writes some messages to a log file and sends an email with the current session’s log entries.

To set up the cron job in Ubuntu I opened a terminal and ran the following command to edit the crontab file:

crontab -e

The crontab file specifies commands to run automatically at scheduled times. I added an entry to the crontab file that runs a script named getdbbak.sh at 6 AM every day. Here is the crontab file:

 
MAILTO="" 

# m h dom mon dow command 

0 6 * * * /home/bill/GetDbBak/getdbbak.sh 

The first line prevents cron from sending an email listing the output of any commands cron runs. The getdbbak.py script will send its own email so I don’t need one from cron. I can always enable the cron email later if I want to see that output to debug a failure in a script cron runs.

Here is the getdbbak.sh shell script that is executed by cron:

 
#!/bin/bash 

/home/bill/GetDbBak/getdbbak.py FTP.EXAMPLE.COM USERNAME PASSWORD /mnt/data2/files/Backup/PairNetworksDb 

This shell script runs the getdbbak.py Python script and passes the FTP login credentials and the destination directory for the backup files as command line arguments.

As I mentioned, the getdbbak.py script deletes older files from the host based on rules. The call to GetDeleteList returns a list of files to delete from the host. That function is implemented in a separate module, DeleteList.py:

#!/usr/bin/env python
# DeleteList.py

from datetime import datetime
import KeepDateList


def GetDateFromFileName(filename, datePos):
    """Expects filename to contain a date in the format YYYYMMDD starting 
       at position datePos.
    """   
    try:
        yr = int(filename[datePos : datePos + 4])
        mo = int(filename[datePos + 4 : datePos + 6])
        dy = int(filename[datePos + 6 : datePos + 8])
        dt = datetime(yr, mo, dy)
        return dt
    except:
        return None
 

def GetDeleteList(fileList, datePos, keepDays, keepWeeks, keepMonths):
    dates = []
    for filename in fileList:
        dt = GetDateFromFileName(filename, datePos)
        if dt != None:
            dates.append(dt)
    keep_dates = KeepDateList.GetDatesToKeep(dates, keepDays, keepWeeks, keepMonths)        
    del_list = []
    for filename in fileList:
        dt = GetDateFromFileName(filename, datePos)
        if (dt != None) and (not dt in keep_dates):
                del_list.append(filename)    
    return del_list

That module in turn uses the function GetDatesToKeep defined in the module KeepDateList.py to decide which files to keep on order to maintain the desired days, weeks, and months of backup history. If a file’s name contains a date that’s not in the list of dates to keep then it goes in the list of files to delete.

#!/usr/bin/env python
# KeepDateList.py

from datetime import datetime


def ListHasOnlyDates(listOfDates):
    dt_type = type(datetime(2009, 11, 10))
    for item in listOfDates:
        if type(item) != dt_type:
            return False
    return True
    

def GetUniqueSortedDateList(listOfDates):
    if len(listOfDates) < 2:
        return listOfDates
    listOfDates.sort()
    result = &#91;listOfDates&#91;0&#93;&#93;
    last_date = listOfDates&#91;0&#93;.date()
    for i in range(1, len(listOfDates)):
        if listOfDates&#91;i&#93;.date() != last_date:
            last_date = listOfDates&#91;i&#93;.date()
            result.append(listOfDates&#91;i&#93;)
    return result
    
    
def GetDatesToKeep(listOfDates, daysToKeep, weeksToKeep, monthsToKeep):
    if daysToKeep < 1:
        raise ValueError("daysToKeep must be greater than zero.")
    if weeksToKeep < 0:
        raise ValueError("weeksToKeep must not be less than zero.")
    if monthsToKeep < 0:
        raise ValueError("monthsToKeep must not be less than zero.")

    if not ListHasOnlyDates(listOfDates):
        raise TypeError("List must only contain items of type 'datetime.datetime'.")
        
    dates = GetUniqueSortedDateList(listOfDates)    
    
    tail = len(dates) - 1
    keep = &#91;dates&#91;tail&#93;&#93;
    days_left = daysToKeep - 1
    while (days_left > 0) and (tail > 0):
        tail -= 1
        days_left -= 1
        keep.append(dates[tail])
        
    year, week_number, weekday = dates[tail].isocalendar()
    weeks_left = weeksToKeep
    while (weeks_left > 0) and (tail > 0):
        tail -= 1
        yr, wn, wd = dates[tail].isocalendar()
        if (wn <> week_number) or (yr <> year):
            weeks_left -= 1
            year, week_number, weekday = dates[tail].isocalendar()
            keep.append(dates[tail])
        
    month = dates[tail].month
    year = dates[tail].year
    months_left = monthsToKeep
    while (months_left > 0) and (tail > 0):
        tail -= 1
        if (dates[tail].month <> month) or (dates[tail].year <> year):
            months_left -= 1
            month = dates[tail].month
            year = dates[tail].year
            keep.append(dates[tail])
        
    return keep

I also put the function SendLogMessage that sends the session log via email in a separate module, getdbbak_email.py:

#!/usr/bin/env python
# getdbbak_email.py

from email.MIMEText import MIMEText
from email import Utils
import smtplib

def SendLogMessage(msgList):
    from_addr = 'atest@bogusoft.com'
    to_addr = 'wm.melvin@gmail.com'
    smtp_server = 'localhost'
    
    message = ""
    for s in msgList:
        message += s + "\n"

    msg = MIMEText(message)
    msg['To'] = to_addr 
    msg['From'] = from_addr 
    msg['Subject'] = 'Download results'
    msg['Date'] = Utils.formatdate(localtime = 1)
    msg['Message-ID'] = Utils.make_msgid()

    smtp = smtplib.SMTP(smtp_server)
    smtp.sendmail(from_addr, to_addr, msg.as_string())

Here is a ZIP file containing the set of Python scripts, including some unit tests (such as they are) for the file deletion logic: GetDbBak.zip

I hope this may be useful to others with a similar desire to automate MySQL database backups and FTP transfers who haven’t come up with their own solution yet. Even if you don’t use Pair Networks as your hosting provider some of the techniques may still apply. I’m still learning too so if you find mistakes or come up with improvements to this solution, please let me know.

PyOhio 2009

I don’t make it to many of the Central Ohio Linux User Group meetings but I happened to be at the one where Catherine Devlin stopped in to announce an upcoming Python conference in Ohio, strangely enough, named “PyOhio.” That was in 2008 and, though it sounded interesting, I couldn’t make it to the conference. Nonetheless I did remain interested and was able to attend the first day, July 25th, of the two day conference this year.

PyOhio 2009 was held in a giant, oddly-shaped, glass-lined, marble-shingle-clad, cement block named Knowlton Hall on the OSU campus. The building is strangely proportioned. There are steps for giants and seats for little people. In fairness, it was a good place for the conference (except for that UPS that apparently lost power behind some locked door beeping constantly all day). The giant steps are meant for sitting on (not a special staircase for the basketball team) and college students typically don’t need as much seat space to be comfortable as I do. I just don’t share architectural aesthetic with the designer of the building.

Getting Started with Django

The first presentation I attended was an introduction to Django by Alex Gaynor. According to the Django web site “Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design.” Alex did a nice job introducing Django. He has obviously worked with it a great deal as evidenced by the live demo he did at the end. The live demo didn’t go perfectly but it went well. It takes courage to do an impromptu live demo since it’s a huge opportunity to crash and burn.

According to Alex, Django doesn’t really use the Model-View-Controller (MVC) pattern precisely but rather uses a pattern more like Model-View-Template (or maybe Model-Template-View – MTV). Django is “opinionated” to be more secure by default, you have to work harder to do things the wrong way (the old “falling into the pit of success” thing). Django views render templates. Templates use a custom template language within HTML. You define base templates that then reference child templates (it’s probably the other way around: child templates reference base templates, but I’m not sure – sorry). In Django a “project” is a web site and an “application” is an individual component of a web site. To learn more about Django (ignore what I’ve written here and) check out The Django Book online.

Django looks very interesting and appears to do a lot of the heavy lifting for you. I hope to explore it on the side sometime over the next year. I’ve done some work with PHP recently, a small project that didn’t use an existing framework. With frameworks like Django available it doesn’t make sense to not use one on anything but the simplest web project (and even then, those “simple” projects tend to become not so simple once you let them out of your head).

Python for Java Developers

The second session I went to was Python for Java Developers presented by Eric Floehr. Eric works at 3x Systems and has been working with both Java and Python. He is also working on starting a Python user group in central Ohio.

Eric talked about the similarities and differences between Java and Python, starting with the histories of the languages. He noted that James Gosling the creator of Java and Guido van Rossum the creator of Python are both cool guys.

The Python language was originally developed for the Amoeba distributed operating system developed by Andrew Tanenbaum who also developed Minix, the inspiration for the Linux kernel (apparently a rather influential fellow).

Eric also mentioned:

Python Operator Overloading

Neil Ludban talked (a little too quietly) about operator overloading in Python. He showed how the special methods, with names that begin and end with double underscores, implement operations on objects in Python. For example, the statement x + y is internally passing y to the __add__ method of object x like this: x.__add__(y)

I have a few terse notes about some things I want to explore further (my note taking diminishes as the day goes on):

Equality is subset of Sortable
x[y] = x[y.__index__()]
__repr__()
__unicode__()
bool(x); __nonzero__(x)
import operator; help(operator)
PEP-3119
Explore: slice, functools
with statement (PEP-343) implicit try/finally block
Attributes versus properties?
f(*iterable, **mapping)

There. Isn’t that helpful?

Python Not Harmful to CS Majors

Bill Punch from Michigan State University talked about the decision to replace C++ with Python in their CS1 class and the results thereof. Python seems to let them spend less time teaching the tooling and more time teaching problem solving. Bill asked us to try to remember what it was like to be a first time programmer (admittedly a hard thing to do after so many years). I appreciate that because over the years I have tried to write software that is accessible to non-computer people, and to do so requires seeing the use of the software from their perspective as best you can.

I enjoyed Bill’s description of “Dung Beetle Programmers” too: Instead of really understanding the program they are writing, and the problem they are solving, they just pile on code creating a ball of dung. Then to get it to work they pile on more for a bigger ball of dung. Finally they end up with an inordinate attachment to dung hence they don’t discard any bad or unnecessary code.

From the statistics they collected as students have progressed past CS1 since the switch to Python in fall 2007 they conclude that Python has not hurt the CS program. On the other hand, there doesn’t seem to be statistical evidence of a great improvement. Bill reported that he has seen a positive change in other ways. Students have come to him with stories of how they have used Python to solve real-world problems.

Form to Database Web Development

I caught the end of a session by Gloria W. Jacobs that I would like to have seen more of. Just a few links from that:

Game Development with Python and Pyglet

The quick and witty Steve Johnson talked about game development with Python. Unfortunately I only caught the end of this one as well, so only a few links:

Lightning Talks

The official PyOhio schedule ended with lightning talks:

Catherine Develin showed off some of the cool stuff you can do with sqlpython, an open source command line interface to Oracle she contributes to (and I believe has taken the lead on).

I failed to get the name of the gentleman who spoke on scaling and suggested using log shipping to update a set of read-only databases from a single write-to database. The read-only databases feed web servers.

Joe Amenta talked about Python 3.0 and a project called lib3to2 he’s working on to help you port backward to the 2.x Python interpreter should you need to do that.

Steve Johnson – funny slides to remind us that the name comes from Monty Python, not the snake.

zsh guy showed us the power of the command line.

Disease modeling in Python guy had scary diagrams where the end result was death (sort of like – life). High powered Python libraries: SymPy, NumPy, SciPy, and WxPython for the GUI.

Gloria W. Jacobs talked about Kamaelia.

Somebody talked about introspection in PyGame but by then my note taking was almost as bad as my memory at the end of a learning-packed day. After the conclusion of the scheduled sessions there were open spaces, sprints, and other evening activities. I couldn’t stick around for those this year.

Wrapping Up

Watching the other attendees working with their notebook PCs (saw a lot of Apples there) made me think I’d like to have some sort of wee PC with good battery life and mobile broadband. My spiral bound steno pad from Wal-mart just didn’t afford me any coolness or connectedness.

I want to thank all the folks who made PyOhio happen. I’d like to thank Catherine Devlin in particular for her role in organizing and spreading the word, and doing so with great enthusiasm. Assuming there will be a PyOhio 2010, I hope to be there for the whole event and maybe even contribute in some way. I don’t know that my Python skills will be up to presenter level by then but maybe I’ll at least come prepared to do an open space of some sort.

Python Imaging Library – Introduction

Sometimes you run across an item in an article or blog post that you don’t take much notice of at the time but it makes just enough of an impression that you recall its existence later, though you may forget the source. I recall reading about working with image files in Python but I don’t remember the source. I do remember there was an example that appeared to be doing some significant image manipulation in just a few lines of code.

It was a few years ago, and some time after that initial encounter, that I found myself with directories full of Windows bitmap files of several megabytes each. These were screen shots captured using either a tool called Screen Seize or using the manual method of pressing Print Screen and pasting into Paint. Regardless of how they got there, it was bugging me that they were taking up so much space. Disks are huge and space is cheap these days but I still recall that the first hard disk drive I used. It had a capacity of 5 MB and cost several thousand dollars. It’s ingrained that I don’t like wasting disk space.

Facing that listing of BMP files, the memory of that image manipulation example in Python came back to me. I searched and found the Python Imaging Library (PIL). You need to have Python installed first. Download and install the version of the PIL to match the installed version of Python and you’re good to go. The Python installer registers the .py extension so typing just the name of a Python script at a command prompt will invoke the Python interpreter to execute the script. I created a script named bmp2png.py (the old ‘2’ for ‘to’) and placed it in a directory that is in the PATH. To use the script, I simply opened a command prompt in the directory containing the bitmap files and ran bmp2png.py to create a smaller PNG file from each BMP file. Of course I looked at some of the PNG files to make sure the conversion went well before manually deleting the original BMP files.

To anyone familiar with Python, the following is a very obvious and simple script. It may also be non-Pythonic, or wrong in some way. I’m no Python guru, just a casual enthusiast at this point. There are a few “extra” lines in the script. The ones with the print statements are just for visual feedback. I like visual feedback (except from other drivers on the freeway).

import os, Image

print 'Converting BMP to PNG in ' + os.getcwd()
ls = os.listdir(os.getcwd())
for f in ls:
    name, ext = os.path.splitext(f)
    if ext.lower() == ".bmp":
        outfile = name + ".png"
        print '  ' + f + ' -> ' + outfile
        Image.open(f).save(outfile)
print 'Done.'

Line 1 imports the os module needed to work with directories and such, and the Image module which contains the Python Imaging Library. Line 4 gets a list of all files in the current working directory, and at line 5 we start working with each file in the list. Line 6 splits the file name and extension into separate variables. We’ll process only the files with a .bmp extension. After making a new file name with the .png extension we get to line 10 where the magic happens. The save method of the Image object will convert the format of the file based on the extension of the given file name. That’s all there is to converting the files. Actually there can be a lot more to it if you want. The PIL uses default options when you don’t specify otherwise, but there are options available if you want more control over the conversion.

I have been impressed with what the Python Imaging Library can do, and I’ve just scratched the surface (oops, better buff that out – sorry). Though I use more efficient screen capture methods these days, I’ve found the above script useful from time to time. It was just a starting point. There are several similar, and slightly more advanced, scripts I plan to share in future posts.