percious.com

As promised, I have created a screencast which shows one how to document a project using sphinx.

http://showmedo.com/videos/video?name=2910020&fromSeriesID=291

The video runs about 16 minutes, I go through a complete example, from installing sphinx, to inserting a doctest in your code, checking it with nosetests, and finally publishing it to the html pages.

Source code for this demo can be found at pythontutorials.googlecode.com.  

I’ve not posted to this blog in over a month, mainly because I have been busy. Busy is a good thing I think. I have been helping clearwired get their newest web application off the ground, working on a number of screencasts, helping Mark with the latest batches of TG2 releases, learning about new and interesting OSS Python projects.

One of the more interesting things I have started to use with some frequency is Sphinx. Sphinx is a documentation system whichallows you to create webpages (among other things) with .rst files. It also grabs directly from your modules and inserts your docstrings (including your doctests.) It is great to have code which is not only documented, but also tested using nose. Look for a screencast from me in the near future on this one.

Two of the recent screencasts I have written show how to use virtualenv, PasteScript, and Nose. I go through a ground-up example showing how to create a virtualenv, a project package, and finally how to test that package and provide robustness with code coverage. The final screencast is about how to install TG2, which has become much easier in the advent of TG2’s first alpha release.

TG2 is actually nearing it’s second alpha release. I think This is the best release yet. I spent some time moving DBSprockets to pep-8 compliance, and the two things TG2 depends on (tg.ext.silverplate and tgcrud) are now both using the newest release of DBSprockets. I also released basketweaver which allows you to create a simple local pypi made of static HTML files. Special thanks to Chris McDonough, who wrote the makeindex.py script. All I did was to fixed it up a little, and package using PasteScript, and provide a console_script for easy usage.

Along with Sphinx, I have been trying to wrap my head around Rum, which is a new project to generate forms from database schema, much like DBSprockets. Although the internals of the system are somewhat mystifying, the API is squeaky clean, and in my mind what DBSprockets was eventually planned to be. The great news is that Alberto has done such a good job laying the groundwork for Rum I should be able to jump right in and apply lessons learned from DBSprockets. At this point it is safe to say that I will continue to maintain DBSprockets for bug fixes, but that the internals of it are going to be converted over to using Rum, before being deprecated altogether. DBSprockets had a great run, but it is time to move on and move forward with a superior design which promises to integrate so many WSGI technologies.

Tomorrow Night I will be speaking about agile technologies with SQLAlchemy at the Front Range Pythoneers monthly meeting. I spent a few hours last night working out my presentation and creating a number of screencasts which show how to use tools like virtualenv, paster, and nosetests.  I also touch apon sqlalchemy, and how one would set up a test environment for their database schema.  Even if you don’t live in the Boulder/Denver area the information could be valuable to you, so I decided to set up a googlecode repository to store all of my tutorial-related materials.  It is called PythonTutorials.

http://wordle.net/

Lately I am finding that it is worthwhile to separate projects into their own packages, but since they are all in the same domain, I want to have them share elements for importing purposes. Enter namespacing, which I believe is a little-used feature of setuptools that people should take a serious look at.

What is namspacing? Well, if you are familiar with creating packages, you know that often times they share similar traits, which means it would be nice to have sort of a global package where they all reside, but then you would not be able to install the components of said package independently. Lets say we have a solar system package, and inside it are elements for each planet. You might import from them like this:

from solarsystem.earth import echosystem

from solarsystem.venus import atmosphere

But what if you didn’t want to package simulations of all the planets, because sometimes you just want to install one or two of them at a time. Namespacing gives you a way of creating packages which are related to each other, but do not clash.

First off, you are going to need setuptools if you don’t already have it. Download the script here and run it with python. This will install the ubiquitous setuptools package which contains easy_install.

The easiest way I have found to create a namespace package is to use zopeskel, which provides a basic_namespace template for your project. First, we easy_install zopeskel:

easy_install zopeskel

Now we can list the templates that paster provides. Paster is a tool which provides a developer with an easy way to create templates for packages, as well as other useful utilities for python file management. (Paster is automatically installed by zopeskel)

$ paster create –list-templates
Available templates:
archetype: A Plone project that uses Archetypes
basic_namespace: A project with a namespace package
basic_package: A basic setuptools-enabled package
basic_zope: A Zope project
nested_namespace: A project with two nested namespaces.
paste_deploy: A web application deployed through paste.deploy
plone: A Plone project
plone2.5_buildout: A buildout for Plone 2.5 projects
plone2.5_theme: A Theme for Plone 2.5
plone2_theme: A Theme Product for Plone 2.1 & Plone 2.5
plone3_buildout: A buildout for Plone 3 projects
plone3_portlet: A Plone 3 portlet
plone3_theme: A Theme for Plone 3.0
plone_app: A Plone App project
plone_hosting: Plone hosting: buildout with ZEO and any Plone version
plone_pas: A Plone PAS project
recipe: A recipe project for zc.buildout
silva_buildout: A buildout for Silva projects
At the top of the list is basic_namespace, which we will be using. For example, lets make a solarsystem.mars package:

paster create -t basic_namespace solarsystem.mars

Paster will then prompt you with a series of questions, of which the first two are the most important:

Selected and implied templates:
ZopeSkel#basic_namespace A project with a namespace package

Variables:
egg: solarsystem.mars
package: solarsystemmars
project: solarsystem.mars
Enter namespace_package (Namespace package (like plone)) [’plone’]: solarsystem
Enter package (The package contained namespace package (like example)) [’example’]: mars

After you answer the rest of the questions, it will create a directory structure that looks something like:

solarsystem.mars/
|– README.txt
|– docs
| `– HISTORY.txt
|– setup.cfg
|– setup.py
|– solarsystem
| |– __init__.py
| `– mars
| `– __init__.py
|– solarsystem.mars.egg-info
| |– PKG-INFO
| |– SOURCES.txt
| |– dependency_links.txt
| |– entry_points.txt
| |– namespace_packages.txt
| |– not-zip-safe
| |– paster_plugins.txt
| |– requires.txt
| `– top_level.txt
`– zopeskel.txt

At which point, you are ready to create a development install of your project. Simply change to the directory:

cd solarsystem.mars

and:

python setup.py develop

Code for your new module would go in the solarsystem/mars/ folder, inside the new package. Any python you have installed this new package will
now be able to:

import solarsystem.mars

Namespacing is a great way to organize your work into digestible morsels, which can be installed one by one as needed by your perspective users. It is a great way to take advantage of all of the tools which have been created to make distribution and versioning easier.

It’s been a while since I have blogged for a few reasons. One, I took a vacation. No work, open source or otherwise for 11 days. I barely even checked my email. One thing that is great about living in Colorado: I don’t have to go anywhere on vacation, and my family will come to me! I took my sister up the First Flatiron, my dad fishing in Cheeseman Canyon, and all of us took a nice stroll through Chautauqua Park. We ate out almost every night and enjoyed catching up since I have moved 2000 miles away.

Previous to that, I managed to win a skirmish in the framework war at work, and spent an obscene amount of time (on and off the clock) bringing my first production-level TG2 site. This was a really fun project because my coworker and I were given the reigns and let loose on implementation. In 3 hours we had a working TG2 site with data in an EXT grid, with database schema expressed as a tree. In 3 weeks we had a complete reporting system (using EXT) with object-based security. We were also able to implement a file upload manager, and a select shuttle (for assigning groups to secure objects) thanks to the help of Sanjiv Singh, my GSoC protege. Thanks to all of you who have helped myself, and TG2 get off the ground.

One of the reasons I was so absent from the OSS community was that I was trying to wrap my head around what was going on with the EXT licensing, which changed under my feet as I was developing our in house app. The move from LGPL to GPL/Commercial was a shock to the community, and in my opinion based primarily on one person’s greed. My personal choice is to finish up whatever EXT support I have to do for our in-house application, and never look back at it again. I will be moving on to Dojo, and have advised my GSoC students and other people involved with TurboGears to do the same.

Now that I am back from vacation/new project hell, I have been able to release a new module for TurboGears, tg.ext.silverplate, which is a plugin for TG2 providing User Management and Profile pages which are customizable. This all fits under the TGTools domain, which has become a home for tg.ext.repoze.who (Authentication for TG2) and tg.ext.geo (An upcoming library for Geographical support in TG2). If you get a chance, you might want to check out the new TGTools googlecode site.

I also did a new release of tg.ext.repoze.who, which provides authentication/authorization to TG2. The only functionality I as able to contribute was a change to the name-spacing, as well as a full test of it’s functionality. Right now I am working on an LDAP bridge which will allow your TG2 site to authenticate against LDAP, and then dump you into the TurboGears Domain for all of the authorization stuff (groups, permissions, etc.) I should have this completed in a few days. Most of the work for tg.ext.repoze.who was done at the last world-wide WSGI-Turbogears sprint. It looks like TG is going to do a release in a few days.

This weekend I am headed to ABQ for some sprinting with clearwired on their next-generation web application. I am hoping this time around TG2 is mature enough in their eyes to take advantage of everything it has to offer, since they have currently chosen to work simply with Pylons as a framework. Either way, I will be working with some of the best people in the business, and we should be able to push TG foreward one way or another.

$ history|awk ‘{a[$2]++ } END{for(i in a){print a[i] ” ” i}}’ |sort -rn|head
241 nosetests
97 ls
85 cd
20 svn
15 python
7 vi
7 rm
5 easy_install
4 pylint
3 history|awk

Are you a software engineer? Programmer? QA specialist? How long have you been at your current position? How long do you expect to be? I bet it’s less than 5 years.

Why would that be an important issue for open source? Most of the time you spend most of your time working OSS projects at home, right? But I bet someday you’d like to get paid for that work, unless like some OSS developers you base your happiness on the proliferation of your software.

So one day you are going to pack up and leave your company, and all of it’s proprietary software behind. And if you are developing Python, that probably means you are leaving a gaping, expensive hole for your former employer. But what if you could keep working for that employer, after you left? What if what you were doing at the new job was so transferable that you could be productive from day 1?

This is the promise that is OSS. And while this promise is probably idealist, it has at least one tentacle sticking out from reality. The fact is, workers are migratory these days. Personally I think this is because short-sighted investors value fiscal gains over a longer payoff, but we do live in the day of day-trading. The fact is, many people don’t see programmers for what they really are, a mixture of engineer, mathematician and artist. I can always seem to find someone who will pay me more money to do the same job I already have. When something I just can’t turn down for my family’s sake comes up, I have to take it. For this reason programmers, and even the greater workforce migrates.

Employers have already realized that employees are migratory. They have all but done-away with pension plans in exchange for portability: the 401k. Who cares what kind of health insurance we offer? 80/20 is now the norm, because employers don’t expect to have to pay in the long run, and employees accept it too, because they don’t expect to be working at the company for the long haul, when health insurance becomes more of a factor.

But lets get back to the original question. What does a migratory worker mean to open source. Portability of work. I can take my Django/Turbogears/Zope knowledge to any other company using these technologies and pick up right where I left off. If I am one of the few developing software in the OSS world (as apposed to just using it) I can actually contribute more to my OSS project on my companies dime. The question is, are the companies really losing out by providing their software to the world?

Definitely not. In fact, companies stand to gain much more by choosing (and contributing to) something “free” and open. They retain some value from their employee after said person has left the company.  Bug patches, new features, etc. all pulled from the same pot they were putting their goodies in before. Since the software is open and free, more people have access to it in the age where replication of information is almost cost-less. That means they have a larger work force to choose from, and more experts and shared expertise to gain from.

So then the question becomes, “As a company, why should I contribute to open source?”  The fact is you are going to get back more than you give.  Your employees will gain greater expertise by helping to build the tools that they use.  Additionally, the company is helping to keep the project’s momentum going, which ensures the OSS project’s longevity.  It is no secret that “feature complete” projects often fall by the wayside, regardless of quality.  By playing an active role in a project’s development, you are ensuring that that project will be available, and have a good support system even if your employees have moved on.

If you are a company who is battling with the decision to contribute to, or even use open source, I implore you to look seriously at your options, your [migratory] workforce, and your ability to develop a product for the long haul. I bet you find more value in free software than the sticker price. Software, more than ever, is about the people, not the product.

TurboGears is undergoing a monumental effort to participate in GSoC.  Ok, maybe not monumental, but at least 5 of the developers have been hard at working putting together an application even Google wouldn’t turn down. Even if we are not accepted, we are planning on participating by way of PSF, as we have done in previous Google Highly Open Participation contests.  We have developed a number of ideas which students can choose from, or students are welcome to come up with their own TurboGears ideas, and I am sure that one of our mentors will be able to match up with you.      

My own ideas revolve around DBSprockets and TwTools, which is not surprising since I am the owner of said projects.  The largest project and the one which I have the most desire to see put into action is that of a TurboGears CMS.  There had been some work done last year by the guys at Pagoda which produced a brilliant screencast, but the project seems stalled out and it would be nice to see it revived.  Furthermore, the solution I proposed is intended to be much more modular, so you could pick apart portions of the CMS and put them in your existing applications.  I think this would be the most flexible solution, and also one which would employ much that DBSprockets, TwTools, and Toscawidgets have to offer.  If you are a student who is interested in working on this project, please don’t hesitate to drop me a line.  You can also track me down at Pycon for the remainder of the week, and at the TurboGears sprint next week.  If you are a student who is eager to get started feel free to participate in our sprint, remotely or in person.   

One of the great things I see coming from this mini-project is that we now have a very nice set of concrete ideas about how to make TurboGears better.  Whether or not students participate in the development we still get a huge benefit from the creation of ideas, and it gives the development team and possibly new developers a target to make TurboGears the best it can be.  My hope is that these ideas will not only bring a students to our project, but also bring some developers out of the woodwork who may have started similar projects and would like to contribute.  The bottom line is, if you aren’t a contributor, and want to be, here is a great place to start.

I am pleased to announce the first DBSprockets 0.2 Release Candidate. This release comes shipped as 100% code-coverage tested for all API modules (non-dbmechanic).

Many thanks to my beta-testers and developers who have submitted patches, especially Nathan McBride, Michael Brickenstein, and Jason Kirtland. Thanks to Alberto Valverde, who’s work with Toscawidgets is invaluable to this project.

The biggest part of v0.2 is probably Primitives, which give developers an easy way to generate web content from database definitions. If you have a sql database, and want fast web content, this might be the
ticket. A simple few simple calls to table reflection with SQLAlchemy, and you can use makeForm, makeTable, and getTableData to generate web content.

The second notable module in this release is the DBMechanic, which allows you to do all of your database crud with about 3 lines of code added to your Controller.

Supported on this release are TG1.0 and TG2.0-preview with both Primitives and DBMechanic. Primitives have also been verified in Grok, which took a certain amount of work with respect to wsgi and Toscawidgets. Mysql and sqlite are supported with this release Posgres has been tested and is shown to work with dbsprockets.

There are no known Issues to date.

Everything you need to know to get up and running with DBSprockets is here:
http://code.google.com/p/dbsprockets/
Thanks,
chris