Pythonic Python - Writing Python code that fits the language's idioms.

A recent posting by +Luke Plant to the Google+ Python community about the importance of PEP-8 reminded me that for quite a while now I've been thinking I should compose a blog posting about idiomatic Python code - and that's what led to this article.

Now I know this is a very small candle trying to light a very large dark space, but so it goes. My own Python experience has so far been limited to small projects and class-work, no larger scale production quality team efforts. The good news is that that limitation in my own experience means there's plenty of room for you to chime in with opinions of your own on this matter. I write posts to my blog in hopes of sparking up some conversations. Do feel free to speak up!

Some Orientation if you're New Around Python

If you are new around Python and aren't entirely clear on what it means when we talk about "the Python community", I offer you this orientation section lest you be confused by technical details in my words here. If any of it remains unclear, do speak up about that and I'll try to expand the material to cover more.

A "PEP" is a "Python Enhancement proposal". There's a web site where all the PEP's are published and shared. PEP 0 is an index to the PEP's. PEP 1 is a guide to how to write and submit a PEP.

Most PEP's are proposals to enhance the Python programming language in some way.

e.g. PEP 435 proposes adding an enum data type to the Python standard library. It's been a long discussed topic. PEP 354 for example proposed something similar but was rejected in 2005. PEP 435 has been accepted and is scheduled to be implemented and released as a feature of Python 3.4.

Other "PEP's" are merely informational, not really enhancement proposals at all.

For example, PEP 429 is the nominal schedule and plans for the Python 3.4 release. So, if PEP's aren't always "Enhancement proposals", what are they? PEP's are public records of consensus opinions of the Python community. Not every voice counts the same. Benevolent Dictator for Life (BDFL) and Creator of the Python language in the first place, Guido Van Rossum, has extraordinary influence on the fate of a PEP. Fortunately, he generally shows good judgement in steering the language.

Among the informational PEP's, PEP-8 is a style guide for Python code. If you don't want your code to look strange to the eyes of experienced Python programmers, you should try to comply with the guidance of PEP-8. To help you do that, there's a PEP-8 checking program. Actually, if you do a Google search, you'll find that there's more than one such PEP-8 checking program available. You should have at least one of these programs in your tool box and make a habit of running it against your code and spending the time to tidy things up to make the checker program happy.

It's not that you can't ever bend the rules, but you darn well better have an excellent reason why deviating from the style guidelines was worthwhile in the exceptional case that you decided not to fix.

Another informational PEP that you should read is PEP-20, the Zen of Python. Unlike PEP-8, the guidance of PEP-20 isn't something so simple that a straight-forward program can look at your code and say whether or not you were thinking like a Python programmer when you wrote your code. PEP-20 does try to get your head into the right way of looking at the things that you wish to program using Python.

Beyond Style

There's more to writing "Pythonic" Python code than following PEP-8's style guidelines and letting PEP-20 shape your thinking. For example, given a collection of things (a "list" being the most typical way of setting that up in a Python program, if your reflex to process that list is to think "DO I=1 TO N"... then you are probably still under the influence of Fortran or C or some other such old programming language. A more Pythonic way is "FOR ITEM IN LIST:"... Now there are lots of special situations that might drive you to explicitly stepping an index through a list, even in Python, but don't do it as a simple automatic habit. Ned Batchelder has written an excellent tutorial on how to "loop like a native".

One other area where I've found I have to fight against habits formed in working in other older programming languages is in conceiving of the types of return values from a function. In some old languages, the return values were limited to something simple, and to exactly one thing. But Python is entirely dynamic, so you can feel free to inexpensively return elaborate data structures or even multiple values (tuples) at a time. It isn't inordinately expensive because the implementation doesn't copy the values around, just pass descriptors of the values. The language's runtime garbage collector takes care of reclaiming the storage space when you no longer need the fancy value that you constructed.

Additional Tools

It is important to understand that Python does incredibly little error checking at "compile" time. Your code will pass without complaint from the Python language processor, even if your program has grossly mis-spelled variable names or calls non-existent functions. You won't hear about the errors in a given line of code until you actually try to execute that line. This makes testing your code incredibly important. You should definitely look into Python unit-test tools so you can embed test cases with your code and be well prepared to routinely re-test after you make revisions to the code. Related to testing, you might also benefit from coverage tools that report which portions of your code remain unexercised. This may guide you into beefing up your unit test cases. Sadly, even 100% test coverage of all the lines of code still can't guarantee that your code has no undiscovered bugs lurking in it. But if you haven't even exercised all the lines of code it is easy to anticipate that there may be easy-to-find errors lying in wait to spring out at you at some inopportune time.

There's no real substitute for good judgement and as the old adage explains good judgement is something you learn from experience and experience is often something you gain from applying bad judgement. Another tool that may draw your attention to places in need of better judgement is sloccount. sloccount tells you how many lines of non-blank, non-comment source code you've written and while it is counting, it computes complexity metrics for your functions. A function that is oversized or that scores as exceptionally complex deserves to be re-considered. More often than not, those are the kinds of functions where your undiscovered bugs are lurking.

There are multiple static checking programs that try to find the more obvious kinds of problems for you. pylint, pyflakes, and pychecker to start you off with 3 names to Google search for. Some of the suggestions from these tools can be very annoying, like if I need a short-lived integer variable for local use, why shouldn't I name that variable "i"? But generally the checkers are tunable to tailor the rules to your taste. Don't just get annoyed with what the checker program tells you. Look at what it thinks it sees and see if you could do better and make it happy while you are at it.

Data Types.

One other aspect of the Python language that you should pay attention to is it's richness in data types. strings, lists, dictionaries, sets, tuples... And those are just "collections" of values. If you find yourself frequently searching through a list, stop and think whether a list is really the right choice. Maybe you really ought to use a dictionary to make it simpler to check if a given value is in the collection.

Modules and Name Spaces

If your Python programs all tend to each live in exactly one file, then odds are you aren't making use of Python's modules and name space capabilities to separate your code into manageable sized pieces. Ultimately, that will limit you in your ability to tackle larger projects such as multiple-person programming teams. Keep an eye out for possible reusable modules that are worth separating from the specific problem at hand so you can use the same code elsewhere in the future. Python is quite liberal in its handling of type ("duck typing"). You can exploit that to make your code quite flexible about what data it is willing to deal with. Take the time to carefully document what the requirements are for the data that your module can handle. e.g. maybe you had in mind that it would handle "employees", but perhaps it could be equally happy with any kind of object that has a mailing address as part of it ("customers" for instance).

Multiple paradigms.

Although Python is not a gigantic language with the sort of sprawl that PL/I was notorious for, Python does allow for more than one programming style. It certainly has support for object oriented programming as well as structured programming. Happily, it doesn't insist that you make use of all of its possibilities, but if you have been shying away from some aspect of the language because it supports a style that you are unaccustomed to, do push yourself toward learning how to use that aspect of the language appropriately. "generators" are a kind of co-routine and you may not have run into such a control structure in other languages, but they are worth the time to learn. Object oriented programming is still a weak area for me, but I've been working on trying to understand how to put that to good use. Test driven development is new to me too, but again I've been trying to regroove my mind to pay attention to doing things that way.

Libraries and Frameworks

One of the mixed blessings of working in Python is that there are many rich libraries of existing code available for your use. Do plan to spend some time searching to find what is available that may be helpful to you. The code is generally free for you to download, but you may need to invest some time to understand it and bend it to your will. Forking it to make a specific-to-you version is probably a bad idea. But reinventing the wheel on your own is probably an even worse use of your time. If the library module isn't going to comfortably fit into your program, that may be nature's way of telling you to return to your favorite search engine to find another alternative implementation to use instead of your initial "find".

Learning the Python language is just a start. Learning to put it to good use is a much taller order. As Peter Norvig promises in his essay, "Yes, you can learn to program in only 10 years". But don't get discouraged. Learning new stuff every day can be great fun.

11/10/2013 - Corrected a typo. "are are". Being your own editor has its hazards.

MOOC's: Be careful what you wish for.

There's an old warning that you should be careful what you wish for, because you just might get it. Here's an article from the Slate "web magazine" cautioning that MOOC's are going to doom us all:

http://www.slate.com/articles/technology/future_tense/2013/07/moocs_could_be_disastrous_for_students_and_professors.html

Countering that point of view, the article does manage to provide a link to a TED talk by coursera founder Daphne Koller, speaking at Edinburgh in June 2012:

http://www.ted.com/talks/daphne_koller_what_we_re_learning_from_online_education.html (21 minutes)

Now my own experience is limited. I took Udacity's CS101 course last year. It has no fixed schedule and I worked on the course when time was available and did eventually complete it with 100% on the final exam. I've signed up for another couple of Udacity courses, but haven't managed to get around to finishing either. Home Internet outages and a few weeks hospitalized distracted me from sticking to it. Of course, with no fixed schedule, I can always return to those courses and get down to work on them.

Koller and coursera seem to put much more emphasis on a fixed schedule than does Udacity. For example, I've been taking coursera's Systematic Program Design Course and have to concede that I have not kept up with it's mandatory schedule of weekly homeworks and quizzes. Realistically, I have to concede I'll not be completing that course this summer and will have to try it again the next time it is scheduled. Not a bad course, but it does get tedious in its attention to microscopic details. I also got distracted with learning a new programming language (Racket) along the way. Too bad as the course was to have 2 peer reviewed assignments and they were to be my first experience with peer reviewed work.

I think the gloom and doom suggestions of the Slate article are a bit too Luddite a point-of-view for my taste. I think MOOC's will lead to a lot of change in education courses in the future. The MOOC courses will establish baselines which future improvements will have to beat. Koller emphasizes that the MOOC offerings collect a lot of data which provide the prospect of guiding future improvements. The one thing that I can see screwing up a bright future of continuous course improvement is if copyrights are used to restrict building a better course based on an existing course. U.S. copyright law has seemed to get repeatedly stretched with additional years to make sure Mickey Mouse, et al, never pass into the public domain. I can only hope that the MOOC courses stick to the tradition of Creative Commons licensing so that it is possible to found an improved course on an existing good course.

On a related topic, although Bill Gates is not a person that I often have praise for, in this 10-minute TED talk by Bill, he does a nice job of arguing that teachers need more feedback. A modest bit of technology, such as a video camera on a tripod, can give teachers a basis for self-assessment and the possibility for peer reviews.

http://www.ted.com/talks/bill_gates_teachers_need_real_feedback.html

FIeld Trip and an Unusual Looking Flag

Flag image by Blas Delgado Ortiz, 27 June 2001

A few weeks ago, while we were visiting my brother-in-law in a nursing home in Queens, we had occasion to stop by the NYPD 110th precinct station house to file a report about an incident. While my wife was chatting with the officer at the desk, I wandered around the lobby looking at the various posters and artifacts on display there. I noticed that in several of the pictures of "events", there was a distinctive flag on display - green & white stripes and 24 stars in a curious looking constellation arrangement, not simple rows or anything simply symmetric. I asked the officer after my wife had finished her business what that flag was and she waved me off, not knowing the answer. But the question stewed in the back of my mind, unsettled.

Today, I decided to try a Google search for:

flag green and white 24 stars

to see what it would find. The search easily turned up exactly the information I was looking for. It's the official flag of the NYC police department. 5 stripes for the 5 boroughs of the city (Manhattan, Brooklyn, Queens, the Bronx and Staten Island) and the 24 stars represent the 3 cities, 9 towns and 12 incorporated villages that were integrated together in forming NYC in 1898. A detailed explanation is given on http://www.crwflags.com/fotw/flags/us-nycp.html. So now I know. Can't say that I'm impressed with the depth of knowledge displayed by the officer at the station house desk.

I can't share this kind of information without mentioning the "Fun with Flags" episode of "The Big Bang Theory". Enjoy!

Recursion

One of the interesting computer science topics that Udacity CS101 introduces is "recursion". Recursion is where you define a function in terms that in some cases require the function to call itself. This is the main topic of Unit 6 of the course. There are important design considerations to be taken into account if you are going to use "recursion" in your implementation of a function.

1. Base case(s) - It is crucial that your function have at least one combination of inputs that do not trigger yet another recursion. This non-recursive case is called the base case of your function. If you don't have at least one base case, then you are fairly certain to have an unending loop that never produces a final result.
2. Progress - It is similarly crucial that your function make progress toward the base case(s) as it recurses. If you have situations where sometimes the function re-invokes itself with the same inputs and state as it had previously, then it may be more subtle, but almost surely you are stuck in an unending loop that never produces a final result.

Not every programming language supports recursion. Cobol and Fortran for example traditionally do not. Some languages (e.g. PL/I) support it, but only if you declare that a specific function may be invoked recursively. ("PROC OPTIONS(RECURSIVE)"). Python supports recursion without any need to declare your intent to use the capability, but Python's support of recursion does not include "tail recursion optimization". Tail recursion optimization is where a language processor recognizes the special case that a procedure is being called recursively, but that when the procedure returns to the point of the call, there's nothing more to do than to return to an earlier call to this procedure. A clever compiler can transform the code for such a program to do a plain loop instead of a recursive call. Alas, in Python, there's much that cannot be known for certain about the code until run time. Guido Von Rossum, the creator of Python and it's "Benevolent Dictator for Life" (BDFL) has blogged about why Python doesn't bother to try harder for this particular case. See: Tail Recursion Elimination.

A key fact to note is that if you've got code with a tail recursion in it, then it is reasonably straight forward for you to restructure that code to explicitly use a loop in place of the recursion. It apparently is in this fact that Guido draws enough comfort to not bother trying to optimize the handling of this kind of code.

Some folks look at the limitation of Python's support of recursion and wrongly conclude that recursion is a feature of the Python programming language that you should avoid. Ned Batchelder did a great job of de-bunking that assertion in his essay: Recursive Dogma.

There are lots of interesting discussions of recursion in the Udacity CS101 forum. Much of the debate is over whether or not recursion is something easy or hard to get your brain wrapped around. Some folks find recursion is an elegant way to express a function while others opine that iteration is a more natural way to conceive of a function's processing. My opinion in this debate is that even if recursion provides a straightforward clean design for a function, do think through how to transform that design into an iteration. Compare the readability, performance and limitations of the 2 alternative designs and pick the alternative that makes the most sense for your needs.

Ken Robinson on how to effectively run an education system - Command and Control isn't the right choice.

Ken Robinson surely must earn a lot of air miles giving talks at TED conferences. Here's another one, from fairly recently. May 2013 is when it was published to Youtube.

Ken Robinson: How to escape education's death valley

I did laugh out loud, so he still manages to give an entertaining talk, but he compares the scale and challenges of Finland's education system to that of a US state. I can accept that for at least some states the scale is comparable. Texas, New York, and California and perhaps Illinois are likely outliers in the size distribution, but that can presumably be solved by carving things into administrative regions. Except on matters of revenue collection, upstate NY surely doesn't want to be lumped in with New York City anyhow. But I do think there are vast differences in the educational challenges faced by the US vs. those faced by Finland. Wikipedia tells me that 91% of the people in Finland speak Finnish and 41% of the population can speak Swedish. There are various other languages spoken by a fraction of the population, but the totals are in the thousand's of people. Compare that to the languages of New York where the total of non-English speakers runs into the millions, thanks to NYC's long term position as a port of entry into the "melting pot" of US culture. I believe there is also more cultural diversity in the US than in Finland and that this makes it easier to run a successful high school in Finland than in Brooklyn.

So, I don't think that Robinson has been entirely convincing in his talk. Nonetheless, he makes lots of valid points about how and how not to effectively run an education system. The prevailing testing culture imposed on US schools seems well captured in this "What real-world skills do you have?" cartoon.

If you didn't see Robinson's previous TED talks on education, see the links from my January, 2013 Blog article "TED Talks on Education". They were enjoyable and informative talks too.

Told You So - MOOC forum participation is important

If you have been a regular reader of this blog and have a long memory, you may recall that back in December of 2012, in my article "Is the Udacity CS101 Course Watered Down?", I strongly emphasized my opinion that a vital part of getting the most out of the course is to participate actively in the forum. As strong as my opinion on this matter was, it was just my opinion. But now I've got data to back me up.

John Duhring has posted to the Google+ STEM Community today a link to an article "Effective Habits of Power Users: A Look At Recent MOOC Research". The article reports on a study of the small percentage of folks who completed the MIT "Circuits and Electronics" course that edX offered as a MOOC in March 2012. About 155,000 folks registered for the online course and only a few more than 7,000 completed the online course and earned a certificate of completion. There was a follow-up study to see what distinguished the folks who made it to the goal line from the rest. The report says the level of participation in the forum is the big differentiator.

Many years ago when I was a college student, I had the pleasure of sharing housing just off-campus with other students taking many of the same courses as I was taking. We had marvelous arguments with each other as to what the professor really was saying in the lectures we'd attended. They were friendly arguments with much hand waving at speeds approaching C to explain space-time dilation and other such hot topics. I'm quite certain that those discussions were an extremely important part of going to a residential college and greatly increased my learning.

I've been watching my wife's kids slow progress as commuter students to a local college and I think part of the problem is that they come home right after class and so don't have any of the discussions that cement into the brain the material of the course lectures. MOOC's bring a similar handicap of splendid isolation of the students from each other, but the forum provides a way for them to virtually get together and have those opinionated arguments with each other through the keyboard (hand-waving is not useful on the forums so some specific verbal skills are called for. At least you are safe from flying meter sticks approaching the speed of light).

So, again, if you take a MOOC, do take the time to participate in the forum associated with the course. Remember what Radio talk-show host Barry Farber used to say at the end of each broadcast: "Keep asking questions!"

A Chem teacher cautions against "Pseudoteaching"

Several times now I've come across mentions of "Pseudoteaching" in my reading, and I confess to not quite grokking what that meant. Today, the Google+ STEM community shared this short video and I think I'm beginning to understand.

"Blow something Up!"

I remember Mr. Greenberg, my high school chemistry teacher, but on the one hand I have trouble cataloging him as an "entertainer', but on the other hand I was always troubled by the seeming "magic" in chemistry class. See, when we stir these 2 clear liquids together, it turns pink. My brain would gripe "Why?". And that's probably why I went into software, not chemistry.

I think this video pairs nicely with the one linked to my blog post of 02/2013 "STEM Education Ideas".