Literate Programming?

Oh, what a tangled web we weave...

Many moons ago (The year 1984 if you want to get specific), Donald Knuth documented the idea of Literate Programming.   Well, he did more than document it.   He concocted software to support it, and applied the technique to his development of TeX.

While researching this topic, I found a page that offers some insight that rings true with me:

Literate programming, however, is not a mainstream technique. Those who use literate programming tools often wonder why not. There have been no studies done of which I am aware, but the basic shortcoming of literate programming is that it is difficult to write a literate program quickly. Yes, once it is written, it is impeccably documented, easily debugged (in those cases where it isn't already provably correct), simply maintained by the original author and others, and in general simply has a far higher quality in every respect than an "illiterate" program. But it takes longer to see results. As we all know, the software industry is an impatient one. And corporate IT in industries other than our beloved software are even less patient and less likely to understand the benefits of good coding style.

So, with some spare time on my hands, my mind has once again turned to trying to reach a level of comfort where I feel I've mastered Python.   My software development style is to write a high level overview of the target program and its major piece-parts as comments first.   My target audience for those comments is mostly just me, so if I get interrupted, I can get re-oriented to the intent of the code by reading what I wrote.   Maybe I'm showing my roots - back in the old days I wrote code in really low level languages (IBM 360 assembler language, for instance).   When developing code, I'm most comfortable sketching out the intent in words and then filling in the code and test cases.

So I was thinking about this and it occurred to me that this is quite similar, I think, to the intent of Literate Programming.  That got me wondering if anyone had done anything with Literate Programming for Python.   I suspected the answer was fated to be "no" since Literate Programming's debut was 1984 and it never really caught on, so far as I can tell, in the real world, and Python didn't debut until 1990, and has only gotten big in the world of software development in recent years (That's from my point of view.   You are free to argue contrary points of view).   So, I did a google search for:

python literate programming

and Bam!    Google found 32,600 web pages to match my search.   What a set back to my journey to that distant lofty land where I feel like I've mastered the fine art of charming Python!   I sure didn't read all 32,600 pages, but after reading a few, I decided to postpone tackling Literate Programming as a formal development methodology at this time.   I found an example of a Python Literate Program using a tool called xlp.   Sorry, but I found the example to be quite off-putting.   Consider for example, this line:

<define procedures 9,10,20,21,28,30,31,32,33>

Literate?   That line did not seem particularly enlightening as a snip of program documentation.   Numbered sections made me think of the programming language 'Basic".   Long ago I learned that if the code resembles Basic, to avert my eyes.  ("It is practically impossible to teach good programming to students that have had a prior exposure to BASIC: as potential programmers they are mentally mutilated beyond hope of regeneration.")  Perhaps this specific literate program is just an unfortunate example... 

The providers of PyLit, another Python Literate Programming tool have provided a web page that is somewhat biased, I believe, towards PyLit, but that provides an overview of other Literate Programming tools for comparison.  Maybe I'm overly sensitive to descriptions, but I was put off by the pro-PyLit page's categorization of PyLit as "semi-literate".   I confess to not having dug hard to find an example of a program written using PyLit, but it annoyed me that there didn't seem to be a conspicuous link on that overview page to an example.

From my reading of that overview page, the tool that left me with the most favorable impression was pyreport, but bear in mind that I didn't actually try the tool and that XML and LaTeX and reStructuredText are all things that from my point of view are potentially "in the way" of my getting comfortable with Python.

In the end, I decided to heed the advice of Monty Python and "Run Away!" from Literate Programming for now.   Things I think I learned from my looking into this today is that Python is reasonably flexible in the order of the code in the source files, particularly given that today there's not much cause to be shy about making a procedure call for the sake of clean packaging of ideas.   And Python already has more than enough provisions for embedding documentation into the program (e.g. docutils).  So, I'm proceeding with the expectation that I can produce reasonably readable programs without having to add another layer of tools to my learning curve.

So, what's to learn to be a Python expert?

1. Python language basics.   (that's the easy part...).  The trick of course is to write idiomatic Python and not to write in, say, C style when programming with Python.   As the old proverb cautions, there is no programming language in which it is not possible to code in Fortran.   Cleanly documenting the code with reStructuredText is sort of part of writing idiomatic Python code, so I'll mention that as part of this item.
2. Python library.   It seems to go on and on forever... (Don't you just hate it when the syllabus handed out on the first day of class tells you: "Required reading includes the Library of Congress, which has been put on reserve.").  There's the Python Standard Library, the odds and ends in the Cheese Shop and there are many frameworks (e.g. CherryPy) and other available bits of possibly useful freely available code.
3. Effective object oriented programming.   (my brain was well molded into seeing top-down structured code as the way to program.   That was back in the days when Dikstra's "Go To considered harmful" was still subject to spirited debate.  For example, see Knuth's wonderfully perverse for its time "Structured Programming with Go To Statements", whose title I more than once have mis-remembered as "Do You Suffer from Painful Elimination?".  So the mechanics of "objects" seem clean enough to me, but darned if I feel like I've got good intuition yet as to when it is appropriate to make something be an "object".
4. Driving a program through effective test coverage by developing good unit tests in conjunction with the code.  From my limited experience so far, in an interpreted language like Python, it is vital to exercise all the code at unit test time as a clean compile means far less than it does in a more "traditional" strongly typed at compile-time language.  Of course there are more than a few tools to choose among for unit test support of Python code (unittest, doctest,py.unit).  Which is best?   Hey!  I said I'm still just learning.  And, no, that isn't an exhaustive list of possibly relevant unit test frameworks.  And I've been sufficiently embarrassed at some dumb mistakes by myself to think pychecker or pylint or pep8.py may be important.   And, no, I don't think that's an exhaustive list of available static-checking tools either.
5. IDE's for Python.    I was awed at how much glop I accidentally introduced into my source when using vi to write Python code.  SPE (Stani's Python Editor) seems to be a much better fit, but as usual, there are lots of alternatives that I may be unwise in passing over.  (eric, idle, eclipse with pydev and ANT, just to name a few that I'm aware of).   I'll toss change control (e.g. subversion) into this item too since the IDE presumably needs to work with the source code control system.  (As I look into this, I'm finding a disappointing lack of integration of subversion and most IDE's.   If it's there, I'm not finding it in SPE).
   
6. Debugging Python code.  Of course there's more than one debugger to choose among.  Profiling/performance tuning needs to be mentioned somewhere, so I'll put it here.
7. GUI's and GUI development tools for Python.  There are too many choices and at this point I haven't a clue which is the right pick.   I suspect I'm best off with plain old least common denominator "tk" as created for TCL and used by Perl even though it is merely functional and not "pretty", at least not when I last looked.
8. Parallelism and Python.   Co-routines, threads, multi-tasking.
9. Packaging for distribution.   Python eggs.
10. Alternate implementations (Pypy, jython, ipython (debugger, IDE, not sure what to call it?), psyco, Shed Skin, Unladen Swallow, cython, Iron Python, Python 3.x).

And when I'm comfortable with all of the above, I'll be comfortable calling myself something of an expert for the Python programming language.  Once upon a time I considered myself an expert in the PL/I programming language.  PL/I is now all-but dead because it was too complex.   But somehow the learning curve for Python looks to be more formidable.  Of course back in the days of PL/I, there were no IDE's, unless you count 80-column coding pads, no GUI's, little concern about multi-processing, and a very limited library to draw upon.  Debuggers are pretty specific to running in an interactive environment.   In a batch processing programming environment, there were SYSUDUMP hex printouts of memory at the time of the crash.  And we walked up hill through the snow to get to school - and up hill to get home too.   Trying to learn OOP at the same time as learning Python is just a matter of trying to catch up with the times for having spent much time and energy on system and network administration instead of keeping my hand in as a software developer in recent years.

Drew
Revised 08/03/2009 to enlarge the list of alternate Python implementations.
Revised 12/20/2012 to clean up the HTML. Darn if I know how the list formatting and whatnot had gone bad.

Revised 08/08/2019 to again replace a failed link to the Knuth "Structured Programming" article with a URL that works at present.

Revised 11/26/2014 to replace a failed link to the Knuth "Structured Programming" article with a URL that works at present. I sure hate it when I use a link and then the web page it references vanishes.