Saturday, November 8, 2014

Playing with my Food

This blog article today is non-technical. It is about food, cooking and grocery shopping. Family members and my grad school room mate who are familiar with my low-level of expertise on these topics are probably doubled over in laughter at my choice of topic today. If you're wondering, today's article was inspired by an impulse purchase at Walmart, a bag of Pecan pieces that I've been especially enjoying.

My first wife, now ex-wife, had her faults, but she was an excellent cook. The difference in how we'd use cookbooks was fascinating to me. I can cook by following a recipe. I do that by leaving the cook book open on the kitchen counter and do the steps of the recipe step by step, measuring each ingredient carefully. The results are generally OK, but rarely amazing. My ex- would go out to the shelf with the cookbooks in the dining room, spend some time looking through them, announce to the air "Oh, I see", put the books back on the shelf, and then go out to the kitchen and cook. Her dishes inevitably included soy sauce and MSG among the ingredients and were generally very good and occasionally amazingly great, though they did lack reproducibility. My daughter's one complaint about her Mom's cooking was that she never did turn out great Brownies. I have heard it said that baking, more than most kinds of food prep, calls for careful measuring of ingredients, time and temperatures. Generally no room for soy sauce nor MSG in most Brownie recipes.

My current wife often tells me how great a cook she is, but she gets busy and rarely spends time in the kitchen. so I mostly have only her word on this. A seasonal specialty of hers is turkey. Most years in the run-up to Thanksgiving she roasts multiple turkeys to give away to friends, to the local police department, and even sandwiches for the day laborers that gather on certain street corners in town here hoping for work. But she manages to violate all the Food Network's tips on how to be sure of having your turkey come out perfectly (At most a 15 pound bird, no stuffing inside the cavity of the bird, no basting, ...). Her birds are inevitably the largest in the store, often purchased fresh, not frozen, because who has time to thaw such a large object? She stuffs it with more stuffing then I'd expect could fit, and then the cooking is extremely labor intensive with endlessly repeated basting to get the skin brown and crispy. The down side for us is that the turkey for our own Thanksgiving dinner often isn't ready until incredibly late on Thanksgiving day.

One of her turkeys, some years ago, actually made its way to the table in Reagan's White House. She got a nice thank-you note from Ronald on White House stationary, attesting to how great her turkey is. Who am I to question her politician nephews to determine if they indeed arranged for her turkey to be served at the White House or just forged a thank you note and ate the turkey themselves?

This year is shaping up to be different. We're planning to visit one of her adult children in Georgia for Thanksgiving dinner. I've actively discouraged all Barbara's thoughts that she should cook a turkey and bring it with us. Dian invited us, so leave it to Dian to prepare dinner. Offer to help in her kitchen, but if she'd rather do it herself, stay out of her way. We'll see how this goes. No guns in Dian's house, but it's hard to guarantee things won't break down into a carving knife fight in the kitchen.

Grocery lists

Much as I was surprised at how my ex- would cook without reference to a cookbook, I'm surprised that my wife goes grocery shopping without bringing along a shopping list. I can go to the store to pick up bread, milk and eggs without a shopping list, but if the trip is to stock up on more than that, I need a list. I keep a list on my desk at home. Whenever I find we are out of something or getting low on something, I add it on to my list. Occasionally, when I am looking at a recipe that calls for an ingredient we don't generally have on hand, that ingredient goes onto the list too.

I found that a problem with grocery-list-driven shopping is that sometimes an item falls out of inventory here and until I specifically miss it, we don't re-stock it, because it wasn't on the list. So now I do 2 things to assure variety in the household pantry:

  1. List things on the grocery list in more generic terms. e.g. instead of listing chicken noodle soup, I'll list "soup" and spend some time looking through what the supermarket has on offer. This does add time to the shopping trip as my wife claims to be allergic to pork, doesn't want too much salt, and says MSG gives her headaches, so I spend a long time reading ingredients lists as I'm picking out soups. We recently tried a Butternut Squash Bisque in a box that way, and I liked it.
  2. I force myself to browse the supermarket for things not on my list. Sometimes that results in fairly frivolous purchases, like flat-bottomed wafer ice cream cones. I do think I eat less ice cream if I pack a small cone with ice cream instead of scooping ice cream into a bowl.

    Seasonal items, like fresh apple cider, find their way into my grocery cart from looking at what the store has, rather than shopping off my grocery list. My wife makes a face and leaves the cider to me. She has memories of a rusty old cider press she saw in use on a farm some years ago and is convinced the cider isn't wholesome.

Pecan Pieces

A recent minor impulse purchase from the grocery section at Walmart was from their "seasonal" aisle, a 24 ounce bag of pecan pieces. I had no idea what I was going to do with them when I bought them, but have been pleasantly surprised at how much I've been enjoying them. Tasty as they are, I'll warn from having read the nutrition box on the side of the bag, that nuts are not a low calorie snack. They are extremely packed with fat, so that keeps them low in sugar and even low in carbohydrates, but high in calories.

So what to do with them? Well, one of my favorite simple uses is to sprinkle a small handful onto my bowl of cold breakfast cereal. This adds some interesting flavor and texture to even a plain old bowl of Cheerios. Sure, the store sells "Honey Nut Cheerios", but have you read the ingredients of those? They add almond flavor, but absolutely no nuts. No thanks. I much prefer to start with plain Cheerios and add my own sprinkling of genuine nuts. Good too on other varieties of cereal, Special K, for instance. One 24 ounce bag is sufficient for weeks of breakfasts.

We recently received a gift of a couple of large cartons of apples from upstate New York. An apple a day makes a nice crisp tasty snack, but what to do with all these apples? Well, one thing I did to use some of them was to make baked apples. Coring the apples with a plain old sharp knife was a bit of a hassle, but by using a covered Corningware baking dish in the oven, the recipe mostly just needed baking (and cooling) time. As Alton Brown was so fond of saying on Good Eats "Your patience will be rewarded".

I did add some pecan pieces to the cinnamon sugar used to fill the cored apples and the result was indeed delicious. My wife's opinion was that New York Delicious apples stay a bit too crunchy through the baking to make a really great baked apple. She thinks it would have been better if I'd used some big round Macintosh apples, but I used what I had on hand. She tells me her baked apples are way better than mine, but we've been together for more than a decade and not once has she served me a baked apple. So I don't argue about it, but if I want a baked apple, I guess, I'll just have to make it myself. On a cool night, it's actually nice to have apples baking in the oven making the kitchen warm and adding a delightful scent to the house. I wonder when I'll get around to doing that again?

Saturday, September 27, 2014

Actual code - C vs. Python for a small problem

If you aren't much interested in writing software, this post is probably not for you. If you read the post without following the links to the example implementations, then I'm not sure what you are doing here, though you are certainly welcome to ask questions.

Recently on someone asked to see C code to find the first 20 prime numbers. Seemed like an easy enough question, but the first guy who answered it, gave a C++-based pseudocode and some tips on how to convert that to C. That answer didn't make me happy. Heck, I figure if you're going to write in some other language, why not Python as a sort of testable pseudocode, but the question really needs that answer to then be re-stated in C.

So I sat down and banged out a Python program to find and print the first 20 prime numbers. That code is here:

I used a list to hold the list of primes. When the list grows to 20 items, I'm done. I start the list by telling it 2 is a prime number. I then only consider odd numbers as additional candidates. If the candidate is divisible by any of the prime numbers we've found so far, it is not a prime number. If the candidate isn't divisible by any of the prime numbers we've found so far, then the candidate is a prime number so we add it to our list of primes and, in ether case, add 2 to the candidate number to get the next candidate number to be tested. In C, we'll need a little more book keeping for the list, but since the max size of the list is bounded (20 items), things should translate from Python to C pretty easily. One trick that isn't entirely obvious is the use I made of Python's for...else control structure to catch the case of the for loop not exiting via break. We can paper that over with a goto in C, or you can add some flags

I was thinking that C has labeled break statements to let me do sort of the same thing as that for...else. But much to my chagrin, that's a feature of Java, not C. Oops. So, goto it is in my C translation of the program.

So I sat down with that Python listing open in one window and typed up find20primes.c That code is here: find20primes.c

I believe it is a straight-forward translation of the Python program into C. Of course, the C program is bigger in the sense of it has more lines of code. There are lines added to delimit the blocks and loops in the program, and lines added for variable declarations and lines added for the annoying book-keeping that C needs me to do where Python just handles those chores. I did run into some trouble where I didn't get the book keeping entirely correct the first time through. The program outputted 20 primes, but it started with 2, followed by 73, followed by 3, and it left out 71 at the end of the list. Huh? Turned out I was mistakenly incrementing primecnt before I'd used it to fill in the next slot in the primes array, so I skipped the primes[1] slot and just got 73 there by bad luck. Python would have told me about an uninitialized variable if there'd been a spot in the Python program for me to have committed that same kind of error.

Having finally gotten both programs working, conventional wisdom is that the C code should be much faster than the Python code. So I used the "time" command to see how the timings compare.

Using cygwin Python 2.7.8 on Windows 7 on a PC with an i5 processor and 8GB of RAM,

$ time python

[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71]

real    0m0.136s
user    0m0.000s
sys     0m0.062s

Using cygwin cc 4.8.3 on that same Windows 7 PC:

$ time cc find20primes.c

real    0m0.238s
user    0m0.045s
sys     0m0.185s

$ time ./a.exe

real    0m0.064s
user    0m0.015s
sys     0m0.015s

The execution time for the compiled C program was way less than the total time for the Python run, but if you add in the compile time for the C program, Python comes out ahead. Your mileage may vary.

By the way, if I throw in a -O option ("optimize the generated code") on the cc command, it further slows the compile down, while the run time is about the same.

$ time cc -O find20primes.c

real    0m0.336s
user    0m0.031s
sys     0m0.185s

$ time ./a.exe

real    0m0.086s
user    0m0.000s
sys     0m0.015s

(the "real" elapsed time varies a lot from run to run. These run times are so short the variability makes it hard to make fair comparisons in races between the programs). I suspect that the real time is dominated by the console I/O to print the results, so optimizing the details of the loops isn't going to do much improving of things.

Now to make things more interesting, suppose I wanted to pass in as a command line argument the number of primes to be found. So long as the primes are still in the range of ints that only needs a malloc of the array in C but if the program is ever asked to get up into really big primes, the Python code will keep on chugging while the C code will need substantial rework to pull in an arbitrary precision integer library module. This blog article is already long enough, so I'll leave that generalization to "findNprimes" up to you. How big does N have to be to break the C code's use of ints? I suppose that with relatively minor surgery, you could convert over to use of long ints (or if your compiler supports them, to long long ints) in the C program. Looking into it lightly, it appears that the cc included with Cygwin supports long long ints. The format strings will need adjusting. Use %lld instead of %d.

If you've feeling anxious to try further experiments, see how pypy does in handling the program. That'll probably be more interesting with with a really big value for N.

The moral to the story is that C may not be the best choice of programming language for this kind of problem.

Did you enjoy this article or was it too technical? Please add a comment down below so I know what you think about this sort of writing.

Tuesday, August 19, 2014

John Cotton Dana

I'm writing this on August 19, 2014. John Cotton Dana was born on August 19, 1856 in Woodstock, Vermont. I looked him up after I came across this snappy graphic on the Web:

I was surprised to learn the extent that he had influenced my life.

Wikipedia says he died in New Jersey on July 21, 1929. I did find another web page that says he died in Manhattan, not New Jersey, so once again we see that Wikipedia is not a definitive source, even if it is darn informative.

So how did this long dead person influence me? Well, for starters, in 1909, he founded the Newark Museum. I grew up in Union, NJ, about 9 miles by car from Newark Museum. My Mom would take me there for an easy day trip when we had nothing specific to do. I particularly enjoyed the science and technology exhibit of simple machines. I haven't been there in many years now and surely there has been turn-over of the exhibits, if nothing else, to make up for mechanical wear and tear.

Besides founding that museum, John Cotton Dana also was a librarian. I think my favorite line from the Wikipedia article was this:

He would have found a library school curriculum intolerable, and doubtless a library school would have found him intolerable.

Before John Cotton Dana, libraries tended to have closed stacks. A librarian would go fetch the book that you wanted to look at. Dana pioneered the radical concept of open stacks. The main library of the Rutger's-Newark campus is named after him. I'll leave it to you to forage around the Internet or your favorite local library to learn more about this man. Hey! At least read through the Wikipedia article, please.

Not all of his point of view would be accepted today. For example, Wikipedia says that he organized the first ever children's library room, but he believed its proper role was to help provide material to teachers. He was opposed to "story-time" at the library.

If you have kids, remember John Cotton Dana today by taking them to your local children's library, where I wager, you will find open stacks for easy browsing of the book collection. Even if you don't have kids, today would be a good day to make sure you know where your library card is. (You do have a library card for your local public library, don't you?) Make sure your card is up to date. If not, renew it, and use it.

Speaking of children's libraries, here in Westbury, NY, the founding of the local children's library pre-dates the establishment of the local public library for adults. The 2 only joined together since 1965. History of Westbury Children's Library

Revised 8/19/2014 to correct a wretched typo: Dana was born in 1856, not 1956!

Wednesday, August 13, 2014

Getting more familiar with Python Classes and Objects.

If you've been following this blog, you're aware that I feel unprepared to make good use of the Object Oriented Programming facilities of the Python programming language. Python is kind enough to allow programming in non-OOP styles, but Python's OOP features keep glaring at me, reminding me of what I don't comfortably know.

I posted a question on asking for real world examples of multiple inheritance being used in Python programs. I was disappointed about how few answers came back. That dearth of response left me with the suspicion that I'm not the only person around who isn't completely comfortable with multiple inheritance. Looking around at other multiple inheritance questions on Quora (, I see some reason to suspect that super serious application of OOP is just going to need more time to sink into the heads of the majority of developers. So, I'll continue to watch and learn, but will try to remember to adhere to the KISS principle to the greatest extent possible.

Additional Lessons

  1. This 40 minute video from Ray Hettinger (The Art of Subclassing) explains how Python classes differ from classes in other languages. Ray tries to reshape people's thinking here, so if you aren't already deeply steeped in OOP lore, you may feel he's dwelling on the obvious. He may give you some ideas of appropriate uses of inheritance in Python objects.

  2. Ray mentions this 2nd talk in his video. This 2nd talk was the next talk after his at Pycon 2012. "Stop Writing Classes", Jack Diederich, 28 minutes. Basically, that video asserts that my own example so far of writing a class for a Python program is not very good. The clue: My example class had only __init__ and one other method. I could have written it as a simple function and used the partial function from the library functools module to handle the initialization.

Further Reading

I have 3 previous blog articles on OOP in Python.

In Creeping up on OOP in Python - Part 1 I described a use of an object-oriented library module, pyparsing, to solve a Project Euler problem.

In Creeping up on OOP in Python - Part 2 I reworked my solution to add a simple class of my own. I was happy that introducing that class made the code cleaner to read. But if you watched the "Stop Writing Classes" video given up above in this blog article, you'll probably notice that my class is an example of exactly what they say you shouldn't do.What can I say? I'm still learning this stuff.

The 3rd in my Creeping up on OOP in Python" series was a bit different from the first 2. It explored an academic question about multiple inheritance. It is exactly the kind of A, B, C example that Ray mentions avoiding in his talk. Creeping up on OOP in Python - Part 3. I haven't forgotten my promise of a Part 4 as soon as I have a practical example of multiple inheritance to substitute for A, B, C and D in that academic question. But so far, there is no Part 4.

Ray mentions "the gang of 4". If you aren't familiar with them, here's a reference for you to pursue: And he mentions "Uncle Bob". I also mentioned Uncle Bob, with some links here: SOLID software design.

Know more about this OOP stuff then I do? Well, don't keep the info to yourself. Please post a comment with a link or example to help me learn more. Have you found a particularly relevant MOOC that you'd suggest?

Friday, August 8, 2014

TED Talk for Mature Audiences - Dismal Science, Disappointing Careers

For many years now, Economics has been known as the "dismal science". So, who better than a (Canadian) economist to deliver a 15 minute TED talk on "Why You Will Fail to Have a Great Career". TED talks, in general, are upbeat talks and this one, despite it having a somewhat cynical tone to it, really does have an upbeat message in there. I do worry in pointing out the talk that it requires a certain amount of maturity to hear the real message. The message an immature audience just might receive from this talk could be extremely negative. You've been warned! So, if you think you have the maturity to find the constructive up-lifting message in his talk, give a listen to Larry Smith of U. of Waterloo on why you will fail to have a great career.

The talk mentions the 2005 Stanford Commencement Address by the late Steve Jobs. If you haven't given that one a listen, I urge you to spend the 22 minutes that it'll take to play that one for yourself too.

you don't have to agree with the speakers in these video talks. If you have an opinion on what you've heard here, you're invited to leave a comment down below. I'm looking forward to hearing from you. And, of course, if you know someone who might benefit from giving a listen to either or both of these videos, please pass along the link to this blog article.

Saturday, August 2, 2014

Python and Parallel Processing

(Image taken from without explicit permission, but with acknowledgement of the source of the work).

I've written about Parallel Processing here before. e.g. See "What's the fuss about Parallel Processing?". I've been worried by my reading that seems to predict that imperative languages such as Python will never be able to safely cope with massively multi-processor architectures in the future. The "Downfall of Imperative Programming Languages" basically says that functional programming languages (such as Haskell and Erlang) are going to be the way of the future. I've been trying to get my head wrapped around Haskell, but so far without much success. So I was happy to hear a clear call from the Python camp that "I'm not dead yet!".

The article that gives me new hope is "Parallelism in One Line" the gist of which is that "Sure you can use the threading library to manage pools of threads (or of processes), but that takes a lot of lines of code and is very error prone, but there's another way to do it." The other way to do it is to use a parallel version of the map function. I, for one, didn't know that there's more than one version of "map" available in the Python library.

If you do a Google search for:

Python map

you will readily find documentation for Python's standard (iterative) map routine. e.g. "Python 2.7.8 builtin functions documentation - map".

If you readily want to find documentation of the parallel version of map that the Parallelism in One Line article talks about, you need to modify your Google search to:

Python map multithreaded

which will take you to a subset of the first search. I was amused to find among the search results a 2010 ActiveState Recipe for building your own "concurrent map" function, which drew a comment from one reader asking why not just use "multiprocessing". The recipe's author admitted not knowing about that one.

From that 2nd search I found "Multiprocessing - process based threading documentation". It does worry me that the documentation seems to have more complexity and gotchas than the Parallelism in One line article owed up to. Arguably I shouldn't be blogging about this at all until I've actually given it a try myself (but I still don't have a multi-core processor here at home).

It's a trick?

If you've been reading this carefully, you might rightfully object that it's all a bit of a trick. The map function is an element lifted from the world of functional programming and then provided in Python. The parallel version is only safe in Python if the function that you are mapping is "pure". If the function code has side-effects, then you will have race conditions and potentially suffer horribly at the hands of multi-cores. The language isn't going to inherently protect you so you have to be careful out there.

If you want to read more about the challenges of Python vs. multi-core architectures, see "Python's Hardest Problem, Revisited", by Jeff Knupp. The piece parts to roll your own multi-processes or multi-threaded Python program remain available, but be sure you have plenty of iodine and bandages on hand if you cavalierly venture into the world of multi-cores in a language that makes no promises of (much) concurrent processing safety. Design your code carefully and watch out!

Wednesday, July 23, 2014

Danger Lurking in the Plastic Packaging of our Food?

I stumbled across this article from a web site unfamiliar to me:

Worrisome, but I didnt want to sound the alarm without checking into it further. So I did some Google searching looking for the same warning from some other source. Found it from National Geographic and that's good enough for me to consider the story to be confirmed.

The troublesome chemical showed up in plastic bags imported from China, but not in bags of local (Spain) manufacture? I wish I had more confidence that the matter would be properly handled and fixed by China. But I've not seen any tendency toward openness in China.

In case you need more to worry about, here's a 17 minute TED talk on some common pesticides in our environment. You say you aren't worried about frogs? Well, you probably should be.

What should you and I do about these problems? Alas, I haven't a clue about what to do. A sternly worded letter to your congress-person (who likely is heavily under the influence of some big agricultural chemical company) would be a beginning, but I've not got much more respect left for our government than I have for that of China.

Am I the only one who has noticed that the food labels track most bad things in milli-grams (mg) but trans-fats on the label are tracked in grams (g)? In case you don't remember, 1 g = 1000 mg. Anything under 500mg of trans-fat in a "serving" is loudly touted as zero grams of trans-fat per serving by the power of rounding down to the nearest whole number of grams. And there are some pretty funny notions of "servings" to be found on those labels. Muffins from my local super-market come in packs of 4 muffins, zero grams of transfat per serving, but a serving is 1/3 of a muffin. Makes me wonder if 1/2 a muffin has enough trans-fat in it that they'd have to round it up to 1g on the label.

Dietary guidelines for trans-fat is to keep the amount that you eat each day as low as possible. Heck, even the guidelines for sodium allow 1500 mg/day of sodium as a recommended daily allowance and 300mg/day of cholesterol. (Speaking of which, have you looked at the nutrition facts for a McDonalds egg and sausage biscuit? 1170 mg of sodium in one. 250 mg of cholesterol in one. But if you are hungry, one of those isn't much of a breakfast. Advice: don't make a habit of McDonald's for breakfast.

Now if our government really had our best interest in mind, wouldn't it push the food company's to report tran-fats in mg (like they do for cholesterol and sodium)? But does that happen? No. Obviously way too controversial a matter? Right up there with identifying which products contain GMO corn.


Sunday, July 6, 2014

The Curious Tale of Epigenetics

My blog article today is a bit off of the beaten path for me.

Instead of looking at a software topic, today's blog article topic is more biological: epigenetics.

I'll confess up front that I'm not a biologist and never took a genetics course in my life. Close as I got to such a course was I had friends in college who took the genetics course and they sweated through worrying about the progress of their cohort of cross-bred fruit flies. And I did work my way through Prof Keeton's Biology 101 course and its Biology 102 sequel. And when my wife was pregnant, when the pregnancy had gotten far enough along we had a sonogram and amniocentesis to make sure all was as well as could be gauged at that time. That's the full extent of my background in genetics. If I'd actually studied the topic in further depth, I suspect I'd now be having one of those "Everything You Know is Wrong" moments.

This article that I read this week is what brings all this up:

Grandma's Experiences Leave a Mark on Your Genes (May 2013, but talking about a discovery that dates back to a couple of guys having beers in a bar in 1992).

I think my favorite part of the article is on page 3, after the researchers had made their outlandish sounding hypothesis and carefully ran 3 different experiments to confirm that they were right and wrote the results up in a paper only to have the paper rejected:

Despite such seemingly overwhelming evidence, when the pair wrote it all up in a paper, one of the reviewers at a top science journal refused to believe it, stating he had never before seen evidence that a mother’s behavior could cause epigenetic change.

“Of course he hadn’t,” Szyf says. “We wouldn’t have bothered to report the study if it had already been proved.”

But there's a happy ending. The paper was published in the June 2004 issue of Nature Neuroscience and here, just about 10 years later, the news has actually caught my attention. Maybe next blog article I'll talk about a Princeton physicist who suggests that E=MC2. :-)

The one thing that bothers me in the linked article about epigenetics is I see nothing that discusses the effects of the mother's contribution to the genes vs. the father's contribution to the genes. Presumably the methyl groups from Mom are not in the exact same places as the ones from Dad. How do the resulting gene pairs interact? Maybe if I dug enough to find the answer to that, I'd lose my amateur standing or something.

And not to be facetious, but remembering the lab struggles of my friends taking that genetics course back in college, I do wonder if epigenetic effects can be demonstrated with fruit flies or something else with a life-cycle that fits within the time constraints of an undergraduate semester. The answer perhaps lies in looking for failed experiments where the dominant gene got trumped by a gene that was supposed to be recessive. I'm pretty sure that not every undergrad experiment produced the expected results. Quoting Issac Asimov: "The most exciting phrase to hear in science, the one that heralds new discoveries, is not 'Eureka!' but 'That's funny...'". There'd be a certain amount of fun in being able to say "I failed the lab part of the course, but got the results published in Nature".

If you found this article to be as eye-opening as I did, and you know someone else who might enjoy it too, please pass along to them the link to this blog article. And, of course, if you know something about this topic, feel free to add a comment to further my education.

Friday, June 27, 2014

Home Networking with FIOS - Don't cross the streams.

As you may remember from my article Adventures in Home Networking, we very badly wanted to get out of using Cablevision. They'd agreed to a price to provide service to our home, but then consistently billed for significantly more per month then the agreed-to price. We got the NY Public Service Commisssion involved, but every month's bill turned into a new argument with Cablevision with the PSC insisting that they accept the payment of the agreed-to amount. Cablevision would accept the payment and leave the service connected, but rebilled the unpaid difference the next month as an amount past due. There were some months they'd insist they hadn't received the payment and we'd have to get a 3-way call between the bank, Cablevision and the PSC to establish that Cablevision indeed had collected the previous month's money from the account and then mis-placed it.

We got tired of that monthly argument, and looked for alternatives. The hard part was finding Internet service. Given Internet, it is easy to get phone service (e.g. Vonage or MagicJack), and there are non-cable alternatives to television service (e.g. DIRECT-TV and Dish satellite service). We had tried Hughes Satellite Data service as an Intenet Service Provider, but found it to be expensive (limited data transfer allowed per month), high latency and noticeably prone to dropping packets. We concluded that Hughes would not be a good substitute as an Internet Service Provider for our home. As I described in that January, 2014 article, we also worked out how to put our home on AT&T's Mifi service (cellular LTE data service). The problem we had with it was again limited data transfer allowed per month, $10/GB for overages. It was fast enough, and seems reliable enough, but at $52/month base, is somewhat expensive. It also had the irresistible temptation to take the access point along when traveling out of the house, but that would leave the house without service while the Mifi was on the road. In the end, we kept the Mifi box and grudgingly pay the $52/month just for the convenience of having Internet access while away from the house. (Well, the other rationalization of the $52/month is that it ensures that I have Internet service even if my home's ISP is out of commission. It's not just that I'm addicted to blogging. I use the Internet for work too). Cablevision, of course, sings a song about having Wifi hot spots in many places. But our experience with those is that often the signal is too weak to be usable. e.g. there is no Cablevision Wifi at our house. If I go down the block, there is Cablevision Wifi in the nearby park. At our local supermarket, there is a signal if you walk out to the sidewalk near the street, but inside the store you are out of reach. Similarly, at some of my doctor's offices, there is a signal out at the street, but inside at the reception desk, the signal isn't strong enough to be reliable. The Mifi box is more secure (uses your own WEP password for encrypted Wifi) then Cablevision's unencrypted Wifi and AT&T Mifi seems to work well even inside low-rise buildings such as are common here in the suburbs.

So, after running out of alternatives, we decided that we'd drop Cablevision and go with Verizon FIOS even though it wasn't going to save us any money. The one advantage seems to be that FIOS includes Jimmy Swaggart's Son Life Network which my wife enjoys. If you want to hear a disgraced minister sing hymns, maybe you'll like it too. It's probably easier to enjoy if you simply reject those repeated charges of cavorting with prostitutes as "obviously" false (New Orleans and Los Angeles). If I was providing a sound track to go with this paragraph, I think I'd opt for the Beach Boys "California Girls". I'm just not creative enough to produce a mix of that and some suitably religious ditty. Not that it is in any way a religious ditty, but how come it seems so rare that the radio ever pairs up California Girls with "Back in the USSR"?

It probably helped minimize any shock from the bill that the price negotiation with FIOS was done by me, not my wife, so we came away with more realistic expectations of what the bill was really going to be. Verizon loves to tack unpleasant surprises onto the bill. e.g. unless you ask, the prices they quote don't include things like $27.98 "Taxes, Governmental Surcharges and Fees" and $25.55 "Verizon Surcharges and Other Charges", but the bill includes those. The first 3 months of service also carry an extra $49.99/month "Activation Fee". There were still surprises to be found by careful inspection of the bill. My wife is flummoxed by the notion of keeping an address book organized by name so you can find people's contact information. Her habit is to write the info on scrap paper, which she then loses, so she repeatedly calls 411 to get the number again. In one month that was 30 calls to 411 at $2.49/call. I keep pleading with her to learn how to use Google, but that just turns into my having to do it for her and getting peeved when I recognize that I'm looking up the same information as I'd given to her before. I wrote some of the contacts into a bright orange address book and she acts surprised every time I show her that she already had the information there.

Video problems with FIOS

The other bad surprise, in our 2nd month bill, was $156.03 for a service call we'd placed with Verizon. Multi-room DVR playback to rooms remote from the actual DVR was unreliable. They sent a guy out to trouble shoot the problem, but he didn't fix it and he didn't write it up as a Verizon problem, so they billed us for an hour of labor. I protested and they did deduct the labor and the tax on the labor from that month's bill. We found more ways to demonstrate the video problem and called for yet another repair visit. The 2nd repair guy was unable to fix the problem too, but we learned a little from chatting with the guy while he was here. In the world of Cablevision, so far as I can discern, data service and television service are quite separate, sharing the same physical cable but traveling in separate channels. But with FIOS, each television set-top box gets an IP address. Some television services depend on being able to talk via the Internet router to get video to the set top boxes. Remarkably, the Verizon service folks don't have any kind of protocol analyzer or even a simple bit-error-rate monitor to give them visibility into what is doing on the home network.

The first repairman's visit wasn't without value. He noticed that our HP printer was assigned IP address This had been our household's long standing convention, so any PC could know how to reach the printer without needing DNS or something complicated to find it. But thru are "reserved" by Verizon for use by their set-top boxes. Oops. So I reassigned the printer to be and adjusted the configurations of each of our PC's accordingly. This did make access to the printer more dependable, without having to fuss with power cycling it after a period of disuse, which had been the state of things since we switched to FIOS. But it didn't get rid of the problem with multi-room DVR playback. We'd also found that the FIOS video tutorials, which are short video-on-demand clips from some central server of theirs also had similar problems of occasional stalls and pixelization in playback. The 2nd repairman saw the problem happening and after examining the router configuration opined that we just had too many devices on our home network for the level of service we'd subscribed to. (Our service is their 50/25 speed. For more money, we can have more bandwidth, but since I'd seen the whole house [excluding FIOS video, of course] doing fine with LTE cellular data service via AT&T Mifi, I didn't think that was a good explanation of what was wrong). To the fellow's credit, he did take the time to disconnect portions of our home network from the router until the problem went away. At that point I had to accept that the problem somehow lay with our home network, though further trial and error would be needed to find the real problem.

So, at this point we had as our only instrumentation, that when the problem wasn't present, we could watch Verizon FIOS training videos without a problem, and that when the problem was present, that a few minutes of playing the training video would result in pixelization and stalled playback. I set about through trial and error to find which of the wires he disconnected mattered. My first suspect was the long cat-5 cable to the router in the garage apartment. The dog had occasionally chewed on that cable and we'd taped it up and it seemed to work well enough, but I certainly wasn't prepared to swear the cable still met all cat-5 specifications. I plugged it back in to the FIOS router and the picture stayed clear, so it wasn't that cable. Only other cat-5 cable I knew about was one that runs through the attic to get to the rear bedroom where the printer resides. Puzzler was that while doing my wall crawl to reconnect things, I found 2 cables running up to the attic. What was the additional cable? Took some more crawling in the rear bedroom to figure it out. Turns out we had 2 cat-5 cables running from the front bedroom to the rear bedroom. Once upon a time, one was for the PC in the rear bedroom and the other was for the printer, but now the PC and the printer are connected to an inexpensive 100-base-T switch so the 2nd cat-5 cable to the room is unneeded. Mistakenly, both cables were plugged into the Verizon router in the front bedroom, and both were plugged into the switch in the rear bedroom. That's just plain wrong. I don't think it ever bothered the Linksys router that used to be in the front bedroom, but the FIOS router showed video playback problems unless you disconnected the erroneous 2nd cable from the router to that 100-base-T switch. Oops. Mixing video playback in with the mostly non-realtime Internet data traffic apparently got fouled up by the wiring mistake. What a pity that Verizon didn't have appropriate tools for their field support people to really find this problem. Anyhow, now that the extra wire is disconnected, things have been much better with the multi-room DVR.

More FIOS Woes - You Own the House Wiring

But we did have other FIOS problems. Cablevision ran a coax cable to a "cable modem" in the front bedroom. There were 2 phone jacks on the cable modem that provided our house phone line and our fax phone line. FIOS doesn't work that way. FIOS mounted a big white box on the outside of the house and ran the fiber to that box. They also installed a large UPS on the wall inside the house and ran power from the UPS to the white box on the outside wall. They then re-used Cablevision's coax cables to connect a coax cable from the FIOS box to the various TV's via a splitter box on the outside wall just below the FIOS box. There is no "cable modem" box as we had with Cablevision, but the Verizon Internet router has a coax connection of its own that it uses to talk to the FIOS network, including the settop boxes. The outside FIOS box also has about 8 phone jacks in it. So, Verizon ran 2 phone lines from those jacks to ancient NY Telephone demarc boxes on the outside wall, to tie the 2 phone lines to the ancient inside house wiring. The deal as I understood it was that as long as Verizon provided dial tone at the demarcation box, that any remaining phone problem was a "house wiring" problem and is only covered by Verizon if you pay them a ridiculous monthly surcharge.

After a particularly rainy Spring night here, the Fax machine lost it's dial tone. I know I had a corded phone around the house, but darn if I could find it. So we traipsed off to Walmart where for $6 I bought a nearly feature-less corded phone. It looks sort of like an old Trimline phone, but I found the base is just a place to set down the handset. The wire from the walljack runs straight through the base and into the handset. The touchtone keypad, ringer, etc. are all in the handset.

Armed with my corded phone, I could verify that things were funky at the demarc boxes. For starters, the boxes weren't labeled with current phone numbers, but worse, the installer had hacked around some problem in at least one box so the modular jack for test access wasn't working at all. So we arranged for a service visit for the coming Monday for Verizon to look at lack of a dial tone on our 2nd line.

More FIOS Woes - the Fragility of the Fiber from the Pole to the House

But Friday, before we ever got to that Monday service visit, something happened to our FIOS service. We lost everything - TV, phone and Internet. Verizon, contacted via cell phone, agreed to send a repair guy out on Saturday. The FIOS box has an LED in it to tell the repairman if the fiber is delivering a signal to the FIOS box. It immediately told the repairman that there was no signal. Alas, he arrived on a truck with ladders, but the fiber drop cable to our house is connected to the distribution fiber mid-span, between 2 telephone poles, so the junction is only accessible with a "cherry picker" truck.

From the ground, something looked dangly at that junction, but nothing was actually laying on the ground. So we waited for them to dispatch a cherry picker truck with a buddy of the repairman's to help him. Once that arrived, they determined that something had severed the junction from the distribution fiber to the drop fiber. So they had to replace the drop fiber. This was surprising to me as this is now my 4th drop fiber in the short times I've had FIOS service. The first drop fiber was installed much like this one, but drooped a little low over the street. One day a large cement mixer truck drove down our street and managed to knock down our fiber and the one to the house next door too. So Verizon sent a repair crew to replace the fibers to our homes. They sent 2 trucks, one for each house, but they decided to work together. The lady with the ladders worked the house end of the job and the guy with the cherry picker truck worked out by the poles on the street. I was impressed with the result. At the house, the fiber came off a board sticking up from the peak of the house's roof. It then flew super high across the street direct to the pole at the corner. It was well clear of any trees and certainly too high to ever get struck by a passing truck on the street. Beautiful looking installation. Alas, it was no match to Hurricane Sandy. Fiber #2 ended up on my lawn after Sandy passed through. Annoyingly, Cablevision's coax to our home survived the storm just fine. Since we were off of FIOS by the time of Sandy, we only had Verizon out for a non-emergency clean-up call to get their fiber off of my lawn. Of course, when we resumed FIOS service this Spring, the installer had been told this was a simple re-connect of a previous customer. Lots more to the job than he'd expected. He installed drop-fiber #3. It ran from the board at the roof peak through the trees to the mid-span point where drop fiber #1 had been attached to the distribution plant. I think the board at the peak added enough height that #3 was going to be safe from passing trucks. But the community center opened up and installed a driveway under the distribution span. The guy who installed drop fiber #4, opined that maybe a school bus turning into the community center disturbed the distribution fiber enough to jostle our drop-cable into failing. He put in a request for the outside plant people to come around to raise the distribution cable a smidge where it crosses the driveway, but I didn't see them come to do that. We'll just have to wait and see how long drop fiber #4 lasts.

The Internet tells me that I'm not the only FIOS customer having repeated service calls for fiber repair.Hungry Ants Knock Out FIOS Service ... Again. I can only assume that these problems are evidence that fiber to the home is new technology and there will be some time before they gain enough experience to work out the Field Engineering kinks to make this reliable. The price of repeatedly replacing the drop cable presumably is enough motivation to encourage Verizon to eventually get it right. The Mifi box and my cell phone at least make it tolerable when we have an occasional day of no FIOS service. Of course, there is the annoying whining sound of my wife complaining when she misses an episode of General Hospital, but so it goes.

When the fiber connection to the outside FIOS box was restored, I asked the Saturday repairman to look at the problem of no dial tone to our Fax machine. He advised that the white FIOS box is now the point of demarcation. He verified that there was dial tone available on lines 1 & 2 at the FIOS box, but that means the crappy old NY Telephone demarc boxes are now part of the house wiring that is my responsibility.

So I looked and found on the house a disused demarc box that then had a phone line running back to the rear bedroom. My wife reminded me that some years ago her Aunt Dolly had lived in that rear bedroom and had a phone line of her own. Aunt Dolly has long since moved out of here and is now deceased. We attended her funeral this Spring. So, I snipped the wire off the old NY Telephone demarc box and ran it into line #2 inside the FIOS box. I added a new modular jack to the spot where that line comes into the rear bedroom and then ran a phone cable from that modular jack to the Fax machine. Voila, the Fax machine has dial tone again and we're using less of the ancient house wiring then we'd been using before.

The future of FIOS??

There's some evidence that Verizon is unhappy with their return on investment in converting their distribution network from copper to fiber. e.g. Wall Street Journal: Verizon to End Rollout of FIOS and DSL Reports: Verizon Again Confirms No Future FIOS Expansion. Will the day come when they try to close down their fiber business? Seems unlikely to me, but we do live in "interesting" times.

I've got to wonder where are the regulators in the process of Verizon making non-aggression pacts with the cable television companies. e.g. Verizon’s Anti-Aggression Treaty With Big Cable May Be the End of FiOS. I really think it is better for the economy when the cable companies have at least one competitor in every neighborhood, preferably one with a different technology and the fire in the belly to want to rewire the country to fiber. Verizon once upon a time had that fire, but now, like AT&T, seem to have lost their sense of direction. Too bad!

Cross the Streams?

I keep forgetting that Ghostbusters was an increasingly long time ago (1984). When our FIOS video problems turned out to be from the mixing of Internet traffic and FIOS video packets, Egon's warning about the dangers of "crossing the streams" immediately sprung to my mind. If you haven't seen Ghostbusters recently, you really should rush out and rent it. But thanks to Youtube, we can give you the relevant clips from the movie. First, the scene where Egon warns "Don't cross the streams!" And, second, the climatic scene where the team makes a slight change of plan and deliberately crosses the streams. A bit of a spoiler, but you still really should see the whole movie.

Message In a Bottle

If you know someone who maybe knows someone in Verizon who'd be in a position to officially react to this note, please pass along the link to this article. I'd be delighted if my woes actually got to the ears of someone who could be properly embarrassed at the lack of network diagnostic equipment in the hands of the Verizon field support staff, or the need for improved field engineering of drop fiber installation and could maybe nudge Verizon in the direction of actually doing something about it.

Sunday, June 22, 2014

How Secure is Your part of the Internet?

Problems with Internet security have been much in the news lately as our homes have been getting more and more connected to the Internet with the growth of the Internet of Things. Please note that I'm one of those people who make a distinction between the terms hacker and cracker.

Image borrowed, without permission, from, apparently the blog of Arief ucHulz.

A specific recent report that caught my eye:

Until We Fix Our Connected Homes, Crackers Will Keep Screaming At Babies

The simple conclusions offered by that article is that if your home is connected to the Internet, to be secure you should:

  1. Take care to set secure passwords on all your devices. Leaving the manufacturer's defaults are just asking for intruders to come by.
  2. Register your devices with their manufacturer so the manufacturer can get in touch with you about security updates. Unsaid is that such registration will open up your e-mail inbox to a likely flood of promotional e-mail (a.k.a. spam).
  3. Keep your device's firmware/software up to date. Unsaid is that not all devices can accommodate updates, and not all manufacturers put much effort into providing updates on old products. Maybe the manufacturer no longer makes that product. Maybe the manufacturer no longer supports that product. Maybe the manufacturer has gone out of business.

The article mentioned briefly that the homeowner had "secured" their home's router that connected their home devices to the Internet. I wished that article had explored that statement in a little more depth. "Securing" a router is quite an essay topic in itself. If you have a router connecting your home to the Internet, please stop and consider how secure is it?

  1. Have you set a secure password so only you have administrator access to your router?
  2. Assuming your router provides you with Wifi (wireless Internet connectivity), have you configured the router to have a serious, non-default password protecting your Wifi network from intruders? There's more than one choice available for Wifi Encryption. Which Wifi encryption option have you picked? If you picked WEP, you'd be well-advised to switch to WPA2. There's freely available software that anyone with a notebook PC within range of your Wifi signal can run (e.g. Automatically crack Wifi with only 30 seconds work. I offer the link as an example, but haven't actually tried to follow that page's instructions myself). The software needs only to listen to your WEP-encrypted traffic for a short time and it will then reveal what the password is that your Wifi network is using. In other words, WEP encryption isn't at all secure if faced with anyone who wants to intrude on your wireless network.
  3. And now we get to the hard question: What is your router configured to do with incoming traffic from the Internet? If the router rejects all packets coming from the outside world, it isn't going to be much use. If you use your browser to visit a web page, you send out packets requesting the web page and web page's server sends back packets that tell your browser what the web page says. If you configured your router to reject those outside packets, you'd likely be most unhappy with your router's behavior.

    Most routers will let you accept only traffic that comes in reply to packets that you sent out to the Internet and that covers most cases. But there are ugly cases where, for example, you initiate an FTP connection and the remote FTP server replies using a different port than the one that you used to initiate the connection. If your router is configured to reject packets that don't look like replies to traffic that you initiated, you're likely going to have trouble doing FTP file transfers.

    A stickier problem is do you ever want to access your home Internet from elsewhere? For example, some folks have home security systems (or perhaps a baby monitor) and want to be able to check in on it from their travel PC while away from home. Almost certainly that requires the router to be configured to allow outside Internet traffic, traffic that isn't in reply to inside-traffic, to come into your home. How secure is your home network to outside traffic coming from a wannabe intruder?

You can try to secure your home, device by device.

Consider, for example, the HP Inkjet printer in my home, an HP 7410 all-in-one printer/scanner/copier with built in Wifi capability. When we first set it up, we put it in a room that didn't have wired ethernet available so we configured it to use that built in wireless capability. Worked wonderfully. Then as I became aware of how easy it was to break WEP encryption, I decided to reconfigure our home router to use WPA2 encryption. Surprise! The HP 7410 printer's built in Wifi support only knew how to handle WEP, not WPA2. I looked into how to upgrade the firmware of the printer, but so far as I can see, the firmware of that printer is permanent, not updateable. Now, for all I know, I could go out and get a new printer that has WPA2 support, and probably has other advantages like, perhaps a faster printer speed and just maybe less expensive ink. (well, I can dream, can't I?). But this printer still works fine, so I'd feel guilty throwing it away just because it doesn't support secure Wifi. So, instead, I grumbled and bought a long Cat-5 cable so I could have a wired ethernet connection from the router to the room with the printer. For less than $10 we also added a wired 8-port 100-base-T ethernet "switch" to that room so we could easily connect all the devices in that room to the wired network. I talked a bit more about my home network back in January. See: Adventures in Home Networking

Barry Shein, one of the early pioneers of the Internet as we know it today, recently posted this note to Facebook:

Internet security is so bad because it was never particularly designed to be secure.

I've been on the net since the 1970s, involved in its technology and politics. I don't remember anyone ever saying or promising "and it has to be secure, right?" until the 1990s.

What happened is in the 1990s a bunch of people figured out how to make A LOT OF MONEY off the net, or save a lot of money, same thing.

But most of their plans required the net to be secure.

So security became a BIG ISSUE. Ok.

It's like coming to a river in your car and thinking hmm, maybe I can just slap some wings on this thing and fly across.

The power of the net is that it enables everyone to share information very easily and very widely.

Now, re-read that sentence with security in mind.

If you aren't uncomfortable yet, I've got more for you to read. Shortly before Barry posted that cautionary note on Facebook quoted up above, he posted on Facebook:

If you try to engage me in a conversation about computer and network security and I don't know for a fact you're an expert I'm going to check whether you read this article. And if you haven't I will politely ice you out.

Everything is Broken

Now there are many different opinions as to what you should do. I don't have the energy or time this morning to track down exact references for what Richard Stallman suggests, but at the risk of mis-reporting what he has in mind, I'll tell you what I think he has said:

  1. Don't trust software that you can't examine and modify on your own.
  2. Don't allow untrusted 3rd parties to have control of the software on your devices. e.g. allowing auto-updates of your PC by Microsoft, Apple,Adobe, Oracle (Java) and Google (Chrome, Android) is imprudent. Even if you trusted the software after careful evaluation of it yesterday, how do you even know what the software you are running today will do?

The trouble with "trust no one!" is that you are cutting yourself off from much of the world. And even if you insist on only running software where you can examine the source code, you likely are only fooling yourself. There's too much software in layers and layers for you to have any hope of being able to detect security problems. Security problems can be quite subtle and hard to recognize. Consider for example the recent brouhaha over the security of OpenSSL in its Debian implementation. The source code was all open and freely available, but it took years for anyone to notice that a security bug had been introduced into the code. The xkcd comic had some good jibes at the security of other open-source systems:

I'll go so far as to suggest that if you refuse to allow auto-update of the software on your devices, you are doomed to never being able to keep anywhere close to current on the latest security updates. There are just too many of them and they come out too often to try to track them by hand. And you'll have a hard time convincing me that the reason you insist on tracking them by hand is you want to research what each one is about before you install it. Good luck with that!

And then there's the problem of web-services like dropbox, gmail, Google drive, Google docs, Facebook, ... Pearltrees, and the list goes on. Generally, you don't get to see the source code that implements those services, and often you have no control over when that service implementation is updated. At some point you have to decide which suppliers you are willing to trust. Stallman will tell you that Facebook surely doesn't belong on that trusted list. My wife has no Facebook account and insists that no one should share her picture or name there. Her children don't buy into that "no Facebook" policy because it would cut them off from keeping in touch with their friends.

I could go on and on, suggesting that you look into "Virtual Private Networks" for securely allowing connections from the outside Internet into your home. But you pretty much have to trust somebody to do the right things to protect you.

But who can you trust? 12 biggest baddest [known] software backdoors of all time. "all time" in that title underestimates what the future could hold. And the article isn't very keen to point out that it is only talking about known backdoors. Goodness knows what unknown backdoors are lurking out there.

In closing, here's a 17 minute TED talk that defends "hackers" as a necessary part of the Internet eco-system. The talk doesn't draw a distinction between hackers and crackers, but so it goes.

Hackers: The Internet's Immune System.

Thursday, May 15, 2014

Comparative Programming Languages

Here are a couple of videos from luminaries in the software world that I believe are worth the time to watch.

First is Rob Pike explaining why he created the Go programming language.

OSCON 2010: Rob Pike, "Public Static Void"

That's a short talk of 12.5 minutes.

Far longer at an hour is this talk by "Uncle Bob" explaining what killed Smalltalk and how the same factors could kill Ruby too.

RailsConf 09: Robert Martin, "What Killed Smalltalk Could Kill Ruby too"

Note, that when Bob finishes with the last of his note cards, the talk is NOT over. I urge you to stay in your seat until the end of the video. Patience!

Now to be fair, I'll have to admit that I haven't programmed in Ruby nor Go nor used Rails and I'll even confess to limited experience with Test Driven Development (TDD), but I think I've got to get one of those green wristbands for myself. So if your reaction to the prospect of watching these videos is "Ain't nobody got time for that!", well, as a consolation prize, I'll give you a link to a related, but short to read XKCD comic:

How Standards Proliferate

Saturday, April 26, 2014

Penny for NASA?

If you've been paying attention to the social media, you may have noticed that there's a lobbying group called "Penny for NASA" that is pushing the idea that we should double federal spending on NASA from it's current level of 1/2 of 1% of the federal budget to a full penny of each dollar spent by Uncle Sam. Back at the height of the moon program in the 1960's apparently NASA's budget had been as high as 4+% of the federal budget. Penny for Nasa web site.

Space exploration is exciting, and I suppose it is unsporting of me to point out that there are some awfully strong other contenders for priority in allocating the federal budget.

Their video makes mention of a "petroleum industry subsidy" that dwarfs NASA's budget. So this got me to wondering where exactly to find a line item for "petroleum industry subsidy". Doing a little digging, I find that apparently it is buried in the complexity of the federal income tax and comes in the form of taxes not levied on petroleum industry profits. e.g. See

A presumably harder to sell proposition is: Let's reform the income tax and raise the effective corporate tax rate and then spend more of the revenues on space exploration. If there was a net increase in federal revenues, there are many contenders for the money with deficit reduction at the top of the list. Even if the feds were persuaded to increase the spending on science and technology, I expect NASA would be in for some stiff competition for the money from folks who'd like to see more medical research or more work on alternative energy sources. The old "If we can send men to the moon, why can't we ...", for your favorite choice of "...".

So far, we have visited the moon and established a "permanent" space station in low earth orbit, and done some limited robotic exploration of Mars. Manned exploration of Mars is exciting to discuss, but difficult problems like fatal levels of radiation exposure while in-transit to Mars remain ahead without obvious solutions. Star Trek-ish "shields" and warp drive (to shorten the transit time) don't seem to be in the cards at present.

Perhaps a more do-able objective would be a permanent manned base on the moon to serve as a lower gravity launch platform for deeper space exploration. To be a "Good Idea" I think it would be dependent on showing the base could be built drawing on materials from the moon. Otherwise, maybe the better choice would be an even lower gravity high orbit space station. If nothing else, digging deep to put some of the mass of the moon into use as radiation shielding might make the moon more suitable than a Disney-inspired high orbit permanent space station.

If we're going to have folks be permanent citizens of the moon, it is probably time to re-read Heinlein's "The Moon Is a Harsh Mistress".

Or has Material Science gotten far enough that we should start by building a space elevator?

Certainly there's no shortage of places to spend money, if we have money for such discretionary expenses and the vision and political will for adventure. The politics of the discussion are likely to devolve into debate on the economic stimulus value of the proposed increase in federal spending. An argument for "Investing in the future" sort of demands the context of a long range plan for the future. Even JFK's "we will go to the moon" had more to it than just an intent to go to the moon.

Am I being too harsh in suggesting that there's much more to consider than just a blind doubling of NASA's budget? The social media would have us believe that the "Penny for NASA" efforts have grass-roots origins. But "Follow the money" is good advice when evaluating any grass-roots movement. If anything, I've over simplified the debate in my article here. e.g. see A Penny for NASA -- or For Congressional Pork

Saturday, April 19, 2014

Bill Gates TED Talks - Ideas Worth Sharing

Bill Gates is the well known founder of Microsoft. I'm not a fan of Bill nor of Microsoft. I am an advocate for "open source" software, and I think Microsoft has brought much harm to the computer industry. But that isn't what this brief article is about.

Bill has given a couple of really good TED talks, short ones, about 10 minutes each. I pointed out this first one before. It advocates more technology in the classroom, specifically video recorders so teachers can assess their teaching sessions.

Teachers Need Real Feedback

A balanced assessment of the idea would have to take George Orwell's concerns from "1984" into account, but as presented, I think the video idea has merit.

A second Bill Gates TED talk, looking at the education problem from an entirely different perspective, is this one:

How State Budgets are Breaking US Schools

Alas, he does better at explaining the problem than he does at suggesting solutions.

What do you think of his suggestions? Is he merely advocating that a larger slice of the education budget should go into technology because that is in his self-interest, or is he showing real forward thinking?

Saturday, March 22, 2014

Things your Guidance Counselor Maybe Didn't Drill into You.

I got the exciting news today that the son of my wife's daughter (which modulo the complications of 2nd marriages would make him my Grandson) has been accepted to the University of West Florida. He lives in Florida and I'm up north on Long Island (New York), so I confess to not having ever met the young man nor even knowing about the University of West Florida. But, as usual, Google is my friend when it comes to learning about places not familiar to me. Having now read a bit about the school, I whole-heartedly endorse his choice of college. Wikipedia had nice things to say about the place and it even fared well in the US News tabulation of colleges. But not actually knowing the plans of the gonna-be Freshman, I've got a few things to pass along to him as guidance.

First Things First

College is an important time when many things are going to be going on in your life. It's an investment in your future, but it comes at a cost. There's the obvious expenses like tuition and fees and books and supplies, but there's also an "opportunity" cost. You've elected to defer getting seriously started on your adult life for the number of years that you'll be tied up pursuing a degree. You've picked a 4 year school, so nominally, the long march to a degree should take you about 4 years. If you manage your time properly, and meet the requirements listed in the catalog, then in about 4 years time, you should indeed be graduating and moving on to the next stage of your adult life. If you do it wrong, 4 years will slip into 5 years and perhaps even longer. The price tag on your college education will continue to mount up, but the prize at the end is still that same degree. My point is, your first priority should be to stay on track to completing your degree. Some of the degree requirements may be quite frustrating, but on the bright side, the rules are at least clearly spelled out in the catalog. I assure you that later in life you will have to work your way through many situations where the prerequisites and the completion requirements are not spelled out with anything close to the clarity of your college degree.

Note that I said completing your degree is your top priority. By no means did I say it was your only priority. Not so clearly spelled out in the catalog is that your college years are when you seek out what it is that really interests you. You have to discover the degree program and subject "major" that you have a passion for - Something that you'd hope to be doing for the rest of your life, whether or not it was attached to a paying job. Make no mistake about it, to keep body and soul together you will need to find and work a paying job after graduation. Food, clothing, shelter, medical care, cell phone service, high speed internet - none of those are "free of charge", so you will need a job to earn a living. But if you force yourself into a particular career (perhaps "plastics") because you heard that it pays well, yet find that it isn't something that you enjoy, then you've made a grave error in planning your path through college. College is like 4 years, but your career is then the rest of your life, hopefully decades.

But wait! There's more!

Besides picking the right course of study and completing the degree requirements, there's an important social side to college. All work and no play doesn't make for a good, complete, college experience. You'll meet a lot of interesting people during your years in college. Some will be greatly challenging to your intellect. Some will be great fun for parties. Some may be strongly negative influences on your life. With luck, you'll even meet someone who you'll deeply love, someone who makes you a better, happier, person. You need to sort out who is who while not getting too distracted from your top priority.

The good news is that you aren't left entirely on your own to navigate this complex mine field. The same social structures that helped you get through life this far, your family, your church, your childhood friends, and so forth, are still places to turn to when you need guidance. The Internet and so forth makes it relatively easy to keep in touch with your old social supports, even if you are physically separated from them by many miles. Time spent socializing on the Internet is yet another "distraction" from your top priority, but keeping in touch with your roots may have real value to you. So make a conscious effort to spend your time wisely.

School Supplies

Next, a few words about school supplies. If you use a smart phone instead of paper products for these, that's fine, but make sure you have things configured so your information is safely backed up outside of the phone. Phones die or get stolen and you don't want to lose your information. What information? That brings us to the first couple of items on this list:

Calendar - You'll be needing a calendar to remember places you have to be at a specific date and time. In high school life is simple enough that perhaps you are used to tracking such commitments in your head with the backup coverage of reminders from your Mom. In college, your calendar can get a lot more crowded than that.

Address Book - You'll be meeting a lot of people at school. Professors, teaching assistants, lab partners, that hot girl in your Psych class. Take a few moments to jot down their name and contact info in a location where you'll be able to find it later. Loose scraps of paper jammed into your pocket or wallet isn't going to be useful to you in a week when it's faded and crumbled and perhaps even run through the laundry.

Alarm Clock - A smart phone can sound reminders through-out the day, but if you are as sound a sleeper as I am, you'll need a loud, persistent alarm clock to get you out of bed in the morning. I suggest a plug-in model, perhaps even a clock-radio, with a backup battery so it doesn't lose the time or settings across brief power outages. The easier it is to set the alarm time, the better. At the very least, look for an "up" and "down" option on the setting buttons. My favorite clock radio had an actual keypad for direct entry of the time, but I'm not sure that that model is on the market any longer. Do remember to buy appropriate batteries for that backup battery feature. And take some care in picking the location for the clock. The night stand next to your bed has the advantage of being right by your ear, but the drawback of putting the snooze button within arm's reach. I find what works for me is to keep the clock on the dresser where I have to get out of bed and walk across the room to turn it off or hit snooze.

Bank Account - You've got to shop around. College is a time where you'll spend your days somewhere between destitute and flat-broke. Every once in a while, you'll receive some money - perhaps a check for your birthday or for Christmas. Nothing will make that money vanish faster than cashing the check and putting the cash in your wallet. You need to set up a bank account for yourself. You probably should find out what banks (or credit unions) have ATM's on or near campus. Banks near colleges often offer accounts structured to attract students: Low fees, low minimum balance, but probably no interest on your deposit. A debit card is almost as easy to use to drain your account as was cash in your wallet, but at least the debit card gives you a statement so you can see where the money went. If they offer you a credit card, you probably should say "no". It is a lot easier to get into credit card debt than it is to get out of it. A debit card inherently caps your spending at how much you have. Living within your means can be tough, but I strongly recommend it.

Fire-resistant lock box - You need to start being responsible for your own life-documents. Birth certificate, social security card, perhaps even a passport. You don't really need a safety-deposit box for these items, or even a secure safe, but you shouldn't be just tossing them into your dresser drawer under the socks either. Visit Walmart or some such store to find a fire-resistant lock box that you can stash in the back of your closet to hold these hard to replace documents. Note that fire-resistant boxes tend to have water entrapped in the walls of the box, which converts to steam in the event of a fire. The physics of boiling keeps the papers inside the box from reaching the ignition point for paper (until the water boils away). The temperature and humidity in the box is meant to be safe for paper, but not for computer media, so such a box isn't a good place to store backups of your data.

File Folders - For all the years of people talking about "paperless offices" and "paperless classrooms", I think you'll be surprised at how much paper comes into your life every day as a college student. The papers are worthless to you unless you can retrieve them when you need them. As a start on getting them organized, I therefore suggest an investment in file folders. If you don't have a place to stash the folders, then an additional expense to anticipate for you is a small file cabinet. (I recently picked up a flimsy 2-drawer metal file cabinet at Walmart for just over $30). How to organize the papers is a whole other essay that I'm not planning to write here. Cheap 3-ring binders and a 3-hole paper punch can be an alternative to the file cabinet to the extent your papers are 8.5"x11". Keep an eye on the back-to-school sales that come up every year as September approaches. Loose-leaf paper and manila notebook divider sheets can be mighty handy. If you use pencil to label the folders and divider sheets, you'll be able to erase and re-use them as the topics to organize shift as time marches on.

Book Shelf - College brings you more (and heavier) text books than you want to haul around in your back-pack every day. And college doesn't come with a hall-locker for you to stash the books you don't need right now. Make sure that no matter where you plan to live, that your room has a bookshelf where you can store your reference books, text books, note books, etc., etc. in a neat and accessible manner.

Desk - In high school the kitchen table or maybe even your lap sufficed as your work place for your writing assignments. In college, you really ought to have a real desk for yourself. Several stacks of cinder blocks and a slab door can make a serviceable desk for little money. Do remember your budget needs to cover a reasonable desk chair too. A suitable computer and some provision for Internet access are key parts of a modern student's desk these days, so that needs to be included in your planning.

Comfie chair and reading light - A startling amount of your college life will be spent on reading. If your habit has been to sit down in the living room with the television on while you read, now is the time to change that habit. It'd be unsporting to tell your family that they aren't to watch television until you graduate. You need a comfortable chair in a quiet place, away from the television, and with a good reading lamp. There are lots of ways to stretch your budget when looking for such items. Some of my college buddies solved the problem by pretty much living in the college library for all their waking hours except for class and meal times. Explore your campus library and find what provisions they have for quiet study space for undergraduates. That wasn't the way that I did it, but I know it can work.

Dorm or commuter student

I see UWF has large dormitories to accommodate students living on campus. They allow Freshmen to park on campus, which implies to me that not everyone is living on campus, some are presumably commuting to school. Home for you is in the Florida Pan-Handle, so I'm guessing that commuting is an option for you. Living at home and commuting to school can be a big cash saving for you. You can forage for food in the kitchen, and your Mom's washer and dryer probably doesn't insist you insert a bunch of quarters for each load. Nevertheless, I urge you to consider living on campus. I believe that moving out of the house is a big step in growing up, and college, if nothing else, is about growing up. Dorm room, meal plan, laundry expenses, ... No doubt living on campus isn't "free", but if you consider the groceries your Mom won't need to buy at home, the gas and car expense that you won't have if you aren't commuting to school each day, maybe you'll be able to manage to live modestly on campus. I think you'll find yourself much more inclined to take advantage of the campus library and of events at school if you are right there.

I can pretty much guarantee that your first trips home after you've been away at school will not be entirely comfortable for you. Quite likely you'll discover that your family and old friends somehow haven't been aware of the way you've grown up while your were away from home. Don't let that disconnect ruin your visit home. If nothing else, it'll help you appreciate the wisdom of that old adage "You can't go home again".

Your Health

Some fraction of kids go away to college and come back dead. The newspapers love to jump on such stories because the news is so universally distressing to parents. Have fun in college, but take care of yourself. Impaired judgement from chemically enhanced fun can lead to tragedy.

There are plenty of other ways to ruin your health, though some of them take time. Lack of sleep, for example. Your alarm clock may faithfully remind you when it is time to get up, but it is up to your own self-discipline to remind yourself when it is time to get some sleep. There will certainly be nights when you are cramming for exams or whatever and absolutely have to cheat yourself out of some sleep time, but I assure you that if you make a habit of that, you'll soon find yourself working much less effectively at your college priorities.

Nutrition is another oft-neglected facet of good health among college students. Pizza and coca-cola are a tasty combination, but not a good choice for breakfast. You really need to make an effort to maintain a reasonably balanced diet. Some oranges for vitamin C, milk for healthy bones, all those things they taught you in health class. Somewhere on campus there's a reasonably accurate scale where you can pop by once in a while to keep tabs on your weight. Perhaps somewhere near the gym? Balancing your calorie intake to your level of physical activity is another important part of a healthy diet. (There are free smart phone apps that make it easy to keep track of your weight over time, producing easy to read graphs of your progress or backsliding). Speaking of physical activity, I'll wager that the outdoor weather in Florida is a lot more hospitable to getting out and about than the gloomy slush that was much of the school year up in Ithaca, NY. Ithication we called it. Like precipitation, but greyer and less comfortable.

You're old enough that I'll not subject you to yet another lecture about the dangers of sex. An incurable STD can certainly take a lot of the fun out of college. An unplanned pregnancy can throw quite the monkey wrench into your graduation plans. So have fun in college, but remember there's no "reset" button like in video games, no rehearsals, just one pass through life so make the best of it.

The Requirements for Graduating are in the Catalog

I mentioned up above how you have to consciously pick your path to graduation. You need to find a "major" that really excites you. One of the things I like about UWF is they have a wide variety of options open to you. Teaching, nursing, biology, social work, engineering. My "thing" is computer software, but that isn't exciting to everyone. I'm not going to be that guy whispering "plastics!" in your ear. You'll have to find your own way.

Do your best to plan ahead. There can be long strings of pre-requisites to work through for some majors, so if you don't make yourself aware of those soon enough that 4 year diploma can start slip-sliding off into the future by years. Try to fit some courses into your schedule that'll give you a taste of something completely different from where you think you are going. Be brave enough to change direction if you find where you thought you wanted to go wasn't as interesting as you thought it was going to be. But if you find yourself having to repeatedly change your mind and your direction, do find yourself a trusted advisor who can help you work through why that keeps happening to you.

There's more to college than passing classes and graduating

There are many more things happening at your college than just your classes. Take advantage of your being on campus by paying attention to what else is going on and attend some of those events. Back in my college years, when dinosaurs roamed the earth, the walls and hallways of campus tended to get plastered with posters announcing "events". It is possible that the Web has completely replaced those flyers, but I doubt it. Don't blindly rush past announcements of symposia and events on campus.

Happy Trails!

You certainly saw quite a fuss today over your getting into college. Just imagine the excitement in your family when you graduate. I'm wishing you the best for your college years. If I've given you any surprises here or left you with any questions, do feel free to contact me and we can talk.

Do well!


Most all of your college papers will be in for severe criticism if they don't include suitable references at the end. So, I close with a few references here for this article.

There's a pretty good book called "How Children Succeed" by Paul Tough. If you don't want to take the time to read the whole book, a possible short cut is to read my book-report on my blog from back when I read the book.

I hate to admit it, but there's an awful lot of wisdom (as well as laughs) about the real world of corporate employment to be found in the Dilbert comic strip by Scott Adams. For example (Dilbert, 02/18/2014:

Click on the dated link above the comic to see the missing punchline in the 3rd frame. Then you can use your browser's "back" button to return here.

Back in January, 2013, I posted "Advice to Undergraduates". It's basically a link to 3 other articles that I enjoyed. It may be a bit too computer-science-centric for general audiences, I still enjoy reading the linked articles there.

"Education" is a recurring topic on my blog and I won't puff up this references section with links to all the related articles. But I would like to point out the TED Talk linked to my May 2013 article "Education's Future: 'Go Do Somethng Interesting'"

If my repeated mention of "plastics" makes no sense to you, use your Netflix subscription to hurry up and rent a copy of 1967's movie "The Graduate".

These references would be far from complete if I didn't also include a link to Dave Barry's 2004 "Commencement Address".
Revised 3/31/2014 - Edited to fix a couple of minor typos.
Revised 04/20/2015 - Revised because Dilbert URL's changed around so link to "sleep" comic broke. I fixed it.

Monday, February 3, 2014

Python Generators


Back in May, 2013, I posted "Cheap, but not Free, Modular Code has a Price". In that blog article I compared the performance of my solution of Project Euler problem 14 to the performance of Nico Ekkart's solution of that same problem. As I explained in that article last May, Project Euler's guidance on performance is that a solution should be able to complete on a typical PC in less than a minute. My initial solution ran well over a minute and I applied a cache to speed my program up. Once I'd gotten it down below the one minute limit, I had concluded "good enough" and went on to other problems. But Nico honed his solution a little tighter and outperformed my program by a factor of 4.

Looks to me that we had basically the same solution, but his was more of an in-line implementation, while I used a Python generator to produce the sequences of numbers the processing of which is the heart of problem 14. I concluded that the Python structuring facilities like generators do add overhead but I opined that the added clarity of packaging the sequence generation as a "generator" made the overhead worthwhile. Nico's code for his cache was so little code that I didn't really defend my use of a Python "class" for my cache implementation. Some further experiments with the code to determine how much of my extra overhead came from the generator vs. how much came from my cache implementation might be interesting. I've posted the source code and invite you to experiment and share your findings.

But that isn't what I came here to talk about today

This blog post today is aimed at expanding your knowledge of Python Generators and share with you a neat trick that I've recently learned: a generator of dictionaries.

Python Dictionaries

Back in January, 2013, I posted "Dictionary Implementation". In that article I pointed to where the Udacity CS101 MOOC introduces Python dictionaries and pointed to further places to look to learn how Python implements dictionaries, hash tables. If you aren't comfortable with Python's notion of a dictionary, I'll sum it up as an easy to use collection of name-value pairs, much like an array is an easy to use collection of subscript-value pairs.

Given an integer value as a subscript into an array, you can easily ask for the value associated with that subscript. (In Python such an array is called a "list". But if you have a dictionary of name-value pairs, in Python it is easy to ask for the value associated with a name. e.g. What value is associated with "eggplant", where "eggplant" in this example is a possible "name" entry in the dictionary. If this is all still mysterious to you, my suggestion is to dig for more info on the web. Google search is your friend. A Google search for:

python dictionary

turned up more than 5,000,000 web pages today and Google looks to have done a decent job of sorting the most helpful ones to the front of the search results.

Or perhaps you should work your way through Udacity CS101 to learn this concept of dictionaries and more.

Python Generators

But as I already said, the topic of my blog post today is Python Generators. Rather than take the time and trouble to create a fresh new tutorial about Python Generators, I'll pause here to provide a link to an article from April 2013 from Jeff Knupp: "Improve your Python: 'Yield' and Generators Explained". That article resonated with me as written in an easy to understand way. It's about 19 screens long, so it isn't a quick read, but you can get back to Facebook and YouTube later.

Knupp's article should certainly get you to the point of being comfortable with my Project Euler problem 14 generator. My generator is very vanilla, just conjuring up a sequence of numbers. The one tricky aspect is my program often doesn't pursue the sequence to the end, but sometimes decides it needs to see a new sequence. I was a little nervous when I did it that way, but now understand that Python doesn't mind dangling generators like that, and even should know enough to be able to garbage collect their saved state, if memory gets tight and it is clear that there's no way the program could get back into that old sequence. Powerful stuff and a feature unlike anything available available in C, Fortran, Basic or Cobol to name 4 examples of older more traditional programming languages.

But Wait! There's more!

If you've read this far but are yawning, thinking that hardly ever do you, yourself, have occasion to use complex logic to generate a sequence of numbers in the programs that you write, well hang on and see what else generators are good for.

Igor Vasilcovsky recently pointed out to me an interesting tutorial from David Beazley about generators. The meat is the slide deck under the presentation link on that page. Even with 2 slides to a page, that slide set runs to 79 pages so it probably is too much to absorb in one sitting. So try to work through it in several sessions. Here's my summary of the key elements I think you should get from wading through that presentation:

  1. A generator can return any kind of Python object. That is, it isn't limited to returning the numbers of a sequence. If you paid close attention to my Pythonic Python from July, 2013, you may remember that one of the points I made there was that Python functions can return large complex objects, not limited to just a number or just a pointer. Better still, Python takes care of the book keeping to automatically garbage collect the returned object if you are no longer using it.

    Beazley has examples with a generator returning strings (lines from log files), and then takes an unexpected turn to have the generator parse the strings into values and stash the values into a dictionary with a name for each value and then return the resulting dictionary as the generator's output. (A dictionary per line of the log file!) My initial reaction to that was to be horrified at the overhead, but if you think about it, looking up a name in a dictionary is quite central to Python's run time modus operandi, so dictionary objects being generated is perhaps more general than generating some more specific Python class object, and looking up values by name is pretty much the same as using a name of a member of a class object to get to a value in the class object. So given that Python is able to get decent performance from run-time interpretive execution, generating dictionaries is a lot more sensible then my initial gut reaction took it to be.

  2. A generator is just Python software so it can do anything that any Python software might want to do to provide the next value in the sequence. It can compute the next value using a formula, or it could access a file to learn the next value. It is even able to make use of another generator to get an input sequence that it manipulates and passes along. So generators can be chained together into long pipelines.
  3. If your program is basically about iterative processing of a series of stuff, then chaining together pipelines of generators can fundamentally shape a clean structure for your program and with a little practice you'll soon have a collection of useful reusable filters that you can use to construct future programs that you haven't even thought of yet. Share them with your friends. If you own the code, consider making it (and it's documentation) GPL licensed and freely shared on Github or some other such public place.
I'm seeing hints on the Internet that future Python releases will have even more use of generators. e.g. standard approaches built in to the language to process streams of events. e.g. Brett Cannon's 40 minute talk: "Python 3.3: Trust me, it is better than 2.7". He foreshadows a Guido Van Rossum talk about Python 3.4 scheduled for that same conference. If you can find a link to a video of Guido's talk about 3.4 and new roles for generators, please share it in the comments section here.