Mon - August 30, 2010
A little Python program to create to-dos in Things from a template file
I quite like the to-do list manager Things from the company
Cultured Code. (Like plenty of other people, I expect, I'm waiting for repeating
to-dos on the iPhone version, but even without that feature it's far the best
one I've seen.) I also have a list of things to do in preparation for going on a
trip that I've refined some over the years. (Make sure I have enough books, get
any foreign currency I'll need, get a hostess present, etc.) Since some of the
items on the list may require that I wait for something to be shipped or for
other people to do things, I have various indications in the list of how long
before I leave that I ought to begin each
item.
I've sometimes thought that it would be convenient if Things could make use of that list without my having to go to the trouble of entering each item. That's the sort of thing that computers are meant to do, after all. I thought of emailing the folks at Cultured Code and suggesting a feature of that sort. And then I noticed that the Macintosh version of Things can be controlled by AppleScript. And there's a Python module that can send the necessary events (so I didn't have to figure out much about AppleScript). So I hacked up a little Python program to read my file and create the appropriate to-dos in Things.
It's unimaginatively called todosfromfile.py and it's licensed under the GPL. You can get it here.
The file it reads has two sorts of lines: "time-before" lines and to-do lines. The time-before lines specify how long in advance the subsequent to-dos should be due. So the file might look like this:
1 month
Make sure I have the clothes I'll need
2 weeks
Get books
Get hostess present
Foreign currency
1 week
Program phone with addresses
Make sure I have enough cash
2 days
Get weather forecast
1 day
Check in
On day
Pack laptop, chargers
Blank lines and lines beginning with a "#" are ignored.
The program creates a project in Things and adds the to-dos to it. It's run from the command-line and takes three arguments: the name of the template file (--file), the name to give the project (--name), and the date to count backward from (--date). The date needs to be in YYYY-MM-DD format. So I might run it as:
$ python todosfromfile.py --file trip.templ --name 'Trip to Maine' --date 2010-10-31
It requires the Appscript library and Python 2.7. It only needs Python 2.7 because I used the argparse module and that could easily be changed.
I've sometimes thought that it would be convenient if Things could make use of that list without my having to go to the trouble of entering each item. That's the sort of thing that computers are meant to do, after all. I thought of emailing the folks at Cultured Code and suggesting a feature of that sort. And then I noticed that the Macintosh version of Things can be controlled by AppleScript. And there's a Python module that can send the necessary events (so I didn't have to figure out much about AppleScript). So I hacked up a little Python program to read my file and create the appropriate to-dos in Things.
It's unimaginatively called todosfromfile.py and it's licensed under the GPL. You can get it here.
The file it reads has two sorts of lines: "time-before" lines and to-do lines. The time-before lines specify how long in advance the subsequent to-dos should be due. So the file might look like this:
1 month
Make sure I have the clothes I'll need
2 weeks
Get books
Get hostess present
Foreign currency
1 week
Program phone with addresses
Make sure I have enough cash
2 days
Get weather forecast
1 day
Check in
On day
Pack laptop, chargers
Blank lines and lines beginning with a "#" are ignored.
The program creates a project in Things and adds the to-dos to it. It's run from the command-line and takes three arguments: the name of the template file (--file), the name to give the project (--name), and the date to count backward from (--date). The date needs to be in YYYY-MM-DD format. So I might run it as:
$ python todosfromfile.py --file trip.templ --name 'Trip to Maine' --date 2010-10-31
It requires the Appscript library and Python 2.7. It only needs Python 2.7 because I used the argparse module and that could easily be changed.
Posted at 07:04 Main Permalink
Wed - April 21, 2010
Errno 35 in Python's socket.sendall() under OS X
A little while ago, on the python-help mailing
list, a question came up that took a bit of work to find the answer to. Since
that list's archives aren't public and Google doesn't seem to have indexed a
page with a good discussion of the issue, I thought I'd post about it here (with
the original poster's permission, of
course).
The poster was using Python 2.6.2 under Mac OS X 10.6.2.
The poster was using Python's ftplib module to upload certain files and some of them would fail consistently with the same number of bytes transferred and with a traceback that ended with:
File "/Library/Frameworks/Python.framework/Versions/
2.6/lib/python2.6/ftplib.py",
line 452, in storbinary
conn.sendall(buf)
File "<string>", line 1, in sendall
error: [Errno 35] Resource temporarily unavailable
It's pretty clear from that that the OS was temporarily running out of network buffers. But why doesn't the socket's send() method just block until it completes?
The reason is that a socket timeout had been set. If you set a socket timeout in Python (whether through the socket module or something that uses the socket module), sockets are set to be non-blocking "under the covers". (That's pretty much the only way to implement that feature.) It's a somewhat awkward side-effect of doing that that errors resulting from timeouts don't always look like what they are.
[Edited April 27, 2010; the earlier version of this post was based on an incomplete understanding of the problem.]
The poster was using Python 2.6.2 under Mac OS X 10.6.2.
The poster was using Python's ftplib module to upload certain files and some of them would fail consistently with the same number of bytes transferred and with a traceback that ended with:
File "/Library/Frameworks/Python.framework/Versions/
2.6/lib/python2.6/ftplib.py",
line 452, in storbinary
conn.sendall(buf)
File "<string>", line 1, in sendall
error: [Errno 35] Resource temporarily unavailable
It's pretty clear from that that the OS was temporarily running out of network buffers. But why doesn't the socket's send() method just block until it completes?
The reason is that a socket timeout had been set. If you set a socket timeout in Python (whether through the socket module or something that uses the socket module), sockets are set to be non-blocking "under the covers". (That's pretty much the only way to implement that feature.) It's a somewhat awkward side-effect of doing that that errors resulting from timeouts don't always look like what they are.
[Edited April 27, 2010; the earlier version of this post was based on an incomplete understanding of the problem.]
Posted at 07:37 Main Permalink
Thu - December 21, 2006
Magellan RoadMate 2200T GPS receiver review
The Magellan 2200T portable GPS unit is mostly
unremarkable. And that's brilliant. But before I tell you why I think that, let
me tell you a little about how I came to buy one because that may be relevant to
knowing why I think what I do.
I'm a confirmed gadget dork. I own more than one iPod. When I saw a buddy's DS Lite, I knew that I had to have one. I have two sets of lightweight headphones for different purposes. I live in Minneapolis, but my cellphone was never sold by an American carrier. I once bought a Japanese-market laptop.
And I considered getting a portable GPS unit more than once. But looking at the products on the websites of two main manufacturers, Garmin and Magellan, is an exercise in annoyance. It's true that, from a marketing perspective, It makes sense to "segment" a market by selling different products at different price points to people who are willing to pay different amounts of money. But these guys are nuts. As of this writing, counting just portable GPS units that are intended for use in cars, Magellan sells 11 of them and and Garmin sells 21. Who wants to read through the marketing drivel for that many products in order to try to figure out which one you really want?
The companies further annoy potential customers by leaving things out of the box that you're going to want and so have to pay extra for. Such as maps in some cases. Or enough storage to hold the maps for the whole U.S. I can't think of a product category in which the manufacturers do a better job of alienating potential customers. They certainly put me off a few times.
But then I used a car GPS unit on unfamiliar roads in difficult driving conditions. I was traveling alone to visit a buddy who lives in a small town in Maine. Northwest Airlines cancelled the leg of my flight from Detroit to Portland, Maine a few days before my trip and so I had to fly into Manchester, New Hampshire. The drive from Manchester into Maine looked like it would take a little more than two hours and I had good maps and good directions. That shouldn't be a big deal. I've done any number of drives like that in the past without any trouble. But the route was reasonably complicated, my plane would be landing late in the afternoon, and the weather forecast for the area wasn't all that good. So I called up Hertz the day before I left and told them I wanted a car with a GPS. They said that that wouldn't be a problem.
Getting in my car in the Hertz lot in Manchester, I found that making their "NeverLost" GPS unit work was snap. Which it would really have to be since otherwise they would have a bunch of annoyed customers going back into airports demanding that someone show them how the thing works. Hertz and Magellan must have done a lot of usability testing to make the unit as easy to use as it is.
Following the unit's guidance, I made a wrong turn just out of the airport. That was because I had mistaken the scale of the map it was displaying. But even that turned out to be just fine because it calmly displayed "Recomputing route" and told me where to go from where I had gone to. From there to the end of the trip (and, indeed, the return to Manchester) I had not the slightest difficulty reading its map or following its spoken directions. The Hertz unit was mounted near the passenger's left knee. It might have annoyed a passenger slightly but it worked fine for me. It was a little like playing Mario Kart DS, with the moving map below the view forward.
And it was great that the NeverLost worked so well and was so easy to use because it rained the whole way and most of the trip was in the dark. The rain varied between heavy and ridiculously heavy for almost all of the trip. I had the car's windshield wipers at their highest setting pretty well all the time and I occasionally wished for a higher setting. People were driving 45 MPH on highways posted for 65 and some of the roads in my buddy's town were flooded.
In circumstances like those, turning on the car's dome light to read a map or directions would have been pretty unsafe. And the limited amount of information I would have managed to gather doing that under those circumstances combined with the very limited visibility outside would certainly have resulted in my making several wrong turns and often having a suspicion that I had gone wrong even when I hadn't.
But the NeverLost's bright moving-map and voice instructions took me directly to where I was going. The noise of the rain on the windshield was often loud enough that I was concerned that I wouldn't be able to hear the voice prompts, but at the loudest setting it shouted comfortably over the racket. I was thoroughly glad to be done driving when it said in its synthesized voice, "You have arrived", but I would have been there considerably later and considerably more tired without it. As it was, the trip was very annoying. Without the NeverLost, it would have been a nightmare.
When I returned the car at the Hertz facility in Manchester, I told the person who was taking down the car's mileage and printing my receipt that I didn't know just what they had charged me for the GPS unit but, whatever it was, that it had been worth ten times the price. And so I returned home willing to go to the trouble of finding which portable GPS would be best.
A bunch of research suggested that the 2200T would probably be best and I suspect that it is. Other than the NeverLost which is similar, the only other car GPS I've used is one that was built into a buddy's car and so I can't give any very useful comparisons, but I can tell you why I like the 2200T.
I like it chiefly because, like the NeverLost, it does its job in an unremarkable way. The thing is most valuable in difficult conditions and the last thing you need then is a confusing display or an elaborate user interface. On the 2200T, both are admirably simple and clear. A geek should need ten or fifteen minutes and a glance or two at the manual in order to get familiar with the unit. A more normal person might take a little longer, but shouldn't take very long.
(I may be able to save you even brief trips to the manual, which is currently only available as a PDF to download, by telling you that "Enhanced POIs" are locations you enter using Magellan's PC software and the "Trip Planner" is for multi-stop trips. Magellan's software is Windows-only but you don't need it to use the normal functions of the unit. Also, many areas of the moving map screen that aren't obviously buttons respond to tapping on them.)
To use the 2200T, you begin by picking a destination. Actually, you don't have to do that. It will be happy to show you a moving map of your vicinity at any of various scales. But the screen is sufficiently small that it can't show an area more than a few miles across in any significant detail, so it's not all that useful for orienting yourself in an unfamiliar area. The same would be true of any map that's slightly smaller than a file card.
So you begin by selecting a destination. That can be a street address, an intersection, a "point of interest", an address that you've previously entered in your addressbook, or a place picked of the map. You pick a place on the map by zooming out, dragging the map around with your finger, zooming back in again, ensuring that where you want to go is at the center of the map, and tapping there. It's not very convenient, but it works and you're unlikely to need to use it much. "Points of interest" is GPS-speak for a sort of telephone book of locations. If you want to go to the nearest gas station or a particular restaurant or shopping mall, you can query the unit's database and pick where you want to go. GPS manufacturers differentiate their units in part by how many zillions of POIs are in their databases. The 2200T has a database of 1.5m POIs, which is small by current standards. Still, it seems to have everything in it that I can think of and it's been useful in a couple of real-world situations. Naturally, a database like that will eventually become out of date, but I'm sure that Magellan will offer updates.
Once you've picked a destination, the unit will display "Calculating route" for a few seconds or a little longer and then will display your route as a magenta line on the moving map. Just start driving along the line. The unit will display the map with your route ahead of you (assuming that you've configured it to show ahead as up). You'll be able to see your route easily because the screen is a wonder. It is perfectly readable in everything from complete darkness to bright sunlight. The map changes scale automatically using what seems to be a pretty intelligent algorithm to determine how much detail you'll want at what time. You can also change the scale yourself.
The unit displays the name of the road you're on, the name of the road you'll turn onto next, the distance to the next turn, the distance remaining for the trip, and the sort of turn you'll make next. (You can vary that a little, but that's the essence of it.) If you've configured it to (and you should, it's useful) the display will change to a split-screen shortly before a turn. Half of the screen displays the moving map and the other half displays a 3-D rendering of the turn to be made. That may not sound especially useful. How interesting is a picture of a left turn? The answer is that it can help to know what to look for if the turn is something other than a plain 90-degree turn. A picture of a right turn at a shallow angle followed by a sharp left turn over a bridge across the road is quite helpful.
But much better than just a moving map, the unit gives you voice prompts. Its synthesized voice informs you about upcoming turns well in advance, repeats the information closer to the turn, and then plays a chime just before you'll be turning the steering wheel. The software also pronounces the name of the road you'll be turning onto. GPS geeks call that "text to speech". Inevitably, there are some slightly odd pronunciations, but I haven't yet run into one that was incomprehensible, and that includes Wayzata Blvd. The virtue of the voice prompts is of course that you don't have to look at the thing much. It may be desirable to glance at it occasionally, but you can take your eyes off the road a lot less often than you would if you were using an ordinary map or directions.
The 2200T has a couple of other small advantages: The battery life seems pretty good. Mine ran about eight hours from a full charge before switching itself off. I strongly suspect that battery life is affected by how much work the unit is doing and since I wasn't asking it to do much that was very hard during those eight hours, I'd suggest taking that as an upper limit. Still, I'm pretty impressed.
The unit feels solidly built and it appears to be sealed against weather pretty well. I wouldn't advise you to take it swimming with you, but I doubt that a little rain would harm it.
The downside to its being pretty well sealed is that the battery isn't user-replaceable. Making a watertight lid for a battery compartment would be hard. That means that when the battery no longer holds a useful charge, you'll need to send it in for service. But you'll probably run it mostly from its cigarette-lighter power cord and so a battery replacement may be pretty far in the future.
I don't have a lot to compare it to, but the unit's reception of GPS signals seems quite good. It works just fine in my apartment as long as I stay pretty close to the windows. In a car, the unit doesn't have to be stuck to the windshield to get a good signal. Oriented randomly in a bag in the back seat seems to work fine.
There are a few other compromises and imperfections.
There are a couple of small problems with the user interface. One is that there's no simple way to change the volume when you're looking at the map screen. There's a mute button that you can tap on that screen to silence the unit, and that's probably a useful thing. But the button should really pop up a volume control. If it starts raining hard enough to make a racket, you're not going to want to go digging through various screens looking for the volume control.
It would also be useful if there were an audio indication of when the split-screen view of the upcoming turn is available. Since the screen returns to the usual map display just before the turn, you pretty much have to guess when to look at it if the unit isn't mounted in your line of sight. We're not supposed to mount things to our windshields here in Minnesota.
There's another small imperfection that's a bit more subtle: When you've specified a route, you can simulate driving it. In a preferences page, you can tell the 2200T that you want it to offer you the option of simulating driving a route after it has calculated it. You can also specify some options for the simulation, such as doing the simulated driving faster than you'd actually drive the route. I can imagine that that's a potentially useful feature. I might like to preview an unusual or complicated route before driving it so as to see what I'd shortly be doing or to have a look at the route to decide if I liked the one that the unit had chosen. (Within limits, you can influence the route that the unit picks.)
But there is a minor problem with the feature. If you have the option to preview a route turned on, it gives you that option even when the unit computes a new route because you've departed from the original one. If you've departed from the route that the 2200T has chosen, it's probably because of bad traffic or to detour around a road that's closed. No one is going to want to preview the new route then. They're driving. It's a relatively minor thing to tap a No button to say that you don't want the preview, but it's a distraction. And the 2200T is good because it almost always doesn't distract you.
If your route has you continuing on the same road for more than a few minutes, the 2200T occasionally speaks, telling you that you'll be continuing on the road. That may not sound like a very useful feature. But the NeverLost didn't have it and, on a stretch of highway that I was on for about an hour, I wished that it would talk to me once in a while. Was the volume set loud enough that I could still hear it? Was it still working correctly? An occasional spoken status message would have spared me a few glances at the NeverLost and a few unnecessary clicks of the volume-up button. The 2200T says "Continue on the current road" every few miles and I think that's a small but valuable feature. It would be even better in my opinion if it incorporated some useful information into the message, such as the number of miles remaining on the current road.
The underlying database of road information that the 2200T uses is licensed from NAVTEQ. It seems that they don't have a lot of competition for the US and Canada. Garmin's GPSs also use their data and so do the map websites of Google, Yahoo, and MapQuest. The database is good but it's not perfect. For example, it doesn't know that in Minneapolis, where I live, 1st Avenue South is two-way between Franklin and 28th St.
The road database knows about the reversible lanes on I-394 near downtown Minneapolis, but it doesn't know all the ways they're connected to the rest of the highway. It correctly identified that it was traveling on them, but at the point where they were about to rejoin the rest of the highway going west, it requested a turn that, while safe, wasn't in the right direction. Proceeding in the right direction on the highway, the unit requested a u-turn, cautioning that it should be a safe and legal one. Still proceeding in the right direction, the unit gave a "You can't get there from here" message and I needed to re-enter the destination. (Happily, I was a passenger at the time and could do that without any fuss.)
In addition, here in Minneapolis, streets and avenues are generally labeled as being North or South or Northeast or Southeast, depending on their position with respect to the Mississippi River and Hennepin Avenue downtown. But we're a bit sloppy about whether we say "First Avenue South" or "South First Avenue". Also, some streets exist only in one of the four divisions. According to the post office, those streets shouldn't have a direction marking since they don't need to be distinguished from different versions.
Entering a street address in the 2200T is admirably simple and hard to do wrong. You begin by spelling the street name on an on-screen keyboard. Letters that can't come next disappear, making it easier to find the ones you want. When you've entered enough letters that there are only a few possibilities for what you're spelling, the unit shows you a list and you pick the street from it. Minneapolitans' casual attitudes to where they put their souths and norths is mirrored in the database. For example, in the database "S. Bryant Ave" has house numbers from 1600 to 3599 and "Bryant Ave South" has house numbers from 1600 to 9399. Holmes Avenue, which exists only in the south section of the city, is listed as "Holmes Ave", "Holmes Ave S.", and "South Holmes Ave". The house numbers listed for the first one don't actually exist.
None of those imperfections particularly surprises me. The 2200T contains a database of all the roads in the United States and Canada. In my experience, a database of that size pretty much can't be perfect. But it's worth remembering that when you're using the unit. Apart from the issue with the reversible lanes, none of the errors would cause any significant trouble, and the unit hadn't recommended taking the reversible lanes.
Once you've specified an address, you can satisfy yourself that you picked the place you meant by tapping a bulls-eye icon. That will show you a map centered on your destination. A text button saying something like, oh, "Show on Map" would have been a little more obvious, but once you know what the little bulls-eye does it's easy enough.
The routes that I've seen 2200T choose have always been good but, in cases in which there are a large number of very similar routes, such as traveling diagonally through a grid of city streets, the routes it has picked haven't always absolutely optimal. That's hardly surprising. It would require information about every stop sign and traffic light in the US and Canada in order to pick the optimal route from dozens or hundreds that are very similar in distance and time. The unit seems to have a preference for broad and one-way streets. That's probably a good preference to have since it's likely to be guiding you on unfamiliar roads.
When you're asking it to calculate a route, you can tell it to pick a route with a minimum use of freeways. That's often handy at rush hour around here. It will also notice that you're in slow traffic on a freeway and indicate that it's willing to find you a different route. There's also an optional traffic receiver (not available yet and probably requiring a paid subscription) which ought to enable it to choose better routes when traffic is heavy.
I mentioned earlier that GPS manufacturers often leave things that you're going to want out of the package so as to make additional sales from accessories. The 2200T is just missing a case and an AC adaptor. That's actually pretty good as these things go. And you may not really need an AC adaptor since you'll probably use the unit in a car almost all the time and it comes with a cigarette-lighter adaptor. Still, I knew that I'd want to learn how to use it at home rather than sitting in my car and I may possibly want to use it on my motorcycle so I wanted a AC adaptor. At the time I ordered the unit, Magellan's website didn't show an AC adaptor as an accessory available for the 2200T. It did show them for other similar units and I supposed that they were the same. Still, I wasn't sure enough to place my order on the website, so I placed the order by telephone, specifying that I wanted an AC adaptor for the 2200T. A few days later the adaptor arrived, but without any prongs to plug it into the wall. Magellan sells their GPS units in the United States and also in Europe. Cleverly, they've designed a universal power supply that just needs to have prongs appropriate to the local outlets clipped to it. So I literally had an AC adaptor, just not a useful one. I called the same number and explained the situation and the nice person who took the call sent me the right prongs without charge. I think that's pretty good. Anyone can make a mistake, but Magellan fixed that one quickly and pretty painlessly.
Magellan still doesn't have a case available for the 2200T. I'm using this, which is a sort of fluffy padded napkin with the hook side of velcro (er, hook-and-loop fastening material) at the corners. You wrap it around something, forming s sort of envelope and then open one corner to slip the thing out and back in. It works as advertised, but the result is a bit bulkier than I'd like.
The unit has an MP3 player and a picture viewer built in. Since I have no idea why anyone would want to use those features, I have nothing else to say about them.
I'm very impressed with the 2200T. It's not the sort of thing I need every day because I travel on familiar routes most of the time. But if it guides me on another trip or two like the one from New Hampshire into Maine and saves me a bit of fuss on an occasional trip to the wilds of St. Paul, it will have been worth the price.
I'm a confirmed gadget dork. I own more than one iPod. When I saw a buddy's DS Lite, I knew that I had to have one. I have two sets of lightweight headphones for different purposes. I live in Minneapolis, but my cellphone was never sold by an American carrier. I once bought a Japanese-market laptop.
And I considered getting a portable GPS unit more than once. But looking at the products on the websites of two main manufacturers, Garmin and Magellan, is an exercise in annoyance. It's true that, from a marketing perspective, It makes sense to "segment" a market by selling different products at different price points to people who are willing to pay different amounts of money. But these guys are nuts. As of this writing, counting just portable GPS units that are intended for use in cars, Magellan sells 11 of them and and Garmin sells 21. Who wants to read through the marketing drivel for that many products in order to try to figure out which one you really want?
The companies further annoy potential customers by leaving things out of the box that you're going to want and so have to pay extra for. Such as maps in some cases. Or enough storage to hold the maps for the whole U.S. I can't think of a product category in which the manufacturers do a better job of alienating potential customers. They certainly put me off a few times.
But then I used a car GPS unit on unfamiliar roads in difficult driving conditions. I was traveling alone to visit a buddy who lives in a small town in Maine. Northwest Airlines cancelled the leg of my flight from Detroit to Portland, Maine a few days before my trip and so I had to fly into Manchester, New Hampshire. The drive from Manchester into Maine looked like it would take a little more than two hours and I had good maps and good directions. That shouldn't be a big deal. I've done any number of drives like that in the past without any trouble. But the route was reasonably complicated, my plane would be landing late in the afternoon, and the weather forecast for the area wasn't all that good. So I called up Hertz the day before I left and told them I wanted a car with a GPS. They said that that wouldn't be a problem.
Getting in my car in the Hertz lot in Manchester, I found that making their "NeverLost" GPS unit work was snap. Which it would really have to be since otherwise they would have a bunch of annoyed customers going back into airports demanding that someone show them how the thing works. Hertz and Magellan must have done a lot of usability testing to make the unit as easy to use as it is.
Following the unit's guidance, I made a wrong turn just out of the airport. That was because I had mistaken the scale of the map it was displaying. But even that turned out to be just fine because it calmly displayed "Recomputing route" and told me where to go from where I had gone to. From there to the end of the trip (and, indeed, the return to Manchester) I had not the slightest difficulty reading its map or following its spoken directions. The Hertz unit was mounted near the passenger's left knee. It might have annoyed a passenger slightly but it worked fine for me. It was a little like playing Mario Kart DS, with the moving map below the view forward.
And it was great that the NeverLost worked so well and was so easy to use because it rained the whole way and most of the trip was in the dark. The rain varied between heavy and ridiculously heavy for almost all of the trip. I had the car's windshield wipers at their highest setting pretty well all the time and I occasionally wished for a higher setting. People were driving 45 MPH on highways posted for 65 and some of the roads in my buddy's town were flooded.
In circumstances like those, turning on the car's dome light to read a map or directions would have been pretty unsafe. And the limited amount of information I would have managed to gather doing that under those circumstances combined with the very limited visibility outside would certainly have resulted in my making several wrong turns and often having a suspicion that I had gone wrong even when I hadn't.
But the NeverLost's bright moving-map and voice instructions took me directly to where I was going. The noise of the rain on the windshield was often loud enough that I was concerned that I wouldn't be able to hear the voice prompts, but at the loudest setting it shouted comfortably over the racket. I was thoroughly glad to be done driving when it said in its synthesized voice, "You have arrived", but I would have been there considerably later and considerably more tired without it. As it was, the trip was very annoying. Without the NeverLost, it would have been a nightmare.
When I returned the car at the Hertz facility in Manchester, I told the person who was taking down the car's mileage and printing my receipt that I didn't know just what they had charged me for the GPS unit but, whatever it was, that it had been worth ten times the price. And so I returned home willing to go to the trouble of finding which portable GPS would be best.
A bunch of research suggested that the 2200T would probably be best and I suspect that it is. Other than the NeverLost which is similar, the only other car GPS I've used is one that was built into a buddy's car and so I can't give any very useful comparisons, but I can tell you why I like the 2200T.
I like it chiefly because, like the NeverLost, it does its job in an unremarkable way. The thing is most valuable in difficult conditions and the last thing you need then is a confusing display or an elaborate user interface. On the 2200T, both are admirably simple and clear. A geek should need ten or fifteen minutes and a glance or two at the manual in order to get familiar with the unit. A more normal person might take a little longer, but shouldn't take very long.
(I may be able to save you even brief trips to the manual, which is currently only available as a PDF to download, by telling you that "Enhanced POIs" are locations you enter using Magellan's PC software and the "Trip Planner" is for multi-stop trips. Magellan's software is Windows-only but you don't need it to use the normal functions of the unit. Also, many areas of the moving map screen that aren't obviously buttons respond to tapping on them.)
To use the 2200T, you begin by picking a destination. Actually, you don't have to do that. It will be happy to show you a moving map of your vicinity at any of various scales. But the screen is sufficiently small that it can't show an area more than a few miles across in any significant detail, so it's not all that useful for orienting yourself in an unfamiliar area. The same would be true of any map that's slightly smaller than a file card.
So you begin by selecting a destination. That can be a street address, an intersection, a "point of interest", an address that you've previously entered in your addressbook, or a place picked of the map. You pick a place on the map by zooming out, dragging the map around with your finger, zooming back in again, ensuring that where you want to go is at the center of the map, and tapping there. It's not very convenient, but it works and you're unlikely to need to use it much. "Points of interest" is GPS-speak for a sort of telephone book of locations. If you want to go to the nearest gas station or a particular restaurant or shopping mall, you can query the unit's database and pick where you want to go. GPS manufacturers differentiate their units in part by how many zillions of POIs are in their databases. The 2200T has a database of 1.5m POIs, which is small by current standards. Still, it seems to have everything in it that I can think of and it's been useful in a couple of real-world situations. Naturally, a database like that will eventually become out of date, but I'm sure that Magellan will offer updates.
Once you've picked a destination, the unit will display "Calculating route" for a few seconds or a little longer and then will display your route as a magenta line on the moving map. Just start driving along the line. The unit will display the map with your route ahead of you (assuming that you've configured it to show ahead as up). You'll be able to see your route easily because the screen is a wonder. It is perfectly readable in everything from complete darkness to bright sunlight. The map changes scale automatically using what seems to be a pretty intelligent algorithm to determine how much detail you'll want at what time. You can also change the scale yourself.
The unit displays the name of the road you're on, the name of the road you'll turn onto next, the distance to the next turn, the distance remaining for the trip, and the sort of turn you'll make next. (You can vary that a little, but that's the essence of it.) If you've configured it to (and you should, it's useful) the display will change to a split-screen shortly before a turn. Half of the screen displays the moving map and the other half displays a 3-D rendering of the turn to be made. That may not sound especially useful. How interesting is a picture of a left turn? The answer is that it can help to know what to look for if the turn is something other than a plain 90-degree turn. A picture of a right turn at a shallow angle followed by a sharp left turn over a bridge across the road is quite helpful.
But much better than just a moving map, the unit gives you voice prompts. Its synthesized voice informs you about upcoming turns well in advance, repeats the information closer to the turn, and then plays a chime just before you'll be turning the steering wheel. The software also pronounces the name of the road you'll be turning onto. GPS geeks call that "text to speech". Inevitably, there are some slightly odd pronunciations, but I haven't yet run into one that was incomprehensible, and that includes Wayzata Blvd. The virtue of the voice prompts is of course that you don't have to look at the thing much. It may be desirable to glance at it occasionally, but you can take your eyes off the road a lot less often than you would if you were using an ordinary map or directions.
The 2200T has a couple of other small advantages: The battery life seems pretty good. Mine ran about eight hours from a full charge before switching itself off. I strongly suspect that battery life is affected by how much work the unit is doing and since I wasn't asking it to do much that was very hard during those eight hours, I'd suggest taking that as an upper limit. Still, I'm pretty impressed.
The unit feels solidly built and it appears to be sealed against weather pretty well. I wouldn't advise you to take it swimming with you, but I doubt that a little rain would harm it.
The downside to its being pretty well sealed is that the battery isn't user-replaceable. Making a watertight lid for a battery compartment would be hard. That means that when the battery no longer holds a useful charge, you'll need to send it in for service. But you'll probably run it mostly from its cigarette-lighter power cord and so a battery replacement may be pretty far in the future.
I don't have a lot to compare it to, but the unit's reception of GPS signals seems quite good. It works just fine in my apartment as long as I stay pretty close to the windows. In a car, the unit doesn't have to be stuck to the windshield to get a good signal. Oriented randomly in a bag in the back seat seems to work fine.
There are a few other compromises and imperfections.
There are a couple of small problems with the user interface. One is that there's no simple way to change the volume when you're looking at the map screen. There's a mute button that you can tap on that screen to silence the unit, and that's probably a useful thing. But the button should really pop up a volume control. If it starts raining hard enough to make a racket, you're not going to want to go digging through various screens looking for the volume control.
It would also be useful if there were an audio indication of when the split-screen view of the upcoming turn is available. Since the screen returns to the usual map display just before the turn, you pretty much have to guess when to look at it if the unit isn't mounted in your line of sight. We're not supposed to mount things to our windshields here in Minnesota.
There's another small imperfection that's a bit more subtle: When you've specified a route, you can simulate driving it. In a preferences page, you can tell the 2200T that you want it to offer you the option of simulating driving a route after it has calculated it. You can also specify some options for the simulation, such as doing the simulated driving faster than you'd actually drive the route. I can imagine that that's a potentially useful feature. I might like to preview an unusual or complicated route before driving it so as to see what I'd shortly be doing or to have a look at the route to decide if I liked the one that the unit had chosen. (Within limits, you can influence the route that the unit picks.)
But there is a minor problem with the feature. If you have the option to preview a route turned on, it gives you that option even when the unit computes a new route because you've departed from the original one. If you've departed from the route that the 2200T has chosen, it's probably because of bad traffic or to detour around a road that's closed. No one is going to want to preview the new route then. They're driving. It's a relatively minor thing to tap a No button to say that you don't want the preview, but it's a distraction. And the 2200T is good because it almost always doesn't distract you.
If your route has you continuing on the same road for more than a few minutes, the 2200T occasionally speaks, telling you that you'll be continuing on the road. That may not sound like a very useful feature. But the NeverLost didn't have it and, on a stretch of highway that I was on for about an hour, I wished that it would talk to me once in a while. Was the volume set loud enough that I could still hear it? Was it still working correctly? An occasional spoken status message would have spared me a few glances at the NeverLost and a few unnecessary clicks of the volume-up button. The 2200T says "Continue on the current road" every few miles and I think that's a small but valuable feature. It would be even better in my opinion if it incorporated some useful information into the message, such as the number of miles remaining on the current road.
The underlying database of road information that the 2200T uses is licensed from NAVTEQ. It seems that they don't have a lot of competition for the US and Canada. Garmin's GPSs also use their data and so do the map websites of Google, Yahoo, and MapQuest. The database is good but it's not perfect. For example, it doesn't know that in Minneapolis, where I live, 1st Avenue South is two-way between Franklin and 28th St.
The road database knows about the reversible lanes on I-394 near downtown Minneapolis, but it doesn't know all the ways they're connected to the rest of the highway. It correctly identified that it was traveling on them, but at the point where they were about to rejoin the rest of the highway going west, it requested a turn that, while safe, wasn't in the right direction. Proceeding in the right direction on the highway, the unit requested a u-turn, cautioning that it should be a safe and legal one. Still proceeding in the right direction, the unit gave a "You can't get there from here" message and I needed to re-enter the destination. (Happily, I was a passenger at the time and could do that without any fuss.)
In addition, here in Minneapolis, streets and avenues are generally labeled as being North or South or Northeast or Southeast, depending on their position with respect to the Mississippi River and Hennepin Avenue downtown. But we're a bit sloppy about whether we say "First Avenue South" or "South First Avenue". Also, some streets exist only in one of the four divisions. According to the post office, those streets shouldn't have a direction marking since they don't need to be distinguished from different versions.
Entering a street address in the 2200T is admirably simple and hard to do wrong. You begin by spelling the street name on an on-screen keyboard. Letters that can't come next disappear, making it easier to find the ones you want. When you've entered enough letters that there are only a few possibilities for what you're spelling, the unit shows you a list and you pick the street from it. Minneapolitans' casual attitudes to where they put their souths and norths is mirrored in the database. For example, in the database "S. Bryant Ave" has house numbers from 1600 to 3599 and "Bryant Ave South" has house numbers from 1600 to 9399. Holmes Avenue, which exists only in the south section of the city, is listed as "Holmes Ave", "Holmes Ave S.", and "South Holmes Ave". The house numbers listed for the first one don't actually exist.
None of those imperfections particularly surprises me. The 2200T contains a database of all the roads in the United States and Canada. In my experience, a database of that size pretty much can't be perfect. But it's worth remembering that when you're using the unit. Apart from the issue with the reversible lanes, none of the errors would cause any significant trouble, and the unit hadn't recommended taking the reversible lanes.
Once you've specified an address, you can satisfy yourself that you picked the place you meant by tapping a bulls-eye icon. That will show you a map centered on your destination. A text button saying something like, oh, "Show on Map" would have been a little more obvious, but once you know what the little bulls-eye does it's easy enough.
The routes that I've seen 2200T choose have always been good but, in cases in which there are a large number of very similar routes, such as traveling diagonally through a grid of city streets, the routes it has picked haven't always absolutely optimal. That's hardly surprising. It would require information about every stop sign and traffic light in the US and Canada in order to pick the optimal route from dozens or hundreds that are very similar in distance and time. The unit seems to have a preference for broad and one-way streets. That's probably a good preference to have since it's likely to be guiding you on unfamiliar roads.
When you're asking it to calculate a route, you can tell it to pick a route with a minimum use of freeways. That's often handy at rush hour around here. It will also notice that you're in slow traffic on a freeway and indicate that it's willing to find you a different route. There's also an optional traffic receiver (not available yet and probably requiring a paid subscription) which ought to enable it to choose better routes when traffic is heavy.
I mentioned earlier that GPS manufacturers often leave things that you're going to want out of the package so as to make additional sales from accessories. The 2200T is just missing a case and an AC adaptor. That's actually pretty good as these things go. And you may not really need an AC adaptor since you'll probably use the unit in a car almost all the time and it comes with a cigarette-lighter adaptor. Still, I knew that I'd want to learn how to use it at home rather than sitting in my car and I may possibly want to use it on my motorcycle so I wanted a AC adaptor. At the time I ordered the unit, Magellan's website didn't show an AC adaptor as an accessory available for the 2200T. It did show them for other similar units and I supposed that they were the same. Still, I wasn't sure enough to place my order on the website, so I placed the order by telephone, specifying that I wanted an AC adaptor for the 2200T. A few days later the adaptor arrived, but without any prongs to plug it into the wall. Magellan sells their GPS units in the United States and also in Europe. Cleverly, they've designed a universal power supply that just needs to have prongs appropriate to the local outlets clipped to it. So I literally had an AC adaptor, just not a useful one. I called the same number and explained the situation and the nice person who took the call sent me the right prongs without charge. I think that's pretty good. Anyone can make a mistake, but Magellan fixed that one quickly and pretty painlessly.
Magellan still doesn't have a case available for the 2200T. I'm using this, which is a sort of fluffy padded napkin with the hook side of velcro (er, hook-and-loop fastening material) at the corners. You wrap it around something, forming s sort of envelope and then open one corner to slip the thing out and back in. It works as advertised, but the result is a bit bulkier than I'd like.
The unit has an MP3 player and a picture viewer built in. Since I have no idea why anyone would want to use those features, I have nothing else to say about them.
I'm very impressed with the 2200T. It's not the sort of thing I need every day because I travel on familiar routes most of the time. But if it guides me on another trip or two like the one from New Hampshire into Maine and saves me a bit of fuss on an occasional trip to the wilds of St. Paul, it will have been worth the price.
Posted at 06:27 Main Permalink
Sat - November 11, 2006
Inheritance vs. delegation in Python
In previous episodes of my tiny introduction to
object-oriented programming in Python, we've had a look at what
objects and classes are and at inheritance.
Then we took a brief detour to look at closures,
which are like objects in some ways. Today let's have a look at
delegation.
We've previously seen that inheritance is a useful way of specializing or otherwise changing the behavior of a class that already exists. Another way of looking at that is that your class is getting some other class to do most of the work.
There's another way of getting another class to do most of the work that's sometimes useful. It involves explicitly handing particular method calls off to an instance of another class. That's such an obvious technique that it seems that it hardly deserves a name, but it's called delegation.
Let's say we wanted a dictionary that didn't raise a KeyError when a key wasn't found, but instead returned a default value of None. (You don't need to implement that since it's in the collections module as of Python 2.4 but it's a useful example.)
One way to implement that using inheritance is like this:
class defaultDict(dict):
def __getitem__(self,key):
if self.has_key(key):
return self.get(key)
else:
return None # Default value
If, for some reason, you didn't want to use inheritance, you could give your defaultDict instances dicts of their own and call those dicts' methods when the same methods of your instances were called, making whatever changes you wanted. Here's one way to implement that:
class defaultDict:
def __init__(self):
self.d={}
def __setitem__(self,key,value):
self.d[key]=value
def __getitem__(self,key):
if self.d.has_key(key):
return self.d[key]
else:
return None # Default
(Here's a bit of terminology: Delegation is also sometimes called "containment". The inheritance relationship is sometimes calls "is-a" or "ISA", as in "the defaultDict is a dict". The delegation relationship is sometimes called "has-a" or "HASA", as in "the defaultDict has a dict".)
You might reasonably ask why anyone would want to do that. It's more code (a lot more if you wanted all of a dict's methods to work). One reason you might have done it once upon a time is that you used not to be able to inherit from built-in types. But that has since been fixed in Python, though the UserDict and UserList modules that did the delegation so that you could inherit from them are still in the standard library.
The real reason that you'd want to use delegation is that the class you were designing has some resemblance to another class, but isn't really enough like it that you'd call it the same kind of thing. That's obviously not the case with our defaultDict. But take, for example, the Message class in Python's email module. You can index a Message object as though it were a dictionary to get at the message's headers, such as "Subject" and "From". Even if you didn't index an email message, it might well make sense to store the headers in something like dictionary. But I don't think that anyone I know would call an email message a kind of dictionary. (The Message class doesn't actually use a dictionary, but that's an implementation detail that's not important for our purposes here.)
One reasonable way to choose between inheritance and delegation is to look at whether you want all of the other class's methods. While it makes sense to get and set an email message's headers, a dictionary's pop() method probably doesn't make sense for an email message and neither, really, do a do a few others. Similarly, a dict's len() method is obvious enough, but finding the length of an email message isn't really related to the number of keys in a dictionary.
So if inheriting would mean that you would need to turn off some methods or implement some in a way that's not related to the parent class's implementation, you may be better off with delegation.
We've previously seen that inheritance is a useful way of specializing or otherwise changing the behavior of a class that already exists. Another way of looking at that is that your class is getting some other class to do most of the work.
There's another way of getting another class to do most of the work that's sometimes useful. It involves explicitly handing particular method calls off to an instance of another class. That's such an obvious technique that it seems that it hardly deserves a name, but it's called delegation.
Let's say we wanted a dictionary that didn't raise a KeyError when a key wasn't found, but instead returned a default value of None. (You don't need to implement that since it's in the collections module as of Python 2.4 but it's a useful example.)
One way to implement that using inheritance is like this:
class defaultDict(dict):
def __getitem__(self,key):
if self.has_key(key):
return self.get(key)
else:
return None # Default value
If, for some reason, you didn't want to use inheritance, you could give your defaultDict instances dicts of their own and call those dicts' methods when the same methods of your instances were called, making whatever changes you wanted. Here's one way to implement that:
class defaultDict:
def __init__(self):
self.d={}
def __setitem__(self,key,value):
self.d[key]=value
def __getitem__(self,key):
if self.d.has_key(key):
return self.d[key]
else:
return None # Default
(Here's a bit of terminology: Delegation is also sometimes called "containment". The inheritance relationship is sometimes calls "is-a" or "ISA", as in "the defaultDict is a dict". The delegation relationship is sometimes called "has-a" or "HASA", as in "the defaultDict has a dict".)
You might reasonably ask why anyone would want to do that. It's more code (a lot more if you wanted all of a dict's methods to work). One reason you might have done it once upon a time is that you used not to be able to inherit from built-in types. But that has since been fixed in Python, though the UserDict and UserList modules that did the delegation so that you could inherit from them are still in the standard library.
The real reason that you'd want to use delegation is that the class you were designing has some resemblance to another class, but isn't really enough like it that you'd call it the same kind of thing. That's obviously not the case with our defaultDict. But take, for example, the Message class in Python's email module. You can index a Message object as though it were a dictionary to get at the message's headers, such as "Subject" and "From". Even if you didn't index an email message, it might well make sense to store the headers in something like dictionary. But I don't think that anyone I know would call an email message a kind of dictionary. (The Message class doesn't actually use a dictionary, but that's an implementation detail that's not important for our purposes here.)
One reasonable way to choose between inheritance and delegation is to look at whether you want all of the other class's methods. While it makes sense to get and set an email message's headers, a dictionary's pop() method probably doesn't make sense for an email message and neither, really, do a do a few others. Similarly, a dict's len() method is obvious enough, but finding the length of an email message isn't really related to the number of keys in a dictionary.
So if inheriting would mean that you would need to turn off some methods or implement some in a way that's not related to the parent class's implementation, you may be better off with delegation.
Posted at 07:43 Main Permalink
Mon - September 18, 2006
Closures in Python
I've previously written two tiny and not at all
exhaustive posts about objects in Python (1,
2).
Here's a similar discussion of another programming topic: closures. I think that
closures are a bit more obscure than objects. The examples here are in Python,
but you can do pretty much the same thing in many other
languages.
Closures fall out of two things that Python can do. The first is that in Python we can treat functions as though there were data. (That's approximately what's meant when some people say that functions in Python are "first-class objects".)
If I define a function:
>>> def f(x):
... print x+1
I can assign it to a new name:
>>> g=f
And call the function with the new name:
>>> g(42)
43
I can put it in a list:
>>> l=[1,"a",f]
>>> l[2]
<function f at 0x4c670>
And call it from there:
>>> l[2](11)
12
It's not amazingly common, but it can occasionally be useful to put functions in various sorts of data structures. Perhaps a dictionary of functions indexed by the sort of data that they work on or produce.
(For example, I once had occasion to parse a calendar format. In calendar formats, repeating events (such as "2:00 pm on the last Thursday of the month") aren't recorded as dates and times since there could be an infinite number of instances of such an event. Instead, they're recorded as specifications for generating the actual instances. It was convenient for me to have the functions that would generate the next event in a dictionary that was indexed by the kind of repetition they did (daily, weekly, etc). But that's by the way for our purpose here.)
The second thing that closures depend on is that Python programs can have nested functions. That is, functions defined inside other functions:
>>> def outer(x):
... def inner(y):
... return y+2
... print inner(x)
...
>>> outer(10)
12
Nesting functions can be useful in making a program more readable. If a particular function can be made clearer by defining a helper function and the helper function is only going to be useful to the first function, it can make sense to define the helper inside the first function. That way someone who's reading your program and sees the helper doesn't need to wonder where it's going to be used.
Now, we can put those two facts together by having a function return a "customized" version of an inner function. For example:
>>> def outer(x):
... def inner(y):
... return x+y
... return inner
...
>>> customInner=outer(2)
>>> customInner(3)
5
The trick that you want to notice in what's going on there is what happens to the value of x. The argument x is a local variable in outer() and the behavior of local variables isn't normally very exciting. But in this case, x is global to the function inner(). And since inner() uses the name, it doesn't go away when outer() exits. Instead inner() captures it or "closes over" it. You can call outer() as many times as you like and each value of x will be captured separately.
The function that's returned is called a closure. The idea is potentially useful because we can specify part of the behavior of a function based on data at runtime.
At this stage you might say, "OK, I followed your tedious explanation, but what good is such a thing? Is it anything more than a curiosity for ordinary programming?" The answer to that is that it is occasionally useful when something, such as a library interface, requires a function and you want to specify more than a couple of them that are very similar.
Imagine that you're designing a GUI interface and you need six buttons that do similar things. Tkinter buttons take a function to call as an argument and it would be tedious to write six very similar functions. Instead you might do something like this:
from Tkinter import *
def makeButtonFunc(buttonName):
def buttonFunc():
print buttonName
return buttonFunc
class mainWin:
def __init__(self,root):
self.root=root
self.createWidgets()
return None
def createWidgets(self):
for buttonName in ("A","B","C","D","E","F"):
b=Button(self.root,text=buttonName,
command=makeButtonFunc(buttonName))
b.pack()
return None
def main():
root=Tk()
mainWin(root)
root.mainloop()
return None
if __name__=="__main__":
main()
That's clearly better than writing six functions that are virtually identical.
There are lots of people who like using closures. I, personally, don't. To me, it feels like using a subtle trick and I prefer my programs to be as obvious as possible. In a similar situation, I'd use a Python object with a __call__() method. If a Python object has that method and it's called as though it were a function, that method is run. In a program I wrote, I'd probably replace makeButtonFunc() with something like:
class makeButtonFunc:
def __init__(self,buttonName):
self.buttonName=buttonName
def __call__(self):
print self.buttonName
Which would do the same thing. Of course, I'd give the class a different name.
Closures fall out of two things that Python can do. The first is that in Python we can treat functions as though there were data. (That's approximately what's meant when some people say that functions in Python are "first-class objects".)
If I define a function:
>>> def f(x):
... print x+1
I can assign it to a new name:
>>> g=f
And call the function with the new name:
>>> g(42)
43
I can put it in a list:
>>> l=[1,"a",f]
>>> l[2]
<function f at 0x4c670>
And call it from there:
>>> l[2](11)
12
It's not amazingly common, but it can occasionally be useful to put functions in various sorts of data structures. Perhaps a dictionary of functions indexed by the sort of data that they work on or produce.
(For example, I once had occasion to parse a calendar format. In calendar formats, repeating events (such as "2:00 pm on the last Thursday of the month") aren't recorded as dates and times since there could be an infinite number of instances of such an event. Instead, they're recorded as specifications for generating the actual instances. It was convenient for me to have the functions that would generate the next event in a dictionary that was indexed by the kind of repetition they did (daily, weekly, etc). But that's by the way for our purpose here.)
The second thing that closures depend on is that Python programs can have nested functions. That is, functions defined inside other functions:
>>> def outer(x):
... def inner(y):
... return y+2
... print inner(x)
...
>>> outer(10)
12
Nesting functions can be useful in making a program more readable. If a particular function can be made clearer by defining a helper function and the helper function is only going to be useful to the first function, it can make sense to define the helper inside the first function. That way someone who's reading your program and sees the helper doesn't need to wonder where it's going to be used.
Now, we can put those two facts together by having a function return a "customized" version of an inner function. For example:
>>> def outer(x):
... def inner(y):
... return x+y
... return inner
...
>>> customInner=outer(2)
>>> customInner(3)
5
The trick that you want to notice in what's going on there is what happens to the value of x. The argument x is a local variable in outer() and the behavior of local variables isn't normally very exciting. But in this case, x is global to the function inner(). And since inner() uses the name, it doesn't go away when outer() exits. Instead inner() captures it or "closes over" it. You can call outer() as many times as you like and each value of x will be captured separately.
The function that's returned is called a closure. The idea is potentially useful because we can specify part of the behavior of a function based on data at runtime.
At this stage you might say, "OK, I followed your tedious explanation, but what good is such a thing? Is it anything more than a curiosity for ordinary programming?" The answer to that is that it is occasionally useful when something, such as a library interface, requires a function and you want to specify more than a couple of them that are very similar.
Imagine that you're designing a GUI interface and you need six buttons that do similar things. Tkinter buttons take a function to call as an argument and it would be tedious to write six very similar functions. Instead you might do something like this:
from Tkinter import *
def makeButtonFunc(buttonName):
def buttonFunc():
print buttonName
return buttonFunc
class mainWin:
def __init__(self,root):
self.root=root
self.createWidgets()
return None
def createWidgets(self):
for buttonName in ("A","B","C","D","E","F"):
b=Button(self.root,text=buttonName,
command=makeButtonFunc(buttonName))
b.pack()
return None
def main():
root=Tk()
mainWin(root)
root.mainloop()
return None
if __name__=="__main__":
main()
That's clearly better than writing six functions that are virtually identical.
There are lots of people who like using closures. I, personally, don't. To me, it feels like using a subtle trick and I prefer my programs to be as obvious as possible. In a similar situation, I'd use a Python object with a __call__() method. If a Python object has that method and it's called as though it were a function, that method is run. In a program I wrote, I'd probably replace makeButtonFunc() with something like:
class makeButtonFunc:
def __init__(self,buttonName):
self.buttonName=buttonName
def __call__(self):
print self.buttonName
Which would do the same thing. Of course, I'd give the class a different name.
Posted at 08:07 Main Permalink
Sun - October 2, 2005
Creating iTunes playlists with Python
I've written a smallish Python program to create iTunes
playlists according to particular rules that suit me. You're welcome to the
program; you can distribute it under the GPL. But since the
particular rules that suit me probably won't suit you exactly, it may not be of
much use to you unless you want to hack on it a bit
yourself.
There are surely bugs in it and legitimate sorts of data in the music library file that it reads that I haven't anticipated. When bugs have caused it to create a bad playlist file, iTunes has rejected the file. But maybe I've just been lucky and a bad playlist file could do something Very Bad. As with programs in general, use it at your own risk. It cheerfully assumes that your terminal wants text output in UTF-8. That may not matter if you don't have any non-ASCII data in your music library. Bug reports and patches are naturally very welcome.
The only non-standard module it requires is Fredrik Lundh's excellent ElementTree module for manipulating XML.
I wrote the program after going down to my local Apple store and having a look at the iPod nano. It's hard to pick one up and not want one. (Though it was immediately clear to me that the thing would scratch easily and therefore really needs a case.) But practicality stopped me from buying one then. I have a reasonably big music collection: one reason that I like my 60GB iPod so much is that it lets me store my 300-some CDs in the basement. Would an iPod that holds only 4GB be of any use?
I imagined that 4GB ought to hold most or all my current favorites, so I created a playlist with my current favorites on it. It fit comfortably in 4GB, but it turned out to be pretty tedious to listen to. Because they were my favorites, I'd listened to most of them recently. I wanted something I hadn't heard recently pretty often.
It would be possible to create a playlist by hand that contained a bunch of favorites and also some albums I hadn't listened to recently, but it would quickly become a nuisance to maintain. So I wondered if it was possible to have a program create a playlist automatically. And it is.
It turns out that iTunes saves an XML description of your music library in:
~/Music/iTunes/iTunes Music Library.xml
That's not iTunes's main data file. As far as I can tell, it's a version that's generated for other programs to read. Changing it doesn't seem to accomplish anything. But iTunes will import playlists that are written in XML. So a program can read the XML version of your music library and generate a playlist based applying whatever rules it likes. And you can then import the playlist.
As far as I'm aware, neither of those file formats is documented publicly. So what the program expects to read and decides to write are based on inspection and experiment. I've very likely missed a few cases. As I mentioned above, bug reports and patches are very welcome.
The program actually creates several playlists, all in one file that's saved on the desktop. It starts with all the music that has been added within the last 60 days and then adds everything that hasn't ever been played. It puts them in a playlist called "AG-Recently Added". It then gets the music that has been played most and puts that in a playlist called "AG-Most Played". Assuming that it hasn't run out of space already, it adds albums from configurable genres chosen quasi-randomly, with a bias toward albums that haven't been played recently. Those playlists are named "AG-" and the genre name.
I still haven't gotten a nano. I'll have to see how I like the playlists that get generated and maybe tweak the rules a bit.
There are surely bugs in it and legitimate sorts of data in the music library file that it reads that I haven't anticipated. When bugs have caused it to create a bad playlist file, iTunes has rejected the file. But maybe I've just been lucky and a bad playlist file could do something Very Bad. As with programs in general, use it at your own risk. It cheerfully assumes that your terminal wants text output in UTF-8. That may not matter if you don't have any non-ASCII data in your music library. Bug reports and patches are naturally very welcome.
The only non-standard module it requires is Fredrik Lundh's excellent ElementTree module for manipulating XML.
I wrote the program after going down to my local Apple store and having a look at the iPod nano. It's hard to pick one up and not want one. (Though it was immediately clear to me that the thing would scratch easily and therefore really needs a case.) But practicality stopped me from buying one then. I have a reasonably big music collection: one reason that I like my 60GB iPod so much is that it lets me store my 300-some CDs in the basement. Would an iPod that holds only 4GB be of any use?
I imagined that 4GB ought to hold most or all my current favorites, so I created a playlist with my current favorites on it. It fit comfortably in 4GB, but it turned out to be pretty tedious to listen to. Because they were my favorites, I'd listened to most of them recently. I wanted something I hadn't heard recently pretty often.
It would be possible to create a playlist by hand that contained a bunch of favorites and also some albums I hadn't listened to recently, but it would quickly become a nuisance to maintain. So I wondered if it was possible to have a program create a playlist automatically. And it is.
It turns out that iTunes saves an XML description of your music library in:
~/Music/iTunes/iTunes Music Library.xml
That's not iTunes's main data file. As far as I can tell, it's a version that's generated for other programs to read. Changing it doesn't seem to accomplish anything. But iTunes will import playlists that are written in XML. So a program can read the XML version of your music library and generate a playlist based applying whatever rules it likes. And you can then import the playlist.
As far as I'm aware, neither of those file formats is documented publicly. So what the program expects to read and decides to write are based on inspection and experiment. I've very likely missed a few cases. As I mentioned above, bug reports and patches are very welcome.
The program actually creates several playlists, all in one file that's saved on the desktop. It starts with all the music that has been added within the last 60 days and then adds everything that hasn't ever been played. It puts them in a playlist called "AG-Recently Added". It then gets the music that has been played most and puts that in a playlist called "AG-Most Played". Assuming that it hasn't run out of space already, it adds albums from configurable genres chosen quasi-randomly, with a bias toward albums that haven't been played recently. Those playlists are named "AG-" and the genre name.
I still haven't gotten a nano. I'll have to see how I like the playlists that get generated and maybe tweak the rules a bit.
Posted at 08:30 Main Permalink
Sun - September 11, 2005
Klipsch ProMedia Ultra 2.0 powered speakers
A while ago I bought a pair of JBL Duet powered
speakers to use in a spot where I'd like to listen to music but where I don't
have a place for a subwoofer. Unfortunately, they
suck.
A little wiser but only $35 poorer, I ordered a pair of Klipsch ProMedia Ultra 2.0 speakers. Happily, they don't suck. Indeed, they're good, or maybe very good for what they are.
Make no mistake, they're not stunningly great speakers, and the absence of a subwoofer shows in the sound they're able to reproduce. If you have a place for a subwoofer, get a decent set of speakers that includes one. But if you don't, the ProMedia Ultras are respectable speakers. Given that they do a good job, $100 isn't an unreasonable price for them. I certainly think that they're more than three times as good as the JBLs.
The ProMedias are pretty highly directional. The stereo "sweet spot" they produce isn't big. But if you wanted to fill a room with music as opposed to listening to something while sitting at a computer, you'd use a set of speakers that has a subwoofer.
On the front of the right speaker, there are volume and bass knobs and another input and an output jack. The output jack is there so that you can plug in headphones if you want to use them occasionally without having to re-cable things. The jack would be much more useful if its output didn't contain really nasty hiss. The hiss isn't in the sound from the speakers.
Their industrial design isn't going to win any awards. They're relatively narrow when viewed from the front (around 3 1/4") but they're pretty tall (10 1/2" or a bit more) and pretty deep (7" or so). They're around the size and shape of a good-sized hardcover book. If you put them next to a monitor that faces a wall, they'd probably be pretty unobtrusive. If you put them on a table that faces a room, they look large and awkward from the side. In addition, the green power LED on the right speaker is brighter than it needs to be. It's bright enough to be a bit distracting in a room that's not brightly lit. Still, decent sound reproduction is more important to me than fabulous industrial design and so I'm pleased with them.
A little wiser but only $35 poorer, I ordered a pair of Klipsch ProMedia Ultra 2.0 speakers. Happily, they don't suck. Indeed, they're good, or maybe very good for what they are.
Make no mistake, they're not stunningly great speakers, and the absence of a subwoofer shows in the sound they're able to reproduce. If you have a place for a subwoofer, get a decent set of speakers that includes one. But if you don't, the ProMedia Ultras are respectable speakers. Given that they do a good job, $100 isn't an unreasonable price for them. I certainly think that they're more than three times as good as the JBLs.
The ProMedias are pretty highly directional. The stereo "sweet spot" they produce isn't big. But if you wanted to fill a room with music as opposed to listening to something while sitting at a computer, you'd use a set of speakers that has a subwoofer.
On the front of the right speaker, there are volume and bass knobs and another input and an output jack. The output jack is there so that you can plug in headphones if you want to use them occasionally without having to re-cable things. The jack would be much more useful if its output didn't contain really nasty hiss. The hiss isn't in the sound from the speakers.
Their industrial design isn't going to win any awards. They're relatively narrow when viewed from the front (around 3 1/4") but they're pretty tall (10 1/2" or a bit more) and pretty deep (7" or so). They're around the size and shape of a good-sized hardcover book. If you put them next to a monitor that faces a wall, they'd probably be pretty unobtrusive. If you put them on a table that faces a room, they look large and awkward from the side. In addition, the green power LED on the right speaker is brighter than it needs to be. It's bright enough to be a bit distracting in a room that's not brightly lit. Still, decent sound reproduction is more important to me than fabulous industrial design and so I'm pleased with them.
Posted at 07:12 Main Permalink
Fri - September 2, 2005
JBL Duet speakers
I recently moved and in my new apartment I have a
use for a pair of desktop speakers. My Monsoon
stereo-plus-subwoofer set (2.1 in the jargon) remains very good indeed, but the
best place for them in the apartment isn't particularly near the best place for
me to work on my laptop.
I could just put them where I work on my laptop, but then they'd be in a lousy place to play when I want music in the living room, and there's no good place for a subwoofer under the table I like to work at.
So I went clicking around, looking for decent desktop or "computer" speakers that included just two stereo speakers and no subwoofer (2.0 in the jargon). I don't need for them to perform well when they're turned up loud since they'll be just a couple of feet from me. The JBL Duets seemed to get reasonably good reviews at Amazon and they were on sale there for $35 delivered. They also have the small advantage that they look reasonably nice from the back which, given how the room is arranged, is how most folks will see them.
Unfortunately, they suck. The highs are muddy and the bass doesn't exist. A cymbal crash sounds like a sketch of a cymbal crash and anything around the tone of a cello or a drum sounds like some irregular thumping that you can just about perceive.
I'm not a lunatic audiophile, but I think I'm also not what Dan Rutter would call, in his inimitable way, a "cloth-eared git". I wasn't expecting wonders from the Duets. They're cheap and it's going to be hard to produce good bass without a subwoofer. But they don't even live up to what I expected for $35. If someone would like a pair and is willing to come to Minneapolis to pick them up, send me an email. You can have them.
If anyone can suggest a 2.0 set of powered speakers that doesn't suck, I'd be glad to hear about them. In the meantime, I'll just wear my Sennheiser PX-100s.
I could just put them where I work on my laptop, but then they'd be in a lousy place to play when I want music in the living room, and there's no good place for a subwoofer under the table I like to work at.
So I went clicking around, looking for decent desktop or "computer" speakers that included just two stereo speakers and no subwoofer (2.0 in the jargon). I don't need for them to perform well when they're turned up loud since they'll be just a couple of feet from me. The JBL Duets seemed to get reasonably good reviews at Amazon and they were on sale there for $35 delivered. They also have the small advantage that they look reasonably nice from the back which, given how the room is arranged, is how most folks will see them.
Unfortunately, they suck. The highs are muddy and the bass doesn't exist. A cymbal crash sounds like a sketch of a cymbal crash and anything around the tone of a cello or a drum sounds like some irregular thumping that you can just about perceive.
I'm not a lunatic audiophile, but I think I'm also not what Dan Rutter would call, in his inimitable way, a "cloth-eared git". I wasn't expecting wonders from the Duets. They're cheap and it's going to be hard to produce good bass without a subwoofer. But they don't even live up to what I expected for $35. If someone would like a pair and is willing to come to Minneapolis to pick them up, send me an email. You can have them.
If anyone can suggest a 2.0 set of powered speakers that doesn't suck, I'd be glad to hear about them. In the meantime, I'll just wear my Sennheiser PX-100s.
Posted at 07:15 Main Permalink
A brief introducton to object-oriented programming in Python: Part 2, inheritance
A while ago I posted
a tiny introduction to object-oriented programming in Python. Here's part two,
about inheritance. It's equally tiny and equally doesn't attempt to be
comprehensive.
Inheritance is a way of specifying a new class that's almost, but not exactly, like a class you already have. Let's extend the example from the page above. Assume that I (or maybe someone who has written a library module) has already defined class c3 for us:
>>> class c3:
... def __init__(self,x):
... self.val=x
... def getValAsDouble(self):
... return self.val*2
And let's assume that, for the purposes of the program we're writing, that's a useful class. But it would be even more useful if it had a getValAsTriple() method. I can save some work by saying that class c4 inherits from class c3. Then I only need to specify what I want to add:
>>> class c4(c3):
... def getValAsTriple(self):
... return self.val*3
...
>>> o6=c4(42)
>>> o6.getValAsDouble() # Comes for free through inheritance from c3
84
>>> o6.getValAsTriple() # Defined in c4
126
As a matter of terminology, the class you inherit from is often called the "superclass" and your class that inherits from it is its "subclass".
Now, it's pretty rare that you find a class that just happens to be useful to inherit from. Many classes that you inherit from were designed specifically to be inherited from.
Inheritance has another advantage: If the class you inherit from is supplied by a library module, you don't have to know or care about the details of how it gets its work done.
Let's say that the author of some module thinks of a better way for one of its classes to do something. Or maybe they fix a bug or two. If you install the new version of the library, you get the improvements "for free". You don't have to go to any extra trouble, because you're now inheriting from the improved version.
There's one more thing: What does inheriting from "object" do? That class is more like a signal than a class that provides useful methods that you'll use every day. It tells Python that you want your class to be a "new-style" class. New-style classes solve a few problems that not very many people have. (For those people they're important, but nobody who's starting out with object-oriented programming is one of them.) So don't worry about inheriting from object any time soon. I've been programming in Python for years and very rarely need to use a new-style class.
Inheritance is a way of specifying a new class that's almost, but not exactly, like a class you already have. Let's extend the example from the page above. Assume that I (or maybe someone who has written a library module) has already defined class c3 for us:
>>> class c3:
... def __init__(self,x):
... self.val=x
... def getValAsDouble(self):
... return self.val*2
And let's assume that, for the purposes of the program we're writing, that's a useful class. But it would be even more useful if it had a getValAsTriple() method. I can save some work by saying that class c4 inherits from class c3. Then I only need to specify what I want to add:
>>> class c4(c3):
... def getValAsTriple(self):
... return self.val*3
...
>>> o6=c4(42)
>>> o6.getValAsDouble() # Comes for free through inheritance from c3
84
>>> o6.getValAsTriple() # Defined in c4
126
As a matter of terminology, the class you inherit from is often called the "superclass" and your class that inherits from it is its "subclass".
Now, it's pretty rare that you find a class that just happens to be useful to inherit from. Many classes that you inherit from were designed specifically to be inherited from.
Inheritance has another advantage: If the class you inherit from is supplied by a library module, you don't have to know or care about the details of how it gets its work done.
Let's say that the author of some module thinks of a better way for one of its classes to do something. Or maybe they fix a bug or two. If you install the new version of the library, you get the improvements "for free". You don't have to go to any extra trouble, because you're now inheriting from the improved version.
There's one more thing: What does inheriting from "object" do? That class is more like a signal than a class that provides useful methods that you'll use every day. It tells Python that you want your class to be a "new-style" class. New-style classes solve a few problems that not very many people have. (For those people they're important, but nobody who's starting out with object-oriented programming is one of them.) So don't worry about inheriting from object any time soon. I've been programming in Python for years and very rarely need to use a new-style class.
Posted at 06:47 Main Permalink
Mon - March 21, 2005
XinFeed passive crossfeed filter for headphones
I've previously mentioned
my AirHead
headphone amplifier from HeadRoom. I
like it very much and I use it routinely when I'm listening to something on my
iBook at home. Ironically, at the moment I don't have any headphones that are
particularly difficult to drive so I mostly use it for its
crossfeed.
Crossfeed?
Yes: When you listen to stereo speakers, most of the right channel goes to your right ear, but some of it goes to your left ear, and vice-versa. Most music is recorded with the expectation that "air mixing" of that sort will happen when it's listened to. But there isn't normally any mixing when you're listening with headphones. The left channel is delivered only to your left ear and the right channel only to your right ear. A crossfeed circuit fixes that; it mixes some of the right channel into the left channel and vice-versa.
Actually, it can be a little more complicated than that. It seems that, on account of the shape of a person's head and ears, the crossfeed that a person perceives from air mixing varies some depending on a sound's frequency. It's straightforward enough to approximate that variation in a crossfeed circuit. Of course, as with many things in audio, it's possible to take matters to an extreme. (There's more at HeadRoom's admirably informative site.)
I'm no nutty audiophile, but after having listened to various things for some time with my AirHead's crossfeed circuit, I find it substantially more pleasant to listen to headphones with crossfeed than without. The sound seems more natural and it seems to be coming from in front of me rather than from beside me.
But what about portable music? It's perfectly possible to carry an AirHead around, and its four AAA batteries last an admirably long time. But it's about the size of a standard iPod, And so while you can carry it, I'm not likely to. And connecting one to an iPod Shuffle would be plain silly.
Happily, there's a solution to the portability problem in the form of XY Computing & Network's XinFeed (specifically, the low-impedance ampless version). The XinFeed is a crossfeed circuit built into a slightly lumpy and asymmetrical widget that's not a lot bigger than the 1/8" mini jack and plug that it has to have. Here's a photo of it plugged into my iPod Shuffle:
For those unfamiliar with the Shuffle's size, the gray control clicker is about 7/8" in diameter, just about the size of a US quarter.
How well does it work? Very nicely. I find that listening with it is much more pleasant than listening without it. It seems to have just about the same effect as my AirHead's crossfeed circuit. I may not be the most discerning listener in the world, and someone else may be able to find a significant difference. But even if someone can, it would be very hard to beat the thing's portability and, at US$30 delivered to me, its price.
Of course, one of the things that makes the XinFeed so portable is that it doesn't run on batteries (it's "passive" as they say). That means that it's going to consume some of the signal in order to do its job. It doesn't eat much, but I found that when I was comparing the sound with it and without it, I needed about two clicks more volume out of my Shuffle to have the sound seem the same after I plugged it in. That means that if your music source is only barely able to drive your headphones satisfactorily, the XinFeed may not be a useful solution for you. Happily, my iPod Shuffle has plenty of power to drive my Sennheiser PX-100s and my Etymotic ER-4Ps even with the XinFeed plugged in. But if I add the cable-thingy that Etymotic sells to make their ER-4Ps work like ER-4Ss, it sounds like the Shuffle is running out of oomph with the XinFeed.
You could probably build something similar to the XinFeed yourself if you know a little about electronics. Googling "passive crossfeed circuit" yields plenty of circuit diagrams and a few pages with a bit more help than that. I know one end of a soldering iron from the other, but I was perfectly happy to pay Mr Xin Feng (who seems to be more or less all of XY Computing & Network) to build it for me. The result is nicer and smaller than anything I'd be able to build and probably cheaper too. Certainly if I valued my time at anything. And, who knows, his circuit may be better than the ones published on the net.
Mr Feng's site is more interesting than elegant, but given a choice between interesting and elegant, I'll take interesting any day.
In all, I'm thoroughly delighted with my XinFeed. When I take my standard iPod on a plane trip or my Shuffle out for a stroll, I'll no longer be thinking that the sound could be a lot more pleasing.
Crossfeed?
Yes: When you listen to stereo speakers, most of the right channel goes to your right ear, but some of it goes to your left ear, and vice-versa. Most music is recorded with the expectation that "air mixing" of that sort will happen when it's listened to. But there isn't normally any mixing when you're listening with headphones. The left channel is delivered only to your left ear and the right channel only to your right ear. A crossfeed circuit fixes that; it mixes some of the right channel into the left channel and vice-versa.
Actually, it can be a little more complicated than that. It seems that, on account of the shape of a person's head and ears, the crossfeed that a person perceives from air mixing varies some depending on a sound's frequency. It's straightforward enough to approximate that variation in a crossfeed circuit. Of course, as with many things in audio, it's possible to take matters to an extreme. (There's more at HeadRoom's admirably informative site.)
I'm no nutty audiophile, but after having listened to various things for some time with my AirHead's crossfeed circuit, I find it substantially more pleasant to listen to headphones with crossfeed than without. The sound seems more natural and it seems to be coming from in front of me rather than from beside me.
But what about portable music? It's perfectly possible to carry an AirHead around, and its four AAA batteries last an admirably long time. But it's about the size of a standard iPod, And so while you can carry it, I'm not likely to. And connecting one to an iPod Shuffle would be plain silly.
Happily, there's a solution to the portability problem in the form of XY Computing & Network's XinFeed (specifically, the low-impedance ampless version). The XinFeed is a crossfeed circuit built into a slightly lumpy and asymmetrical widget that's not a lot bigger than the 1/8" mini jack and plug that it has to have. Here's a photo of it plugged into my iPod Shuffle:
For those unfamiliar with the Shuffle's size, the gray control clicker is about 7/8" in diameter, just about the size of a US quarter.
How well does it work? Very nicely. I find that listening with it is much more pleasant than listening without it. It seems to have just about the same effect as my AirHead's crossfeed circuit. I may not be the most discerning listener in the world, and someone else may be able to find a significant difference. But even if someone can, it would be very hard to beat the thing's portability and, at US$30 delivered to me, its price.
Of course, one of the things that makes the XinFeed so portable is that it doesn't run on batteries (it's "passive" as they say). That means that it's going to consume some of the signal in order to do its job. It doesn't eat much, but I found that when I was comparing the sound with it and without it, I needed about two clicks more volume out of my Shuffle to have the sound seem the same after I plugged it in. That means that if your music source is only barely able to drive your headphones satisfactorily, the XinFeed may not be a useful solution for you. Happily, my iPod Shuffle has plenty of power to drive my Sennheiser PX-100s and my Etymotic ER-4Ps even with the XinFeed plugged in. But if I add the cable-thingy that Etymotic sells to make their ER-4Ps work like ER-4Ss, it sounds like the Shuffle is running out of oomph with the XinFeed.
You could probably build something similar to the XinFeed yourself if you know a little about electronics. Googling "passive crossfeed circuit" yields plenty of circuit diagrams and a few pages with a bit more help than that. I know one end of a soldering iron from the other, but I was perfectly happy to pay Mr Xin Feng (who seems to be more or less all of XY Computing & Network) to build it for me. The result is nicer and smaller than anything I'd be able to build and probably cheaper too. Certainly if I valued my time at anything. And, who knows, his circuit may be better than the ones published on the net.
Mr Feng's site is more interesting than elegant, but given a choice between interesting and elegant, I'll take interesting any day.
In all, I'm thoroughly delighted with my XinFeed. When I take my standard iPod on a plane trip or my Shuffle out for a stroll, I'll no longer be thinking that the sound could be a lot more pleasing.
Posted at 07:09 Main Permalink
Tue - October 26, 2004
Casio EX-S100 digital camera
The best way I can explain what happened to me
when I first
read about Casio's Exilim EX-S100 camera is to paraphrase Penny Arcade's guest columnist
Storm
Shadow and say that I missed my saving throw against cool gadgets.
It's not surprising that I missed; the camera is small and shiny and sleek and
has a really big LCD display. From the front, it's barely bigger than a credit
card and it's no thicker than a slice of toast. And it records 3.2m pixels. How
could any self-respecting gadget geek not be
entranced?
I did read a couple of reviews (1, 2) first, and if both of them had said "Turned our reviewer into a bug-eyed zombie" I probably would have thought twice about buying one. But neither did. And as it turns out, I'm very happy with the camera. I won't bother to repeat what's in those reviews, but there are a few things I think are worth mentioning.
Casio had to make a compromise or two in making the camera as small as they have. For one, there's no viewfinder. But I find that doesn't bother me. The LCD is big and comfortable to use and I'm sure that I'll eventually stop putting the camera up to my eye before remembering that that doesn't work.
The most significant compromises seem to be in the lens and the size of the image sensor. It seems that resolution suffers a bit. I bought the EX-S100 to replace my mildly-antique Canon Elph PowerShot S200, so that's what I have to compare it to. Here's a magnified detail from a shot with the Casio at its highest resolution:
And here's the same detail from the same scene shot a moment later with the Canon:
The Casio's image is bigger because it records more dots, but there's no more detail. (The Casio's color is a little more accurate.)
To go with detail that's not better than a 2.0m-dot camera, there's some image noise. The EX-S100's sensor is 1/3.2 (or in sane units about 4.5 x 3.4 mm) vs. 1/2.7 (5.3 x 4.0 mm) for the Canon. The Casio squeezes 3.2m pixels into that area while the Canon has only 2.0m on its larger sensor. That means that the individual photosites on the Casio are going to be considerably smaller. If I'm counting right, the Casio's dots are slightly less than half the size of the Canon's. That seems to cause a little trouble because the camera has to turn up the gain a fair amount in low-light situations. When the camera decides that it needs ISO 400 sensitivity, there's visible noise in the picture. Here are enlarged details of a shot of a blank wall shot at ISO 100, 200, and 400:
(The color varies a bit because I didn't bother to fiddle the white balance.) Noise is visible in the middle one, but it's pretty pronounced in the last. In practice, images taken at ISO 200 don't look noisy. At ISO 400, they do. Happily, it seems that the camera doesn't think it needs ISO 400 very often.
I think that the EX-S100 may be able to record a slightly larger range of brightnesses than the S200 can. But if that's true, the difference is marginal.
On the good side, Casio uses the big, clear LCD to provide a the user interface that's very clean. Indeed, it's as good as I've seen on a digital camera. And that's matched by control buttons that work nicely and feel good. And in addition, there's one feature that I haven't seen before and which I'm very glad to have: You can specify which settings revert to default after power-off and which are remembered. I like that because I think it's rude to take flash pictures in most public places. My Canon would enable automatic flash each time I switched it on. I'm sure that most people like that; they won't miss a photo if the light is dim. For me, it meant that I had to remember to switch the flash off if I didn't want it each time I switched the camera on. With the Casio, I can tell it to remember the last flash setting.
You use a cradle both to import photos and to charge the camera. You can set the camera to look like a USB mass-storage device when it's in the cradle and so it works just fine under OS X and ought to work equally wall under any sane operating system. The cradle and its wall-plug transformer aren't big or heavy, but people who want to travel as light as possible can buy a travel charger for the battery and an SD card-to-USB dongle. Interestingly, the connector that mates with the camera when it's in the cradle looks a lot like a USB Mini-B plug. I have one of those cables, but since nothing I've found in the documentation or elsewhere mentions USB Mini-B cables, I think I'll let someone else try plugging in the camera that way first. (If you do, I'd be glad to know what happens.)
The camera doesn't fit into Casio's "EXCASE3" very exactly:
Happily, the cooler Japanese cases seem to be available here in the US. I plan to order one soon.
In the end, I like the camera a lot. I could wish that there weren't any compromises involved in its engineering. And I could also wish for a pony. It works perceptibly better than my S200 and it's half the size of that small camera. That's quite good enough for me.
Update October 29, 2004:
My Japanese-market case arrived today and it's much nicer than the American-market one. I ordered the soft case in "ebony". The case that arrived is an attractive shade of very dark brown. The packaging is marked "cioccolato" and "chocoraato". The leather is pleasantly soft and it fits the camera snugly. Even in the case, the camera fits tidily in a shirt pocket. The case's flap is secured with the sort of magnetic disks you've almost certainly seen before. The camera's power switch is designed in such a way that it seems unlikely that it will get switched on accidentally. The case doesn't have a belt-loop or clip; it's suitable for protecting the camera in a pocket or a bag. That suits me just fine because I don't like carrying things on my belt. You may have a different opinion.
I did read a couple of reviews (1, 2) first, and if both of them had said "Turned our reviewer into a bug-eyed zombie" I probably would have thought twice about buying one. But neither did. And as it turns out, I'm very happy with the camera. I won't bother to repeat what's in those reviews, but there are a few things I think are worth mentioning.
Casio had to make a compromise or two in making the camera as small as they have. For one, there's no viewfinder. But I find that doesn't bother me. The LCD is big and comfortable to use and I'm sure that I'll eventually stop putting the camera up to my eye before remembering that that doesn't work.
The most significant compromises seem to be in the lens and the size of the image sensor. It seems that resolution suffers a bit. I bought the EX-S100 to replace my mildly-antique Canon Elph PowerShot S200, so that's what I have to compare it to. Here's a magnified detail from a shot with the Casio at its highest resolution:
And here's the same detail from the same scene shot a moment later with the Canon:
The Casio's image is bigger because it records more dots, but there's no more detail. (The Casio's color is a little more accurate.)
To go with detail that's not better than a 2.0m-dot camera, there's some image noise. The EX-S100's sensor is 1/3.2 (or in sane units about 4.5 x 3.4 mm) vs. 1/2.7 (5.3 x 4.0 mm) for the Canon. The Casio squeezes 3.2m pixels into that area while the Canon has only 2.0m on its larger sensor. That means that the individual photosites on the Casio are going to be considerably smaller. If I'm counting right, the Casio's dots are slightly less than half the size of the Canon's. That seems to cause a little trouble because the camera has to turn up the gain a fair amount in low-light situations. When the camera decides that it needs ISO 400 sensitivity, there's visible noise in the picture. Here are enlarged details of a shot of a blank wall shot at ISO 100, 200, and 400:
(The color varies a bit because I didn't bother to fiddle the white balance.) Noise is visible in the middle one, but it's pretty pronounced in the last. In practice, images taken at ISO 200 don't look noisy. At ISO 400, they do. Happily, it seems that the camera doesn't think it needs ISO 400 very often.
I think that the EX-S100 may be able to record a slightly larger range of brightnesses than the S200 can. But if that's true, the difference is marginal.
On the good side, Casio uses the big, clear LCD to provide a the user interface that's very clean. Indeed, it's as good as I've seen on a digital camera. And that's matched by control buttons that work nicely and feel good. And in addition, there's one feature that I haven't seen before and which I'm very glad to have: You can specify which settings revert to default after power-off and which are remembered. I like that because I think it's rude to take flash pictures in most public places. My Canon would enable automatic flash each time I switched it on. I'm sure that most people like that; they won't miss a photo if the light is dim. For me, it meant that I had to remember to switch the flash off if I didn't want it each time I switched the camera on. With the Casio, I can tell it to remember the last flash setting.
You use a cradle both to import photos and to charge the camera. You can set the camera to look like a USB mass-storage device when it's in the cradle and so it works just fine under OS X and ought to work equally wall under any sane operating system. The cradle and its wall-plug transformer aren't big or heavy, but people who want to travel as light as possible can buy a travel charger for the battery and an SD card-to-USB dongle. Interestingly, the connector that mates with the camera when it's in the cradle looks a lot like a USB Mini-B plug. I have one of those cables, but since nothing I've found in the documentation or elsewhere mentions USB Mini-B cables, I think I'll let someone else try plugging in the camera that way first. (If you do, I'd be glad to know what happens.)
The camera doesn't fit into Casio's "EXCASE3" very exactly:
Happily, the cooler Japanese cases seem to be available here in the US. I plan to order one soon.
In the end, I like the camera a lot. I could wish that there weren't any compromises involved in its engineering. And I could also wish for a pony. It works perceptibly better than my S200 and it's half the size of that small camera. That's quite good enough for me.
Update October 29, 2004:
My Japanese-market case arrived today and it's much nicer than the American-market one. I ordered the soft case in "ebony". The case that arrived is an attractive shade of very dark brown. The packaging is marked "cioccolato" and "chocoraato". The leather is pleasantly soft and it fits the camera snugly. Even in the case, the camera fits tidily in a shirt pocket. The case's flap is secured with the sort of magnetic disks you've almost certainly seen before. The camera's power switch is designed in such a way that it seems unlikely that it will get switched on accidentally. The case doesn't have a belt-loop or clip; it's suitable for protecting the camera in a pocket or a bag. That suits me just fine because I don't like carrying things on my belt. You may have a different opinion.
Posted at 04:27 Main Permalink
Thu - September 2, 2004
Book: Network Security Hacks by Andrew Lockhart
Andrew
Lockhart
Network Security Hacks: 100 Industrial-Strength Tips & Tools
O'Reilly, 2004
ISBN: 0-596-00643-8
$24.95
280 pages (main text)
As you can tell from this book's subtitle, the word "hacks" in the title Network Security Hacks is used in one of its original senses: a good or ingenious idea. This book isn't a catalog of computer break-ins.
Each of the 100 hacks is described in a short section with a title, a one-sentence description, and generally a couple of pages of discussion. The hacks are organized into eight chapters:
Unix Host Security
Windows Host Security
Network Security
Logging
Monitoring and Trending
Secure Tunnels
Network Intrusion Detection
Recovery and Response
The hacks aren't evenly divided among chapters; "Monitoring and Trending" is pretty short and "Network Security" is pretty long. Each chapter has an introduction that isn't very interesting ("In this chapter, you'll learn....").
The hacks themselves are pretty good. I'm not qualified to comment on the ones that have to do with Windows, but on Unix and network security there's plenty of good sense here. The discussions are of varying value. That is, once you've said "Run ntop for Real-Time Network Stats" (hack 63), someone with clues (and this book is addressed to people with geeky clues) probably doesn't need a lot more help. I mean, you'd Google for ntop's site and read the documentation there to see if it does something you'd find useful. On the other hand, the discussions for "Firewall with Netfilter" (hack 33) and "Firewall with OpenBSD's PacketFilter" (hack 34) have useful examples of the sorts of rules you'd want to run on a firewall host. Most of the discussions are useful but few are vital.
One thing that's missing is any indication of why you would or wouldn't want to use a given hack. That is, "Test Your Firewall" (hack 38) probably makes sense for any network admin. But "Create a Static ARP Table" (hack 32) would be a big nuisance on any but the smallest networks. I'd need to be pretty scared of ARP-table poisoning attacks before I went to that much trouble. The book is a toolbox, not a tutorial.
If these 100 hacks here were made available as a list of possibly-useful security practices for free on a website, that list would be less useful than this $25 book. But maybe not a whole lot less useful. Still, they're not available that way and some of the discussions are quite good.
Neither of my two favorite security hacks is mentioned. The first was told to me and I haven't yet used it, but I expect to eventually. It is: Use a dedicated log host and cut its transmit pair. The second is from my own experience: Don't run Sendmail, BIND, rsync, or Kerberos. And try really hard not to run any IMAP server or an FTP server that allows non-anonymous logins.
Network Security Hacks: 100 Industrial-Strength Tips & Tools
O'Reilly, 2004
ISBN: 0-596-00643-8
$24.95
280 pages (main text)
As you can tell from this book's subtitle, the word "hacks" in the title Network Security Hacks is used in one of its original senses: a good or ingenious idea. This book isn't a catalog of computer break-ins.
Each of the 100 hacks is described in a short section with a title, a one-sentence description, and generally a couple of pages of discussion. The hacks are organized into eight chapters:
Unix Host Security
Windows Host Security
Network Security
Logging
Monitoring and Trending
Secure Tunnels
Network Intrusion Detection
Recovery and Response
The hacks aren't evenly divided among chapters; "Monitoring and Trending" is pretty short and "Network Security" is pretty long. Each chapter has an introduction that isn't very interesting ("In this chapter, you'll learn....").
The hacks themselves are pretty good. I'm not qualified to comment on the ones that have to do with Windows, but on Unix and network security there's plenty of good sense here. The discussions are of varying value. That is, once you've said "Run ntop for Real-Time Network Stats" (hack 63), someone with clues (and this book is addressed to people with geeky clues) probably doesn't need a lot more help. I mean, you'd Google for ntop's site and read the documentation there to see if it does something you'd find useful. On the other hand, the discussions for "Firewall with Netfilter" (hack 33) and "Firewall with OpenBSD's PacketFilter" (hack 34) have useful examples of the sorts of rules you'd want to run on a firewall host. Most of the discussions are useful but few are vital.
One thing that's missing is any indication of why you would or wouldn't want to use a given hack. That is, "Test Your Firewall" (hack 38) probably makes sense for any network admin. But "Create a Static ARP Table" (hack 32) would be a big nuisance on any but the smallest networks. I'd need to be pretty scared of ARP-table poisoning attacks before I went to that much trouble. The book is a toolbox, not a tutorial.
If these 100 hacks here were made available as a list of possibly-useful security practices for free on a website, that list would be less useful than this $25 book. But maybe not a whole lot less useful. Still, they're not available that way and some of the discussions are quite good.
Neither of my two favorite security hacks is mentioned. The first was told to me and I haven't yet used it, but I expect to eventually. It is: Use a dedicated log host and cut its transmit pair. The second is from my own experience: Don't run Sendmail, BIND, rsync, or Kerberos. And try really hard not to run any IMAP server or an FTP server that allows non-anonymous logins.
Posted at 08:16 Main Permalink
Sat - August 14, 2004
Mini WiFi access points
I don't travel all that much, but in the next few
months my iBook and I are likely to travel to a couple of places where there's
high-speed internet access, but where the Ethernet jacks aren't necessarily in
the places most convenient for me. So it would be nice to bring along an 802.11g
wireless access point so that I could use my iBook in a convenient spot. A
Linksys WRT54G
access-point with appropriately hacked firmware is
unquestionably cool, but I hoped to find something a bit friendlier to my
carry-on bag.
Apple's AirPort Express looked like just the thing and I promptly got one. How well does it work? Heck if I know. You see, you need OS X 10.3 (the latest version of OS X) in order to configure it and my iBoork runs OS X 10.2. I'm not going to upgrade my iBook's OS right now because dong that would break too many things. (It seems that you can use various flavors of Windows, but only one flavor of OS X to do the configuration.) I have no idea what the people at Apple were thinking when they decided that you'd have to use an OS version that's less than a year old in order to make the AirPort Express work. I'm pretty sure that when I ordered the AirPort Express, Apple's website said that under OS X 10.2 the thing had "limited functionality". I interpreted that to mean that the music-streaming and/or printer-sharing features probably wouldn't work. I didn't interpret it to mean that I'd be completely SOL because I couldn't even configure the thing.
Yes, I could have borrowed a 10.3 machine to do the configuration. But the idea is to have a portable access-point and I'm quite sure that if I took an access-point that I couldn't reconfigure on a trip, something would require that I reconfigure it.
OK, fine, then. Does anyone else have something similar? It turns out that they do. Tom's Networking had a review of the ASUS WL-330g and they rather liked it. So I ordered one.
The WL-330g is undeniably small. Here's a photo of it, its AC adaptor, and a ballpoint pen for scale:
Taken together with its small AC adaptor, it's about the same size as an AirPort Express, though not nearly as attractively designed. For some reason that I can't guess, ASUS decided to use high-intensity blue LEDs for the status LEDs. They're bright enough and blinky enough that they'll be distracting if they're in your line of sight. I can live with all that, but we're clearly not talking about Apple-like attention to design.
Configuration was easy. (Kind of ironic, huh?) There's a configuration utility that I didn't use because I'm sure it runs only under Windows. Instead I did the configuration by web browser. The WL-330g has a tiny webserver in it and its default address is 192.168.1.1. So all you have to do is turn on your machine's 802.11b/g interface, manually assign it an IP address and subnet mask on that puts you on the WL-330g's network (such as 192.168.1.2 and 255.255.255.0), and point a web browser at http://192.168.1.1. The configuration pages aren't very pretty but they're straightforward to use. If you get something badly wrong, just press the reset button for ten seconds or so and the defaults are restored.
Of course, one of the ways that the WL-330g manages to be compact is that it doesn't have an external antenna (or indeed a jack for one). So it's going to be important that its internal antennas give it reasonable range. For that reason, I conducted a very scientific experiment. I put the WL-330g on top of a desk in a Mark A1 standard suburban office building. To ensure randomness, I paid no attention at all to its orientation. I then picked up my iBook and walked as far away from the access-point as possible while staying in the building. From there, the access-point was on the other side of a wall, around 120 feet away. The signal strength shown in my iBook's menu bar had gone down one bar (actually it's an arc) and Apple's Internet Connect showed about half signal strength. In keeping with the spirit of the experiment, I didn't do any detailed throughput experiments, but web pages loaded at what looked like the full speed of the 640kbps connection there.
So color me thoroughly satisfied; at least as satisfied as I can be without having used it "in anger". At $75 from NewEgg, I can't see anything to complain about in this handy little gadget.
The WL-330g can also be used as a client. (There's a switch on the bottom to change modes.) That might be useful in some conceivable circumstances, but I haven't tried it because I don't think I'm likely to run into any of those circumstances. There's a cable in the box that allows the WL-330g to run from the power available on a USB port, which might be useful if you were using it as a client. The quick-start guide is reasonably good and there's a carrying case and a short Cat-5 cable in the box. As for the carrying case, I continue to prefer AeroStich's nylon envelope bags.
Update: September 2, 2004
The WL-330g worked very well in practice. In an ordinary house, I plugged it into an unused cable-modem port and I got excellent signal strength 20 meters or so away. Or rather, I did after I realized that the switch on the thing's bottom that's labeled Access Point/Ethernet Adaptor had gotten moved by accident while it was in my luggage. It seems that it may not have been the best idea for ASUS to put that switch on the bottom since I didn't notice it or think of it for the first ten minutes of troubleshooting. The switch has a ridge around it that's intended to prevent it from being moved accidentally, but it's obviously not completely effective when the thing is packed in a suitcase.
After I thought of the switch on the bottom, I found that it's much more reliable to configure the WL-330g by wired Ethernet than by its radio link. After the configuration is done, the radio link is perfectly reliable. But when I was applying the various configuration options by radio, the link to my iBook would often drop and when I reconnected, the options generally hadn't been applied. Plugging the WL-330g's Ethernet cable into my iBook solved the problem. In that process, I also found that it may be desirable to configure the WL-330g to use an IP other than the default of 192.168.1.1 since it's quite possible that whatever you to plug it into is already using that address because that is its default. I set the WL-330g to use 192.168.1.150 which I judged likely to be outside the range that the network's DHCP server was using and everything was fine.
Apple's AirPort Express looked like just the thing and I promptly got one. How well does it work? Heck if I know. You see, you need OS X 10.3 (the latest version of OS X) in order to configure it and my iBoork runs OS X 10.2. I'm not going to upgrade my iBook's OS right now because dong that would break too many things. (It seems that you can use various flavors of Windows, but only one flavor of OS X to do the configuration.) I have no idea what the people at Apple were thinking when they decided that you'd have to use an OS version that's less than a year old in order to make the AirPort Express work. I'm pretty sure that when I ordered the AirPort Express, Apple's website said that under OS X 10.2 the thing had "limited functionality". I interpreted that to mean that the music-streaming and/or printer-sharing features probably wouldn't work. I didn't interpret it to mean that I'd be completely SOL because I couldn't even configure the thing.
Yes, I could have borrowed a 10.3 machine to do the configuration. But the idea is to have a portable access-point and I'm quite sure that if I took an access-point that I couldn't reconfigure on a trip, something would require that I reconfigure it.
OK, fine, then. Does anyone else have something similar? It turns out that they do. Tom's Networking had a review of the ASUS WL-330g and they rather liked it. So I ordered one.
The WL-330g is undeniably small. Here's a photo of it, its AC adaptor, and a ballpoint pen for scale:
Taken together with its small AC adaptor, it's about the same size as an AirPort Express, though not nearly as attractively designed. For some reason that I can't guess, ASUS decided to use high-intensity blue LEDs for the status LEDs. They're bright enough and blinky enough that they'll be distracting if they're in your line of sight. I can live with all that, but we're clearly not talking about Apple-like attention to design.
Configuration was easy. (Kind of ironic, huh?) There's a configuration utility that I didn't use because I'm sure it runs only under Windows. Instead I did the configuration by web browser. The WL-330g has a tiny webserver in it and its default address is 192.168.1.1. So all you have to do is turn on your machine's 802.11b/g interface, manually assign it an IP address and subnet mask on that puts you on the WL-330g's network (such as 192.168.1.2 and 255.255.255.0), and point a web browser at http://192.168.1.1. The configuration pages aren't very pretty but they're straightforward to use. If you get something badly wrong, just press the reset button for ten seconds or so and the defaults are restored.
Of course, one of the ways that the WL-330g manages to be compact is that it doesn't have an external antenna (or indeed a jack for one). So it's going to be important that its internal antennas give it reasonable range. For that reason, I conducted a very scientific experiment. I put the WL-330g on top of a desk in a Mark A1 standard suburban office building. To ensure randomness, I paid no attention at all to its orientation. I then picked up my iBook and walked as far away from the access-point as possible while staying in the building. From there, the access-point was on the other side of a wall, around 120 feet away. The signal strength shown in my iBook's menu bar had gone down one bar (actually it's an arc) and Apple's Internet Connect showed about half signal strength. In keeping with the spirit of the experiment, I didn't do any detailed throughput experiments, but web pages loaded at what looked like the full speed of the 640kbps connection there.
So color me thoroughly satisfied; at least as satisfied as I can be without having used it "in anger". At $75 from NewEgg, I can't see anything to complain about in this handy little gadget.
The WL-330g can also be used as a client. (There's a switch on the bottom to change modes.) That might be useful in some conceivable circumstances, but I haven't tried it because I don't think I'm likely to run into any of those circumstances. There's a cable in the box that allows the WL-330g to run from the power available on a USB port, which might be useful if you were using it as a client. The quick-start guide is reasonably good and there's a carrying case and a short Cat-5 cable in the box. As for the carrying case, I continue to prefer AeroStich's nylon envelope bags.
Update: September 2, 2004
The WL-330g worked very well in practice. In an ordinary house, I plugged it into an unused cable-modem port and I got excellent signal strength 20 meters or so away. Or rather, I did after I realized that the switch on the thing's bottom that's labeled Access Point/Ethernet Adaptor had gotten moved by accident while it was in my luggage. It seems that it may not have been the best idea for ASUS to put that switch on the bottom since I didn't notice it or think of it for the first ten minutes of troubleshooting. The switch has a ridge around it that's intended to prevent it from being moved accidentally, but it's obviously not completely effective when the thing is packed in a suitcase.
After I thought of the switch on the bottom, I found that it's much more reliable to configure the WL-330g by wired Ethernet than by its radio link. After the configuration is done, the radio link is perfectly reliable. But when I was applying the various configuration options by radio, the link to my iBook would often drop and when I reconnected, the options generally hadn't been applied. Plugging the WL-330g's Ethernet cable into my iBook solved the problem. In that process, I also found that it may be desirable to configure the WL-330g to use an IP other than the default of 192.168.1.1 since it's quite possible that whatever you to plug it into is already using that address because that is its default. I set the WL-330g to use 192.168.1.150 which I judged likely to be outside the range that the network's DHCP server was using and everything was fine.
Posted at 06:19 Main Permalink
Sun - June 27, 2004
A patch for SpamBayes to record URLs' IPs and a cache for PyDNS
Non-geeks and geeks not interested in the
details of Bayesian spam filtering may prefer to skip this
post.
A while ago I mentioned the spam filter SpamBayes. I've used it almost from the beginning and it works very well for me.
Starting early this year, I found that spammers had begun sending messages with bland or almost entirely nonsense text and a link to click on. SpamBayes would generally score them as unsure because they contained so little information that it could make use of. (In his original article, Paul Graham predicted that spammers would respond that way to the widespread use of Bayesian filters.)
Turning on SpamBayes's mine_received_headers option helps, but not enough in my experience. Especially if you have any legitimate correspondents on Comcast's network.
In April, I posted a patch for SpamBayes's tokenizer to the spambayes-dev list that creates synthetic tokens for the IP addresses that the host part of the URLs in a message resolve to. That turns out to help a lot on those messages. That's because the IPs of spammers' webservers aren't uniformly distributed. Indeed, it seems that there are relatively few networks that are willing to host spammers' websites and it doesn't take very long for SpamBayes to start using those tokens as evidence.
At first, it didn't seem to help. At least not much. It even produced a small decrease in accuracy in some cases. But that was on historical data. On more recent data, it's a significant win for me.
If you're doing a lot of scoring all at once (as you might with certain training regimes), doing lookups that way generates a lot of DNS traffic. Unless your resolving DNS server is electronically very close to you (like on the same Ethernet segment), that's going to slow scoring down a fair amount. Depending on the details of your situation, it may also be a significant load on your (or your ISP's) DNS server. To deal with that, I've hacked up a cache for PyDNS and a slightly different version of the patch. (With the new version, the clue "timeout" is now a slight misnomer. A better name would be something more generic like "error", but I've left it as it was for compatibility with the data in my database.)
By default, the cache respects the time-to-live of the data returned by the resolving name server it uses. The resolving name server component of D.J. Bernstein's djbdns returns TTLs of zero under most circumstances. Probably some others do too. Dan explains that that's to prevent cache snooping. If the cache doesn't seem to speed scoring up, you can set its attribute printStatsAtEnd to see if you're getting any cache hits. If low TTLs turn out to be the problem, you can set the attribute minTTL to 300 or 600 seconds or something harmlessly small like that and the cache will cache everything for at least that long.
A while ago I mentioned the spam filter SpamBayes. I've used it almost from the beginning and it works very well for me.
Starting early this year, I found that spammers had begun sending messages with bland or almost entirely nonsense text and a link to click on. SpamBayes would generally score them as unsure because they contained so little information that it could make use of. (In his original article, Paul Graham predicted that spammers would respond that way to the widespread use of Bayesian filters.)
Turning on SpamBayes's mine_received_headers option helps, but not enough in my experience. Especially if you have any legitimate correspondents on Comcast's network.
In April, I posted a patch for SpamBayes's tokenizer to the spambayes-dev list that creates synthetic tokens for the IP addresses that the host part of the URLs in a message resolve to. That turns out to help a lot on those messages. That's because the IPs of spammers' webservers aren't uniformly distributed. Indeed, it seems that there are relatively few networks that are willing to host spammers' websites and it doesn't take very long for SpamBayes to start using those tokens as evidence.
At first, it didn't seem to help. At least not much. It even produced a small decrease in accuracy in some cases. But that was on historical data. On more recent data, it's a significant win for me.
If you're doing a lot of scoring all at once (as you might with certain training regimes), doing lookups that way generates a lot of DNS traffic. Unless your resolving DNS server is electronically very close to you (like on the same Ethernet segment), that's going to slow scoring down a fair amount. Depending on the details of your situation, it may also be a significant load on your (or your ISP's) DNS server. To deal with that, I've hacked up a cache for PyDNS and a slightly different version of the patch. (With the new version, the clue "timeout" is now a slight misnomer. A better name would be something more generic like "error", but I've left it as it was for compatibility with the data in my database.)
By default, the cache respects the time-to-live of the data returned by the resolving name server it uses. The resolving name server component of D.J. Bernstein's djbdns returns TTLs of zero under most circumstances. Probably some others do too. Dan explains that that's to prevent cache snooping. If the cache doesn't seem to speed scoring up, you can set its attribute printStatsAtEnd to see if you're getting any cache hits. If low TTLs turn out to be the problem, you can set the attribute minTTL to 300 or 600 seconds or something harmlessly small like that and the cache will cache everything for at least that long.
Posted at 08:46 Main Permalink
Wed - May 19, 2004
Headphone amplifier and audio-image enhancer
I don't mean to be an audio geek. I blame Steve
Jobs for it. I was perfectly happy just being a coder and sysadmin geek. But,
you see, an OS X laptop is really good for a Unix coder geek. And it's hard or
maybe impossible to have an OS X machine and not subsequently be sucked into
getting a cool iPod. I call it the iPod
tax.
That's the beginning.
It seems that in order to sound at all good, earbuds need to fit your ears correctly. The earbuds that ship with the iPod sound somewhere between OK and lousy depending how they fit your ears. It seems that my ears (more exactly, ear canals) are bigger than average (no doubt to match my big mouth) so I fall on the lousy end of that spectrum. But that's easily remedied by buying a pretty-inexpensive set of headphones. And maybe a not-so-inexpensive set of earbuds that sound very good indeed and block a lot of external sound, for when that's desirable. And maybe a set of powered speakers since the iPod is more convenient to use than a CD changer and also sounds better through them than my unremarkable stereo sounds through its speakers. And if you're going to listen to an iPod that way, you might want a remote control.
Whew! Well, at least we're done now and can close the chapter on audio geekery.
Um, maybe not. You see, there's something called a headphone amplifier. On the face of it, that sounds like the silliest thing ever. I mean, how much power can it take to drive a pair of headphones? And wouldn't every manufacturer of something with a headphone jack put at least that much power on the jack? As it turns out, not a lot of power and, even so, no they don't. There's no shortage of headphones that require more power than is available on many headphone jacks, especially headphone jacks on portable players. Dan over at Dan's Data has a cool review of a hand-built headphone amplifier and more background on the subject.
But I don't actually have that problem. The headphones I have can easily be driven by the power available on my iPod's and iBook's headphone jacks.
"Sowhat'syerprollem?" I hear you ask. Well, it has to do with "imaging". It's like this: Almost all recordings are made with the idea that you'll listen to them using stereo speakers. With speakers, you get some crossover; you hear the left channel some with your right ear and vice-versa. Feeding the left channel only to your left ear and the right channel only to your right ear, as headphones do, is apt to make what you're listening to sound at least a little funny. The sound that OS X makes when you drop a file in the trash is an extreme example. On speakers it sounds fairly well like something bouncing between the sides of a trash basket that's in front of you. With headphones, it sounds like a ping-pong game.
A good audio-image enhancer for headphones doesn't just mix the channels. It also delays very slightly the arrival of the portion of each signal that it sends to the opposite ear, since that's what happens in real life. The appropriate delay is a small portion of a millisecond, but you still can hear it, albeit not consciously. A really good audio-image processor will also vary the mixing depending on the frequency, since that happens in real life too. There's more detail about that on HeadRoom's admirably informative site. Follow the links on the left margin ("How We Hear", etc) for the other pages of the article. HeadWize also has more on the subject.
As with many things in audio, you can take price and even, perhaps, quality to absurd heights. (That's one reason that I'm glad to be a digital geek: with bits, the engineers know when the circuit is done because the bits come out right; with analog circuits, you can spend as much time and money as you like improving the waveform just a little more.)
So I was chiefly looking for something to do audio-image enhancement. There's no important reason that it's necessary to combine the imaging function with the amplification function, but it's convenient to. And there's always the possibility that I'll want the amplification function later.
So I got a HeadRoom AirHead. Actually, I swallowed their marketing pitch and got the Total AirHead version. And it sounds very nice. When I plug my headphones into it and switch on the crossover circuit, the apparent sources of sound migrate from the sides of my head to places in front of me. Depending on what I'm listening to, the difference may be subtle, but in all cases it's significantly more pleasant in my opinion. As for amplification, the AirHead appears to be able to do plenty.
The thing sounds very good. The way it looks is, um, another story. If you put it next to an iPod, I suspect that the best thing you'd say is, "At least it's black".
You can see the input from the AC adapter (sold separately) on the left of the photo, line-in at the top, a mini-plug plugged into one of the headphone output jacks on the right, and a second headphone jack unused at the bottom. It looks like the AC adapter isn't plugged all the way in, but that's as far in as it will go. The green LED indicates that it's switched on and the red one indicates when it doesn't have as much power as the volume level requires (most likely because the batteries are running low). Between them is a thumbwheel for volume.
I can't yet verify HeadRoom's claim that it will run for 40 hours on a set of batteries (it takes 4 AAAs). Indeed, I'm also not yet sure that I'll take it with me when I travel. It's just about the same size and weight as an iPod and it's not immediately clear to me that the improvement it produces would be worth carrying it with me. Clipping my iPod's case to the pouch on the back of the seat in front of me in an airliner is easy enough. I'm not sure that cabling an AirHead to it and stuffing it in there would be all that much fun. Time will have to tell on both those counts. Nevertheless, for headphone listening at home, I'm very pleased with my AirHead.
Coincidentally, Dan has a review of a related product from the same company just now on his site.
I promise that I am not going to become one of those people who think that cables need to be broken in before they will conduct electricity well.
That's the beginning.
It seems that in order to sound at all good, earbuds need to fit your ears correctly. The earbuds that ship with the iPod sound somewhere between OK and lousy depending how they fit your ears. It seems that my ears (more exactly, ear canals) are bigger than average (no doubt to match my big mouth) so I fall on the lousy end of that spectrum. But that's easily remedied by buying a pretty-inexpensive set of headphones. And maybe a not-so-inexpensive set of earbuds that sound very good indeed and block a lot of external sound, for when that's desirable. And maybe a set of powered speakers since the iPod is more convenient to use than a CD changer and also sounds better through them than my unremarkable stereo sounds through its speakers. And if you're going to listen to an iPod that way, you might want a remote control.
Whew! Well, at least we're done now and can close the chapter on audio geekery.
Um, maybe not. You see, there's something called a headphone amplifier. On the face of it, that sounds like the silliest thing ever. I mean, how much power can it take to drive a pair of headphones? And wouldn't every manufacturer of something with a headphone jack put at least that much power on the jack? As it turns out, not a lot of power and, even so, no they don't. There's no shortage of headphones that require more power than is available on many headphone jacks, especially headphone jacks on portable players. Dan over at Dan's Data has a cool review of a hand-built headphone amplifier and more background on the subject.
But I don't actually have that problem. The headphones I have can easily be driven by the power available on my iPod's and iBook's headphone jacks.
"Sowhat'syerprollem?" I hear you ask. Well, it has to do with "imaging". It's like this: Almost all recordings are made with the idea that you'll listen to them using stereo speakers. With speakers, you get some crossover; you hear the left channel some with your right ear and vice-versa. Feeding the left channel only to your left ear and the right channel only to your right ear, as headphones do, is apt to make what you're listening to sound at least a little funny. The sound that OS X makes when you drop a file in the trash is an extreme example. On speakers it sounds fairly well like something bouncing between the sides of a trash basket that's in front of you. With headphones, it sounds like a ping-pong game.
A good audio-image enhancer for headphones doesn't just mix the channels. It also delays very slightly the arrival of the portion of each signal that it sends to the opposite ear, since that's what happens in real life. The appropriate delay is a small portion of a millisecond, but you still can hear it, albeit not consciously. A really good audio-image processor will also vary the mixing depending on the frequency, since that happens in real life too. There's more detail about that on HeadRoom's admirably informative site. Follow the links on the left margin ("How We Hear", etc) for the other pages of the article. HeadWize also has more on the subject.
As with many things in audio, you can take price and even, perhaps, quality to absurd heights. (That's one reason that I'm glad to be a digital geek: with bits, the engineers know when the circuit is done because the bits come out right; with analog circuits, you can spend as much time and money as you like improving the waveform just a little more.)
So I was chiefly looking for something to do audio-image enhancement. There's no important reason that it's necessary to combine the imaging function with the amplification function, but it's convenient to. And there's always the possibility that I'll want the amplification function later.
So I got a HeadRoom AirHead. Actually, I swallowed their marketing pitch and got the Total AirHead version. And it sounds very nice. When I plug my headphones into it and switch on the crossover circuit, the apparent sources of sound migrate from the sides of my head to places in front of me. Depending on what I'm listening to, the difference may be subtle, but in all cases it's significantly more pleasant in my opinion. As for amplification, the AirHead appears to be able to do plenty.
The thing sounds very good. The way it looks is, um, another story. If you put it next to an iPod, I suspect that the best thing you'd say is, "At least it's black".
You can see the input from the AC adapter (sold separately) on the left of the photo, line-in at the top, a mini-plug plugged into one of the headphone output jacks on the right, and a second headphone jack unused at the bottom. It looks like the AC adapter isn't plugged all the way in, but that's as far in as it will go. The green LED indicates that it's switched on and the red one indicates when it doesn't have as much power as the volume level requires (most likely because the batteries are running low). Between them is a thumbwheel for volume.
I can't yet verify HeadRoom's claim that it will run for 40 hours on a set of batteries (it takes 4 AAAs). Indeed, I'm also not yet sure that I'll take it with me when I travel. It's just about the same size and weight as an iPod and it's not immediately clear to me that the improvement it produces would be worth carrying it with me. Clipping my iPod's case to the pouch on the back of the seat in front of me in an airliner is easy enough. I'm not sure that cabling an AirHead to it and stuffing it in there would be all that much fun. Time will have to tell on both those counts. Nevertheless, for headphone listening at home, I'm very pleased with my AirHead.
Coincidentally, Dan has a review of a related product from the same company just now on his site.
I promise that I am not going to become one of those people who think that cables need to be broken in before they will conduct electricity well.
Posted at 03:51 Main Permalink
Fri - March 5, 2004
Headphones for my iPod
A while ago I mentioned
how much I like the Etymotic ER-4P earplug-style earbuds that I use with my
iPod. My opinion of them hasn't changed a bit since then: they're fabulous. On
the last flight I took, I was seated in among two families that had a total of
eight children. A baby in the row behind me spent most of the flight crying, a
teenager across the aisle watched a portable DVD player without headphones, and
a youngster to my left played a portable videogame without headphones. I put my
ER-4Ps in my ears, switched my iPod on, and had a pleasant
flight.
But the sound-isolation that makes the ER-4Ps so nice in situations like that isn't always desirable. If I were to use them while I was sitting in a departure lounge, I might well miss my flight being called. Eric Rescorla has remarked on the same thing, and he's talking about ER-6s which provide slightly less isolation. When I needed to be able to hear things over my iPod, I used to use Apple's cool-looking white earbuds. But after listening with my ER-4Ps and my Monsoon speakers (which I also mention in the post linked above), I decided that it ought to be possible to find something that sounds better than Apple's earbuds but doesn't provide much isolation and is easily portable.
I found HeadRoom's site to be admirably informative and I judged from what they have to say that Sennheiser PX-100s ought to be suitable. And at $40 from HeadRoom, there's nothing to complain about in the price. Still, I was a bit leery. Would $40 headphones be any good? So I emailed their sales manager, explaining what I was looking for, and asked if the PX-100s were what he'd recommend, or if I should spend a bit more because something else would likely suit me better. He mailed back quite quickly and said that the PX-100s were what he'd recommend for me. In my experience, if a someone in sales doesn't recommend that you spend more money, you've found someone who really knows their products and is interested in making their customers happy. Naturally, I ordered the PX-100s from them.
And I think that the PX-100s sound very nice indeed. They sound a lot like my Monsoon speakers. They're certainly a considerable improvement over Apple's earbuds. Next to the PX-100s, Apple's earbuds sound thin and lacking in bass. The PX-100s aren't as good as the ER-4Ps, but it would be silly to expect them to be.
In addition to sounding good, the PX-100s are small and light and quite comfortable. The headband has two hinges and the earpads rotate 90 degrees, so they fold up like a pair of glasses. They come in a hard-plastic case which some folks may like but I think is too clever by half. If I want to put things in my shoulder-bag and have them not rattle around, I prefer the nylon envelope bags from Aerostich. Velcro wire-ties solve the snarled-cable problem.
Update July 8, 2004
Dan over at Dan's Data also likes the PX-100s.
But the sound-isolation that makes the ER-4Ps so nice in situations like that isn't always desirable. If I were to use them while I was sitting in a departure lounge, I might well miss my flight being called. Eric Rescorla has remarked on the same thing, and he's talking about ER-6s which provide slightly less isolation. When I needed to be able to hear things over my iPod, I used to use Apple's cool-looking white earbuds. But after listening with my ER-4Ps and my Monsoon speakers (which I also mention in the post linked above), I decided that it ought to be possible to find something that sounds better than Apple's earbuds but doesn't provide much isolation and is easily portable.
I found HeadRoom's site to be admirably informative and I judged from what they have to say that Sennheiser PX-100s ought to be suitable. And at $40 from HeadRoom, there's nothing to complain about in the price. Still, I was a bit leery. Would $40 headphones be any good? So I emailed their sales manager, explaining what I was looking for, and asked if the PX-100s were what he'd recommend, or if I should spend a bit more because something else would likely suit me better. He mailed back quite quickly and said that the PX-100s were what he'd recommend for me. In my experience, if a someone in sales doesn't recommend that you spend more money, you've found someone who really knows their products and is interested in making their customers happy. Naturally, I ordered the PX-100s from them.
And I think that the PX-100s sound very nice indeed. They sound a lot like my Monsoon speakers. They're certainly a considerable improvement over Apple's earbuds. Next to the PX-100s, Apple's earbuds sound thin and lacking in bass. The PX-100s aren't as good as the ER-4Ps, but it would be silly to expect them to be.
In addition to sounding good, the PX-100s are small and light and quite comfortable. The headband has two hinges and the earpads rotate 90 degrees, so they fold up like a pair of glasses. They come in a hard-plastic case which some folks may like but I think is too clever by half. If I want to put things in my shoulder-bag and have them not rattle around, I prefer the nylon envelope bags from Aerostich. Velcro wire-ties solve the snarled-cable problem.
Update July 8, 2004
Dan over at Dan's Data also likes the PX-100s.
Posted at 03:29 Main Permalink
Sun - December 21, 2003
Voice-over-IP is interesting but not in the way most people seem to think
It's a tempting thought: In most offices and many
homes there are two networks, one for voice and one for data. Why not run voice
over the data network? In a lot of places, including here in Minneapolis, it
looks like that would be convenient and cheap. It also doesn't look very hard.
Voice is easily digitized and compressed. Your cell phone does it and even
land-line telephone companies routinely carry voice over Asynchronous Transfer
Mode networks. (ATM networks are data networks where the packets are 53 bytes
long and are called cells.) The highest data rate that's ever used for voice is
64 Kbps and that wouldn't make much of a dent in my DSL line's 256 Kbps upload
speed. In practice you could use much a much lower
rate.
So why aren't we all using phones that have Ethernet ports to make calls that don't cost us anything beyond our monthly fee for internet service? Well, actually some people are. And telephone companies are understandably somewhat concerned about losing voice revenue. But there's a little more to it than that. The problem is that we have different expectations for voice networks and data networks. If an email were delayed by three seconds, I wouldn't notice or care. But if my "Hello" were delayed by three seconds, it would be an awkward way to start a conversation. I'm no fan of giant telephone companies but at least part of the reason that voice is expensive and data is cheap (where that's true) comes from the different expectations we have for those networks.
In addition to expecting little latency when we send voice, we also expect our voice networks to be extremely reliable. I'm pretty sure that that annoying "five nines" thing started with telephone companies. If any telephone company actually achieves 99.999% reliability (that's about 5 1/4 minutes of downtime per year) by any meaningful calculation it's news to me. Still, I have to admit that they're much more reliable than even quite good data networks.
Beyond low latency and high reliability, voice networks also have lots of capacity. It's often the case that something causes lots of people to want to make telephone calls all at the same time. Whether it's a holiday or a disaster, it's very possible that I can have a particularly strong desire to make a telephone call just at the time that everyone else does. As Steven den Beste has observed, building capacity that goes unused almost all of the time is expensive, but the capacity of the voice network around here is great enough that I don't remember the last time I got a fast busy signal or an "all circuits busy" recording.
My ridiculously cheap data network can be laggy, it goes down a couple of times a year, and I routinely run into its capacity limits. And that's just fine. We use data networks in ways that mean we can tolerate those things. The interesting thing about voice-over-IP isn't that voice can be sent using some particular protocol over a cheap data network. Rather, it's how cheap you can keep that data network while providing what people want from a voice network. Here's hoping that's pretty cheap.
Update March 3, 2004
The Register reports on similar issues.
Update May 30, 2004
It seems that some other folks have come to the same conclusion from a different direction.
So why aren't we all using phones that have Ethernet ports to make calls that don't cost us anything beyond our monthly fee for internet service? Well, actually some people are. And telephone companies are understandably somewhat concerned about losing voice revenue. But there's a little more to it than that. The problem is that we have different expectations for voice networks and data networks. If an email were delayed by three seconds, I wouldn't notice or care. But if my "Hello" were delayed by three seconds, it would be an awkward way to start a conversation. I'm no fan of giant telephone companies but at least part of the reason that voice is expensive and data is cheap (where that's true) comes from the different expectations we have for those networks.
In addition to expecting little latency when we send voice, we also expect our voice networks to be extremely reliable. I'm pretty sure that that annoying "five nines" thing started with telephone companies. If any telephone company actually achieves 99.999% reliability (that's about 5 1/4 minutes of downtime per year) by any meaningful calculation it's news to me. Still, I have to admit that they're much more reliable than even quite good data networks.
Beyond low latency and high reliability, voice networks also have lots of capacity. It's often the case that something causes lots of people to want to make telephone calls all at the same time. Whether it's a holiday or a disaster, it's very possible that I can have a particularly strong desire to make a telephone call just at the time that everyone else does. As Steven den Beste has observed, building capacity that goes unused almost all of the time is expensive, but the capacity of the voice network around here is great enough that I don't remember the last time I got a fast busy signal or an "all circuits busy" recording.
My ridiculously cheap data network can be laggy, it goes down a couple of times a year, and I routinely run into its capacity limits. And that's just fine. We use data networks in ways that mean we can tolerate those things. The interesting thing about voice-over-IP isn't that voice can be sent using some particular protocol over a cheap data network. Rather, it's how cheap you can keep that data network while providing what people want from a voice network. Here's hoping that's pretty cheap.
Update March 3, 2004
The Register reports on similar issues.
Update May 30, 2004
It seems that some other folks have come to the same conclusion from a different direction.
Posted at 08:05 Main Permalink
Sun - December 14, 2003
Free vs. Open-Source software
The folks at the Free Software Foundation and the folks
behind the Open-Source
Initiative seem to want about the same thing. They both want software
that everyone is free to inspect, modify, improve, and re-distribute. A lot of
software that's identified by its authors as being open-source is licensed under
the FSF's GNU General Public
License. But if their aims are so similar, why would Richard M. Stallman, the founder of
the FSF say, "I disagree
with the Open Source-movement" and why would Eric S. Raymond, a promoter of
open-source software, just about say that Richard Stallman should "shut
up"? And those are moderate examples of the acrimony between the two
camps.
The reason for the acrimony is that the two groups arrive at the same place from entirely different directions. RMS believes that my ability to give software that you wrote to a third person is a "natural" right, up there with life, liberty, and the pursuit of happiness. I don't see it that way, but there's little point in arguing with someone about what are and aren't natural rights.
By contrast, ESR and the OSI folks, whatever their personal principles, make pragmatic, utilitarian arguments that I should be able to give away software that you wrote. Over here they say, "Open source promotes software reliability and quality by supporting independent peer review and rapid evolution of source code." And, referring to the RMS's position slightly obliquely, "We think the economic self-interest arguments for open source are strong enough that nobody needs to go on any moral crusades about it."
It's easy for me to see things the open-source way. Some time ago, I released some software under the GNU General Public License. It was far from complete, but it was in a state where I thought that it might possibly be of some use to someone else. Pretty late at night, I typed the upload command and went to bed. The next morning, I checked my mail and found a very nice note from someone who had downloaded my program and had found a bug and sent me a patch to fix it. Literally, someone had improved my software while I slept. I had previously liked the idea of open-source rather abstractly, but the patch in my morning's mail was concrete. Since that time, I've gotten plenty of improvements to my software from strangers.
It's much more difficult for me to try to see things RMS's way. I have an ownership interest in the labor of my hands and I don't know why I shouldn't have an ownership interest in the labor of my brain. But RMS thinks that I shouldn't. On the other hand, if I look at a place where I do see a natural right, for example in free speech, RMS's impatience becomes clearer. If someone came to me and made utilitarian arguments in favor of free speech, I'd probably come off as pretty peevish in reply. Even if the utilitarian arguments were correct, I wouldn't want my right to free speech to depend on free speech being useful.
I don't expect to see an end to the arguments any time soon. There's no way to persuade someone who sees a natural right that they should be pragmatic about it, and no way that pragmatists are going to start seeing natural rights in software distribution.
The reason for the acrimony is that the two groups arrive at the same place from entirely different directions. RMS believes that my ability to give software that you wrote to a third person is a "natural" right, up there with life, liberty, and the pursuit of happiness. I don't see it that way, but there's little point in arguing with someone about what are and aren't natural rights.
By contrast, ESR and the OSI folks, whatever their personal principles, make pragmatic, utilitarian arguments that I should be able to give away software that you wrote. Over here they say, "Open source promotes software reliability and quality by supporting independent peer review and rapid evolution of source code." And, referring to the RMS's position slightly obliquely, "We think the economic self-interest arguments for open source are strong enough that nobody needs to go on any moral crusades about it."
It's easy for me to see things the open-source way. Some time ago, I released some software under the GNU General Public License. It was far from complete, but it was in a state where I thought that it might possibly be of some use to someone else. Pretty late at night, I typed the upload command and went to bed. The next morning, I checked my mail and found a very nice note from someone who had downloaded my program and had found a bug and sent me a patch to fix it. Literally, someone had improved my software while I slept. I had previously liked the idea of open-source rather abstractly, but the patch in my morning's mail was concrete. Since that time, I've gotten plenty of improvements to my software from strangers.
It's much more difficult for me to try to see things RMS's way. I have an ownership interest in the labor of my hands and I don't know why I shouldn't have an ownership interest in the labor of my brain. But RMS thinks that I shouldn't. On the other hand, if I look at a place where I do see a natural right, for example in free speech, RMS's impatience becomes clearer. If someone came to me and made utilitarian arguments in favor of free speech, I'd probably come off as pretty peevish in reply. Even if the utilitarian arguments were correct, I wouldn't want my right to free speech to depend on free speech being useful.
I don't expect to see an end to the arguments any time soon. There's no way to persuade someone who sees a natural right that they should be pragmatic about it, and no way that pragmatists are going to start seeing natural rights in software distribution.
Posted at 02:09 Main Permalink
Sat - December 13, 2003
Who's trusting what in "trusted computing"
"Trusted computing" is a marvelous bit of
marketing weasel-speak. I mean, who wouldn't want to trust their computer more?
I know several people whose computers can't be trusted to get through the day
without falling over at least once. Alas, that's not the kind of trust that
trusted computing refers to. The Trusted Computing
Group's "backgrounder
" (warning, as is often the case with tedious documents, it's a PDF) says (p.
6):
As an example, per TCG PC Specific Implementation Specification
v.1.0, the CRTM for PCs is the BIOS or BIOS boot block and the BIOS
is required to load HASHES of pre-boot information into various
PCRs of the TPM. This establishes the "anchor" for the chain of trust
and the basis for platform integrity metrics. This can be used to
validate that the platform configuration has not changed and that the
BIOS has not been changed by malicious code such as a Trojan
horse. While not required, verifiable attestation of the platform
configuration can be extended by a chain of trust to the boot loader,
operating system, and applications if software support for this is
provided. TCG does not provide specifications for how this is
accomplished, as this is under the control of these software suppliers.
Un-obfuscated, that means that when you turn on a computer that conforms to their specification, the first thing that happens is that some hardware gets to decide if the BIOS gets to run. In order to be acceptable, the BIOS is likely to need to be cryptographically signed with some particular secret key by the people who wrote it. The BIOS then becomes trusted and can choose to load only an operating system that has been similarly cryptographically signed. The operating system can choose to run only programs that have also been cryptographically signed. All this is couched in terms of "opt-in" and larded with "may"s but it's easy enough to see where this is going: your computer's ROMs would be picky about what operating system their code was willing to pass control to. And your operating system could be picky about what programs it was willing to run. That might increase security slightly since it's unlikely that a virus would be signed by Microsoft, but it's an even better way to reduce choice.
In general, elaborate restrictions of this sort cause users more problems than they solve. Such a system would be sufficiently elaborate that it's virtually certain that there would be flaws in its implementation and quite possibly errors or limitations in its design. The bad guys would go to the trouble of finding and exploiting those errors and limitations and the ordinary users would be left with less choice of software. So "trusted computing" doesn't mean that you can trust your computer to do what you want, but rather that the software publishers can trust your computer to do what they want.
Microsoft's somewhat different trusted computing project used to be called Palladium but they changed the name to "Next-Generation Secure Computing Base", presumably to make it less memorable. "Palladium sucks" has a ring to it that "NGSCB sucks" just doesn't have.
Here's a bit from Microsoft's NGSCB FAQ:
Q: I have heard that NGSCB will force people to run only
Microsoft-approved software.
A: This is simply not true. The nexus-aware security chip (the SSC)
and other NGSCB features are not involved in the boot process of the
operating system or in its decision to load an application that does not
use the nexus. Because the nexus is not involved in the boot process,
it cannot block an operating system or drivers or any nexus-unaware
PC application from running.
That's fine as far as it goes, but Microsoft's claim that they're only interested in the OS and not in the boot process matches up surprisingly well with the Trusted Computing Group's claim that they're only interested in the boot process and not in the OS. Indeed, one looks pretty useless without the other; for the OS to trust what the underlying hardware tells it, it would want a trusted BIOS. And a trusted BIOS isn't of much use alone. Indeed, in the same FAQ Microsoft says a little later:
Q: How is NGSCB related to the Trusted Computing Group (TCG) and
the Trusted Computing Platform Alliance (TCPA)?
A: ... Microsoft is a founding member of TCG and anticipates that some
of the industry standards being developed by the group will be
incorporated into NGSCB.
It seems that that convenient match-up may not be a coincidence.
It may be that it's only some future version of Windows Media Player that will say "I don't trust your system" and refuse to run. But on past form, people betting on the most benign interpretation are likely to be disappointed.
For myself, I'm not interested in anyone but me deciding what programs will run on hardware that I pay for. I'm very ready to deal only with publishers who trust me rather than my hardware.
As an example, per TCG PC Specific Implementation Specification
v.1.0, the CRTM for PCs is the BIOS or BIOS boot block and the BIOS
is required to load HASHES of pre-boot information into various
PCRs of the TPM. This establishes the "anchor" for the chain of trust
and the basis for platform integrity metrics. This can be used to
validate that the platform configuration has not changed and that the
BIOS has not been changed by malicious code such as a Trojan
horse. While not required, verifiable attestation of the platform
configuration can be extended by a chain of trust to the boot loader,
operating system, and applications if software support for this is
provided. TCG does not provide specifications for how this is
accomplished, as this is under the control of these software suppliers.
Un-obfuscated, that means that when you turn on a computer that conforms to their specification, the first thing that happens is that some hardware gets to decide if the BIOS gets to run. In order to be acceptable, the BIOS is likely to need to be cryptographically signed with some particular secret key by the people who wrote it. The BIOS then becomes trusted and can choose to load only an operating system that has been similarly cryptographically signed. The operating system can choose to run only programs that have also been cryptographically signed. All this is couched in terms of "opt-in" and larded with "may"s but it's easy enough to see where this is going: your computer's ROMs would be picky about what operating system their code was willing to pass control to. And your operating system could be picky about what programs it was willing to run. That might increase security slightly since it's unlikely that a virus would be signed by Microsoft, but it's an even better way to reduce choice.
In general, elaborate restrictions of this sort cause users more problems than they solve. Such a system would be sufficiently elaborate that it's virtually certain that there would be flaws in its implementation and quite possibly errors or limitations in its design. The bad guys would go to the trouble of finding and exploiting those errors and limitations and the ordinary users would be left with less choice of software. So "trusted computing" doesn't mean that you can trust your computer to do what you want, but rather that the software publishers can trust your computer to do what they want.
Microsoft's somewhat different trusted computing project used to be called Palladium but they changed the name to "Next-Generation Secure Computing Base", presumably to make it less memorable. "Palladium sucks" has a ring to it that "NGSCB sucks" just doesn't have.
Here's a bit from Microsoft's NGSCB FAQ:
Q: I have heard that NGSCB will force people to run only
Microsoft-approved software.
A: This is simply not true. The nexus-aware security chip (the SSC)
and other NGSCB features are not involved in the boot process of the
operating system or in its decision to load an application that does not
use the nexus. Because the nexus is not involved in the boot process,
it cannot block an operating system or drivers or any nexus-unaware
PC application from running.
That's fine as far as it goes, but Microsoft's claim that they're only interested in the OS and not in the boot process matches up surprisingly well with the Trusted Computing Group's claim that they're only interested in the boot process and not in the OS. Indeed, one looks pretty useless without the other; for the OS to trust what the underlying hardware tells it, it would want a trusted BIOS. And a trusted BIOS isn't of much use alone. Indeed, in the same FAQ Microsoft says a little later:
Q: How is NGSCB related to the Trusted Computing Group (TCG) and
the Trusted Computing Platform Alliance (TCPA)?
A: ... Microsoft is a founding member of TCG and anticipates that some
of the industry standards being developed by the group will be
incorporated into NGSCB.
It seems that that convenient match-up may not be a coincidence.
It may be that it's only some future version of Windows Media Player that will say "I don't trust your system" and refuse to run. But on past form, people betting on the most benign interpretation are likely to be disappointed.
For myself, I'm not interested in anyone but me deciding what programs will run on hardware that I pay for. I'm very ready to deal only with publishers who trust me rather than my hardware.
Posted at 07:00 Main Permalink
Sun - December 7, 2003
The DMCA and what it says about open-source software
There's no reason here to go into all the reasons
that the Digital
Millennium Copyright Act is a bad law. Plenty of other people have done that. But there's an
implication of one aspect of it that I haven't seen mentioned
elsewhere.
As most every geek knows these days, the DMCA makes it illegal (with a few exceptions) in the US to circumvent anti-piracy functions built into software. A non-geek might find that odd. Why should a big software company need to make subverting part of their software illegal? Couldn't they just make it impossible or at least very hard? Alas, on past form, they can't. Every form of copy protection I've heard of have been cracked sooner or later. And it's not large companies that crack copy-protection systems. It's mostly teenagers working in their spare time. The DMCA is just a matter of the big software companies running up a white flag. They're sure that whatever code they write will be cracked by a socially irresponsible teenager so they need to be able to sue.
But how is it that they can admit that they can't outwit teenagers and then say with a straight face that they produce better code than the thousands of dedicated people who work on open-source projects such as Linux and Python? Doesn't make sense, does it?
As most every geek knows these days, the DMCA makes it illegal (with a few exceptions) in the US to circumvent anti-piracy functions built into software. A non-geek might find that odd. Why should a big software company need to make subverting part of their software illegal? Couldn't they just make it impossible or at least very hard? Alas, on past form, they can't. Every form of copy protection I've heard of have been cracked sooner or later. And it's not large companies that crack copy-protection systems. It's mostly teenagers working in their spare time. The DMCA is just a matter of the big software companies running up a white flag. They're sure that whatever code they write will be cracked by a socially irresponsible teenager so they need to be able to sue.
But how is it that they can admit that they can't outwit teenagers and then say with a straight face that they produce better code than the thousands of dedicated people who work on open-source projects such as Linux and Python? Doesn't make sense, does it?
Posted at 01:53 Main Permalink
Quick Links
Calendar
| Sun | Mon | Tue | Wed | Thu | Fri | Sat |
Categories
Archives
XML/RSS Feed
Me
Statistics
Total entries in this blog:
Total entries in this category:
Published On: Aug 30, 2010 07:44
Total entries in this category:
Published On: Aug 30, 2010 07:44