Stuff I've read lately

"Nebula Awards 33" (Connie Willis ed.) - They've changed the format, and I must say I like it. Each story is followed by a note from the author describing the conditions that led up to writing it. It's interesting how good stories come from flimsy circumstances, and less interesting stories often come from well thought out approaches (and vice versa). Also, there is a classic story from the golden age of SF.

Stuff I've read lately

"Act of War" (Dale Brown)(audio) - I have to wonder who is the target audience for Dale Brown's work. Do they appreciate the cheesiness? Or do they actually take it seriously? I mean, the "good" people are all super-good (and beautiful) - while the "bad guys" eat kittens (and are usually ugly and scarred).

That said, I did have to appreciate this book codifying the phrase "he's got the atomic bomb!" (said in the same vein as "he's got a knife" or "he's got a gun"). Not that anyone actually said it, but Brown set up the situation.

Brown did manage to set up a surprise at the end, although the voice acting gave it away somewhat earlier than would probably come about from just reading it.

Aerostats

Interesting news article I found in my archives:

"This so-called graphene sealed microchamber is impermeable to even the tiniest airborne molecules, including helium."

Fiction involving nanotechnology often has objects called "aerostats", basically floating machines (usually equipped with fans or jets for movement).

If the balloon wall is impermeable to most atmospheric gases, then the balloon might be able to be "filled" with a vacuum (assuming air pressure won't collapse it). Making it the best lighter-than-air material.

Stuff I've read lately

"iWoz" (Steve Wozniak)(audio) - I was really looking forward to this. Woz was somewhat of a hero for me, what with my first computer being an Apple II. It also seemed like he had had just a bad experience with Apple as I had.

Sadly, Woz gives Apple a bye. He even lets Jobs off the hook.

It was interesting to hear the story of how Jobs used to wirewrap boards for Woz.

I say

we take off - nuke the site from orbit. It's the only way to be sure.

I just read something at Ars that makes me physically sick. Farmville for Dummies.

At least the interview with the author is good for a laugh:

"The first time Orland played Farmville he had to give himself a goal to maintain interest: to recreate the first level of Super Mario Bros. out of haybales. After that, his interest waned and he left the game."

NedNews: XML, Again

Isn't the point of XML that you can load in all sorts of data, and everyone can pick out what they need?

I just try fetching from a different RSS feed, and all the data is in different places! What a pain.

Regardless, I'm now satisfied that I can get all the data into the database. That will be the next step before trying to show article bodies (because article lookup should pass through the database).

NedNews: Pics

Ok, I'm slowing down, the fatigue is starting to build up!

I need to have something I can show, in case my brain shuts down entirely.

Let's get the titles from our RSS feed into out header window.



Ok, that is a screenshot of the Tcl window (still called tMain for now) over top of NewsFox (with my subscriptions and the other columns hidden).

Not too bad. I had to shuffle things around, because I forgot the scrollbar (I always do that!)

NedNews: HTML

Ok, I little distraction from the HTML. Sadly, there isn't a really good Tcl widget that automatically handles HTML (this is something I'd like to work on, I'd call it InterNed Exploder).

There is an old project called tkHtml. Surprisingly, I was able to get the source and build it with no problems. I have it in a git repo, and can push it to github if anyone cares.

I think I'm tackling enough with NNTP, RSS, and a database backend. I am going to punt on the HTML, knowing that there is one ok solution, and it's something I'd like to work on.

NedNews: XML

Ok, now we have our XML data (in the string variable httpData).

Let's see if it looks good (in my wish session):

% llength $httpData
list element in braces followed by "&lt;/div&gt;</summar" instead of space
% string length $httpData
101373

I am so used to using llength instead of string length, I mistyped it the first time. No worries, Tcl tries to interpret the string as a list, but it is malformed. The second time around I get the length in characters (Tcl will handle multibyte character strings automatically).

Let's parse this into a tree.
set doc [dom parse $httpData]

Ok, let's look through the tree (the dom interface returns a list of child nodes), and see what we got (llength is "list length" and lindex is "list index", where 0 is the first in the list):

% llength [$doc childNodes]
2
% [lindex [$doc childNodes] 0] asText
type="text/css" href="http://semipublic.comp-arch.net/wiki/skins/common/feed.css?270"
% [lindex [$doc childNodes] 1] asText
(pages and pages)

Ok. Pesky. The top of the tree has just two nodes. The first gives us the stylesheet. The second has all the real content...

% set docTop [lindex [$doc childNodes] 1]
domNode00BFCF48
% llength [$docTop childNodes]
38

In my final code, I will try to use XPath to extract the vital goodies. But we can poke around by hand here.

I can see that there are some initial fields ("id" - the URL I used to get this, "title", "link" - which also seems to be the URL, "updated" - with some time info, "subtitle", and "generator"). Then there are "entry" fields with the posts. So, the "updated" field will come in handy when I want to look for updates, "title" might be useful for populating a description. Otherwise, I just want the "entries".
set entries [$doc selectNodes -namespaces {atom http://www.w3.org/2005/Atom} /atom:feed/atom:entry]
There is a slight complication here. The Atom tags are referenced against a URL, so I need to supply that for XPath...
I can see each entry has 6 fields, an "id" - which is the link to the full post, "title", "link" - which also links to the full post, "updated", "summary" - the body, "author". Perfect.

I would pick the one feed and entry which is full of HTML. I will post a bit on that next...

NedNews: HTTP

At this point, I would like to set up my database schemas. I'm very familiar with Usenet, so I think I can lay out everything there. However, I am less familiar with the XML that comes in RSS feeds, so I want to sift through some of that before committing to a database layout.

An RSS feed is indicated by a URL, and is XML content delivered via HTTP. We will need to load packages to handle this for us:
package require http
package require tdom

tdom is the package I use for parsing the NewStars XML turn files. It has an excellent XPath system.

We'll need a URL to fetch the data from (this should come from a subscription list, but we will hack it in for now):
set url "http://semipublic.comp-arch.net/wiki/index.php?title=Special:RecentChan
ges&feed=atom"

This is the list of recent edits from Andy Glew's informative Comp.Arch wiki.

Now, let's fetch some data. Things are somewhat complicated by the asynchronous nature of HTTP (we fire off a request, then some period of time later, we get data). So, we will need a callback to handle the eventual arrival of data. Ideally, we would handle multiple outstanding requests, but let's deal with one right now...
set httpData ""

This is a global variable which will store the result of our http request (this is why we can only handle one request at a time).

And the callback:

proc httpDone {token} {

  set fail 0
  switch [::http::status $token] {
     ok {
        if {[::http::ncode $token] != 200} {
           puts "HTTP server returned non-OK code [::http::ncode $token]"
           set fail 1
        }
     }

     eof {
        puts "HTTP server returned nothing"
        set fail 1
     }

     error {
        puts "HTTP error [::http::error $token]"
        set fail 1
     }
  }

  if {!$fail} {
     set ::httpData [::http::data $token]
  }
  ::http::cleanup $token
  puts "HTTP Done"
}

The handler is complicated by how the HTTP package decided to handle errors and odd conditions. Ideally, it would roll bad "ok" states into "error" (and probably the "eof" condition as well), but there it is. Again, ideally, we would handle what errors we could (possibly with a retry), and notify the user in a better way (but I am the main user!)

Here we can see Tcl's switch statement. Pretty straightforward. The first argument provides the string that will be matched. The body consists of tuples. The first part of the tuple being the string to match, the second being code to execute on a match. There is no provision for fall-through, although you can handle multiple patterns with a single body:

switch $status {
   eof -
   error { puts "Not good" }
}

Now we are set to fire off a request:
set httpToken [::http::geturl $url -timeout 5000 -command httpDone]

I punch all this right into my wish session. Then I wait for some output... My first attempt used the "-channel" option, but it appears that requires an already open channel - so the command failed. I can rewrite my handler to not use the channel, paste it into my session, and reissue the command, without restarting (if you create a proc that matches an existing proc, the old one is overwritten).

Which reminds me. proc is just another command in Tcl. It takes three arguments: the name of the procedure, the arguments, and the body. Any Tcl command can be overriden this way (although it is best to rename the original to something else so you can get to it). I use this to trap the exit functionality:

rename exit origExit
proc exit {} {
   # closeout some resources and cleanup some stuff
   origExit
}

The second time I get:
HTTP Done

NedNews: GUI

The other "Hello world to RSS reader" series tack on a GUI at the very end. One of the draws for Tcl is the easy integration with a GUI (using Tk). For small projects, seeing something right away is helpful for keeping focused and motivated.

Let's start with the GUI right away!

I have set up my code on Github, so you can see it all there!

Getting started:
I am on a Windows machine, using a "batteries included" Tcl install from ActiveState. Tcl is cross platform, so everything should just work on Linux or MacOS. You may need to install various packages (probably Tcl, Tk, and Sqlite). Also, I heavily recommend tkcon on Linux. I am using Tcl 8.5 which includes a lot of improvements over 8.4.

I do most of my development "live" (inside a Tcl windowed shell - wish). That gives me instant feedback on changes, and allows me to test code in the running environment. No compile, breakpoint, test loop!


wm withdraw .
toplevel .tMain

This is a little bit of boilerplate I use on all projects.

Wish will give you a default top level window (named "."). The problem is, if "." is destroyed (closed), then wish will exit, and you lose your session.

Using the window manager command (wm), I "withdraw" the default toplevel (which causes it to disappear). That way, I can't accidentally close it.

I then create a new toplevel. Putting everything into a new top level allows me to close it, and start over, while keeping the rest of my session.

Let's create our three pane display. There is one pane on the left, to show all the news sources. The right side is split into a top and bottom, showing headers and bodies.

## a splitter for left and right windows
pack [ttk::panedwindow .tMain.splitLR -orient horizontal] -expand 1 -fill both
# a tree for news sources
.tMain.splitLR add [ttk::treeview .tMain.splitLR.tSrcs]

Here we can see a comment (starting with "#"), and a line using nested execution. Execution begins inside the square brackets. I use the new (to Tk 8.5) "Themed Tk" (ttk) namespace command "panedwindow", which will divide up my window into panes (duh).

This is our left-to-right split. All of the Tk widget commands return the widget name (.tMain.splitLR in this case). That allows me to chain the command into pack.

pack activates the packing algorithm. Many GUI systems require you to place widgets at specific x,y points. This then requires a GUI builder to arrange everything so it looks nice (there is a GUI builder for Tcl called "v", which uses the place layout algorithm).

Not so pack. I can just give all my widgets over to pack, and it does a good job of arranging them all neatly. The arguments -expand and -fill tell it I want all the window space given to this big widget (it helps to have all the documentation open, so you can see all the options for these commands).

We pile in some more widgets in a similar way (note that the panedwindow is its own sort of packer for the sub-windows)

## a splitter for top and bottom (headers and bodies)
.tMain.splitLR add [ttk::panedwindow .tMain.splitLR.splitRTB -orient vertical]

# a tree for headers (threaded view)
.tMain.splitLR.splitRTB add [ttk::treeview .tMain.splitLR.splitRTB.tHdr]

# textbox for bodies
.tMain.splitLR.splitRTB add [text .tMain.splitLR.splitRTB.xBdy -state disabled]

Eighteen lines of code, five of which are blank, and six are comments. And a lot of explanation on this blog. That's why I like Tcl. Very concise.

NedNews: Overview

Ok, that was long and involved! Let's get coding!

First, we should do a little planning...

Most news readers are three panel. That will be fine for our purposes.

We are going to support both RSS (XML) and Usenet (NNTP). This will add a little complexity, but it is what I want :)

The other thing I want, which the other series did not cover, is persistence. I want to remember what posts I have read, and I want to store posts as long as I want (not as long as the server decides to keep them!)

digitalmars.D (one of the Usenet groups I read) has 20,000 posts. I previously mentioned that NewsFox breaks down at several thousand posts (which tells you I often let my RSS feeds accumulate thousands of posts).

Storing this amount of data should be done in a database.

Fortunately, the popular embedded database engine Sqlite is actually a Tcl extension (it also provides a C api, which is what most people use).

We will need to think through some database schemas, but it will be worth it. The database file can be shared between machines, and across the network through a service like DropBox. (As long as we are careful not to update it simultaneously from different machines.)

NedNews: Background

The main idea behind Tcl is "everything is a string". Code is a series of strings evaluated in a particular context. Lists are strings formatted in a particular way. (There are also maps from string to string)

This might seem stupid or limiting, but it actually flattens interfaces - now you don't need to worry as much about how to interface different components.

You have production and consumption of strings (somewhat like command pipelines in Unix).

Tcl looks enough like C that it can be misleading:

for {set i 0} {$i <= 10} {incr i} {
   puts "Hello $i"
}

It can be more helpful at times to actually think of it as a stripped down Lisp-like:

(for, (set, i, 0), (<=, $i, 10), (incr, i) (puts, "Hello $i"))
(which makes more sense if you are used to Lisp, else little sense)

• The first command on each line in Tcl doesn't need parens (and the whole program doesn't need parens).
• Arguments are separated by whitespace instead of commas. Double quotes surround arguments which contain white space.
  
• Curly braces are used to protect strings from variable substitution (when variables will get substituted later).
• Square brackets enclose nested commands (arguments which need to be evaluated as function calls)
  
• New lines are significant, for is really a procedure which takes the body as an argument - lambda programming from the beginning!

That last point means:

for {set i 0} {$i <= 10} {incr i}
{
   puts "Hello $i"
}

won't work (for will fail to find its body argument).

In other words, the Tcl parser is very simple. It processes one line at a time (concatenating lines inside curlies or that end in "\"). It then breaks the line up into "words" based on spaces and tabs. The first word is used as the command, and the other words get passed along as arguments. Substitution is done to all words, except inside of of curly braces ("{}"). Square braces ("[]") trigger nested execution. That's it!

Let's say you want to display the results of some math:

puts "Two plus two = [expr 2 + 2]"


Again, in Lisp, this is:

(puts, "Two plus two (expr, 2, +, 2)")
(assuming Lisp would expand inside quotes, which I don't think it does)

The brackets trigger execution inside the currently executing line of code. The first word is the command ("mathematical EXPRession"). expr takes any number of word arguments, which get evaluated as math.

2 + 2 can't work because "2" is not a command. Some people create commands for "+,-,/,*" - then you can do "+ 2 2". That is good for RPN people, but I don't see the benefit.

NedNews: Motivation

I've long desired to write my own news reader. I currently use the Firefox plugin Newsfox for RSS, and Outlook Express (which is far superior to Outlook) for old-school Usenet (NNTP) news.

I would like something which I can access from multiple computers, but all the web based news readers I looked at did not store which messages are read versus unread. They also tend to only show the last few posts (I sometimes leave posts in the queue for months, and my NNTP backlog goes back two years).

Plus, Newsfox has eaten my archives on two occasions now (it seems to have problems once a feed gets to several thousand posts). And Outlook Express uses some cryptic Microsoft method for storage, which would be a pain to try and share across computers.


I have seen two posts now on Reddit, talking about "Hello world to RSS reader" (Scala and Prolog). I figure, this is the perfect time for one on Tcl!

The redditor takes a month for his projects. Given that I already have experience with Tcl, and have thought a fair amount about my newsreader - I am going to see how much I can get done in one day!

Got a funny feeling

you don't love me anymore.

If you don't mind me asking,
what's this poisonous cobra
doing in my underwear drawer.

Got a funny feeling, you don't love me anymore.

You drilled a hole in my head,
then you dumped me in a drainage ditch
and left me for dead.

Ooo, oh, oh, you never acted this way before,
honey something tells me,
you don't love me anymore.