git avert disaster

Let me start by saying I hate hg. I should like it, since it is fundamentally the same as git. I want to like it, but it just is unlikeable.

1. It is too complicated. In git there are commits (patches, content) and heads (names). Mercurial has versions, patches (in mq), branches, tags, bookmarks. Maybe more.
2. It is disjointed. In git, everything is integrated. In hg, everything is an extension. The extensions work fine, one at a time. But bringing multiple extensions together doesn't always work well. Also, the tab completion only supports some things.
3. It keeps messing up. This is the really scary part. I haven't lost anything, but twice now all my subrepos have gone into a "disconnected head" state (once after a crash, but once all by itself). I also sometimes get weird "unknown version" errors. It generally just requires "hg onsub 'hg up'", but sometimes greater voodoo is required.

That said, I encountered my first git problem recently.

1. Install Cygwin git on Windows
2. Intall Linux dual boot, boot in Linux, install git
3. Mount the NTFS under Linux (hmm, maybe not a good idea here!)
4. Use Linux git to manipulate the Cygwin repo (playing with fire!)
5. Rebase the repo
6. Boom

Yea, maybe I was asking for this one. The problem is that under Cygwin all the files have 644 permission, while Linux mounts NTFS with 755. Git stores the permissions flag under version control, so it is seeing every file change all the time. Rebasing causes massive conflicts all the time (since the history is all 644, but the mount point won't allow anything but 755).

So, here I am in some disconnected head state, with all my patches gone from the rebase. What to do?

I forgot what I did, but I got most of my patches applied. Except one fell on the floor, and didn't show up in the history anymore (well, the commit was there with the log message, but the patch was empty).

Fortunately, there is a file in .git called "ORIG_HEAD" or something similar. Cat it, and "git checkout". Cp the bad files off to a separate space and "git checkout" back. Diff and patch, commit. All fixed!

Yea git!

Stuff I've read lately

"The Oxford Book of Science Fiction" (ed. Tom Shippey) - This was really excellent. It covers about 100 years of SF, and most of the stories I hadn't read before. Really unusual.

String Madness

I read earlier today that Denis Ritchie (or Brian Kernigan, I get them mixed up) hates C++.

Now, why would anyone hate C++?

Oh wait, I had to do some case insensitive string compares today in C++. Let's see how to do it:

1. Solution 1, download hundreds of megabytes of Boost - meh
2. Comment #2, Unicode is broken, yea Unicode!
3. Solution three:

struct ci_char_traits : public char_traits<char> {
    static bool eq(char c1, char c2) { return toupper(c1) == toupper(c2); }
    static bool ne(char c1, char c2) { return toupper(c1) != toupper(c2); }
    static bool lt(char c1, char c2) { return toupper(c1) <  toupper(c2); }
    static int compare(const char* s1, const char* s2, size_t n) {
        while( n-- != 0 ) {
            if( toupper(*s1) < toupper(*s2) ) return -1;
            if( toupper(*s1) > toupper(*s2) ) return 1;
            ++s1; ++s2;
        }
        return 0;
    }
    static const char* find(const char* s, int n, char a) {
        while( n-- > 0 && toupper(*s) != toupper(a) ) {
            ++s;
        }
        return s;
    }
};

typedef std::basic_string<char, ci_char_traits> ci_string;

Muahahahahahaa! That's evil (Note: I handled all the freaky <> using Tcl, yea Tcl!). In case you're wondering:

string map "< {&lt;} > {&gt;}" $code

Solution 4, fall back to C. Thanks, that's great. Also, note it is different on different platforms, yea C. It also doesn't work on Unicode.

In Tcl?

string compare -nocase $str1 $str2

Works on most Unicode, but, hey Unicode is broken what do you want?

True Atlanteans

Lots of progress, haven't had time for updates. I am building a client for Atlantis, similar to Atlantis Little Helper (see screenshot).

Here is what I have so far:

Obviously, not as much functionality right now, but it is improving quickly. The main advantages will be that I can understand this code (currently less than 500 lines of Tcl).

Also, the code uses an internal database to store the turn information, so should be less vulnerable to disruptions (and more easily support multiple games).

This will be the main visualization tool for any changes we make to Atlantis (currently, I have already hacked in the new turn format).

Tcl Object Notation

In the beginning, everything was a plain text file or a binary file. Binary files were easy for computers (just copy the bytes from disk into memory) and hard for people. Text files were easy for people and hard for computers.

On top of this, things were always changing. So the binary files from Word 1.0 didn't want to be read into Word 2.0 (well, 2.0 would upgrade them - but eventually, support could disappear). Text files have a way of expanding into their own Turing-complete language.

Then some mega-genius said, "Hey, let's make a text format that easy for computers and people, and backwards/forwards compatible! We'll call it XML."

And so we got this:

<?xml version="1.0"?>
   <NSTARS_UNIV DIMENSIONS="2" X="400" Y="400">
      <UNIV_HDR>
         <STAR_COUNT>2</STAR_COUNT>
      </UNIV_HDR>
      <GAME_YEAR>2406</GAME_YEAR>
      <TECH_COSTS>50 80 130 210 340 550 890 1440 2330 3770 6100 9870 13850 18040
 22440 27050 31870 36900 42140 47590 53250 59120 65200 71490 77990 84700</TECH_C
OSTS>
      <COMPONENTFILENAME>newStarsComponents.xml</COMPONENTFILENAME>
      <HULLFILENAME>newStarsHulls.xml</HULLFILENAME>
      <PLAYERFILEBASENAME>tiny_sparse</PLAYERFILEBASENAME>
      <MASTER_COPY>1</MASTER_COPY>
      <NUMBER_OF_PLAYERS>2</NUMBER_OF_PLAYERS>
      <PLAYERDATA>
         <RACELIST>
            <RACE>
               <SINGULARNAME>Ugly Duckling</SINGULARNAME>
               <PLURALNAME>Baby Swans</PLURALNAME>

That's the first 771 bytes of the NewStars master file (which is 3530 bytes in full). That's turn 6 for a tiny universe. I generated a huge turn once, it was many megabytes... Every AJAX request is generating and shipping around stuff like this. If you wonder why servers can't handle many clients, why the Internet is so slow (even though we have a lot more bandwidth than the old 56k modems), and clients are so slow - XML is a big part of it.

Then another (smarter) genius said, "All these angle brackets and matching tags are just a pain. Why can't we have a simpler format?" That gave us JSON (JavaScript Object Notation, and AJAJ). Here is the JSON file from my Space Battle project:

{
        "force1" : {
                "name" : "Imperials",
                "ships" : [
                        "idest"
                ]
        },

        "force2" : {
                "name" : "Bugs",
                "ships" : [
                        "bship"
                ]
        }
}

A lot more concise. But, why do I need all the quotes and colons? Why are some things in square brackets, while others are in curlies? We can do better:

Name {Tester (3)}
FactionType { (War 1, Trade 1, Magic 1)}
Month March
Year 1
VerString {4.1.0}
Rulesetname {Ceran}
Rulesetversion {2.0.4 (beta)}
Newssheet 1
Password {none}
TurnCountdown -1
Quit 0
TaxRegion 0
MaxTax 10
TradeRegion 0
MaxTrade 10
NumMage 1
MaxMage 2

This is "Tcl Object Notation" (TON). Just as JSON yields a JavaScript dictionary, this is a Tcl dictionary. Since everything is a string, no quotes needed. No colons (dictionaries are lists with "key" "value" pairs). You just need {} for things which might have spaces (or sub-dictionaries).

This will be the turn file format for the True Atlanteans, Atlantis PBEM GUI Client.

Peak Civilization

(a recent article at CNN has got me all depressed)

A few years ago the drumbeat of "peak oil" picked up. Now, it seems to be mostly taken as a given. With the recent Japanese nuclear disaster (among other things), I can't help but think we have reached "peak civilization".

Early science fiction (1900, up to the 60's, even parts of the 70's and 80's) was decidedly optimistic (although not entirely). Mankind was seen as making use of greater and greater stores of energy, and going far. Stories assumed galactic civilization (some took place after many cycles of rise and fall of galactic empires).

At some point, something changed. Perhaps it was the nuclear accidents of the 70's and 80's, or the failure of fusion power. The failure of the space program (which we now see the culmination of, in the last shuttle flights - which will end the American manned space program). Science had promised us the stars, and failed to deliver.

Now, the spectre of global warming is pushing back against greater carbon based energy production. The Fukushima disaster will likely hinder, if not stop nuclear production. Green sources are notoriously inefficient, in terms of land use and construction overhead - they might replace our current production, but growth will be limited.

Science fiction has picked up on this ennui (I don't think it has caused it). Stories of galactic empire are few and far between (John Scalzi is the only current writer I know). Most stories tell of introspective and decadent remnants of humanity, living in dirt (at least, metaphysically, if not physically).

Much of our economy is based on an assumption of growth. It's unclear how we can adapt (we see this in the repeated "bubble" growth and pop cycles). Even population growth is leveling off.

Asimov wrote about computers stretching underneath the whole world (plantary AC). Now, we strive to make computers smaller and less powerful.

It seems we are destined to simply fizzle out. Staring at our belly buttons until the lights go out.

Edge of Battle

"Edge of Battle" (Dale Brown)(audio) - This one was bizarre. I don't know if his editor came to him and said "You're scenarios are too believable, it's scaring people". Here we have a group of Mexicans who believe the western US belongs to them. Then they engineer border skirmishes and steal nuclear weapons (not sure what they want the nukes for, they get blown up (reeeaal goood) before we can find out).

Why? To force the US to implement a guest worker program.

The book turns all cheery 1984 at the end too; with the Mexican and American presidents gulping down nano-tracking pills and smiling for the camera (only people with something to hide avoid swallowing nano-tracking pills!).

Why Not Linux

(This is about why NedOS will not be a Linux clone)

There are three main operating systems today: Windows, Linux, and MacOS.

Windows is just terrible in every way, and getting worse. It used to be that Windows had the best support for hardware. I believe Linux is better now and can only get better, and Windows will get worse. This is because Microsoft has taken to enjoying changing their driver model periodically, and that breaks old stuff. I do most development on Windows under Cygwin, and I would very much like to stop using Windows entirely.

MacOS is a non-starter. I'm not going to give Apple more money (long story). Also, MacOS is basically BSD (a Unix derivative), plus some stuff.

That leaves Linux. Linux is really very good. But it is a Unix derivative. Some say Unix is bad because it is old. Being old is not necessarily bad, but you must re-examine the assumptions, circumstances, and goals that went into the old design. Where there are changes, it may necessitate a new design.

1. Unix was designed as a simplified alternative to Multics (hence the pun in the name). That no one is making Multics clones today tells you this was probably a good move. That Unix is considered complex hints at how complex Multics must of been (I have no idea)
2. Unix was meant to be run on multi-user computers. You would have one computer for possibly a dozen people to use at the same time. There might be dozens of accounts, some of whom are total strangers (paid accounts).
3. Unix was created for hackers, by hackers. As much as possible, it is intended that users should be able to write their own programs, and those programs should be able to access only the resources available to that user - and no more. Thus we see all the jockeying for "root shells" and counter security practices to prevent that.
4. Unix was meant to run on different types of machines. Linux has done really well here. You can run Linux on a "wall wart" (computer in a power plug), laptop, or supercomputer. In every case, you get the most out of your computer.
   

#1 - Simplicity. This was key to Unix's success. It will have to be a main stay for NedOS. Complexity leads directly to effort, and effort is going to be a key bottleneck.

#2 - Multi-user. This is a source of some complexity that I will be able to avoid. NedOS will be largely single user, with the possibility of some user specific configuration (a family likely has multiple computers, but different people may be on different boxes at any given time).

#3 - Software development. This will be the biggest change. All software functionality will be set at boot time. This should make things simpler, and more secure. I am considering some level of user scripting, but I'm not sure about that.

#4 - Multi-platform. This is a sore spot. I would love to run on all the hardware that Linux supports. But this is a source of complexity (and a huge effort sink). I will be forced to stick to Bochs (which should transition to Qemu for speed). I might be able to run on some real hardware at some point. But consider that my developing a 64 bit kernel means only 64 bit hardware support. 32 bit would double most of my effort.

It's a Cookbook!

A COOK BOOK!
(apologies to the Twilight Zone)

I have completed my read through of "Tcl/Tk 8 Programming Cookbook". It reads very much more like a cookbook than a novel or textbook.

I see three possible consumer groups:

1. People who are experts at finding information from man pages (like me, and 1% of the rest of the world)
   
2. People who don't like reading man pages, and are interested in learning Tcl.
3. People who have "that Tcl app" at work, that they need to maintain once every six months.

I don't think group 1 will get a lot out of this book (I did find some lesser known switches of some commands).

However, groups 2 and 3 should benefit.

Beginners will need to read the whole book to cover everything they need. However, the "cook book" style allows you to jump to any point and try something out. That is important for a beginner, to keep the interest up ("is there something new and interesting here that is worth my learning it").

You can jump right into GUI development. That is what got me hooked on Tcl/Tk.

For people less interested in learning Tcl, but needing to solve a particular problem - the table of contents should allow you to jump right to it. No need to sift through man pages of sometimes historically named commands.

This book is up to date, preferring dictionaries (new) over arrays (old). Most of the other Tcl books are from before dictionaries. It also mentions using the new themed widgets (ttk).

Tcl and Greek

While some may say that Tcl looks like Greek to them, I have Tcl doing my Greek homework!


This is a window with just an single line text entry. It is built with the following code:

toplevel .t
pack [entry .t.e]
bind .t.e <return> {
  set s [.t.e get]
  .t.e delete 0 end
  .t.e insert 0 [revGreek $s]
}

When I hit enter, I get:


My English/Greek transliteration uses the "_" character as an escape (to handle the two letter English versions, and s/s at the end). The revGreek function is a simple state machine to check for _, and build the escaped sequence. Everything is passed through a dictionary which maps English sequences to Greek letters.

Here is the Greek to English map:
set greekDict {α a β b γ g δ d ε e ζ z η _ae θ th ι i κ k λ l μ m ν n ξ x ο o π p ρ r σ s ς _s τ t υ u φ _ph χ _ch ψ _ps ω _oe}

Greek can be converted to English with a simple function:

string map $greekDict $s

This is because each Greek letter is all by itself.

Building the English to Greek map is easy:

foreach {g e} $greekDict {
  dict lappend revGreekDict $e $g
}

Where the Greek to English map is a simple list (where the first subelement is one Greek letter, and the second element is the English transliteration), the use of multiple English characters requires a full map (or "dict" in Tcl).

Strike Force

"Strike Force" (Dale Brown)(audio) - Brown must have a lot of fun writing these books. The protagonist is an American general in charge of the "Air Battle Force" (basically guys with rudimentary orbital dropships and power armor). He is so aggressive as to make Douglas Macarthur look like a peacenik.

In order to make this guy look sensible and righteous, the evil guys are almost comically evil (although, they are in Iran, so it's not entirely unbelievable). Many explosions ensue (including a ground to orbit laser attack from the Russians! pew! pew!).

Book Review Coming

I have been asked to review a recently published book on Tcl (the cover is below)

I'm always glad to see any interest in Tcl, so I am looking forward to this.

Houston, we have a keyboard

Nice!

I had a feeling once I got past the interrupt barrier things would move a lot more smoothly.

I quickly got the keyboard interrupt working (it puts the scancodes into a queue and returns as quickly as possible).

Then, I changed the final idle loop into a loop to empty this queue. Right now, it is printing the scan codes - you can see the keyboard scancodes (in hex) for "Shift, h, i, space, k, e, y, b, o, a, r, d". I am ignoring all break codes, so you don't see the Shift key released.

It was a real pain getting the screen shot for this one. I couldn't hit print screen in Bochs, no telling what that funky scan code would do to my handler :) I had to make a tiny notepad window active (you can see it peeking over the left edge), and do a full screen capture.

How Things Work

and Stuff. (At CMU, CFA - which some people said stood for "Can't Friggin Add", while others said it stood for "Can't Friggin Spell" :P - had three courses, the first "How People Work", and the second "How Things Work", and the third "How People Work and Things").

I've been studying computers for a long time. I got started seriously the summer I visited colleges (between Sophomore and Junior year in High School). I got a book on DOS programming from the MIT bookstore, and read it the whole rest of the trip.

It's only today that I figured out why the "mouse" and "keyboard" PS/2 connectors on the back of the computer are different!

You know which ones I mean. The are different colors, with the plugs on the mouse and keyboard color coded to match.

I always looked at them, and said "Why should it matter?" The plugs are the same, why can't the computer just figure it out!

Then I saw the figure at the OSDev Wiki page for the keyboard.

The IRQ line from the keyboard connector is hardcoded to IRQ 1, and the mouse goes to IRQ 12. There is no identifier from the hardware to easily figure out which is which (they both send bytestreams of input, worse they send the data to the same port). You might be able to decode which is more likely, but it isn't easy. Then you would have to switch your interrupt handlers. I guess most people don't bother.

Interrupts Online!

Wow! That was a pain!

This isn't much to look at, but it is vital to making any more progress. I now have interrupts turned on - the interrupt handler increments r9 and returns. You can see r9 is 1, then four interrupt/exceptions occur, then r9 is 5.

There were a lot of barriers to this:

1. Several system tables had to be filled in: the IDT, TSS, and IOAPIC redirection table being the biggest ones (I am making things somewhat harder on myself by skipping the legacy interrupt mode). The TSS required another entry in the GDT. Also, I had to add a page table entry for the LAPIC, and set it up as well.
2. I had to go through several rounds of IOAPIC settings. At first I was using lowest priority, logical, active high. Then I tried ExtInt. Finally, I figured out that I needed physical mode (the default LAPIC id is 0, which doesn't match any logical bit setting - I peeked at the Bochs LAPIC implementation!).
3. I encountered a couple of triple faults. The first was due to trying to poke directly into upper memory region (mov [0xfee0_00f0], 0x100). The immediate value gets sign extended, and ends up off the end of virtual memory. Another was due to using segment selector "1" for my interrupt handler (because I want entry 1). Of course, the bottom three bits of the selector are shifted off, so I need "8".
4. You can't just iret from a 64 bit interrupt handler. For some reason, you have to hard code in the word size override ("db 0x48; iret").
   

Amanda Knox

I just finished watching the Lifetime movie about Amanda Knox (the American student charged with murder in Italy).

First off, way too soon. The appeal is still underway!

Second, I admit to going into it thinking she was totally guilty (yea, I'm a cynic - I prefer the term "realist"). If the goal of the producers was to portray her as guilty (which seemed to be the complaint some people had), the producers failed.

If the goal of the producers was to make the Italians look inept and corrupt, they succeeded beyond their wildest dreams.

My biggest take away was that the Italian police MO is similar to that of the Inquisition. Round up some likely suspects, torture them into confessing and implicating others; pick up those implicated; rinse, torture, repeat.

Sci Fi

I have found the most remarkable collection of SF short stories. It is called "The Oxford Book of Science Fiction Stories". It has stories from 1903 to 1990.

Just the first three stories:
"The Land Ironclads" (H.G. Wells, 1903) - a surprisingly prescient description of tanks used in trench warfare. Of course, in researching this post, I checked Wikipedia, and it cites this story... They fail to mention that Wells was somewhat off on the scale:

"It might have been from eighty to a hundred feet long"

He also describes them as using automatic weapons (which had been deployed in 1889 according to Wikipedia).

They also had an interesting fire control and "fly by wire" (or fire by wire) system:

"sights which threw a bright little camera-obscura picture... marked with two crossed lines... A little twisted strand of wire like an electric-light wire ran from this implement up to the gun... when the land ironclad moved forward the sights got a compensatory deflection in the direction of its motion"

That last bit is what's is called a fire control system and wasn't deployed until WWII (on ships as analog computers, and in planes as the Norden bombsight). I don't think it was until modern times (say, Abrams) that tanks got them.

"Finis" (Frank Pollack, 1906) - An interesting story that reminded me of a story where the sun releases a huge flare that cooks the planet. In this version, scientists calculate that there is a giant sun at the center of the universe, whose light is just about to reach Earth (after "a thousand years"?). This is interesting because the idea of a limited universe didn't become popular until Hubble's investigation of 1919-1929.

"As Easy as ABC" (Rudyard Kipling, 1912 - expansion of a story from 1909) - Most people know of Kipling for his poems, and "Jungle Book". I also remember him for Gunga Din. I never knew he wrote SF! And some remarkable stuff:

"An ABC [Aeriel Board of Control] boat does not take the air with the level-keeled lift of a liner, but shoots up rocket-fashion like the 'aeroplane' of our ancestors" (the story is set in 2065).

The ships appear to be some sort of blimp (there is reference to floating and mooring). They also travel a shocking "320 m. p. h.", and cross the Atlantic in 10 hours - not bad considering the Wright brothers had just gotten started in 1900.

There is also a very weird society - world population is down to 600e6 (in 1900, the world population was around 1.5e9). People no longer gather in crowds, and there is no voting for things (sort of a hyper-individualism).

The story involves a group of "voting evangelists" who trigger a riot. The ABC is forced to suppress the rioters with sensory overload (intense light and sound). There is also an interesting sort of force field ("ground-circuits") which can slow or stop movement.

Stuff I've read lately

"Nebula Awards 33" (Connie Willis ed.) - They've changed the format, and I must say I like it. Each story is followed by a note from the author describing the conditions that led up to writing it. It's interesting how good stories come from flimsy circumstances, and less interesting stories often come from well thought out approaches (and vice versa). Also, there is a classic story from the golden age of SF.

Stuff I've read lately

"Act of War" (Dale Brown)(audio) - I have to wonder who is the target audience for Dale Brown's work. Do they appreciate the cheesiness? Or do they actually take it seriously? I mean, the "good" people are all super-good (and beautiful) - while the "bad guys" eat kittens (and are usually ugly and scarred).

That said, I did have to appreciate this book codifying the phrase "he's got the atomic bomb!" (said in the same vein as "he's got a knife" or "he's got a gun"). Not that anyone actually said it, but Brown set up the situation.

Brown did manage to set up a surprise at the end, although the voice acting gave it away somewhat earlier than would probably come about from just reading it.

Aerostats

Interesting news article I found in my archives:

"This so-called graphene sealed microchamber is impermeable to even the tiniest airborne molecules, including helium."

Fiction involving nanotechnology often has objects called "aerostats", basically floating machines (usually equipped with fans or jets for movement).

If the balloon wall is impermeable to most atmospheric gases, then the balloon might be able to be "filled" with a vacuum (assuming air pressure won't collapse it). Making it the best lighter-than-air material.