03.26.04

A Message From The George Herbert Walker Bush Houston Intercontinental Airport

They insist on calling the main airport in Houston "Bush Intercontinental", which sounds like a Texas-size case of overcompensation to me. Heaven forfend we should assume that planes from Houston only fly to Canada and Mexico, after all - they go to other continents, continents less fortunate then ours, since they don't have a Lone Star State.

So Bush Intercontinental it is, and it is a spacious and pleasant new airport, except for the following unsavory announcement that you will hear repeated every ten minutes or so, buried in among warnings about leaving baggage unsupervised. I paraphrase from memory - I'd be grateful if anyone can send me the verbatim text:

The Homeland Security Office advises you that anyone making jokes or inappropriate remarks about terrorism WILL be arrested.

Which is the most un-American thing I've heard in a long while. There is no conceivable way an announcement like this makes us safer; all it does is create the illusion of a stern and vigilant authority, and we had better not step out of line. To some people (third graders come to mind) that may be a comfort, to the rest of us it's noxious.

Sooner or later there's going to be another big terrorist attack in the United States - maybe by Al Qaeda, maybe by some completely different entity. We need to be adamant about resisting the temptations of authoritarianism now, because the momentum for curtailing our liberties right after the next Oklahoma City bombing or September 11 will be irresistible. The passengers at Bush Intercontinental display backbone every time they fly (who has forgotten the sight of those planes?). I wish the airport authorities would display the same level of backbone and refuse to participate in fearmongering.

03.19.04

Introducing Loaf

For the past few days, Joshua Schachter and I have been working on a neat application of Bloom filters, an ancient and underappreciated algorithm from the days when men were men, women were women, and computers read data from perforated stone tablets.

The fruit of our labors is a little social network extension to email we call Loaf. You might say it's a breadfruit.

Configure Loaf on your machine, and suddenly you are attaching your address book to every outgoing message. And through the miracle of 1970's computer science, you're doing it in a way that doesn't completely destroy your privacy.

Loaf tells you whether strangers sending you email are people that your friends know. And it can tell you a little bit about the structure of your own social network. And it can probably be used in cool ways that haven't occurred to us yet.

For the moment, Loaf is just a reference implementation for Unix systems, and we need serious hackers to help us write email plugins and installers for the rest of the world. But a friendlier, happier Loaf should be just around the corner. You can read all about the project and track its progress at loaf.cantbedone.org.

03.15.04

Your Literary Masterpiece Was Delicious

For the past few days I've been working on a little project that reads Russian novels and converts them to pretty pictures. Specifically, it turns them into pastel-colored graphs that show you the contextual relationships between characters, based on how many times their names occur together in the text.

The point of the exercise is to try and create character maps for novels that can then serve as clickable, interactive maps to the text, and to do so with as little supervision as possible. Combining such an interface with flexible, full-text search and annotation tools could open up a whole new way of interacting with literary texts.

Here, for example, is a section of the graph for Mikhail Bulgakov's "The Master and Margarita". The size of the rectangles indicates the relative importance of each character, and the thickness of the connections shows you the strength of the relationship between them (you can click through to a full version of this graph if you have the SVG plugin):

This project poses an interesting problem in natural language processing - given a text, how can you isolate the names of characters? Is it possible to do it just based on capitalization patterns? And how do you resolve characters that have multiple variant names? For example, the character of Pontius Pilate appears in the Russian text as "Pontii Pilat", "Pilat", "prokuror", "hegemon" - is there any way to flag the many variants as one?

If you're working in Russian (as in this example), you also have to face the issue of normalization (Russian names can have one of seven case endings, which depend on gender and number), and in whatever language you work in, you run into the problem of anaphoresis (figuring out which antecedent words like "he" and "it" refer to, sometimes over a span of many paragraphs). The key to success in NLP is knowing where the computer should apply brute force, and where it needs to call in a human being to help with the hard bits. For example, to generate the graph above, the machine asked me to validate some guesses it made ("Is 'Margarita' the same person as 'Margo'?"), and then I had to give it a final push over the cliff, as a Spinal Tap fan might say, by disabusing it of the notion that "Petersburg" was a proper name, or that "Ivan" and "Bosoi" were two different characters.

More importantly, this kind of project poses an interesting cultural litmus test. To a computer hacker, treating a literary work like a data stream and mining it for patterns is an amusing challenge, but there are many people who would frown on this kind of thing as a reductionist attack on a living text. When I was first learning Perl, I remember writing a toy program that read all my letters to my girlfriend, and her letters to me, and spit out lists of words that only one of us had ever used. I thought it was a neat trick, while she was livid with me for treating our correspondence as just so much raw material for my computer games. It was a surprisingly touchy topic.

For as long as there has been writing, there have been cabalists eager to quantify, rearrange, and play with texts to reveal hidden inner meanings. The humanities have an understandable suspicion of such textual games, and a less understandable reluctance to explore the many worlds that fast computers and the mass digitization of texts are opening up. But as natural language programming techniques improve, the humanities are going to have to come to terms with computers, whether they like it or not. We geeks are going to eat all of their favorite books.

eachers are already battling the problem of Google plagiarism, or in its mild form, Google-only research. And soon it will be possible to auto-generate meaningful summaries and essays on many topics with minimal effort - software already exists that grades essays as well as a human TA, it's not hard to imagine a nice NLP essay generator that would withstand the scrutiny of a harried graduate student with two hundred papers to grade. The only way around this challenge will be to adapt to the new technologies, or find ways to successfully exclude them. Just like the programming example I gave, the trick will be finding where to draw the line between the computer and the reader. It will be interesting to see how the humanities cope with the problem that math ran into twenty years ago, with the advent of the cheap pocket calculator.

My own interest is finding a way to make these pretty pictures useful in some way to the reader. For example, the image above links to a full-size version of the graph in scalable vector graphics (SVG) format, which among other things lets you create hyperlinks to web pages and interact with the drawn image. So you can do things like link every edge in the graph to a page that displays all the paragraphs where those two characters in the book interact, or even start to explore connections across many books (consider the grand cycles of Balzac or Zola). Perhaps scripted visualizations like this will open the texts to more intensive study, and enable some kinds of scholarship that up to now have been very tedious. There's already been a small revolution in lexicography thanks to automated corpora, why not extend it to literature?

I'm happy to announce that my talk proposal on hacking literature has been accepted for this year's O'Reilly Open Source Conference, which takes place in July. If the topic is something you find interesting, I hope you will come to the presentation, or download the notes and code from this site as they become available. And please email me your own "wouldn't it be cool if" ideas about visualizing literature, so I can steal them and win glory and fame. I'll keep posting little demos (and begin posting links to code) as time allows.

For the technically curious, this graph was generated by an unholy combination of Perl, neato graph layout software), SVG and the amazing lib.ru website, which has full-text copies of every Russian novel you can name. Because of the way Russian copyright law works, you can use the texts for educational purposes without restriction.

03.01.04

Warszawa

An old joke: Two trains pull in to Warsaw Central Station; one of them is the westbound Moscow-Paris express, the other is the return train from Paris, heading in the opposite directon.

A Frenchman steps out of eastbound train, thinking that he's reached the terminus, and walks out into downtown Warsaw. What he sees roots him to the spot. "My god, the filth! The poverty! Look at these monstrous buildings, the ratty little cars, the clouds of black smoke - look at the people standing in line, in such cheap clothes - the shelves bare in every store! Moscow is every bit as desolate as I expected!"

Meanwhile, a Russian gets out the westbound train, takes one look around and cries "Ah, que c'est beau, Paris!"

As with so many things in life, you will get much, much more out of a visit to Warsaw if you take appropriate care to minimize your expectations. A backpacker freshly arrived from Prague, Stockholm, or (God forbid) Paris might feel an understandable sense of panic wandering into the raspberry confection we call Okęcie International Airport, or taking the first few slushy steps out of Warsaw Central Station into the ugly building contest we call downtown. But by simply adopting the Polish national motto ("it could always get worse"), the prudent visitor can inoculate himself against all manner of disappointment. And after a couple of donuts, a bus ride or two on the scarily modern new coaches that wish you a happy name day in glowing LEDs, and a few strategically timed glasses of hot spiced wine, Warsaw will start to acquire a happy, rosy glow.

The saving grace of the city is that there are any number of interesting nooks and crannies worth exploring. And once you leave the heart of downtown, the entire stretch along the Vistula is genuinely beautiful, from the Citadel through the Old Town, all the way down to Łazienki park and south to the old royal residence at Wilanów (e.g., ville neuve). Warsaw University lies right in the middle of the New Town (like every New Town in Europe, one of the oldest parts of the city), and masses of students help keep the city alive, awake, and interesting. And because this is Eastern Europe, the national capital naturally lives serious cut above the rest of the country, with a pleasing whiff of cosmopolitanism that you won't find in, say, Grzęzidło Dolnośląskie. There are even hipsters now - a whole posh row of trendy cafs and restaurants tucked in behind the Aleje Jerozolimskie - so that the smug Americanized expat no longer feels like he is at the top of the coolness pyramid. When the summer months come, and constant darkness gives way to the beautiful late nights of June, it is hard to be unhappy in Warsaw. The locals manage, mind you - dissatisfaction is a natural sport - but it takes hard concerted effort, and a good helping dose of misgovernment, one of the few things that has never been in short supply.

I was born and very slightly raised on the western side of the city, in a district known as Ochota. You would not have recognized me back then - I was a little blond kid, spoke no English, and had a lively interest in automotive trivia that has disappeared as completely as the blond hair. Warsaw was somewhat slim pickings for an automotive buff in the late 1970's - there weren't many cars on the roads, and what there were tended to be unimpressive socialist putt-putts - but I perservered and had pretty much the whole line of Eastern Bloc models memorized by the time we left for America, that automotive paradise. My other interests at the time involved rocketry (specifically, pretending to be a pilot steering the carcass of my prone sleeping father), ham sandwiches (rationed!), and elaborating complicated and probably workable plans for how to trap a street pigeon in a shoebox and take it home to be my pet.

My parents' apartment wasn't far from the Central Station referenced in the joke. It overlooked a middle school playground - a good place to play badminton with my grandmother, but also home to fierce tribes of giant older children that I found terribly intimidating. There was a grocery store across the street, a playground not far from the back entrance, and a set of parallel bars in the courtyard where every Saturday you could hear the rhythmic thwack! thwack! thwack! of women beating the dust out a defenseless carpet. My closest brush with the yoke of totalitarian rule was at the state preschool, where I was forced to wear scratchy wool tights and eat foul foods like rice with apples by a certain Miss Lusia.

The Warsaw skyline of my childhood consisted of exactly one building - the Palace of Culture and Science, the ultimate in unreturnable gifts, built for the grateful inhabitants of Warsaw by our fraternal allies to the east. The Palace is an oppressive sixty-story erection of cheap sandstone in the architecural style known as Stalinist Gothic, or Stalinist Wedding Cake. In the total absence of other tall buildings, the Palace quickly became an icon of the city, squatting on an immense open square with inspirational Orwell-in-toga statuary of workers' heroes arranged in niches around its base. It may not have been completely original (Moscow has eight of the things), but it was definitely memorable in its ugliness, and a good place to catch a concert.

There was a moment in the immediate aftermath of 1989 when the Palace's fate was uncertain, so it's pleasing to see that it survived the initial backlash and is still standing, alive and well. It is beginning to assume a campy, retro-socialist feel, and irony becomes it. When I made my first trip back to Poland in 1990, the Palace was still rather filthy and unlit, except for a giant floodlit billboard towards its spire (symbolic of the glued-on capitalism of those years). Later in the 1990's, it seemed to be falling into disrepair - the plaza at its base turned into a permanent bazaar, and for a happy few years, two of the neon lights in the theater lobby marquee stayed broken, so that instead of spelling out "Palace of Culture", at night they brightly shone with the words "Palace of the Cult". But now the Palace has been scrubbed, ornamented with a nice bright set of lights, and generally made to look like a respectable citizen of the urban landscape. At least after dark.

It helps that the Palace is no longer the only high-rise in the entire capital. There was a little building boom in the 1990's, during which architects seemed intent on proving that capitalism could build things just as hideous as socialism had, and do it in a tenth of the time. So in the space of the decade, the city acquired a little patch of provincial skyscrapers, each one designed in five minutes on a blank sheet of graph paper, and each one clad in reflective glass to better mirror all the others. The net effect has been more glory for the Palace, which at least now has a crowd to stand out from, and mercifully absorbs most incident light.

Like so many things in life, responsibility for Warsaw's architectural trouble rests squarely with the Germans. Not only did they design much of what went up in the 1990's, but they were responsible for setting the city up with so much vacant land to begin with.

To say the city had a rough time of it during the war would be putting it mildly. Warsaw effectively ceased to exist, going from a bustling capital of 1,300,000 on the eve of the fighting to an uninhabited expanse of ruins when the Red Army finally arrived. While the city was heavily damaged in the 1939 attack, and lost a whole section of its downtown during the liquidation of the Ghetto in 1943, the real destruction did not occur until August of 1944. That month marked the start of the Warsaw Uprising, a general insurrection that lated for three months, and ended with the city's entire population either killed or deported.

The idea behind the uprising was to tie up German forces, speeding the advance of the rapidly approaching Red Army, as well as carve out some kind of official status and future political role for the underground Polish resistance before the Soviet armies took over completely. It was the largest resistance action fought during the war. Most of the soldiers in the underground Home Army were veterans with a vivid memory of the Soviet invasion in 1939, and most were aware that the Soviets had earlier executed the bulk of the Polish officer corps in Katyń Forest. So they had few illusions about their likelihood of ultimate success, even if they failed to anticipate the full extent of Stalin's cold-bloodedness.

For Stalin, the Warsaw Uprising was an excellent opportunity to be rid of some troublesome political opponents. Although advance units of the Red Army were already in the eastern districts of Warsaw when the fighting began in August, they halted for almost five months, giving the Germans time to bring in thousands of SS troops to quell the revolt. It was a horribly bloody affair - all told, the city lost nearly two hundred thousand people, most of them civilians massacred in cold blood. Those left alive when the hostilities ended in November were deported or sent to labor camps in Germany, leaving the city completely empty. Hitler was so enraged at the insurrection that he ordered the methodical destruction of what remained of Warsaw, and the engineers sent in to accomplish the task were able to finish their work before the Russians finally crossed the river in January of 1945. Eighty-five percent of the city was razed.

Because the destruction was so total, paradoxically there are few traces of the war left in Warsaw, far fewer than you might see in Kraków or Łódź. The only reminders are the several large public monuments, along with the innumerable, numbing plaques on nearly every street corner, indicating the number of civilians shot in reprisal killings on the spot on such and such a date. The numbers grow as the war progresses, the events the plaques describe impossibly remote even though the war remains in living memory.

Poland voted to restore the capital in an otherwise dubious 1948 referendum that also brought in sweeping constitutional change. The decision to rebuild was not a simple one - the country had lost a third of its population and sustained damage estimated at 18 times the 1939 gross national product, so devoting the nation's resources to rebuilding the capital meant real sacrifice. There was some talk of moving the capital back to Kraków, or even rebuilding the city somewhere other than on a gigantic pile of ruins that would have to be cleared. Still, the reconstruction went ahead and proved to be one of the most genuinely popular actions by the newly-installed socialist government (the Red Army was kind enough to bring a Polish government with it, saving the trouble of having to fetch the old one all the way back from London). Postwar photographs and films of the capital show a lunar landscape - entire streets invisible under mountains of rubble - with long bucket brigades of people painstakingly clearing the debris and picking out unbroken bricks for reuse.

Unlike the city of Rotterdam, which suffered similar devastation, it was decided to rebuild at least a part of Warsaw as it had been before the war, and so architects and planners meticulously collected old photographs, paintings, maps and other documentary evidence needed for a faithful re-creation. A series of 18th century cityscapes proved particularly useful. In the end, a large swath of the medieval Old Town and 18th-century New Town were rebuilt, making Warsaw a little bit like the joke about the 200 year old axe, where the handle has been replaced twice and the head three times. You find yourself walking through a four hundred year old square that was built all of fifty years ago. The Royal palace, with its sumptuous interiors and stately air of antiquity, was built a year before I was born.

All the war statistics and numbers seem impossibly abstract. It's a jolt to think that only fifty years separate here from then. My grandfather, alive and well at 91, remembers it all, and every time I go back I try to note it down, imagine it through his eyes, and otherwise fix the unreachable past in my own memory. But strange, misplaced nostalgia is a symptom of emigrating too young, and I'm glad my own city is moving on. In five years it will be like any other second-string European capital, miraculously, gloriously normal. It makes me happy to see it.