Sunlight Labs

Quantifying Data Quality

You've already heard me complain about data quality -- how it's a bigger problem than most people realize, and a harder problem than many people hope. But let's not leave it there! Perfect datasets mostly exist in textbooks and computer simulations. We need to figure out what we can do with what we have. In this and other posts, I hope to give the developers in our community some idea of how they can deal with less-than-perfect data.

The first step is to figure out how bad things actually are. To do that, we'll use some simple statistics -- those of you with a strong stat background can skip to the next entry in your RSS reader (or better yet, correct my mistakes in comments).

More About the Door

The above video -- put together by Noah, Ali & Greg, and featuring star turns by Daniel and Luigi's phone -- shows the current state of the door project I wrote about on Tuesday. It's working pretty well! I think I still need to add a bypass capacitor to improve the circuit's stability, but it's certainly good enough for our uses.

But the electronics are just one part of the system. As I mentioned at the end of that last post, my colleagues did an impressive job of springing into action and building out the systems necessary to turn an SSH-accessible script into a useful interface. Here's how they did it.

Our Door Opener (A Science Project)

Life in the labs has been pretty good since we moved into our current offices. Before, we were spread out over two floors: my team was upstairs in a stuffy law office sublet, and the rest of our colleagues were stuck in a homey but increasingly cramped and run-down space four floors below. Since moving everyone to the third floor we've found ourselves with plenty of room, lots more light and a nicer kitchen. It's just a more pleasant working environment in general.

But there's always room for improvement. For one thing, the new space came with new locks -- ones with really expensive keys. Issuing keys to the entire staff wasn't practical, and coordinating door-opening responsibilities in a way that accommodated team members' occasionally odd schedules was inconvenient. Fortunately, the space also came with the button you see to the right.

Located near the reception desk, this button opens an electronic latch on the front door. Pulling the assembly out of the wall revealed the system to be about as simple as possible: the button simply connects two wires. Bridging them with a screwdriver fired the latch (from their small gauge and uninsulated connections, it was obvious we weren't dealing with dangerous voltages, but please don't start pulling cable from your walls unless you know what you're doing).

Connecting two low-voltage wires electronically isn't a particularly hard trick, so I decided it'd be fun to spend some evenings building a system to expose that switch to our network.