Leisure Reading: The Existential Pleasures of Engineering

I was browsing the book selection at the Harvard Coop a few weeks ago for a graduation present and came upon this little gem. The Existential Pleasures of Engineering by Samuel C. Florman is a book about the engineering profession, how it came to be, where it is now, why its important, and just what makes it special. The title in particular is what caught my eye because when I thought about it I couldn’t think of better words to describe what I feel when I’m doing meaningful engineering work. To put it another way, I enjoy my work not because it’s easy but because the act of creatively solving complex problems is intrinsically rewarding to me. On the inside, it was Florman’s timeless insight into the issues that face our profession and where we stand in society that kept me reading. It taught me a lot about my professional responsibility to society and gave me a greater appreciation for the work we do. Hands down, I strongly suggest this book to anyone passionate about engineering.

It starts out talking about the rise of our profession and the golden age of engineering to really hook the reader in. This golden age being the period of time between the 1850’s to the 1950’s where we began to see the completion of macro-scale engineering works. You get a sense of how young and naive the profession is compared to the institutions of business, law, and medicine. That’s not to say that engineers haven’t existed for thousands of years, but rather the institutions of the profession is still in its infancy which is one of the main themes of the book. Because the institutions are weak, engineers are unable to truly fulfill their duty to serve society.

The second part then goes into a long discussion on the claims of anti-technologists, in other words the people against the wide spread application of engineering solutions. They argue that the rise of technology has not only decreased the quality of human life but in many ways made it worse. Further, they argue that it was the engineers that brought this on society. Their argument primarily centers on the negative externalities of technology such as pollution or the creation of powerful weapons and the idea that old simple lifestyles of the past were inherently better. Fact of the matter however is that the fault lay not in the engineer, but the choices made by society which are more often than not poor in judgment. Sad thing is that these beliefs have gained much traction with society because the arguments are so plainly laid out in language that’s easy to relate to and understand. Due to the naivete of the engineering organizations we have not mastered the organizational skills to mold the sentiments of society. While a politician can empower a nation to spend billions of dollars to land a man on the moon, engineering institutions can barely lobby enough funds to maintain critical infrastructure. For our profession to be successful in our mission to serve the people, we must be able to take a unified stance on issues and address the anti-technologists.

Florman then ends this piece on suggestions for the engineering profession and the adoption of an existential philosophy. A philosophy that hinges on the idea that engineers choose the profession because they find interest in conquering the natural world. What makes engineering special is that we don’t choose the profession for prestige or materialism. If that’s what we wanted, we would’ve taken up another profession with just a fraction of the effort. However, the psychological rewards of engineering are far greater in value to the individual who pursues them. How this applies to our profession is that it’s not really an ethical guideline but rather a contract to take pride in our work. We are to be professional in our conduct, in that we must deliver the best we can based on all possible considerations given the resources available. A good existential engineer must be well aware of the consequences of his/her actions and must be willing to take responsibility for them (Whereas other professions excel at placing blame).

Rsync in a Week

This past semester I had the luxury of taking a course in Advanced Data Structures part time as I slowly work on my master’s. It was probably one of the most difficult classes I’ve taken so far. Given my background focusing on the lower half of the technical stack and having never taken a course in algorithms, it was a rough start. We just covered so much material and I spent a lot of time just trying to reabsorb it all. For those in the know, we went through CLRS cover to cover in a single semester. Most of all though, it was a test of my discipline as it was the first course I’ve taken outside of undergrad while working full time. I learned just as much about time management as I did about data structures. Overall, it was a rewarding experience and I feel that I learned many invaluable skills from this course and I’m happy to have taken it.

My favorite part of the course was the final project. We were tasked with implementing what’s known as a File Reconciler which takes a file on one computer and recreates its changes onto a similar file stored elsewhere. The idea is that this should take a minimal amount of bandwidth compared to simply sending over the file in its entirety by only sending differences. At first it was a daunting challenge, but with a firm understanding of running times and data structures the problem suddenly became very tractable. I took on the project lead by delegating work, architecting the software, and most importantly implementing the core algorithm behind our program. The latter being self-imposed after I discovered a fundamental flaw in my peers’ approach during a code review.

I modeled our program against Rsync which was a utility written in the dial-up days to handle synchronization of large code bases over high latency low bandwidth networks. It basically works by splitting a reference file into blocks that are then matched on the file we are trying to reconcile. Then we come up with a list of steps that either provide literal data or a reference to a matched block. The brilliant part of the algorithm is how it matches blocks. For each block, a pair of independent hashes(MD5 and Adler-32) are generated that uniquely describe it. The probability of two random blocks having matching signatures is on the order of 10^(-100). MD5 was chosen because it’s a strong hash that generates a unique value that isn’t likely to collide in the same file. More importantly, Adler-32 was chosen for its rolling checksum variation. It has the wonderful property of constant time computation on single byte offsets and incorporates byte order into its entropy. This allows us to slide through a file and compute each and every possible block and search for a match. On a match of the weak, we can then compute the strong and verify. With the two hashes, the algorithm is both computationally cheap while being robust against hash collisions.

By far, the most interesting exercise of the project for me was the chance to take a concept from an academic paper and turn it into code that actually works. Further, due to time constraints, I had to do this in about a week using only my few hours after work. Usually I read papers for the high level theoretical understanding, but rarely is it ever the even deeper understanding of translating academic jargon into a usable object. After understanding how it all worked, I then had to begin the greater engineering challenge of deciding what parts of the algorithm to implement. This was a question of what is an essential component, what gave us the most performance for the effort, and what I could conceivably implement given my intellectual resources. Most notably, the thesis provided many suggestions for using parallelism to improve performance, I felt that it would add too many risk factors and development effort that I could not spare. It’s also important to realize that while performance is great, the project requirements (50MB of bandwidth) were trivially met by even the most naive implementation of Rsync.

To present my work I prepared a few slides to illustrate what Rsync does visually, I am by no means a graphic artist but it does the job. My goal was to highlight the data structures which data structures were used, how they were used, and why.  These were included in a video created by my group for the class’s critique. I apologize in advance for the awkward choppy audio. This is due to me having to heavily edit my fellow group members’ audio tracks to fit within the 10 minute presentation window.

Github Project Link

This slideshow requires JavaScript.