Guess I'm a little late to the party, but I figure that its still worth talking about. Throughout the summer, Cisco Systems intern Greg Justice has been releasing a bunch of videos where he claims to be the world's most interesting intern, and surprisingly, he's managed to gain a remarkable amount of popularity and even inspired numerous video responses.  I don't know Greg personally, but like him, I too am a Cisco summer intern at the San Jose campus (along with a few hundred others). Here's a few of his videos:

• http://blogs.cisco.com/news/comments/i_am_the_worlds_most_interesting_intern/ (this was the first one)
• http://www.youtube.com/watch?v=TphCCh5iuTU
• http://www.youtube.com/watch?v=MKZk7Xlhvsk&feature=channel
• http://www.youtube.com/watch?v=-87j7xWJUdw&feature=channel

Quite frankly, I'm amazed that he's managed to garner so much attention, since his videos aren't exactly gut-busting hilarious, but rather, just simply amusing. But of course, lamer things have somehow managed to gain more popularity on the inter-webs (I'm looking at you double-rainbow-man). I'm not going to make a case that I'm the world's most interesting intern (I know I'm not) but as my internship at Cisco draws to a close, I figure it might be worthwhile to at least mention some of my experiences this summer.

The work has been intellectually interesting, which is more then I can say for some of my previous internship experiences, and I'm happy to say that I was not relegated to the post of code monkey, although I did pump out a fair bit of code. Whether or not I made a positive contribution to the company as a whole, I cannot truly say, since  some of the aspects of the product I'm working on are not set in stone and if product specs change again my make may have to be discarded. Overall the work environment is fairly nice, the other engineers highly intelligent and helpful, and the management friendly and unobtrusive, so I cannot complain about this summer. My greatest fear, as a software engineer, is to be left in an uninspiring occupation, banging out unoriginal code for rarely used and uninteresting programs. I fear that I may become an out-source-able code monkey. It is often felt that large companies, like Cisco, that have literally buildings filled with engineers, are often prone to relegate their engineers to excruciatingly boring and tedious code-monkey-like tasks, and treating them like cheap, interchangeable workers in a factory. But I'm happy to say that this was not the case for me. So high-fives all around...

Also, might I just say, that the laptops they gave us interns (not to keep of course), are insanely powerful. Which is a little odd considering that we do almost all our development work on the servers, which are even more powerful.

Stability has always been a much touted feature of Linux in general, but some distributions lay a greater claim to that attribute then others. Arch in particular has tended to be slightly more bleeding-edge then other distros, sacrificing stability for the newest features; packages are updated in the repository as soon as new versions are released and with  a relatively minimal amount of time (extremely minimal compared to other Linux distros like Debian) spent in testing in an Arch environment, just enough to ensure that the packages don't completely break the system. While this strategy has its benefits, namely that it allows users to get the latest and greatest software right when it comes out, it comes at the cost of stability (and security to some degree). And the more packages that I've added to my system, the more I've started to notice just how unstable Arch can be.

I generally run "pacman -Syu" to do a full system update at least every week, and I try not to let my system stay without an update for three weeks at the longest, so in general I'll stay pretty well up-to-date. But it has not been uncommon, that after performing a full update, that my system completely locks up or goes completely nuts. Take for example, just a few weeks ago, when a full system update made my Arch Linux partition completely unbootable and required that I boot a live CD and futz around in the configuration files. When my laptop was finally usable again, I had to mess around some more with my wireless drivers to get them working again. And lately, after my most recent system update, I've been having some problems where my laptop will occasionally freeze up and become totally unresponsive to everything except a hard reboot, and system logs show no behavior to be out of the ordinary. Of course, not all of the breaks in Arch have been this bad. About four months ago, a system update made it so that I could no longer hibernate, a problem which was easily remedied by a quick visit to the Arch wiki and a few short commands. Sadly, the list of weird errors goes on (although its not that long).

Two years ago, when I was moving off of Debian testing, Arch's bleeding edge packages were quite welcoming, but not that I've matured a little bit, I don't care as much about the latest features (Lets face the facts, the programs I've been using the most these past few weeks, have been vim, GCC, SVN, and Zoom). My first priority these days, is getting shit done. And if my laptop decides to go bat-shit-crazy now and then, it seriously hampers my ability to work properly. I don't mind a few bugs now and then, and I could probably even live with a rare kernel panic, but sometimes I get the feeling that Arch is maybe just a little too bleeding edge for me.

I mentioned earlier that another one of the costs of having the latest and greatest software, is security. A lot of the newest software releases tend to be not as well hammered out and therefore are slightly more prone to have security holes. I'm only a slightly paranoid Linux user, so while the lack of security is a little worrying to me, its not a huge deal breaker. Arch's lack of solid support for more powerful RBAC security modules like SELinux or AppArmor has also been a little worrying to me. I would love to be able to slap on some powerful RBAC policies on my laptop to give me greater piece of mind, but Arch's normally awesome wiki is a little lacking in help (although it seems that reccently, the SELinux page has gotten a little more meat to it).

This next point is a rather silly and illogical thing to hold against a distribution, but I feel that it needs to be said because its entered my thoughts a few times in the past year. Whenever I go in for an interview, I generally try to play up my Linux expereince (which is not incredible, but still fairly impressive enough). The logical question for an interviewer to ask of course, is "what distribution(s) do you use?" As soon as the words "Arch Linux" comes out of my mouth, I can see the interviewers knocking some points off of my interview. People always assume that Arch is just another one of those random "edge" distros that is basically just an Ubuntu/Fedora knockoff with some sparkles thrown in, and no maybe how much I explain it to them, I know that they don't respect an Archer as much as they respect a Slacker. So yeah, I'm a little shallow, but I do care about what people think about me, especially in interviews. A part of always wishes that Arch was just a little more mainstream and a little more well known.

So I've taken some pretty mean shots at Arch, but my comments shouldn't be miscontrued to indicate that hate I Arch. Quite the contrary in fact, I've loved using Arch. My old Arch review enumerates out more clearly the points of Arch that I really like, but I'll list them out here quickly.

• Fast! - Compiled for i686 and lightweight with no extra cruft thrown in
• Clean - there is nothing on my Arch system that I didn't put there
• Simple - things tend to be very straightforward and elegently simple
• Awesome documentation and user community - Arch's comprehensive wiki is in my opinion, one of its strongest selling points, also the forums are quite helpful.
• Rolling updates - its nice not to have do some big update every six months...

All the reasons that I first came to love using Arch still hold true, its simply that as time has worn on, I've changed a bit: I don't care as much for bleeding edge features, stability and security have become bigger issues, and I've started caring about what other people think about me. So I've been asking myself, "is Arch still for me?" And I think the answer might be no. I purchased a used IBM Thinkpad reccently, and I don't think Arch is going to be my first choice for it.

It seems like its time for Arch and I to "take a break" in our long relationship. But don't worry Arch, its not you, its me.

A quick intro to buffer overflow attacks for the unlearned (feel free to skip this bit).

I would highly recommend reading AlephOne's Smashing the Stack for Fun and Profit if you really want to learn about buffer-overflow attacks, but you can read my bit instead if you just want a quick idea of whats its all about.

Programs written in non-type safe languages like C/C++, do not perform bounds checking on memory when doing reads and writes, and are therefore often to susceptible to what is known as a buffer-overflow attack. Basically, when a program allocates some array on the program stack, the possibility exists that if the programmer is not careful, her program may accidentally overwrite the bounds of the array. One could imagine a situation where a program allocates N amounts of bytes on the stack, reads in input from stdin using scanf (which terminates reading input when it hits a newline) and stores the read bytes in the allocated array. However, the user sitting at the terminal might decide to enter in more then N bytes of data, causing the program to overwrite the bounds of its array. Other values on the stack could then be unintentionally altered which could cause the program to execute erratically and even crash.

But a would-be attacker can do more then just crash a program with a buffer-overflow attack, they could potentially gain control of the program and even cause it to execute arbitrary code. (By "executing arbitrary code" I mean that the attacker could make the program do anything.) An attacker can take control of a program by writing down the program stack past the array's bounds and changing the return address stored on the stack, so that when the currently executing function returns control to the calling function, it actually ends up executing some completely different segment of code. At first glance, this seems rather useless. But an attacker can set the instruction pointer to anything, she could even make it point back to the start of the array that was just overwritten. A carefully crafted attack message, could cause the array to be filled with some bits of arbitrary assembly code (perhaps to fork a shell and then connect that shell to a remote machine) and then overwrite the return address to point to the top of the overwritten array.

A problem with the generic buffer overflow attack is that the starting location of the stack is determined at runtime and therefore can change slightly. This makes it difficult for an attacker to know exactly where the top of the array is every single time. A solution to this, is to use a "NOP slide," where the attack message doesn't immediately begin with the assembly code but rather begins with a stream of NOPs. A NOP is an assembly language instruction that basically does nothing (I believe that it was originally included in the x86 ISA to deal with hazards) so as long as the instruction pointer is reset to point into somewhere the NOP slide, the computer will "slide" (weeeeee!!!!) into the rest of the injected assembly code.

Sounds simple so far? Just you wait....

The trials and tribulations of my feeble attempt to "smash the stack."

The professor for my security class threw in a nice little extra credit problem on a homework assignment last quarter. One of the problems in the homework asked us to crash a flawed web-server using a buffer overflow attack, but we could get extra credit if we managed to root the server with the buffer overflow. Buffer overflow attacks on real software are nontrivial, something my professor made sure to emphasize when he told us that in his whole experience of teaching the class, only one student had ever successfully rooted the server. Now Scott Adams, author of Dilbert, mentioned in one of his books that the best way to motivate an engineer, is to tell them that a task is nearly impossible and that if they're not up for the challenge, then its no big deal because so-and-so could probably do it better. This must be true, because after discussion section, I went straight to a computer and started reading Smashing the Stack for Fun and Profit, intent on being the second person to "smash the stack" in my school.

I was able to crash the server software within less then one minute of swapping the machine's disk image in, as it was a ridiculously simple task to guess where an unbounded buffer was being used in the server code and force a segmentation fault. It took me a few more minutes to trace down the exact location of where the overflow was occurring (there was a location in the code where the program would use strcat() to copy a message), but as soon as I did, I booted up GDB and started gearing up for some hard work.

I created a Python script to generate the message that would crash the server, and ran it a few times against the server, stepping through instructions in GDB to get a better look at what was happening. The first thing I noticed was that there were local pointers and variables on the stack that would get over written when I overflowed the unbounded buffer. Since these values were used after the server called strcat() (which would copy my message into an unbounded buffer on the stack), corrupting these values could very well result in the program crashing prematurely, before it even reached the end of the function. This is generally a bad thing, since my intent is not to crash the software, but rather to make it do whatever I want it to. So the first thing I had to do, was figure out how to generate a message that would overwrite the local stack variables with values that would allow the function to think that everything was functioning properly. Since the size of the local stack is known, I was able to adjust my message length so I could modify stack variables. I had to modify pointers on the stack (since they were above the stored instruction pointer on the stack) and changed them (which were supposed to point to dynamically allocated memory on the heap) to point to someplace much further down on the stack; this corrupted the values near the base of the stack but prevented segmentation faults from occurring. It took me a while to remember that x86 is a little-endian machine, and that I would have to be careful how I arranged the bytes in my message. It wasn't long however, before I hit a brick wall.

I had already copied the shell code from Aleph One's Smashing the Stack for Fun and Profit into my message and filled the space before it with a nice little no-op slide, so at this point I figured I was ready for overwriting the function's return address. I tried for over an hour to get the return address to point to somewhere in my overflowed buffer but no matter how I set it up, the function always crashed when it returned. On a whim, I set the return address on the stack to point to a location not on the stack, but rather in the data segment where program code is stored. This worked beautifully. Unfortunately, this was quite useless since nowhere in the program code, did it execvp() a bash shell and connect it to my machine. Sure I could make the program do some really crazy things, but this still wasn't the same as rooting the machine.

By this point in time, I was spending a lot of time checking out register values, and was making extensive use of the "disassemble" instruction in GDB. After spending a few minutes searching the inter-webs, I learned about an instruction in GDB called "si" which is basically a "step instruction" kind of instruction that steps through each assembly level instruction in the code. Using "si" I found that the server was crashing on the "ret" assembly instruction, and that more specifically, it was crashing right as it tried to load the $EIP register. I guessed that what was happening was that the stack pointer had been reset to the former base pointer and so my overflowed buffer which resided outside of the program stack was being viewed by the machine as unallocated memory so attempts to access it were resulting in a segmentation fault, but this turned out to be wrong.

It took a few minutes of pondering, but then I realized, that clearly, the machine was operating with executable bit protection (where a control bit on each page of memory allows a page to be either executable or writable but not both) and there was no way it was going to run an instruction that wasn't sitting in a page of memory that had the executable bit set. (Note that executable bit protection requires not only a processor that has the necessary technology, but also an operating system that can recognize.) So there was no way I could get the machine to run a single instruction on my carefully crafted no-op slide.

I had already invested a great deal of time into this attack by this point. I've explained things pretty simply up until now, but make no mistake, every bit of progress that I made was bought with hours of work (for the most part, I was learning completely new things). I was faced with two choices at this point, give up on trying to root the system, thereby wasting many hours of work, and commit myself to finishing the rest of the homework, or else continue with my attack. My obsession won out and I chose the hard route; continue with the attack.

Executable bit protection, which is a sort of protected memory, is not the end all method to protecting against buffer overflow attacks as one might first naively think; there are still ways to finagle root access to a machine with a buffer overflow. One in particular, the return-to-libc attack, makes use of the knowledge that on Unix systems, libc is always linked to the program and is accessed at a well known location in memory. Libc contains many useful functions for an intrepid hacker, including system(). In a libc attack, instead of filling the stack with some assembly code and then getting the return address to point to that assembly code, the intent instead is to point the return address to a specific function in libc and then pass as arguments, some value on the stack. Calling a function in libc is perfect legitimate behavior for a program so this kind of buffer-overflow based attack is much harder to protect against. However, it does complicate an attacker's life...

Now heres a rough ASCII picture of the stack for the web-server program that I was trying to crash (note that the stack grows up).
======top of stack=====
+++++++array+++++++
|                 |
|                 |
|                 |
|                 |
+++++++++++++++++++
++local variables++
++frame pointer++++
++return address+++
++socket id++++++++
=====bottom of stack====

Note that below the return address, is a socket identifier that is passed as an argument to the callee function. In the web-server program, the value of the socket identifier is used after a message is copied into the array. What this means, is that if I wanted to write down the stack past the return address, I had to make sure to maintain the value of the socket identifier in order to prevent the machine from crashing on a bad socket id. This was an important fact to know, because in order to be able to make a useful call to a function in libc, I had to be able to pass arguments to the function. Arguments are of coursed passed to other functions by pushing values onto the top of the stack, which in this case would mean overwriting the socket id.

And at this point, I hit a problem that I simply couldn't shift.

I thought at first, that all I had to do was figure out how to put a value in the socket id that would allow the program to execute normally, and could also double as an argument to excevp (which is by no means a trivial task). However, the big problem was that the socket id number was generally a small number, in this case, typically ranging from 4 to 7. The socket id was a 4 byte signed integer, and 7 represented in 4 bytes in hex is:

0x0000 0007

Now recall that the way I was achieving a buffer overflow, was to send a really long message, and then expect the flawed web-server to use strcat() to copy that message into a buffer that was too small to properly accommodate the data. The function strcat() copies bytes from a source pointer to a destination pointer until it hits a null byte. Now look at the hex value above, and notice all the leading zeros! Those leading zeros would be interpreted by strcat() as a null byte which would cause it to immediately stop copying bytes! When I realized this, I was a little stunned, but quickly recovered when I figured that I could simply open a bunch of different connections to the server and hold them open, thereby increasing the socket number used with each successive connection. Lucky me, the server software was quite stupid, and it didn't use SYN cookies and nor did it make any attempt to close connections that were left open and inactive for too long. In order to remove all traces of null bytes from the 32 bit integer, I would need at the very least, a socket number value of:

0x0101 0101

This is a value slightly larger then 16 million, and it gives rise to yet another problem. So lets say that for whatever reason, the server's OS doesn't grant socket numbers in perfect successive values, and instead gives socket ids as multiples of 10 (10, 20 30, 40...), then I would only have to open up 1.6 million connections instead of 16 million! Each full connection that I opened up to the server would cause the server to fork a process to handle the new data connection, so if I opened up 1.6 million connections, the server would have to fork 1.6 million children.

Question: How many processes can a vanilla Red Hat Linux installation handle?

Answer: A lot less then 1.6 million...

Likewise, there was no way the server had the ability to handle 1.6 million half-open connections. So there was no way I was going to be able to pump the socket id up to a high enough value without DOSing the hell out of the server. As my OS professor would say, I was hosed. At this point in time, I was only about a third done with my homework and it was due in about two hours. For the sake of my grade, I was forced to forget the extra credit challenge and pour my efforts into trying to finish the rest of the assignment. (I ended up getting about a 40% on the assignment.) For all the hours and hours of effort that I put into smashing the stack, the only things I succeeded in doing were to make the server crash in several ways and to DOS it. Over the weekend, I put a few more hours of thinking into the problem, but was unable to come up with a satisfactory solution.

So what did I learn from this 12 hour ordeal?

A good buffer overflow attack on real software is a non-trivial problem. AlephOne and every other buffer overflow explanation on the Internet use highly contrived attack scenarios that make it seem like a simple thing, but the fact remains that the devil is in the details. All the little things, like how to get around needed values on the stack, and how to copy a null byte into a buffer are completely ignored in introductory tutorials.

Buffer overflows are hard!

meh.

Pros:

• shiny
• long battery life
• more portable then a typical laptop

Cons:

• almost impossible to achieve a reasonably fast typing speed on it
• shiny (which results in fingerprints and glare)
• does nothing that my laptop can't do
• can't do a lot of things my laptop can do
• costs $150 more then the refurbished laptop I just bought
• closed platform
• wide-aspect movies look weird on a 4:3 screen
• back-lit screens are not ideal for reading books
• development work for the ipad must be done in Objective  C
• less portable then a Motorola Droid or Nexus One (or even an iphone)
• made by Apple

In other fail news:

A week ago, I got a big batch of images that needed to be resized and displayed on one of the websites that I manage. This required that I crop and resize each photo so that it be exactly the correct size to be displayed on the site, a time consuming and quite laborious task. So I figured I could probably whip up a script with Python and ImageMagick to help automate the process, with idea being that my program would allow the user to simply highlight the "relevant" area of an image and then the program would crop and resize it to the correct size.

I ended up having to use wxPython to do all the GUI type stuff, which meant I had to spend some time learning the ins and outs of GUI programming seeing as how my experience with that kind of stuff is fairly limited. So for the past week, I've been spending about an hour a day learning wxPython and knocking together a sort of program to make my life easier. Today, I looked at my image resizer program and realized I had created some of the most god-awful code ever seen by mankind.  It was basically 100+ lines of uncommented and completely unintelligible spaghetti code.

I threw my monster out and did the cropping and resizing by hand, which ended up taking me less than an hour.

• procrastinated
• cooked enough food to last me to the middle of the week
• ate all the food I cooked (I was hungry)
• tried to sleep in but failed miserably (I ended up waking up at 7:50am)
• rode my bike 25 miles, stopped and stared at the houses in Brentwood that probably cost more money then I'll ever make in ten lifetimes
• procrastinated by looking at various electronic gadgets on-line that I have no need for and couldn't possibly afford
• tried to work on my lab but was distracted by food
• sat in the computer lab for about three hours, wrote two lines of code, and tried unsuccessfully to help someone with his Linux troubles
• procrastinated by doing laundry and then sewing up the holes in my black jeans (there were a lot more holes the I realized)
• tried to study but somehow ended up watching old Justice League Unlimited episodes on youtube
• finally got my butt in gear around 7pm on Sunday night and hit the library

On another note, I added a basic captcha to the comment box on this blog in order to reduce the amount of spam Akismet had to handle (Akismet is great, but it does occasionally mark stuff incorrectly). Amazingly enough, a few spam bots are making their way past my captchas! Modern image processing is impressive stuff...

Now, enter OCaml, a functional language free from legacy cruft, and inspired in some form from LISP. I thought I would love being able to develop in a LISP-like language without having to worry about ending statements with a dozen end-parenthesis (which I thought to be LISP's only big syntax flaw), but I found the lack of required parenthesis initially quite awkward. Yes, there are a lot of insipid little parenthesis in LISP, but the point of them is to clarify the code and they do! Parenthesis are what allow LISP to have such simple and easy to understand syntax. (Lets face it, Ocaml just doesn't have the "Its full of cars!" sort of easily understood syntax.) Of course, since OCaml makes parenthesis optional in most cases, one could simply add parenthesis to everything in OCaml, much as one would in LISP. I got over the parenthesis business in OCaml quite quickly, although I still miss them quite a bit.

The one thing I could never get over however, was the weird way OCaml uses built-in operators (like '+', '-', '/', etc.) in infix notation, but has all other functions used in typical prefix notation. By enclosing an operator in parenthesis, it can then be used in prefix notation, but this is a little clunky. I would think that it would be better to force all aspects of the language to follow the same common rules in order to reduce confusion, but the OCaml designers were apparently following a different line of reason from mine.

Although OCaml's odd mix of infix and prefix notation has remained a thorn in my side whenever I lay hand to keyboard to bang out some OCaml code, I've nevertheless managed to gain a good understanding of the language. I've also started to find the usefulness of the language, but my heart still pines for LISP...

E-ink, the display technology behind most e-book readers, is an amazing technology, and not just because of its superior battery life, but because reading it is like reading a newspaper. Anyone who has read plain text on a computer screen for hours at a time, knows that it is not a particularly fun experience. Back-lit screens are stressful on the eyes after long periods of time, whereas reading a good-old-fashioned paper book is a much easier experience. I own several e-books and I actually read Stephanie Meyer's God-awful Twilight book on my computer screen and it was not an experience that I want to repeat, not just because of the terribleness of the book but also because of how my eyes were starting to burn from staring at a back-lit screen for so long. Now I have a friend that says he likes to read books on his iphone for extended periods of time, and I'm fairly certain that hes either a freak of nature, or a bold faced liar. But aside from the scarce masochistic few who enjoy burning their retinas out staring at glowing boxes, most everybody else would rather read paper books.

E-ink has allowed for electronic reading devices that are easy and comfortable to read on. This is something that the multi-use ipad does not offer, and it is the reason that dedicated e-book readers like Amazon's Kindle aren't going anywhere just yet. I don't mean to imply that the ipad is doomed to failure, but rather that the ipad is a device meant to do many things and cannot compete with dedicated e-book readers like the Kindle.