6.858 Fall 2013 Lab 2: Privilege separation

Handed out:	Monday, September 16, 2013
Parts 1 and 2 due:	Friday, September 27, 2013 (5:00pm)
Parts 3 and 4 due:	Friday, October 4, 2013 (5:00pm)
All parts due:	Friday, October 11, 2013 (5:00pm)

Introduction

This lab will introduce you to privilege separation, in the context of a simple python web application called zoobar, where users transfer "zoobars" (credits) between each other. The main goal of privilege separation is to ensure that if an adversary compromises one part of an application, the adversary doesn't compromise the other parts too. To help you privilege-separate this application, the zookws web server used in the previous lab is a clone of the OKWS web server, discussed in lecture. In this lab, you will set up a privilege-separated web server, examine possible vulnerabilities, and break up the application code into less-privileged components to minimize the effects of any single vulnerability.

To fetch the new source code, use Git to commit your Lab 1 solutions, and merge them into our lab2 branch:

In some cases, Git may not be able to figure out how to merge your changes with the new lab assignment (e.g. if you modified some of the code that is changed in the second lab assignment). In that case, the git merge command will tell you which files are conflicted, and you should first resolve the conflict (by editing the relevant files) and then commit the resulting files with git commit -a.

Once your source code is in place, make sure that you can compile and install the web server and the zoobar application:

Part 1: Application vulnerabilities

To understand the zoobar application itself, the first exercise looks at fixing some bugs in the zoobar web application code. Securing the infrastructure for an application has little value if the application itself has bugs that can be exploited directly.

One of the key features of the zoobar application is the ability to transfer credits between users. This feature is implemented by the script transfer.py. Unfortunately, transfer.py has some logical bugs that may result in wrong transfers.

Now, make sure you can run the web server, and access the web site from your browser, as follows:

In this particular example, you would want to open your browser and go to http://172.16.91.143:8080/zoobar/index.cgi, or, if you are using KVM, to http://localhost:8080/zoobar/index.cgi. You should see the zoobar web site.

In your browser, connect to the zoobar Web site, and create two user accounts. Login in as one of the users, and transfer zoobars from one user to another by clicking on the transfer link and filling out the form. Play around with the other features too to get a feel for what it allows users to do. In short, a registered user can update his/her profile, transfer "zoobars" (credits) to another user, and look up the zoobar balance, profile, and transactions of other users in the system.

Read through the code of zoobar and see how transfer.py gets invoked when a user sends a transfer on the transfer page. A good place to start for this part of the lab is templates/transfer.html, __init__.py, transfer.py, and bank.py

Exercise 1. Fix as many logical bugs as you can find in the transfer code. Don't worry about browser-side attacks such as XSS for now, but focus on bugs that an adversary can trigger by giving unanticipated values to the zoobar transfer page. Think carefully about what kinds of inputs an attacker might provide, and try them out by entering them on the zoobar transfer page. In our solution, there are three vulnerabilities.

Privilege separation

Having removed logic bugs in the application that can be directly exploited, it is worth starting to think about how to apply privilege separation to the zookws and zoobar infrastructure so that bugs in the infrastructure don't allow an adversary, for example, to transfer zoobars to the adversary account.

The web server for this lab uses the /jail directory to setup chroot jails for different parts of the web server, much as in the OKWS paper. The make command compiles the web server, and make setup installs it with all the necessary permissions in the /jail directory.

As part of this lab, you will need to change how the files and directories are installed, such as changing their owner or permissions. To do this, you should not change the permissions directly. Instead, you should edit the chroot-setup.sh script in the lab directory, and re-run sudo make setup. If you change the permissions in a different script, your server might not work.

Two aspects make privilege separation challenging in the real world and in this lab. First, privilege separation requires that you take apart the application and split it up in separate pieces. Although we have tried to structure the application well so that it is easy to split, there are places where you must redesign certain parts to make privilege separation possible. Second, you must ensure that each piece runs with minimal privileges, which requires setting permissions precisely and configuring the pieces correctly. Hopefully, by the end of this lab, you'll have a better understanding of why many applications have security vulnerabilities related to failure to properly separate privileges: proper privilege separation is hard!

One problem that you might run into is that it's tricky to debug a complex application that's composed of many pieces. To help you, we have provided a simple debug library in debug.py, which is imported by every Python script we give you. The debug library provides a single function, log(msg), which prints the message msg to stderr (which should go to the terminal where you ran zookld), along with a stack trace of where the log function was called from.

If something doesn't seem to be working, try to figure out what went wrong, or contact the course staff, before proceeding further.

Part 2: Privilege-separate the web server setup using Unix principals and permissions

As introduced in Lab 1, the zookws web server is modeled after OKWS from lecture 4. Similar to OKWS, zookws consists of a launcher daemon zookld that launches services configured in the file zook.conf, a dispatcher zookd that routes requests to corresponding services, as well as several services. For simplicity zookws does not implement helper or logger daemon as OKWS does.

The file zook.conf is the configuration file that specifies how each service should run. For example, the zookd entry:

specifies that the command to run zookd is zookd, that it runs with user and group ID 0 (which is the superuser root), in the jail directory /jail.

The zook.conf file configures only one HTTP service, zookfs_svc, that serves static files and executes dynamic scripts. The zookfs_svc does so by invoking the executable zookfs, which should be jailed in the directory /jail by chroot. You can look into /jail; it contains executables (except for zookld), supporting libraries, and the zoobar web site. See zook.conf and zookfs.c for details.

The launcher daemon zookld, which reads zook.conf and sets up all services is running under root and can bind to a privileged port like 80. Note that in the default configuration, zookd and vulnerable services are inappropriately running under root and that zookld doesn't jail processes yet; an attacker can exploit buffer overflows and cause damages to the server, e.g., unlink a specific file as you have done in Lab 1.

To fix the problem, you should run these services under unprivileged users rather than root. You will modify zookld.c and zook.conf to set up user IDs, group IDs, and chroot for each service. This will proceed in a few steps: first you will modify zookld.c to support chroot, second you will modify zookld.c to support user and group IDs other than root, and finally you will modify zook.conf to use this support.

Exercise 2. Modify the function launch_svc in zookld.c so that it jails the process being created. launch_svc creates a new process for each entry in zook.conf, and then configures that process as specified in zook.conf. Your job is to insert the call to chroot in the specified place. You want to run man 2 chroot to read the manual page about chroot. If you do this correctly, services won't be able to read files outside of the directory specified. For example, zookd shouldn't be able to read the real /etc/passwd anymore.

Run sudo make check to verify that your modified configuration passes our basic tests in check_lab2.py, but keep in mind that our tests are not exhaustive. You probably want to read over the cases before you start implementing.

Exercise 3. Modify the function launch_svc in zookld.c so that it sets the user and group IDs and the supplementary group list specified in zook.conf. For example, you want to set in zook.conf the uid in zookd's entry to, say, 61011, and have zookld.c change the user ID to 61011. You should do the same for group IDs. You will need to use the system calls setresuid, setresgid, and setgroups.

Think carefully about when your code can set the user ID. For example, can it go before setegid or chroot?

This will also require you to modify chroot-setup.sh to ensure that the files on disk, such as the database, can be read only by the processes that should be able to read them. You can either use the built-in chmod and chown commands or our provided set_perms function, which you can invoke like so:


set_perms 1234:5678 755 /path/to/file

which will set the owner of the file to 1234, the group to 5678, and the permissions to 755 (i.e., user read/write/execute, group read/execute, other read/execute).

Run sudo make check to verify that your modified configuration passes our basic tests.

Now that none of the services are running as root, we will try to further privilege-separate the zookfs_svc service that handles both static files and dynamic scripts. Although it runs under an unprivileged user, some Python scripts could easily have security holes; a vulnerable Python script could be tricked into deleting important static files that the server is serving. Conversely, if the static service is compromised, it might read the databases used by the Python scripts, such as person.db and transfer.db. A better organization is to split zookfs_svc into two services, one for static files and the other for Python scripts, running under different users.

Exercise 4. Create two new HTTP services (replacing zookfs_svc), along the lines of the existing zookfs_svc service, such that one will execute dynamic content, and one will serve static files. Modify the configuration file zook.conf to split the zookfs_svc service into two services running under different users: the static_svc service that only serves static files, and the dynamic_svc service that only executes the intended Python scripts (i.e., /zoobar/index.cgi).

Set file and directory permissions (using chroot-setup.sh) to ensure that the static service cannot read the database files from the dynamic service, that the dynamic service cannot modify static files, and that the dynamic service cannot be tricked into executing other scripts, such as the various .py programs under zoobar/.

This separation requires zookd to determine which service should handle a particular request. You may use zookws's URL filtering to do this, without modifying the application or the URLs that it uses. The URL filters are specified in zook.conf, and support regular expressions. For example, url = .* matches all requests, while url = /zoobar/(abc|def)\.html matches requests to /zoobar/abc.html and /zoobar/def.html.

Do not rely on URL filters for security; it is exceedingly difficult to do so correctly. For example, even if you configure the filter to pass only URLs matching .cgi to the dynamic service, an adversary can still invoke a hypothetical buggy /zoobar/foo.py script by issuing a request for /zoobar/foo.py/xx.cgi.

There are many executable files on the filesystem, and we can not mark all of them non-executable for the zookfs service. For example, the zookfs service needs to execute /jail/usr/bin/python to run the zoobar website. We have added a feature to zookfs to only run trusted executables marked by a particular combination of owner user and group. To use this function, add an args = UID GID line to the service's configuration. For example, the following zook.conf entry:

[safe_svc]
    cmd = zookfs
    uid = 0
    gid = 0
    dir = /jail
    args = 123 456

specifies that safe_svc will only execute files owned by user ID 123 and group ID 456.

Run sudo make check to verify that your modified configuration passes our tests.

Part 3: RPC library

In this part, you will privilege-separate the zoobar application itself in several processes. In the first exercise of this lab, we fixed the bugs in the transfer code; now we would like to make sure we can deal with any future such bugs that come up. That is, if one piece of the zoobar has an exploitable bug, then an attacker cannot use that bug to exploit other parts of the zoobar application. A challenge in spitting the zoobar application in several processes running with their own privileges is that the different processes must interact and have a way to communicate. You will first modify a Remote Procedure Call (RPC) library that allows processes to communicate over a Unix socket. Then, you will use that library to separate the zoobar in several processes that communicate using RPC.

The RPC library itself shouldn't have any exploitable bugs, but historically parsing of messages has been a problem. We provide you with a buggy RPC library that you need to fix. In the process, you will learn how to use the RPC library so that you can use it for privilege separation of the zoobar application.

To illustrate how the RPC library might be used, we have implemented a simple "echo" service for you, in zoobar/echo-server.py. This service is invoked by zookld; look for the echo_svc section of zook.conf to see how it is started.

echo-server.py is implemented by defining an RPC class EchoRpcServer that inherits from RpcServer, which in turn comes from zoobar/rpclib.py (the echo-server.py actually code uses a copy of this code, from zoobar/rpclib_orig.py, so that it runs the same way after you modify rpclib.py in a later exercise). The EchoRpcServer RPC class defines the methods that the server supports, and rpclib invokes those methods when a client sends a request. The server defines a simple method that echos the request from a client, along with an unlink method that you will be using in your exploit in the next exercise.

echo-server.py starts the server by calling run_sockpath_fork(sockpath). This function listens on a UNIX-domain socket. The socket name comes from the argument, which in this case is /echosvc/sock (specified in zook.conf). When a client connects to this socket, the function forks the current process. One copy of the process receives messages and responds on the just-opened connection, while the other process listens for other clients that might open the socket.

We have also included a simple client of this echo service as part of the Zoobar web application. In particular, if you go to the URL /zoobar/index.cgi/echo?s=hello, the request is routed to zoobar/echo.py. That code uses the RPC client (implemented by rpclib) to connect to the echo service at /echosvc/sock and invoke the echo operation. Once it receives the response from the echo service, it returns a web page containing the echoed response.

The RPC client-side code in rpclib is implemented by the call method of the RpcClient class. This methods formats the arguments into a string, writes the string on the connection to the server, and waits for a response (a string). On receiving the response, call parses the string, and returns the results to the caller.

As we mentioned before, the RPC library contains a bug that can be misused by an adversary to invoke an unintended RPC function on the server. Your job is to find this vulnerability and develop an exploit for it.

Exercise 5. Modify echo-exploit.py to construct an exploit that unlinks /jail/echosvc/sock using /zoobar/index.cgi/echo?s=xxx. Hint: what if echo-exploit.py uses spaces or a newline in an argument?

Run sudo make check to verify that your exploit works correctly.

Now that you understand how important it is to avoid parsing bugs in the RPC library, your job is to fix the RPC library so that the caller can safely pass arbitrary arguments, without worrying that the server might mis-parse them.

To help you test your RPC library, we have provided an RPC client and server, zoobar/rpctest-server.py and zoobar/rpctest-client.py. The server is similar to the echo service from the previous exercise. The client performs a variety of RPC calls on the server, trying to pass different kinds of arguments and return values, and checks whether it obtains the proper response. To run the server by hand, type:

As you can see, the initial version of rpclib.py works fine with simple alphanumeric values, but starts to fail when punctuation is involved (rpctest-client.py prints the failing arguments and the expected and received return values).

Exercise 6. Modify the RPC library to avoid parsing bugs. A good plan would be to use a standard encoding scheme such as JSON. Note that you should not use Python's pickle module, because a malicious pickled object representation can cause the unpickling function can execute arbitrary Python code.

Modify rpclib.py, but do not modify rpclib_orig.py.

Run sudo make check to verify that your RPC library passes our tests.

Part 4: Privilege-separating the login service in Zoobar

Now we have a correct RPC library, we will use it to improve the security of the user passwords stored in the Zoobar web application. Right now, an adversary that exploits a vulnerability in any part of the Zoobar application can obtain all user passwords from the person database.

The first step towards protecting passwords will be to create a service that deals with user passwords and cookies, so that only that service can access them directly, and the rest of the Zoobar application cannot. In particular, we want to separate the code that deals with user authentication (i.e., passwords and tokens) from the rest of the application code. The current zoobar application stores everything about the user (their profile, their zoobar balance, and authentication info) in the Person table (see zoodb.py). We want to move the authentication info out of the Person table into a separate Cred table (Cred stands for Credentials), and move the code that accesses this authentication information (i.e., auth.py) into a separate service.

The reason for splitting the tables is that the tables are stored in the file system in zoobar/db/, and are accessible to all Python code in Zoobar. This means that an attacker might be able to access and modify any of these tables, and we might never find out about the attack. However, once the authentication data is split out into its own database, we can set Unix file and directory permissions such that only the authentication service---and not the rest of Zoobar---can access that information.

Exercise 7. Implement privilege separation for user authentication, as described above.

Don't forget to create a regular Person database entry for newly registered users.

Run sudo make check to verify that your privilege-separated authentication service passes our tests.

Now, we will further improve the security of passwords, by using hashing and salting. The current authentication code stores an exact copy of the user's password in the database. Thus, if an adversary somehow gains access to the cred.db file, all of the user passwords will be immediately compromised. Worse yet, if users have the same password on multiple sites, the adversary will be able to compromise users' accounts there too!

Hashing protects against this attack, by storing a hash of the user's password (i.e., the result of applying a hash function to the password), instead of the password itself. If the hash function is difficult to invert (i.e., is a cryptographically secure hashes), an adversary will not be able to directly obtain the user's password. However, a server can still check if a user supplied the correct password during login: it will just hash the user's password, and check if the resulting hash value is the same as was previously stored.

One weakness with hashing is that an adversary can build up a giant table (called a "rainbow table"), containing the hashes of all possible passwords. Then, if an adversary obtains someone's hashed password, the adversary can just look it up in its giant table, and obtain the original password.

To defeat the rainbow table attack, most systems use salting. With salting, instead of storing a hash of the password, the server stores a hash of the password concatenated with a randomly-generated string (called a salt). To check if the password is correct, the server concatenates the user-supplied password with the salt, and checks if the result matches the stored hash. Note that, to make this work, the server must store the salt value used to originally compute the salted hash! However, because of the salt, the adversary would now have to generate a separate rainbow table for every possible salt value. This greatly increases the amount of work the adversary has to perform in order to guess user passwords based on the hashes.

A final consideration is the choice of hash function. Most hash functions, such as MD5 and SHA1, are designed to be fast. This means that an adversary can try lots of passwords in a short period of time, which is not what we want! Instead, you should use a special hash-like function that is explicitly designed to be slow. A good example of such a hash function is PBKDF2, which stands for Password-Based Key Derivation Function (version 2).

Exercise 8. Implement password hashing and salting in your authentication service. In particular, you will need to extend your Cred table to include a salt column; modify the registration code to choose a random salt, and to store a hash of the password together with the salt, instead of the password itself; and modify the login code to hash the supplied password together with the stored salt, and compare it with the stored hash. You can store the hashed password in the existing password column you have in the Cred table.

To implement PBKDF2 hashing, you can use the Python PBKDF2 module. Roughly, you should import pbkdf2, and then hash a password using pbkdf2.PBKDF2(password, salt).hexread(32). We have provided a copy of pbkdf2.py in the zoobar directory. Do not use the random.random function to generate a salt as the documentation of the random module states that it is not cryptographically secure. A secure alternative is the function os.urandom.

Run sudo make check to verify that your hashing and salting code passes our tests. Keep in mind that our tests are not exhaustive.

A surprising side-effect of using a very computationally expensive hash function like PBKDF2 is that an adversary can now use this to launch denial-of-service (DoS) attacks on the server's CPU. For example, the popular Django web framework recently posted a security advisory about this, pointing out that if an adversary tries to log in to some account by supplying a very large password (1MB in size), the server would spend an entire minute trying to compute PBKDF2 on that password. Django's solution is to limit supplied passwords to at most 4KB in size. For this lab, we do not require you to handle such DoS attacks.

Challenge! (optional) For extra credit, implement the honeywords proposal from Ari Juels and Ron Rivest in your authentication service. Consider implementing the honeychecker as a separate service running with its own user ID.

Part 5: Privilege-separating the bank in Zoobar

Finally, we want to protect the zoobar balance of each user from adversaries that might exploit some bug in the Zoobar application. Currently, if an adversary exploits a bug in the main Zoobar application, they can steal anyone else's zoobars, and this would not even show up in the Transfer database if we wanted to audit things later.

To improve the security of zoobar balances, our plan is similar to what you did above in the authentication service: split the zoobar balance information into a separate Bank database, and set up a bank_svc service, whose job it is to perform operations on the new Bank database and the existing Transfer database. As long as only the bank_svc service can modify the Bank and Transfer databases, bugs in the rest of the Zoobar application should not give an adversary the ability to modify zoobar balances, and will ensure that all transfers are correctly logged for future audits.

Exercise 9. Privilege-separate the bank logic into a separate bank_svc service, along the lines of the authentication service. Your service should implement the transfer and balance functions, which are currently implemented by bank.py and called from several places in the rest of the application code.

You will need to split the zoobar balance information into a separate Bank database (in zoodb.py); implement the bank server by modifying bank-server.py; add the bank service to zook.conf; modify chroot-setup.sh to create the new Bank database and the socket for the bank service, and to set permissions on both the new Bank and the existing Transfer databases accordingly; create client RPC stubs for invoking the bank service; and modify the rest of the application code to invoke the RPC stubs instead of calling bank.py's functions directly.

Don't forget to handle the case of account creation, when the new user needs to get an initial 10 zoobars. This may require you to change the interface of the bank service.

Run sudo make check to verify that your privilege-separated bank service passes our tests.

Finally, we need to fix one more problem with the bank service. In particular, an adversary that can access the transfer service (i.e., can send it RPC requests) can perform transfers from anyone's account to their own. For example, it can steal 1 zoobar from any victim simply by issuing a transfer(victim, adversary, 1) RPC request. The problem is that the bank service has no idea who is invoking the transfer operation. Some RPC libraries provide authentication, but our RPC library is quite simple, so we have to add it explicitly.

To authenticate the caller of the transfer operation, we will require the caller to supply an extra token argument, which should be a valid token for the sender. The bank service should reject transfers if the token is invalid.

Exercise 10. Add authentication to the transfer RPC in the bank service. The current user's token is accessible as g.user.token. How should the bank validate the supplied token?

Although make check does not include an explicit test for this exercise, you should be able to check whether this feature is working or not by manually connecting to your transfer service and verifying that it is not possible to perform a transfer without supplying a valid token.

Submit your answers to the first part of the lab assignment by running make submit to upload lab2-handin.tar.gz to the submission web site.