6.858 Spring 2020 Lab 2: Privilege separation and server-side sandboxing

Handed out:	Wednesday, February 19, 2020
Part 1 due:	Friday, February 28, 2020 (5:00pm)
Parts 2 and 3 due:	Friday, March 6, 2020 (5:00pm)
All parts due:	Friday, April 3, 2020 (5:00pm)

Introduction

This lab will introduce you to privilege separation and server-side sandboxing, in the context of a simple python web application called zoobar, where users transfer "zoobars" (credits) between each other. The main goal of privilege separation is to ensure that if an adversary compromises one part of an application, the adversary doesn't compromise the other parts too. To help you privilege-separate this application, the zookws web server used in the previous lab is a clone of the OKWS web server, discussed in lecture. In this lab, you will set up a privilege-separated web server, examine possible vulnerabilities, and break up the application code into less-privileged components to minimize the effects of any single vulnerability. The lab will use modern support for privilege separation: Linux containers, which the OKWS paper doesn't use (because they didn't exist at the time).

You will also extend the Zoobar web application to support executable profiles, which allow users to use Python code as their profiles. To make a profile, a user saves a Python program in their profile on their Zoobar home page. (To indicate that the profile contains Python code, the first line must be #!python.) Whenever another user views the user's Python profile, the server will execute the Python code in that user's profile to generate the resulting profile output. This will allow users to implement a variety of features in their profiles, such as:

Supporting this safely requires sandboxing the profile code on the server, so that it cannot perform arbitrary operations or access arbitrary files. On the other hand, this code may need to keep track of persistent data in some files, or to access existing zoobar databases, to function properly. You will use the remote procedure call library and some shim code that we provide to securely sandbox executable profiles.

To fetch the new source code, use Git to commit your Lab 1 solutions, and merge them into our lab2 branch:

In some cases, Git may not be able to figure out how to merge your changes with the new lab assignment (e.g. if you modified some of the code that is changed in the second lab assignment). In that case, the git merge command will tell you which files are conflicted, and you should first resolve the conflict (by editing the relevant files) and then commit the resulting files with git commit -a.

Once your source code is in place, make sure that you can compile and install the web server and the zoobar application:

Prelude: What's a zoobar?

To understand the zoobar application itself, we will first examine the zoobar web application code.

One of the key features of the zoobar application is the ability to transfer credits between users. This feature is implemented by the script transfer.py.

The first time you run zookld.py it will run for minutes, because it is building the base container, which involves building a Linux image and installing all the software that zoobar needs. Then, it builds three other containers: main, zookfs, and echo. These containers are stored in the ~/.local/share/lxc/ directory. zookld.py uses zookconf.py to build and configure the containers. If you get errors about creating or starting containers, try resetting the container state using the ./zookclean.py command and/or rebooting your VM.

All of these commands are wrappers around the LXC Python API. You can also run many of the LXC commands from command line; in particular, lxc-attach -n name will give you a root shell in container name, which you might find useful for debugging.

Run zookps.py to see if your containers are running. Then, make sure you can run the web server, and access the web site from your browser, as follows:

In this particular example, you would want to open your browser and go to http://192.168.24.128:8888/zoobar/index.cgi/, or, if you are using KVM, to http://localhost:8888/zoobar/index.cgi/. You should see the zoobar web site.

Note the different port, 8888 rather than 8080, in the URLs above. This is because we actually want to connect to the main container, but it's on an internal virtual network that's only reachable from the VM itself. We configured the Linux kernel in the VM to forward connections to port 8888 to port 8080 on the main container. You can see the iptables rules we used to achieve this in /etc/rc.local in your VM.

If you are having issues with seeing the web site, one possible cause is that the fixes that you made in Lab 1 to fix the buffer overflow bugs may be overly strict. If so, please fix it before continuing.

In your browser, connect to the zoobar Web site, and create two user accounts. Login in as one of the users, and transfer zoobars from one user to another by clicking on the transfer link and filling out the form. Play around with the other features too to get a feel for what it allows users to do. In short, a registered user can update his/her profile, transfer "zoobars" (credits) to another user, and look up the zoobar balance, profile, and transactions of other users in the system.

Read through the code of zoobar and see how transfer.py gets invoked when a user sends a transfer on the transfer page. A good place to start for this part of the lab is templates/transfer.html, __init__.py, transfer.py, and bank.py in the zoobar directory.

Note: You don't need to turn in anything for this exercise, but make sure that you understand the structure of the zoobar application--it will save you time in the future!

Privilege separation

Having surveyed the zoobar application code, it is worth starting to think about how to apply privilege separation to the zookws and zoobar infrastructure so that bugs in the infrastructure don't allow an adversary, for example, to transfer zoobars to the adversary account.

The web server for this lab uses containers for different parts of the web server. The containers are defined in zook.conf. zookld.py uses zookconf.py to read zook.conf (via readconf.py) and sets up the containers. As part of this lab you will define new containers and modify how containers are set up. For that purpose you need to work with LXC and you should read the documentation on LXC.

Two aspects make privilege separation challenging in the real world and in this lab. First, privilege separation requires that you take apart the application and split it up in separate pieces. Although we have tried to structure the application well so that it is easy to split, there are places where you must redesign certain parts to make privilege separation possible. Second, you must ensure that each piece runs with minimal privileges, which requires setting permissions precisely and configuring the pieces correctly. Hopefully, by the end of this lab, you'll have a better understanding of why many applications have security vulnerabilities related to failure to properly separate privileges: proper privilege separation is hard!

One problem that you might run into is that it's tricky to debug a complex application that's composed of many pieces. To help you, we have provided a simple debug library in debug.py, which is imported by every Python script we give you. The debug library provides a single function, log(msg), which prints the message msg to stderr (which should go to the terminal where you ran zookld), along with a stack trace of where the log function was called from.

All output from code running in various containers that you see in your terminal should also be prefixed with the name of the container. For example, when you see main: zookd2: Forwarding to 10.1.1.4:8081 for /zoobar/media/zoobar.css, it means the message is coming from the main container.

If something doesn't seem to be working, try to figure out what went wrong, or contact the course staff, before proceeding further.

Part 1: Privilege-separate the web server setup using containers

In lab 1, zookws consisted of essentially one process: zookd. From the security point of view, this structure is not ideal: for example, any buffer overrun you found, you can use to take over zookws. For example, you can invoke the dynamic scripts with arguments of your choice (e.g., giving many zoobars to yourself), or simpler, just write the database that contains the zoobar accounts directly.

This lab refactors zookd following OKWS from lecture 5. Similar to OKWS, zookws consists of a launcher program zookld.py that launches services configured in the file zook.conf, a zookd that just routes requests to corresponding services, as well as several services. For simplicity zookws does not implement helper or logger daemon as OKWS does.

We will run each component in a separate Linux container. Linux containers provide the illusion of a virtual Linux machine without using virtual machines; they are implemented using Linux processes. A process in a container is stronger isolated than standard Linux processes: a process inside a container has limited access to kernel names spaces, has limited access to systems calls, and doesn't have access to the file system. In many ways, they behave like virtual machines: they are started from a VM image, have their own IP address, their own file system, etc. You assign IP addresses to them, copy the right files into them, and arrange for remote procedure calls between them.

We will use unprivileged containers, which run as an unprivileged user process (i.e., not root). Even if the process running inside the container runs with root privileges, the container itself runs as an unprivileged user.

With containers, even if there is an exploit (e.g., another buffer overrun in zookd), the container running zookd will give the attacker little control. For example, taking over zookd, will not allow the attacker to invoke the dynamic scripts or write the database directly that runs inside another container. Furthermore, zookd cannot break out of its container and take control over the underlying Linux kernel.

The file zook.conf is the configuration file that specifies how each container should run. For example, the main entry:

specifies that the command to run main is zookd2, in directory /home/student on a container connected to virtual network 0 (lxcbr, short for LXC bridge), and that cmd gets port 8080 to receive/send requests on.

In the lab VM that you are using, we pre-created 10 virtual networks, lxcbr0 through lxcbr9, which you can use, corresponding to network addresses 10.1.0.* through 10.1.9.* respectively (or 10.1.0.0/24 through 10.1.9.0/24 in CIDR notation). Each container gets the .4 IP address in its virtual network's subnet; for example, if you assign some service to lxcbr = 7, it will have the IP address 10.1.7.4.

The reason for having multiple virtual networks is to provide strong network isolation between containers. Containers connected to the same virtual network can send each other packets directly, and a compromised container could spoof packets from another container's IP address. On the other hand, containers on different virtual networks must go through the host's Linux kernel to route packets, and the Linux kernel ensures that packets coming from one virtual network do not spoof a source address of a different virtual network (enabled by the rp_filter kernel option).

By default, each container allows incoming packets from any other container. You will restrict this later in the lab, and the use of separate virtual networks will ensure that a compromised container cannot bypass these network-level restrictions.

The zook.conf file configures only one HTTP service (through the line http_svcs), zookfs, that both serves static files and executes dynamic scripts. Later on in this lab, you will need to run multiple HTTP services, which you can do by listing all of them, separated by comma, in the http_svcs line; for example, http_svcs = first,second,third.

The zookfs service works by invoking the executable zookfs, which runs in the directory /home/student on the container connected to lxcbr = 1 (and thus with IP address 10.1.1.4).

The fwrule lines that you see commented out in the zookfs service description in zook.conf specify filters that control communication to that container:

For example, the commented-out rules allow packets from the main and echo containers, but block all other packets. You will change these rules (and define similar rules for other containers) as we further privilege separate zookws.

We will start to further privilege-separate the zookfs service that handles both static files and dynamic scripts. Although it runs in a container, some Python scripts could easily have security holes; a vulnerable Python script could be tricked into deleting important static files that the server is serving. Conversely, the static file serving code might be tricked into serving up the databases used by the Python scripts, such as person.db and transfer.db. A better organization is to split zookfs into two services, one for static files and the other for Python scripts, running as different users.

Exercise 2. Modify zook.conf to replace zookfs with two separate services, dynamic and static. Both should use cmd = zookfs.

dynamic should execute just /zoobar/index.cgi (which runs all the Python scripts), but should not serve any static files. static should serve static files but not execute anything.

To remove the zookfs container so that zookld.py doesn't start it, run ./zookclean.py zookfs.

Run the dynamic and static services on different virtual networks.

This separation requires zookd to determine which service should handle a particular request. You may use zookws's URL filtering to do this, without modifying the application or the URLs that it uses. The URL filters are specified in zook.conf, and support regular expressions. For example, url = .* matches all requests, while url = /zoobar/(abc|def)\.html matches requests to /zoobar/abc.html and /zoobar/def.html.

For this exercise, you should only modify zook.conf; don't modify any C or Python code.

Run make check to verify that your modified configuration passes our tests.

We would like to control with whom dynamic and static can communicate. For example, even if static were compromised, we want the attacker be unable to communicate with dynamic, making it harder for the attacker to compromise dynamic too. Therefore, we would like to set up firewall rules in each container to enforce this isolation. We will use iptables to insert filter rules in each container, so that static will only accept packets from main (which runs zookd). This ensures that dynamic cannot send packets directly to static. Similarly, dynamic should only accept packets from main, and not from static. Setting security rules likes these is often referred to as network segregation.

Exercise 3. Write appropriate fwrule entries for main, static, and dynamic to limit the communication as specified above.

If you get the filters wrong, you may be unable to connect with any container. You can reset the firewall to allow all communication by stopping and starting the containers again, using zookstop.py followed by zookld.py.

Submit your answers to the first part of this lab assignment by running make submit-a. Alternatively, run make; prepare-submit-a and upload the resulting lab2a-handin.tar.gz file to the submission web site.

Interlude: RPC library

In this part, you will privilege-separate the zoobar application itself in several processes, running in different containers. We would like to limit the damage from any future bugs that come up. That is, if one piece of the zoobar application has an exploitable bug, we'd like to prevent an attacker from using that bug to break into other parts of the zoobar application.

A challenge in splitting the zoobar application into several containers is that the processs inside containers must have a way to communicate with each other. You will first study a Remote Procedure Call (RPC) library that allows processes to communicate. Then, you will use that library to separate zoobar into several processes, each inside its own container, that communicate using RPC.

To illustrate how our RPC library might be used, we have implemented a simple "echo" service for you, in zoobar/echo-server.py. This service is invoked by zookld and runs inside its own container; look for the echo section of zook.conf to see how it is started.

echo-server.py is implemented by defining an RPC class EchoRpcServer that inherits from RpcServer, which in turn comes from zoobar/rpclib.py. The EchoRpcServer RPC class defines the methods that the server supports, and rpclib invokes those methods when a client sends a request. The server defines a simple method that echos the request from a client.

echo-server.py starts the server by calling run_fork(port). This function listens on a TCP socket. The port comes from the argument, which in this case is 8081 (specified in zook.conf). When a client connects to this socket, the function forks the current process. One copy of the process receives messages and responds on the just-opened connection, while the other process listens for other clients that might open the socket.

We have also included a simple client of this echo service as part of the Zoobar web application. In particular, if you go to the URL /zoobar/index.cgi/echo?s=hello, the request is routed to zoobar/echo.py. That code uses the RPC client (implemented by rpclib) to connect to the echo service at (host IP address, port). The client looks up the (host IP address, port) in zook.conf. The client invokes the echo operation. Once it receives the response from the echo service, it returns a web page containing the echoed response.

The RPC client-side code in rpclib is implemented by the call method of the RpcClient class. This methods formats the arguments into a string, writes the string on the connection to the server, and waits for a response (a string). On receiving the response, call parses the string, and returns the results to the caller.

Part 2: Privilege-separating the login service in Zoobar

We will now use the RPC library to improve the security of the user passwords stored in the Zoobar web application. Right now, an adversary that exploits a vulnerability in any part of the Zoobar application can obtain all user passwords from the person database.

The first step towards protecting passwords will be to create a service that deals with user passwords and cookies, so that only that service can access them directly, and the rest of the Zoobar application cannot. In particular, we want to separate the code that deals with user authentication (i.e., passwords and tokens) from the rest of the application code. The current zoobar application stores everything about the user (their profile, their zoobar balance, and authentication info) in the Person table (see zoodb.py). We want to move the authentication info out of the Person table into a separate Cred table (Cred stands for Credentials), and move the code that accesses this authentication information (i.e., auth.py) into a separate service.

Note that it is not completely necessary to split the data into separate tables for security: each container would end up with its own copy of the database, and would have no data in the parts of the database that it never populated. We split up the data into separate tables anyway, because it helps understand how to split up the services correctly.

Exercise 4. Implement privilege separation for user authentication, as described above.

Don't forget to create a regular Person database entry for newly registered users.

Run make check to verify that your privilege-separated authentication service passes our tests.

Exercise 5. Specify the appropriate fwrule entries for auth.

Now, we will further improve the security of passwords, by using hashing and salting. The current authentication code stores an exact copy of the user's password in the database. Thus, if an adversary somehow gains access to the cred.db file, all of the user passwords will be immediately compromised. Worse yet, if users have the same password on multiple sites, the adversary will be able to compromise users' accounts there too!

Hashing protects against this attack, by storing a hash of the user's password (i.e., the result of applying a hash function to the password), instead of the password itself. If the hash function is difficult to invert (i.e., is a cryptographically secure hash), an adversary will not be able to directly obtain the user's password. However, a server can still check if a user supplied the correct password during login: it will just hash the user's password, and check if the resulting hash value is the same as was previously stored.

One weakness with hashing is that an adversary can build up a giant table (called a "rainbow table"), containing the hashes of all possible passwords. Then, if an adversary obtains someone's hashed password, the adversary can just look it up in its giant table, and obtain the original password.

To defeat the rainbow table attack, most systems use salting. With salting, instead of storing a hash of the password, the server stores a hash of the password concatenated with a randomly-generated string (called a salt). To check if the password is correct, the server concatenates the user-supplied password with the salt, and checks if the result matches the stored hash. Note that, to make this work, the server must store the salt value used to originally compute the salted hash! However, because of the salt, the adversary would now have to generate a separate rainbow table for every possible salt value. This greatly increases the amount of work the adversary has to perform in order to guess user passwords based on the hashes.

A final consideration is the choice of hash function. Most hash functions, such as MD5 and SHA1, are designed to be fast. This means that an adversary can try lots of passwords in a short period of time, which is not what we want! Instead, you should use a special hash-like function that is explicitly designed to be slow. A good example of such a hash function is PBKDF2, which stands for Password-Based Key Derivation Function (version 2).

Exercise 6. Implement password hashing and salting in your authentication service. In particular, you will need to extend your Cred table to include a salt column; modify the registration code to choose a random salt, and to store a hash of the password together with the salt, instead of the password itself; and modify the login code to hash the supplied password together with the stored salt, and compare it with the stored hash. Don't remove the password column from the Cred table (the check for exercise 5 requires that it be present); you can store the hashed password in the existing password column.

To implement PBKDF2 hashing, you can use the Python PBKDF2 module. Roughly, you should import pbkdf2, and then hash a password using pbkdf2.PBKDF2(password, salt).hexread(32). We have provided a copy of pbkdf2.py in the zoobar directory. Do not use the random.random function to generate a salt as the documentation of the random module states that it is not cryptographically secure. A secure alternative is the function os.urandom.

Run make check to verify that your hashing and salting code passes our tests. Keep in mind that our tests are not exhaustive.

A surprising side-effect of using a very computationally expensive hash function like PBKDF2 is that an adversary can now use this to launch denial-of-service (DoS) attacks on the server's CPU. For example, the popular Django web framework posted a security advisory about this, pointing out that if an adversary tries to log in to some account by supplying a very large password (1MB in size), the server would spend an entire minute trying to compute PBKDF2 on that password. Django's solution is to limit supplied passwords to at most 4KB in size. For this lab, we do not require you to handle such DoS attacks.

Challenge 1! (optional) For extra credit, implement the honeywords proposal from Ari Juels and Ron Rivest in your authentication service. Consider implementing the honeychecker as a separate service running with its own user ID. If you decide to complete this challenge, please include a file named honeywords.txt in the top level of your lab submission that gives a brief overview of your approach and solution.

Part 3: Privilege-separating the bank in Zoobar

Finally, we want to protect the zoobar balance of each user from adversaries that might exploit some bug in the Zoobar application. Currently, if an adversary exploits a bug in the main Zoobar application, they can steal anyone else's zoobars, and this would not even show up in the Transfer database if we wanted to audit things later.

To improve the security of zoobar balances, our plan is similar to what you did above in the authentication service: split the zoobar balance information into a separate Bank database, and set up a bank service, whose job it is to perform operations on the new Bank database and the existing Transfer database. As long as only the bank service can modify the Bank and Transfer databases, bugs in the rest of the Zoobar application should not give an adversary the ability to modify zoobar balances, and will ensure that all transfers are correctly logged for future audits.

Exercise 7. Privilege-separate the bank logic into a separate bank service, along the lines of the authentication service. Your service should implement the transfer and balance functions, which are currently implemented by bank.py and called from several places in the rest of the application code.

You should split the zoobar balance information into a separate Bank database (in zoodb.py); implement the bank server by modifying bank-server.py; add the bank service to zook.conf; create client RPC stubs for invoking the bank service; and modify the rest of the application code to invoke the RPC stubs instead of calling bank.py's functions directly.

Don't forget to handle the case of account creation, when the new user needs to get an initial 10 zoobars. This may require you to change the interface of the bank service.

Run make check to verify that your privilege-separated bank service passes our tests.

Finally, we need to fix one more problem with the bank service. In particular, an adversary that can access the bank service (i.e., can send it RPC requests) can perform transfers from anyone's account to their own. For example, it can steal 1 zoobar from any victim simply by issuing a transfer(victim, adversary, 1) RPC request. The problem is that the bank service has no idea who is invoking the transfer operation. Some RPC libraries provide authentication, but our RPC library is quite simple, so we have to add it explicitly.

To authenticate the caller of the transfer operation, we will require the caller to supply an extra token argument, which should be a valid token for the sender. The bank service should reject transfers if the token is invalid.

Exercise 8. Add authentication to the transfer RPC in the bank service. The current user's token is accessible as g.user.token. How should the bank validate the supplied token?

Although make check does not include an explicit test for this exercise, you should be able to check whether this feature is working or not by manually connecting to your transfer service and verifying that it is not possible to perform a transfer without supplying a valid token.

Exercise 9. Specify the appropriate fwrule entries for the bank service.

Submit your answers to parts 2 and 3 of this lab assignment by running make submit-b. Alternatively, run make prepare-submit-b and upload the resulting lab2b-handin.tar.gz file to the submission web site.

Part 4: Server-side sandboxing for executable profiles

In this final part, you will implement sandboxing for executable profiles, which is challenging because we need to isolate the profiles of different users from one another and from the rest of the system.

We will use two methods for privilege separation. We will use a container to run the profile service, much as in the earlier parts of this lab. The profile service will have several Unix processes: a process for the profile server, which forks for each request two other processes (i.e., one for the ProfileAPIServer and one for the sandbox). To separate the sandbox process (which runs untrusted code) from profile server, we will use Unix mechanisms to reduce the privileges of the sandbox process.

You should familiarize yourself with the following new components related to executable profiles:

One challenge is that the profile service performs rpc_xfer from the profile owner's account, which requires a token for the owner. You cannot just add an RPC to the auth service to obtain a token for the profile owner, because then any service could ask for it; we want only the profile service to be able to do this transfer. Similarly, we cannot add an RPC to the bank service to do a transfer from anyone's account without a token.

We want to do the transfer from the profile owner's account only if the request came from the profile service. Thus, the bank service must be able to authenticate that a request came from the profile service. To help you do so, rpclib.py provides the name of the calling service in self.caller, based on the IP address of the connection from which it received the request.

To get started, you will need to add profile service to your zook.conf. Run the service in its own container and on its own network link.

Exercise 10. Add profile-server.py to your web server.

Make sure that your Zoobar site can support all of the five profiles.

Run make check to verify that your modified configuration passes our tests. The test case creates some user accounts, stores one of the Python profiles in the profile of one user, has another user view that profile, and checks that the other user sees the right output.

If you run into problems from the make check tests, you can always check /tmp/html.out for the output html of the profiles.

The above design has two security problems. First, the sandbox runs with the same privileges as the profile server and can access any file in the file system on the profile container, as well as talk on the network to issue RPCs directly to the bank. Second, profiles from different users create files in the same directory /tmp. The next exercise solves both of these problems by sandboxing the profile code using Unix mechanisms for privilege separation.

Exercise 11. Modify rpc_run in profile-server.py so the sandbox runs with a non-root UID (e.g., 6858 instead of 0). Furthermore, ensure that each user's profile has access to its own files, and cannot tamper with the files of other user profiles (e.g., create a unique directory for each user, and use chown and chmod to control ownership and access to that directory).

Consult the man pages for chown and chmod to see what they do. Pass the your UID and userdir to Sandbox.run(). The code in Sandbox.run() calls setresuid and chroot for you, and also calls unshare(CLONE_NEWNET) to prevent the sandboxed code from communicating directly over the network. Look at the man pages to learn more about what those operations are doing.

Remember to consider the possibility of usernames with special characters.

Run make check to see whether your implementation passes our test cases.

Exercise 12. Specify the appropriate fwrule entries for profile and other services. For example, profile should not be able to communicate with auth.

Challenge 2! (optional) Think of some interesting features that you could implement using Python server-side profiles, possibly in combination with extending the sandboxing infrastructure (e.g., providing an API for sending messages between users, or for sharing files between users). For example, can you build profile code that analyzes the social graph of who visited whose profile, or an equivalent to a Facebook wall, all using untrusted profile code?

Write a profile that demonstrates this functionality in profiles/my-profile.py. Describe what your profile is implementing in a comment at the top of the profile source code. Make any changes to your ProfileAPIServer necessary to support your feature.

Challenge 3! (optional) Now that profiles contain Python code, and can give away the user's zoobars, it's important that the user's profile code is not modified by an attacker, and only the correct profile code is executed by profile-server.py.

Create an RPC server that is in charge of modifying user profiles, and which requires a valid user token in order to modify a user's profile. Change the rest of the Zoobar application code to modify user profiles via this RPC server. Set permissions on the profile database so that the rest of the Zoobar application cannot modify profiles directly. Change profile-server.py to read profile code directly from the profile database, instead of accepting it as input to the run RPC call.

make check only does a cursory inspection of the person db, so it may be that your solution is correct but the test fails, or that the test succeeds but your solution is wrong. Therefore, if you've completed the challenge and want us to grade it, add an empty file named challenge3.txt to the lab directory so we know to take a look at your solution.

You are done! Submit your answers to the lab assignment by running make submit. Alternatively, run make prepare-submit and upload the resulting lab2-handin.tar.gz file to the submission web site.