_id
field if not provided: http://docs.mongodb.org/manual/tutorial/insert-documents/#review-the-inserted-documentLook at MongoDB's example queries: http://docs.mongodb.org/manual/tutorial/insert-documents/
They recommend embedding tables (documents) in one another as opposed to using foreign keys
https://stackoverflow.com/questions/5841681/mongodb-normalization-foreign-key-and-joining
MongoDB doesn't support server side foreign key relationships, normalization is also discouraged. You should embed your child object within parent objects if possible, this will increase performance and make foreign keys totally unnecessary. That said it is not always possible, so there is a special construct called DBRef which allows to reference objects in a different collection. This may be then not so speedy because DB has to make additional queries to read objects but allows for kind of foreign key reference.
Still you will have to handle your references manually. Only while looking up your DBRef you will see if it exists, the DB will not go through all the documents to look for the references and remove them if the target of the reference doesn't exist any more. But I think removing all the references after deleting the book would require a single query per collection, no more, so not that difficult really.
If your schema is more complex then probably your should choose a relational database and not nosql.
You can query for a subset of a document
You can insert anywhere in an array (not clear how fast)
$ a = db.arrays.find({_id: ObjectId("546fa4e840694746c9f063d6")}).next().array; a.length;
2304
$ db.arrays.update({ _id: ObjectId("546fa4e840694746c9f063d6")}, { $push: { array: { $each: a, $position: 0 }}});
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
Meteor.users = {
username: String
emails: [ String ] array
profile: {
username: String
masterKeySalt: Salt // used to derive the masterKey = KDF(password, masterKeySalt), masterKey != password hash that Meteor stores
encAuthKeyPair: WrappedAsymmetricKeyPair // an asymmetric key pair like RSA for this user, encrypted and authenticated under the user's masterKey
publicKey: PublicKey // this user's public key, for the server's convenience
}
}
FriendList = {
username: String
friends: [ Friend ] array
merkleTree: MerkleTree ID // v2: a merkle tree over the 'friends' array
date: Date // v2: last time FriendList has changed (new friend or deleted friend)
signature: Signature // v1: SIG_ownerSK("friendlist", username, aclRootData);
// v2: root hash signature -> SIG_ownerSK("friendlist", username, rootHash(merkleTree))
aclTreeId: AclTree ID // the ID of the ACL tree over all friends
}
Friend = {
username: String
publicKey: PublicKey
macPublicKey: MAC of PublicKey // v1: MAC_masterKey("friend", "publickey", "mac", publicKey)
// v2: we don't need this, we have a Merkle tree over all Friend objects
encAuthFriendKey: WrappedSymmetricKey // RAND_masterKey(friendKey)
}
Question and answer: do we need merkle tree here? what happens if the server tricks us into believing a
defriended friend is still a friend? Will the JS code automatically send him sensitive
keys? Not the wall key which is only sent once at friend-time. Maybe new album keys will be sent to him? Yes, indeed. Creating
a new album means we'll need an ACL tree, to which we'll add all of the user's friends.
Thus, we need to be able to remove people from the list of friends and update the client's
root hash fast. With hash chains this is slow because removing a link means updating all
the next ones: O(n)
, with Merkle Trees this is faster in O(log n)
but we'll have to
manage the resulting free space. It seems that for any Merkle-tree like structure we can
keep a free-space list with the leaf # where we can add new nodes.
Note: In our first implementation we just have one single content key
that is used to encrypt everything, so we do not need freshness on the friend list because we never send new keys to all of our friends. We only change the content key
by reencrypting an ACL tree path when we remove a friend. And we only add new users to the ACL tree.
Note: The server can lie across multiple clients for the same user. And the user is unlikely to detect it.
Here there's an initiator who sends the friend request and a target who receives it.
// v1
FriendRequests = {
initiatorUsername: String
targetUsername: String
initiatorIsDone: Signature // SIG_initiatorSK("friend request finished", full database row), target verifies and deletes row
}
// v2
FriendRequests = {
initiatorUsername: String
targetUsername: String
initiatorDhGa: Null or DiffieHellmanPublicKey
signDhGa: Null or Signature
targetDhGb: Null or DiffieHellmanPublicKey
signDhGb: Null or Signature
initiatorIsDone: Null or Signature // SIG_initiatorSK("friend request finished", full database row), target verifies and deletes row
}
Once the initiator has filled in the initiatorIsDone
field, it updates its Friends table.
Once the target has verified the initiatorIsDone
field, it updates its Friends table and deletes the database row.
DefriendRequests = {
initiatorUsername: String
//initiatorSig: Signature // could also authenticate these, but it might be overkill
targetUsername: String
}
The target user (i.e. the defriended user) periodically checks this table or gets notified of a defriend (can hide this in the UI since typically people don't want to see they've been defriended. Facebook certainly does not let you know).
Then, the target user will rekey its: albums and wall. What about comment threads? Those are keyed with the parent object's key (more or less), so the key changes automatically.
What about conversations? One-on-one private conversations between you and the user can remain unaffected and stay active. Group conversations that you own can also remain active, since it might not always make sense to kick out the defriended person. Group conversations that you do not own can remain active, since you have no authority on the ACL edits. - The UI can display non-friend conversation members in red, or something of the sort.
Profile = {
objid: Bytes // Randomly selected. Everything needs objid for ACLs.
username: String
aclVer: AclVersion
// TODO: typically when we include identity info in signatures (like in
// a quack), we only sign the username and not the full name, so it seems
// that the fullname should never even be displayed because the client
// cannot authenticate it.
// We can maybe have each user keep a signature in their profile that binds
// their username to their full name?
fullname: String
nameSignature: Signature // SIG_ownerSK("fullname", username, fullname)
profile: {
photo: Bytes // AE_ownerSK(photo_bits)
field1: value1
field2: value2
...
}
signature: Signature // SIG_ownerSK("profile", username, date(), aclVer, profile)
}
Server can replay old profiles.
Albums = {
objid: Bytes // Randomly selected object ID
name: Bytes // the encrypted name of the album
owner: String // the owner's username
timeStamp: Date // the last modified date
aclVer: AclVersion // the current version of the ACL on the album
photos: {
objid: Bytes // Randomly generated.
owner: String // the photo's owner, would be the same as the album, so we might not need this here (gotta be careful about integrity though)
date: Date // the last modified date
aclVer: AclVersion // the current version of the ACL for this photo
encName: Bytes
encAttrs: {
field1: value1
...
}
encData: Bytes
}
merkleTreeId: MerkleTree ID // the merkle tree over the pictures in the album (can lookup root hash)
signature: Signature // root hash signature -> SIG_ownerSK("album", owner, date, aclVer, objid, encNameAndAttrs, merkleTreeId)
}
Quack = {
id: AutoInteger
type: String
date: Date
author: String
encMsg: Bytes
signature: Signature
aclVer: Integer
prevQuackHash: Hash
prevQuackId: Quack ID
}
Dialogue = {
id: AutoInteger
type: String // 'wall' or 'conversation'
owner: String // the username of the owner
title: String // the title
lastQuackId: Quack ID
}
See the dialogue/quack code in lib/dialogue.js
See the dialogue/quack code in lib/dialogue.js
See the dialogue/quack code in lib/dialogue.js
See the dialogue/quack code in lib/dialogue.js
MerkleTree = {
id: AutoInteger
rootNode: MerkleTreeNode ID
leafNodes: [ MerkleTreeNode IDs ] array
emptyLeafNodes: [ MerkleTreeNode IDs] array // TODO: do we need integrity on this? what happens if the server tricks us into overwriting something? could delete an arbitrary photo: increases ability of server to forks us. Ans: No. If we fill empty leaf nodes with the value 0, this can be verified using the Merkle tree.
}
MerkleTreeNode = {
id: AutoInteger
isEmpty: Boolean
hash: Bytes // Should contain hash(leftChild.isEmpty, leftChild.hash, rightChild.isEmpty, rightChild.hash) if not leaf, hash(content) (or objid for photos) if leaf.
parent: Null or MerkleTreeNode ID
leftChild: Null or MerkleTreeNode ID
rightChild: Null or MerkleTreeNode ID
}
// helps us find an object (specified by its objId) in the merkle tree
MerkleTreeLeafIndex = {
treeId: MerkleTree ID
objId: Bytes
nodeId: MerkleTreeNode ID
}
We only append leafs to Merkle Trees, possibly creating new internal nodes or root nodes. We never delete nodes from the tree. If we delete a leaf, we mark it as deleted and keep track of it so we can place something in it later.
Unlike Merkle Trees, ACL trees have copy on write semantics, so they need to be easily copiable. When someone is revoked access a new ACL version is created and a new tree which excludes that person needs to be forked off.
We implement this in an easy way by keeping all versions of the root node and allowing all of its children to reach it. MongoDB allows us to easily append to arrays using $push
.
AclTree = {
id: AutoInteger
objid: Bytes // The object ID of the object the ACL is protecting
aclVer: Integer // The current version of this ACL. Starts at 1.
owner: String // ACL owner's username
signature: Signature // SIG_ownerSK(hash(this)), includes the hash of the root node hash
// We can't have this be a typical AclNode because the encKeys field needs to be
// an array of encrypted tuples of keys. Can't we? MongoDB is typed? Seems like we
// can actually :)
ownerEncKey: [ WrappedSymmetricKey ] // The content key encrypted under the owner's key
rootNode: AclNode ID
// Don't see a good reason for this when we have AclIndex
//leafNodes: [ AclNode ID ] array // In insertion order
numUsers: Integer // the number of users on this ACL (excluding owner)
//numNodes: // might need
}
// TODO: adding a new user to version 2 of a tree will need to be done carefully. place the hashes in the 2nd entry of the array.
// or have an initial version number in the node
AclNode = {
id: AutoInteger
// The node key encrypted under the ACL owner's node key. We need this for
// quickly retrieving a node's key during updates, etc.
ownerEncKey: WrappedSymmetricKey
// If this is the root node, then this is an array of all versions of the content key,
// where each entry is a version and is encrypted under the root's left & right children
// keys.
// - [ ( WrappedSymmetricKey, WrappedSymmetricKey ) tuple ] array
//
// If this not the root node, this this is just a tuple with the encrypted node keys
// under the keys of this node's children.
// - (WrappedSymmetricKey, WrappedSymmetricKey) tuple
encKeys: Array or Tuple, see above
hash: Bytes // H(encKey, H(leftChild), H(rightChild))
parent: AclNode ID // null for the root node
leftChild: AclNode ID // null for leaf nodes
rightChild: AclNode ID // null for leaf nodes
}
// Maps a user to its leaf node in a specified ACL tree
// Note: we can enumerate all leafs in a tree with this
AclIndex = {
treeId: AclTree ID // The tree ID
username: String // The username
isDeleted: Boolean // True if this user was deleted from the tree
nodeId: AclNode ID // The node where you can start the search for the keys for this user.
keyIdx: Integer // Each node stores 2 keys, this index tells you where to find the key for this user.
}
If you look at ACL index, you notice that it maps a username to its leaf ID in the ACL tree, but it does not include the ACL version number. Presumably, the username could be remapped to different leafs as the ACL changes. We explicitly disallow this (to simplify implementation work). ACLs are add-only, which means deleting someone from an ACL means marking them as deleted and reencrypting the keys on their path.
TODO: make sure to use authenticated encryption along the path of some usr in the acl tree.