Programmable Dataflows: Abstraction and Programming Model for Data Sharing

Abstract

Data sharing is central to various applications such as fraud detection, ad matching, and improving patient care. However, each solution to data sharing is bespoke and cost-intensive, hampering value generation. We identify the lack of abstractions to control data release as the culprit of the problem. For example, it is common to have constraints on whether to share data that depend on the result of sharing, and evaluating these constraints requires sharing in the first place, leading to a standstill. To help people build solutions to a wide variety of data sharing applications, we propose programmable dataflows, which consist of two components. The first component is an abstraction, the contract, which agents use to communicate the intent of a data sharing action and evaluate its consequences before the dataflow takes place. This helps agents control the release of their data. The second component is a contract programming model (CPM), which allows agents to program data sharing applications catered to each problem’s needs with the contract abstraction. We describe how to deploy those applications on a data escrow to ensure data remains protected from unintended data releases. Our evaluation shows 1) the contract abstraction permits representing a wide range of sharing problems, 2) CPM permits writing programs for complex data sharing problems and 3) quantitatively, our improvements to CPM make sharing programs run efficiently.

1 Introduction

Data-sharing consortia are associations between individuals and organizations that come together to extract value from data. Examples include cross-organizational and cross-jurisdictional data sharing efforts between government agencies geared towards improving decision-making [52], the multi-party market between individuals, ad marketers, and brokers that powers the multi-billion ad-tech business [6], health organizations sharing data to improve patient care [41, 42, 48, 53, 72], financial institutions sharing data to improve fraud detection [3, 4, 10], citizens sharing data with governments to facilitate resource allocation [2], and data sharing to empower IoT technologies [65, 69].

Today, the cost of setting up such consortia is so high that only a few form and remain sustainable. We identify a lack of abstractions to control data release as the culprit of such high cost. Controlling data release often requires input from multiple parties, including legal experts, consultants, and technicians. This process then evolves into a bespoke solution tailored to the hyper-specific compliance, regulatory, commercial, and privacy constraints of each data-sharing scenario. Consider the following illustrative example:

Example 1

(Financial Fraud Detection) Banks are interested in pooling credit card transaction data to train more accurate fraud detection models [3, 24], but they are generally not allowed to reveal their raw data. Moreover, it is not commercially interesting to pool their data, unless every bank has the guarantee that the joint model benefits them, instead of only helping others. Thus, the banks may participate in data sharing if the following requirements are met: i) no raw data is disclosed; ii) each bank contributes sufficient training data; and iii) the joint model can achieve some minimum accuracy on each bank’s test data.

Achieving controlled data release introduces several challenges illustrated by the example. At time 0, the banks do not know whether their data is compatible with the joint model training. Banks need to release some information about their data to prepare the data, but they also cannot release too much, which may violate the privacy of their customers. Additionally, banks want to ensure that the model trained on the joint data is better than the model trained only on their data, but this creates a catch-22 situation: to decide whether to share the data, it is necessary for them to share the data first. Today, these roadblocks to controlled data release prevent the formation of many potentially beneficial data sharing consortia.

In this paper, we propose programmable dataflows to achieve controlled data release, and guarantee that data is only released when it meets the privacy, compliance, and commercial constraints in a given data sharing consortia. We make two primary contributions:

• A conceptual contribution of the paper is a data contract abstraction (Section 3) that permits controlling data release according to a set of constraints as specified by each sharing consortium. It bounds the consequences of sharing by making it explicit who contributes data, what computation is allowed on that data, who receives the result of the computation, and under what conditions such sharing occurs. Crucially, it lets people understand the consequences of sharing before the sharing takes place, addressing the catch-22 situation. We derive the contract abstraction from a model of data sharing which we present in Section 2.

• A technical contribution of the paper is the design and implementation of a contract programming model (CPM) that permits developers to adapt applications to the needs of each consortium (Section 4). The CPM is deployed on top of a data escrow system with special properties, including encrypted memory and remote attestation [86], to ensure that contracts are enforced, i.e., preventing unintended data releases. Once the app is running, agents connect to a server and propose, approve, or edit contracts iteratively, concentrating their efforts on converging to a desired state. All stakeholders have a central place to express their constraints, with the guarantee that data releases are fully under their control.

A data sharing application written with the CPM is deployed on a server, and exposes the functionality available to the participating agents. The agents make calls to REST APIs to manage the contract lifecycle and iteratively work toward their sharing goal. Under this paradigm, stakeholders communicate their preferences and constraints by creating and refining the contracts, collectively converging towards their sharing goal. For example, legal experts may add conditions under which the data can be shared to incorporate legal requirements, and engineers may request changes to formats and sharing procedures to prepare data ahead of sharing. In addition to the primary contributions, we investigate methods to reduce the computational and human cost of running data sharing applications, ensuring high performance of the sharing applications written with the CPM (Section 5).

We ask three questions in the evaluation, which represent the main hypotheses of our work (Section 6). In RQ1, we explore whether the contract abstraction is general enough to model varied and common data sharing problems. In RQ2, we study the effectiveness of the CPM in implementing a diverse set of data sharing consortia (including the fraud detection problem above, a national health consortium for medical data sharing [75], and ad matching done usually with data clean rooms [6]), compared to using alternative data sharing technologies. Third, we quantitatively evaluate the performance of data sharing programs written using the CPM.

To the best of our knowledge, this is the first paper that concentrates, holistically, on the challenge of data sharing and specifically on controlled data release. Notice that controlled data leakage technologies, such as differential privacy, are geared towards providing guarantees over the released data (i.e., that no individual will be at more risk after releasing a result) and not towards controlled data release. In fact, controlled leakage requires controlled release, i.e., differential privacy needs a server where to apply the mechanism and control the hyperparameters, such as privacy budget. Most existing work concentrates on point problems, such as regulation (the legal literature [39, 47, 71, 82]), incentives (the economics literature [22, 30, 67]), or specific execution strategies, which only become relevant after different parties agree on the data to be shared and the conditions under which the data can be shared [26, 28, 50, 66, 81, 89].

Overall, our paper contributes to ongoing calls from government [49] and funding agencies [14], for improving technical data sharing infrastructure. Data sharing is today thought of as an organizational process; this paper contributes technology to make that process more flexible and secure, thus contributing to the formation of more beneficial data-sharing consortia.

The paper is structured as follows. Section 2 presents a data sharing model that conceptualizes a wide range of data sharing problems and distills a set of common challenges. Section 3 introduces the contract abstraction to address these challenges. Section 4 and 5 present the technical contributions of the paper, including the implementation and optimization of a contract programming model that permits developers to write data-sharing applications. Section 6 gives evaluation results. Section 7 discusses related work. Lastly, Section 8 presents conclusion.

2 The Data Sharing Model

In this section, we present a data sharing model we use to systematically think of data sharing problems.

Overview of the Model In the model, data sharing is represented as a series of data sharing states. A state describes which agent has access to what data. Agents have a preference to achieve certain states (goal states) but also need to prevent certain states (constraint states), for example, because of regulatory, compliance, and privacy reasons. Agents move from one state to another via a dataflow.

2.1 Concepts of the Data Sharing Model

Definition 1

(Agent) An agent, $a_i$, is an entity with the agency to participate in data sharing.

In the fraud detection example from Section 1, agents are the banks. The concept of an agent encompasses various entities in data sharing literature, including data owners [68], data providers [35, 66], data consumers [35], data scientists [68], and data requesters [66].

Definition 2

(Data Element) A data element $d_i$ represents some logical grouping of data.

The granularity of data elements is specific to each sharing problem; it may be a text document, image, database, table cell, etc. Data elements may be derived from other data elements via a function f(), e.g., a classifier ML model is a data element built off some input training data (the input data element).

Definition 3

(Data Sharing State) A data sharing state, q, is defined by a set of agents A, a set of data elements D, and a correspondence $s :A \rightarrow D$, that yields the subset of D that is accessible to each agent. That is, $q = \langle A, D, s \rangle$.

Initially, there exists a set of agents $A_0$ with a set of data elements $D_0$. We denote this initial state by $q_0$. Whenever a data element is shared with another agent, there is a transition to a new state; the details of the transition are explained in the following subsection. Data sharing is a series of transitions from states $q_0, q_1, \cdots , q_i, \cdots$. For state i, $q_i = \langle A_i, D_i, s_i \rangle$. Figure 1 shows three sharing states. In state i, $A_i$ = {$a_1$, $a_2$}, $D_i$ = {$d_1$, $d_2$}, $s_i(a_1)$ = {$d_1$}, and $s_i(a_2)$ = {$d_2$}. In state i+1, A and D remain unchanged, and $s_{\text {i+1}}(a_2)$ = {$d_1, d_2$}.

Transitions (i.e., data sharing) are explicitly triggered by agents. Agents want to reach some final state where they have access to certain data elements. In the banking example, each bank wants to access the joint model that achieves some minimum accuracy. Such desired final states are examples of goal states.

Definition 4

(Goal States) For an agent $a_i\in A_i$, their goal states are the set of states $q^g_i$ where $a_i$ gets access to the data elements that they want to access.

Each agent has their own set of goal states. For a set of agents, their common goal states $q^G$ are the intersection of each individual’s goal states: $q^G = \{q: q \in q^g_1 \cap q^g_2 \cap \cdots \cap q^g_m\}$. Reaching a common goal state is the desired outcome of a given data sharing problem.

While agents would like to get access to data elements by entering their goal state, they would also like to avoid states where other agents have access to certain private, restricted data elements. As such, they want to avoid states that violate such restrictions. Such states are represented by constraint states.

Definition 5

(Constraint State) An agent $a_i$’s constraint states are the set of states $q^c_i$ that are disallowed by $a_i$. From $a_i$’s perspective, data sharing should never reach $q^c_i$.

Analogous to common goal states, the common constraint states for a set of agents $q^C$ are the union of each individual’s constraint states: $q^C = \{q: q \in q^c_1 \cup q^c_2 \cup \cdots \cup q^c_m\}$. In the fraud detection example, if a joint model is shared with any of the banks without meeting the accuracy requirement, then these banks have entered a common constraint state.

2.2 Driving the Data Sharing Process

Agents want to transition from an initial sharing state into a goal state while avoiding falling into a constraint state. Transitions between data sharing states are triggered by dataflows, which indicate the exchange of data elements among agents. Dataflow is the central concept in the data sharing model.

Fig. 1

Illustration of three consecutive data sharing states

Definition 6

(Dataflow) A dataflow $t_i()$ is a function that takes: a state $q_i$, a set of destination agents $A_{\text {dest}} \subseteq A_i$, a set of source data elements $D_{\text {src}} \subseteq D_i$, and a function f to be applied on the source data elements. It produces the next state $q_{i+1}$, in which each destination agent $a_j \in A_{\text {dest}}$ gets access to a new data element $d^j$ $=$ $f(D_{\text {src}})$.

$$\begin{aligned} t_i(q_i, A_{\text {dest}}, D_{\text {src}}, f) = q_{i+1} = \langle A_i, D_{i+1}, s_{i+1} \rangle , \ \text {where }\end{aligned}$$

$$\begin{aligned} D_{i+1} = D_i \cup d^j, \ \ \ s_{i+1}(a_j) = {\left\{ \begin{array}{ll} s_i(a_j) \cup d^j & a_j \in A_{\text {dest}}\\ s_i(a_j) & \text {otherwise } \end{array}\right. }\end{aligned}$$

In Figure 1, there are two dataflows. The first one gives $a_2$ access to $d_1$, and the second one gives $a_1$ access to $f(d_2)$. Each dataflow leads to a change in sharing state.

Definition 7

(Data Sharing Goal) Agents $a_i \in A_i$ want to find a sequence of dataflows $t_0, t_1,..., t_n$, which take them from the initial state $q_0$ to a goal state $q_i \in q^G$ (i.e., the agents have reached a common goal state), such that none of the intermediate states is in one of the common constraint states $q^C$ .

We make two important observations based on the model.

Observation 1: In the initial state, agents often lack information about others’ data elements, goal states, and constraint states. In the fraud detection example, banks must know others’ data formats to integrate data before building the model, but these details may not always be available initially. This lack of information makes it hard for agents to know 1) if there exists a sequence of dataflows leading to their common goal states; and 2) what that sequence is.

Observation 2: Agents learn more information as they make progress through the sharing process, considering one dataflow at a time. In general, an agent cannot unilaterally trigger a dataflow from one state to another without agreement from the other agents involved, because a dataflow that helps one agent move towards their goals may enter another’s constraint state. To reach an agreement, agents must communicate their intent of the dataflow and then assess whether the outcome of the dataflow moves the sharing state closer to a goal state while avoiding their constraint states. In some cases, it is not possible to reach a common goal state (e.g., no joint model can meet all banks’ accuracy requirements), and agents will learn this information as they progress.

Main Challenge What makes sharing challenging is the resulting catch-22 problem: to approve a dataflow, agents want to see the outcome of the dataflow first, but to see the outcome, they need to approve the dataflow. For example, the banks need the joint model to check if it meets each bank’s accuracy requirement. Agents do not want to commit to a dataflow without knowing whether the outcome takes them into a constraint state. This uncertainty results in dataflows that do not materialize, even in situations where they would advance the sharing state in a positive direction.

3 The Contract Abstraction

To address the above challenge, we introduce a new contract abstraction that lets agents understand the consequences of a dataflow before the dataflow materializes, so they can decide whether the dataflow should happen.

3.1 Introducing the Contract Abstraction

A contract determines: i) what data is shared; ii) who decides to share data with whom; and iii) when data can be shared.

What data is shared Data shared in a contract is f(DE), where $DE \subseteq D$ is some subset of the data elements, and f is a transformation to be applied on DE before it is shared. Examples of f include training an ML model, running a SQL query over the data, and the identity function, in which case the original data is shared directly.

Who decides to share with whom A dataflow takes place if it is approved by a set of source agents, $A_{\text {src}}$. If it is approved, the outcome of the dataflow is that a set of destination agents, $A_{\text {dest}}$, gains access to f(DE). Typically, $A_{\text {src}}$ needs to contain agents who contribute data elements to the dataflow, which is the set of all $a_i$, such that $s(a_i) \cap DE \ne \emptyset$. In addition to the agents who contribute data elements, $A_{\text {src}}$ may also contain other agents, such as compliance officers, who must sanction the proposed subsequent state. For example, a compliance officer may require that a dataflow can only happen if it does not leak any PII.

When can data be shared In every sharing problem, source agents need to consider common factors when they decide whether to share: what data is shared (f(DE)), and who receives the data ($A_{\text {dest}}$). In addition to these, $A_{\text {src}}$ may have additional requirements on what conditions need to be met for the data to be shared. We categorize those into preconditions and postconditions:

• precondition: these conditions can be checked before f(DE) is computed. In the fraud detection example, the banks’ precondition is to contribute sufficient data.

• postcondition: these conditions can only be checked after f(DE) is computed. The banks’ postcondition is that the joint ML model achieves a target accuracy on every bank’s test data. This ensures that all banks benefit from sharing.

We denote each source agent’s decision on whether the data can be shared with a function, $M_{k} (A_{\text {dest}}, DE, f, C_{\text {pre}}, C_{\text {post}}) \rightarrow \{0,1\} \quad \forall a_k \in A_{\text {src}}$, with 1 representing approval and 0 denial. Then, a dataflow, $t_i(q_i, A_{\text {dest}}, DE, f)$, takes place if $M_k()=1, \forall a_k \in A_{\text {src}}$.

Definition 8

(Contract) A contract is a 6-tuple:

$$\begin{aligned} \langle A_{\text {dest}}, DE, f, A_{\text {src}}, C_{\text {pre}}, C_{\text {post}} \rangle \end{aligned}$$

with a function $M_{k}(A_{\text {dest}}, DE, f, C_{\text {pre}}, C_{\text {post}}) \rightarrow \{0,1\}$ for every $a_k \in A_{\text {src}}$. $A_{\text {dest}}$ represents a set of destination agents, DE represents a set of data elements as inputs to f, $C_{\text {pre}}$ and $C_{\text {post}}$ represent the preconditions and postconditions, and f(DE) is the data to be shared with all of the destination agents. $A_{\text {src}}$ is the set of agents who can approve the contract with $M_k$. The content of a contract is visible to all of $A_{\text {src}}$.

3.2 Using the Contract Abstraction

The contract abstraction captures both the intent and the outcome of a dataflow, so agents can use contracts to communicate the intent of a dataflow, and understand the outcome of a dataflow before committing to it. Agents first propose a contract to the other agents, specifying the destination and source agents, data elements, f(), and the preconditions and postconditions. Once the contract is seen by the other source agents, it may get approved. If a contract instance is not approved, a new refined one may be proposed, expressing a new intent. If a contract is approved, it can be executed to complete the dataflow, when all preconditions and postconditions are met. Together, propose, approve and execute form the three contract operations, which together manage the lifecycle of each contract, from its creation until its completion.

3.2.1 Contract operations help agents work towards the sharing goal

As stated in Section 2, agents may not have complete knowledge of each other’s goal and constraint states at the beginning, and what sequence of dataflows will lead them into a common goal state. The contract abstraction helps agents communicate their goals and constraints through the proposal of contracts, and move towards a common goal while avoiding the constraint states through the execution of approved contracts only. The contract abstraction hides away the complexity of the sharing process by allowing agents to share data through simple operations on the 6-tuple that constitutes a contract.

Contrast the simplicity of the contract abstraction with the costly process that agents must endure today to share data. At time 0, the banks may not know whether their data is compatible, but preparing data for the joint model may reveal more information than they would have liked. Even after solving the preparation problem, they still need to enforce the accuracy and input cardinality requirements. To address these challenges, banks must rely on consultants and manually create data-sharing agreements, making sharing a costly and ad-hoc process. This lack of guarantee, coupled with the costly process of designing bespoke solutions, prevents the formation of many potentially beneficial data sharing consortia.

Finally, we note that while our model needs no assumption on agents’ prior knowledge about others’ goals and constraints, in practice, the use of contracts can be expedited by standard domain and sector-specific DSAs (i.e., data sharing agreements). Examples of these DSAs include Open Banking’s DSA for financial API performance data [25]; California Health & Human Services’ DSA for PHI/PII [32]; and state health departments’ DSAs with explicit breach-notification and re-disclosure limits. [46] Because these agreements are standardized within domains and institutions, participants usually enter consortia with existing background knowledge about each other (e.g., common clauses, review paths, and compliance norms). This makes the revision of contracts more flexible and more feasible to enforce technically than revising lengthy legal paperwork and agreements, even in cases where several rounds of modifications may be needed before reaching consensus.

Fig. 2

Examples of how contracts enable data sharing in the fraud detection example

3.2.2 Example Usage of Contracts

We illustrate how agents use the contract abstraction to enable sharing in the fraud detection example with two banks in Fig. 2. At time 0, bank 1 has the goal state of accessing a model trained over both banks’ joint data. To reach this goal, bank 1 proposes a contract that specifies: destination agents are both banks, DEs are both banks’ data, and f() is the model training function. If bank 2 approves this contract, bank 1 can execute it and achieve their goal. However, this contract does not capture bank 2’s constraints because bank 2 requires that every bank must contribute sufficient data (precondition), and that the result model must meet an accuracy threshold (postcondition). Thus, bank 2 does not approve the original contract, but proposes a new one with these conditions added. Bank 1 reviews the new contract and decides to approve it because the new changes do not violate bank 1’s constraints. The contract is now ready for execution, and both banks have the guarantee that the model will not be released unless all conditions are met. The use of contracts ensures that the only dataflow that can occur is one that leads to a goal state without reaching any constraint states.

3.2.3 The contract abstraction can form a legally binding contract

In law, a contract is an enforceable promise between two or more parties. The following definitions are legal terminology different from those used so far. Contracts require four elements.

• Offer: Proposal that indicates a willingness to be bound on the specified terms.

• Acceptance: Agreement to the terms of the offer.

• Consideration: Value exchanged by parties.

• Manifestation of Mutual Assent: Objective indication that parties intend to form an agreement.

If any element is missing, no contract is formed. Contracts can be written, oral, or implied in actions, and they can be expressed in different forms and languages, including via programming code, as long as they fulfill the elements above.

Therefore, the contract abstraction could be designed to form a legally enforceable contract if the parties so choose. Offer, Acceptance, and a Manifestation of Mutual Assent follow from the contract abstraction definition, and each party’s data and any promised dataflows constitute Consideration.

The legal interpretation of the contract abstraction is a crucial characteristic, and it allows agents to solve data sharing problems that include sensitive data with contracts. Since legal contracts are often subject to cumbersome data sharing and use agreements, it can be beneficial to agents to supplement or displace them with the simplified contracts that the abstraction facilitates.

4 The Contract Programming Model

The contract abstraction is a conceptual contribution of the paper derived from the data sharing model. We now present the technical contribution of the paper: a contract programming model (CPM) that permits developers to write data-sharing applications using contracts. A programming model that allows contract programming needs to meet two requirements.

• First, the programming model has to allow agents to represent a wide range of sharing problems, i.e. supporting arbitrary number of agents, functions, and data elements. We achieve this by providing a cpm_api library with APIs to manage both 1) state about contracts, including their proposal, approval, and execution, which are needed in any sharing application, and 2) application specific state. (Section 4.1)

• Second, data-sharing applications written with the contract programming model should be able to control the release of data: that is, only approved dataflows run. We achieve this through delegated computation with a data escrow. (Section 4.2 and 4.3)

Fig. 3

Examples of APIs from the fraud detection example deployed on a FastAPI server

Solution Overview At a high level, agents implement their data sharing problem with a program written with the CPM. The program exposes a set of functionalities to the agents via a set of APIs, and is hosted on a data escrow server with in-memory encryption and remote attestation (we elaborate on what these characteristics are and why they are needed in Section 4.3). Figure 3 shows an example of the deployed fraud detection example, which includes functions for agents to upload data, propose contracts, share schema of their data elements, etc. Agents then connect to the server via a client to call the APIs. An instance of the application running will be shared among all agents who participate in this data sharing problem (e.g. sharing a link to the same instance among all the banks). This deployment mode is akin to how people use collaborative software such as Google Docs today.

We will be referring to two roles: developers and agents, throughout this section. We first give a brief overview of these roles.

Agent An agent follows its definition from earlier sections, and refers to an entity who participates in a specific data sharing problem, with certain goal and constraint states.

Developer A developer is an entity whose sole responsibility is building a program using the CPM that reflects a set of agents’ desired sharing problem. Therefore, there is no assumption or requirement on whether the developer needs to be an agent itself who participates in a sharing problem. For example, in the banking scenario, the developer may be a software engineer from one of the banks, or an external entity who is familiar with writing CPM programs.

4.1 Programming Data Sharing Applications

While existing data sharing solutions are often built with a mixture of hyper-specific legal clauses and ad-hoc technologies, data sharing solutions implemented with the CPM are programs written by a developer¹. In particular, the program implements the following: i) the endpoints accessible to the intended participating agents of the data sharing application; ii) the functions (f()) available to participating agents. The developer uses the following annotations:

@api_endpoint This makes the annotated function publicly available to participating agents. The developer must choose carefully which functions to expose.

@contract_function This indicates the annotated function accesses data elements and thus requires a contract; they represent the f() in the contract abstraction. Functions annotated with @contract_function may also be annotated with @api_endpoint. The pre- and post-conditions are implemented programmatically within these @contract_function-annotated functions, thus making full use of the programming language’s conditional logic and ability to compute arbitrary functions.

Fig.  7 shows how the annotations are used in the fraud detection example. In the program, upload_credit_ transaction_data is annotated with @api_endpoint, and banks invoke this to upload their data. Meanwhile, show_ schema is annotated with both @api_endpoint and @contract_function. Banks call this to see the schema of every data element in the contract.

Functions may access external software libraries as long as these are available in the data escrow server. For example, they may use a library that implements differential privacy to add noise to a function’s output to control data leakage. A CPM program may contain functions without any annotations, thus permitting developers to factor out common tasks and follow standard software engineering practices to write more complex applications.

4.1.1 Manage Contract and Application States

The CPM provides developers with a cpm_api library to manage both contract and application state. We start by discussing the contract state. Recall that contracts are made of agents, data elements, and functions. Having discussed how functions are supported, we now introduce APIs needed to handle agents and data elements.

register_agent(id, name) Registers an agent in the sharing app to make the agent’s ID available to others. These IDs are used to identify agents in contracts.

DEStore.read($\bullet$), DEStore.write($\bullet$) A DEStore permits: i) writing data elements so they can be referred to during contract creation; ii) reading registered data elements during the execution of an approved contract. We provide different implementations catering to different data element formats. For example, we have implemented a FileStorageDEStore() that provides access to the file system to handle files in various formats (e.g., CSV and Parquet).

Agents operationalize contracts through proposition, approval, and execution. The CPM provides the following APIs to support these contract operations:

propose_contract($\bullet$), approve_contract($\bullet$), deny_ contract($\bullet$), get_pending_contracts(agent_id) are functions available to developers. They may be exposed to participating agents through a @api_endpoint-annotated function in a CPM program.

Notably, cpm_api does not provide a function to execute contracts. Agents call @contract_function-annotated functions in the sharing program and, if there is a contract available that permits the dataflows created by such functions, then these execute.

The above APIs allow developers to manage contract state, which are needed across any sharing application that leverages the contract abstraction. In addition, for any particular sharing application, there may be application specific state that needs to be tracked. For example, in an application that allows agents to pose differentially private queries, the developer may want to track the remaining privacy budget. Another example is that in fraud detection, the banks may want to record the contribution of each bank’s data, then use the contribution as input to some subsequent revenue distribution mechanism. To support these use cases, the cpm_api additionally includes the following APIs:

save_state(key, value), load_state(key) We implement an encrypted key-value store that lets developers save and load arbitrary application states for their needs.

The cpm_api library implements a set of simple primitives for manipulating functions, data elements, and contracts. It offers the above APIs that can be mixed and matched to create a wide suite of more complex applications. As we illustrate in a variety of scenarios in Section 6.2, developers can combine these APIs with arbitrary code and existing libraries to implement the specific requirements of their sharing problem.

4.2 Controlling Data Release with Contracts

In the CPM, a contract can execute if it is approved by all source agents, when an agent invokes a contract_function. During contract execution: i) preconditions are checked, ii) f(DE) is computed, and iii) postconditions are checked. If conditions are met, the dataflow materializes. The key challenge is to ensure that f runs on DE to produce a result conforming to the conditions, without agents seeing the original data. If a participating agent (e.g., a bank) can access the contract DEs on the server on which the sharing program runs, that would violate the contract’s promise that destination agents only gain access to f(DE), not DE.

Instead, contract execution must be delegated to an entity that is not a participating agent, who can ensure that the terms of the contract are implemented. There are two options:

• Participants may delegate their computation to a trusted third-party (e.g., a consultant) [51], and/or rely on a functioning legal system for enforcing the terms of the contract [78].

• Contract execution may be delegated to a trustworthy intermediary built with security and cryptographic techniques, such as data escrows [73, 86], secure multi-party computation [26, 89], homomorphic encryption [7], etc.

Of all solutions that can be used for delegated contract execution, we prefer using a trustworthy intermediary over a trusted one. The key difference is that the latter requires trusting other agents to not leak data. For example, delegating computation to a consultant requires relying on legal safeguards to disincentivize the consultant from releasing the data further. This is cost-intensive and slow, and agents are still prone to others not enforcing the contract’s terms. On the other hand, having access to a trustworthy intermediate removes the trust on other agents to not leak data, leading to much stronger security guarantees.

Requirements for the Trustworthy Intermediary The intermediary must fulfill the following requirements to control data release without trusting agents to not leak data.

• R1: Data is encrypted end-to-end. DEStore.read() and DEStore.write() permit saving and accessing data elements. These functions encrypt and decrypt data on-demand to keep it encrypted at all times.

• R2. Contracts state and application state cannot be fabricated to permit unapproved dataflows. In particular, contract state should only be modified through propose_contract(), approve_contract(), and deny_ contract(), and application state should only be modified through save_state().

• R3. cpm_api implementation cannot be tampered without notice. Examples include installing a backdoor to leak data.

We choose to implement the CPM on top of a data escrow architecture [86]. In particular, a data escrow architecture guarantees that all data on the escrow are only accessible to the escrow by default, and can only be accessed through explicitly approved computation delegated from the agent querying the data. This makes it a suitable technology for implementing the three requirements above for trustworthy contract programming. There can be multiple ways to implement data escrows to meet R1-3, using technologies such as [73, 77, 86]. Our implementation follows that proposed in Data Station [86]. In the following sub-sections, we first give an overview of the escrow’s architecture, then explain how our implementation meets R1-3.

Controlled Data Release vs. Controlled Data Leakage Before we dive into the implementation details, we note the distinction between controlled data release versus controlled data Leakage. In the scope of this paper, ensuring trustworthy contract programming is equivalent to ensuring that no unapproved contracts can execute, i.e. controlling data release. This is not equivalent to controlling data leakage. For example, if agents approve a contract that releases the weights of a joint model, an attacker may be able to reconstruct the data [74, 87]. Techniques that control data leakage are orthogonal and complementary techniques to control data release such as contract programming, and can be implemented in data sharing programs written with the CPM. We will show an example of incorporating differential privacy, a technique for achieving controlled data leakage, in the banking application, combining controlled data leakage and data release, later in the evaluation.

4.3 Data Station Escrow Overview

Fig. 4

Figure from [86]: an overview of the data escrow architecture

An overview of Data Station’s architecture is given in Figure 4. Data Station supports the delegated execution of functions through several coordinated steps.

Upon Data Station’s initialization, its App Register module registers a list of functions that need to access data elements. For a program written with the CPM, this would be all @contract_functions. Agents need to first register themselves to call the escrow application’s APIs. When an agent makes an API call, the Protocol Coordinator authenticates the identity of the agent, and checks if it is a registered function that needs data element access. If true, it delegates the call to the Gakekeeper, which is responsible for checking if the invocation can proceed, according to some defined policies. In the context of the contract programming model, we implement these policies with contracts and the corresponding approval statuses.

If the execution can proceed, the Gatekeeper instantiates a task to run that function using the execution environment, where the function runs in isolation. All data elements in Data Station reside in the storage manager, and are encrypted with their corresponding keys managed by the Volatile Key Manager. As functions in the execution environment run, they will attempt to access data elements. These accesses are mediated by an Interceptor component, which intercepts all I/O calls from the running function to the actual path of the data elements, so it can record information such as which data elements are accessed during the function’s execution.

Lastly, when the function finishes running, the result is returned to the Gatekeeper and, in turn, sent to the agent. Data Station additionally keeps a log that records all computations that occurred inside, which supports auditability when necessary.

4.4 Escrow Security Analysis

We now give a detailed security analysis of our implementation of the CPM on top of the Data Station escrow. We first present a threat model, then explain what security mechanisms are available on the Data Station escrow, and how we leverage these mechanisms to meet R1-3.

Threat Model We consider a set of agents who want to compute an arbitrary function f() over their data elements, following the definitions in Section 2. The Data Station escrow is the architecture that will execute f() on the data. We assume a strong adversary who can gain access to the escrow infrastructure on which the sharing applications run. Thus, the adversary can read disk and memory contents, including data elements in processing, at rest, and any storage of contract and application states. Our implementation of the CPM has to meet R1-3 against such an adversary. We do not assume protection against denial-of-service or side-channel attacks on underlying hardware.

R1: Ensure end-to-end data encryption Since the adversary can gain access to both memory and disk, Data Station must ensure that data remains encrypted end-to-end throughout its lifecycle. To achieve this, the escrow uses a combination of secure hardware enclaves with symmetric key encryption.

During execution, when a function f() needs to process a DE, Data Station leverages AMD’s Secure Encrypted Virtualization (SEV) technology. SEV provides a hardware-backed protection of confidentiality by automatically encrypting all of the operating system’s writes to memory. This ensures that even privileged system components such as the hypervisor, BIOS, or other tenants cannot inspect plaintext data. In particular, the SEV-SNP (Secure Nested Paging) variant used by Data Station reduces the Trusted Computing Base (TCB) to only two elements: AMD’s hardware/firmware and the OS image running on the node. Within the enclave, the DE is decrypted only in encrypted memory before being processed by the function. Furthermore, to guarantee integrity, every message exchanged between agents and Data Station is signed with the sender’s private key, preventing an adversary from undetectably altering computation inputs or outputs.

Every agent has a symmetric key that is shared with the escrow’s Volatile Key Manager that resides in-memory, which remains encrypted. When agents upload data elements, these data are encrypted with the corresponding keys at origin. When a data element is being processed, Data Station first decrypts it in memory using its corresponding symmetric key. To make the result of a function available to the agents, Data Station decrypts the DE in memory, then re-encrypts it with the symmetric key of the destination agent. Data elements remain encrypted throughout the entire process. Combining these mechanisms allows Data Station to ensure that sensitive data never appears in plaintext outside of the protected enclave, while maintaining full end-to-end encryption guarantees.

R2: Prevent fabrication of contract and application states An adversary who gains access to the infrastructure learns all information about the contract states, including both the contents of the contracts and their approval states. Our CPM implementation on the escrow leverages its in-memory database to store the contract states, which are encrypted using SEV. To ensure recoverability of the in-memory database, the escrow maintains an encrypted write-ahead log (EWAL). Every database update is first encrypted with the symmetric key of the agent that issued it and tagged with that agent’s ID. On restart, the escrow replays the EWAL using keys resent by agents, restoring the database. Together, these mechanisms ensure that 1) the contract states are protected against an adversary who gains access to the infrastructure, and 2) the contract states are preserved during machine failures.

For application states, we implement the key-value store that enables save_state() and load_state() in the cpm_api to be encrypted in the same way that data elements are encrypted, ensuring that application-specific states cannot be tampered with without notice.

R3: Prevent undetected modification of the CPM implementation To ensure that the cpm_api implementation is correct (that no one tampered with the deployed software), the server hosting the sharing instance employs remote attestation via AMD’s SEV. Remote attestation produces a cryptographically signed report with a launch measurement (hash of the loaded memory image and configuration) that reflects the binaries, firmware, and parameters in use [20]. The report is signed with a chip-unique Versioned Chip Endorsement Key, issued by AMD and traceable to its root of trust, enabling agents to verify both authenticity and integrity. In the context of the CPM, this allows agents to verify that the sharing program’s implementation is as expected.

Agents check that the attestation certificate chain is valid, that the measurement matches the expected build of the sharing program, and that the platform policies (e.g., firmware version) are acceptable [19]. If the software restarts, attestation must be repeated; any attempt to swap binaries or insert a backdoor would change the measurement, alerting agents about the change in the sharing program.

Executing Arbitrary f() The flexibility of f() being an arbitrary function may pose security risks to the system. To mitigate this, the escrow executes all f() within a jailed environment that mediates all of its I/O, which is a Docker container configured with a FUSE-based storage engine and network redirection, so that all file operations and communication attempts are intercepted. The FUSE layer records attempted data access, while network traffic is controlled by the container to prevent unauthorized channels. In the tradeoff of performance vs security, the escrow biases to security and intercepts every call. Yet, we will show in the evaluation that our CPM implementation remains performant, even when comparing against strong baseline.

Limitations As the underlying escrow uses SEV, it is susceptible to attacks against the SEV technology, such as various side-channel attacks as discussed in [33, 61, 84]. Data Station also reports to not protect against denial-of-service attacks.

4.5 Centralized vs Distributed Escrow Implementation

We now briefly discuss one performance aspect of our CPM’s implementation. Since we implement CPM on top of a centralized solution [86], our contract execution also runs in a centralized fashion (i.e., on the server that runs the sharing application). Yet, we note that the concept of the CPM is not inherently tied to a centralized or a distributed contract execution engine, and a distributed trustworthy third party can also be used to execute contracts, if it brings improved efficiency while meeting the above security requirements R1-3. Note that having a centralized contract execution engine does not prevent opportunities for local optimizations. For example, agents can apply filters to their data locally, before combining them in the CPM and running other computations over the join data. We later show in our evaluation that our CPM implementation on top of the escrow is performant, compared to other sharing technologies, matching the observations made in [86].

5 Improving CPM Efficiency

In the previous section, we introduced components of the CPM for people to program contracts while controlling data release. We now introduce optimizations along two dimensions to further improve the efficiency of contract programming. First, we discuss mechanisms to mitigate the computational inefficiencies introduced by contract management. Then, we discuss how we minimize human intervention by including humans (agents) only when strictly necessary to control dataflows.

5.1 Maintain Computational Performance

Programming applications with contracts affects performance in two ways. First, it potentially introduces unnecessary computation. Second, it complicates the program’s state management. We augment our implementation of the CPM to address both.

5.1.1 Early Termination of Function Execution

Developers can write arbitrary @contract_function in the sharing program. One promise of the CPM is that the result of a function cannot be released if it accesses data elements that are not part of an approved contract, whether unintentionally or maliciously. Recall that the interceptor component on the escrow architecture is capable of recording data accesses. One way to keep the promise above is for the interceptor to track the set of all accessed data elements, then return the set to the gatekeeper that checks the validity of the set. Yet, this approach is inefficient, because the function needs to run to completion before the set of agents accessing the data element is available. We introduce an early termination technique to skip computation whenever possible, thus leading to performance improvements.

Upon invocation of a @contract_function, we pass all sets of approved data elements for the current user and function call combination to the execution environment. The environment then starts two processes in parallel: one runs the function, the other runs the interceptor, which also gets information about the sets of approved data elements. With each access to a new data element (i.e., as the function executes), the interceptor filters the remaining sets of accessible data elements to only keep those that is a superset of the current data elements accessed. The environment runs a loop to continuously check that the set of accessible data elements is non-empty. If it is, the environment suspends (i.e., early terminates) the function execution, skipping computation that would otherwise be wasted.

This optimization is particularly effective, because a typical pattern in many data sharing problems is that data elements are first combined from different agents, before some expensive computation (e.g. running a SQL query or training a joint model) runs on the combined data. Examples of systems that support such workflow include Conclave [81], where mutually distrusting parties pool their private datasets so that queries can be run over the joint dataset; and CrypTen [59], where tensors are first secret shared among parties before expensive operations like neural network layer runs. Early termination allows detecting whether a function call corresponds to the execution of an approved contract before the expensive computation runs. The environment tracks the data element accesses closely and therefore introduces a slight overhead, but it is insignificant in comparison to the large gains of skipping computation, which we later demonstrate in the evaluation.

5.1.2 Materializing Intermediate Dataflows

Ensuring that no unintended dataflow takes place means that only the final result of a contract can be revealed. Sometimes, this leads to developers writing sharing programs that are less efficient than those written without contracts. Consider a group of agents who want to run function f, which first combines data from different agents, and then trains a model over the combined data. f can be decomposed into the sequential execution of two functions: g, which combines the data, and h, which trains the model. h(g) is equivalent to f. But, if agents want to train a different model in f, f still needs to execute g to combine the data from scratch, even though the result g will not change. This repeated execution introduces inefficiencies. Unfortunately, if the developer instead writes an application that separates g and h into two functions, then a contract over g will disclose the combined data to the participating agents. This disclosure does not correspond to an approved dataflow from contracts, and may violate agents’ constraints.

The solution is to allow developers to write intermediate data elements that are computed during the execution of a function, but will not become accessible to any destination agent, i.e., the destination agents are empty. At the same time, we keep this transparent to developers, who will write the intermediate data elements in the same way they write any other data element: with DEStores. To implement this feature, we leverage the execution environment and the interceptor, and modify the behaviour of DEStore.write() to support intermediate DEs. When DEStore.write() is called outside of function execution, it simply stores the DEs. On the other hand, if it is called during the execution of a @contract_function, it will let the system keep a list of intermediate DEs to be created, along with information like the IDs to assign to them, their formats, and the actual content. Importantly, we also need to track these DEs’ origins: which DEs are accessed to create these intermediate DEs. This information is recorded by the interceptor.

Upon the function’s completion, information about the intermediate DEs gets sent back to the system, which then encrypts and stores the DEs. Additionally, each of these DEs will have their origins stored in the contract states. To correctly manage access to these intermediate DEs, when a @contract_function is invoked, the system passes the intermediate DEs along with their origins to the interceptor. The interceptor will set up the set of approved DEs by converting all intermediate DE IDs to the original DEs that are uploaded by agents using the origins in a recursive fashion. Likewise, when there is a DE access, the interceptor will also convert the ID of the accessed DE into IDs of original DEs. Importantly, agents who write sharing programs are agnostic to this complexity, as they are exposed to the same DEStore interface, whether dealing with original or intermediate DEs.

Reusing intermediates effectively decouples the execution of a @contract_function, without actually separating it into different functions. The output remains the same, but the potentially large performance impact is avoided. We demonstrate in the evaluation (Section 6) how this optimization contributes to the performant execution of data sharing applications.

5.2 Improve Human Efficiency

Data sharing requires human (agent) intervention. Source agents need to closely inspect proposed contracts and discuss them with the relevant stakeholders before carefully choosing an action. However, there are situations where agents propose the same contract repeatedly. In those cases, human intervention could be saved if the approval process is codified ahead of the contract proposal. We introduce contract management rules to minimize agent intervention for such cases.

Contract Management Rule In some applications, the source agents know their constraints on what contracts they will approve from the beginning, potentially even without an active contract proposal. In the fraud detection example, one bank may approve any other bank from querying the schema of their data. For such instances, they can express these through a contract management rule using a register_cmr($\bullet$) function in the contract_api.

At a high level, a contract management rule simply specifies which sets of contracts can be automatically approved, in terms of a subset of destination agents, data elements, functions and conditions. These rules interact with propose_contract($\bullet$) to reduce the amount of human intervention in two ways. First, whenever propose_contract($\bullet$) is invoked, in addition to creating a new contract in the contract state, for each source agent involved in this contract, we build two maps. The first map has all destination agents as keys, and the data elements each destination agent tries to access in this contract from the current source agent as values. The second map has the same keys, and the set of data elements automatically approved for this destination agent as values. This map is constructed by combining all existing contract management rules specified by the source agent. If the first dictionary is a subset of the second, then we automatically invoke propose_contract($\bullet$) on behalf of the source agent. Similarly, whenever a new contract management rule is registered, we fetch all contracts for which the agent who registered the rule has not approved yet, and go through a similar process to determine which of these contracts can be automatically approved. Both of these lead to the reduction of agent intervention.

A contract management rule is related to access control mechanisms, such as role-based access control and attribute-based access control. Information conveyed in a rule automatically specifies a set of contracts to be approved or rejected. Contract management rules greatly reduce human intervention in many sharing problems. In the fraud detection example, one @contract_function will return the schema of all contract data elements. If one bank does not consider this information as sensitive, they can specify a rule that automatically approves any contracts that execute this function, saving them from the need to manually approve each one.

6 Evaluation

We design the evaluation to address the main hypotheses of the work, listed below as research questions:

• RQ1: Can we model varied data sharing problems with the contract abstraction? (Section 3)

• RQ2: Is it easier to implement sharing problems using the CPM than using alternative data sharing technologies? (Section 4)

• RQ3: Do the programming model extensions permit writing efficient data sharing programs? (Section 5) In addition to the above RQs, we also include experiments to evaluate our CPM’s performance and scalability against practical baseline. The results are presented in the final subsection.

Table 1 Collection of dataflows that arise in various data sharing problems

6.1 RQ1: Expressiveness of Contract Abstraction

We show that the contract abstraction applies to a variety of data sharing problems. We compile sharing problems from descriptions of real deployments and from the literature (see column "Sharing Problem") and show them as rows in Table 1. For each problem, we include: i) agents’ goals, ii) agents’ constraints, and iii) a diagram of dataflow(s) that take place, where nodes represent agents, and (directed) edges represent dataflows.

We classify the sharing problems into a set of canonical problems described by their dataflow patterns. Qualitatively, we show that the contract abstraction can represent the underlying dataflows in all of the patterns.

Many-to-many pattern In this pattern, a set of agents wants to run computation over their joint data. Every agent contributes data and observes the results of the computation. For example, consider the mobile network federation [80]. Let the providers in the federation be $A_{\text {net}}$ and their data $D_{\text {net}}$. Then, a contract to enable this federation has $\langle$ $A_{\text {dest}}$ = $A_{\text {src}}$ = {$A_{\text {net}}$}, DE = {$D_{\text {net}}$}, f = {spatial queries}$\rangle$. Many sharing problems with this dataflow pattern have the constraint that agents cannot share the raw data with each other. The contract enforces this by construction, as the destination agents would only get access to f(DE).

One-to-many pattern In this pattern, one source agent wants to share data with multiple destination agents, provided they meet certain conditions. For example, an online medical database [54] (MIMIC-II, denoted $a_{\text {m}}$), requires users $A_{\text {u}}$ to have completed a training program before accessing the database (denoted $d_{\text {m}}$). Let the requirement for providing training completion be denoted $c_{\text {train}}$. Then a contract that meets MIMIC-II’s requirements is as follows: $\langle$ $A_{\text {dest}}$ = {$A_{\text {u}}$}, $A_{\text {src}}$ = {$a_{\text {m}}$}, DE = {$d_{\text {m}}$}, f = {direct share}, $C_{\text {pre}}$ = {$c_{\text {train}}$}$\rangle$. Implementation-wise, $c_{\text {train}}$ can be encoded with a function that takes in proof of training completion, then returns the data.

One-to-one pattern In this pattern, a destination agent wants to access data that meets certain requirements. They choose from a pool of potential source agents who want to get something in exchange. This pattern is common in crowdsourcing platforms [11] and online data marketplaces [23, 79]. For example, an e-commerce company is looking for training data to improve the accuracy of their recommendation system by some percentage. An important challenge in these problems is Arrow’s information paradox [12]: destination agents are not ready to pay before knowing the value of data, but source agents will not grant access without payment. The contract abstraction helps by incorporating buyers’ requirements into the contract’s conditions. For example, in a data market for ML, a buyer requires that a private model augmented with the seller’s data should achieve a performance increase of three percent (denoted $c_{\text {perf}}$). The following contract captures this requirement: $\langle$ $A_{\text {dest}}$ = {$a_{\text {buyer}}$}, $A_{\text {src}}$ = {$a_{\text {seller}}$}, DE = {$d_{\text {seller}}$}, f = {direct share}, $C_{\text {pre}}$ = {$c_{\text {perf}}$}$\rangle$.

Many-to-one pattern In this pattern, one destination agent wants to collect data from many source agents. One example is Google training its keyboard prediction algorithm from user-generated text [43]. Today, source agents have little control over their data: once their data is collected, they are "at the mercy" of how the destination agent will use their data. The contract abstraction helps the source agents gain more control over their data by allowing them to specify their constraints explicitly. For example, denote Google by $a_g$, the users from which Google collects data by $A_u$, users’ data by $D_u$, and the keyboard prediction algorithm by $f_{\text {key}}$. Then, the following contract ensures that Google only gets the model trained from the users’ data, but not their raw data: $\langle$ $A_{\text {dest}}$ = {$a_g$}, $A_{\text {src}}$ = {$A_u$}, DE = {$D_u$}, f = {$f_{\text {key}}$}$\rangle$.

Mix of patterns Other sharing problems exhibit multiple patterns. We take the government census as an example. At a high level, the census consists of two sequential patterns: a many-to-one followed by a one-to-many. The bureau first collects data from every citizen, then publishes aggregate statistics about the collected data. There exists one challenge: the census bureau is subject to law [16] that they cannot publish information that identifies individuals. Yet, after the data collection period, the bureau already has every citizen’s original data, and citizens have no way of ensuring, technically, that bureau will keep their data confidential. A contract remedies this situation by only giving access to f(DE), where f() can be a function that adds noise to individual’s data, or a function that aggregates multiple agents’ data together.

Summary We show that the contract abstraction represents all the canonical patterns identified, and that these canonical patterns can express a multitude of data sharing problems.

6.2 RQ2: Effectiveness of the CPM

We now show the effectiveness of the CPM in implementing a wide variety of sharing problems. To this end, we introduce three representative, real-life sharing scenarios: healthcare data sharing [75], ad matching [6], and financial fraud detection. These sharing problems exhibit multiple dataflow patterns, comprise a diverse set of goals and constraints, and span multiple stages of the sharing process, from initial data preparation to final results sharing. Together, they are representative of a large space of existing sharing problems.

To highlight the qualities of the implementation, we compare the CPM implementation of these sharing problems qualitatively with alternative sharing baselines. We first describe the sharing problems in details.

Table 2 LoC Needed to Implement Certain f() Using MPC Frameworks

Table 3 The Space of Supported Dataflows by a Collection of Existing Data Sharing Technologies

Healthcare Data Sharing The Danish National Patient Registry (DNPR) has been collecting medical data on Danish citizens since 1977, which is valuable for research and discovery. One such good use case for the DNPR data, as noted in [75], is to “obtain information on confounders, particularly co-morbidities.” Consider the Danish Adult Diabetes Registry (DADR), who has data on patients’ smoking status and HbA1c (blood sugar level). They want to calculate the causal effect of smoking on HbA1c, and they know that depression is a confounder for these two variables. As discussed earlier, DNPR has data on patients’ historical record of depression, and having access to this data can help DADR get more accurate estimate of the causal effect. While allowing access to DNPR’s data is clearly beneficial, DNPR is subject to the Danish Data Protection Act [5], which specifies that their data can only be processed for scientific purposes. So DNPR cannot directly send their data to DADR, or make their data public, because there is no guarantee that the above constraint will be satisfied.

This sharing problem has three requirements. i) DNPR needs to ensure that their data is only processed for scientific purposes. Thus, they only allow one type of computation that joins agent data with additional confounders (i.e., columns) from DNPR’s data, then calculates the causal effect between a treatment variable and an outcome variable. ii) Agents’ data should include the attribute "CPR", which is a join key that uniquely identifies each Danish citizen [75], so that their data can join with DNPR’s. iii) A large number of agents want to use DNPR’s data, so DNPR wants to automatically approve all those requests to avoid human intervention.

Ad Matching Advertisers want to run analysis over data collected by media publishers to study the effect of their ad campaigns [6]. For example, Don, an advertiser, may want to study the click-through rate of an ad over data collected by Facebook and Youtube, which jointly includes audiences’ responses to an ad, their demographics, and their social media interactions. Don has some idea of what analysis are helpful, such as identity matching (return overlapping users across Facebook and Youtube’s data), user profile expansion (join information about overlapping users), user discovery (find user groups with certain characteristics), and machine learning (identify factors that influence users’ click-through rate). But Don cannot specify all the computations he will run at time 0 (e.g., what queries to ask, what types of model to train, etc.), because such analysis is exploratory in nature. Meanwhile, Don knows that the media publishers will not reveal their plaintext data, but certain computations on the data (e.g., aggregate analysis) may be allowed.

This sharing problem has two requirements. i) Because Don’s analysis is exploratory, he wants functionalities to run a wide range of f(), ranging from analytical queries to ML models. ii) Don’s analysis should be approved on-the-fly, because Don does not know Facebook and YouTube’s constraints at time 0, nor do Facebook and YouTube know what analysis Don wants to run.

Financial Fraud Detection We revisit the example from the Introduction, which has three requirements. i) At time 0, banks do not know if their data is readily compatible or what other banks are willing to disclose about their data. Thus, the banks want support for data preparation functions that disclose information about input data (e.g., show sample, show schema, etc.). ii) Banks not only need functionality to train a model over their joint data, but they also need to incorporate the precondition about input size requirement, and the postcondition about the accuracy requirement. iii) Similar to ad matching, banks want to approve data access on-the-fly.

Summary of Agents’ Requirements We summarize the agents’ requirements across the three sharing problems as follows:

1. Support a wide range of dataflows. f() includes SQL, causal queries, and ML with customized conditions. The cardinality of agents also differs in each problem.

2. Support data preparation. Without knowledge over other agents’ data elements, agents want to ensure that their data is compatible for the desired computation to run. In healthcare data sharing, DNPR wants to ensure that other agents’ data can join with theirs. In fraud detection, the banks want to ensure that their data is combined properly before a joint model is trained.

3. Ensure computational efficiency. In ad matching, Don may need to train multiple models over the joint data for comparison due to the exploratory nature of his analysis. This process should be efficient.

4. Effectively manage contract lifecycles. In the ad matching and fraud detection, agents have to propose and approve contracts on-the-fly, because they do not have complete information over others’ goals and constraints. On the other hand, in healthcare data sharing, DNPR wants to automatically allow all causal queries on their data, as this does not violate their constraints.

Note that these distilled requirements generalize to a large number of sharing problems. For example, a variety of sharing problems in Table 1 require data preparation for joint computation to run (e.g. mobile network federations, data markets, etc), need support for many different dataflows, and demand computational efficiency. By showing how the CPM achieves these requirements across the three examples in the following discussion, we emphasize that the results extend to a large set of sharing problems.

6.2.1 Implementing the Sharing Problems Using the CPM vs Using Other Technologies

To answer the research question, we conduct a qualitative comparison of each data-sharing problem implemented using programmable dataflows and contrast that implementation with a medley of existing data sharing technologies, which we summarize in Table 3. Our criteria for including a technology in the table is that it has to prevent unintended dataflows.

Throughout this section, we use a few selected code snippets (Figures 5, 6, and 7) to illustrate aspects of the CPM.

Fig. 5

Program snippet for healthcare sharing. upload_data_with_CPR ensures all agents’ data includes a "CPR" column. run_causal_query uses Python’s dowhy package to calculate causal effects. upload_cmr uses the default implementation from the CPM. DNPR call this to automatically allow agents to run run_causal_query over their data

Fig. 6

Program snippet for ad matching. propose_contract and approve_contract allow contracts to be proposed and approved one at a time. train_advertising_model allows reusing the saved joined result from Facebook and YouTube’s data

Fig. 7

Program snippet for fraud detection. show_schema, show_sample, and ask_de_aggregate_info support data preparation. train_fraud_model checks the input cardinality precondition and the accuracy postcondition

Support a wide range of dataflows Fig. 5 shows an implementation of the causal query for healthcare data sharing. It first augments users’ data with additional confounders from DNPR’s data, then uses Python’s dowhy package to calculate a causal effect. Fig. 7 shows an implementation of training the joint fraud detection model over all banks’ data, while checking for both the input cardinality and accuracy conditions. Note that methods like train_model_with_conditions and run_casaul_query are APIs implemented by developers that are tailored to their particular sharing problem. Importantly, these APIs are implemented by mixing and composing more general primitives offered by the cpm_api library, such as DEStore.read and save/load_state, with both standard Python and external libraries, as discussed in Section 4. The suite of APIs provided by the CPM allows agents to implement a wide range of dataflows using contracts, while requiring minimal changes from writing regular Python programs. On the other hand, existing technologies are highly constrained in the dataflows they support. For example, none of the technologies in Table 3 allow developers to implement the causal query from the healthcare example easily because most only support specific functions. Some only support SQL [26,27,28, 57, 70, 81, 83, 85], some only ML [29, 88,89,90], and some only private set intersection [31, 36, 44, 60]. In addition, many of them cannot scale to dataflows with more than three source agents [26,27,28, 45, 57, 81, 85], so they resolve the fraud detection problem, nor the ad matching problem with more media publishers. Most of them also do not offer native support for implementing arbitrary preconditions and postconditions, so agents would need to rely on additional technologies or mechanisms (e.g., legal) to enforce these conditions.

While there also exist secure computation frameworks that are meant to be general-purpose such as ABY, Moose, and MPyC [8, 37, 76], users have to write every functionality needed by their sharing problem from scratch, which is cost-intensive. Table 2 shows the lines of code needed to implement certain computations using these frameworks. While these functionalities take at most a few lines to implement in Python with an existing Python library in the CPM, they are complex to implement with secure computation frameworks (e.g., >200 lines to implement set intersection, 800 lines to implement AES with ABY).

Support data preparation upload_data_with_CPR in line 4-10 from Fig. 5 checks that all agents’ data contains the "CPR" column, before they are uploaded. This specialized @api_endpoint ensures that all agents’ data contains a standard attribute that is used downstream for joining the data. In Fig. 7, the banks implement a suite of functionalities for data preparation, including show_schema, share_sample, and get_de_aggregate_info. With these functions, the banks disclose information to prepare their data in a controlled way, without entering constraint states.

The banks can further incorporate controlled data leakage into their application with differential privacy, using the CPM’s save_state and load_state for managing application states. In line 21-28 Fig. 5, the banks implement a function for the other banks to query aggregate information about their data. Each query will produce a noised output that consumes some amount of privacy budget. The CPM program allows the banks to specify how much total budget they tolerate, and automatically tracks the remaining budget with the encrypted key-value store that implements application states.

Support for data preparation is lacking from existing data sharing technologies. For example, a large number of works in Table 3 assume that data are horizontal partitions of the same table that share the same schema [26,27,28, 81, 89]. However, in most cases, it is rare that data from different agents are schema-aligned. Hence, these technologies assume the existence of additional data preparation tools that still need to share data, so not integrating them into the end-to-end data sharing problem is a limitation.

Ensure computational efficiency Line 13-17 in Fig. 6 shows how to reuse the combined data from Facebook and YouTube. train_advertising_model first checks if the joined data exists. If not, it creates and saves the joined data. If yes, it directly goes to model training. Recall that when DEStore.write() is called inside a @contract_function, the system creates an intermediate DE along with its content, format and origin DEs. Importantly, the intermediate DE itself (in this example, the joined data) will never be directly accessible to any agents. Upon accessing this intermediate DE, the system will automatically manage the DE accesses using the recorded origins. These features together permit the reuse of computation.

Such capability is missing from the technologies in Table 3. Most use secure multiparty computation (MPC) to prevent unintended dataflows. To use MPC, agents define what computation to run a priori, and the technology constructs a circuit for that computation so that all agents get access to the output of the circuit. Since the entire purpose of these technologies is that agents do not learn anything about each other’s data elements other than the output, these technologies offer no support for reusing computation. On the other hand, while there exist technologies for running SQL and ML individually, they cannot be chained together because no agent can have the joined results, as that would violate the agent’s constraints.

Fig. 8

Left: Saving Intermediate’s Effect in train_advertising_model with Logistic Regression. Middle: Saving Intermediate’s Effect in train_advertising_model with MLP. Right: Early termination’s Effect in train_fraud_model

Effectively manage contract lifecycles Since agents do not know each other’s goals and constraints completely in ad matching and fraud detection, they need to propose and approve contracts on-the-fly. Therefore, they include propose_contract and approve_contract explicitly in their sharing program, using the default implementation provided by the CPM, shown in Fig. 6. On the other hand, DNPR wants to approve all incoming causal queries automatically, so they include the upload_cmr function to achieve that.

We are not aware of any other technology that enables the kind of interaction facilitated by CPM. Some technologies assume that agents go through an initial phase to agree upon a set of computations to run, without giving sufficient details on how that initial phase can be enabled. For example, Conclave [81] assumes that "parties agree via out-of-band mechanisms on the (relational) query to run". Cerebro [89] assumes that "parties come together in an agreement phase" to decide on the computation and other parameters of the system, and that "this agreement is enforced by an external mechanism, e.g., through a legal agreement". Many other SQL-based systems [26,27,28, 85] assume that the queries to run have already been agreed upon. However, agreeing on the initial phase involves revealing information about agents’ data, and it is challenging for agents to reveal sufficient information to reach an agreement while avoiding the common constraint states, without support from the sharing technology. Our CPM offers a suite of functionalities for agents to manage the lifecycle of contracts effectively.

Summary CPM permits implementing complex sharing problems that include data preparation. It ensures the efficient management of contract lifecycles and computation of contract functions.

6.3 RQ3: Improving Computational Efficiency

We first study the effect of saving intermediates DEs, and then the impact of early termination on the execution time of contracts. Both extensions lead to significant performance gains. In all of our following experiments, we use n2d-highmem-8 instances (8 virtual CPUs and 64GB of RAM) from Google Compute Engine, with 100GB of disk. We generate datasets with sizes ranging from 10MB to 1GB, which are comparable to a variety of real-world, sharable datasets (e.g. popular datasets on NYC Open Data include DOB Job Applications [9] that is 1.5G, and monthly NYC taxi trip data [1] that are hundreds of MB.)

6.3.1 Saving Intermediate DEs

As explained in Section 5, the CPM permits reusing intermediate computation results from contracts, which can be accessed by @contract_functions but no agents. Such feature is desired in many examples. In the ad matching problem, Don may need to train multiple models over Facebook and YouTube’s combined data to obtain a desired one. We study the benefits of reusing intermediates in the ad matching problem.

Experimental Setup We manually downloaded data that Facebook and YouTube collect about their users, then generated a simplified schema based on the downloaded data [17, 18] to ensure the practicality of our example. Both datasets contain attributes including name and email, which will be used as join keys. Each dataset includes some additional attributes, such as age, marriage status, have users liked a Facebook page with games-related content, and have users clicked on some ad displayed on YouTube. We run the experiment on various dataset sizes: 50MB, 100MB, and 200MB, corresponding to 500K, 1M, and 2M dataset rows.

We compare the performance of two implementations of train_advertising_model: one uses the CPM’s extension to store intermediates, and the other does not. In both implementations, train_advertising_model first runs a fuzzy join between Facebook and YouTube’s data on names and email, and then trains either a logistic regression model or a multilayer perception model over the joined data. Without intermediates, the implementation has to combine the data from scratch every time. With intermediates, the implementation only needs to combine the data the first time it is called, and subsequent calls to this function can skip this step. Line 13 to 17 in Fig. 6 show this implementation.

Metrics We evaluate the total time it takes to run the train_fraud_model function over five runs. The runtime is measured end-to-end.

Results The left and middle subplots of Fig.  8 show the total runtime of the LR model and MLP models. The runtime is broken into: 1) constant time introduced by the system to run every function, which is fixed at about 2 seconds, 2) time to train the model, and 3) time to combine the data.

Reusing intermediates for the LR model is effective. With small datasets (500K rows), the join finishes fast, so the two implementations have similar runtime. As the number of records increases, the implementation without reusing intermediates takes proportionally longer to finish (>2X for 2M records). The more times f() is run, the more significant the saving become. For the MLP model, there is still runtime saving, albeit less significant than the LR model. This experiment shows that with little additional effort from the developers to reuse intermediates, agents can implement sharing problems more efficiently.

6.3.2 Early Termination of Function Execution

When the data elements accessed by a contract_function are not part of any approved contract, we want to terminate function execution early to skip computation. We study the efficiency improvement in running the train_fraud_model function in fraud detection.

Experimental Setup We implement 2 baselines. i) No early termination. In this case, f() runs to completion, even if it accesses unapproved data elements. ii) With early termination. Here, one bank has an approved contract to access some other banks’ data, but the execution of train_fraud_model will access data not included in the contract. As soon as unapproved data access is detected, the execution stops.

To simulate the banking example, we generate data using a Kaggle schema, containing synthetic data for fraud detection [15]. We run the experiment on the following dataset sizes: 10MB, 100MB, and 1GB.

Metrics We evaluate the average time the system spends on running the train_fraud_model function over five runs.

Results The right subplot in Fig.  8 shows the average runtime for the baselines for different data sizes. The y-axis is in log scale. Early termination leads to runtime savings. Over all sizes of data, early termination consistently shows performance gains in orders of magnitude. Note that, when the data size is small (<= 100 MB), the time spent running f() with early termination is about 2s for every run. This is a parameter set by the system to run the @contract_function, before checking that the remaining set of approved data elements from contracts is still non-empty. When the data size is larger (1GB), the system spends more time reading the data, before it can determine that there are no more approved data elements. As a result, time spent running f() also increases.

Importantly, in many data sharing problems, the computation agents want to run has a similar structure to the train_fraud_model function, where they first read the data from different agents, combine the data, and then perform some computation over the combined data. These computations particularly benefit from the early termination optimization, as this technique can filter the set of approved data elements before the computation on the combined data element starts, which is typically the bottleneck.

6.4 CPM’s Efficiency in Running Sharing Programs

We now compare the CPM’s efficiency in running sharing programs against practical baseline to show it is performant. In particular, we compare against Secretflow [64], which is a SOTA, open-source framework for building privacy-preserving data sharing applications that has been shown to outperform other MPC-based ML sharing frameworks (e.g., MP-SPDZ [58], TF Encrypted [21]) in many settings.

Fig. 9

Top: Two agents, end-to-end runtime of the CPM program vs Secretflow, for a logistic regression model on replicated breast cancer dataset. Bottom: Five agents with other settings kept the same.

Experimental Setup We follow the example given in Secretflow’s repository of running a logistic regression model over UCI’s breast cancer data [38]. To test the scalability of both our CPM’s implementation and Secretflow, we run the experiment over various dataset sizes: 10MB, 100MB, and 1GB, by replicating the original dataset 100, 1000, and 10000 times. We also run our experiment over a two-agent setting and a five-agent setting. In both settings, the dataset is distributed evenly across all the agents. We exactly replicate Secretflow’s model implementation with a CPM program. We use the same machine spec as the previous subsection.

Metrics We evaluate the average, end-to-end time it takes to finish running the sharing program over five runs. For Secretflow, this includes the time to run the sharing protocol to train and validate the model. For the CPM, this includes the time for agents to upload their datasets, propose and approve contracts, and execute the model training and validation function.

Results The top and bottom subplot of Fig.  9 show the runtime of the sharing problem, under the two-agent and five-agent settings, respectively. We further break down the runtime of the CPM into: 1) CPM data upload time: time for agents to upload data elements, and 2) CPM execution time: time to train and validate the model. This breakdown is given in the two subplots in Fig.  9 as proportions. We observe that the runtime of other cpm_api like contract proposition and approval is minimal, since they are just simple database read/write operations. Thus, they are omitted from the plot, and merged into the data upload time. The x-axis shows the total size of data, and the y-axis shows the average runtime in seconds, given in logarithmic scale.

We make two main observations. First, across all settings of the experiment, the CPM implementation of the sharing problem is an order of magnitude more efficient than the Secretflow version. The CPM’s runtime scales near linearly with dataset size, mostly from the symmetric encryption of datasets during uploading, and the symmetric decryption of datasets to execute f(). Once the data is decrypted in memory, the time to execute f() is close to that of the plaintext version. This is due to the efficiency of the underlying escrow for function execution, compared to other sharing technologies like MPC, matching observations made in [86]. Second, the runtime of the CPM implementation remains consistent across the two-agent and five-agent settings. For CPM, as long as the total dataset size remains the same, the total encryption/decryption time will stay consistent, thus contributing to little change in runtime. On the other hand, Secretflow’s runtime scales with the number of agents, due to the increased communication cost between more agents. For example, for 100MB of data, the average two-agent Secretflow runtime is about 200s, and the five-agent runtime is >1200s. Note that on 1GB of data, Secretflow exhibits an out-of-memory error on our experiment machine spec (n2d-highmem-8 instances). Overall, these results show that CPM offers an efficient and scalable solution for implementing data sharing problems.

7 Related Work

In this section, we first present various technologies designed to enable the formation of data sharing consortia, and discuss their connection to our work. We then revisit examples of real-life data sharing consortia across multiple domains and show how our solution can help implement such consortia.

Sharing Technologies Many systems have explored how to let organizations perform relational analytics without exposing raw data, typically by re-purposing SQL to operate securely across silos. One line of work begins with compiling SQL into secure multiparty computation [26], and later provided extensions to control information leakage through differential privacy and to improve scalability via approximate query processing [27, 28]. Other research has focused on the secure execution of core relational operators, such as joins and aggregates [85], on making secure SQL more efficient by splitting computation between local cleartext execution and MPC [81], or by tolerating adversarial behavior among participants [70]. In industry, Snowflake’s Data Clean Rooms provide governed environments where companies can combine data and run approved SQL workflows without violating confidentiality. However, as noted in Section 6.2, many of these works have various limitations on the data sharing consortia they support, such as limiting max number of source agents, restricting the types of SQL operators they support such as UDFs [26,27,28, 57, 81, 85], and/or making strong assumptions about the underlying data (e.g., all agents’ data are horizontal partitions of the same table with the same schema [26,27,28, 81]). Our CPM is designed to address many of these limitations to enable many sharing scenarios.

Sharing Technologies for Machine Learning Machine learning application is another important target of data-sharing consortia, which aims to enable collaborative model training or inference without sharing raw data. Various federated learning techniques [29, 56, 63, 90] allow multiple parties to jointly train models by keeping data local and only exchanging updates, often augmented with privacy mechanisms such as differential privacy or secure aggregation to mitigate information leakage. In addition to compiling ML programming into MPC protocols, Cerebro [89] additionally provides auditing that allows parties to backtrack the input during private computation, increasing the participating parties’ accountability. While these works make valuable contributions to the technical aspect of secure ML collaboration, addressing the full deployment lifecycle is out of their scope. In practice, ML-based consortia require extensive preparation and validation steps that demand flexible computation composition [55]. CPM supports such end-to-end workflows by allowing arbitrary contract functions to combine ML training with data preparation and evaluation, governed by explicit pre/postconditions. This flexibility enables participants to enforce their customized sharing policies within one framework, whereas existing federated or MPC-based systems would rely on external coordination or legal mechanisms, adding operational complexity.

General-purpose Sharing Technologies In addition to SQL and ML workloads, there are also work that provides a more general-purpose computation framework [8, 37, 76], and work that supports smaller primitives such as private set intersection that could be combined with other technologies to build data-sharing consortia [31, 36, 44, 60]. In particular, Aby [37], Moose [8], and MPyC [76] enable developers to implement arbitrary secure computations by offering abstractions over Boolean, arithmetic, and mixed circuits, with protocol libraries that can be composed for a wide range of applications; however, as shown in Table 2, they generally remain as low-level tools that are not straightforward for non-experts to adopt.

Data Sharing Consortia in Practice There are numerous data sharing consortia that span diverse application areas, and new initiatives emerge as people see their potential. We highlight a few representative examples. In finance, regulators and banks form consortia to jointly monitor transactions and detect illicit behavior. For example, Australia’s AUSTRAC [4] aggregates reports from financial institutions to flag money laundering and terrorism financing, and TMNL (Transaction Monitoring Nederland) in the Netherlands enables major banks to share cross-bank payment data to detect suspicious patterns not visible within any single institution [10]. For any of these analyses that can be defined programmatically, the CPM can provide additional technical guarantee in addition to these institutions’ existing legal and regulatory mechanisms. In healthcare, the DNPR report is one example of institutions realizing the benefits of sharing data for research and analysis. Several works propose techniques to facilitate privacy-preserving health data sharing [41, 42, 48, 53, 72]. Notably, VaultDB [72] was piloted across three health systems in the Chicago area to answer public-health queries under HIPAA compliance. Finally, the advertising industry is another domain where data-sharing consortia are impactful. For example, Data Clean Room [57] has reported collaborations between Snap and Booking.com to run analyses on their joint customer data [13]. While technologies like VaultDB and Data Clean Room are limited to SQL-based analyses, the CPM supports a broader set of computations and enables explicit governance through programmable contracts, which can extend the capabilities of existing data sharing consortia. For example, in addition to letting agents run the usual SQL-based analyses, the CPM can additionally incorporate data preparation and integration functionalities.

8 Conclusions

While sharing data increases its value for agents, sharing problems are diverse and have their own bespoke solutions. We create a model that represents any data sharing problem, and derive from it the contract abstraction that allows agents to express the intent and understand the consequences of a dataflow, before the dataflow takes place. To permit agents to program their sharing problems with contracts, we implemented a contract programming model. The abstraction and the programming model allow agents to implement complex sharing problems that tailor to their needs.