Diary of a Tech Savant and Servant Leader - All things technology, product, and engineering leadership.

Abhishek Tiwari

Privacy in data systems has traditionally focused on protecting sensitive information as it enters a system - what we call input privacy. However, as systems become more complex and capable of inferring sensitive information from seemingly harmless data, the importance of output privacy has gained significant attention.

Input vs Output Privacy

Secure multi-party computation (SMPC) enables organisations to collaborate on sensitive data analysis without directly sharing raw information. However, seemingly harmless aggregate outputs, particularly private set intersection (PSI), can leak individual-level information when analysed strategically over time.

Birds of a Feather Leak Together: The Set Bias Privacy Problem

Tech companies and large consumer businesses are grappling with how best to protect end-user data while maintaining pace of innovation and competitive edge. Two distinct approaches have emerged: top-down and bottom-up privacy. Understanding these approaches is essential for anyone involved in privacy engineering, product development, or driving tech policy decisions.

Top-down vs. Bottom up Privacy

Safeguarding individual privacy inherently means data minimisation i.e. limiting the collection and disposal of data. This principle has been a cornerstone of privacy advocacy and is even enshrined in regulations like the EU's General Data Protection Regulation (GDPR). However, a research published by Ponte et. al is challenging this fundamental assumption, introducing what they call the "Where's Waldo effect.

The Where's Waldo Effect in Privacy

Homomorphic encryption is a powerful cryptographic technique that allows computations to be performed on encrypted data without decrypting it first. This blog post will introduce the concept of homomorphic encryption and demonstrate implementations using Python.What is Homomorphic Encryption? Homomorphic encryption is a form of encryption that allows specific types of computations to be carried out on ciphertext.

Exploring Homomorphic Encryption with Python

Microservices have become the dominant architectural paradigm for building large-scale distributed systems, but until now, their inner workings at major tech companies have remained shrouded in mystery. A recent new paper by Huye et. al  - researchers at Tufts University and Meta - provides an unprecedented look under the hood at Meta's massive microservices architecture.

Microservice architecture of Meta

For big data and complex data processing, data pipelines have emerged as a popular solution for managing and manipulating data. They provide a systematic approach to extract, transform, and load (ETL) data from various sources, enabling organizations to derive valuable insights. However, as with any technology trend, data pipelines have not been immune to misuse and overuse.

Data Pipelines: The Hammer for Every Nail

Having a well-defined strategy is crucial for success of a modern business. A good strategy should address the core problems or challenges faced by a business, provide guiding principles on how to tackle them, and be supported by the coherent actions and right resources. In this article, we will explore the core elements of a good strategy.

Kernel of a good strategy - Why, How and What

The ability to deploy applications quickly and efficiently is crucial for organizations. Agile methodologies and continuous integration/continuous deployment (CI/CD) pipelines have become the norm, enabling rapid release cycles and frequent updates. However, amidst the drive for speed, ensuring policy compliance is often overlooked, leading to potential security vulnerabilities and compliance risks.

Pre-Deployment Policy Compliance

In digital landscape user experience is paramount and organisations must ensure their applications and systems are functioning optimally at all times. Traditional testing and monitoring practices have long been relied upon to achieve this goal.

Sufficiently Advanced Monitoring is Indistinguishable from Testing

The promise of differential privacy is compelling. It offers a rigorous, provable guarantee of individual privacy, even in the face of arbitrary background knowledge. Rather than relying on anonymization techniques that can often be defeated, differential privacy works by injecting carefully calibrated noise into computations.

Managing Differential Privacy in Large Scale Systems

Differential Privacy (DP) is a mathematical framework that protects individual privacy in data analysis while allowing useful insights to be extracted. It works by adding carefully calibrated noise to data or query results, ensuring that including or excluding any single individual's data doesn't significantly change the analysis outcomes.

Differential Privacy: A Primer

In 1982, Andrew Yao proposed the Millionaire Problem which discusses how two millionaires can learn who is richest one without disclosing their actual wealth. They solve this problem by comparing their wealth using secure two party computation to ensure that they learn only the richest one and nothing else is revealed. The problem was later generalised for
<strong>
</strong>
secure multiparty computation by Goldreich et al in 1987.

Privacy Preserving Measurement

When a machine learning model is trained on a dataset, not all data points contribute equally to the model's performance. Some are more valuable and influential than others. Unfortunately value of data for training purposes is often nebulous and difficult to quantify. Applying data valuation to large language models (LLMs) like GPT-3, Claude 3, Llama 3.1 and their vast training datasets has faced significant scalability challenges to date.

What is worth of your data to LLM?

Anyone who has worked with advanced search systems knows the intricacy of boolean operators. AND, OR, NOT - these seemingly simple words become powerful tools in the hands of search specialists. Add in proximity operators like WITH and NEAR, throw in some parentheses for nesting, and you've got yourself a query that can pinpoint exactly what you're looking for... or so we thought. In practice, these queries can become unwieldy beasts.

The funnel of relevance

Multi-touch attribution is considered as holy grail in advertising industry. As advertisers are targeting users with multiple advertisements across different platforms and publishers, understanding how each of these touch points contributes to conversion is crucial—but this understanding has traditionally come at the cost of user privacy.

Privacy-Preserving Multi-Touch Attribution at TikTok

In last post we covered, Privacy Preserving Measurement (PPM) and discussed how Distributed Aggregation Protocol (DAP)  works. Today, we'll explore how to implement a simplified version of the DAP using Python with Prio3 as our Verifiable Distributed Aggregation Function (VDAF). This implementation will support multiple clients, demonstrating how DAP can aggregate data from multiple sources while maintaining privacy.

Distributed Aggregation Protocol (DAP) Primer

We've all experienced the frustration of our smartphones crashing or freezing at the worst possible moment. So how is it that NASA can land a rover on Mars, millions of miles away, with software that works flawlessly? The answer lies in a combination of factors that set space-grade software apart from your average app.

Engineering excellence is how NASA flawlessly landed Curiosity on Mars

Recently I have been reading Art of Leadership, The: Small Things, Done Well by Michael Lopp. This book is an excellent read and it covers small things that you can do to build trust and to become an authentic and true leader to your team at different stages of your leadership journey.

Engineering excellence: the art of doing small things really well

A mathematical guarantee is a formal, provable assurance about the behavior, performance, or properties of a system, algorithm, or process, derived from rigorous mathematical analysis or proof. The guarantee is based on mathematical logic and can be demonstrated through mathematical reasoning. It provides a definitive statement about what can be expected under specified conditions.

Mathematical Guarantee

Personal data processing forms the backbone of many big tech service providers. Tech giants like Netflix, Meta, and Amazon employ intricate networks of microservices to automatically process user data, creating complex data flows that span multiple layers of computation.

Modelling data flows as graphs to apply user privacy constraint

For big data and complex data processing, data pipelines have emerged as
a popular solution for managing and manipulating data. They provide a
systematic approach to extract, transform, and load (ETL) data from
various sources, enabling organizations to derive valuable insights.
However, as with any technology trend, data pipelines have not been
immune to misuse and overuse. In this article, we explore the idea that
data pipelines have become the new anti-patterns, as they are often
misapplied and misunderstood. I have written my thoughts about this
topic [a few years
back](https://cdn.abhishek-tiwari.com/friends-dont-let-friends-build-data-pipelines/)
but not a lot changed since then.

## The DAG Model and the Misconception

Data pipelines are commonly implemented as Directed Acyclic Graphs
(DAGs), where data flows through a series of processing steps, with each
step represented as a node and the dependencies between steps
represented as edges. The DAG model offers advantages such as
parallelism, fault tolerance, and scalability, making it suitable for
many data processing scenarios. However, it is crucial to recognize that
a DAG is not synonymous with a workflow, despite sharing some
similarities.

Data pipelines, when visualized as DAGs, can create an illusion of
workflow-like behavior due to the sequential flow of data. This illusion
often leads to confusion, as users may mistakenly assume that a DAG can
capture the complexities and nuances of a complete workflow. In reality,
a DAG lacks the necessary workflow context, and relying solely on it can
result in incomplete solutions and missed opportunities.

For instance, [Apache Airflow](https://airflow.apache.org/) uses DAGs to
describe finite-running batch processing workloads. Airflow provides
rich scheduling and execution semantics enabling data engineers to
easily define complex pipelines, running at regular intervals. Firstly,
Airflow does not support infinitely-running (days, weeks, or even
longer) asynchornus processes. Secondly, it processes data in batch
context and not at individual context. So if you are processing customer
orders using Airflow, context is batch of customer orders and not
individual customer order. Thirdly, Airflow follows a task-based
approach and primarily concerned with executing individual tasks.
Lastly, Airflow is not suited for complex state management and
coordination requirements.

### Task-based vs. State-based

While task-based DAGs can emulate certain aspects of a workflow, they
lack the comprehensive state management offered by state-based workflow
orchestration systems. Primary emphasis a DAG is executing tasks in a
specific order and handling dependencies. A state-based workflow
encompasses the entire process, including the logic, decision-making,
and human interaction. , the focus is on the states that the workflow
goes through and the transitions between those states. State-based
workflows are well-suited for modeling long-running processes with
complex state management and coordination requirements.

## The Danger of Overusing Data Pipelines

The danger lies in perceiving every problem as a nail and using data
pipelines as the hammer. While data pipelines excel at handling data
transformations and aggregations, they may not be the most suitable
solution for all scenarios. Overusing data pipelines without considering
the broader workflow context can lead to several issues.

Data pipelines are designed to operate within a fixed structure defined
by the DAG. This rigidity can limit adaptability and make it challenging
to incorporate dynamic or ad-hoc decision-making processes into the
workflow.

When data pipelines are misused as complete workflow solutions, they
tend to become overly complex. Trying to represent the entire workflow
as a single DAG can result in convoluted pipelines that are difficult to
understand, debug, and maintain.

As data pipelines grow in size and complexity, maintaining them becomes
increasingly cumbersome. Changes or additions to the workflow often
require modifications to the underlying DAG, leading to a tedious and
error-prone process.

While data pipelines are known for their scalability, overreliance on
them can lead to scalability issues at a workflow level. The absence of
a holistic view of the workflow can result in bottlenecks or inefficient
resource allocation.

Orchestrating complex workflows involves more than just coordinating
data processing steps. It requires managing dependencies, handling
failures, handling user interactions, and handling exceptions, which are
not inherent capabilities of a simple DAG-based data pipeline.

## Use The Right Tool for the Job

To address the limitations of data pipelines, businesses should adopt
dedicated workflow orchestration tools or frameworks that provide
features beyond the scope of data pipelines. State-based workflow
orchestration systems, like Cadence, Amazon Simple Workflow Service
(*Amazon SWF*), Temporal, or AWS Step Functions, offer robust workflow
modeling, execution, and monitoring capabilities. Such tools enable the
modeling and execution of complex workflows, offering capabilities like
conditional branching, event-driven triggers, user interactions, and
exception handling.

## Every Company has a Workflow Problem

In today\'s digital age, companies across various industries are
recognizing the importance of effectively managing their business
processes. From transportation and logistics to e-commerce and food
delivery, the core operations of many successful companies can be viewed
as workflow problems. Workflow platforms have emerged as a crucial
component for these companies, enabling them to orchestrate complex
tasks, automate processes, and ensure efficient collaboration.

Let\'s take Uber as an example. Uber, the ride-hailing giant, has
disrupted the transportation industry by leveraging a robust workflow
platform at the heart of its operations. From connecting drivers and
riders to managing payments and handling customer support, Uber\'s
entire business model can be conceptualized as a well-designed workflow.

\(1\) Rider Request and Driver Assignment: When a rider requests a ride
through the Uber app, a series of tasks are initiated. These tasks
involve finding available drivers in the vicinity, calculating the
estimated time of arrival, and selecting the most suitable driver based
on various factors such as distance, rating, and availability. This
process involves complex algorithms and real-time data processing, all
orchestrated through a workflow platform.

\(2\) Ride Execution: Once a driver is assigned, the workflow
transitions into the ride execution phase. This includes tasks such as
navigation guidance, tracking the ride\'s progress, and providing
real-time updates to both the rider and the driver. The workflow
platform ensures seamless communication and coordination between all
parties involved, ensuring a smooth and efficient ride experience.

\(3\) Payment and Rating After the ride is completed, the workflow
transitions to the payment and rating phase. This involves tasks such as
calculating the fare based on distance and time, processing payment
transactions securely, and prompting the rider and driver to rate each
other. The workflow platform handles these tasks, ensuring accurate
billing, secure payments, and valuable feedback for continuous
improvement.

\(4\) Customer Support: In case of any issues or queries, Uber relies on
its workflow platform to manage customer support. This involves tasks
such as receiving support tickets, routing them to the appropriate
teams, and tracking the progress of issue resolution. The workflow
platform ensures timely and efficient customer support, maintaining high
service standards.

By modeling its core business as a workflow problem, Uber has been able
to optimize its operations, enhance user experience, and scale its
services globally. The workflow platform acts as the backbone of Uber\'s
business, enabling seamless coordination, automation, and monitoring of
tasks throughout the entire lifecycle of a ride.

While Uber is a prime example of a workflow-driven business model, the
concept can be extended to various other industries. For instance:

\(1\) E-commerce companies like HelloFresh heavily rely on workflow
platforms to manage order processing, inventory management, fulfillment,
and delivery logistics. Workflows ensure smooth coordination between
various stakeholders, from order placement to final delivery, while
maintaining high efficiency and customer satisfaction.

\(2\) Food Delivery: Services like UberEats, DoorDash, and Grubhub
utilize workflow platforms to manage the entire food delivery process.
From receiving orders, assigning delivery drivers, tracking deliveries,
and handling payments, workflows ensure the seamless execution of food
delivery operations.

\(3\) Logistics and Supply Chain: Companies involved in logistics and
supply chain management, such as DHL and FedEx, employ workflow
platforms to handle tasks such as order tracking, warehouse management,
shipment routing, and delivery scheduling. Workflows streamline these
complex operations, ensuring accurate and timely deliveries.

## Conclusion

While data pipelines have undoubtedly revolutionized data processing and
analysis, it is important to recognize their limitations and avoid
treating them as one-size-fits-all solutions. Mistaking data pipelines
for complete workflows can lead to the emergence of anti-patterns,
resulting in reduced flexibility, increased complexity, and maintenance
challenges. To build effective workflows, organizations should embrace
state-based workflow orchestration tools that go beyond the capabilities
of data pipelines.
