Data Consumers and Data Producers
Why you need to change how you think about data and people
Today I want to talk about two new terms I’ve been thinking about… Data Consumer and Data Producer. The idea is simple, really. There are people or things that produce data, and there are those that consume data. I think this is a new way of looking at data processes and, especially, the human element in our software. I think this mental model helps when thinking about where data is going and where it comes from in our increasingly data-centric environment. I hope you find it useful as well!
The Technical Component
First let’s start with the technical component of data consumers and data producers. If we think about a modern application, we usually have some sort of architecture that consists of a database, some form of an API, and a frontend either web or mobile. This is an old and established method of building applications and is great. Some newer developments allow us to relocate our APIs to be more integrated with source code (NextJS) or to not need traditional CMS database but utilize a headless variant (Strapi). My argument is that all of these services, and in fact, most technical services, can be categorized as either a Data Producer or Data Consumer… OR BOTH!
A service can be BOTH a Data Consumer and a Data Producer, usually in that order.
Embedded services/systems can be hard to categorize sometimes but if they are distinct enough do still fit this model. So let’s look at a modern web app example…
As you can see, we do, in fact, have an API that both consumes data from the database and produces data for the frontend. This brings up our next two key points… directional flow and context.
Directional Flow
Directional Flow is an important concept here since we do not have bi-directional data flow in this mental model. The reason for this is to reduce cognitive load and its why we introduced the next topic which is Context.
But first let’s clarify this Directional Flow concept. What this means, in essence, is that data only flows downstream (or upstream, depending on your perspective/context). What this means for our battery charts is that the arrows will always point in one direction. Let’s look at another example:
In this example we have an automated analysis script that does some descriptive analysis on the data in our database. It consumes data from the database, and then produces it for the report to consume. The dynamic report then consumes the updated data. As we can see, we have a valid Directional Flow (DB -> Script -> Report). You will also notice that in almost every situation there is a “partner” Consumer to every Producer. In this case the Script partners with the DB Producer and the Report partners with the Script Producer. If you are missing a Consumer for your Producer, it is likely you are Producing data without a target for it…
All Producers should have Consumers!
You will also notice that almost always the “doubled” services, those that function as both consumers and producers, will “start” with a consumer. This is very common of services that translate or transform data.
But couldn’t you argue that in the first example the API also produces data for the Database (via put/post/delete methods)? YES. This is why we have our last concept… Context.
Context
Our final concept is Context and it addresses the issue of a never-ending cycle of consumers/producers. For example, couldn’t we add a (human) step to the previous example where now the Report produces data for a human to consume (read). Below I will discuss human elements, but let’s just use this as an example. In this case we would have the following chart:
WOAH! Our Report turned from a consumer to a consumer/producer. This is the affect of adding context. It isn’t a problem that our service has a new meaning in a new context, it is a problem that we didn’t establish the different context.
Thus, all mental models of producers/consumers are valid for a specific context, preferably as small and discrete as possible. So our first example context could be for report generation, and then we could have a second one for report consumption which consists of just the report and the human elements.
Alright, with that I hope you have a good understanding of producers/consumers. Now let’s get to the good part! Humans!
A commodity often missed in technical ventures is people, we are everywhere and in most cases, without us, the technical service is completely meaningless.
So how do we account for humans more? We use them as producers and consumers!
The Human Element
So how do people fit into all of this? Well… humans are constant producers and occasional consumers. Walking around, humans are constantly interacting with their environment, changing it, modifying it, and generally making things happen. The simplest example is footprints in the sand. But we also have examples like heart rate (a produced signal), breathing, and the list goes on. The “occasional” consumer bit refers to that humans are not always consuming things (although an argument could be made for that, we are focusing more on data-centric “things”). Companies like Meta (formerly Facebook) have tried to harness your attention to make you a constant consumer and we have seen the affects of that on the human psyche.
Ok, so humans produce a lot, and consume a lot, but less than they produce… how does this help me? Well… since humans are the center of all business ventures, from startup companies to research projects, you need to try and take advantage of either the producer or the consumer component of humans.
Most people opt for the latter and try to make things people enjoy (entertainment industry) and often capturing a little of the producer component (usually an email or phone number). You could write research papers or books for others to consume or opt for something monetizeable and modern such as YouTube videos or blogging. Either way, you are now producing an “artifact” that other humans can consume. This a great business model and the effectiveness usually depends on the content being produced and how relevant it is for the consumers.
Another option is to focus on the producer side of humans. In fact, public health researchers spend A LOT of time on this and study human patterns and even Medical Examiner/Coroner reports of death records to gather additional data that people produce(d), even after their death. Researchers consume this data (that humans originally produced) and then produce a paper or article that can then be consumed by more humans still. Thus is the cycle of the producer and consumer.
Overview and Recap
So where does your venture fall? Where can you act as a producer for others? Where do you need to act as a consumer to gather more data in order to be an effective producer? Do you need a survey, a website form, or an in-depth interview? What will you do with the data you gather? All of these questions can be answered by thinking about your producer and consumer roles throughout the process in various context.
Personal Opinions
I want to finish up with some personal thoughts on this. It may seem like I am putting new labels on things that have existed for a while, and maybe I am but this mental model has helped me to think more concretely about many problems I face both personally and professionally. I also have some opinions about how to use this model:
- You should always emphasize producing more than consuming and attempt to consume the minimum required for effective producing
- Effective producing can be defined by either quality OR quantity depending on the industry and intended consumer, but for most cases quality reigns
- Producing is easy, but producing for a paired consumer is hard
- This is the old adage of having a “target audience”
- CONSUME MORE!
- This tends to contradict the first item, but in our increasingly data-driven and data-saturated world, there is so much data for you to consume. I suggest trying to bring in many source that already exist and thus don’t need to be “harvested” from humans directly (i.e. raw data) but can be consumed by prepared datasets/APIs.
- Once you see what is out there of interest for your domain, you can attain a minimum set of the data for you to be an effective producer.
- Chain your processes!
- Once you become a producer, think about how you could re-consume your produced data and then re-produce it again in a new way. A great example of this is Twitch streamers who post YouTube videos of clips of their stream… one source producer (the streamer) resulting in two produced source (twitch stream and YouTube video) for different consumers
I hope you like this mental model, thank you for reading this far if you made it and consider following me!
-Nick
