Introducing Streambatch

Agricultural Metrics On Demand

My new company, Streambatch.io, is on a mission to empower agricultural analysts with on-demand earth metrics. If you are doing agricultural analysis of any sort, reliable ground truth is essential even though timely and accurate measurements can be difficult and expensive to acquire. The rapidly growing Earth Observation (EO) industry has been helpful for many in this regard because not only does it have the potential to generate instant, accurate ground truth cheaply, it also offers a path to new measurements that might well enable better, more effective analysis.

However, there are many more organizations for whom the EO route is not viable or cost effective. This has to do with the nature of what is on offer. Satellites offer petabytes of raw  spectrometric data; they do not tell you when and where you need to do something, let alone what that something is. They simply give you the raw material that, when processed, can help you figure it out.

Transforming raw spectrometric imagery into metrics – i.e. numbers on which you can do the analysis you need to do– is, for the most part, a solved problem. But that does not mean it is easy or cheap to do. In fact, it’s the opposite: it’s hard and expensive. It’s hard and expensive because it demands the deployment of skilled people on a tedious, repetitive and error prone problem.

Many organizations have talented analysts who could do this work if not for other, higher priority tasks.  Other organizations simply don’t have that particular technical skillset at all. This is why so many companies that could be using remote sensing are not.

A Solved Problem

This situation is not unique to agriculture. In almost any industry, data arrives raw and transforming it into an analysis-ready form is prerequisite for analytical success. There is one important difference however: in more “mature” data-driven industries, there are companies that make it their business to do the data chores so that they can offer practitioners analysis-ready data that is truly “plug-and-play”.

Take the finance industry for example. Because of companies like Bloomberg and many others, financial analysts don’t spend time on data processing. Macroeconomic data comes normalized and seasonally adjusted. Stock price data comes corrected for dividends and stock splits. Financial analysts have this luxury because of the many companies that do the difficult and tedious work to take raw data and make it analysis-ready.

For EO data to be plug-and-play in the way financial data is, there has to be organizations doing – to quote Aravind from Terrawatch – “the boring work” that enables downstream analysis. This is exactly what Streambatch does.

Streambatch

The idea of Streambatch is to enable consumers of remote sensing data to enjoy the same luxury as the finance industry. Just like a stock market analyst has every financial metric they desire available on demand, we want every agricultural analyst to have any and all Earth metrics at their fingertips.

Analysts should not need to know or care about satellites, sensors and raw images. Generating measurements like temperature, precipitation, moisture and vegetation health from satellite data is a solved problem and organizations should not have to reinvent that wheel again and again. Streambatch’s mission is to enable analysts to invest in applying Earth metrics as opposed to generating them.

Here are some of the key aspects of Streambatch:

The RUDS Principle

RUDS stands for “Reliable, Usable, Documented and Supported". RUDS was the foundation of my last data company Quandl, (today Nasdaq Data Link). Basically, if you are in the business of delivering data to customers, you better be zealous on these four principles:

1. Reliability: the data you bring customers had better be 100% correct, 100% of the time.
2. Usability: your consumption experience should be nothing short of delightful.
3. Documentation: your documentation had better be organized and well written (something that is chronically lacking industry wide).
4. Support: you need to be obsessed with your customer’s success.

We’re fanatical about these four principles at Streambatch.

API-First

Offering our customers a simple, powerful API follows directly from our commitment to usability. Our API is intuitive and can be learned in minutes. Need a daily time series of NDVI for 36 different farms at 10 meter resolution? That’s one API call. 

Abstraction backed by Transparency

Streambatch customers get analysis-ready data; they never have to think about satellites and sensors. Streambatch does all image processing, fusion and calculations “by the book”. However, customers are not expected to take our word on that. We publish every data source, formula and algorithm we use to generate the analysis-ready data we offer. Customers can validate our methodologies once and then leverage our data ever after.

ESA/NASA First

The satellites operated by NASA and ESA make accurate scientific measurements their highest priority. As a result, spectrometry from their sensors is exceptionally good for calculating agricultural metrics. Streambatch has a bias towards NASA and ESA data because in most cases it offers the best measurements for agriculture and it comes with no “platform risk”, i.e. the risk that a satellite owner can squeeze you once you're dependent on their data. (The combination of using open data and being completely transparent on methodology means that Streambatch customers take no such risk.)

In fact–to the financial benefit of private satellite operators– the measurement potential of the NASA and ESA satellite constellations has simply not been tapped to its full potential. This is because, for many agricultural applications, no one constellation can offer a complete solution. NASA’s MODIS constellation, for example, has high temporal resolution but low spatial resolution; ESA’s Sentinel-2 constellation has high spatial resolution but low temporal resolution. You either sacrifice something or pay extremely high fees for access to private satellite data.

It turns out there is a third choice: It’s not trivial but it’s entirely possible to fuse data from all the public satellites to achieve something that gives you high spatial resolution and high temporal resolution. For many applications this will yield a complete solution at costs well below what private satellite operators will charge to address the same use case.

NDVI and Beyond

This is exactly what Streambatch is doing today. Our first product, daily NDVI time series at 10m resolution for anywhere on Earth, is built by fusing outputs from multiple NASA and ESA satellite systems. It’s all accessible via our API and can be trialed today.

These are our first steps towards our vision of a world where agricultural analysts are empowered with on-demand access to any measurement they need, for any location on this planet – without ever having to process a single byte of sensor data.