Kawa Space: Achieving 5x Latency Improvement in Satellite Imagery Processing

Role: Principal Data Scientist Duration: Apr 2020 - Apr 2022

Executive Summary

At Kawa Space, I led the data products team to build a scalable geo-intelligence platform for fintech customers. The core challenge was processing heavy satellite imagery in real-time for applications like land boundary detection and property verification. Through architectural innovation and a novel data standardization algorithm, we achieved a 5x improvement in inference latency, which directly enabled us to land our first major contract worth $10K+.

The Problem: Latency Bottlenecks in Satellite Image Processing

Business Context

Fintech companies need real-time geo-intelligence for:

  • Land boundary verification for loan collateral
  • Property valuation through satellite imagery analysis
  • Agricultural land assessment for rural lending
  • Fraud detection in property transactions

Technical Challenges

1. Heavy Data Volumes

  • Satellite images ranged from 50MB to 500MB per tile
  • High-resolution multispectral imagery (RGB + NIR bands)
  • Customers required processing of 100+ images per request

2. Heterogeneous Data Sources

  • Multiple satellite providers (Sentinel-2, Landsat, Planet Labs)
  • Different resolutions: 10m, 30m, 3m per pixel
  • Varying band configurations and metadata formats
  • Inconsistent coordinate reference systems (CRS)

3. Real-Time Requirements

  • Customer SLA: < 30 seconds for boundary detection
  • Initial system: ~150 seconds average latency
  • Inference time dominated by data loading and preprocessing

4. Cloud Cost Constraints

  • GCP Cloud Storage egress costs were significant
  • Compute costs for preprocessing on every request
  • Need to balance performance with cost efficiency

The Solution: A Multi-Pronged Optimization Strategy

1. Data Standardization Algorithm

The breakthrough came from realizing that preprocessing was being repeated for every inference request. I designed a data standardization pipeline that normalized all satellite imagery at ingestion time.

Key Components:

a) Spatial Standardization

# Conceptual approach (proprietary implementation details omitted)

def standardize_spatial_resolution(image, target_resolution=10):
    """
    Resample all images to a common resolution using bilinear interpolation
    for continuous data (bands) and nearest neighbor for discrete data (labels)
    """
    # Reproject to common CRS (EPSG:4326)
    # Resample to target resolution
    # Clip to standard tile size (256x256 or 512x512)
    pass

def normalize_band_configuration(image, target_bands=['R', 'G', 'B', 'NIR']):
    """
    Ensure all images have the same band order and configuration
    """
    # Map source bands to target bands
    # Handle missing bands (interpolation or zero-fill)
    # Standardize band order
    pass

b) Radiometric Normalization

def apply_radiometric_correction(image):
    """
    Normalize pixel values to a common scale accounting for
    atmospheric conditions and sensor characteristics
    """
    # Apply Top-of-Atmosphere (TOA) reflectance conversion
    # Normalize to 0-1 range or standardize (mean=0, std=1)
    # Apply histogram matching for consistency
    pass

c) Metadata Standardization

{
  "tile_id": "unique_identifier",
  "acquisition_date": "2021-03-15T10:30:00Z",
  "satellite": "sentinel-2",
  "resolution_m": 10,
  "bands": ["R", "G", "B", "NIR"],
  "crs": "EPSG:4326",
  "bbox": [lat_min, lon_min, lat_max, lon_max],
  "cloud_cover_pct": 5.2,
  "preprocessing_version": "v2.1"
}

Impact: This allowed us to store “inference-ready” images that required zero preprocessing at inference time.

2. End-to-End GCP Pipeline Architecture

I designed and implemented a complete data pipeline on Google Cloud Platform:

[Satellite Data Sources]
         ↓
[Cloud Storage - Raw Bucket]
         ↓
[Cloud Function - Trigger on Upload]
         ↓
[Dataflow Pipeline - Standardization]
    ├── Spatial Resampling
    ├── Radiometric Correction
    ├── Band Normalization
    └── Metadata Extraction
         ↓
[Cloud Storage - Processed Bucket]
    ├── TFRecord format (for TensorFlow models)
    ├── Optimized for streaming reads
    └── Indexed by geohash for spatial queries
         ↓
[AI Platform - Model Serving]
    └── Land Boundary Detection Model
         ↓
[API Gateway - Customer Endpoint]

Key Design Decisions:

a) Storage Format: TFRecord

  • Chose TFRecord over PNG/TIFF for 40% faster loading
  • Serialized preprocessed tensors directly
  • Enabled efficient batching for inference

b) Geospatial Indexing

  • Implemented geohash-based indexing
  • Enabled fast spatial queries: “Find all tiles within bounding box”
  • Reduced unnecessary data loading

c) Caching Layer

  • Added Cloud Memorystore (Redis) for frequently accessed tiles
  • Cache hit rate: ~60% for repeat customers
  • Reduced latency by additional 30% for cached requests

3. Land Boundary Detection Model

Built a semantic segmentation model using TensorFlow/Keras:

Architecture:

  • Base: Modified U-Net with ResNet34 encoder
  • Input: 4-channel (RGB + NIR), 512×512 pixels
  • Output: Binary mask (boundary vs. non-boundary)

Training Details:

  • Dataset: 10,000 manually annotated satellite images
  • Augmentation: Rotation, flip, brightness, contrast, Gaussian noise
  • Loss: Combined Binary Cross-Entropy + Dice Loss
  • Optimizer: Adam with learning rate scheduling
  • Validation IoU: 0.87

Inference Optimization:

  • Converted model to TensorFlow Lite for 2x speedup
  • Quantization (float16) reduced model size by 50% with <1% accuracy loss
  • Batch inference: Process 8 tiles simultaneously

4. Performance Benchmarking & Optimization

Latency Breakdown (Before vs. After):

Stage Before After Improvement
Data Loading 45s 3s 15x
Preprocessing 80s 0s ∞
Model Inference 20s 12s 1.67x
Post-processing 5s 3s 1.67x
Total 150s 18s 8.3x

Note: Achieved 5x improvement on average across diverse query patterns. The 8.3x represents best-case scenario.

Cost Optimization:

  • Reduced compute costs by 60% (preprocessing done once vs. per request)
  • Storage costs increased by 40% (storing processed data)
  • Net cost reduction: ~35%

The Impact: From Technical Win to Business Win

Technical Achievements

  1. 5x Latency Reduction: Average latency from 150s → 30s
  2. Scalability: System now handles 1000+ concurrent requests
  3. Accuracy: Land boundary detection IoU of 0.87
  4. Reliability: 99.5% uptime over 12 months

Business Impact

  1. First Major Contract: The performance improvements directly enabled us to win a $10K+ contract with a major fintech client
  2. Product-Market Fit: Latency reduction made the product viable for real-time use cases
  3. Customer Satisfaction: NPS improved from 6 to 8.5 after deployment
  4. Platform Foundation: Architecture became the foundation for additional geo-intelligence products

Learnings & Trade-offs

What Worked:

  • Preprocessing at ingestion time (do work once, not repeatedly)
  • Investing in data standardization paid massive dividends
  • GCP-native services (Cloud Functions, Dataflow) reduced operational overhead

What Was Challenging:

  • Balancing storage costs vs. compute costs required careful analysis
  • Handling edge cases (cloud cover, seasonal variations) in standardization
  • Migrating from ad-hoc processing to structured pipeline required customer coordination

What I’d Do Differently:

  • Implement standardization earlier in the product lifecycle
  • Add more comprehensive monitoring/alerting from day one
  • Consider multi-cloud strategy for redundancy (single-cloud risk)

Technical Deep Dive: The Standardization Algorithm

For those interested in implementation details, here’s the core workflow:

Step 1: Metadata Parsing & Validation

class SatelliteImageStandardizer:
    def __init__(self, target_resolution=10, target_crs='EPSG:4326'):
        self.target_resolution = target_resolution
        self.target_crs = target_crs

    def parse_metadata(self, image_path):
        # Read satellite-specific metadata
        # Validate required fields
        # Extract acquisition parameters
        pass

Step 2: Coordinate System Transformation

    def reproject_to_standard_crs(self, image, source_crs):
        # Use GDAL/rasterio for reprojection
        # Handle datum transformations
        # Preserve geospatial accuracy
        pass

Step 3: Resolution Harmonization

    def harmonize_resolution(self, image, source_resolution):
        if source_resolution > self.target_resolution:
            # Downsample using bilinear interpolation
            resampled = self.downsample(image)
        else:
            # Upsample using bicubic interpolation
            resampled = self.upsample(image)
        return resampled

Step 4: Radiometric Calibration

    def calibrate_radiometry(self, image, metadata):
        # Apply TOA reflectance conversion
        toa = self.to_toa_reflectance(image, metadata)

        # Atmospheric correction (optional, simplified DOS method)
        corrected = self.dark_object_subtraction(toa)

        # Normalize to [0, 1] range
        normalized = (corrected - corrected.min()) / (corrected.max() - corrected.min())

        return normalized

Step 5: Band Standardization & Serialization

    def standardize_and_save(self, image, output_path):
        # Ensure band order: [R, G, B, NIR]
        standardized = self.reorder_bands(image)

        # Convert to TFRecord
        tf_example = self.create_tf_example(standardized)

        # Write to Cloud Storage
        self.write_tfrecord(tf_example, output_path)

Conclusion

The Kawa Space project demonstrated that architectural innovation can be as impactful as algorithmic innovation. By rethinking when and how we process data, we achieved an 8x latency improvement that transformed a proof-of-concept into a production-ready product.

The key insight: Do expensive work once, at ingestion time, rather than repeatedly at inference time.

This principle applies beyond satellite imagery—to any domain where preprocessing dominates latency.

Technologies Used: Python, TensorFlow, Google Cloud Platform (Cloud Storage, Cloud Functions, Dataflow, AI Platform), GDAL, Rasterio, Redis

Code Samples: Due to proprietary constraints, full source code cannot be shared. The conceptual implementations above illustrate the approach.

← Back to Portfolio