ALEXANDRE PAIVASystems · Data · Technology · Engineering
Home
Data & AnalyticsCASE 01

ObraIndex

In development

Data Engineering applied to the historical analysis of construction material prices.

Official data sourceOpen Data API — Compras.gov.br
Purpose

Turning government procurement records into a reliable view of construction price behavior.

Core deliverable

A traceable foundation for comparing price trends, regional differences and volatility.

01 / Architecture

An architecture to preserve, process and serve data.

The flow separates responsibilities to keep sources traceable and deliver structures suitable for analysis.

ObraIndex architecture diagram: Compras.gov.br, Airflow, Bronze, Silver and Gold layers in MinIO, PostgreSQL and Metabase Enlarge diagram

ClassificationData Lake + Medallion Architecture + Analytical Serving Layer

02 / From API to decision

How the project was built.

Each technology has a clear responsibility in the flow, from operational data to business insight.

  1. 01

    Defining the problem

    The project began with the need to track trends, regional differences and volatility in construction material prices.

  2. 02

    Traceable ingestion

    Python queries the Compras.gov.br API with pagination, retries and backoff, adding technical metadata to Bronze records.

  3. 03

    Layered storage

    MinIO separates the Bronze, Silver and Gold stages to preserve traceability and enable reprocessing.

  4. 04

    Processing and quality

    The Silver layer standardizes types, deduplicates records, creates analytical attributes and flags invalid data and outliers.

  5. 05

    Analytical modeling

    The Gold layer was structured to organize monthly data marts by material and state, preparing historical and regional comparisons.

  6. 06

    Planned publishing

    PostgreSQL and Metabase form the planned layer for queries, indicators and accessible exploration of processed data.

03 / Technical decisions

Details that underpin the implementation.

Engineering decisions that make collection auditable, quality verifiable and the MVP scope transparent.

Resilient collection

Ingestion was designed to handle volume, pagination and API instability.

  • Retries for HTTP failures
  • Exponential backoff
  • Page-by-page processing

Batch traceability

Each record receives technical context to support auditing and source investigation.

  • Run ID and timestamp
  • CATMAT and source page
  • Record SHA-256 hash

Quality while preserving evidence

Rules flag problems without deleting the original data or hiding relevant exceptions.

  • Typing and deduplication
  • Business key
  • Invalid-record and outlier flags

Current MVP boundaries

The published DAG runs validations and the Bronze layer; other integrations remain under development.

  • Silver and Gold implemented in code
  • Persistent watermark pending
  • Analytical serving planned
04 / Results and insights

The research will take shape here.

ObraIndex findings will be published here as the analysis advances, with evidence, context and conclusions derived from the data.

Research in progress Coming soon
01

Price indicators

Historical series, changes and comparisons for the materials analyzed.

Content in preparation
02

Regional patterns

Differences between states and regions identified from processed data.

Content in preparation
03

Research insights

Findings, conclusions and relevant signals to support new decisions.

Content in preparation