Backlog

Iteration +0

• Folksonomy – define categories

  • File formats

  • Filesystems (local vs. remote vs. cloud storage)

  • Databases

  • Data warehouses

  • Streams

  • Services (System vs. Cloud/SaaS?)

  • Open table formats

• Misc

  • Cloud storage

  • Pipeline frameworks

  • Metrics, telemetry, logs

Iteration +1

• Documentation: Rework section “supported-sources”

• Documentation: Advertise OCI image

• Code layout: Refactor omniload.src into omniload.x.{category}

• Tutorial “CSV to Elasticsearch” https://github.com/bruin-data/ingestr/issues/487

• Tutorial “Parquet to CrateDB”

• Tutorial “HDFS to CrateDB”

• Check other people’s dlt component collections https://github.com/thealtoclef/dlt-providers https://github.com/thealtoclef/dlt/commits/devel

Iteration +2

• Add Helm chart, and Improve Cloud deployment docs & resources https://images.minimus.io/

• Emphasize and improve file-based formats (CSV, JSON, Parquet, BSON, etc.)

• Refurbish example-uris subcommand

• Support 3-component table names (catalog.schema.table). Make catalog/database/project-qualified names parse consistently. https://github.com/bruin-data/ingestr/pull/856

• Make Delta Lake and Iceberg support merge, delete+insert, truncate+insert, and SCD2 strategies. https://github.com/bruin-data/ingestr/pull/912

• StarRocks https://github.com/bruin-data/ingestr/blob/main/docs/supported-sources/starrocks.md?plain=1

Iteration +3

• Add scheduling unit

• Add notification unit

• Validate support for CrateDB https://github.com/bruin-data/ingestr/pull/284

• Support streaming sources https://github.com/bruin-data/ingestr/issues/282