Onit Documentation

Reporting Data Flow

by Michael Nadeau Updated on

Everyday, the App Builder and BillingPoint data is refreshed into warehouse, which Tableau then consumes to refresh the Tableau data source extracts. Here is the high-level overview diagram that details how the data flows into the warehouse. The ETL process is developed using Python scripts that do the following:

  • Extract data from MongoDB (Atoms in Mongo) or Postgres (Atoms in Postgres)
  • Transforms the data into warehouse as rows and columns
  • Loads the data into warehouse
  • Recreates the required views
  • Grants permissions for the tableau user
  • Triggers the Tableau data source refresh for the corporation.   Data sources must be owned by domain_publisher and not tagged as do not refresh.
  •  

Data Flow:
 

  1. For corporations with atoms in MongoDB, the nightly refresh Jenkins job reads the data from the MongoDB read replica host. The nightly refresh job reads the atoms from the atoms table in the app builder schema for corporations with atoms in Postgres.
     
  2. The nightly refresh job transforms the data using a Python script and is scheduled on Jenkins.
     
  3. Once the data is transformed, the nightly refresh job starts writing data into warehouse tables in batches. If metadata changes, it will drop and recreate the tables to reflect the updated columns.
     
  4. At the end of the nightly refresh job, it triggers the Tableau data source refresh so that Tableau can pull the data from the warehouse into Tableau data source extracts.
     
  5. The data from the Tableau data source extracts is then consumed by the workbooks in Tableau.
Previous Article Reporting Database Objects
Next Article Advanced Reporting End User Guide

© 2024 Onit, Inc.

docs.onit.com contains proprietary and confidential information owned by Onit, Inc. that is subject to copyright. Onit presents it exclusively to you for your sole use in conjunction with using Onit products. No portion of the materials contained herein may be used for any other purpose. No portion of the materials contained herein may be shared with third parties or reproduced in any form.