Building data civilizer pipelines with an advanced workflow engine

Essam Mansour; Dong Deng; Raul Castro Fernandez; Abdulhakim A. Qahtan; Wenbo Tao; Ziawasch Abedjan; Ahmed Elmagarmid; Ihab F. Ilyas; Samuel Madden; Mourad Ouzzani; Michael Stonebraker; Nan Tang

doi:10.1109/ICDE.2018.00184

Building data civilizer pipelines with an advanced workflow engine

Essam Mansour, Dong Deng, Raul Castro Fernandez, Abdulhakim A. Qahtan, Wenbo Tao, Ziawasch Abedjan, Ahmed Elmagarmid, Ihab F. Ilyas, Samuel Madden, Mourad Ouzzani, Michael Stonebraker, Nan Tang

Qatar Computing Research Institute

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

5 Citations (Scopus)

Abstract

In order for an enterprise to gain insight into its internal business and the changing outside environment, it is essential to provide the relevant data for in-depth analysis. Enterprise data is usually scattered across departments and geographic regions and is often inconsistent. Data scientists spend the majority of their time finding, preparing, integrating, and cleaning relevant data sets. Data Civilizer is an end-To-end data preparation system. In this paper, we present the complete system, focusing on our new workflow engine, a superior system for entity matching and consolidation, and new cleaning tools. Our workflow engine allows data scientists to author, execute and retrofit data preparation pipelines of different data discovery and cleaning services. Our end-To-end demo scenario is based on data from the MIT data warehouse and e-commerce data sets.

Original language	English
Title of host publication	Proceedings - IEEE 34th International Conference on Data Engineering, ICDE 2018
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1593-1596
Number of pages	4
ISBN (Electronic)	9781538655207
DOIs	https://doi.org/10.1109/ICDE.2018.00184
Publication status	Published - 24 Oct 2018
Event	34th IEEE International Conference on Data Engineering, ICDE 2018 - Paris, France Duration: 16 Apr 2018 → 19 Apr 2018

Publication series

Name	Proceedings - IEEE 34th International Conference on Data Engineering, ICDE 2018

Conference

Conference	34th IEEE International Conference on Data Engineering, ICDE 2018
Country/Territory	France
City	Paris
Period	16/04/18 → 19/04/18

Keywords

Data Cleaning
Data Discovery
Data Integration

Access to Document

10.1109/ICDE.2018.00184

Cite this

Mansour, E., Deng, D., Fernandez, R. C., Qahtan, A. A., Tao, W., Abedjan, Z., Elmagarmid, A., Ilyas, I. F., Madden, S., Ouzzani, M., Stonebraker, M., & Tang, N. (2018). Building data civilizer pipelines with an advanced workflow engine. In Proceedings - IEEE 34th International Conference on Data Engineering, ICDE 2018 (pp. 1593-1596). Article 8509405 (Proceedings - IEEE 34th International Conference on Data Engineering, ICDE 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICDE.2018.00184

@inproceedings{b16ee105c2584e66b51e8c0f6ebafdb9,

title = "Building data civilizer pipelines with an advanced workflow engine",

abstract = "In order for an enterprise to gain insight into its internal business and the changing outside environment, it is essential to provide the relevant data for in-depth analysis. Enterprise data is usually scattered across departments and geographic regions and is often inconsistent. Data scientists spend the majority of their time finding, preparing, integrating, and cleaning relevant data sets. Data Civilizer is an end-To-end data preparation system. In this paper, we present the complete system, focusing on our new workflow engine, a superior system for entity matching and consolidation, and new cleaning tools. Our workflow engine allows data scientists to author, execute and retrofit data preparation pipelines of different data discovery and cleaning services. Our end-To-end demo scenario is based on data from the MIT data warehouse and e-commerce data sets.",

keywords = "Data Cleaning, Data Discovery, Data Integration",

author = "Essam Mansour and Dong Deng and Fernandez, {Raul Castro} and Qahtan, {Abdulhakim A.} and Wenbo Tao and Ziawasch Abedjan and Ahmed Elmagarmid and Ilyas, {Ihab F.} and Samuel Madden and Mourad Ouzzani and Michael Stonebraker and Nan Tang",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 34th IEEE International Conference on Data Engineering, ICDE 2018 ; Conference date: 16-04-2018 Through 19-04-2018",

year = "2018",

month = oct,

day = "24",

doi = "10.1109/ICDE.2018.00184",

language = "English",

series = "Proceedings - IEEE 34th International Conference on Data Engineering, ICDE 2018",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1593--1596",

booktitle = "Proceedings - IEEE 34th International Conference on Data Engineering, ICDE 2018",

address = "United States",

}

Mansour, E, Deng, D, Fernandez, RC, Qahtan, AA, Tao, W, Abedjan, Z, Elmagarmid, A, Ilyas, IF, Madden, S, Ouzzani, M, Stonebraker, M & Tang, N 2018, Building data civilizer pipelines with an advanced workflow engine. in Proceedings - IEEE 34th International Conference on Data Engineering, ICDE 2018., 8509405, Proceedings - IEEE 34th International Conference on Data Engineering, ICDE 2018, Institute of Electrical and Electronics Engineers Inc., pp. 1593-1596, 34th IEEE International Conference on Data Engineering, ICDE 2018, Paris, France, 16/04/18. https://doi.org/10.1109/ICDE.2018.00184

Building data civilizer pipelines with an advanced workflow engine. / Mansour, Essam; Deng, Dong; Fernandez, Raul Castro et al.
Proceedings - IEEE 34th International Conference on Data Engineering, ICDE 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 1593-1596 8509405 (Proceedings - IEEE 34th International Conference on Data Engineering, ICDE 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Building data civilizer pipelines with an advanced workflow engine

AU - Mansour, Essam

AU - Deng, Dong

AU - Fernandez, Raul Castro

AU - Qahtan, Abdulhakim A.

AU - Tao, Wenbo

AU - Abedjan, Ziawasch

AU - Elmagarmid, Ahmed

AU - Ilyas, Ihab F.

AU - Madden, Samuel

AU - Ouzzani, Mourad

AU - Stonebraker, Michael

AU - Tang, Nan

PY - 2018/10/24

Y1 - 2018/10/24

N2 - In order for an enterprise to gain insight into its internal business and the changing outside environment, it is essential to provide the relevant data for in-depth analysis. Enterprise data is usually scattered across departments and geographic regions and is often inconsistent. Data scientists spend the majority of their time finding, preparing, integrating, and cleaning relevant data sets. Data Civilizer is an end-To-end data preparation system. In this paper, we present the complete system, focusing on our new workflow engine, a superior system for entity matching and consolidation, and new cleaning tools. Our workflow engine allows data scientists to author, execute and retrofit data preparation pipelines of different data discovery and cleaning services. Our end-To-end demo scenario is based on data from the MIT data warehouse and e-commerce data sets.

AB - In order for an enterprise to gain insight into its internal business and the changing outside environment, it is essential to provide the relevant data for in-depth analysis. Enterprise data is usually scattered across departments and geographic regions and is often inconsistent. Data scientists spend the majority of their time finding, preparing, integrating, and cleaning relevant data sets. Data Civilizer is an end-To-end data preparation system. In this paper, we present the complete system, focusing on our new workflow engine, a superior system for entity matching and consolidation, and new cleaning tools. Our workflow engine allows data scientists to author, execute and retrofit data preparation pipelines of different data discovery and cleaning services. Our end-To-end demo scenario is based on data from the MIT data warehouse and e-commerce data sets.

KW - Data Cleaning

KW - Data Discovery

KW - Data Integration

UR - http://www.scopus.com/inward/record.url?scp=85057102546&partnerID=8YFLogxK

U2 - 10.1109/ICDE.2018.00184

DO - 10.1109/ICDE.2018.00184

M3 - Conference contribution

AN - SCOPUS:85057102546

T3 - Proceedings - IEEE 34th International Conference on Data Engineering, ICDE 2018

SP - 1593

EP - 1596

BT - Proceedings - IEEE 34th International Conference on Data Engineering, ICDE 2018

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 34th IEEE International Conference on Data Engineering, ICDE 2018

Y2 - 16 April 2018 through 19 April 2018

ER -

Mansour E, Deng D, Fernandez RC, Qahtan AA, Tao W, Abedjan Z et al. Building data civilizer pipelines with an advanced workflow engine. In Proceedings - IEEE 34th International Conference on Data Engineering, ICDE 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 1593-1596. 8509405. (Proceedings - IEEE 34th International Conference on Data Engineering, ICDE 2018). doi: 10.1109/ICDE.2018.00184

Building data civilizer pipelines with an advanced workflow engine

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this