A framework for exploration and cleaning of environmental data : Tehran air quality data experience

Shamsipour, Mansour and Farzadfar, Farshad and Gohari, Kimiya and Parsaeian, Mahboubeh and Amini, Hassan and Rabiei, Katayoun and Hassanvand, Mohammad Sadegh and Navidi, Iman and Fotouhi, Akbar and Naddafi, Kazem and Sarrafzadegan, Nizal and Mansouri, Anita and Mesdaghinia, Alireza and Larijani, Bagher and Yunesian, Masud. (2014) A framework for exploration and cleaning of environmental data : Tehran air quality data experience. Archives of Iranian medicine, Vol. 17, H. 12. pp. 821-829.

PDF - Published Version

Official URL: http://edoc.unibas.ch/dok/A6328998

Downloads: Statistics Overview


Management and cleaning of large environmental monitored data sets is a specific challenge. In this article, the authors present a novel framework for exploring and cleaning large datasets. As a case study, we applied the method on air quality data of Tehran, Iran from 1996 to 2013. ; The framework consists of data acquisition [here, data of particulate matter with aerodynamic diameter ≤10 µm (PM10)], development of databases, initial descriptive analyses, removing inconsistent data with plausibility range, and detection of missing pattern. Additionally, we developed a novel tool entitled spatiotemporal screening tool (SST), which considers both spatial and temporal nature of data in process of outlier detection. We also evaluated the effect of dust storm in outlier detection phase.; The raw mean concentration of PM10 before implementation of algorithms was 88.96 µg/m3 for 1996-2013 in Tehran. After implementing the algorithms, in total, 5.7% of data points were recognized as unacceptable outliers, from which 69% data points were detected by SST and 1% data points were detected via dust storm algorithm. In addition, 29% of unacceptable outlier values were not in the PR.  The mean concentration of PM10 after implementation of algorithms was 88.41 µg/m3. However, the standard deviation was significantly decreased from 90.86 µg/m3 to 61.64 µg/m3 after implementation of the algorithms. There was no distinguishable significant pattern according to hour, day, month, and year in missing data.; We developed a novel framework for cleaning of large environmental monitored data, which can identify hidden patterns. We also presented a complete picture of PM10 from 1996 to 2013 in Tehran. Finally, we propose implementation of our framework on large spatiotemporal databases, especially in developing countries.
Faculties and Departments:09 Associated Institutions > Swiss Tropical and Public Health Institute (Swiss TPH)
09 Associated Institutions > Swiss Tropical and Public Health Institute (Swiss TPH) > Department of Epidemiology and Public Health (EPH) > Environmental Exposures and Health Systems Research > Physical Hazards and Health (Röösli)
UniBasel Contributors:Amini, Hassan
Item Type:Article, refereed
Article Subtype:Research Article
Publisher:Acad. of Medical Sciences of I. R. Iran]
Note:Publication type according to Uni Basel Research Database: Journal article
Related URLs:
Identification Number:
edoc DOI:
Last Modified:31 Dec 2015 10:56
Deposited On:06 Feb 2015 09:58

Repository Staff Only: item control page