Data filtering framework for preserving meaningful data records from streams of unstructured weather data

• Main Author: Al-Zyadat, Wa'el Jum'ah
• Format: Thesis
• Language: English
• Published: 2014
• Online Access: http://psasir.upm.edu.my/id/eprint/60463/1/FSKTM%202014%2017IR.pdf; http://psasir.upm.edu.my/id/eprint/60463/

The aim of this research is to design and implement a data filtering framework for preserving meaningful data records from streams of unstructured weather data based on data collection, data pre-process, data filtering, classification, and visualization. The data collection involved monitoring data and data structuring; data pre-process, error checking, error validation, and correction of error; data filtering involved filtering concept, sequential process, and particles filtering while classification data involved the proposed 5M method. The environmental data creates challenges in the context of storage and data processing, which producing large volume of data collection including a massive 30% of unwanted data. It affects the correctness of data in term of usage The rationale and importance of this research comes in forms of creating step by step operations of data filtering method using local weather data that combines data preprocessing and data filtering steps to enhance the accuracy of data classifications. This research successfully reformatted data collected from sensor-boards which are unstructured, with features that structure raw data to a standardized data format. The analysis of the proposed framework employed three measurements approaches for the validation purposes. First, data pre-process component measurement using correctness and precision measures; filtering components are measured by data (indexing and item) using the context of records to discover the duplicates between two different datasets, and finally the 5M classification is measured by the percentage of total and meaningful_ sensitivity to indicated classification of relevant items which explore the meaningful data of measurements based on percentages of classifier data from relevant data. Result shows the pre-processed data collected is reduced by 69.23 % with similar accuracy as compared to the raw data. Classification hit is 84.6 % accuracy to each clustered data. This research has been able to classify streaming weather data. The execution of data filtering framework for preserving meaningful data records from streams of unstructured weather data produced highly accurate classification clusters.