Reliable analysis results depend on high-quality datasets. Engineers aggregating data from multiple sources must identify and correct discrepancies in these datasets before this downstream analysis can be performed. For instance, the column titles may vary from one source to another, time stamps may be in different formats, some data columns need to be inserted or deleted, data sets need to be sliced, merged or joined, etc.
Data manipulation tools such as Pandas, a widely adopted software library written for Python, eases the pain of data-wrangling and can ingest raw data from a variety of formats like csv, JSON, Parquet, HDF5, etc. These well-known capabilities of Pandas can be used within the intuitive drag and drop interface of nCodeDS, an analytical tool that enables ultra-fast engineering analysis on high volumes of sensor data. The new Data Frame Connector node provides tremendous value to analysts who can benefit from both pandas and the highly specialized engineering algorithms in nCodeDS for digital filtering, cycle counting, time at level, damage calculations and FFT algorithms.
This presentation will introduce how the link between the Python ecosystem and nCodeDS is strengthened through the Data Frame Connector node, enabling engineers to quickly turn data into a form suitable for large scale data reduction and processing.
Originally presented on October 20, 2021
