Smart City Gnosys
Smart city article details
| Title | A Novel Spatial Data Pipeline For Orchestrating Apache Nifi/Minifi |
|---|---|
| ID_Doc | 3535 |
| Authors | Carthen C.D.; Zaremehrjardi A.; Le V.; Cardillo C.; Strachan S.; Tavakkoli A.; Maketitle; Harris F.C., Jr.; Dascalu S.M. |
| Year | 2023 |
| Published | International Journal of Software Innovation, 12, 1 |
| DOI | http://dx.doi.org/10.4018/IJSI.333164 |
| Abstract | In many smart city projects, a common choice to capture spatial information is the inclusion of lidar data, but this decision will often invoke severe growing pains within the existing infrastructure. In this article, the authors introduce a data pipeline that orchestrates Apache NiFi (NiFi), Apache MiNiFi (MiNiFi), and several other tools as an automated solution to relay and archive lidar data captured by deployed edge devices. The lidar sensors utilized within this workflow are Velodyne Ultra Puck sensors that produce 6-7 GB packet capture (PCAP) files per hour. By both compressing the file after capturing it and compressing the file in real-time; it was discovered that GZIP and XZ both saved considerable file size being from 2-5 GB, 5 minutes in transmission time, and considerable CPU time. To evaluate the capabilities of the system design, the features of this data pipeline were compared against existing third-party services, Globus and RSync. © 2023 Taru Publications. All rights reserved. |
| Author Keywords | Big Data; Data Pipeline; Data Transfer; Edge Computing; IOT; Lidar; MiNiFi; NiFi; PCAP; Smart City; Spatial Data |
Similar Articles
| Id | Similarity | Authors | Title | Published |
|---|---|---|---|---|
40971  | 0.989 | Carthen C.; Zaremehrjardi A.; Le V.; Cardillo C.; Strachan S.; Tavakkoli A.; Harris F.C.; Dascalu S.M. | Orchestrating Apache Nifi/Minifi Within A Spatial Data Pipeline | Proceedings - 2023 IEEE/ACIS 21st International Conference on Software Engineering Research, Management and Applications, SERA 2023 (2023) |
| 3536 | 0.861 | Carthen, CD; Zaremehrjardi, A; Le, VD; Cardillo, CG; Strachan, S; Tavakkoli, A; Harris, FC Jr; Dascalu, SM | A Novel Spatial Data Pipeline For Streaming Smart City Data | INTERNATIONAL JOURNAL OF SOFTWARE INNOVATION, 12, 1 (2024) |
| 4864 | 0.861 | Carthen C.; Zaremehrjardi A.; Le V.; Cardillo C.; Strachan S.; Tavakkoli A.; Dascalu S.M.; Harris F.C. | A Spatial Data Pipeline For Streaming Smart City Data | 2024 IEEE/ACIS 22nd International Conference on Software Engineering Research, Management and Applications, SERA 2024 - Proceedings (2024) |
| 5094 | 0.854 | Kim S.-S.; Lee W.-R.; Go J.-H. | A Study On Utilization Of Spatial Information In Heterogeneous System Based On Apache Nifi | ICTC 2019 - 10th International Conference on ICT Convergence: ICT Convergence Leading the Autonomous Future (2019) |