Abstract
Any Big Data scenario eventually reaches scalability concerns for several factors, often storage or computing power related. Modern solutions have been proven to be effective in multiple domains and have automated many aspects of the Big Data pipeline. In this paper, we aim to present a solution for deploying event-based automated data processing tools for low code environments that aim to minimize the need for user input and can effectively handle common data processing jobs, as an alternative to distributed solutions which require language specific libraries and code. Our architecture uses a combination of a network exposed service with a cluster of “Data Workers” that handle data processing jobs effectively without requiring manual input from the user. This system proves to be effective at handling most data processing scenarios and allows for easy expandability by following simple patterns when declaring any additional jobs.
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
