mapreduce, hadoop, etl, elt, infrastructure, cloud, workflow, meta-scheduler and cost optimisation
<img src=“/resources/whitepaper-big-data-orchestration/images/meta-scheduling-big-data-apps.png” alt=“big data job scheduling over secured environments” class=‘img-fluid ‘
Storing, processing and extracting value from the data are becoming IT department’s’ main focus. The huge amount of data, or as it is called Big Data, have four properties: Volume, Variety, Value and Velocity. Systems such as Hadoop, Spark, Storm, etc. are de facto the main building blocks for Big Data architectures (e.g. data lakes), but are fulfilling only part of the requirements.
Moreover, in addition to this mix of features which represents a challenge for businesses, new opportunities will add even more complexity. Companies are now looking at integrating even more sources of data, at breaking silos (variety is increasing with structured and unstructured data), and at real-time and actionable data. All those are becoming key for decision makers.