Control analytics at scale on large amount of IoT data generated every day
Every mining machine carries more than 10 sensors
Tasks executed every hour
Integration with various Big Data tools
A Japanese multinational corporation manufacturing construction, mining, industrial and military equipment works with Activeeon to retrieve data from sensors situated on machines in use, in order to perform predictive analytics on their state. These analytics help ensure the health and performance of mining machines and prevent issues, by applying real-time control and optimizations. To do so, the company needed to perform streaming over hot and cold storage and batch analytics, which represents 1200 tasks per hour.
Some of the requirements were:
Workflows & Scheduling solution from Activeeon is used to orchestrate and schedule analytics workflows at scale over the cloud. Activeeon provided the customer with a unique workflow solution to handle job submission based on dynamic inputs. Data analytics orchestrated by Workflows & Scheduling can be executed and paused on distinct machines or groups of machines, triggered on events and/or on schedule.
The solution provided is currently used in dev and prod environments. It handles node failures and recovery, as well as fault tolerance for every application and service managed by ProActive Workflows & Scheduling. The solution includes a powerful interface and features data search by status, groups or distinct machines, or by execution time. A REST interface is provided for third-party tools integration.
With ProActive Workflows & Scheduling, the customer benefits from distributed and parallelized data processing, allowing to retrieve reliable results to make business decisions. Activeeon also helped the customer migrate from AWS to Azure cloud infrastructure and benefit from its scalability capacities.
As shown above, the data is ingested and stored in different storage types. ProActive from Activeeon is consuming this data based on parameters collected from the master data storage to perform analytics. The results are then published in multiple places to be consumed by operators.
Activeeon has achieved this integration thanks to its open architecture. ProActive scheduler has been integrated with multiple big data solutions such as:
Moreover, the various languages supported by the solution enabled the data analytics team to select their favorite language such as R, Python or Matlab.