News | 29.10.2024
Integration of Shopfloor Data
International chemical group integrates production data with Azure Data Lake Storage Gen2 (ADLS)
More efficiency and secure data management for industrial environments thanks to simplified data pipeline
For a PoC, the large chemical company collected production and laboratory data at high frequency to use it for analysis and BI via the cloud. The PoC successfully demonstrated how Cybus Connectware can seamlessly synchronize unstructured data such as images and files with Azure Data Lake. Additionally, scalability was proven by extending the integration from one to multiple endpoints.
What is Azure Data Lake Storage Gen2?
Azure Data Lake Storage Gen2 is Microsoft’s Azure scalable and secure data lake solution, designed to handle massive amounts of data in a cost-effective manner. Built on Azure Blob storage, it provides hierarchical namespace support and is optimized for big data analytics, making it ideal for organizations looking to store, process, and analyze large volumes of structured and unstructured data.
Seamless data transfer to the cloud with Cybus Connectware
The Group’s Manufacturing Data Integration team relies on Azure Data Lake as the primary repository for all production and manufacturing data. This includes structured data from production and laboratory environments, such as OPC UA devices, as well as unstructured data such as images or test files. The ability to collect and analyze data from different sources is an essential prerequisite for making informed decisions.
However, the challenge lies in the seamless transfer of this data from on-site devices to a cloud environment where it can be further analyzed and used. Traditional methods often involve complex and time-consuming processes that create a bottleneck in the data pipeline. This is where Connectware comes in, providing a reliable and efficient way to collect, manage, and transfer this data to the cloud.
Integration and data synchronization via the protocol connector
To overcome this challenge, Connectware’s protocol connector was used, which can be seamlessly integrated into Azure Data Lake Storage Gen2. Connectware’s native HTTP request capabilities also enabled data to be uploaded directly from OPC UA devices in labs and workshops to the cloud-based data lake.
One of the key components that could be used in Azure was the blob storage artifact. Specifically to secure the data generated throughout the day in a single CSV file within the blob. To achieve this, we implemented the Blob Append feature, which allows new data to be continuously added to the existing file without creating multiple files. This approach ensures a lean and organized data storage, which is crucial for efficient data analysis.
In addition, Azure’s AZCopy tool was used in conjunction with a Connectware agent to securely synchronize other file types, such as images, and ensure that all relevant data was transferred to the cloud securely and efficiently.
Secure and efficient data integration with Connectware
Cybus Connectware is characterized by its ability to integrate seamlessly into existing industrial infrastructures and offers a flexible, scalable solution for efficient data management. By connecting directly to Azure Data Lake Storage Gen2, Connectware enables reliable and secure transfer of production data to the cloud without manual intervention or complex customization. This combination offers companies decisive advantages that go far beyond conventional data solutions.
- Seamless integration:
By using Connectware, data from OPC UA devices is automatically uploaded to Azure Data Lake Storage Gen2 without manual intervention, reducing the risk of data loss or delays. - Efficiency:
Using Blob Append ensures that all data is contained in a single, easily accessible CSV file on a daily basis, simplifying data management and analysis. - Security:
The integration of the AZCopy tool and Connectware agent ensures files are transferred securely and the integrity and confidentiality of sensitive information are maintained. - Scalability:
Azure Data Lake Storage Gen2 is designed to scale with your needs, making it suitable for both small businesses and large enterprises handling large amounts of data. - Cost efficiency:
By leveraging Azure’s cloud infrastructure, organizations can reduce the costs associated with storing and processing data on-premises while benefiting from the flexibility and scalability of the cloud.
Together with Azure, Connectware creates an end-to-end data pipeline from production to the cloud, while increasing efficiency and security of data management in industrial environments. Take a closer look at Connectware here or contact us directly to find out more about how you can efficiently realize your initiatives and projects yourself with Connectware.
