News | 29.10.2024
Integration of Shopfloor Data
Large chemical manufacturer integrates production data with Azure Data Lake Storage Gen2 (ADLS)
Simplified data pipeline: more efficiency and secure data management for industrial environments
In a proof of concept (PoC), a major German company in the chemical industry tested Cybus Connectware as a solution for data integration in production. The defined target was to record production and laboratory data at a high frequency while taking data quality into account. The software also needed to be able to seamlessly synchronize unstructured data, such as images and other files, with Azure Data Lake. The PoC also served to demonstrate the scalability of the solution, in particular how to expand from the integration of a single endpoint to many.
What is Azure Data Lake Storage Gen2?
Azure Data Lake Storage Gen2 is Microsoft’s Azure scalable and secure data lake solution, designed to handle massive amounts of data in a cost-effective manner. Built on Azure Blob storage, it provides hierarchical namespace support and is optimized for big data analytics, making it ideal for organizations looking to store, process, and analyze large volumes of structured and unstructured data.
Seamless data transfer to the cloud with Cybus Connectware
Our customer’s Manufacturing Data Integration team relies on Azure Data Lake as the primary repository for all production and manufacturing data. This includes structured data from production and laboratory environments, such as OPC UA devices, as well as unstructured data such as images or test files. For BASF, the ability to collect and analyze data from different sources is an essential prerequisite for making informed decisions.
However, the challenge lies in the seamless transfer of this data from on-site devices to a cloud environment where it can be further analyzed and used. Traditional methods often involve complex and time-consuming processes that create a bottleneck in the data pipeline. This is where Connectware comes in, providing a reliable and efficient way to collect, manage, and transfer this data to the cloud.
Implementation of integration and data synchronization
To overcome this challenge, Connectware’s protocol connector was used, which can be seamlessly integrated into Azure Data Lake Storage Gen2. Connectware’s native HTTP request capabilities also enabled data to be uploaded directly from OPC UA devices in labs and workshops to the cloud-based data lake.
One of the key components that could be used in Azure was the blob storage artifact. Specifically to secure the data generated throughout the day in a single CSV file within the blob. To achieve this, we implemented the Blob Append feature, which allows new data to be continuously added to the existing file without creating multiple files. This approach ensures a lean and organized data storage, which is crucial for efficient data analysis.
In addition, we used Azure’s AZCopy tool in conjunction with a Connectware agent to securely synchronize other file types, such as images, to ensure that all relevant data is transferred to the cloud in a secure and efficient manner.
Data integration with Connectware: The benefits
Cybus Connectware stands out for its ability to seamlessly integrate into existing industrial infrastructures and offers a flexible, scalable solution for efficient data management. By connecting directly to Azure Data Lake Storage Gen2, Connectware enables reliable and secure transfer of production data to the cloud without manual intervention or complex customization. This combination offers companies decisive advantages that go far beyond conventional data solutions.
- Seamless integration:
By using Connectware, data from OPC UA devices is automatically uploaded to Azure Data Lake Storage Gen2 without manual intervention, reducing the risk of data loss or delays. - Efficiency:
Using Blob Append ensures that all daily data is contained in a single, easily accessible CSV file, simplifying data management and analysis. - Security:
The integration of the AZCopy tool and Connectware agent ensures files are transferred securely and the integrity and confidentiality of sensitive information are maintained. - Scalability:
Azure Data Lake Storage Gen2 is designed to scale with your needs, making it suitable for both small businesses and large enterprises handling large amounts of data. - Cost efficiency:
By leveraging Azure’s cloud infrastructure, organizations can reduce the costs associated with storing and processing data on-premises while benefiting from the flexibility and scalability of the cloud.
Together with Azure Data Lake Storage Gen2, Cybus Connectware offers a solution that simplifies the data pipeline while increasing the efficiency and security of data management in industrial environments. Take a closer look at Connectware here or contact us directly to find out more about how we can fulfill your individual requirements.
