Project At a Glance
Somexsoft implemented a BizTalk 2004 solution for a utilities leader, allowing them to retire two costly legacy applications while expanding functionality for an update to their product monitoring pipeline and Customer Management System (CMS). The solution incorporates a common schema format, supporting future scalability so additional systems can be configured to share data with CMS.
About Our Client
Our client is a multi-billion dollar master limited partnership with three business segments: transportation and storage of refined products, liquefied petroleum gases (LPGs) and petrochemicals (downstream segment); gathering, transportation, marketing and storage of crude oil, as well as the distribution of lubrication oils and specialty chemicals (upstream segment); and gathering of natural gas, fractionation of natural gas liquids (NGLs) and transportation of NGLs (midstream segment).
Background
Our client had an aging system in place for exchanging data between two of their business support servers: SCADA, a system responsible for monitoring the flow of products through their pipelines, and CMS, their customer management system. The data coming from SCADA was used in CMS to bill our client’s customers for the use of the pipelines.
Our client was getting ready to implement a new version of SCADA and saw the need to update their communication lines between CMS and the new system at the same time. Additionally, they saw the possibility that other systems may need to share data with the CMS system. The Somexsoft team recognized this as an excellent fit for BizTalk Server to be put into use. This project was mission-critical since it directly affected our client’s revenue stream.
Solution Detail
The first stage of the solution concentrated on replacing the existing communication lines between CMS and SCADA with a BizTalk solution. Because SCADA was being replaced, it did not make business sense to spend money on it to get it to work with BizTalk, so the solution was architected such that SCADA did not need to be modified. Instead, BizTalk was programmed to receive and send flat files in the format that SCADA expected, and converted the data into an internal XML format to be used in the orchestrations that implemented the business logic.
The second stage of the solution focused on connecting CMS to the new DNA system. Whereas SCADA communicated using flat files placed on a network share, DNA communicated with MSMQ. Because the type of data flowing between DNA and CMS was the same, the business logic remained the same, and so the BizTalk orchestrations could be reused in support of the interface to DNA. The only new development for implementing the DNA interface was creating maps to map DNA’s XML schemas to the internal schema used by the existing orchestrations and to slightly modify the orchestrations so that they no longer produced the command files that the SCADA system required.
Benefits
Since this project effort supported a legacy application rewrite, the new functionality was achieved at a lower cost than alternative options. It is also easier to maintain, reducing labor costs, and allows our client to support any future expansion efforts.
The BizTalk orchestrations use a common schema format and the connecting systems have maps to transform their data into this common format. Consequently, additional systems can be configured to share their data with CMS in the future.
Because CMS used an Oracle database and BizTalk had no out-of-the-box Oracle send or receive adapters, the Somexsoft team created generic send and receive BizTalk adapters that use the Microsoft data access application blocks to access the Oracle database. The receive adapter operates on a scheduled basis and can retrieve data based on a SQL statement, a stored procedure name, or a table name. Under the covers it uses ADO.NET in conjunction with the data access application block to store and retrieve data.
Reliability and scalability were provided by the out-of-the box features of BizTalk 2004. The solution has 10 different daily interfaces and five hourly interfaces. After the application had been in production for more than a month it had zero outages or errors of any kind.
Deployment of this solution is allowing our client to retire two costly legacy applications on the UNIX platform.
Challenges
The files exchanged with the SCADA system were tab-delimited text files. It was relatively straightforward to create flat-file schemas using the BizTalk schema editor, but the files were required to have a specific name in order for the SCADA system to recognize them. Searching through Microsoft online resources provided the solution to this problem.
It was known that the SCADA system would be replaced by a system that was still in development. The business rules were the same for each system, so the solution reused the orchestrations for the new system. New maps were added to allow communication to the new system using XML and MSMQ. The orchestrations used schemas that were in a generalized format which were mapped to both the flat file for the old system and XML schema for the new system. This made conversion between systems easy.
Technically Speaking
This Microsoft BizTalk Server 2004 solution replaced a functionally similar solution that was implemented using Perl scripts which accessed the C Server Oracle database. The solution was architected so that the Perl scripts were taken completely out of production, but the SCADA server side was unchanged.
Half of the Perl scripts were for sending data from CMS to SCADA, and the other half received the data from SCADA by first sending a command file to the SCADA system indicating the data for the SCADA system to send back to CMS. These scripts were run on a UNIX system and were run on a scheduled basis by using “cron.”
The files were transferred between systems using FTP. The SCADA system would send any files that needed to be transferred to the CMS system. The files were named with “.out” and “.in” file extensions to indicate whether the file should be directed to SCADA or to CMS.