Discovering Data Services through Use Cases: Guaranteed Delivery

[admin]

Talend recently announced the availability of the Talend Platform for Data Services. Built around Talend v5’s data integration and ESB technologies, it lets IT organizations operationalize data through the single development, deployment, runtime and monitoring environment of the Talend Unified Platform.

Through a couple blog posts, we look at typical use cases of the Talend Platform for Data Services. In a recent post, we looked at data virtualization. This post explains the use of ESB as a persistent transport layer for guaranteed message delivery.

Unlike point to point interfacing between source and target, the ESB abstracts the transport layer and handles issues such as latency, reliability, security – to ensure a message is guaranteed to be delivered to its recipient. As an example, we can consider a retailer with many small stores, each of them running no more than a few point of sales (PoS) systems. Each of these PoS systems must feed its transactions in near-real-time to a centralized application. Polling each system from the centralized location would place a huge burden on the central system, and would be highly inefficient in term of performance.

Instead, data integration jobs are developed that, upon the execution of a transaction, creates a message containing the relevant information, and sends this message to the central system via the ESB’s messaging system. The ESB handles latency and throttling, network instabilities, and provides guaranteed delivery of the message. At the other end (on the central system side), data integration jobs are being triggered by incoming ESB messages and populate the consolidated transaction database. Because of the high volumes, these jobs are being managed as a pool of services that can scale elastically based on demand and volumes.

In order to ensure quality of the data coming through these messages, data quality jobs can also be developed, exposed as SOAP or REST services, and used by the ESB to deduplicate, cleanse and otherwise enrich the data prior to its insertion in the consolidated database.

An additional benefit is derived in case of the addition of heterogeneous PoS systems (M&A, new version, etc.). Only the data integration jobs need be modified to extract data and transform it to the pivot message format – no change is required on the central system side. Data Services add a layer of “insulation”, moreover it can support multiple formats and versions of the interface at the same time, whilst still ensure an architectural consistency across the enterprise, meaning that as other applications are integration or new business applications developed, they can be easily added in a very cost effective manner.

Finally, when changes are required to the data integration jobs, the ESB handles their deployment to all the PoS systems, greatly facilitating the management of the integration infrastructure.

Yves






More from the Talend blog ...