Yes, I admit it, I was a mainframe developer and DBA for the first 20 years of my career. I did a lot of COBOL, then moved into IDMS database adminstration. But after the Y2K (year 2000 date “fix” projects), I moved into the world of Microsoft, first as a trainer, but then I narrowed my focus down Mirosoft BizTalk Server.

BizTalk is a server product from Micosoft, that allows companies to automate business processes, and is often used for Business to Business (B2B) and Enterprise Application Integration (EAI). I’ve been working with BizTalk now over 12 years, in a variety of companies and industries.

So why is BizTalk the new COBOL? About 90% of the work I’ve seen on BizTalk runs in batch. What is batch? Today, the hot thing seems to be user experience (UX), designing interactive web pages and screens using the latest JavaScript and interactivity. EAI and B2B most frequently deals with moving data back and forth between systems according to business processes.

In the old COBOL days, we would receive a magnetic tape from another company, then run a COBOL program to load it into a VSAM file or database (or even, heaven forbid, a flat file). Nowadays, with BizTalk, I receive data from other companies via various methodologies, often FTP, MQSeries, or web, then process that data in batches. Okay, maybe 30% of the time we receive the data in “one-sies” – i.e. one single order at a time, but I would say most of the time it comes in batches. Then with BizTalk we often have to debatch the data, then call a a web service or a SQL stored procedure to store the data into our back end enterprise systems.

With BizTalk, I usually deal with incoming data and outgoing data from trading partners, such as vendors, customers, or service providers. A company sends data to us, we process it, and then send data back to the other company. I’ve worked in cases where I receive airline fueling data, then after the plane is fueled, we receive the transaction, then at night, we send batches of data back to the airlines or other parties like the fueling companies.

In another company, we receive a list of what a company is going to ship, we store that into our backend ERP system. When the items arrive physically, we send back a confirmation that we received the items. The items are then repaired, and we send them an a notice that the repair has completed, and then another one when we ship the items. In some case the items are shipped directly to the customer, and thus individual transactions, but other times we ship full palettes of items, thus the messages are backed up, and sent together.

For another past client, we received CSVs (Comma Separated Value files) from Universities, and had to debatch and load into our system related to student loans. For a dot com company, we created a list (i.e. a batch) of all the products we had to sell, and uploaded to Amazon via web services.

Several things made me think of this analogy, of BizTalk to COBOL. One is how I often debug. With BizTalk, I have created my own Trace facility, that dumps the message to a SQL table; to me, it’s a little fancier and much easier to use than using the built-in BizTalk message tracking. It reminds me of the days where I did similar, or added many Display statements to long COBOL programs. Then to test and debug, I pour over pages of output, looking for issues, anomalies, and patterns. Online and interactive developers are more likely to use an interactive debugger, setting breakpoints, and/or walking through code.

Batches are often processed at night for several reasons. 1) They represent the end of a days activity, or 2) a batch of orders for the next day, or 3) the computers are less overloaded at night, so it’s just a good time to do the processing. In the old mainframe days, we used to have this horrible concept called the night processing window. The online and the batch world didn’t seem to coexist very well. Thank goodness, this is less of an issue today. However, there are still many companies that make copies of their database on which to run queries, so as to not slow down the primary system.

Since batches are often run at night, BizTalk guys are often on-call during off-hours. BizTalk developers tend to have know a lot about infrastructure and communications, and have to be involved with fixes to certificates, IIS, FTP, and even router and data pipeline issues.

BizTalkers are often viewed at as “those other guys”. Everyone else is playing with the latest GUI widget, and we are still working with core processes that keep the business running. The web guys often get all the glory, and the BizTalk guys get blamed when anything goes wrong.

BizTalk has two modes of processing messages: 1) content-based routing, and 2) orchestrations and sometimes an 3) Enterprise Service Bus (itineraries) for those companies that use that). With content-based routing, there is less coding, and the coding that is done is done in something called Pipelines (written in C#). With the second, orchestrations are graphical representations of the message/data flow created in Visual Studio. Within an orchestration, we create maps to translate the data (for example, the customer calls the data Order_No, and we called it PO_NUM). We also insert short bits of X/Lang code, which is about 90% the same as C#, and we can invoke methods in C# class libraries.

Instead of JCL (Job Control Language), we create ESB’s and orchestrations, tying it all together with SendPorts and ReceivePorts taking the place of DDNames, such as the the two mainframe DDnames below (“INFILE” and “OUTFILE”):

 

//MyJobNam JOB Class=E
//Step01 Exec pgm=MyProgNam
//INFILE   DD DSN=MYAPP.PROD.MYFILE1,DISP=SHR
//OUTFILE DD DSN=MYAPP.PROD.MYFILE2,DISP=SHR

 

Now we create a SendPort, and point to a disk file.

BizTalk “Batch Jobs” are monitored on Biztalk Admin Console, which is basically the equivalent of SDSF (Spool Display and Search Facility in TSO/ISPF – for you mainframe guys, or some mainframe sites used a tool called “JARS” instead). Orchestrations are run in Host Instances, instead of Job Run Classes. Orchestrations can be terminated, just like mainframe jobs. Of course, most batch COBOL program used “DISPLAY” statements to output their results. BizTalk doesn’t quite have this, but Biztalk Admin console can allow you to view XML messages processed by the orchestration. And of course, Orchestrations can blow up or “ABEND” just like COBOL jobs, we just call them “SUSPENDS” instead.

 

 

Can BizTalk be interactive? Well, yes, it actually can. Orchestrations can be published as web services. So when a client calls your web service, it actually can run a BizTalk orchestration under the cover. Since the client is expecting a quick response, there are special considerations and practicalities for providing for this real time response. Many times the client is running in a batch mode at the client companies. I have two cases in my career where we actually invoked an orchestration real-time. One was interesting, because it was connecting a CICS mainframe 3270 application to BizTalk via HIS (Host Integration Services), in order to call an external vendor’s web service and return insurance data.)

So I hope the title of the article wasn’t too misleading. COBOL is not just a batch language, it used for online as well. What I’m really trying to say is that batch processing is alive and well. Data still comes in batches, whether in XML stored in a queue, or whether a CSV or flat file.

The author, Neal Walters worked on mainframes for about 20 years, until the Y2K date conversion was complete. He worked with COBOL, CICS, end-user tools (Focus, SAS) and was an IDMS Database Administrator and Developer (and still occasionally sells an IDMS Tutorial Course and Documentation). After Y2K, Neal became a Microsoft Certified Trainer, and from there started training BizTalk and then moved into BizTalk Consulting for since 2004.