Suppose you walk into a new company, they don’t have documentation, and they use a lot of content-based routing. The first thing I do is usually draw a Visio diagram. There never really been one widely adopted standard for how to draw such diagrams. The secret is to pack the most useful information that you can in the smallest space.

When drawing BizTalk diagrams, I like to keep everything one page if possible. Of course, sometimes you have to break on multiples pages, and when I do, I don’t like cross-page lines or pointers, if they can be avoided.

In BizTalk, a Send Port or a SendPort Group can subscribe to the data coming in from a Receive Port (and of course one Receive Port can have multiple Receive Locations). The subscription however, is by the ReceivePortName, not the ReceiveLocationName. You can have multiple send ports subscribe to the same message (thus the name Pub-Sub, short for Publisher/Subscriber model, upon which BizTalk is based).

BizTalk_Filter_Subscriptions

To get to the above screen, I found a SendPort in BizTalk Admin Console, right clicked on it, then selected “Filters” on the left side. The above has one simple filter. All messages received from a ReceivePortName of MSMQDemo_RcfFromMSMQ_biztalktrans will be routed to this SendPort. You can add and change the filter/subscriptions on the top part of the screen. For convenience, the filters are shown at the bottom (and you can copy/paste them from there). It’s hard to see the entire names on the top, because of the limited width of the columns.

So often you can use business logic, naming convention, and and guessing to determine which SendPorts subscript to which ReceivePorts. Or you can open each SendPort and write them down, or copy/paste the above filters into a Word Doc, Spreadsheet, or Visio diagram.

But all the data in BizTalk Admin Console is stored in one BizTalk’s SQL Databases (BizTalkMgmtDB). The query below allows you find all SendPorts subscribing to a ReceivePort.

The code below was original found here: gallery.technet.microsoft.com/scriptcenter.

use BizTalkMgmtDb
go  -- If you have the "use" above, without the "GO" here you will get an error on the "With" statement
WITH
TmpXMLNode
  ( SendPortName, ApplicationName, tempXMLcolumn )
AS
  (SELECT
    SP.nvcName AS SendPortName
    , APP.nvcName AS ApplicationName
    , CAST(REPLACE(REPLACE(REPLACE(CONVERT(NVARCHAR(MAX), SP.nvcFilter),'>','>'),'<','<'),
      'xmlns="http://www.w3.org/2001/XMLSchema-instance"','') AS XML) AS tempXMLcolumn
  FROM
    bts_sendport AS SP
    INNER JOIN bts_application AS APP ON SP.nApplicationID = APP.nID
  WHERE
    CONVERT(VARCHAR(MAX), nvcFilter) <> ''
  )

SELECT 
  SendPortName,
  ApplicationName,
  CONVERT(VARCHAR(255), nref.query('data(@Value)')) AS FilterValue,
  CONVERT(VARCHAR(255), nref.query('data(@Property)')) AS FilterProperty,
  CONVERT(VARCHAR(255), nref.query('data(@Operator)')) AS FilterOperator
FROM
  TmpXMLNode
  CROSS APPLY
    TmpXMLNode.tempXMLcolumn.nodes('/Filter/Group/Statement') AS R(nref)
WHERE
  CONVERT(VARCHAR(255), nref.query('data(@Property)')) = 
     'BTS.ReceivePortName' -- filter type  (do not change the value in quotes) 
  AND CONVERT(VARCHAR(255), nref.query('data(@Value)')) = 
     'YourRcvPortHere'  -- change this value in quotes  

You can of course put a LIKE clause on the final where clause:

  AND CONVERT(VARCHAR(255), nref.query('data(@Value)')) like 'MSMQ%'  

Results of query in SSMS:
Result_of_BizTalkMgmtDB_Sql_Query_Against_SendPorts_Filters2
You may recall that you can also filter on your own schema/promoted fields. So you could have one SendPort subscribe to PurchaseOrderAmount > 5000 and a different one subscribe to PurchaseOrderAmount <= 5000.

Note in the above query, I added the “Operator” column, which was not in the original SQL sample.

The filters above are stored in a text column called nvcFilter. Part of the trick of the SQL is convert this to an XML column, so XQuery statement can be performed on it.

So to demonstrate what is going on internally, I created a dummy SendPort with several filters just to see something beyond the most simple example.

BizTalk_SendPort_Complex_Filters

I then ran this query

SELECT *
  FROM bts_sendport AS SP
    INNER JOIN bts_application AS APP ON SP.nApplicationID = APP.nID
  WHERE  nvcFilter like '%msmq%'

I then copied the contents of the nvcFilter column to NotePad++ and formatted it:

<Filter xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<Group>
		<Statement Property="BTS.ReceivePortName" Operator="0" Value="ABC" />
		<Statement Property="MSMQ.Priority" Operator="3" Value="5" />
		<Statement Property="MSMQ.SourceMachine" Operator="0" Value="Machine1" />
	</Group>
	<Group>
		<Statement Property="BTS.ReceivePortName" Operator="0" Value="DEF" />
		<Statement Property="BTF2.commitmentRct_decidedAt" Operator="0" Value="test" />
	</Group>
	<Group>
		<Statement Property="BTF2.commitmentRct_commitmentCode" Operator="0" Value="test" />
	</Group>
</Filter>

 

So from this, we can discern that the > Operator is stored as a 3. One could continue this exercise and map each of the operators to the numbers, and then decode the numeric values back to the humanly readable values to show in the query results.

Notice also that the “and”s and “or”s are saved in separate “Group” elements.

So now, you know how to find and cross-reference all the SendPorts that are used by your ReceivePorts, and hopefully that will help you figure out content-based routing and better understand and document your system.

You can specify more than one email in the to: of the SendPort.  Just be sure and use a semicolon as the separator, and not a comma.

SMTP Send Port Configuration

Mainframe_TapeDriveYes, I admit it, I was a mainframe developer and DBA for the first 20 years of my career. I did a lot of COBOL, then moved into IDMS database adminstration. But after the Y2K (year 2000 date “fix” projects), I moved into the world of Microsoft, first as a trainer, but then I narrowed my focus down Mirosoft BizTalk Server.

BizTalk is a server product from Micosoft, that allows companies to automate business processes, and is often used for Business to Business (B2B) and Enterprise Application Integration (EAI). I’ve been working with BizTalk now over 12 years, in a variety of companies and industries.

So why is BizTalk the new COBOL? About 90% of the work I’ve seen on BizTalk runs in batch. What is batch? Today, the hot thing seems to be user experience (UX), designing interactive web pages and screens using the latest JavaScript and interactivity. EAI and B2B most frequently deals with moving data back and forth between systems according to business processes.

In the old COBOL days, we would receive a magnetic tape from another company, then run a COBOL program to load it into a VSAM file or database (or even, heaven forbid, a flat file). Nowadays, with BizTalk, I receive data from other companies via various methodologies, often FTP, MQSeries, or web, then process that data in batches. Okay, maybe 30% of the time we receive the data in “one-sies” – i.e. one single order at a time, but I would say most of the time it comes in batches. Then with BizTalk we often have to debatch the data, then call a a web service or a SQL stored procedure to store the data into our back end enterprise systems.

With BizTalk, I usually deal with incoming data and outgoing data from trading partners, such as vendors, customers, or service providers. A company sends data to us, we process it, and then send data back to the other company. I’ve worked in cases where I receive airline fueling data, then after the plane is fueled, we receive the transaction, then at night, we send batches of data back to the airlines or other parties like the fueling companies.

In another company, we receive a list of what a company is going to ship, we store that into our backend ERP system. When the items arrive physically, we send back a confirmation that we received the items. The items are then repaired, and we send them an a notice that the repair has completed, and then another one when we ship the items. In some case the items are shipped directly to the customer, and thus individual transactions, but other times we ship full palettes of items, thus the messages are backed up, and sent together.

For another past client, we received CSVs (Comma Separated Value files) from Universities, and had to debatch and load into our system related to student loans. For a dot com company, we created a list (i.e. a batch) of all the products we had to sell, and uploaded to Amazon via web services.

Several things made me think of this analogy, of BizTalk to COBOL. One is how I often debug. With BizTalk, I have created my own Trace facility, that dumps the message to a SQL table; to me, it’s a little fancier and much easier to use than using the built-in BizTalk message tracking. It reminds me of the days where I did similar, or added many Display statements to long COBOL programs. Then to test and debug, I pour over pages of output, looking for issues, anomalies, and patterns. Online and interactive developers are more likely to use an interactive debugger, setting breakpoints, and/or walking through code.

Batches are often processed at night for several reasons. 1) They represent the end of a days activity, or 2) a batch of orders for the next day, or 3) the computers are less overloaded at night, so it’s just a good time to do the processing. In the old mainframe days, we used to have this horrible concept called the night processing window. The online and the batch world didn’t seem to coexist very well. Thank goodness, this is less of an issue today. However, there are still many companies that make copies of their database on which to run queries, so as to not slow down the primary system.

Since batches are often run at night, BizTalk guys are often on-call during off-hours. BizTalk developers tend to have know a lot about infrastructure and communications, and have to be involved with fixes to certificates, IIS, FTP, and even router and data pipeline issues.

BizTalkers are often viewed at as “those other guys”. Everyone else is playing with the latest GUI widget, and we are still working with core processes that keep the business running. The web guys often get all the glory, and the BizTalk guys get blamed when anything goes wrong.

BizTalk has two modes of processing messages: 1) content-based routing, and 2) orchestrations and sometimes an 3) Enterprise Service Bus (itineraries) for those companies that use that). With content-based routing, there is less coding, and the coding that is done is done in something called Pipelines (written in C#). With the second, orchestrations are graphical representations of the message/data flow created in Visual Studio. Within an orchestration, we create maps to translate the data (for example, the customer calls the data Order_No, and we called it PO_NUM). We also insert short bits of X/Lang code, which is about 90% the same as C#, and we can invoke methods in C# class libraries.

Instead of JCL (Job Control Language), we create ESB’s and orchestrations, tying it all together with SendPorts and ReceivePorts taking the place of DDNames, such as the the two mainframe DDnames below (“INFILE” and “OUTFILE”):

 

//MyJobNam JOB Class=E
//Step01 Exec pgm=MyProgNam
//INFILE   DD DSN=MYAPP.PROD.MYFILE1,DISP=SHR
//OUTFILE DD DSN=MYAPP.PROD.MYFILE2,DISP=SHR

 

Now we create a SendPort, and point to a disk file.

BizTalk “Batch Jobs” are monitored on Biztalk Admin Console, which is basically the equivalent of SDSF (Spool Display and Search Facility in TSO/ISPF – for you mainframe guys, or some mainframe sites used a tool called “JARS” instead). Orchestrations are run in Host Instances, instead of Job Run Classes. Orchestrations can be terminated, just like mainframe jobs. Of course, most batch COBOL program used “DISPLAY” statements to output their results. BizTalk doesn’t quite have this, but Biztalk Admin console can allow you to view XML messages processed by the orchestration. And of course, Orchestrations can blow up or “ABEND” just like COBOL jobs, we just call them “SUSPENDS” instead.

 

Sample IBM SDSF Job Monitor

IBM SDSF Sample – Monitoring Batch Jobs

 

BizTalk_Admin_Console_Montior_Active_Jobs

BizTalk Admin – Active Service Instances (i.e. Jobs)

Can BizTalk be interactive? Well, yes, it actually can. Orchestrations can be published as web services. So when a client calls your web service, it actually can run a BizTalk orchestration under the cover. Since the client is expecting a quick response, there are special considerations and practicalities for providing for this real time response. Many times the client is running in a batch mode at the client companies. I have two cases in my career where we actually invoked an orchestration real-time. One was interesting, because it was connecting a CICS mainframe 3270 application to BizTalk via HIS (Host Integration Services), in order to call an external vendor’s web service and return insurance data.)

So I hope the title of the article wasn’t too misleading. COBOL is not just a batch language, it used for online as well. What I’m really trying to say is that batch processing is alive and well. Data still comes in batches, whether in XML stored in a queue, or whether a CSV or flat file.

The author, Neal Walters worked on mainframes for about 20 years, until the Y2K date conversion was complete. He worked with COBOL, CICS, end-user tools (Focus, SAS) and was an IDMS Database Administrator and Developer (and still occasionally sells an IDMS Tutorial Course and Documentation). After Y2K, Neal became a Microsoft Certified Trainer, and from there started training BizTalk and then moved into BizTalk Consulting for since 2004.

I have pretty much standardized on SSO as a resource for getting parameters and values for Orchestrations.  I started doing this with the BizTalk Deployment Framework (BTDF).

Recently, I was at a client who chose not to use BTDF, but we still needed a way to store and retrieve values from SSO.

So here’s what we did.

  1. The first thing you probably want, is a free tool to update SSO.   Microsoft now provides this, as an MMC (Microsoft Management Console) Snap-in.Once installed, you have to remember the directory it’s stored in:
    SSO_MMC_Tool_Location

When you click on the “mmc” (the .msc file), it loads a snap-in in MMC, allowing you to maintain your SSO applications and variables.

2. Use the tool to add applications, and to add/modify key/value pairs in the app.

           SSO_Application_Configuration_MMC_Tool

 

Below shows the pop-up menu when you right-clik on an application.

 

            SSO-App-Right-Click-Menu

When you choose “Add Key Value Pair” you can enter the values as shown below:

                    SSO_App_Configuration_Key_Value_Properties

Years ago, Richard Seroter laid the groundwork, and created an SSO C# Windows-Form utility that essentially does the same.

The code he provided still works with the MMC snap-in above.

3.  Access your variables (key/value pairs at run-time).

a) Create a C# helper utility to share across your applications

    // had to remove "Static" from class below to get it to appear in BizTalk Map 
    public class SSOConfigHelper
    {
        private static string idenifierGUID = "ConfigProperties";

        ///

        /// Read method helps get configuration data
        ///

///The name of the affiliate application to represent the configuration container to access ///The property name to read /// /// The value of the property stored in the given affiliate application of this component. /// public static string Read(string appName, string propName) { try { SSOConfigStore ssoStore = new SSOConfigStore(); ConfigurationPropertyBag appMgmtBag = new ConfigurationPropertyBag(); ((ISSOConfigStore) ssoStore).GetConfigInfo(appName, idenifierGUID, SSOFlag.SSO_FLAG_RUNTIME, (IPropertyBag) appMgmtBag); object propertyValue = null; appMgmtBag.Read(propName, out propertyValue, 0); return (string)propertyValue; } catch (Exception e) { System.Diagnostics.Trace.WriteLine(e.Message); throw; } } }

b) Call that C# helper from your orchestrations

In the example below, strSSOAppName and strYNTraceOnFromSSO are first defined as orchestration variables (of type “string”):

strSSOAppName = "Demo";

strYNTraceOnFromSSO = MyLib.Common.BizTalk.Components.SSOConfigHelper.Read
                                              (strSSOAppName, "YNTraceOn");

strYNArchiveXML = MyLib.Common.BizTalk.Components.SSOConfigHelper.Read
                                              (strSSOAppName, "YNArchiveXML");

Then for example, later in the orchestration, I can have a “Decide” shape with a rule that tests:

strYNArchiveXML == "Y"

Based on the value of that flag, I can choose whether or not to send a message to an archive port.

Note for security: Your orchestration will need to run in a host instance that has been defined at least an “Application User” to App.  That can be set in the BizTalk Admin Utility that is installed when BizTalk is installed.  (See screen shot towards the end of this article.)

4. Deploy from Dev to QA/PROD

Export your SSO “app” from the Dev system. Right click on the app, and choose “Export Application”. The following screen will appear. You must give it an encryption key. If you don’t have any super secret variables in you SSO, then can set a standard of using your app name for the key name. Once exported, the file will have a .sso file suffix. You can open it with notepad, but it will be encrypted. To import into the target system, the person doing the import will have to know the key you assigned.
SSO_Export_App

Comparison to BTDF’s SSO tool BTDF uses an Excel spreadsheet to enter all the variables and values. The advantage of BTDF is you can put in the values for each and every environment across the columns of that spreadsheet. Then, if I recall, BTDF stores an XML in SSO, and that SSO contains all the key-value pairs. So the tool above and the BTDF tool work differently, the the variables stored in one cannot really be viewed or updated by the other. BTDF also provides a C# windows form utility to update their SSO variables (sometimes needed, for example, in Production, to change some value on the fly).

By the way, when you change a value in SSO, it’s available “almost” immediately. Maybe it’s cached for a few seconds, but not for long.

Why is all the above needed when BizTalk installs with Administration and Client SSO utilities?

These are the two utilities included when you install BizTalk on a machine:
SSO_BuiltIn_Utilties

These utilities have never been that useful for me, except to change the SSO security for an app.

Here’s a trick you need to know. When you oepn the SSO Admin tool, you won’t see the applications above without clicking on the following obscure settings. You have to go to “Affiliate Applications”, right click then “View”, then check “Config Store”.

SSO_Admin_Utility_View_Options

Then when you scroll down to the bottom, you will see the types of SSO applications created by the above utility (or BTDF). There will be a very long list of GUIDs in the “Name” column, then your apps will be at the bottom.

SSO_Admin_Utility_View_Apps

Now, you can right click on the ‘Affiliate Application” named “Demo” and select “Properties”. The following screen appears, and for the screen shot, I have already clicked on the “Fields” tab at the top.

SSO_Admin_Affiliate_Application_Properties

You can see that the values in the “Label” column match the “keys” of our “Key/Value” pairs. So for whatever reason, Microsoft did NOT provide the ability to add or modify what are called “Labels” above. It seemed to take a few years later for them to create the MMC snap-in (described at the top of this blog).

Unless you are new to BizTalk, you maybe more familiar with the “Accounts” tab, shown below.

SSO_Admin_Utility_Accounts

Only users defined as either an “Application Administrator” or an “Application User” can read/write values in the SSO application. Your BizTalk orchestration will be associated to a Host Instance (the process under which BizTalk runs).  That Host Instance runs under the security of a userid, defined in BizTalk Admin console. Typically, a user group is defined as the Application User, and the Host Instance userid is added to that group.

I hope this gives you an overview of how to use SSO with your BizTalk Orchestrations.  I used to put some of my variables in the BizTalk application configuration file (BTNTSVC.exe.config and BTNTSVC64.exe.config), but using SSO is much more “elegant.”  Those config files had several downsides:

  1. There are two files to update, and you have multiple servers in your group (for failover/high-availability), you have to update them, or copy them, to each server.
  2. You have to cycle the host instances for any change in the .config files to become effective.
  3. You don’t have to risk messing up a critical file. If these BizTalk .config files are edited improperly, then BizTalk might not come back up until the mistake is corrected.

Especially in production, it’s a lot easier to use the little GUI utility above to update SSO.

Neal Walters is a BizTalk Architect with over 14 years BizTalk experience in Dallas, Texas (or more specifically Irving/Las Colinas). Beyond that Neal has over 30 years experience in the I.T., starting with Fortran in college and COBOL on his first job. He was an IDMS database developer and database administrator for most of his mainframe career, then switched to the client/server world around Y2K (the year 2000, after the big date change).

The video below can be seen on YouTube here: Dallas TX BizTalk Consultant.

What is BizTalk?

BizTalk is an enterprise level server product sold by Microsoft. It is often called “middle ware”, because it is it acts as a go between by either:

  1. Your company and your trading partners (vendors, customers, etc…)
  2. Various applications within your company

Other may call BizTalk a “messaging system,”, but don’t get is confused with simple queues like MSMQ and MQ-Series. BizTalk can also call web services and provides dozens of adapters, some are technology-based and some are business-based. For example, it can communicate with Microsoft or Oracle SQL, FTP, and it can read and write xml, comma separated files, or “flat files”. It has interfaces for popular back-end systems like SAP, JD Edwards, and PeopleSoft. Click here for a complete list of BizTalk adapters.

While not all companies take advantage of it, BizTalk has an optional Enterprise Service Bus (ESB Toolkit) that is available and supported. It always a developer to create on and off ramps, and to direct messages through a system using something called “itineraries.” Compare these to a multi-legged flight, where you go first from city A to city B, then from city B to city C.

What does BizTalk compete against

The basic competitors of BizTalk are IBM’s WebSphere, Tibco, and MuleSoft.

What does it take to be a BizTalk developer?

Unlike many programming jobs, BizTalk requires one to see the big picture to the whole system. So in addition to programming skills, a senior level BizTalk developer must have a good working knowledge of the Windows operating system, IIS, certificates, security, and performance. Thus basic administrator skills are required. BizTalk developers typically need “admin” access to their machines, to be able to deploy programs to the “GAC”, and to install BizTalk initially.

Major Components of BizTalk

Here are some of the major pieces that make up the system:

  1. Visual Studio plug-ins to allow easy design of maps and orchestrations
  2. Maps – a GUI designer for XSLT maps
  3. Orchestrations – a process flow
  4. Business Rules – allows logic “rules” to be entered in a GUI tool, and changed as needed, without having to redeploy the underlying software
  5. BAM – Business Activity Monitoring – tools for checking the status of the system and selected business data
  6. BizTalk Admin Console – A tool to create applications, Send Ports, Receive Ports, and to define, configure, and control the BizTalk host processes
  7. Adapters – see above
  8. Pipelines – BizTalk includes commonly needed pipelines, and pipeline components and has a way to create your own custom pipelines to manipulate and massage data on the way in or out.
  9. ESB – Enterprise Service Bus

Check the video above to see if you need Neal Walters o help you with your BizTalk needs. Neal is the author of over 300 hours of BizTalk training videos for older releases of BizTalk 2004 and 2006. He has contributed those videos to YouTube under the channel name “MrBizTalk”. Neal now works only on a consulting basis as a BizTalk Architect, Team Lead, or Senior Developer.