Error

error btm1044: Input validation error: Non Segment level : [3] Transaction Set Control Number Mismatch

Situation

I was doing a test map on a BizTalk map in Visual Studio.
The data was in XML format, not EDI, so I’m surprised it tried to enforce this error. I would expect that to only occur a Receive Pipeline with real EDI data.

Data

<ns0:X12_00401_204 xmlns:ns0="http://abc.com/X12/204">
	<ST>
		<ST01>204</ST01>
		<ST02>0007</ST02>
	</ST>

        <!-- middle part omitted --> 	

	<SE>
		<SE01>22</SE01>
		<SE02>0001</SE02>
	</SE>
</ns0:X12_00401_204>

Fix

In the example above, I had to set the SE02 value to the same as the ST02 value. Note also that SE01 should be the number of EDI segments, including the ST and SE segments.

NOTE: You would presumably get the same error if you drop an EDI file into a Receive Location with an EDI pipeline.

If you’ve been around BizTalk for a while, like me, you most certainly have gotten this error many times.

The normal solutions are:
1) Make sure you schema is deployed
2) Make sure your schema is not deployed more than once
3) Set “AllowUnrecognizedMessage” to True in the XML Disassembler.
4) Make sure the file you dropped has the proper namespace and root element.

To review, in BizTalk MessageType is the TargetNamespace#RootElement.

Here was the exact message I got when I dropped my test file:

The Messaging Engine failed while executing the inbound map
 for the message coming from source URL:"e:\Integration\Inbound\CPChemNew\204\*.xml" 
 with the Message Type "http://abc.com/X12/204/CPChem2#X12_00401_204". 
 
Details:"Finding the document specification by message type "ST" failed. Verify the schema deployed properly. " 

The message type looked okay.
http://abc.com/X12/204/CPChem2#X12_00401_204

But what was this "ST" in the second message?

Solution

I was converting a regular schema to an envelope schema in order to accomplish debaching. "Body XPath" is a parm you set so that the receive XML pipeline will automatically split the message into multiple messages).

I put the "Body XPath" one element too low. It should have split on X12_00401_204, but actually split the message into an ST, and other similar segments.

Insights

Here's what I didn't understand. I'm not sure why, but after debatching, the XML Disasembler on the Receive Pipelines, decides that it needs to verify that the target schema exists as well. Thus, the error on "ST".

How I Solved It

1. I did a test map in Visual Studio and got the results of the test map.
2. And then dropped that into the appropriate Receive Location (different from the one above).
3. Since I had tracking turned on, I check the "Tracked Message Events", and saw multiple "Transmission Failed" and looking at the body of each, I see one file has as the root, one has as the root, and so on.
4. The other thing I should have noticed was that in the error, the message type should have been: "http://abc.com/X12/204/CPChem2#X12_00401_204Looping" instead of "http://abc.com/X12/204/CPChem2#X12_00401_204". That should have hinted to me that maybe the debatching was happening. X12_00401_204Looping and X12_00401_204 were so similar I didn't notice, so lesson learned here is don't assume and be exact.

The file I'm trying to build looks like this:

<ns0:X12_00401_204_LOOPING xmlns:ns0="http://abc.com/X12/204/CPChem2">
	<X12_00401_204>
           <ST> 
              etc... 

In EDI, a customer can send multiple 204s, but we wanted a single file with multiple 204s in it (too complex to explain here).

The next thing I need to do is make sure I have a scheme that matches , and I need to add a namespace to that as well. So more work to be done, but now I'm back on track. What I need is two schemas, and the Envelope schema can import and reference the other one.

What if you need to import a bunch of BizTalk vocabulary rules (.xml files) into the Biztalk BRE (Business Rule Engine)? You don’t want to do it one at time using the Business Rule Deployment Wizard. The real question, is why Microsoft didn’t provide a batch version of that tool that does all the same features, or incorporate those features into the BTTask command.

Microsoft used to provide a tool called ImportExportRuleStore.exe. A reference to this tool can be found on Tallan Blog.

When you search on Microsoft site or MSDN, you can no longer find it nor download it, although there are some .exe copies of it on the old CodePlex site (under the BiztalkBatchBuild project). However, I didn’t want to download an un-trusted .exe. Apparently Microsoft never provided the source for it.

BTDF (BizTalk Deployment Framework) surely has some way to do import rules as well, but I didn’t dig through it. If they do, it’s probably in an MSBuild file.

At least part of that utility now seems to be replaced with a tool called “Microsoft.Practices.ESB.RuleDeployer.exe” in the Microsoft ESB directory (if you have that installed).

You can specify parameters of “-v filename.xml” to import a vocabulary.

See related blog where I have a PowerShell script to batch import all vocabulary files in a directory.

While the GitHub says it’s only for BizTalk 2013-R2 and lower, it’s just C# code that should work with any version of BizTalk, unless BizTalk makes some breaking changes in future releases. But it’s unlikely they would do that, as many customers have pipeline components and business rules.

Visual Studio 2015 Builds

To get it to work with BizTalk 2016, download the code from GitHub, and open it with Visual Studio 2015. Open the solution called BREPipelineFramework.sln. For each of the four projection in that solution, right click “properties” and set the .NET framework to 4.6 as shown below.

NOTE: I had to rename BREPipelineFramework.PipelineComponent.BRPEPiplineFramework.Component, a ridiculously long name, to something shorter, such as: BREPipelineFramework.PipelineComponent. The compile was given me an issue about exceeding the max file size. So in other words, my GitHub repository was something like C:\users\myname\Source\Repos\BizTalkPipelineFramework, and the subfolder structure within that just made some of the names a little too long.

I will add any additional notes here as I work with installing on BizTalk 2016.

Run GACUtil on these .DLLS

I used my PowerShell version of GacUtil to deploy the following four DLLs:

$dllpath1 = "c:\Users\MyName\Source\Repos\BizTalkPipelineFramework\BREPipelineFramework\BREPipelineFramework\bin\Debug\BREPipelineFramework.dll"
GacDll $dllPath1 

$dllpath2 = "c:\Users\MyName\Source\Repos\BizTalkPipelineFramework\BREPipelineFramework\BREPipelineFramework.Helpers\bin\Debug\BREPipelineFramework.Helpers.dll"
GacDll $dllPath2 

$dllPath3 = "c:\Users\MyName\Source\Repos\BizTalkPipelineFramework\BREPipelineFramework\BREPipelineFramework.SampleInstructions\bin\Debug\BREPipelineFramework.SampleInstructions.dll"
GacDll $dllPath3

$dllPath4 = "c:\Users\MyName\Source\Repos\BizTalkPipelineFramework\BREPipelineFramework\BREPipelineFramework.PipelineComponent\bin\Debug\BREPipelineFrameworkComponent.dll"
GacDll $dllPath4 

Create a Test Application Import Pipelines Assembly

I created a new BizTalk Application called BizTalk.Common, and imported the resource/assembly:
c:\Users\nealw\Source\Repos\BizTalkPipelineFramework\BREPipelineFramework.BizTalk2016\BREPipelineFramework.TestProject\bin\x86\Debug\BREPipelineFramework.TestProject.dll

This project contains the .btp Pipeline files that could be used for testing.
I chose not to run the full BizUnit test, as I didn’t want to take the time to install BizUnit, at least or today.

I then created a Receive Port and Location, and I was able to see and pick the pipelines from the BREPipelineFramework.TestProject:

Import Vocabulary and Policies

Import the BizTalk Business Rule Vocabularies:

“c:\Users\MyName\Source\Repos\BizTalkPipelineFramework\BRE Artifacts\Vocabularies\BREPipelineFramework.JSONInstructions.1.0.xml”

When I tried this, I got the error:
“Unable to load assembly BREPiplineFramework.JSON”. Earlier, I had not compiled this because we had not immediate need to do JSON. So at this point, I went back and compiled it and GACed it (after setting it’s .NET framework to 4.6 and building it).

then Import the BizTalk Business Rule Policies:
“c:\Users\MyName\Source\Repos\BizTalkPipelineFramework\BRE Artifacts\Policies\BREPipelineFramework.JSONPolicy.1.0.xml”

But then I realized there was a whole directory of vocabularies to import (25 files in this directory):
“c:\Users\MyName\Source\Repos\BizTalkPipelineFramework\BREPipelineFramework\BRE Artifacts\Vocabularies\..”

So I did some research, and found the Microsoft.Practices.ESB.RulesDeployer.exe replacement for the old ImportExportRulestore.exe utility. If you don’t have the ESB Toolkit installed, then you might have to manually import all 25 files.

So I wrote a little PowerShell to loop through the files in the directory and call that utility:

#
#  Import an entire directory of vocabulary files into BizTalk BRE (Business Rule Engine) 
# 
cls
$sqlServer = "DLBizTalkDev1" #not sure this is supported in this utility 
$commandPath = "e:\BizTalkServer2016\ESB Toolkit 2.4\Bin\Microsoft.Practices.ESB.RulesDeployer.exe"

$vocabDir = "c:\Users\nealw\Source\Repos\BizTalkPipelineFramework\BREPipelineFramework\BRE Artifacts\Vocabularies"
$files = Get-ChildItem -Path $vocabDir -Filter "*.xml"

foreach($file in $files)
{
   $processedCounter = $processedCounter + 1 
   Write-Host "Filename=$($file.Name)"
   $cmdResults = &"$commandPath" -v $($file.FullName)
   Write-Host $cmdResults 
 }

I needed to implement RawString functionality to be able to read a non-XML file into an orchestration. Scott Colestock had a great article on this on his TraceOfThought blog, but when I needed it the other day, it had been hacked (hopefully only temporary).

1. Deploy the C# code towards the bottom of this log; it is not a pipeline, so can be added to new class to an existing C# helper library you might have, or you can put it in its own assembly.

2. Add a reference to this C# Assembly in your orchestration.

3. In your orchestration, create a multipart message type. It will have only one part, but that will be the RawString class from the code below.

4. In the orchestration, create a nessage of type System.Xml.XmlDocument, and assign this to your cativating receive. But wait Neal, I thought you said we wanted to receive a non-XML message. Trust me, this will work.

5. In the orchestration, add a Message Assignment shape and code similar to below:

//convert the message received to a new message with a multiple-part-type of RawString 
msg_rcv_RawString.MessagePart_1 = msg_XML_Doc; 

6. At this point, you can access the message as a string, for example:

varText = msg_rcv_RawString.MessagePart_1.ToString();

Voila – you have now have the text of the file in a variable called varText. You can then parse it, or do whatever you want with it.

The code below needs to be in a C# helper class that you can access within your orchestration. (Obviously give it a strong name and GAC it.) I can’t remember which site I got this code from, but I think it’s out there on several.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Xml;
using System.Xml.Serialization;
using System.Runtime.Serialization;
using System.IO;
using Microsoft.XLANGs.BaseTypes;

namespace REI.BizTalkNonDB.Helpers
{

    public abstract class BaseFormatter : IFormatter
    {
        public virtual SerializationBinder Binder
        {
            get { throw new NotSupportedException(); }
            set { throw new NotSupportedException(); }
        }

        public virtual StreamingContext Context
        {
            get { throw new NotSupportedException(); }
            set { throw new NotSupportedException(); }
        }

        public virtual ISurrogateSelector SurrogateSelector
        {
            get { throw new NotSupportedException(); }
            set { throw new NotSupportedException(); }
        }

        public abstract void Serialize(Stream stm, object obj);
        public abstract object Deserialize(Stream stm);
    }


    public class RawStringFormatter : BaseFormatter
    {
        public override void Serialize(Stream s, object o)
        {
            RawString rs = (RawString)o;
            byte[] ba = rs.ToByteArray();
            s.Write(ba, 0, ba.Length);
        }

        public override object Deserialize(Stream stm)
        {
            StreamReader sr = new StreamReader(stm, true);
            string s = sr.ReadToEnd();
            return new RawString(s);
        }
    }

    [CustomFormatter(typeof(RawStringFormatter))]
    [Serializable]
    public class RawString
    {
        [XmlIgnore]
        string _val;

        public RawString(string s)
        {
            if (null == s)
                throw new ArgumentNullException();
            _val = s;
        }

        public RawString()
        {
        }

        public byte[] ToByteArray()
        {
            return Encoding.UTF8.GetBytes(_val);
        }

        public override string ToString()
        {
            return _val;
        }
    }


}

Error:

the type or namespace ‘BaseCustomTypeDescriptor’ could not be found

Solution:

If you are working on BizTalk pipeline components, you may have copied over parts of c:\Program Files (x86)\Microsoft BizTalk Server 2013\SDK\Samples\Pipelines\ArbitraryXPathPropertyHandler to your own solution or project. You either did not add the required project references, or some of the .cs file that are referenced. Specifically BaseCustomTypeDesciptor can be found in the FixMsgDescription.cs C# module.

Also: If you get error on “BitMap” could not be found, you need the System.Drawing namespace.

Below is a screen shot of the References you may need to add, and the .cs modules you might need to copy:

To add the reference, you can open the sample project, click the reference, and find the “Path” in the properties window. You can copy that path, then go to your project, and when you do an “Add Reference” you can browse to that path and select it.

Sometimes you ask yourself what version of BizTalk is running on a certain server. Maybe you’re a consultant or employee stepping into a new job/contract and you are trying to get the lay of the land. Or maybe you have some old servers that are not documented.

The following script will answer that question. Might save keystrokes of opeining Control Panel or RegEdit to find out.

See this page to get a list of all the BizTalk versions. Hope you don’t find anything older than BizTalk 2010!

cls
#read the registry 
$item = Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\BizTalk Server\3.0"
Write-Host "Biztalk Internal Version: $($item.ProductVersion)"
$major,$minor, $x, $y = $item.ProductVersion.split('.') 
#Write-Host "$major $minor" 

# see https://social.technet.microsoft.com/wiki/contents/articles/7915.biztalk-server-versions.aspx 

$version = switch ($minor)
{
    13 {"2020"}  #assumed 
    12 {"2016"}
    11 {"2013/R2"}
    10 {"2013 (not R2)"}
    9 {"2010"} 
    8 {"2009"} 
    6 {"2006/R2"} 
    5 {"2006 (not R2)"} 
    0 {"2000-2004 Past Time to Upgrade! "} 
}

Write-Host "BizTalk Common Name: $version" 
Write-Host "Product Edition: $($item.ProductEdition)" 

Example Output:

Biztalk Internal Version: 3.10.229.0
BizTalk Common Name: 2013 (not R2)
Product Edition: Enterprise

BizTalk has a schedule job that does incremental backups, usually every 15 minutes. But what if you want to do a full backup immediate?

Run this command in SQL Server Management Studio (SSMS):

USE BizTalkMgmtDb
EXEC sp_ForceFullBackup

Then manually start the SQL agent job “Backup BizTalk Server (BizTalkMgmtDb)” (or wait until the next scheduled 15 minute interval).

The above is required if you get in the situation where you see this error in the 3rd step of the above backup agent job:

Error: BACKUP LOG cannot be performed because there is no current database backup.

Full error looks something like this:

Send Port: “sp_Vendor_Vendor_FilesToSendToAS2” URI: “http://as2.vendorsite.com/”
Reason: The Encryption Certificate has not been configured for AS2 party. AS2-From: YourAS2ID AS2-To: VendorAS2ID

Solution:
Open the send port, click the “Certificate” item on the left.
Then click the “Browse” button under the certificate, and select the certificate that applies to your vendor. The certificate has to be first stored in the “Local Computer/Other People Store”, as described on this MSDN doc.

The selection for the certificate will look something like this. I had to black out our actual vendor/certificate names.

test

This script can be scheduled to create a nightly backup of all the bindings and MSIs for all the applications in BizTalk. It uses the Powershell Extensions for BizTalk to enumerate the application names, then calls BTSTask once for the bindings, and once for the MSI.

It gives you the ability to customize your backup folder names and files names. For example, if your application name is “ABC.Orders”, the backups are:
ABC.Orders_2019_05_17__09_10_21.xml
ABC.Orders_2019_05_17__09_10_21.msi

You could make changes to create a new folder for each set of backups, example: Backups_2019_05_17__09_10_21.

Another enhancement could be to zip the archives, or to purge archives over x days.

There is one trick. You may also want to backup the Parties. I’m currently on BizTalk 2013, and I know there are some improvements for supporting importing/exporting parties individually in 2016. Given that I’m on BT2013, I decided to backup the parties just once. So if the application name = “BizTalk EDI Application”, I add the parameter for the party backup, and they will be in that file. You could also create a dummy empty application and use it.

<#

       Author: Neal Walters 
  Description: Backup BizTalk Applications (both Bindings and Application/MSIs) via Script 
         Date: May 2019 

#>

Add-PSSnapIn -Name BiztalkFactory.PowerShell.Extensions
#This next line only needed when SQL server is not on same machine as BizTalk
New-PSDrive -Name BizTalk -Root BizTalk:\ -PsProvider BizTalk -Instance "YourSQLServer" -Database BizTalkMgmtDb

$exportFolder = "c:\Backup\BizTalk" 
$server = "YourSQLServer"
$database = "BizTalkMgmtDB"

cls

$apps = Get-ChildItem -Path "Biztalk:\Applications"

foreach ($app in $apps) 
    {
        Write-Host "===== Application: $($app.Name) ==== " 

        if ($app.Name -ne "BizTalk.System")   # avoid error trying to export this app 
          {
            $fmtDateTime = $(get-date -f yyyy_MM_dd__HH_mm_ss)
            $bindingsDestFilename = "${exportFolder}\$($app.Name)_$fmtDateTime.xml" 
            $msiAppDestFilename = "${exportFolder}\$($app.Name)_$fmtDateTime.msi" 
           
            write-host $bindingsDestFilename 
            $allArgs = @("ExportBindings", "/Destination:$bindingsDestFilename",  "/ApplicationName:$($app.Name)",
                "/Server:$server", "/Database:$database") 
            if ($app.Name -eq "BizTalk EDI Application") 
            {
               #just backup the parties one time with this one app 
               $allArgs += "/GlobalParties"
            }

            write-host "Args=$allArgs"
            & "c:\Program Files (x86)\Microsoft BizTalk Server 2013\BTSTask.exe" $allArgs

            $allArgs = @("ExportApp", "/Package:$msiAppDestFilename",  "/ApplicationName:$($app.Name)",
                "/Server:$server", "/Database:$database") 
            write-host "Args=$allArgs"
            & "c:\Program Files (x86)\Microsoft BizTalk Server 2013\BTSTask.exe" $allArgs
          }
    }