The one thing I didn’t account for (yet), is the case where different files can have different EDI delimiters. Technically, you should look for the end of the ISA segment to find the delimiters, and use those in the RegEx match. For now, I’m assuming the field delimiter is * and the segment delimiter is the tilda (~).
I was archiving the EDI files in BizTalk with the filename set to “%datetime%_%MessageID%_EDI.txt”. I decided it would be better to name the files COMPANYNAME_DOCTYPE_ORDERNO_ORDERDATE_%datetime%_%MessageID%_EDI.txt.
NOTE: I could have done this logic in a custom C# BizTalk Pipeline, but decided to do it after the fact with a more simple Powershell than would be easier for administrative staff to maintain and update.
Sample 1 – Just test the parsing
With this sample, you can copy the contents of a file into the $ediText string, and test.
#Note subsituted " with `" in the string to escape the quotes within quotes issue
$ediText = "ISA*00* *00* *ZZ*MYCUSTOMER*ZZ*MYCOUNTRY*170823*1610*U*00401*000000117*0*T*:~GS*PO*BTS-SENDER*RECEIVE-APP*170823*1610*117*T*00401~ST*850*0117~BEG*00*NE*391949**20170828~N1*BY*DELIVERY-ADDRESS~N1*ST*DELIVERY-ADDRESS~N3*1420 MAINSTREET DR~N4*DALLAS*TX*12345~PO1*1*5.00*EA*4.350**IN*106889~PID*F****SAND MIX ( SSM80 )~PO1*2*1.00*etc...~";
$CompanyID = [regex]::match($ediText,'.*ISA\*.*?\*.*?\*.*?\*.*?\*.*?\*(.*?)\*.*').Groups.Value
$OrderNum = [regex]::match($ediText,'.*BEG\*.*?\*.*?\*(.*?)\*.*').Groups.Value
$OrderDate = [regex]::match($ediText,'.*BEG\*.*?\*.*?\*.*?\*.*?\*(.*?)[~\*].*').Groups.Value
$EdiDocType = [regex]::match($content,'.~ST\*(.*?)[~\*].*').Groups.Value
Write-Host "CompanyID = $CompanyID";
Write-Host "OrderNum = $OrderNum";
Write-Host "OrderDate= $OrderDate";
Write-Host "EdiDocType= $EdiDocType";
Having a filename like this will make it faster to search the archives for certain types of orders or files from a certain partner, or do do quick counts, based on the filename alone. For example, how many files did we get from XYZ company yesterday and today? This could be done in BizTalk with BAM as well, but my current client opted out of the overhead and complexity of BAM, especially since BizTalk was (for the most part), just passing the files around, not creating them.
The variable $EdiDocType above represents something like and 850, 855, 856, 810, 997, etc…
I might add one more feature. Many of the trading partner don’t use name, but some Dun number, phone number, or other ID number. I might have a lookup table to translate the code to a shortname that represents that trading partner.
The underlying connection was closed: The connection was closed unexpectedly.
I’m sure there are numerous reasons for this error, but here’s one I encountered today.
First the background. We have BizTalk server connecting to an internal Open/AS2 server running on Linux. That Linux server was re-installed on a Virtual Machine; and I had agreed with the Linux guy to change the AS2-ID (defined in the Trading Parties) from BiztalkTest to BizTalkProd. However, he didn’t make the change on his side.
So when he looked at the logs, he could see that it was receiving data, but as he described it “the server didn’t know what to do with it.” That means, it didn’t generate the MDN and send it back to BizTalk.
I had defined my SendPort was a two-way (Static Solicit-Reponse) and was using content-based routing. So we sent the message to the AS2 server, and didn’t get a response back, giving the “underlying connection closed unexpectedly” error.
Suppose you need to create an EDI file with the ISA header that looks like this:
You need put “ZZ” in the ISA05 field, and “ABCDEFGPLUMBING” in the ISA06 field. How do you do that in BizTalk?
In the Trading Partner Management (TPM) of the BizTalk Admin Console,
when you set up a “party”, you can specify the identifiers. For the “ZZ” identiier, select “Mutually Defined (X12)” from the drop down list under the “Name” column, and specify “ZZ’ as the qualifier, and type in the value “ABCDEFGPLUMBING” or whatever the desired value is in the “Value” column.
The ISA and GS headers will NOT be in your BizTalk map. They will be generated when you use the EDI Pipeline to read or write a file in a send or receive port.
The GS indicates this is the “GS Segment”, the fields are numbered GS01, GS02, GS03, GS04 etc… based on the separator character, which in this case is the Shift-6 carrot symbol.
In BizTalk, this field is NOT mapped by the developer. Instead, the EDI functions of BizTalk creates the line and formats the data on it.
You have to set up Parties in the BizTalk Admin console to use EDI in BT2010. Then you create an “agreement” between those two parties.
Select the agreement, click the proper tab on the top (I had to blur out the company names), then click on the “Envelopes” option on the very lower left.
Scroll right until you find the GS4 segment, then select the desired date format.
You can also type in the desired GS02 and GS03 values above.
Exporting and importing EDI Parties in BizTalk has a few challenges. How do you migrate just one or two parties and agreements? As BizTalk developers, we typically have to move bindings around 2 to 4 times, such as developer machine to DEV or TEST environment, then to UAT (User Acceptance Testing) or QA (Quality Assurnace) environment, and then ultimate to the production environment.
This MSDN post in the BizTalk Forum the question was asked how to export a single party from the Trading Party Management (TPM) system. Sounds like a great opportunity for someone to create a helper solution, but I don’t have time for that right now.
I have tested this following.
1. As a proof of concept, I wanted to see if exporting parties from two different apps gives me the same parties in xml format.
a. From BizTalk Admin Console, pick an application and do right-click and select “Export” then “Bindings”. Click the checkbox in the lower left
b. Now do the same thing for a different application, even one that doesn’t use EDI at all.
c. Edit the two binding files that were created. The goal is to eliminate everything except the parties. So delete delete from <ModuleRefCollection> to line before <PartyCollection> (which is probably </ReceivePortCollection>, then save each file. Now use you favorite file compare utility, such as WinCompare, BeyondCompare, or Total Commander’s built-in file compare, and you will see the PartyCollection sections are identical.
2. Before testing the migration of your EDI Parties from one system to another, you want to be sure you backup the parties on the target system. This is done the same manner as above; there doesn’t seem to be a way to export only parties, you have to pick one app, then export it with “Export Global Party information” checked. If you have to restore, you will have to delete parties and agreements in BizTalk Admin console, then re-import the binding file. (Just importing the binding file would not delete anything new that you would have added.)
A message sent to adapter "FILE" on send port "sp_856_EDI_Out_FILE"
with URI "e:\Messages\MyApp\Process\856EDI_Temp\%SourceFileName%" is suspended.
Error details: Unable to read the stream produced by the pipeline.
Details: Error: 1 (Field level error)
Position in TS: 9
Data Element ID: N401
Position in Segment: 1
6: Leading or trailing space found
Solution 1 – Fix the bad data
The N401 refers to the first element of the N4 segment, which is the city of the city, state, address.
In our case, the client sent us this in the EDI 850 message.
They should have trimmed the spaces after the city name of “CLOVIS”. Oddly enough, the error did not occur on the 850, but on the 856. So we stored the data in our database with the trailing blanks. We extract data to an XML file and saved to disk, so I was able to edit the XML and resubmit it. If you can do this or not depends on how you have your set up your send/receive port structure (or your orchestration flow).
Solution 2 – Turn off EDI validation
Validation can be changed as follows:
1) From BizTalk Admin console, expand the “Parties”
2) Find the Profile, then the Agreement for the specific party. Open the agreement.
3) Click on the top tab that indicates message sent to that party.
4) Click on the left option called “Validation” and you will see something like the screen below.
I had to spend a few hours today to dig into the batching features of BT2010 (I don’t think 2013 is much different, but don’t know about earlier releases).
The MSDN walkthrough can be found here: 2010 or 2013. Finding the MSDN doc for BT2010 itself was a bit of a challenge. In my case, I was needing to debatch 855 and 856 EDI/X12 messages.
Built-In Convoy-Style Orchestration
So basically, there is a sequential convoy style orchestration that can be started, runs continuously, waiting on a receive to feed it more messages, and can spit out a batch occasionally (we will discuss what causes the batch to be created shortly).
Four Orchestrations included with the BizTalk.EDI.Application
The following shows the batching orchestration running in the Group Hub / Running Instances.
Running Orchestration to do Batching
Now, how do we start this orchestration, how do we stop it, and what causes the batched EDI files to be emitted?
Setting up the Agreement
I’m not going to cover BizTalk EDI from the very beginning, that would be another blog and a much longer one. So I’m assuming you have already created some EDI files, and now you need to batch them, and thus you are already familiar with the Party/Agreements.
In the agreement, there is a “link” or item on the left called “Batch Configuration”. From there, you can click “New Batch” and fill out the screen. At first, I didn’t realize the screen was scrollable, so I didn’t see that about 70% of the options were not visible until one scrolls down. So here is a screen shot of the “Batch Configuration” full expanded in length. (I already clicked “New Batch” and entered some basic dummy data.)
You set up the Filter to some Receive Location that you have created. You will drop the files you want to batch there.
When you click the “Start” button at the top, an orchestration will start running within a few seconds (the length of time might be dependent on your SQL polling interval BatchControlMessageRecvLoc, I’m still a little fuzzy on that).
Where do the batched EDI message go?
Per the walkthrough (link at the top), you create a SendPort with the following filters:
On the first line of the grid, select EDI.DestinationPartyName for Property, == for Operator, the name you selected for the party to send the batch to for Value, and And for Group by.
On the second line, select EDI.ToBeBatched for Property, == for Operator, and False for Value, and And for Group by.
On the third line, select EDI.BatchName for Property, == for Operator, and the name of the batch for Value. (Batch1 in my screen shot above).
When do the batch file get released (created)?
Back in the “Batch Configuration” (the tall screen above), there is a section called “Release”. You can select four options:
2. Maximum number of transaction sets in group or interchange
3. Maximum number of characterrs in interchange
4. External release trigger.
(You could have different batches, with different release mechanism.)
Use the schedule to release a file every 30 minute, every hour, every day, etc…
Here are samples of the schedule options:
Batch Schedule – Weekly
Batch Schedule – Daily
Batch Schedule – Hourly
I think the other option, the max number of transactions and characters, is fairly self explanatory.
The last option, “External release trigger”, is a little more interesting. It is explained in detail on MDSN here. The concept might be that you want to send a batch of EDI files based on some other factor. Imagine for example that someone clicks a “Send” button on a web page. That web page you would then be the “external trigger” that would cause a batched EDI file to be emitted.
In summary, the “external release trigger” involves creating a ReceiveLocation with the BatchControlMessageRecvPipeline Pipeline, then dropping a file like the one below in that Receive Location. In the above example I made up, the “Send” button from the web page would create this XML and write the file in the BizTalk pickup directory tied to the appropriate Receive Location.
EDI Batching – External Release Trigger XML Message
I hope this is a good summary. I believe that if you have read this, then reading the MSDN articles and walk throughs will make a lot more sense, and you will get your EDI batching up and working faster. Please comment below if you found this helpful, or if you see any corrections or improvements.
So the “Start” button at the top of the Batch Configuration starts the orchestration. What if you want to quiesce your BizTalk system? As usual, just stopping the host instances should to that. If you need to redeploy your application, you would have to stop the orchestration; the question then is whether the last messages in the orchestration all get batched into one final file. What if your trading partner just wanted one file per day? Might be some logic you have to add here. If I find answers, I will update the blog.
Error encountered during parsing. The X12 transaction set with id '0001' contained in functional group with id '850009661', in interchange with id '850009661', with sender id '9333516114 ', receiver id '956837686 ' is being suspended with following errors:
Error: 1 (Miscellaneous error)
4: Number of included segments do not match
Below is an the last few lines of an EDI file in NotePad++. You can do replace ~ with ~\r\n to format to make it more readable (be sure and check the “Search Mode” to “Extended”.
The number after “SE” should be one less than the line it is on. In this case, I removed an MTX segment to test, but didn’t change segment count in the “SE” segment. So if you add or remove segments, you have to update this counter.
After installing BizTalk, this file can be found here: “c:\Program Files (x86)\Microsoft BizTalk Server 2010\SDK\EDI Interface Developer Tutorial\ProcessEDI_TestLocations\SamplePO.txt”. It is part of the SDK (Software Developers Kit), which includes common usage scenarios of BizTalk.
A list of the common EDI X12 message can be found here on Wikipedia.
I did the following Google search beg03 x12 401, which resulted in several companies providing public PDF descriptions describing their requirements for sending or receiving 850 documents. For example, this 850 document was very useful in describing how to break down the above 850 message.
In the above sample document, the asterisk “*” is used as the primary separator character. Each line begins with a two or three character prefix (for example, ISA, GS, ST, BEG, etc… called a Segment Identifier (thus each line is a “Segment”). One advantage of XML over EDI is that every data value is identified by an name (either an element or attribute). When looking at an EDI document, you have to have the “decoder” ring or past experience to interpret the document by just looking at it.
Here’s a short clip from the above PDF document. That document is an “Implementation Guide” which gives the specifics about how two companies should implement the more general standard.
Sample EDI 850 Implementation Guide
For example, in the above you see “PO1*2*25*EA*35.68**CB*1093-4927-001”.
Each data value is given a name based on the Segment ID and then followed by a two digit position. So for example, the first field of interest to us is PO102, which is the “Quantity Ordered” which has a value of 25. PO103 is the “Unit or Basis for Measurement Code”. Looking at the EDI values, we see “BX”, “EA”, and “PC” on different P01 segments, with specifically the value of “EA” on the segment we are looking at. It’s not shown in the above clip, but you might guess that is PO104 is the price (value of 35.68), because it has two decimal places.
When you see several asterisks side by side, that means that a field value in between has been omitted.So PO105 is missing or null, and PO106 has the value of “CB” for the “Product/Service ID Qualifier”. Note that in the above PDF, that specific company accepts three values MF, MG, VP, so obviously our document from the BizTalk sample library was not specific to the rules of the specific company providing that PDF.
BizTalk internally treats all messages as XML, allowing them to be mapped with the BizTalk mapper (which is actually XSLT under the cover). It is the BizTalk Adapter specified in the Receive Pipeline that converts the EDI data to XML data. Below is an example of a BizTalk map in the SDK, showing the mapping of the P01 segment to an internal XML format with more meaningful names: