No, BizTalk 2013 (nor prior release of BizTalk) do NOT support IBM WebSphere MQ 8.0. We confirmed this with a ticket to Microsoft.

What is confusing is the following on Microsoft page: http://msdn.microsoft.com/en-us/library/jj248684%28v=bts.80%29.aspx#BKMK_MQSeries

According to current information, supported IBM WebSphere MQ versions in Microsoft BizTalk Server 2013 and 2013 R2 :

    IBM WebSphere MQ 6.0.2.12 and later
    IBM WebSphere MQ 7.0.1.9 and later
    IBM WebSphere MQ 7.1.0.5 and later
    IBM WebSphere MQ 7.5.0.3 and later

Further information: Install MQSeries Prerequisites

“And later” apparently means only in the same 7.x series. Apparently WebSphere 8.0 added 64-bit support, and BizTalk has not yet been updated to use it. BizTalk will try to work, but apparently it has an issue not being able to load some IBM WebSphere .DLL that was moved to a different folder path.

If you see 8007005, it may mean “access denied”, but it can also mean that it cannot find a .dll, such as “MQM.dll”.

Your environment variables for the “PATH” must include the directory that contains “MQM.dll”. Use must include the regular directory, and the one that ends with bin64 or bin32 (depending on which mode you are running in). Your 32 or 64 bit mode can be seen “MQ_ENV_MODE”.

In theory, the MQ install itself should update the Path statement for you, but we’ve two cases where it was not updated.

Example: D:\Program Files (x86)\IBM\WebSphere MQ\java\lib;D:\Program Files (x86)\IBM\WebSphere MQ\java\lib64;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;D:\Program Files (x86)\IBM\WebSphere MQ\bin64;D:\Program Files (x86)\IBM\WebSphere MQ\bin;D:\Program Files (x86)\IBM\WebSphere MQ\tools\c\samples\bin;D:\Program Files (x86)\Microsoft BizTalk Server 2010\;D:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\VSShell\Common7\IDE\;D:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\;D:\Program Files\Microsoft SQL Server\100\Tools\Binn\;D:\Program Files (x86)\Microsoft SQL Server\100\DTS\Binn\

Other error messages which may lead to the above:
Retrieving the COM class factory for remote component with
CLSID {86E96D72-0011-4B28-B1AC-BF52AB47F1B4} from machine 10.1.140.197 failed due to the Following error: 80070005
10.1.140.197.

NOTE: One way for testing to see if a BizTalk Server “more or less” talks to an MQ server, is by going into BTAdmin Console, bringing up the configuration of the MQ port, and clicking the down arrow to see if the list of QueueNames is properly enumerated (or whether an error is thrown.)

Error received in Windows Application EventLog:

Error encountered on QueueManager.open Queue Manager name = QTAAFrank Reason code = 2058.

MQ Queue Manager names are case sensitive. In our case, it should have been QTAAFRANK instead of QTAAFrank.
You could get this error by specifying any non-existent, misspelled, or name that is not properly matching based on being case-sensitive.