That will really depend on how Cognos BI works. The main thing to look for is how to make it return a URL like:
Tests from going through the URL in browser gave me this:
http://< canvas server > /< canvas app >/?cam_passport=XXXXX&CAMNamespace=CCCCCC
That is, after passing in a URL with c_cmd value as http://< canvas server > /< canvas app >/#/configuration. And as noted, it strips down the #/configuration part, and just replaces it with ?cam_passport and CAMNamespace parameter/value.
So if there is a simple setting or flag maybe in Cognos BI that allows redirect with the full URL passed, then that should help on this.
Then on using Canvas’ SSO mechanism, the returned response by Cognos BI should have in its HTML response / body, details about the CAMPassport. There should be visible information in there about:
You can check this out by opening the Developer Console of Chrome for example and clicking on the request to Cognos. It will look something like:
From there, it should be able to give you more information as to what the error is (if there is any, such as Forward URL is Invalid, etc.).