Xml or json - use one format passing to sense-o

[juztas]

As reported by Sunami and seen in the past, it all starts from here: https://github.com/sdn-sense/sense-o-py-client/blob/main/sense/client/workflow_combined_api.py#L674 (wrapping json inside xml). Need to rewrite to use in either xml or json

Traceback (most recent call last):
  File "src/python/RTMon/worker.py", line 271, in <module>
    worker.startwork()
  File "src/python/RTMon/worker.py", line 230, in startwork
    self._startwork()
  File "src/python/RTMon/worker.py", line 253, in _startwork
    self.main()
  File "src/python/RTMon/worker.py", line 217, in main
    self.submit_exe(filename, fout)
  File "src/python/RTMon/worker.py", line 50, in submit_exe
    manifest = self.s_getManifest(instance)
  File "/opt/devrtmon/src/python/RTMonLibs/SenseAPI.py", line 79, in s_getManifest
    response = wApi.manifest_create(dumpJson(template, self.logger))
  File "/usr/local/lib/python3.6/site-packages/sense/common.py", line 21, in wrapper
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sense/client/workflow_combined_api.py", line 680, in manifest_create
    body_xml, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.6/site-packages/sense/common.py", line 21, in wrapper
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sense/client/workflow_combined_api.py", line 735, in instance_si_uuid_manifest_post_with_http_info
    query_params=query_params)
  File "/usr/local/lib/python3.6/site-packages/sense/common.py", line 21, in wrapper
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sense/client/requestwrapper.py", line 145, in request
    f"Returned code {ret.status_code} with error '{error_message}'")
ValueError: Returned code 500 with error 'Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (io.undertow.servlet.spec.ServletInputStreamImpl); line: 1, column: 2]'

[xi-yang]

@juztas Can you elaborate? If this is a bug report, what are the steps / input to reproduce?

[xi-yang]

I see this from the sense-rtmon issue: esnet/sense-rtmon#127 (comment). But it remains unclear what was the input to the API call.

[juztas]

@xi-yang We have discussed this here: https://stackv.slack.com/archives/CFSBT7TFC/p1718389638923429

Issue is on sense-o-py-client. To repeat issue (it happens sometimes):

        template = {"Ports": [{
            "Port": "?terminal?",
            "Name": "?port_name?",
            "Vlan": "?vlan?",
            "Node": "?node_name?",
            "Peer": "?peer?",
            "Site": "?site?",
            "Host": [{
                "Interface": "?host_port_name?",
                "Name": "?host_name?",
                "IPv4": "?ipv4?",
                "IPv6": "?ipv6?",
                "MAC": "?mac?",
                "sparql": "SELECT DISTINCT ?host_port ?ipv4 ?ipv6 ?mac WHERE { ?vlan_port nml:isAlias ?host_vlan_port. ?host_port nml:hasBidirectionalPort ?host_vlan_port. OPTIONAL {?host_vlan_port mrs:hasNetworkAddress  ?ipv4na. ?ipv4na mrs:type \"ipv4-address\". ?ipv4na mrs:value ?ipv4.} OPTIONAL {?host_vlan_port mrs:hasNetworkAddress  ?ipv6na. ?ipv6na mrs:type \"ipv6-address\". ?ipv6na mrs:value ?ipv6.} OPTIONAL {?host_vlan_port mrs:hasNetworkAddress  ?macna. ?macna mrs:type \"mac-address\". ?macna mrs:value ?mac.} FILTER NOT EXISTS {?sw_svc mrs:providesSubnet ?vlan_subnt. ?vlan_subnt nml:hasBidirectionalPort ?host_vlan_port.} }",
                "sparql-ext": "SELECT DISTINCT ?host_name ?host_port_name  WHERE {?host a nml:Node. ?host nml:hasBidirectionalPort ?host_port. OPTIONAL {?host nml:name ?host_name.} OPTIONAL {?host_port mrs:hasNetworkAddress ?na_pn. ?na_pn mrs:type \"sense-rtmon:name\". ?na_pn mrs:value ?host_port_name.} }",
                "required": "false"
            }],
            "sparql": "SELECT DISTINCT ?vlan_port ?vlan WHERE { ?subnet a mrs:SwitchingSubnet. ?subnet nml:hasBidirectionalPort ?vlan_port. ?vlan_port nml:hasLabel ?vlan_l. ?vlan_l nml:value ?vlan. }",
            "sparql-ext": "SELECT DISTINCT ?terminal ?port_name ?node_name ?peer ?site WHERE { {?node a nml:Node. ?node nml:name ?node_name. ?node nml:hasBidirectionalPort ?terminal.  ?terminal nml:hasBidirectionalPort ?vlan_port. OPTIONAL {?terminal mrs:hasNetworkAddress ?na_pn. ?na_pn mrs:type \"sense-rtmon:name\". ?na_pn mrs:value ?port_name.} OPTIONAL {?terminal nml:isAlias ?peer.} OPTIONAL {?site nml:hasNode ?node.} OPTIONAL {?site nml:hasTopology ?sub_site. ?sub_site nml:hasNode ?node.} } UNION { ?site a nml:Topology. ?site nml:name ?node_name. ?site nml:hasBidirectionalPort ?terminal. ?terminal nml:hasBidirectionalPort ?vlan_port. OPTIONAL {?terminal mrs:hasNetworkAddress ?na_pn. ?na_pn mrs:type \"sense-rtmon:name\". ?na_pn mrs:value ?port_name.} OPTIONAL {?terminal nml:isAlias ?peer.}}}",
            "required": "true"
        }]}
        wApi = self.s_getWorkflowApi()
        wApi.si_uuid = instance['referenceUUID']
        response = wApi.manifest_create(dumpJson(template, self.logger))

From Chat, Alberto said:

it is sending request in JSON, but the data is xml: Accept: application/json\r\nConnection: keep-alive\r\nContent-type: application/json\r\

So client - should say it is xml and not json (or we should migrate calls to json format on the client)

[juztas]

I see few issues here:

1. py client should set correct headers (either it is xml or json)
2. What is not clear - why it sometimes works and sometimes not? we need to run this in a full debug mode and compare the failing one with succeeding.

[xi-yang]

Accepting XML content with JSON header is possibly environmental. But the code is indeed flawed.

sense-o-py-client/sense/client/apiclient.py

Lines 38 to 41 in 19ddaa2

	def _setHeaders(self):
	"""Set Headers for API calls"""
	self.config['headers'] = {'Content-type': 'application/json', 'Accept': 'application/json',
	'Authorization': 'Bearer ' + self.token['access_token']}

There should be an option (xml or json with default to json) in the request calls.

[xi-yang]

I patched up the API client with both content-type and accept-type options for request calls.

The manifest POST will use XML content and accept JSON return.

The change is in PyPi now. Try the latest sense-o-api==1.35.

[juztas]

That was still an issue. Fix is here: #37

We need to set headers before issuing post/get/put call.