Missing endpoints (in case it becomes unstable)

[juztas]

The following intent [1] becomes unstable at some point (due to ESnet lost information in the model). In case RTMon ran once it was stable, it will prepare correct diagrams. If it run once it is unstable, diagram will be missing ESnet [2]. Orchestrator returns to us the following information (once unstable) [3] - which is missing all ESnet endpoints. Looking at the Viz on Orchestrator [4] - it has all information.

1. How we can ensure RTMon always get latest manifest (even if there are unstable items?) Is there a way for RTMon to get same as Orchestrator at [4]. RTMon uses the following query to get info about intent and connections https://github.com/esnet/sense-rtmon/blob/main/autogole-api/src/python/RTMonLibs/SenseAPI.py#L49-L76
2. discover_service_instances_get does not provide stable/unstable status [5] - It would be nice to have it (and show in Grafana dashboard (either stable/unstable).

@xi-yang @sunami09

[1]

[2]

[3]

Thu, 05 Dec 2024 12:32:55.566 - RTMonLibs.LogLib - INFO - Here is manifest for the following instance:
Thu, 05 Dec 2024 12:32:55.566 - RTMonLibs.LogLib - INFO - {'Ports': [{'IPv4': '?port_ipv4?',
            'IPv6': '?port_ipv6?',
            'Mac': '8c:04:ba:e9:0e:a9',
            'Name': 'Port-channel 102',
            'Node': 'T2_US_Caltech:dellos10_s0',
            'Peer': 'urn:ogf:network:sc-test.cenic.net:2020:aristaeos_s0:Port-Channel501',
            'Port': 'urn:ogf:network:tier2.ultralight.org:2024:dellos10_s0:Port-channel_102',
            'Site': 'urn:ogf:network:tier2.ultralight.org:2024',
            'Vlan': '3875'},
           {'IPv4': '?port_ipv4?',
            'IPv6': '?port_ipv6?',
            'Mac': '8c:04:ba:e9:0e:a8',
            'Name': 'Port-channel 101',
            'Node': 'T2_US_Caltech:dellos10_s0',
            'Peer': 'urn:ogf:network:ultralight.org:2013:dellos9_s0:Port-channel_103',
            'Port': 'urn:ogf:network:tier2.ultralight.org:2024:dellos10_s0:Port-channel_101',
            'Site': 'urn:ogf:network:tier2.ultralight.org:2024',
            'Vlan': '3875'},
           {'IPv4': '?port_ipv4?',
            'IPv6': '?port_ipv6?',
            'Mac': '28:e7:1d:3f:53:48',
            'Name': 'Ethernet13/1',
            'Node': 'NRM_CENIC:aristaeos_s0',
            'Peer': 'urn:ogf:network:es.net:2013::losa-cr6:2_1_c4_1:+',
            'Port': 'urn:ogf:network:sc-test.cenic.net:2020:aristaeos_s0:Ethernet13-1',
            'Site': 'urn:ogf:network:sc-test.cenic.net:2020',
            'Vlan': '3875'},
           {'IPv4': '?port_ipv4?',
            'IPv6': '?port_ipv6?',
            'Mac': '28:e7:1d:3f:53:88',
            'Name': 'Port-Channel501',
            'Node': 'NRM_CENIC:aristaeos_s0',
            'Peer': 'urn:ogf:network:tier2.ultralight.org:2024:dellos10_s0:Port-channel_102',
            'Port': 'urn:ogf:network:sc-test.cenic.net:2020:aristaeos_s0:Port-Channel501',
            'Site': 'urn:ogf:network:sc-test.cenic.net:2020',
            'Vlan': '3875'},
           {'IPv4': '?port_ipv4?',
            'IPv6': 'fc00:1798:0:0:0:0:0:2/64',
            'Mac': '?port_mac?',
            'Name': 'Ethernet1/36',
            'Node': 'T1_US_FNAL:cisconx9',
            'Peer': 'urn:ogf:network:es.net:2013::fnalfcc-cr6:lag-2:+',
            'Port': 'urn:ogf:network:fnal.gov:2023:cisconx9:Ethernet1-36',
            'Site': 'urn:ogf:network:fnal.gov:2023',
            'Vlan': '3610'},
           {'IPv4': '?port_ipv4?',
            'IPv6': 'fc00:1798:0:0:0:0:0:1/64',
            'Mac': '4c:76:25:e8:44:c2',
            'Name': 'Port-channel 103',
            'Node': 'T2_US_Caltech_Test:dellos9_s0',
            'Peer': 'urn:ogf:network:tier2.ultralight.org:2024:dellos10_s0:Port-channel_101',
            'Port': 'urn:ogf:network:ultralight.org:2013:dellos9_s0:Port-channel_103',
            'Site': 'urn:ogf:network:ultralight.org:2013',
            'Vlan': '3875'}]}

[4]

[5]

 {'alias': '[SC23] Rucio-DMM-Caltech-FNAL',
 'archived': False,
 'intents': [{'creation_time': '2024-12-02 19:45:21',
              'id': '523b8237-1fb4-46d2-9d85-ab272e4b77ba',
              'json': {'data': {'connections': [{'bandwidth': {'capacity': '1000',
                                                               'qos_class': 'guaranteedCapped'},
                                                 'ip_address_pool': {'name': 'RUCIO-BGP-P2P-Slash64-Pool',
                                                                     'netmask': '/64'},
                                                 'name': 'Connection 1',
                                                 'path_profile': {'exclusion_list': [{'uri': 'urn:ogf:network:stack-fabric:2024:topology'}]},
                                                 'terminals': [{'assign_ip': True,
                                                                'ipv6_prefix_list': '2605:d9c0:2:fff3::/64',
                                                                'uri': 'urn:ogf:network:ultralight.org:2013',
                                                                'vlan_tag': 'any'},
                                                               {'assign_ip': True,
                                                                'ipv6_prefix_list': '2620:6a:0:2843::/64',
                                                                'uri': 'urn:ogf:network:fnal.gov:2023',
                                                                'vlan_tag': '3610-3612'}]}],
                                'type': 'Site-L3 over P2P VLAN'},
                       'options': [],
                       'queries': [],
                       'service': 'dnc',
                       'service_instance_uuid': '555da2e2-0c40-4e6d-b819-262515a05571',
                       'service_profile_uuid': '2d450838-96db-4b98-870c-3552a7a99c79'},
              'provisioned': True,
              'serviceDeltaUUID': 'f5a8ac97-520c-43d4-b0d7-f7c657149333',
              'serviceInstanceUUID': '555da2e2-0c40-4e6d-b819-262515a05571'}],
 'lastState': 'COMMITTED',
 'owner': '[email protected]',
 'profileUUID': '2d450838-96db-4b98-870c-3552a7a99c79',
 'referenceUUID': '555da2e2-0c40-4e6d-b819-262515a05571',
 'state': 'CREATE - READY',
 'timestamp': '2024/12/02 19:45:20'}

[juztas]

Instance 555da2e2-0c40-4e6d-b819-262515a05571 left on sense-o-dev (https://sense-o-dev.es.net:8443/StackV-web/portal/details/555da2e2-0c40-4e6d-b819-262515a05571)

[juztas]

What is strange, is a new submission. Submitted, got create-ready (unstable) and sense-o manifest query returned ESnet part.
. So it only happens if at some time in future RM looses information