Component Monitoring in ES : not all fields available

[arrabito]

Dear all,

I'm using Elastic Search for the WMS History, RMS Monitoring and Component Monitoring.

However, I've noticed that for the Component Monitoring I have only some of the fields that are available, like:
Service Response Time, Agent Cycle duration, etc.
while I don't have other fields like:
Memory Usage, CPU Usage, etc.

Note that I have all these fields filled at some moment in the past months, but it's not the case anymore (see the attached plots

). I also checked the data directly in ES and I just see the fields that are also visible on the Web App Monitoring.
I don't remember if I

changed anything in the Configuration, but I don't think so.

In my configuration I have:

Operations/CTA/EnableActivityMonitoring = True
Systems/Framework/CTA/Services/SystemAdministrator/DynamicMonitoring = True

Note also that the Framework_Monitoring Service is also running, I don't know if it can cause conflict with the Monitoring System.

Could you please advise me on the correct configuration to get all the fields available?

Thank you in advance,

Luisa

[rupozzi]

Hello @arrabito,

We are aware of that issue as we also noticed how some monitoring fields do not appear in the web app.
We have fixed it in Pull Request #5944, in which ComponentMonitoring will actually be replaced by two new monitoring types: ServiceMonitoring and AgentMonitoring.
The pull request will soon be merged and will be available in release 8.0.

Best,
Ruben

[arrabito]

Thank you for your answer.
However, I cannot see those fields also when I look directly at the raw data directly in Elastic Search. So it's not just that they do not appear in the Web app, but they are missing at all. Does it mean that we are not filling correctly Elastic Search?
Thank you.
Luisa

[rupozzi]

Yes you are right, they were not being pushed correctly to ES with the current code.

[arrabito]

@rupozzi ok good to know. I will then wait for release 8.0.
Could you please also confirm that the old system for component monitoring visible from the Activity Monitor is completely replaced by the Monitoring System? If yes, which are the components that should be stopped/uninstalled? I tried to follow the documentation, but I have some doubts. Are the Framework/Monitoring and Framework/ComponentMonitoring services the components in use by the old component monitoring system or they are used in a different context?
Thank you in advance for the explanation.

[rupozzi]

The activity monitor is being removed from the WebApp, yes. We will compile all the necessary info about the modifications and updates in the wiki for the release.
Regarding the Framework system, those are being used in a different context. Also, Framework/Monitoring has been removed, but Framework/ComponentMonitoring is staying.

Best,
Ruben

[arrabito]

OK thanks a lot for the explanations.

I would also have another question. I've noticed that in the values collected in the host field, there are all our DIRAC servers, as expected, but not only. I also found the names of the workers nodes where the jobs ran. Investigating a little bit, I found that this is due to the reports of the JobAgent.

I guess that the Monitoring of the JobAgent should be desactivated, right?

Even if we have the ComponentMonitoringSystem globally activated through:

Operations/CTA/EnableActivityMonitoring = True

Can I desactivate it for the JobAgent by setting:

EnableActivityMonitoring = False
in the configuration of the Agent?

Thanks

[rupozzi]

I apologize for the late answer @arrabito.
However, it is not possible to desactivate the ComponentMonitoring for a single agent. The enableActivityMonitoring flag is only global.
Hope this helped.

Best,
Ruben

[arrabito]

Hi @rupozzi
Thank you for your answer. In this case I'm very interested to know how for instance LHCb does to not get hosts other than the DIRAC servers when displaying some metrics grouped by 'Host'.

Indeed if I display some metrics by "Host" I get plots of this type:

which is not really what we want right?

Thanks

[rupozzi]

To be honest, I am not sure myself about this and I would not rely too much on what is being shown. ComponentMonitoring was implemented quite a while ago and hasn't really been used by LHCb since it has never been working properly (afaik it is not even enabled for LHCb).
However, I hope the new implementation of Agent and ServiceMonitoring may solve all of these issues in v8.

Best,
Ruben

[arrabito]

@rupozzi ok good to know... I thought it was in use in LHCb because I've seen some plots in some past presentations.

Anyway, I will wait for v8 to test it again.

Thanks.

[arrabito]

Hi @rupozzi
I would also have another question. Since long time ago, I've also tried to enable the use of ES to store JobParameters, but I'm missing something since I did not get any Job Parameters stored in ES.

The only thing I've done is to set the following option:

/Operations//useESForJobParametersFlag = True

but it doesn't seem to be enough.
Is there anything else that should be done?
Maybe create some index in ES?

Thank you.

[fstagni]

That's a different one. The flag to use is Operations/<>/Services/JobMonitoring/useESForJobParametersFlag (see https://dirac.readthedocs.io/en/latest/AdministratorGuide/Systems/WorkloadManagement/architecture.html?highlight=useESForJobParametersFlag#databases)

[arrabito]

OK Thank you I will try this one.

[arrabito]

Hi @fstagni I've tried the option you suggested for job parameters but I don't see any new type of index created in ES neither any new content in the existing wmshistory indexes in ES.

I thought that indexes for job parameters would have been created automatically if they were not existing.

Could you please tell me if this is the case or if some action is needed on ES beforehand?

Also which is the index pattern that should contain job parameters?

Currently we have:

wmshistory
componentmonitoring
rmsmonitoring

Thank you.

[fstagni]

A new index would be created, not connected with any of the existing ones.

I honestly do not know what is missing. You should have access to the CS used in the DIRAC certification machine: https://lbcertifdirac70.cern.ch/DIRAC/?theme=Crisp&url_state=1|*DIRAC.ConfigurationManager.classes.ConfigurationManager:,

Please check there and compare it with your configuration.

[arrabito]

Ok thank you.
I don't see for the moment any difference wrt to this, but I will check further.
Can you please tell me which is the index pattern created for job parameters?

[fstagni]

it is "%s_elasticjobparametersdb" % yourSetup.lower()

[arrabito]

OK thank you, so I confirm that they are not created in our case.
I will try to investigate further why.