Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AMD Slow Memory Bandwidth Allocation(SMBA) 0.2 #259

Closed
wants to merge 32 commits into from

Conversation

babumoger
Copy link
Contributor

Add new AMD feature X86_FEATURE_SMBA. With this feature, the QOS
enforcement policies can be applied to external slow memory connected
to the host. QOS enforcement is accomplished by assigning a Class Of
Service (COS) to a processor and specifying allocations or limits for
that COS for each resource to be allocated.

Kernel changes are already committed. Here are the commit ids.
5b6fac3fa44baf x86/resctrl: Detect and configure Slow Memory Bandwidth Allocation
a76f65c89f928 x86/resctrl: Include new features in command line options
78335aac6156 x86/cpufeatures: Add Bandwidth Monitoring Event Configuration feature flag
a5b699665580 x86/resctrl: Add a new resource type RDT_RESOURCE_SMBA
f334f723a63cfc x86/cpufeatures: Add Slow Memory Bandwidth Allocation feature flag

Description

This feature is identified by the CPUID Function 8000_0020_EBX_x0.

CPUID Fn8000_0020_EBX_x0 AMD Bandwidth Enforcement Feature Identifiers
(ECX=0)

Bits Field Name Description
2 L3SBE L3 external slow memory bandwidth enforcement

CXL.memory is the only supported "slow" memory device. With the support
of SMBA feature, the hardware enables bandwidth allocation on the slow
memory devices. If there are multiple slow memory devices in the system,
then the throttling logic groups all the slow sources together and
applies the limit on them as a whole.

The presence of the SMBA feature(with CXL.memory) is independent of
whether a slow memory device is present in the system. If there
is no slow memory in the system, then setting an SMBA limit will have no
impact on the performance of the system.

Presence of CXL memory can be identified by numactl command.

$numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
node 0 size: 63678 MB node 0 free: 59542 MB
node 1 cpus:
node 1 size: 16122 MB
node 1 free: 15627 MB
node distances:
node 0 1
0: 10 50
1: 50 10

CPU list for CXL memory will be empty. The cpu-cxl node distance is
greater than cpu-to-cpu distances. Node 1 has the CXL memory in this
case. CXL memory can also be identified using ACPI SRAT table and
memory maps.

Feature description is available in the specification, "AMD64
Technology Platform Quality of Service Extensions, Revision: 1.03
Publication # 56375 Revision: 1.03 Issue Date: February 2022".
Link: https://www.amd.com/en/support/tech-docs/amd64-technology-platform-quality-service-extensions

Affected parts

  • library
  • pqos utility
  • rdtset utility
  • App QoS
  • other: (please specify)

Motivation and Context

With this feature, the QOS enforcement policies can be applied
to the external slow memory connected to the host. QOS enforcement
is accomplished by assigning a Class Of Service (COS) to a processor
and specifying allocations or limits for that COS for each resource
to be allocated.

How Has This Been Tested?

Tested on AMD machine and sanity tested on Intel machine to verify for regression.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.

Adds the new AMD feature X86_FEATURE_SMBA. With this feature, the QOS
enforcement policies can be applied to external slow memory connected
to the host. QOS enforcement is accomplished by assigning a Class Of
Service (COS) to a processor and specifying allocations or limits for
that COS for each resource to be allocated.

This feature is identified by the CPUID Function 8000_0020_EBX_x0.

CPUID Fn8000_0020_EBX_x0 AMD Bandwidth Enforcement Feature Identifiers (ECX=0)
Bits    Field Name      Description
2       L3SBE           L3 external slow memory bandwidth enforcement

Feature descriptions are available in the specification,
"AMD64 Technology Platform Quality of Service Extensions, Revision: 1.03
Publication # 56375 Revision: 1.03 Issue Date: February 2022".

Link: https://www.amd.com/en/support/tech-docs/amd64-technology-platform-quality-service-extensions
Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537

Signed-off-by: Babu Moger <[email protected]>
This feature is identified by the CPUID Function 8000_0020_EBX_x0.

CPUID Fn8000_0020_EBX_x0 AMD Bandwidth Enforcement Feature Identifiers (ECX=0)
Bits    Field Name      Description
2       L3SBE           L3 external slow memory bandwidth enforcement

Discover the feature and its details using the CPUID Function 8000_0020.

Signed-off-by: Babu Moger <[email protected]>
The sysfs interface will have a new directory "/sys/fs/resctrl/info/SMBA/"
if the feature SMBA is supported. Read the information from the directory.

Signed-off-by: Babu Moger <[email protected]>
SMBA allocation is achieved by writing to the MSRs called
L3QOS_SLOWBW_CONTROL_n where "n" is the corresponding COS number.
These MSRs begin at C000_0280h.

Signed-off-by: Babu Moger <[email protected]>
The pqos_cpu_get_one_by_smba_id retrieves core information from
cpu info structure for smba_id.

Also initialize the info->smba_id to l3_id. Resource ids are same across
all the resources for AMD systems.

These functions will be used in API functions smba_get and smba_set.

Signed-off-by: Babu Moger <[email protected]>
The function pqos_smba_get_cos_num retrieves the total number of slow
memory B/W allocation classes of service from cap structure.

These functions will be used in API functions smba_get and smba_set.

Signed-off-by: Babu Moger <[email protected]>
Introduce following wrapper APIs. Specific implementation of these
APIs will be introduced later.
smba_get : Gets the current SMBA schemata value for a given SMBA
           class of service.
smba_set : Sets the SMBA schemata value to a given SMBA class of service.

Signed-off-by: Babu Moger <[email protected]>
Reads the current SMBA allocation for the specific COS using the
MSR method.

Signed-off-by: Babu Moger <[email protected]>
The function hw_smba_set_amd writes the given SMBA allocation value to
the specific class of service using the MSR method.

Signed-off-by: Babu Moger <[email protected]>
Add the SMBA related data structure in resctrl_schemata structure.

Signed-off-by: Babu Moger <[email protected]>
The function pqos_cpu_get_smba_ids retrieves the smba id's from cpu info
structure. These are support functions which will be used to implement
os_smba_get and os_smba_set routines.

Signed-off-by: Babu Moger <[email protected]>
The function resctrl_schemata_smba_get reads the SMBA class of service
information for a given smba_id. These are support functions which will
be used to implement os_smba_get and os_smba_set routines.

Signed-off-by: Babu Moger <[email protected]>
Update the function resctrl_schemata_alloc to allocate memory for SMBA
schemata structure if the SMBA feature is supported.

Signed-off-by: Babu Moger <[email protected]>
Update the functions resctrl_schemata_type_get, resctrl_schemata_type_set
and related functions to support the SMBA feature.

Signed-off-by: Babu Moger <[email protected]>
The function os_smba_get_amd reads all the COS(class of service)
information from the sysfs interface.

Signed-off-by: Babu Moger <[email protected]>
Add resctrl_schemata_smba_set and resctrl_schemata_write functions
to support the SMBA feature.

Signed-off-by: Babu Moger <[email protected]>
The function os_smba_set_amd updates the schemata value of a given class
of service (COS), using the sysfs interface.

Signed-off-by: Babu Moger <[email protected]>
Introduce the function set_smba_cos to apply the class of service
allocations. This will be called from alloc_apply if SMBA feature is
supported.

Signed-off-by: Babu Moger <[email protected]>
Users can now set SMBA feature setting using the pqos tool.
E.g.
"pqos -e smba:1=64"

This means that COS1 on all sockets, can utilize up to 64 units of
available slow memory bandwidth.

Signed-off-by: Babu Moger <[email protected]>
Introduce cap_print_features_smba to print SMBA capabilities.

Signed-off-by: Babu Moger <[email protected]>
Update print_per_socket_config to print the SMBA capabilities for each socket.

Signed-off-by: Babu Moger <[email protected]>
Introduce _pqos_cap_smba_change to reset SMBA settings when the user
resets the allocations.

Signed-off-by: Babu Moger <[email protected]>
Updates the following functions to reset the SMBA schemata to default value.
hw_alloc_reset
hw_alloc_reset_cos
os_alloc_reset
pqos_alloc_reset
os_alloc_reset_schematas
os_alloc_reset_light
os_alloc_reset_full
resctrl_schemata_reset

Signed-off-by: Babu Moger <[email protected]>
Update the rdt structure rdt_cfg and rdt_config to include smba fields.

Signed-off-by: Babu Moger <[email protected]>
Introduce SMBA feature string in rdtset command interface.

E.g.
rdtset -t 'smba=128;cpu=8-15' -c 8-15 ls

The above command creates a the class of service with
cores 8-15 and SMBA settings to 128 and then runs the command "ls".

Signed-off-by: Babu Moger <[email protected]>
Update alloc_init to detect SMBA feature and alloc_fini for cleanup.

Signed-off-by: Babu Moger <[email protected]>
Updates the function cfg_configure_cos to accept the new feature SMBA.

Signed-off-by: Babu Moger <[email protected]>
Update the function cfg_set_pids to accept the new parameter for SMBA feature.

Signed-off-by: Babu Moger <[email protected]>
Update the function cfg_set_cores_os to accept the new parameter
to support SMBA feature.

Signed-off-by: Babu Moger <[email protected]>
Update the function cfg_set_cores_msr to accept the new parameter
for SMBA feature.

Signed-off-by: Babu Moger <[email protected]>
@babumoger
Copy link
Contributor Author

Opened a new pull request.
#260
This can be closed.

@babumoger babumoger closed this Jan 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant