Managed Components not part of project in separate directory? (PACMAN-744)

[gojimmypi]

Related area

component directory

Is your feature request related to a problem?

This is similar to #13 and #40 but I would like to be able to manage components in a shared project directory.

Describe the solution you'd like

I'd like to be able to idf.py add-dependency or perhaps have new idf.py install-component into a directory that is not part of a project, but is otherwise shared by possibly multiple projects or examples.

I didn't see anything mentioned in the IDF Component Manager Documentation.

Note I was able to use add-dependency, but needed to create a hacky main directory:

gojimmypi:/mnt/c/homekit/esp32-homekit-demo
$ idf.py add-dependency "wolfssl/wolfssl^5.6.0-stable"
/mnt/c/SysGCC/esp32/esp-idf/v5.1/tools/check_python_dependencies.py:12: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
Executing action: add-dependency
ERROR: Directory "/mnt/c/homekit/esp32-homekit-demo/main" does not exist! Please specify a valid component under /mnt/c/homekit/esp32-homekit-demo or try to use --path
gojimmypi:/mnt/c/homekit/esp32-homekit-demo
$ mkdir main
gojimmypi:/mnt/c/homekit/esp32-homekit-demo
$ idf.py add-dependency "wolfssl/wolfssl^5.6.0-stable"
/mnt/c/SysGCC/esp32/esp-idf/v5.1/tools/check_python_dependencies.py:12: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
Executing action: add-dependency
Created "/mnt/c/homekit/esp32-homekit-demo/main/idf_component.yml"
Successfully added dependency "wolfssl/wolfssl^5.6.0-stable" to component "main"
Done

Any guidance as to Best Practices would also be appreciated.

Describe alternatives you've considered

Install the component to each example.

Install the component to the HomeKit submodule. (see below)

Additional context

This is specifically in reference to @AchimPieters using wolfSSL as a managed component in the esp32-homekit project.

In particular, note there's a shared solution components directory that's used by the examples.

I have somewhat of a workaround as described in wolfSSL/wolfssl#5909 (comment). However it's not a very graceful solution in that I copied the managed component source into a new, non-managed component directory.

I have checked existing list of Feature requests and I have provided enough information.

• I confirm I have checked existing list of Feature requests and I have provided enough information.

[kumekay]

Hello @gojimmypi
Thank you for raising the topic,
The proposal of install-component sounds sensible, but it's very hard to implement because components are not simply downloaded and unpacked. As the very first step the full tree of dependencies is built. For this the context of the project (IDF version, current target) is required to choose the right versions of all dependencies. Even if you add only one dependency it may have intricate web of sub-dependencies.

At the same, what is stopping you from adding the dependency from the component registry to all the examples?

[gojimmypi]

Hi @kumekay -

Thanks for your feedback!

it's very hard to implement because components are not simply downloaded and unpacked

hm, yes, I suspected as much.

what is stopping you from adding the dependency from the component registry to all the examples

Ah, a good question! What I neglected to explain in enough detail here (but explained more in the long thread of wolfSSL/wolfssl#5909), is that in the case of the esp32-homekit example, there's some intricacy of where the wolfSSL settings are defined (i.e. the CMakeLists.txt for the esp32-homekit vs the managed component user_settings.h in the example code).

I'm thinking in the end, the best place to add the managed component in this example is the esp32-homekit submodule, as that's where the crypto.c uses wolfssl. Do you agree? Those examples don't even use wolfSSL directly.

Somewhat related to this topic is the desire / ability to not only use a managed component, but also be able to configure it. For example, today if the wolfSSL component is used as a managed component, I believe the user_settings.h is then locked in place with regards to the .component_hash file. Once the user_settings.h is changed, the hash no longer matches, and there's a resulting idf.py warning / suggestion to move the code to a non-managed component directory. What are your thoughts on this topic? Do you see managed components as only a getting started capability?

[igrr]

Regarding user_settings.h: in IDF we generally follow the approach that users should not need to modify source or header files of libraries they didn't write. Where compile-time configuration is necessary, it is achieved using Kconfig.

It is also possible to let the wolfssl component find user_settings.h in another location — for example, in the build directory. The user's project can then use CMake configure_file command to generate the user_settings.h file at build time.

However, I would generally suggest creating a Kconfig file for configuration options. Here's an example how this is done with MbedTLS:

• https://github.com/espressif/esp-idf/blob/master/components/mbedtls/port/include/mbedtls/esp_config.h — the equivalent of user_settings.h, part of the component, not intended to be modified.
• Kconfig options are defined here: https://github.com/espressif/esp-idf/blob/master/components/mbedtls/Kconfig

[gojimmypi]

We were able to resolve the ESP32 HomeKit issues (it's really quite cool, and I created a little blog on the topic).

Given the comment from @kumekay regarding the complexity of implementing install-component, I'll close this issue.

I plan to follow @igrr 's suggestion and use the kconfig for wolfSSL user_settings.h configuration. I'll look into if I can use the file from another directory outside of the component structure.

Thank you.