How to assign a key to the created topic

[Sano123456]

Hi
actually all topic created are without any key and this making me some problem when i want consume this data do any db

how can i set the key for each table?

[sbcd90]

Hello @Sano123456 ,
There is a setting called

{topic}.pk.fields

Please refer to Configuration section for details.
Kindly let me know if this doesn't help.

[Sano123456]

Hi @sbcd90
this means that i need to define for each table? it doesn't read it automatically ?
Actually i am using about 500-1000 tabls

[igorcosta]

{topic}.pk.fields - This setting can be used to specify a comma-separated list of primary key fields when {topic}.pk.mode is set to record_key

which means if you specifiy the record_value instead of key should go automatically.

[elakito]

I am wondering why we are not extracting the primary keys into the key part by default. Shouldn't we be at least provide an option to extract the keys? I suppose that was what @Sano123456 asked.

[AndreasKlein]

Hi, as @elakito stated, I think this is about the source (not sink) connector, where afaik there is no option to provide a recipe for building the keys of Kafka topics yet. I am very interested in this feature as well to prevent the need for rekeying in Kafka and would be happy to provide a PR.

My approach would be to have three possible modes:

1. No key mode (the current implementation)
2. Auto key mode (Read and use the current primary key(s) from the db)
3. Manual key builder (Define the fields in a comma separated list)

Generally I am leaning toward 2. being the default, but it might be better to stick with 1. as default behaviour. What are your thoughts?

[pantaoran]

The comments from @sbcd90 and @igorcosta are not helpful because they reference sink connector settings when the question is about the source connector.

Just as a workaround to achieve setting a key: one can use general Kafka Connect SMTs (single message transforms) to achieve this with any connector. See here for details: https://rmoff.net/2020/12/09/twelve-days-of-smt-day-2-valuetokey-and-extractfield/

Note: this will only work if the full key is contained in one field, it cannot concatenate a primary key which uses multiple fields.