Define how the connector is deprecating certain fields, while others remain required for backwards compatibility

[raulb]

For context, this is a proposed solution in order to fix this issue: #1558.

Problem statement:

Connector developers may sometimes want to rename a specific field and deprecate the original one. Currently, this process is not automated. Instead, developers document the change in the connector's readme and use the idiomatic Go way to indicate the deprecation in the field description.

In situations where this field is required, we cannot rely solely on the required validation, as only one of the fields may be provided. To illustrate this scenario, I’ll use a recent change in our conduit-posgres-connector:

Context

• Original required field was table, and when adding support for multiple collections, this field was renamed to tables.
• We removed the required validation that’s automatically checked by our conduit-connector-sdk, but instead we perform a custom validation in the connector to ensure that one (and not both) values were provided: conduit-connector-postgres/source/config.go at main · ConduitIO/conduit-connector-postgres.
• We could have moved the required validation to the new tables field, but we wanted to maintain backward compatibility while providing the new collections feature.

Drawback

• Our GUI relied on these required attributes to determine a minimalist layout. This presented only the essential fields upfront, allowing users to access additional parameters as needed.
• By removing the required attribute originally linked to the table parameter, we lost the ability to indicate that tables is now a required field while maintaining backwards compatibility.

Proposed solution

The proposed solution considers the following scenarios.

1. A field is being deprecated and no other field is replacing it.
2. A field is being deprecated and being replaced by another one (field renamed).
3. A field that’s being deprecated could have been required or not.

Adding a new validation and rely on // Deprecated

The proposed solution suggests two approaches to address both scenarios: deprecating and changing the required validation.

1. Deprecating fields

A field will be marked as deprecated automatically by our paramgen in connector-sdk relying on the idiomatic Go way to indicate such thing (// Deprecated).

Example: To continue with our previous example, imagine we’re renaming the Table field that was required to a new field named Tables (for simplicity, I’m not including what to do with the required field here).

Before

type Config struct {
 // Table indicates the table name to read from
 Table string `json:"table" validate:"required"`
 // ...
}

After

We include the // Deprecated notation and remove the required validation from the old field, and we add the new field.

type Config struct {
 // Deprecated: use `tables` instead.
 Table []string `json:"table"`

 // Tables is a List of table names to read from, separated by a comma, e.g.:"table1,table2".
 // Use "*" if you'd like to listen to all tables.
 Tables []string `json:"tables"`

 // ...
}

2. Require dependency

In order to maintain backwards compatibility with the old field and indicate that one of the two fields (old and new one) is required, we’ll add a new validation named require_unless . This will be added to both fields while maintaining compatibility. This validation could be removed and simply use require in the new field once the deprecated field is removed completely.

Before

type Config struct {
 // Table indicates the table name to read from
 Table string `json:"table" validate:"required"`
 // ...
}

After

type Config struct {
 // Deprecated: use `tables` instead.
 Table []string `json:"table" validate:"required_unless=tables"`

 // Tables is a List of table names to read from, separated by a comma, e.g.:"table1,table2".
 // Use "*" if you'd like to listen to all tables.
 Tables []string `json:"tables" validate:"required_unless=table"`

 // ...
}

GUI implementation

Our GUI currently relies on the required validation to show these fields upfront letting users to show additional fields if they need to.

When parsing the connector configuration fields, paramgen will automatically add a deprecated value that will let the GUI distinguish these fields and ignore them altogether. Example:

// transformFields will include types used and whether they're required and deprecated or not.
func transformFields(cFields map[string]plugin.Parameter) map[string]connectorSpecField {
	fields := make(map[string]connectorSpecField)
	for k, v := range cFields {
		csf := connectorSpecField{
			Type:        fieldType(v.Type),
			Required:    fieldRequired(v.Validations),
			Description: v.Description,
			Default:     v.Default,
			Deprecated:  v.Deprecated, // bool
		}
		fields[k] = csf
	}
	return fields
}

Default values

Fields can specify a default value. Default values on deprecated fields will be ignored.