Configuration as a Service (CaaS)

Team: Payment Routing · Status: In production

This case study describes the problem, my contribution, the design decisions, and the production rollout.

For the same story on one page, read the CaaS one-screen brief.

For a structured account of my contribution, read the CaaS STAR account.

Business context

The Payment Routing team owned a control plane microservice. This service stored business configuration for other microservices that processed payments.

The Product and Operations group, called Prod-Ops, managed this configuration through a graphical user interface (GUI).

The existing model used business contracts. A business contract was an internal configuration object. It was not a legal agreement.

A business contract described a payment processing situation. It could contain information about a country, a payment service provider (PSP), or a gateway.

Contracts also contained configuration keys and values. Services used this configuration for different business flows and operational needs.

For example, services used configuration to connect to PSP APIs or identify available payment methods. These were examples of a broader set of uses. Each consumer service had its own configuration requirements.

Existing system

The control plane stored configuration in predefined database structures. Different configuration types used separate tables.

Consumer services retrieved configuration only through Spring Cloud Config (SCC). The administration interface used APIs to manage configuration.

After a configuration change, notifications passed through Amazon Simple Notification Service (SNS) and Amazon Simple Queue Service (SQS).

In the previous integration, services required a restart to load the latest SCC configuration after an administration change. A notification alone did not make the new values active in a running service.

The system also had to support existing business contract data during migration.

Problems

The existing configuration model did not sufficiently control key names, value types, or value formats.

For example, a field that required a number could contain an invalid value. A consumer could then receive configuration that it could not use.

Two failure conditions occurred:

• A service received invalid configuration and could not start.
• A service could not apply updated configuration and continued to use its previous configuration.

The predefined database structures restricted changes to configuration structure. Prod-Ops could not freely adapt the structure to new business requirements through the GUI.

Structural changes required code changes in the control plane. They also required engineering work on database structures, configuration versions, compatibility, and migration.

This process took time and was prone to errors. It limited flexibility and made service teams dependent on control plane changes when their configuration requirements changed.

Prod-Ops needed more flexibility to define and maintain configuration. The engineering team also needed to address the technical debt in the existing design.

Approval and my responsibility

A request for comments (RFC) introduced CaaS as an extension of the existing control plane. Representatives from different teams reviewed and approved this RFC.

Our team then prepared an internal RFC for implementation. We compared alternatives and obtained approval for the selected approach.

I took responsibility for developing the internal RFC and the detailed solution design. My work covered:

• The data model.
• The APIs and processing flows.
• Observability requirements and operational guidance.
• Implementation tasks and tickets.
• The implementation roadmap and preparation for production.

I also planned the rollout phases and agreed on scope trade-offs with Product.

I implemented almost all implementation tickets. Other team members reviewed my changes. One colleague and I developed the client library for service integration.

Storage decision

We compared two storage options: a NoSQL database, such as MongoDB, and the existing PostgreSQL relational database.

The Fintech tribe had no existing experience with MongoDB setup and maintenance. Its introduction could increase costs and operational risk.

We agreed to use PostgreSQL for the first implementation. This decision allowed us to use the existing database technology and operational experience.

CaaS design

CaaS used schemas to define configuration. Each schema specified the structure, field names, value types, and validation rules.

A schema could include reusable schema components. This allowed staff to use the same definition in more than one schema.

Prod-Ops managed schemas through a new web interface. The interface used APIs to create and change schemas and configuration data.

Engineers did not review each schema definition. The GUI and API performed correctness checks.

Prod-Ops could create a configuration instance from a schema. The system supported versions of both schemas and configuration data.

Validation and publication

Validation occurred during data entry. The system checked entered data against the applicable schema.

Publication was a separate, final step. It created a new configuration bundle version and announced that version to consumers.

Consumer services continued to retrieve configuration through SCC. They did not use the administration APIs to retrieve configuration.

A publication notification prompted consumers to retrieve the new version. However, publication did not confirm that every consumer had retrieved it.

If a consumer could not retrieve the update, it could continue to use its previous configuration version.

Phased implementation and rollout

The old configuration data was already stored in the control plane. I planned a gradual rollout with feature flags, starting with one platform in one region or country.

Phase 1: Convert existing data

The new GUI required substantial development time. I therefore planned the first phase around the existing configuration data.

This phase required three steps:

1. Read the existing configuration data.
2. Define suitable schemas for that data.
3. Convert the data into configuration instances in the new model.

The system continued to read the current business contracts. It converted their data into the new model, created new bundle versions, and published them automatically.

Consumer services retrieved configuration from the new model through SCC. This phase allowed us to introduce the new storage and retrieval flow before the GUI was ready.

Phase 2: Integrate the new GUI and APIs

The next phase introduced schema and configuration management through the new GUI and APIs.

I started with one configuration type that had few dependencies on the existing structure. Feature flags controlled its rollout.

We then added more configuration types in stages until all configuration types were supported in the new system.

Phase 3: Migrate consumer services

After the production release, we progressively required service teams to adopt the new configuration process.

Staff used the new GUI to manage configuration. Their services continued to use SCC to retrieve and synchronize configuration.

Scope agreements with Product

Product wanted complex data structures and strict validation, including checks for data types and length limits.

We could not deliver all requested features in the first release. I discussed the constraints with Product and agreed on three stages:

1. Support simple structures with primitive field types, without nested objects.
2. Add support for lists of values in the GUI and APIs.
3. Add support for nested objects.

This sequence reduced the scope of the first release while retaining a path to more complex configuration.

Example: Expression entry

Some configuration fields required Spring Expression Language (SpEL) expressions.

Initially, staff entered an expression as a string. The API validated the expression.

We later improved the GUI so that staff could construct expressions without knowing the SpEL syntax. We also added input components for specific formats, including expressions and maps of keys and values.

Configuration refresh improvement

Some consumer services had many configuration entries. The initial design created a new bundle version for all configuration belonging to a service after any change.

Even a change to one configuration entry required a new bundle and a notification. We accepted this limitation for the initial release.

One colleague and I later developed a client library, or software development kit (SDK), for configuration refresh.

The library retrieved the complete configuration bundle through SCC. It then detected changes locally and supplied the changed configuration to the service components that listened for those changes.

This reduced the configuration changes passed to those components. It did not reduce the bundle retrieved from SCC.

Services that adopted this integration could apply configuration updates without restarting the whole service.

We provided this library to other service teams to simplify adoption.

Main delivery challenges

Divide the work for a safe rollout

I had to divide the implementation into phases that could enter production without disrupting consumer services.

The initial conversion flow separated the new configuration model from delivery of the final GUI. Feature flags and a configuration type with few dependencies limited the initial rollout scope.

Maintain configuration during migration

The migration had to convert existing data safely while services continued to operate.

We kept reading and converting the current business contracts during the transition. This synchronization continued until the new integration was in place.

Protect tasks already in progress

Some consumer services had tasks that were still running when configuration changed.

The refresh design had to apply new configuration to new tasks without changing the configuration used by tasks already in progress.

My colleague and I had to account for Spring’s refresh context behavior when developing the SDK. Safe handling of running tasks was a requirement of this integration.

Delivery timeline

We planned to deliver the solution within one quarter. The final GUI took longer than planned.

The phased approach allowed work on configuration conversion and SCC integration to proceed before the final GUI was available.

Production outcomes

CaaS is in production. It provides schema-based configuration, validation during data entry, reusable schema components, and versioned publication.

During the rollout, we observed these improvements:

• Service teams could introduce new configuration and change existing structures more easily.
• Prod-Ops had better control over operational data. Configuration structures could more closely match the business data they represented.
• Reusable configuration definitions reduced duplication and made configuration easier to manage.
• Validation rejected unexpected data. We observed fewer incidents related to invalid configuration.
• Services that adopted the SDK could apply configuration updates without a full service restart.

These outcomes are qualitative. This case study does not include measured incident reductions, adoption counts, or time savings.