Simplifying AWS Parameter Store structure (use case)

Parameters in the AWS Parameter Store can become a big hairball of complexity. For the sake of example, imagine we work for Tesla in Europe and we use Stripe for payments, with a bunch of Stripe accounts. We started working on an internal invoicing app. We checkout the AWS Parameter Store and we find this:

/TESLA_BE_BRUSSELS/stripe.api.key
/TESLA_BE_BRUSSELS/stripe.webhook.url
/TESLA_BE_FLANDERS/stripe.api.key
/TESLA_BE_FLANDERS/stripe.webhook.url
/TESLA_BE_WALLONIA/stripe.api.key
/TESLA_BE_WALLONIA/stripe.webhook.url
/INVOICING-APP/stripe.api.key.be
/INVOICING-APP/stripe.api.key.nl
/STRIPE/api.key.be
/STRIPE/api.key.nl
/SPACEX_BRUSSELS/stripe.api.key
/SPACEX_BRUSSELS/stripe.webhook.url
/SPACEX_FLANDERS/stripe.api.key
/SPACEX_FLANDERS/stripe.webhook.url
/SPACEX_NL/stripe.api.key
/SPACEX_NL/stripe.webhook.url
/SPACEX_WALLONIA/stripe.api.key
/SPACEX_WALLONIA/stripe.webhook.url

So that can be hard to get your head around: Why are there parameters for SpaceX if this is Tesla? Which API key should I use for the app I'm building? Are the keys somehow synchronized? What does the country code BE and NL mean? What for are the Belgium areas? Are those webhooks provided by Stripe or by us?

This hairball can be made robust, self-explanatory & DRY. Here's a way how to do it:

  1. It is hard to understand the names because there are multiple orthogonal concepts intertwined in the names. We need to identify the orthogonal concepts. For example:

    • There are companies belonging to one family: Tesla & SpaceX
    • There are countries: Netherlands & Belgium
    • There are Belgium areas: Flanders & Wallonia
    • There are external services: Stripe
    • There are internal services: Invoicing App
    • There are config types: API key & Webhook URL
  2. We need to understand the business rules & business cruft accumulated in the names. For example:

    • Stripe accounts are per country where Tesla is incorporated.
    • Stripe accounts have subaccounts, which are not related to areas. In fact, they relate to customer segments.
      • In Belgium, we have customer segments Basic and Premium
      • In Netherlands we have one subaccount for all customer segments
    • SpaceX parameters are obsolete & not used
    • API keys are issued by Stripe.
    • Webhooks are issues by the invoicing API.
  3. We need to understand our development practices. For example:

    • Our apps most typically work with all accounts for a given service.
  4. With these findings, we can come up with organization rules. For example:

    • We organize parameters around services & apps.
    • We couple API keys & webhook with a service/app that issued the key/webhook. Otherwise, we risk proliferation of parameters.
    • We organize API keys around accounts, where each account is identified by an account ID. For an account ID, we use standard codes where possible, such as ISO 2-letter country codes
  5. Having the rules, we draft the shape of the parameter. For example:

    #content of /services/Stripe/accounts, in YAML format
    NL:
      description: Tesla incorporated in Netherlands
      subAccounts:
        AllSegments:
          apiKey: "324234234234234"
    BE:
      description: Tesla incorporated in Belgium
      subAccounts:
        BasicSegment:
          apiKey: "324234234234234"
        PremiumSegment:
          apiKey: "32423423423423"
    
  6. We try this out in a pilot: pick a few apps, create a parameter with this setup, and evaluate. If needed, improve the structure. Once validated, start migrating other parameters. For example: migrate a group of parameters every time you update an app that uses a parameter that hasn't been migrated yet.

With such steps, we can boil down the 18 parameters to much less, with consistent naming rules & orthogonal concepts on every hierarchy level1. For example:

/services/Stripe/accounts
/apps/Invoicing/webhooks

  1. If you use AWS Lambda, you can create middleware to resolve service configuration based on the account id

-

If you need help with building the tech products get in touch.