Troubleshooting the deployment

Helm Installation and Upgrade Issues

Failed Upgrade #1

It might happen that the helm installation or upgrade fails with the following error:

Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

This error occurs when a previously triggered upgrade of the deployment is still in progress. This can be the case if a pipeline is run multiple times within a short period of time or if the previous deployment attempt was not / only partially successful.

The issue can be resolved by resetting the helm release to a previous state. Use the following helm commands:

# get your release name
helm ls -a

# get the history of past releases and look up the last working version
helm history <releasename>

# roll back to the last working release
helm rollback <releasename> <releasenumber>

Failed Upgrade #2

You might experience the following error message:

Error: UPGRADE FAILED: timed out waiting for the condition

Verify the configuration of the deployed cadenza. Check the Kubernetes logs for any hints on why cadenza didn’t start up.

Failed Startup

A common error is that the Cadenza Database Migration Tool was not deployed or executed. See Using the Database Migration Tool.

Enabling Debug Logging

Cadenza on Kubernetes supports two approaches for enabling debug logging. For general information about logging subsystems see Logging.

Debug logging produces a significant amount of output and will seriously impact the performance of Cadenza. It should never be used in production for a sustained period.

Option 1: Targeted Subsystem Log Levels

To increase the log level for specific subsystems, set cadenza.logging.subsystemLogLevels in your values:

cadenza:
  logging:
    subsystemLogLevels:
      outgoingHttp: "DEBUG"
      outgoingHttpWire: "DEBUG"
      cluster: "DEBUG"

Option 2: Full Debug Logging

To enable DEBUG logging across all components, provide a custom log4j2 configuration via cadenza.logging.customLog4J2configurationFile. See the Apache Log4j2 configuration documentation for details on the configuration format.

cadenza:
  logging:
    customLog4J2configurationFile: |
      <?xml version="1.0" encoding="UTF-8"?>
      <Configuration strict="true">
        <Appenders>
          <Appender type="Console" name="console">
            <Layout type="PatternLayout" pattern="%d{ISO8601} %-5p [%t] %-c - %m%n" />
          </Appender>
        </Appenders>
        <Loggers>
          <Root level="DEBUG">
            <AppenderRef ref="console"/>
          </Root>
        </Loggers>
      </Configuration>

Only use the Console appender as shown above. This ensures that all log output is written to stdout and becomes part of the container logs, which can then be retrieved with kubectl logs.

The Helm chart creates a ConfigMap from this content and mounts it as the active log4j2 configuration. After the rollout, retrieve the logs with kubectl logs.