Administrator's Guide

DevOps Configuration

Defining the DevOps Project Name

Before starting the configuration, it is necessary to define a name for the DevOps project. This name will be used in the subsequent instructions, so it should be remembered.

Rules for forming the DevOps project name:

• The name should consist only of English alphabet characters, be in lowercase, not contain any special characters (including "-", "_"), or spaces. The use of numbers is allowed if they are not the first characters of the name.
• The name should either indicate the product name (not the module) or the bank name for which the project is being created.

The following names are prohibited for the project: dev, usr, release, rc.

Creating a Reference Database for the Project

Before creating the project, it is necessary to prepare a reference database for the project. To do this, determine the list of existing database schemas that will be used in the project.

List of modules and their databases:

1. QBPD - Business Process Designer: qbpmdesigner
2. QC - Cockpit: qbpmcockpit
3. QHT - Human Task: dqhumantask
4. QBPET - Q.BPM Player: qbpetqbpmplayer
5. QAL - Application List: qbpmapplist

Generating a List of Images for Installation on the Stand

The application installation tool supports two options for installing Docker images on the stand:

• Installing an image by link (on the base image);
• Installing an image from the corresponding project branch in the image registry.

The most appropriate setup method for a production stand involves the following:

• Install all adjacent modules (system environment and other adjacent modules not developed by the team) from a ready-made image;
• Install modules developed by the team from a separate branch of the image registry.

To support this installation option, do the following:

• In config.yml of the project branch in devops_configs (opens in a new tab), add/change the following parameter: module_list_from_file: "combo";
• Add a file module-list.yml in the same folder as the config.yml file, containing the used modules.

Images in the project branch of the image registry are automatically created during the initial DevOps setup in the respective module.

On the devops VM, in the project folder, place an sh file to retrieve images from the registry.

Creating a Namespace

For each user who needs a personal stand, it is necessary to create a task for creating/recreating a personal stand.

To create a namespace, provide the following information:

• devopsProject – the project branch in the devops_configs repository;
• stage - the stage in the project branch in the devops_configs repository. By default, it is dev;
• registryStage - the stage in the Docker image registry. For regular stands, it is usually equal to dev, and for personal ones, it is the user's login for whom the stand is created;
• namespace - the name of the created namespace in Q.RunneR;
• team – the name of the production team responsible for the namespace;
• devops - EMail of the employee responsible for the namespace.

Creating a Branch in the devops_configs Repository

Request permissions to work in the GIT repository devops_configs; and log in.

Navigate to the interface for creating a new branch in the GIT repository devops_configs;. In the opened interface, enter the name of the new branch in the Branch name field, corresponding to the name of the DevOps project for which the configuration is being done (see item 1). If the branch is created for a project based on a microservices or hybrid architecture, enter the value template_micro_pg in the Create from field.

Click the Create branch button.

Creating a Repository and Building a Microservice Based on Q.BPM Player

To create application microservices based on Q.BPM Player, the base image Q.BPM Player must be used. The algorithm for creating a repository and building an application microservice is as follows:

• For each deployed microservice based on Q.BPM Player (hereinafter module), a separate repository must be created in GitLab.

  The repository structure for the microservice should be as follows: Module Directory

  • Project (main)
    • Resources (resource)
      • Schemas (schemas)
        • Schema – .json file
      • Process – .bpmn file
  • Docker
    • Dockerfile-database.txt
    • Dockerfile-service.txt
    • properties.properties
  • pom.xml

• The module repository should have a project directory "main" where there is no java code, a "Docker" directory, and a pom.xml file.

• In the "main" directory, there should be a "resource" directory containing only resources (process + schema (object)). To store schemas, a "schemas" directory should be designated in the "resource" directory. Storing schemas (objects) in a separate directory is necessary to ensure the unambiguous identification of a business object required to run the module. Thus, the dynamic attribute library uniquely identifies the schema required to start the module. In special cases, the service can start without schemas.

• The "Docker" directory should contain Dockerfile-database.txt and Dockerfile-service.txt - instructions for building the Q.BPM Player Docker image. The Dockerfile-database.txt and Dockerfile-service.txt files should specify the location of the repository where the Q.BPM Player image of a certain version is stored, the location of the file.jar, and the startup parameters.

• Example GIT QHT/qhtbpm (opens in a new tab);

• Next, the Jenkins configuration is identical to the MSA setup, and the release process is set up exactly the same way.

Updating the Q.BPM Player Base Image

To update the base image of the player, follow these steps:

1. Go to the player repository in the docker directory.
2. In the Dockerfile-database.txt and Dockerfile-service.txt files, specify the current version of the image from registry-new.diasoft.ru/release.

3. Commit the changes. The module will start building on the updated image.
4. If the release version is specified in docker, just restart the build.
5. Update the module on the stand.

Installing an Application Service Based on the Q.BPM Player Image

In the devops repository, add the configuration file for the deployed (created) MS in the namesService.property format.

The configuration file describes standard parameters related to the Digital_Q platform, for example: AUTH_SERVER_URL, AUTH_PROVIDER, AUTH_SERVER_CONTEXT, DATASOURCE_URL, KAFKA_URL etc.

Parameters related to Q.BPM Player:

• QBPM_PROCESS_INPUT_TOPICS - topics for subscription separated by semicolons
• QBPM_PROCESS_OUTPUT_TOPICS - separated by semicolons, topics to which sending is allowed (Example: QBPM_PROCESS_OUTPUT_TOPICS = dsTaskCreateEvent;dsTaskCancelEvent;dsTaskActionEvent)
• QBPM_MODE - player mode dev/prod/run
• QBPM_DLQ_TOPICS - creating topics with dlq postfix (true - creation enabled; false - creation disabled)
• QBPM_LIMIT_DEFINITION_VERSIONS_ENABLED - auto-deletion of business processes - default false (true - auto-deletion is enabled; false - auto-deletion is disabled)
• QBPM_LIMIT_DEFINITION_VERSIONS_COUNT - number of saved processes (default 10)

Player Modes: dev/prod/run

Division of diagrams into areas/namespaces/profiles has been implemented in two operating modes (each mode has a corresponding tenantId parameter for diagrams):

1. DEV - all diagram versions are marked with a special label. An active job is activated, which "cleans up" active processes - terminates (having the status "Externally Completed") all active process instances if they have been running for more than 12 hours. Also, one day after the process instance has been executed, its history is cleared - entries in the qbpetqbpmplayer service table are deleted: act_hi_*
2. PROD - the default mode, the platform works as before. Diagrams are published, complete history is stored, etc.
3. RUN - all versions of diagrams are marked with a special label. A job is activated, which “cleans” active processes - terminates (will have the “Externally Completed” status) all active process instances if they have been running more than 5 minutes. The history is cleared - records in the qbpetqbpmplayer service table are deleted: act_hi_*

At the same time, Swagger controllers are also divided by profiles and do not intersect.

The player mode, QBPM_MODE configuration setting, is set manually via Kubernetes configs. This enhancement has several advantages and solves several problems:

1. Conducting demonstrations and important presentations in PROD mode. Eliminates the need to demonstrate multiple versions used for debugging (clutter).
2. In DEV mode, versions can be deleted as much as needed. PROD mode is for using the final, working, stable version of the diagram (without the ability to delete diagram versions).
3. Reduces memory consumption by the service.

Additional features include the implementation of test diagram instance launch functionality. The Play button opens a modal window where input and output parameters are specified, after which the diagram instance can be launched. This mode allows debugging small "pieces" of a large diagram that are divided into subsets of diagrams. Thus, it allows finding and fixing all defects during design. During such a test instance launch, all active diagram instances will be terminated if they have been running for more than 5 minutes. Also, one day after the process instance has been executed, its history is cleared - entries in the qbpetqbpmplayer service table are deleted: act_hi_*.

To perform a test launch of a process instance, fill in the "Test Value" column in the input parameters in the modal window.

Storing process execution history

To save memory in the database, you can configure clearing the history of executed processes. Number of days of storage of the execution history of processes registered in act_re_procdef.history_ttl_

You can set the value history_ttl_ - History Time To Live in several ways:

1. You can completely disable history. In the Business Process Designer, in the process properties, enable the Do not store process execution history option. Pay attention to each process separately.
2. Set the number of days to store history. In the Business Process Designer, in the process properties, set the value for the History Time To Live parameter. Configured separately for each process.
3. Set the Q.BPM Player operating mode via the QBPM_MODE parameter
4. Add the QBPM_HISTORY_TIME_TO_LIVE parameter to Business Process Monitoring (qbpmcockpit) and specify its value in days. (Example QBPM_HISTORY_TIME_TO_LIVE: P1D)

Logging Capabilities and Levels

logging:
    config: classpath:log4j2.xml
level:
    root: ${LOGGING_ROOT_LEVEL:INFO} # Logging everything
    ru.diasoft.micro: ${LOGGING_DIASOFT_LEVEL:${LOGGING_ROOT_LEVEL:INFO}} # Logging Diasoft classes
    org.hibernate.type: ${LOGGING_HIBERNATE_LEVEL:INFO} # To log values set TRACE // Logging database queries
    org.apache.kafka.clients: ${LOGGING_KAFKA_LEVEL:INFO} # Logging Kafka client
    org.springframework.kafka.listener: ${LOGGING_KAFKA_LEVEL:INFO} # Logging listeners
    org.camunda.bpm.engine: ${LOGGING_CAMUNDA_LEVEL:INFO} # Logging all Camunda actions
    org.springframework.transaction: ${LOGGING_TRANSACTION_LEVEL:INFO} # Logging transactions

Logging should always be enabled. By default, logging is set to the ERROR level, and the use of all modules is allowed. For the PROD mode, it is recommended to set the level to ERROR and only for the necessary module. For the DEV or RUN mode, it is recommended to set the level to ALL for the "root" module.

Available logging levels:

• INFO - only informative messages and what is logged in the code with the INFO level;
• ERROR - information about all errors is output;
• WARN - information about various warnings;
• ALL - the maximum level that outputs absolutely everything;
• DEBUG - debug mode.

Q.BPM Installation Guide

Setting up the project environment and configuration in the repository

devops_configs (opens in a new tab)

You can configure it similarly to the [omniplatform] (https://gitflex.diasoft.ru/devops/devops_configs/-/tree/omniplatform/devops/stages/dev (opens in a new tab)) branch. You need to create your branch: branch name = namespace name

Configuring services

Examples of general namespace settings (PostgreSQL database):

• config.yml - config.yml (opens in a new tab)
• Overall stand database settings - dbservers.properties (opens in a new tab)
• Gateway settings - mdpgateway.properties (opens in a new tab)
• Rulelist settings for the gateway in the new format (opens in a new tab) - rulelist.yaml (opens in a new tab)

Examples of required settings for modules (PostgreSQL database):

1. Q.BPM UI (Root Application)

   • QBPM UI Settings (Standard Authorization mdpauth)
   Example configuration file content
   • QBPM UI Settings (Keycloak Authorization)
   Example configuration file content

2. QBPD - Business Process Designer

   qbpmdesigner.properties

3. QC - Cockpit

   qbpmcockpit.properties

4. QHT - Human Task backend

   dqhumantask.properties

5. QHT - Human Task

   humantaskui.properties

6. QBPET - Q.BPM Player

   qbpetqbpmplayer.properties

Copying images

Latest release versions of the image:

• registry-new.diasoft.ru/release/qbpmui:24050501
• registry-new.diasoft.ru/release/qbpmdesigner:24042206
• registry-new.diasoft.ru/release/qbpmcockpit:24042501
• registry-new.diasoft.ru/release/dqhumantask:24042708
• registry-new.diasoft.ru/release/humantaskui:24042711
• registry-new.diasoft.ru/release/qbpetqbpmplayer:24042703

License Configuration

After March 10, 2023, license verification is enabled in Q.BPM services.

To obtain a license, you must send a request to «CTL - incoming requests ctl@diasoft.ru»

Modules that must be specified in the license:

• QBPD - Business Process Designer: qbpmdesigner
• QC - Cockpit: qbpmcockpit
• QHT - Human Task: dqhumantask
• QBPET - Q.BPM Player: qbpetqbpmplayer

Setting up in namespace via Kubernetes

1. Open Kubernetes, Config Maps section
2. Find the configuration named share-config-volume
3. Open for editing
4. 
5. Find the license.xml parameter in the data block:
6. 
7. If this parameter is missing, add a parameter called license.xml
8. Add the entire contents of the received license.xml file to the parameter value
9. Save the configuration share-config-volume using the Update button
10. In system-config or in the service configuration, add the LPATH parameter: /share/config/
11. Check the value of the LPATH parameter in the Q.BPM services. If necessary, change the value to the correct one, or disable (delete) the parameter
12. After setting the parameter, restart Q.BPM services

Setting up authentication in Kafka via SSL (Secure Sockets Layer)

Description of parameters

• KAFKA_SECURITY_PROTOCOL - the protocol that the service supports for the broker, the default is "PLAINTEXT" to enable SSL you must set it to "SSL".
• KAFKA_TRUSTSTORE_LOCATION - path to the trusted certificate store, according to The default is empty, the value must begin with the protocol instructions ”file://” after which the full path to the file is reached.
• KAFKA_TRUSTSTORE_PASSWORD - password for the trusted certificate store, empty by default.
• KAFKA_KEYSTORE_LOCATION - path to the private key store, by default empty, rules for handling KAFKA_TRUSTSTORE_LOCATION symbols.
• KAFKA_KEYSTORE_PASSWORD - password for the private key store, empty by default.
• KAFKA_KEY_PASSWORD - password for the private key, empty by default.

Parameters must be added to the service settings file on the stand and to the file that is loaded onto the stand when installing/updating the service (application.properties or application.yaml)

Interaction of Services with Keycloak Authorization

To use VPM services in a Keycloak-enabled environment, follow these steps:

1. Follow the steps according to the instructions (opens in a new tab)

2. Install the latest versions of services:

   • qbpmdesigner
   • qbpetqbpmplayer
   • dqhumantask
   • qbpmcockpit
   • mdpemployee
   • mdpdepartments
   • mdpusers
   • mdproles
   • mdprefs

3. If you need a directory of employees and departments with qwork, configure a connection to the qwork database in the mdpemployee and mdpdepartments platform services in dbsecret

4. In Kubernetes, add parameters to the service configuration files:

   • QOAUTH_MODE
   • RESOURCE_SERVER_ID
   • AUTH_SERVER_PUBLIC_KEY

5. Remove the parameter AUTH_PROVIDER=digitalq.

6. In the system config map, add the parameter KUBERNETES_LEGACY_MOD=true

7. Enable the parameter SECURITY_ENABLED=true in BPM services.

8. Parameters for adding a technical user to the Q.BPM Player:

• KEYCLOAK_AUTH_URL
• KEYCLOAK_CLIENT_ID
• KEYCLOAK_CLIENT_SECRET
• KEYCLOAK_SCOPE
• KEYCLOAK_GRANT_TYPE
• TECH_AUTH_SERVICE_NAME // authorization service name
• TECH_USERNAME // technical user login
• TECH_PASSWORD // technical user password

9. After configuring, restart the services. Example of a configured stand (opens in a new tab)

Final Stage

After the settings have been completed, you need to run the stand re-creation assembly (RecreateNamespace), which was created for your namespace. The stand must rise.

Starting services

Recommended resource values ​​for starting the service:

Check the settings in the Deployment of the service in the resources/limits section. Recommended limits for Q.BPM services:

          resources:
            limits:
              cpu: '2'
              memory: 2Gi

Memory allocation pool options:

JAVA_XMS: 1g
JAVA_XMX: 2g

• JAVA_XMS - initial memory allocation pool for a Java Virtual Machine
• JAVA_XMX - allocation pool for a Java Virtual Machine

How much memory is allocated will be visible when the service starts. If you do not configure the JAVA_XMS and JAVA_XMX parameters, their default values ​​are as follows:

JAVA_XMS: 64m
JAVA_XMX: 512m

The service restarts with the error Killed in the logs:

• Solution: Raise the limits on service deployment

The service restarts with a Java heap space error in the logs:

• Solution: Check the JAVA_XMS and JAVA_XMX settings as described above

Pod ImagePullBackOff status:

Failed to pull image "registry-new.diasoft.ru/dvt/development/dev/qbpmdesigner:24010101": rpc error: code = NotFound desc = failed to pull and unpack image "registry-new.diasoft.ru/dvt/development/dev/qbpmdesigner:24010101": failed to resolve reference "registry-new.diasoft.ru/dvt/development/dev/qbpmdesigner:24010101": registry-new.diasoft.ru/dvt/development/dev/qbpmdesigner:24010101: not found

Solution: Register the current image from the image registry

License:

Error creating bean with name 'LStart': Invocation of init method failed; nested exception is java.lang.Exception: Signature is invalid

Solution: The LPATH parameter is set incorrectly. Check the value of the LPATH parameter. This error can also occur if the license file has an incorrect structure.

Error creating bean with name 'LStart': Invocation of init method failed; nested exception is java.lang.Exception: Expired

Solution: The license has expired. The license needs to be updated.

Error creating bean with name 'LStart': Invocation of init method failed; nested exception is java.lang.Exception: Error reading file

Solution: The license.xml section on the stand is empty or has incorrect content. It is necessary to check its contents at the stand.

If the database flushing is not completed or the flushing is incorrect, an error will appear in the interface:

ResultSet; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: could not extract ResultSet

Solution: Check the database flush log. If necessary, re-flush. A successful flush is indicated by the presence of the message: The DB-manager worked SUCCESSFULLY

Q.BPM Player does not start

The deployment contains definitions with same key '<key value>' (id attribute), this is not allowed

Solution: Open a repository with a bpm player. Check processes in the src/main/resources directory for duplicate keys. Remove repetition. Rebuild bpm player.

The service cannot create topics:

Failed to create topics. error: Creating topics with default partitions/replication factor are only supported in CreateTopicRequest version 4+. The following topics need values for partitions and replicas: [qpbcd-pbc-ext-bussiness-platform-platform-type-reply]

Solution: Add a parameter to the Config Maps service spring.cloud.stream.kafka.binder.replication-factor: '1'

Authorization

Failed to instantiate [javax.servlet.Filter]: Factory method 'springSecurityFilterChain' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mdpAuthJwtDecoder' defined in class path resource [ru/diasoft/micro/mdp/lib/qoauth/QSecurityConfig.class]

Solution: Update mdpauth version