First of all, the key metrics of growth:

the key milestones of 2024:

and part of our global wide enterprise users:

For more detailed annual report, please visit our online vercel app, the last but not the least, Happy Chinese New Year to everyone, wish you happy and healthy !

Introduction

Configuration Management and Configuration-as-Code(CaC) are pivotal concepts in modern software development. At Ant Group, as the product development and operations scale, and business portfolio diversifies, the demands for rapid iteration and stringent regulatory compliance present substantial challenges to effective configuration management.

This series of articles is dedicated to exploring the intricacies of configuration management at Ant Group, walking through challenges and solutions in detail, discussing architectures, proposing best practices that have proven effective in production at massive scale, and looking ahead into the future of configuration management.

In this first article, we will examine the specific challenges we encountered over the years, the strategies we devised to address them, and the resulting patterns that have emerged as what we believe to be best practices — Generated Manifest & Immutable Desired State. Through this exploration, we aim to provide valuable insights and practical guidance for navigating the complexities of configuration management in a dynamic and highly regulated environment.

Challenges & solutions

1. Infrastructure Complexity

Infrastructure complexity is the elephant in the room for any modern-day software deployments.

The inherent intricacy of the Kubernetes ecosystem is a reality that developers face from Day 1 and continues to evolve as the product matures. Ant Group runs over 90% of the workload on Kubernetes, which means this complexity is passively falling upon all the developers. This complexity is compounded by the myriad of operational considerations, including the cross-region or multi-cluster setups, resource allocations, networking configurations and such, all of which demand considerable amount of attention.

Applications today also often needs to connect to a variety of non-Kubernetes services, such as managed databases, message queues, and cloud storage, etc. New features are also continously getting added by the infrastructure providers, but this means that developers must commit to continuously learning and adapting to keep up with the pace. This results in massive cognitive loads on the developers.

Abstractions, Models & DSLs

As software engineers, abstraction is a natural response that comes to mind. We introduced an abstraction layer and a dedicated DSL to do so. KCL(Kubernetes Configuration Language) was born. Instead of writing static Kubernetes YAMLs that are both weakly-typed and complicated, we encourage our developers to use a strongly-typed DSL with semantic validation, structs and inheritance, constraint definitions, conditional statements, functions and such.

With abstractions, we have defined a top level model named AppConfiguration. It covers all of the resources an application needs to run. Both Kubernetes and Non-Kubernetes resources are included in the same set of application configurations to ensure uniformity and minimize developer’s burden to understand every little detail of infrastructure. But that creates new problems.

2. Mismatched Rights and Obligations & Platform Orchestrator

The resulting configuration is made of both Kubernetes and non-Kubernetes resources, which means it controls both application workload (where application code runs) itself, and its environment-specific, cloud-specific, dependent infrastructures. One of the infamous side effect with Infrastructure-As-Code is its scary blast radius, especially when some of the infrastructure could be serving more than just a single application.

As an application developer, these infrastructure settings might just be the least they want to worry about. Developers are not familiar with them, yet they will have to be held accountable for the adequacy of their configurations. This is undeniably horrifying experience and afflicts countless confusion and suffering to the application developers.

Take sidecar as an example, the platform could have provided an abstraction for provisioning a sidecar container, but most developers won’t know or even care whether their application should have one. Alternatively, it could even be a security mandate to use one. In these cases, developers are forced upon the accountability to maintain such a sidecar configuration as part of the AppConfiguration model, and subsequently responsible for its entire lifecycle.

Separation of concerns

Our solution to this is separation of concerns. In modern day software architecture, you can’t expect one person to know everything about every corner case of every infrastructure. The launch of an application generally involves a number of systems, platforms and infrastructures, each backed with different teams or organizations. Different parties may carry different intents on how to provision resources, without a single person knowing all of them. This is the very source for the proliferation and complexity in application configuration management. But the need for these diverse infrastructures is real. The need for inputs from multiple roles in an organization is real. So we need a way to break down the application configurations into smaller pieces so each role can only care about what they NEED to care about.

The infrastructure teams may have the knowledge on the infrastructure they manage, but not in the application itself (i.e. how do they want to use the infrastructure). The application team goes the other way around. A gap therefore emerges. The reconciliation needs to happen somewhere in the workflow. In most Platform-As-A-Service pattern, this is when we see developers consistently talking to infrastructure or platform teams and trying to figure out what those configurations mean and what they do.

The problem is, in a lot of the times, this interaction is extremely people-driven. Or in other words, you’ll have to find “THE right person” and expect them to be familiar with the exact context you need. Not only is this a counterproductive and repetitive effort, the resulting knowledge in most of the time cannot be effectively documented or codified for future iterations and success. This is what we consider an ineffective, people-driven collaboration pattern, which we wish to pivot to a protocol-driven one, where no one gets blocked when this key person inevitably takes a two-week vacation in Hawaii.

Platform Engineers & A proliferation of viewpoints

What we also see going on in the industry and with the Platform Engineering evolution is the emergence of “Platform Engineers”, how it’s distinct from the “Application Developers”, and how they will be in charge of managing things that “Application Developers” don’t. In an ideal world, “Application Developers” are in charge of implementing the business logic, while “Platform Engineers” cover the rest, including those that make the business logic run and last. Another observations we made when we practiced separation of concerns is that we often discovered multiple roles underneath this hat of “Platform Engineers”, like compliance, security, network engineers, etc. And in some cases, one can be in multiple roles. Each role will have their own perspective when expressing intents.

What we have decided the key to the problem is this mismatch between rights and obligations, between “What one actually cares about” and “What one should care about”. The key is NOT identifying those fields that change more frequently than others, or to exhaust all the possible abstractions. The key is to provide a mechanism that allows engineers in different roles to each express their share of the intents, both environment-agnostic and environment-specific ones, and a golden path for them to do so.

Hence, the need for a Platform Orchestrator was identified. The Platform Orchestrator is a system designed to capture and “orchestrate” intents from different viewpoints, and connecting them with different infrastructures. It serves as the glue for different intents throughout the software development lifecycle, from deployment to monitoring and operations, ensuring that users’ intentions can seamlessly integrate and flow across different environments and infrastructures.

3. Massive scaling & Dynamic Configuration Management

Again that doesn’t solve all the problems. Scaling, for one, poses significant challenges when we tried to roll out the new paradigm.

Based on the idea of separation of concerns, we can split the configurations into smaller pieces based on roles and “merge” them into a complete one all we want. In reality, applications usually needs to be deployed to multiple clouds, sites, regions, tenants, datacenters, environments, or in some cases, even smaller units of deployment. They may share 90% of the configurations, but the other 10% could be entirely different, namely environment variables, or database sizes, or container images for example. In Ant Group, there are 16 combinations of these different layers that may have different configurations from one another.

From the perspective of configuration management and keeping your configurations DRY (Don’t Repeat Yourself), it would be a terrible idea to copy a set of configurations for each possible deployment targets. Rather, if you think the likes of Helm or Kustomize, what we are proposing here is a similar pattern where the final resource manifests are dynamically rendered from a series of inputs and some rendering logic. The difference being:

1. The manifest will cover non-Kubernetes resources so it would be application-scoped, rather than only Kubernetes-resource-scoped. This manifest will serve as the single source of truth for all the resources needed (and we have a plan for not just resources in the future) to idempotently pull up an application at any time or from the ground up.
2. The rendering logic are written in a General Programming Language rather than a templating language to support rather complex logic. We knew we need this when parts of the resource specifications need to be retrieved dynamically from a centralized location, for example a CMDB. This is actually a pretty common case in our practice.

Generated Manifests — Specs

In comes the Specs. The spec is a system-generated, immutable, declarative representation of the resources involved in a particular deployment. As opposed to the static configurations that are stored in a Git repository, which may or may not be scoped to a given deploy target, a spec is dynamically rendered from the aggregated intents from multiple sources, including those that are target-specific, and those aren’t (e.g. global configs, constraints posed by security, compliance and so on, for example what kind of application may have Internet access). Specs are resource-facing desired states and are always rendered on the spot based on all the relevant inputs.

The Specs are designed to be THE intermediate data layer between configuration code and actual resources. It is designed to be a structured data format that is both machine-friendly (so that we can use the proper libraries to process and actualize them) and human-friendly (so that it provides a readable reference to the resource perspective of an application). In this workflow, the component that processes the resources in the Spec is called a Runtime. Runtimes are in charge of bridging the resource specification to the actual infrastructure API. For Kubernetes resources, its runtime uses client-go to connect to the clusters. For the cloud resources, we are using IAC tools like Terraform/Crossplane and their providers to connect to the cloud control APIs. Runtimes are also extensible.

The rendering logic in this case are produced by “Generators”, which are pieces of code written and distributed in Go. Generators are in charge of converting configuration code written in KCL into resource specifications in the Spec. They are packaged and wrapped inside a GRPC server whose lifecycle are dynamically managed as individual go-plugins.

Once a Spec is generated for a given application deployment, it is considered THE immutable source of truth for its resources. You can think of it like an artifact of the configuration code, similar to a JAR file built from the Java source code. If anything in the configuration is changed, a new Spec shall be re-generated when a new deployment happens. Similar to that of an artifact, we also opt for proper version control for audit purposes and potential rollbacks.

Intents Grid & Deployment-Specific Constraints

Now back to scaling. Remember in the beginning when we talked about how there may be a proliferation of layers of different configurations based on deployment targets, and a proliferation of roles that are individually responsible for a different set of infrastructures, each needs to express their intents on deployments. This forms a so-called “Intents Grid”.

Splitting the grid vertically are the different roles and viewpoints from infrastructures that form the actual runtime environment for an application. Each infrastructure viewpoint may raise certain constraints or have opinionated defaults that may affect the outcome resources. For example, the storage team requires the use of standard-tier S3, or the security team mandates Customer-Managed-Keys for data-at-rest encryption in production environments. Horizontally diving the grid are the physical or logical deployment targets for an application. Together they form a grid that captures the operational intents from all the possible sources into a set of deployment-specific constraints. In the prior S3 example, when application team declares “my application needs 50 Gigs of object storage”, and when this application is deployed to production in us-east-1 region on AWS, the final resource generated will be a standard-tier S3 with 50 Gigs size, CMK enabled in the us-east-1 region.

Note that this may generally be a sparse grid, where not every single intersection represent an existing intent. But by collecting all the intents on this grid, we form a finalized, orchestrated intent that encapsulates the input from all the relevant platform perspectives. Combine it with the application developer’s input, they are fed into the generators to produce the resource-facing manifest, aka. the Spec.

4. The need for an open platform

Up until here, it’s very likely that we will end up with a highly-bespoke system if the platform team holds the key to all the rendering logic. In fact we have unsurprisingly been in this exact situation where the platform team becomes the bottleneck and the system gradually turns into a black box over time to the application teams. The need for new infrastructures is always expanding and it’s evolving fast in a growing organization, which calls for a mechanism to build and sustain an ecosystem upon an open platform, where everyone can come in and define their own rendering logic. This founds familiar isn’t it?

Naturally, the solution to enable Bring-Your-Own-Logic is to make them out-of-tree. So we’ve introduced the concept of a “Kusion Module”. Each is made of two things:

• A user friendly schema (written in DSL) that is consumed directly by the developers
• The rendering logic (written in GPL) that takes all the end-user inputs, mix them with opinionated platform defaults, and produce the resource-facing Spec

We encourage Platform Engineers to produce modules as needed and we are providing the frameworks to do so. A Generator interface requires a Generate() method that takes in a GeneratorRequest and returns a GeneratorResponse with the generated resources, both are types that is defined by the standardized Kusion Module Framework.

Best Practices

To summarize, what we are proposing and ending up with is a pattern that involves the following:

1. Expressing Intents: The end users will express their intents and write configurations in user-friendly DSLs. This includes more than just the application developers, but also those that are setting the platform or environment-wide standards, for example, security engineers, network engineers, or DBAs, each having their own perspectives on what needs to be configured on the application infrastructures.
2. Dynamically Assemble Configurations: These configurations will then be put together dynamically in a “reasonable” way. For example, a security engineer defines the minimum TLS version to be 1.2 when the application is using databases. This gets combined with the application developers’ need for a database (among others) into what we call a Final Intent.
3. Handle Final Intent: The “final intent” then gets feed into the module generators, which will then transform the “final intent” into a resource-facing data format. Generators are written in General Programming Languages and can contain complex logic, including fetching data from other sources like a CMDB.
4. Generate Immutable Specs: This “Generated Manifest” is considered to be THE immutable, declarative desired state to a given application deployment. It serves as the Single Source Of Truth for the application resources. Should anything go wrong in a future release, it can be used to roll back to this specific version of resources that stay exactly the same as it was (There could be new problems, we will raise them as a question at the end).
5. Extensible Runtimes: Between the Spec and the actual resource APIs are Runtimes. They can be considered the “top-level SDKs” that take resources in the Spec as input, connect with the underlying infrastructure provider, and manage the lifecycle of said resources via their current and desired states. In the case of Kubernetes resources, the runtime is built on top of client-go. For Terraform resources, it’s on top of the Terraform CLI. While this pattern works well with public cloud providers, in the case of on-premise infrastructures, this will require a certain level of automation and maturity from the infrastructure service providers that enables resource lifecycle management via robust CRUDL APIs.

Open Discussions & Up Next

In the concluding section, we will pose a question (on closure, no pun intended 😝) instead:

In our previous assumptions, we considered the Spec as the SSOT (single-source-of-truth) for resources. Our expectation for the configuration management workflow is that it should possess the property of closure — that is, the same Configuration Code give to the same version of the Kusion Module Generators should produce the same Spec. Similarly, the same Spec given to the same version of the Runtime and infrastructure should ideally produce the same set of Resources idempotently.

But this property of closure is not guaranteed. Without it, “perfect” rollbacks are hard to achieve. Even with Generators remain the same, they could be retrieving data from external sources. If an external data source, such as a CMDB changes, the generated Spec will also change. Likewise, if the version of the infrastructure API changes (for instance, if a parameter changes), the behavior of rolling back resources with the same Spec is also unpredictable. In this scenario, what can truly be considered the Single Source Of Truth?

In the next article, we will delve into the large-scale dynamic configuration management and its implementation practice at Ant Group, providing a more detailed introduction to the relevant technologies and architecture.

Stay tuned!

With the holiday season approaching, we have wonderful news to share.

We are thrilled to announce that KusionStack has joined the Cloud Native Computing Foundation (CNCF) as a Sandbox Project following a vote by the Technical Oversight Committee (TOC) in Q4 2024. We would like to extend our heartfelt gratitude to all the CNCF staff involved for the invaluable guidance throughout the process, and to our community and the team for the overwhelming support that made this happen.

What is KusionStack? A Brief History

KusionStack was first incubated at Ant Group in 2020 as an Infrastructure-As-Code tool, accompanied by KCL, designed to simplify and standardize cloud-native resource management. Throughout the years, with the concept of Platform Engineering emerging, KusionStack has growed into a loosely-coupled Platform Engineering toolkit, intend to empower platform teams altogether in developing enterprise grade, production-ready, self-service-able Internal Developer Platforms (IDPs).

KusionStack provides a comprehensive set of modular components that teams can use to build every layer of their IDPs. These components are both composable and programmable, offering the flexibility to use them as a complete platform solution or to integrate specific elements into your existing IDPs through extensive APIs. Whether you’re building from the ground up or enhancing what you have, KusionStack offers the tools to simplify the process.

• Kusion — A Platform Orchestrator that “orchestrates” codified intents from multiple parts of an organization, and connects with heterogeneous infrastructures to substantially improve application delivery experience
• Karpor — A data visualization tool for Kubernetes that provides multi-cluster search, insight, and AI-powered data intelligence that enables interactions with clusters via natural language
• Kuperator — A set of Kubernetes controller extensions that provides more complete workload lifecycle management, traffic management, progressive rollouts, and better observability
• More to come in 2025…

Intended scope for KusionStack

Looking forward…

Being an incredible honor and a major milestone as it is, this is a new beginning for the KusionStack community. We have great plans for 2025 and beyond. We believe Platform Engineering is a multi-year journey to fundamentally transform the way software engineering organizations build and manage their technologies.

It will be a journey filled with challenges, but we have every confidence in the world we can navigate through, thanks to having such an amazing community. We are very excited about the opportunities to be part of the force that advances cloud-native technologies.

We wish see you along the way!

The KusionStack team is delighted to announce that Karpor v0.5.0 release is now available! This update brings four major highlights for users:

• Upgraded Multi-Cluster Management Capabilities: Added support to import AWS EKS clusters and cluster certificates based on Tokens, providing a more flexible cluster management experience.
• Enhanced Security: Introduced a new Token-based login mechanism with RBAC(Role-Based Access Control) support, effectively tightening access to better secure the production environments.
• Optimized Installation: Provided over 30 Helm parameters during installation, fully supporting custom image repositories and arm64 architecture, making on-prem and private deployment scenarios more convenient.
• Improved User Experience: Newly supported features such as intelligent YAML folding, cluster health checks and version management, delivering a smoother user experience.

Karpor, as a sub-project of the KusionStack (part of the CNCF Sandbox) ecosystem, is dedicated to building an intelligent Kubernetes data plane. By providing advanced search, insights, and AI capabilities, it helps users gain critical visibility into their Kubernetes clusters in any cloud environment. We aspire to become a small yet beautiful, vendor-neutral, developer-friendly, and community-driven open-source project!

• Github Repo: https://github.com/KusionStack/karpor
• Website: https://www.kusionstack.io/karpor
• Live Demo: https://karpor-demo.kusionstack.io

This update includes contributions from 12 contributors across 58 PRs, including 14 significant new features and 7 bug fixes. For detailed update information, please refer to the Karpor v0.5.0 Release Notes.

❤️ Special Thanks

We would like to extend our special thanks to the 9 new code contributors (in alphabetical order):

@CirillaQL、@cheny-alf、@JasonHe-WQ、@mryanchia、@peter-wangxu、@rajeshkio、@regend、@solarhell、@z1cheng

We also want to thank the following community members for their valuable suggestions (in alphabetical order):

@arrowfeng、@danielstankw、@Edwin-Li-01、@eryajf、@haiwu、@kinbod、@T1-leiyang、@wenxuanwu、@ywgx

Thank you all for your contribution to the development of Karpor 🎉, we are delightedly shocked by the power of the community!

Welcome to claim the task in “Community Tasks” and participate in the community!

Link: Community Tasks

🚀 Important New Features

Cluster Management Upgrades

1. Support for Token-based Kubeconfig Import
Contributed by @peter-wangxu, this feature provides a more flexible way to access clusters.

2. AWS EKS Cluster Support
Contributed by @CirillaQL, you can now access the EKS cluster directly using AWS ACCESS_KEY and SECRET_ACCESS_KEY. For detailed configuration instructions, please refer to the EKS Cluster Access Guide.

Security Enhancements

1.Token Login Based on RBAC
Contributed by @ruquanzhao and @hai-tian, this feature supports logging into the Dashboard using a Token issued by Karpor-server. For more details, please refer to the Token Creation Guide.

Deployment Optimization

1. More flexible Helm configuration (@elliotxx)

2. arm64 architecture support(@elliotxx)

3. Cert-manager Optimization (@elliotxx)
Migrating Cert-manager into the image solved the issue of slow installation in certain network environments.

Experience Improvement

1. Health Check API (@JasonHe-WQ)
   Added /livez and /readyz health check APIs.
2. YAML Display Optimization (@regend)
   Automatically collapse non-critical YAML fields in the Dashboard to improve readability.

3. Version Information Display

• Cli supports --version parameter (@rajeshkio)
• Automatic version number generation (@rajeshkio)
• Version display on the Dashboard interface (@solarhell）

4. UI customization (@z1cheng)
Support configuration for displaying and hiding the GitHub icon.

🔧 Bug Fix

1. README video encoding optimization (@ruquanzhao)
2. Namespace addition to Cert-generator template (@elliotxx)
3. REST API response improvement (@elliotxx)
4. Fix for Syncer Transform Patch function (@iamryanchia)
5. Enhancement of Transform error logging (@iamryanchia)
6. Correction of Swagger path (@regend)
7. Fix for Hub Cluster Config Endpoint (@elliotxx)

📈 Future Plan

The v0.5.0 release primarily focuses on optimizing features based on feedback from the community users. Enthusiastic users have already tried Karpor in their internal corporate environments to address the needs of multi-cluster management and data plane requirements.

We have now started planning for the v0.6.0 version, which will enhance usability and AI integrations. For example:

• AI-based Kubernetes troubleshooting (similar to K8SGPT)
• Natural language resource search
• Display of real-time logs and event aggregations
• More features to come…

Please refer to the detailed milestone for more information. [v0.6 Milestone]

Feel free to bring your suggestions to the [Karpor v0.6.0 discussion] or submit an Issue on [GitHub].

🔎 Feature Preview

Searching for Kubernetes resources via natural language (developed by @jueli12 and @hai-tian) has entered the optimization phase and will be officially released in v0.6.0, stay tuned!

As always, we are grateful to the growing KusionStack community for their invaluable contributions, which helped us reach this remarkable milestone. In this post, you will discover some of the exciting new features, selected based on our strong belief in their potential to enhance the user experience of Kusion.

Kusion Release Command

In v0.13.0 , we have introduced a new command, kusion release , which allows users to observe and update the status of Kusion Release files for the Project in a specified Workspace . This can help users understand the contents of the Release files after executing the kusion apply and kusion destroy commands. Additionally, in case of an unexpected interruption during applying and destroying applications, users can revise the locked status through the command for further operations.

The kusion release command includes the following 3 sub-commands.

• kusion release list lists all the Release files of the current Project in the specified Workspace
• kusion release show shows the details of a Release file of the current or the specified Project
• kusion release unlock unlocks the status of the latest Release file (applying & destroying ->failed) of the current or the specified Project

Kusion Resource Command

We have also added the kusion resource command in v0.13.0 , which allows users to observe the information of Kusion Resources related to the applications after executing kusion apply or kusion destroy . This will include the actual infrastructural resource names corresponding to the Kusion Resources , the dependency resources of Workload , the IDs of the cloud resources, and the resources’ deployment statuses. The kusion resource command aims to provide application developers and platform engineers with all the infrastructural resource information and relationships corresponding to the config intent, helping users better manage the Day 1 operations.

The kusion resource command includes the following sub-command.

• kusion resource graph displays a graph of all the resource information of the target Project and Workspace

Kusion Project List Command

Additionally, we have implemented a useful feature in v0.13.0 , allowing users to list all the applied Projects in the target Backend and Workspace through the kusion project list command. This command supports specifying multiple Workspaces simultaneously.

KCL-based Customized Health Check

In previous versions, Kusion would by default to waiting for the reconciliation of kusion-supported Kubernetes resources, including some common resources like Namespace, Deployment, Job and Service during kusion apply . However, for the other resources such as Kubernetes CRD, Kusion could not effectively check their reconciliation status. In v0.13.0 , we support users adding KCL-based customized health check conditions for specified Kusion Resources in the Workspace configurations of a Kusion Module . Kusion will add these customized health check conditions to the Extensions field of the Kusion Resource upon Spec generation and use them to check whether a Kubernetes resource is successfully reconciled during kusion apply .

Besides declaring the customized health check conditions in the Workspace configurations of a Kusion Module , we also provide the SDK support in kusion-module-framework, helping platform engineers to add these health check conditions to the Extensions field of the generated Kusion Resources during the Module development.

LLM Ops with Inference Module for AI Applications

In the recent wave of Artificial Intelligence (AI), the Large Language Model (LLM) has been becoming a key factor in driving innovation and productivity. As the scale of the LLMs grows, the development and operations of these complex models are becoming increasingly challenging, leading many developers and companies to find more efficient ways to manage the large-scale and intricate AI models and applications.

Our KusionStack community members have provided a Kusion Module named Inference , aimed at simplifying the complexities of deploying and invoking the LLM inference service for the AI application developers, thus enhancing the efficiency of AI application deployment and operations.

import kam.v1.app_configuration as ac
import service
import service.container as c
import inference.v1.inference
import network as n
​
inference: ac.AppConfiguration {
    # Declare the workload configurations. 
    workload: service.Service {
        containers: {
            myct: c.Container {image: "kangy126/app"}
        }
        replicas: 1
    }
    # Declare the inference module configurations. 
    accessories: {
        "inference": inference.Inference {
            model: "llama3"
            framework: "Ollama"
        }
        "network": n.Network {ports: [n.Port {
            port: 80
            targetPort: 5000
            public: True
        }]}
    }
}

Above is the sample config codes with Inference module as an Accessory to provide LLM inference service for AI application, which declares the required config items for the Inference module, model and framework for the developers to select the types of LLM models and frameworks. And the AI application workload can call the corresponding inference service through the INFERENCE_URL environment variable injected by this module. For more details, please refer to the user guides of the Inference module.

Bug Fixes

• Fixed some bugs related to performing JSON Patch on Workload
• Fixed some bugs of initializing Kubernetes and Terraform Runtime
• Fixed some issues with updating Release files during kusion apply and kusion destroy
• Fixed some bugs of not printing all the Kubernetes resources with multiple replicas while applying

Summary

In this version, we have implemented some new commands of kusion release , kusion resource and kusion project list , helping users better observe and manage the application resources. In addition, we support the KCL-based customized health check for Kubernetes resources. And we provide the Inference module for AI applications to enjoy a more efficient LLM service deployment and integration. Aside from the ones mentioned above, there are tons of small improvements and changes in this release as well. You can find the full details in the Release Note.

Welcome to our Github and leave some feedback! ⭐️

Github: https://github.com/KusionStack/kusion

Website: https://kusionstack.io

However, in some specific scenario, users may also have the need to directly use Kusion CLI to apply the raw Kubernetes manifests, such as taking over some existing resources and deploying some special resources like CRDs (CustomResourceDefinition), benefitting from Kusion’s previewing resource diffs and managing the application Releases .

Hence, we are exciting to announce that, to help users directly apply raw K8s manifests, the KusionStack community has provided a k8s_manifest module.

The module definition and implementation, as well as the example we are about to show can be found here.

Pre-requisites

Please refer to the pre-requisites in the guide for deploying an application with Kusion.

The example below also requires users to have initialized the project with the command of kusion workspace create , kusion project create , and kusion stack create , which will create a workspace and a project, and also generate a kcl.mod file under the stack directory.

Managing Workspace Configuration

Steps to initialize a workspace with an empty configuration can be found at this document. And the same empty configuration will still work in this guide, no changes are required here.

However, if you (or the platform team) would like to set some default paths for the raw K8s manifests to standardize the behavior of applications in the dev workspace, you can do so by updating the dev.yaml with the following config block:

modules: 
    k8s_manifest: 
        path: oci://ghcr.io/kusionstack/k8s_manifest
        version: 0.1.0
        configs: 
            default: 
                # The default paths to apply for the raw K8s manifest YAML files. 
                paths: 
                    - /path/to/k8s_manifest.yaml
                    - /dir/to/k8s_manifest/

Please note that the paths declared by the platform engineers in the workspace configs will be merged with the ones declared in the AppConfiguration in main.k .

The workspace configs need to be updated with the command:

kusion workspace update dev -f dev.yaml

Example

To apply the specified raw K8s manifests with k8s_manifest module, please use the v0.2.1 version of kam , whose workload is no longer a required field in the AppConfiguration model.

An example is shown below:

kcl.mod :

[dependencies]
kam = { git = "https://github.com/KusionStack/kam.git", tag = "v0.2.1" }
k8s_manifest = { oci = "oci://ghcr.io/kusionstack/k8s_manifest", tag = "0.1.0" }

stack.yaml :

# Generate a specified namespace 
name: dev
extensions: 
  - kind: kubernetesNamespace
    kubernetesNamespace: 
      namespace: test

main.k :

import kam.v1.app_configuration as ac
import k8s_manifest

test: ac.AppConfiguration {
    accessories: {
        "k8s_manifests": k8s_manifest.K8sManifest {
            paths: [
                # The `test.yaml` should be placed under the stack directory, 
                # as it is declared using a relative path. 
                "./test.yaml"
            ]
        }
    }
}

test.yaml :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: test
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

Generate Spec

Execute the kusion generate command, the Deployment in the test.yaml will be generated into a Kusion Resource with a Kusion ID in the Spec .

Preview and Apply

Execute the kusion preview command, we can review the resource three-way diffs for a more secure deployment.

Execute the kusion apply command to deploy the K8s resources.

As always, we are grateful to the growing KusionStack community for their invaluable contributions, which helped us reach this remarkable milestone. In this post, you will discover some of the exciting new features, selected based on our strong belief in their potential to improve the user experience.

Optimize Storage Backend Management

In this version, we add support for AWS S3 path-style endpoint for Kusion backend storage, and provide a new kusion release unlock command to unlock the latest Release file of the current Stack . This can prevent Phase in the Release file from being locked in an unfinished states such as previewing , applying , and destroying due to unexpected interruptions, which may cause users unable to perform subsequent operations with Kusion on the Stack .

Optimize the Display of Sensitive Information

In the previous versions, the kusion preview and kusion apply command would directly expose the plaintext data and stringData when displaying Kubernetes Secret resources, posing a risk of sensitive information leakage. In this version, we optimize the display of the sensitive information, and the data and stringData field of Kubernetes Secret resources will not be displayed in plaintext.

Optimize Terraform Cloud Resource Management

In this version, Kusion provides the ability to import existing cloud resources, and for the imported resources, Kusion will skip their deletion during kusion destroy , as there may be multiple applications sharing the same resources. Additionally, Kusion also supports concurrent operations on independent Terraform resources, thereby improving the operational efficiency.

A sample configuration is shown below.

modules: 
    mysql: 
      path: oci://ghcr.io/kusionstack/mysql
      version: 0.2.0
      configs:
          default: 
            cloud: alicloud
            size: 20
            instanceType: mysql.n2.serverless.1c
            category: serverless_basic
            privateRouting: false
            subnetID: vsw-2zem*********
            databaseName: "wordpress-mysql"
            # Import an existing alicloud_db_instance. 
            importedResources: 
              "aliyun:alicloud:alicloud_db_instance:wordpress-mysql": "rm-2zem********"
    network:
        path: oci://ghcr.io/kusionstack/network
        version: 0.2.0
        configs:
            default: {}

Support Runtime Configuration in Workspace

In this version, users can decalre the Kubernetes cluster configs and Terraform Provider credentials as runtime configurations in the context field in Workspace with the following configuration items.

• KUBECONFIG_PATH : Kubernetes cluster config path
• KUBECONFIG_CONTENT : Kubernetes cluster config content, supporting plaintext or referring AWS and Alicloud Secrets Manager URL for declaration, e.g. ref://secret-store//kubeconfig
• AWS_ACCESS_KEY_ID : Terraform AWS Provider access key ID, supporting plaintext or referring AWS and Alicloud Secrets Manager URL for declaration
• AWS_SECRET_ACCESS_KEY : Terraform AWS Provider secret key, supporting plaintext or referring AWS and Alicloud Secrets Manager URL for declaration
• ALICLOUD_ACCESS_KEY : Terraform Alicloud Provider access key ID, supporting plaintext or referring AWS and Alicloud Secrets Manager URL for declaration
• ALICLOUD_SECRET_KEY : Terraform Alicoud Provider secret key, supporting plaintext or referring AWS and Alicloud Secrets Manager URL for declaration

A sample configuration is shown below.

modules: 
    mysql: 
      path: oci://ghcr.io/kusionstack/mysql
      version: 0.2.0
      configs:
          default: 
            cloud: alicloud
            size: 20
            instanceType: mysql.n2.serverless.1c
            category: serverless_basic
            privateRouting: false
            subnetID: vsw-2zem*********
            databaseName: "wordpress-mysql"
    network:
        path: oci://ghcr.io/kusionstack/network
        version: 0.2.0
        configs:
            default: {}
# The remote secrets manager for sensitive data storage. 
secretStore:
  provider:
    aws:
      region: us-east-1
# Runtime configs. 
context: 
  # Kusion will retrieve the encrypted kubeconfig with the secretStore provider declared above. 
  KUBECONFIG_CONTENT: ref://secret-store/kubeconfig
  ALICLOUD_ACCESS_KEY: LTAI5*********
  ALICLOUD_SECRET_KEY: LNOmf*********

Support Using Spec for Preview and Apply

In this version, Kusion supports using the Spec file as the input for the kusion preview and kusion apply command. Users can specify a Spec file in the Stack directory with the --spec-file flag. The Kusion engine can directly consume the Spec file and update the operation results to the corresponding Release file for the Stack .

Complete the Modularization of the Workload

In the version of v0.12.0 , we have decoupled the schema definitions of Service and Job workloads separately from the codes of Kusion. And in this version, we have completely modularized the Workload , both of the schema definitions and module generator implementations for Service and Job workloads have been moved to the catalog repository.

A sample configuration is shown below.

# kcl.mod
[package]

[dependencies]
kam = { git = "https://github.com/KusionStack/kam.git", tag = "0.2.0" }
service = { oci = "oci://ghcr.io/kusionstack/service", tag = "0.2.0" }

# main.k
import kam.v1.app_configuration as ac
import service
import service.container as c

quickstart: ac.AppConfiguration {
 workload: service.Service {
  containers: {
   quickstart: c.Container {
    image: "kusionstack/kusion-quickstart:latest"
   }
  }
 }
}

Please note that Kusion v0.12.1 is not compatible with the locally cached 0.1.0 version of Service and Job modules. Users may need to delete the locally cached directories of $HOME/.kcl/kpm/service_0.1.0 and $HOME/.kcl/kpm/job_0.1.0 , or use the 0.2.0 version of Service and Job modules.

Bug fixes

• Fix a bug of volume directory overwrite misbehave
• Fix a bug of reusing cached Terraform providers
• Fix a bug of JsonPatch error message exposure

New Contributors

Sincere gratitude to @hoangndst and @vietanhtwdk from Vietnam for contributing to the Kusion repository!

Summary

In this version, we optimize the storage backend management, the display of sensitive information, and Terraform cloud resource management. Besides, we add support for runtime configurations in Workspace and using Spec file as the input for kusion preview and kusion apply . In addition, we completely implement the Workload modularization.

Aside from the ones mentioned above, there are tons of small improvements and changes in this release as well. You can find the full details in the Release Note.

Welcome to our Github and leave some feedback! ⭐️

Github: https://github.com/KusionStack/kusion

Website: https://kusionstack.io

🔔 What is Karpor?

Today we are excited to announce that Karpor is now open-source! 🎉🎉🎉

Karpor is a Modern Kubernetes Visualization Tool. Its core features focus on 🔍 Search, 📊 Insight and ✨ AI. The goal is to connect platforms and multi-clusters more easily and quickly, and use AI to empower kubernetes to extract key insights from proliferations of cluster resources and provide them to end users.

Karpor is designed to reduce the complexity to use kubernetes, so that developers and platform teams can extract the most valuable information more effectively and intuitively.

GitHub：https://github.com/KusionStack/karpor

🚀 Why Karpor?

The increasing complexity of the kubernetes ecosystem is an undeniable trend that is becoming more and more difficult to manage. This complexity not only entails a heavier burden on operations and maintenance but also slows down the adoption of new technologies by users, limiting their ability to fully leverage the potential of kubernetes.

As an experienced Kubernetes YAML Engineer, you may have also encountered the following perplexities:

• The cluster is like a black box, sometimes all you can see is a KubeConfig, and we can’t see what happens behind it
• The team/company has a specific business domain model and needs to establish a mapping between existing systems and kubernetes resources
• The application has been deployed to multiple kubernetes clusters, but its topology is not fully visible

We have used several kubernetes visualization tools over time, such as Lens, k9s, kube-explorer, and the kubernetes dashboard, among others. Some are commercialized, some do not support self-host, and some are rudimentary for production needs… In short, we have not yet encountered a product that completely satisfied with.

The recent rise of large language models has sparked an unprecedented wave of artificial intelligence innovations. This time, AI technology has remarkably infiltrated its way into people’s everyday life. Even my retired parents have started using AI services, which makes me believe that we are at a historical moment that is reshaping the world.

So naturally we started building a lightweight, AI-empowered kubernetes visualization tool to solve the problems mentioned earlier. It features the following:

• Fully empowers Kubernetes with AI.
• Identify potential risks and provide solutions based on AI.
• Customized logical views to fit the resource organization models for different scenarios, such as applications, environments, etc, which may have different interpretations at places.
• Travel back in time via timeline, time machine to quickly diagnose and troubleshoot based on historical snapshots.
• Intuitive and effective search, providing a number of user-friendly ways to locate resources across clusters, such as keywords, SQL, and natural language.
• Low cognitive burden, it is read-only, non-invasive to the cluster it’s watching, and users can deploy it to their private environments with one click.
• Cross-cluster topological views, providing a global perspective of resources no matter where they are.

We have named this tool Karpor. In general, we wish Karpor to focus on 🔍 search, 📊 insights, and ✨ AI, to break through the increasingly complex maze of kubernetes, achieving the following value proposition:

As of today, we have built the initial version of Karpor based on this vision, which features the following:

• An optimized search experience for kubernetes:

• Discover potential problems through compliance reports:

• Manage the customized logical views:

🙌 Karpor vs. Kubernetes Dashboard

In today’s kubernetes ecosystem, there are multiple tools and platforms that can manage and visualize clusters. Kubernetes Dashboard is an officially provided universal web UI for managing and troubleshooting kubernetes clusters. Karpor, as an emerging Kubernetes Visualization Tool, is designed to provide more advanced features and a better user experience.

Here are some key comparisons between Karpor and Kubernetes Dashboard:

🎖 ️Vision: Embracing the Community

We firmly believe that a successful open-source project should be community-driven. For open-source projects, we come up with an idea and build an initial version. The final form of the project, we believe, should be well guided by the community.

Therefore, we are committed to shaping Karpor into something that is:

• Small and beautiful: Focused on an excellence of user experience.
• Vendor-neutral: Independent from any specific cloud services or companies.
• Developer-friendly: Community-friendly with high-quality documentations and support.
• Community-driven: Encouraging and welcoming contributors to participate in and even lead the development of the project.

We place great emphasis on community participation and contribution. To this end, we have specially put together a community task list to help those that are interested quickly get started and participate in the project. The tasks are categorized by difficulty, ranging from simple tasks such as document translation, bug fixes, and unit testing, to medium-difficulty tasks like log/event aggregators, risk audit enhancements, and automatic cluster imports, to challenging tasks such as OpenCost integration and login authentication.

We encourage every developer interested in Karpor to visit our GitHub page, review the task list, and contribute at ease.

🎖︎ Community Task List: https://github.com/KusionStack/karpor/issues/463

All developers who participate in the community will be featured in the Contributors section on the README and the homepage of the official website. We extend our sincerest thanks to all developers and contributors who are already active in the Karpor open-source project, for your efforts and creativity! We look forward to working with the community to make Karpor an even more powerful and comprehensive open-source tool.

🌈 Moving Forward

We are actively soliciting feedback and suggestions from the community to plan the next version of Karpor — v0.5. We want to hear your voice, whether it’s feature requests, improvement suggestions, or bug reports; please leave a comment in the corresponding issue.

Our ultimate goal is to shape Karpor into a community-driven Kubernetes Visualization Tool in the AI era. Currently, what we have is a usable version with basic functionalities.

In the next version, we will solidify the basic functionalities and fully embrace AI. We have preliminarily planned some new features, such as support for natural language search of cluster resources, AI-driven diagnostic suggestions, timelines, etc., to help users better understand resources in multiple clusters, identify issues, and troubleshoot. We welcome everyone’s feedback!

If you like this project, welcome to Star on GitHub 🌟🌟🌟

https://github.com/KusionStack/karpor

If you are also building an IDP, our practical experience might be helpful to you.

About the Author

Mr. Wang Zijian is a senior staff and PaaS team leader in the Platform Engineering and Technology Risk Department at Ant Group. This article is transcribed from his speech “A Blend of Fire and Ice: Accelerating innovations across Internet and Financial Services via Platform Engineering at Ant Group” at PlatformCon 2024.

Hello everyone, welcome to join us at PlatformCon 2024. My name is Zijian Wang. I’m from the Platform Engineering and Technology Risk Department at Ant Group.

Yes, you heard that right. Ant Group has specifically created this department to implement our platform engineering strategy. In this post-COVID era, it has become particularly important to drive the innovation and growth of the company’s business. To this end, we are focusing on building a self-service Internal Developer Platform to improve developer efficiency and accelerate innovation within the company. Currently, I am the leader of the company’s PaaS team, responsible for building our IDP.

AntGroup: Develop digital technology to serve the real economy

Our platform serves as a fundamental pillar that supports the diverse business scenarios within Ant Group, including digital payments, connectivity, finance, technologies, and globalization. We have more than 10,000 developers who build software on global infrastructures across multiple clouds. To ensure this extensive and complex production is running smoothly, Ant Group has established a Platform Technologies Business Group, responsible for the technological foundation beneath all the lines of business.

My department, the Platform Engineering and Technology Risk Department, is at the heart of this organization, providing support upwards to the application teams and collaborating downwards with our computing and AI infrastructure. The goal is to provide efficient, secure, and risk-adjusted platform technology products and services to all our lines of business (LOBs).

Primary technical challenges: Ice and Fire

In our journey, the primary challenge comes from Ant Group’s diversified business landscape. Serving over a billion users and 80 million merchants, the Alipay app processes hundreds of millions of transactions daily. Furthermore, Ant’s digital finance serves more than 2,000 financial institutions in China and has provided digital credit services to over 50 million small and micro businesses.

As a consequence, the developers at Ant Group are facing a quite unique challenge. On one side, they need to quickly innovate and test out new ideas on the internet business side. On the other, digital finance requires the utmost stability, safety, and risk management. It’s like trying to mix fire with ice.

In the past few years, cloud-native technologies are flourishing, which introduced a great deal of complexity of using multi-clouds infrastructure to the developers. Along with this, the popularization of DevOps and the shift-left approach has significantly increased the responsibilities and cognitive burden on our developers, lowering their productivity and efficiency, and further hindering the company’s ability to quickly innovate.

The Strategy of Platform Engineering at Ant Group

To address this challenge, reconcile the conflict between this ice and fire situation, and further reduce the burden on developers, Ant Group has developed a strategy to implement platform engineering philosophies, which includes three major points:

• First of all, we have built a unified PaaS product for the entire business: Jiuzhou. A unified control plane and constraints on the application architectures are essential premises for an effective risk and security control on a global scale. Jiuzhou was initially positioned as Ant Group’s operations platform, primarily serving operations engineers and SREs. However, with the evolution of the DevOps movement, about 70% of our users are now application developers. As a response, we have also transformed our platform product strategy, upgrading Jiuzhou to an IDP, a.k.a an Internal Developer Platform. We intend to focus on developers as the primary users of our platform, improving the developer experience through a one-stop developer portal that hides the complexity beneath the surface. This approach helps developers achieve efficient software development, operations, and delivery, thereby accelerating TTM(Time To Market).
• Secondly, unlike the shift-left movement, which transfers the complexities and responsibilities to the developers, we are continuously shifting operations, risk management, security, and trust capabilities downwards into our platform by shipping it with Kubernetes and other cloud-native technologies, into the platform layer. This approach decouples the advanced platform capabilities required by financial services from the application code, allowing developers to focus solely on the business logic, thereby balancing the contradictions between competing demands.
• Finally, we’ve discovered in practice that a platform product that is natively stacking capabilities and accommodating all the applications, often gradually turns the platform team into a bottleneck trying to meet all the customer needs. The diverse business requirements and standardized platform capabilities are difficult to reconcile. In some cases where rapid innovations are justified, the required platform capabilities might not always be considered the most critical ones to prioritize from a holistic view. However, these seeds of innovation could represent the infinite potential looking into the future. Therefore, openness is the most important strategy following a unified-paas and shift-down. For platform capabilities, we have built a programmatic interface based on the concept of “Everything as Code”, thereby enabling the SREs and Platform Engineers to quickly pave golden paths for developers, customized for specific scenarios. While reusing the platform’s core capabilities, this approach aims to meet the diverse business needs as much as possible.

Unified PaaS — Platform + Service accelerates business innovations

Back in 2014 and 2015, Ant Group created its first-generation PaaS platform product, Jiuzhou 1.0, to facilitate the needs for application operations and technical support. However, with the evolution into a more diversified business landscape, small platforms under different business groups started to naturally appear, leading to many independent “siloed” architectures across the company. These fragmented platforms resulted in inconsistent standards, especially in dealing with global architecture issues such as the ability to manage technical risks, which became more inefficient and difficult, posing challenges to business development.

As a result, we initiated an upgrade for the PaaS platform product, which launched Jiuzhou 2.0. A major highlight was the centralization into a single application control plane across the company, which not only achieved consolidation of the application’s metadata, operations, and change management but also allowed each business group to benefit from the high-quality technical risk management capabilities, underpinning the rapid growth of internet finance businesses over the years.

However, this approach also introduced more constraints into software development, which soon led to a loss of diversity in business development. As a natural response, We launched a strategy for platform engineering, continuously upgrading the PaaS product to an IDP (Internal Developer Platform). This time, our primary goal was to improve developer efficiency and enable self-service by providing a set of core capabilities covering the software development lifecycle. At the organizational level, we re-scoped the responsibilities of SREs (Site Reliability Engineers) and formed a dedicated team of platform engineers to fill the gap. This team was responsible not only for managing risks but also for improving efficiency.

The platform was divided into two parts: The bottom layer leverages core cloud-native capabilities to provide general functionalities for various business scenarios, including identity, permissions, metadata, deployment, changes, configuration, and capacity management. Meanwhile, the top layer equips SREs (Site Reliability Engineers) and platform engineers with a Golden Path Builder, enabling them to serve various relevant business teams by paving opinionated golden paths based on these core capabilities. This meets the diverse needs of developers across different business groups. Essentially, we’ve established a layered architecture in which the platform, together with SREs and platform engineers, supports all the developers at Ant Group, accelerating the Time To Market.

Declarative Workloads — Automating the procedural operations

In addition to building a consolidated platform, we have also committed to shift-down various technical capabilities to the infrastructure layer. Operations and changes to an application, being a key part of the software development lifecycle (SDLC), have been re-implemented through the Kubernetes Operator. This approach has redefined the traditional operational tasks, making them accessible to developers in a declarative manner. This transformation has significantly streamlined the operations process, similar to how automated driving works: users just need to declare their intended destination, and the remaining process is handled automatically by the car, or in this case, the platform.

Evidently, this represents an ideal scenario. To distinguish ourselves from the various K8S Workloads found within the community, we have developed our own workloads and extensions for cloud-native resources. These, such as CollaSet, PodDecoration, and OperationJob, are used to manage application workloads, sidecars, and traditional procedural operations, respectively. They are consolidated and abstracted as Kubernetes resources for streamlined management and integration with Ant Group’s technical risk system, achieving a shift-down of risk management capabilities. This includes service level protection, change governance, and circuit breaker mechanisms.

For Pod lifecycle management, PodOpsLifecycle extends and integrates Pod and container state management across all workloads. This approach enables developers to execute necessary actions at specific phases of a Pod operation — before, during, or after. For instance, excluding the Pod’s IP from the traffic routing rules before initiating changes, performing the actual changes, and adding it back after the it is completed, to achieve a graceful Pod operation, and prevent any traffic loss.

Alter Shield — Empowering DEV to change with confidence

From a stability perspective, about 60% of incidents within Ant Group are caused by changes. The adoption of the unified declarative operations not only improves developers’ efficiency but also provides an excellent tangent for centralized technical risk assurance. As a result, we have developed a cloud-native risk defense product: Alter Shield.

On one hand, we introduced the Open Change Management Specification protocol to standardize change information, which is defined and managed through the Altershield platform. On the other hand, the platform implements a unified change Pipeline across multiple environments within the SDLC, automating the batch processing and canary release of changes at scale, and adding Hooks before and after each batch of changes. These Hooks are connected to change Policies, capable of conducting impact analysis, change interruption, and business and system metrics measurement.

In the early stages, most of these policies were written by SREs, implementing universal risk prevention and control. As we implement our platform engineering strategy and focus on improving developer efficiency, we are also making change policies accessible to developers. For example, a declarative policy might state that if a certain business metric drops during a change process, the platform can automatically roll back the change or take other actions, enabling unattended change processes and empowering developers to change with confidence.

Trust-Native IAM — Balancing safety and efficiency

Efficiency, risk control, and security are the ‘Iron Triangle’ of our core platform metrics. Within this premise, the IAM provides the core capabilities to manage identities and permissions across the platform. It is responsible for issuing universal identities for both users and applications, while also keeping track of the organizational structures and resource topologies across the entire company. This enables the platform to construct a Resource Graph that connects resources to their corresponding applications and organizations. Based on this graph, the platform is able to automatically derive the relations among application assets and result in less permission approvals.

In the meantime, secret managements, application permission requests, and policy-based access control, we’ve also shift-down this entire set of capabilities by leveraging cloud-native technologies. For example, we’ve implemented an identity injection mechanism based on Kubernetes. When a POD is created, an identity is automatically provisioned and injected into each container. By leveraging the Distributed App Runtime, which offloads the communication of distributed applications to the Sidecar component, decoupling them from the application logic. In this way, the distributed applications runtime controls access and secures connections to external service calls based on injected identities.

Most Importantly, all these security features are transparent to the application and does NOT require any prior knowledge. Therefore, we’ve named this mechanism Trust-Native, emphasizing that our platform’s security and trust capabilities are built-in natively, achieving a balance between efficiency and security.

Kusion — Fusion of K8s and all infra enables open collaboration

Eventually, we’ve codified all capabilities and built the platform centered around a Platform Orchestrator named Kusion. We’ve connected to various types of infrastructures through an expandable driver-like mechanism, including Kubernetes as well as other cloud infrastructures. Based on KCL (Kusion Configuration Language) and an configuration monorepo, we provided the developers with a consistent, efficient, and secure programmable interface. Moreover, downstream teams can build new configuration models leveraging these existing and reusable ones to meet the specific business needs, thus creating a thriving ecosystem.

In the early days, our first and foremost goal was to fully codify Ant infrastructure, in order to accelerate the adoption of the IaC (Infrastructure as Code) strategy. However, we’ve noticed that when various perspectives of the configurations were merged together, especially when it’s mixing up the viewpoints representing applications and infrastructure opinions, developers are more likely to find it both challenging and inefficient when managing those configuration code.

We received a substantial amount of feedback and learned valuable lessons from these experiences. Based on the division of labor and the collaboration patterns among developers, SREs, and platform engineers, we’ve defined and implemented a mechanism for separation of concerns. By drawing insights from DCM (Dynamic Configuration Management), we have upgraded the entire Kusion configuration model and the platform orchestrator.

The upgraded model distinguishes between application configurations that are environment-agnostic and those that aren’t. In terms of division of labor, global configurations and environment-specific ones, especially the complex, cell-based deployment configurations are usually maintained by SREs, who also select the scopes for them to take effect. (A cell is the smallest deployable unit that represents a logical isolation for an application).

Correspondingly, developers only need to define the part of the configurations that is closely relevant to the application itself, hopefully mostly are environment-agnostic. When the configurations are consumed by Kusion, it automatically associates the two based on previously defined scopes of effectiveness, dynamically generates the final infrastructure specifications, and then applies them to the infrastructure. This significantly reduces developers’ awareness of various configurations which heavies their burden. Of course, Ant Group’s cell-based deployments has a long history and penetrates all aspects of the infrastructures. This work continues as we implement our platform engineering strategy, with the ultimate goal to completely eliminate developers’ need to be aware of cell-based and all other environment-specific deployment configurations.

Welcome everyone to join our communit

We have already partially open-sourced our platform capabilities, which include the KusionStack and the AlterShield projects. KusionStack is a platform engineering technology stack, aimed at helping platform engineers build their own Internal Development Platforms (IDP).

Meanwhile, AlterShield represents the open-source implementation of Ant Group’s Technological Risk-defense as a Service. In the future, we plan to open-source more platform capabilities. Here are the related links and I welcome everyone to join our community.

KusionStack：

• Website: https://kusionstack.io
• Github: https://github.com/KusionStack/
• Slack: kusionstack.slack.com
• Twitter: @KusionStack

AlterShield:

• Website: https://altershield.io
• Github: https://github.com/traas-stack/

A Blend of Fire and Ice: Accelerating Innovations Across Internet and Financial Services via… was originally published in KusionStack on Medium, where people are continuing the conversation by highlighting and responding to this story.

As always, we are grateful to the growing KusionStack community for their invaluable contributions, which helped us reach this remarkable milestone. In this post, you will discover some of the exciting new features, selected based on our strong belief in their potential to improve the team collaboration workflow.

Kusion Apply Enhancement

In the previous versions, the kusion apply command did not automatically wait for all the resources to be available, which might lead to users’ confusions. In this version, we have upgraded the kusion apply command to default to waiting for all the resources to be available before completing the execution. Meanwhile, it also supports the configurable timeout and optimizes the command line terminal interface.

State Upgrading to Release

In the preview versions, when a Stack is applied or destroyed, Kusion would create or update a state file in the backend storage to record the state information of the application resources. In this version, we have upgraded the state file to a release file, which is used to record the metadata, resource spec, the state of the stack resources, and the operation phases. Besides, we have also implemented a locking mechanism for the release file to ensure the concurrent read and write safety in team collaboration scenarios and to meet the needs of audit management.

Kusion Module Management Upgrade

In this version, we added kusion mod list and kusion mod add to the kusion mod command, allowing developers to better manage the Kusion modules. kusion mod list can help developers obtain the name, version and OCI artifact URL of the available Kusion module in a specific workspace, while kusion mod add can assist developers in automatically adding the dependency declarations for a specified module in the kcl.mod file and fetching the corresponding module OCI artifact, which is similar to go get .

Workload Modularization

In previous versions, the Workload was a component tightly coupled in the AppConfiguration model. In this version, we have modularized the built-in workloads and move the schema definitions of the Service and Job from the codes of Kusion CLI to the catalog module repository. Developers can now declare and use Service and Job just as they would with other modules. Moreover, platform engineers can also implement customized workloads according to their own needs, thereby enhancing the extensibility of the AppConfiguration model.

The definitions and related examples of the modularized Service and Job can be found at:

• https://github.com/KusionStack/catalog/tree/main/modules/workload/service
• https://github.com/KusionStack/catalog/tree/main/modules/workload/job

Production Practice Case

In this version, we provided an example production-ready practice case of how Kusion integrates with GitHub Actions to help developers and platform engineers collaborate on application delivery and operation. You can refer to the following document for more details:

• https://www.kusionstack.io/docs/user-guides/production-practice-case/collaborate-with-github-actions

In the above article, we have defined the workflows for platform engineers and application developers based on the concept of separation of concerns:

• Platform engineers are primarily responsible for creating centralized Kusion backend storage, developing customized Kusion modules, and creating and maintaining the standardized platform configurations in the workspace.
• Application developers can focus on the application business logic and configuration codes, and trigger the automated GitHub Actions pipelines by submitting configuration codes to complete the preview and apply process.

Bug fixes

• Fix a bug of customized namespace to correctly get Kubernetes namespace
• Fix a bug of secrets management
• Fix some bugs of Kubernetes resource normalization and conversion
• Fix some bugs of applying and destroying Terraform resources on the Windows system

New Contributors

Sincere gratitude to @ Liuxingyu1111111 and @ wolfcode111 for contributing to the kusion repository.

Summary

In this version, we have enhanced the ability and user interface of the kusion apply command, and upgraded the state file in the storage backend into release , while also optimizing the experience for application developers to use Kusion modules. Aside from the ones mentioned above, there are tons of small improvements and changes in this release as well. You can find the full details in the Release Note.

Welcome to our Github and leave some feedback! ⭐️

Github: https://github.com/KusionStack/kusion

Website: https://kusionstack.io