Chapter 3: Packages

Getting a package #

Packaging in kpt is based on Git forking. Producers publish packages by committing them to a Git repository. Consumers fork the package to use it.

Let’s revisit the Wordpress example:

$ kpt pkg get https://github.com/kptdev/kpt.git/package-examples/wordpress@v0.9

A package in a Git repo can be fetched by specifying a branch, tag, or commit SHA. In this case, we are specifying tag v0.9.

Refer to the get command reference for usage.

The Kptfile contains metadata about the origin of the forked package. Take a look at the content of the Kptfile on your local filesystem:

# wordpress/Kptfile
apiVersion: kpt.dev/v1
kind: Kptfile
metadata:
  name: wordpress
upstream:
  type: git
  git:
    repo: https://github.com/kptdev/kpt
    directory: /package-examples/wordpress
    ref: v0.9
  updateStrategy: resource-merge
upstreamLock:
  type: git
  git:
    repo: https://github.com/kptdev/kpt
    directory: /package-examples/wordpress
    ref: package-examples/wordpress/v0.9
    commit: b9ea0bca019dafa9f9f91fd428385597c708518c
info:
  emails:
    - kpt-team@google.com
  description: This is an example wordpress package with mysql subpackage.
pipeline:
  mutators:
    - image: gcr.io/kpt-fn/set-labels:v0.1
      configMap:
        app: wordpress
  validators:
    - image: gcr.io/kpt-fn/kubeval:v0.3

The Kptfile contains two sections to keep track of the upstream package:

1. The upstream section contains the user-specified Git reference to the upstream package. This contains three pieces of information:
   • repo: The Git repository where the package can be found
   • directory: The directory within the Git repository where this package can be found
   • ref: The Git reference for the package. This can be either a branch, tag, or commit SHA.
2. The upstreamLock section records the upstream Git reference (exact Git SHA) that was fetched by kpt. This section is managed by kpt and should not be changed manually.

Now, let’s look at the Kptfile for the mysql subpackage:

# wordpress/mysql/Kptfile
apiVersion: kpt.dev/v1
kind: Kptfile
metadata:
  name: mysql
info:
  emails:
    - kpt-team@google.com
  description: This is an example mysql package.
pipeline:
  mutators:
    - image: gcr.io/kpt-fn/set-labels:v0.1
      configMap:
        tier: mysql

As you can see, this Kptfile doesn’t have the upstream and upstreamLock sections. This is because there are two different package types in kpt:

• Independent package: A package where the Kptfile has upstream defined.
• Dependent package: A package where the Kptfile doesn’t have upstream defined.

In this case, the mysql subpackage is a dependent package. The upstream package for mysql is automatically inferred from the parent package. You can think of the Kptfile in the mysql package as implicitly inheriting the upstream section of its parent, with the only difference being that upstream.directory in the subpackage would instead point to /package-examples/wordpress/mysql.

Package Name and Identifier #

It is possible to specify a different local directory name to the get command. For example, the following fetches the packages to a directory named mywordpress:

$ kpt pkg get https://github.com/kptdev/kpt.git/package-examples/wordpress@v0.9 mywordpress

The name of a package is given by its directory name. Since the Kptfile is a KRM resource and follows the familiar structure of KRM resources, the name of the package is also available from the metadata.name field. This must always be the name of the directory, and kpt will update it automatically when forking a package. In this case, metadata.name is set to mywordpress.

In general, the package name is not unique. The unique identifier for a package is defined as the relative path from the top package to the subpackage. For example, we could have two subpackages with the name mysql having the following identifiers:

• wordpress/backend/mysql
• wordpress/frontend/mysql

Exploring a package #

After you fetch a package to your local filesystem, you typically want to explore the package to understand how it is composed and how it can be customized for your needs. Given a kpt package is just an ordinary directory of human-readable YAML files, you can naturally use your favorite file explorer, shell commands, or editor to explore the package.

kpt also provides the tree command which is handy for quickly viewing package hierarchy and the constituent packages, files, and resources:

$ kpt pkg tree wordpress/
Package "wordpress"
├── [Kptfile]  Kptfile wordpress
├── [service.yaml]  Service wordpress
├── deployment
│   ├── [deployment.yaml]  Deployment wordpress
│   └── [volume.yaml]  PersistentVolumeClaim wp-pv-claim
└── Package "mysql"
    ├── [Kptfile]  Kptfile mysql
    ├── [deployment.yaml]  PersistentVolumeClaim mysql-pv-claim
    ├── [deployment.yaml]  Deployment wordpress-mysql
    └── [deployment.yaml]  Service wordpress-mysql

Refer to the tree command reference for usage.

In addition, you can use a kpt function such as search-replace to run a query on the package. For example, to search for resources that have a field with path spec.selector.tier:

$ kpt fn eval wordpress -i search-replace:v0.1 -- 'by-path=spec.selector.tier'

Editing a package #

kpt does not maintain any state on your local machine outside of the directory where you fetched the package. Making changes to the package is accomplished by manipulating the local filesystem. At the lowest-level, editing a package is simply a process that either:

• Changes the resources within that package. Examples:
  • Authoring new a Deployment resource
  • Customizing an existing Deployment resource
  • Modifying the Kptfile
• Changes the package hierarchy, also called package composition. Examples:
  • Adding a subpackage.
  • Create a new dependent subpackage.

At the end of the day, editing a package will result in a Git commit that fully specifies the package. This process can be manual or automated depending on your use case.

We will cover package composition later in this chapter. For now, let’s focus on editing resources within a package.

Initialize the local repo #

Before you make any changes to package, you should first initialize and commit the pristine package:

$ git init; git add .; git commit -m "Pristine wordpress package"

Manual edits #

As mentioned earlier, you can manually edit or author KRM resources using your favorite editor. Since every KRM resource has a known schema, you can take advantage of tooling that assists in authoring and validating resource configuration. For example, Cloud Code extensions for VS Code and IntelliJ provide IDE features such as auto-completion, inline documentation, linting, and snippets.

For example, if you have VS Code installed, try modifying the resources in the wordpress package:

$ code wordpress

Automation #

Oftentimes, you want to automate repetitive or complex operations. Having standardized on KRM for all resources in a package allows us to easily develop automation in different toolchains and languages, as well as at levels of abstraction.

For example, setting a label on all the resources in the wordpress package can be done using the following function:

$ kpt fn eval wordpress -i set-labels:v0.1 -- env=dev

Chapter 4 discusses different ways of running functions in detail.

Rendering a package #

Regardless of how you have edited the package, you want to render the package:

$ kpt fn render wordpress

Refer to the render command reference for usage.

render is a critical step in the package lifecycle. At a high level, it perform the following steps:

1. Enforces package preconditions. For example, it validates the Kptfile.
2. Executes functions declared in the package hierarchy in a depth-first order. By default, the packages are modified in-place.
3. Guarantees package postconditions. For example, it enforces a consistent formatting of resources, even though a function (developed by different people using different toolchains) may have modified the formatting in some way.

Chapter 4 discusses different ways of running functions in detail.

Updating a package #

An independent package records the exact commit where the local fork and the upstream package diverged. This enables kpt to fetch any update to the upstream package and merge it with local changes.

Commit your local changes #

Before you update the package, you want to commit your local changes.

First, to see the changes you’ve made to the fork of the upstream package:

$ git diff

If you’re happy with the changes, commit them:

$ git add .; git commit -m "My changes"

Update the package #

For example, you can update to version v0.10 of the wordpress package:

$ kpt pkg update wordpress@v0.10

This is a porcelain for manually updating the upstream section in the Kptfile :

upstream:
  type: git
  git:
    repo: https://github.com/kptdev/kpt
    directory: /package-examples/wordpress
    # Change this from v0.9 to v0.10
    ref: v0.10
  updateStrategy: resource-merge

and then running:

$ kpt pkg update wordpress

The update command updates the local wordpress package and the dependent mysql package to the upstream version v0.10 by doing a 3-way merge between:

1. Original upstream commit
2. New upstream commit
3. Local (edited) package

Several different strategies are available to handle the merge. By default, the resource-merge strategy is used which performs a structural comparison of the resource using OpenAPI schema.

Refer to the update command reference for usage.

Commit the updated resources #

Once you have successfully updated the package, commit the changes:

$ git add .; git commit -m "Updated wordpress to v0.10"

Creating a package #

Creating a new package is simple: create a new directory and author resources:

$ mkdir awesomeapp
# Create resources in awesomeapp/

For convenience, you can use pkg init command to create a minimal Kptfile and README files:

$ kpt pkg init awesomeapp
writing Kptfile
writing README.md

Refer to the init command reference for usage.

The info section of the Kptfile contains some optional package metadata you may want to set. These fields are not consumed by any functionality in kpt:

apiVersion: kpt.dev/v1
kind: Kptfile
metadata:
  name: awesomeapp
info:
  description: Awesomeapp solves all the world's problems in half the time.
  site: awesomeapp.example.com
  emails:
    - jack@example.com
    - jill@example.com
  license: Apache-2.0
  keywords:
    - awesome-tech
    - world-saver

Composing a package #

A package can be composed of subpackages (HAS A relationship). Package composition is when you change the package hierarchy by adding or removing subpackages.

There are two different ways to add a subpackage to a package on the local filesystem:

1. Create a new package in a subdirectory
2. Get an existing package in a subdirectory

Let’s revisit the wordpress package and see how it was composed in the first place. Currently, it has the following package hierarchy:

$ kpt pkg tree wordpress/
Package "wordpress"
├── [Kptfile]  Kptfile wordpress
├── [service.yaml]  Service wordpress
├── deployment
│   ├── [deployment.yaml]  Deployment wordpress
│   └── [volume.yaml]  PersistentVolumeClaim wp-pv-claim
└── Package "mysql"
    ├── [Kptfile]  Kptfile mysql
    ├── [deployment.yaml]  PersistentVolumeClaim mysql-pv-claim
    ├── [deployment.yaml]  Deployment wordpress-mysql
    └── [deployment.yaml]  Service wordpress-mysql

First, let’s delete the mysql subpackage. Deleting a subpackage is done by simply deleting the subdirectory:

$ rm -r wordpress/mysql

We’re going to add back the mysql subpackage using the two different approaches:

Create a new package #

Create the directory:

$ mkdir wordpress/mysql

Initialize the package:

$ kpt pkg init wordpress/mysql
# author resources in mysql

This creates a dependent package.

Get an existing package #

Remove the existing directory if it exists:

$ rm -rf wordpress/mysql

Fetch the package:

$ kpt pkg get https://github.com/kubernetes/website.git/content/en/examples/application/mysql@snapshot-initial-v1.20 wordpress/mysql

This creates an independent package. If you wish to make this a dependent package, you can delete the upstream and upstreamLock sections of the Kptfile in mysql directory.

Publishing a package #

A kpt package is published as a Git subdirectory containing KRM resources. Publishing a package is just a normal Git push. This also means that any existing Git directory of KRM resources is a valid kpt package.

As an example, let’s re-publish the local wordpress package to your own repo.

Start by initializing the the wordpress directory as a Git repo if you haven’t already done so:

$ cd wordpress; git init; git add .; git commit -m "My wordpress package"

Tag the commit:

$ git tag v0.1

Push the commit which requires you to have access to the repo:

$ git push origin v0.1

You can then fetch the published package:

$ kpt pkg get <MY_REPO_URL>/@v0.1

Monorepo Versioning #

You may have a Git repo containing multiple packages. kpt provides a tagging convention to enable packages to be independently versioned.

For example, let’s assume the wordpress directory is not at the root of the repo but instead is in the directory packages/wordpress.

Tag the commit:

$ git tag packages/wordpress/v0.1

Push the commit:

$ git push origin packages/wordpress/v0.1

You can then fetch the published package:

$ kpt pkg get <MY_REPO_URL>/packages/wordpress@v0.1