Version after Version; How the Open Source Project Kubernetes Releases its Software

In this article, we look at how the open source project Kubernetes manages its software releases. By exploring the established and evolved community structures, this discussion will delve into the collaborative dynamics that underpin the management of Kubernetes releases, shifting the focus away from Kubernetes itself.

Open Source is fascinating ✨

Open source is undeniably fascinating. Just think about how big projects like Kubernetes are, where thousands of people around the world come together to build software that is changing entire industries. The contributors devote their free time and passion to work on this project, alongside their professional and personal commitments. Through initiatives like Kubernetes, the cloud-native field has emerged, revolutionizing how applications are developed and deployed.

Open source serves as a powerful driver of innovation. By embracing open source, companies can tap into a pool of collective knowledge and expertise, benefiting from advancements and fresh perspectives that emerge from collaborative efforts. Neglecting open source means potentially missing out on innovation and talent that could propel a company ahead of its competitors. Therefore, actively contributing to open source projects is not only a way to stay connected with the latest advancements, but also a means to actively participate in shaping the future of technology.

Open source challenges traditional hierarchies and structures by letting different voices and ideas come together in a democratic process. The democratization and inclusion of principles in software development may encourage companies to adopt similar approaches to their internal source projects. Establishing a culture of collaboration, transparency, and ownership within an organization can foster greater creativity and innovation within the organization.

The appeal of open source lies not only in its ability to drive innovation, but also in the opportunities it presents for personal and professional growth. Participating in open source projects lets people learn from top-notch professionals and connect with a global community of people who are passionate about pushing the boundaries of technology.

Challenges in Open Source Software Projects

It's fun to work on open source software projects, but it's important to know that they have their own problems. The chaotic nature of open source software development is one of the most intriguing aspects, and it's also where some obstacles come from.

• Coordination and Collaboration: Open source projects typically involve a geographically distributed community of contributors who work remotely. It can be hard to coordinate efforts, set goals, and manage contributions from many different people. Communication and collaboration requires proactive measures to ensure everyone is on the same page and working towards common objectives.
• Lack of Centralized Control: In open source projects, there is typically no central authority or control exerting dominance over the development process. Instead, decisions are collectively made through discussions and consensus-building. This consensus-driven approach can be time-consuming and necessitate compromise.
• Community Management: Open source projects rely heavily on community involvement and engagement. Managing the diverse community, encouraging participation, and fostering a positive and inclusive environment require effective community management strategies. A strong community ecosystem is important for the project's long-term sustainability.

While there are certainly additional challenges in the realm of open source, such as licensing and intellectual property, the aforementioned list should provide an adequate foundation for proceeding. Addressing these challenges is essential to the successful management of releases within the Kubernetes project.

Kubernetes Community Structure: SIG-Town Analogy

Inspired by Paris Pittman's insightful analogy in a recent episode of the Kubernetes Podcast by Google, the Kubernetes Community can be compared to a city. This analogy resonated and led to the creation of the SIG-Town diagram, illustrating the organizational structures within the community.

The Kubernetes Community is vast and diverse. The CNCF Devstats data highlights its scale, with nearly 3,000 individual contributors and approximately 60,000 commits recorded in the project over the past 12 months. Managing such a large and active community, driving decisions, fostering inclusivity, and ensuring effective collaboration can be a significant challenge. It's important to learn about the Kubernetes Community's different SIGs, roles, and processes. The SIG-Town analogy shows how big, complicated, and patient the Kubernetes Community is.

Collaborate

To make open-source projects work well together, it is important to set rules and guidelines that we all agree on. These rules help everyone work together in a good way.

Concepts of Cross-Cutting Collaboration

Improving collaboration is key in open-source projects like Kubernetes. Here are some concepts that help people work together in the community:

1. Everything is a file: every contribution is managed through the version control system, Git. This means that changes, discussions, and decision-making processes revolve around files within the project's repository. Recognizing and honoring the contributions made by project contributors is essential. But it's worth noting that in some discussions, it can be hard to keep everything in a markdown file or similar format. In such cases, opting for alternative platforms like public Google Docs files or GitHub issues proves more suitable and conducive to effective collaboration.
2. Ownership and Maintainer Roles: without maintainers taking ownership, the community would not be able to agree on decisions and ensure the project matures. Ownership is defined in files OWNERS that list approvers and reviewers. Approvers and reviewers take ownership of specific areas and contribute their expertise to guide and review contributions. There are other roles next to code approver and reviewer roles like SIG Chairs, Release Team Roles, etc. Serving in these roles implies a commitment to the community for a specific duration.
3. Meritocracy: recognition and influence are earned based on demonstrated skills, knowledge, and contributions. This approach emphasizes the importance of individual contributions rather than hierarchical positions or affiliations.
4. Transparency and Openness: includes open communication, decision-making processes, and community-driven initiatives. Encouraging participation from others and giving them a platform to express their opinions and concerns is vital. It is important to avoid private direct messages (DMs) and ensure that all communication happens publicly.

Collaboration in open-source projects is a vast topic, and there is much more to explore. However, understanding these key concepts provides a solid foundation for comprehending the Kubernetes release processes and how collaboration operates within the community.

Kubernetes Enhancement Proposals (KEPs) - The "Feature Design Contract"

In the Kubernetes Project, the management of contributions is facilitated through the creation of Kubernetes Enhancement Proposals (KEPs). A KEP serves as a design document that outlines the desired feature to be implemented. It acts as a contractual agreement, allowing for evaluation, discussion, and assessment of proposed changes. The KEP process is mandatory for any modifications made to Kubernetes. Simply opening a pull request (PR) to introduce support for a new protocol or similar changes is not sufficient. To ensure proper documentation, the provided template below illustrates the required structure and information that needs to be included in a KEP.

    Summary
    Motivation
    - Goals
    - Non-Goals
    Proposal
    - User Stories (Optional)
    - Notes/Constraints/Caveats (Optional)
    - Risks and Mitigations
    Design Details
    - Test Plan
    	- Prerequisite testing updates
    	- Unit tests
    	- Integration tests
    	- e2e tests
    - Graduation Criteria
    - Upgrade / Downgrade Strategy
    - Version Skew Strategy
    Production Readiness Review Questionnaire
    - Feature Enablement and Rollback
    - Rollout, Upgrade and Rollback Planning
    - Monitoring Requirements
    - Dependencies
    - Scalability
    - Troubleshooting
    Implementation History
    Drawbacks
    Alternatives
    Infrastructure Needed (Optional)

KEPs follow the concept and principles of the Google Design Document. Just like Google Design Documents, KEPs serve as a structured framework for proposing and documenting feature enhancements in the Kubernetes Project. They provide a systematic approach for outlining the design, rationale, and implementation details of a proposed feature.

Each Kubernetes Special Interest Group (SIG) is responsible for managing and overseeing its respective Kubernetes Enhancement Proposals (KEPs). These KEPs can be in various stages of development within the SIG's purview. They may be under discussion, being considered for inclusion in a release (opted-in), or have already been merged and categorized as alpha, beta, or stable, indicating different levels of maturity and readiness. KEPs can vary significantly in size, but generally, smaller KEPs that enable atomic changes are preferred.

SIG-Release

As mentioned earlier, managing and monitoring all Kubernetes Enhancement Proposals (KEPs) and related discussions is an incredibly challenging task. SIG-Release takes on this responsibility, as outlined in its charter. The primary objectives of SIG-Release are to ensure the delivery of high-quality Kubernetes releases, continuously enhance release and development processes, and foster collaboration with downstream communities that utilize Kubernetes releases to build their own artifacts.

Release Team Structure

Within sig-release, there are two teams: the release engineering team, responsible for developing release tooling, with a more stable team of contributors, and the release team, which changes with each cycle. The primary role of the release team is to facilitate the release process.

Referencing the diagram above, the team consists of Leads and Shadows who oversee different areas of interest throughout the release process. The shadow program serves as an onboarding pathway to Sig-release, the Kubernetes community, and the wider open-source realm. You can contribute to a release cycle by assisting in tasks such as collecting enhancements, monitoring test pipelines, and more. You can find out more about the release team roles and team here.

Contributing to open-source projects extends beyond coding. You can make valuable contributions through documentation, community engagement, operations, governance, infrastructure work, and various other areas of work. The release team is a prime example.

Overview of the Release Cycle

A typical Kubernetes release cycle spans approximately three months and consists of several key deadlines throughout this period. You can find the release timeline for the v1.28 release cycle here. In past releases, the milestones and deadlines have not changed significantly, and the schedule is fairly consistent.

• Assemble Release Team: The release team begins its work, and individuals have the opportunity to apply and join the team each cycle by participating in a survey.
• Start Enhancement Tracking: SIGs opt-in their KEPs in the release cycle. The release team makes sure that opted in KEPs fulfill all requirements to be included in the release cycle.
• Enhancement Freeze: KEPs need to be merged up until enhancements freeze hits. All KEPs that do not make it in time need to file an exception. With enhancements freeze the release team know which changes will be made.
• Feature Blog Freeze: KEP authors have the option to indicate their interest in writing a blog post about their respective changes.
• Code Freeze: All code changes which related to approved KEPs must be successfully merged. After Code freeze pull requests will not be merged and need to file an exception which gets evaluated case by case by the release lead.
• Major Themes Deadline: Marks the deadline until the team collects significant changes which are set to be made in the release cycle. These major themes are highlighted to the media.
• Final Release Cut: The final release artifacts are pushed, the release blog is published, the version is announced through communication channels, the documentation is updated, and the repository restrictions are lifted.

Every Kubernetes Release Cycle demands a significant amount of effort. It involves numerous individuals, multiple deadlines, and a multitude of tasks that must be completed. The release lead is responsible for maintaining a comprehensive tracking issue for the entire release cycle, which currently includes more than 100 checkboxes representing various “formal” tasks. Additionally, a considerable amount of extra work, especially in terms of communication and enhancing the release team's performance from one cycle to another, is necessary.

During my tenure as the release team lead for version 1.26, I vividly recall receiving a notification from the Slack bot that I have sent over 600 messages in a single week, which clearly indicates the substantial workload involved. The team continuously strives to enhance the overall experience and automate tasks as much as possible, but such improvements require time and iterative efforts across multiple cycles.

Incorporating KEPs into the release process: How does it work?

Ensuring the inclusion of KEPs (Kubernetes Enhancement Proposals) in a Kubernetes release involves two key aspects that deserve attention. Firstly, there are tasks that the KEP author must undertake, and secondly, there are deadlines and tasks that the release team is responsible for.

To begin with, it is crucial for the KEP author to engage in discussions about the KEP with their relevant SIG (Special Interest Group). The SIG leads plays a pivotal role in opting the KEP into the release. The KEP author must diligently complete all the necessary tasks to prepare the KEP for inclusion. This includes obtaining approval and merging from the SIG, opening pull requests (PRs) to update the code, ensuring code review and approval, and updating the documentation. Additionally, the KEP author may choose to write a blog post to provide further insights. The release team ensures that the KEP author completes these steps and closely monitors the process to prevent the SIG leads responsible for the KEP from missing anything.

Why are certain features not included in releases?

It is important to remember that most maintainers contribute to the project in their spare time. This is a volunteer work and they are free to spend as much time as they want on things of their choice.

If you notice a missing feature, you have the opportunity to contribute yourself. Contact your engineering manager at work and explain the significance of this feature for your company's IT systems. Investing in the Kubernetes project not only benefits your company but also allows you to collaborate with talented engineers and potentially gain valuable knowledge in the process. Moreover, your contributions can benefit the wider cloud native community, as they may find your changes extremely useful.

That’s it! If you would like to contact me, the best way to do so is via Slack in the Kubernetes or CNCF workspaces.