Commit Graph

168 Commits

Author SHA1 Message Date
Oleg Matskiv
e60f7ecc19 Don't verify the node password if the local host is not running an agent
Signed-off-by: Oleg Matskiv <oleg.matskiv@gmail.com>
(cherry picked from commit e3b237fc35)
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2024-02-10 00:49:54 -08:00
Brad Davidson
893eabd128 Bump runc and helm-controller versions
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2024-02-01 18:52:32 -08:00
Aofei Sheng
65495bb65e Use ipFamilyPolicy: RequireDualStack for dual-stack kube-dns (#8984)
Signed-off-by: Aofei Sheng <aofei@aofeisheng.com>
(cherry picked from commit 8d2c40cdac)
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2024-01-18 13:52:28 -08:00
Derek Nola
8a00e7ab32 Add a retry around updating a secrets-encrypt node annotations (#9123)
* Add a retry around updating a se node annotations

Signed-off-by: Derek Nola <derek.nola@suse.com>
2024-01-04 09:51:30 -08:00
Brad Davidson
dba56d046c Add jitter to client config retry
Also:
* Replaces labeled for/continue RETRY loops with wait helpers for improved readability
* Pulls secrets and nodes from cache for node password verification
* Migrate nodepassword tests to wrangler mocks for better code reuse

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-11-16 09:56:05 -08:00
Harsimran Singh Maan
0c9b48f4ef Disable helm CRD installation for disable-helm-controller (#8702)
* Disable helm CRD installation for disable-helm-controller
    The NewContext package requires config as input which would
    require all third-party callers to update when the new go module
    is published.

    This change only affects the behaviour of installation of helm
    CRDs. Existing helm crds installed in a cluster would not be removed
    when disable-helm-controller flag is set on the server.

    Addresses #8701
* address review comments
* remove redundant check

Signed-off-by: Harsimran Singh Maan <maan.harry@gmail.com>
(cherry picked from commit abc2efdd57)
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-11-16 09:56:05 -08:00
Brad Davidson
09ab9b7e21 Store extra metadata and cluster ID for snapshots
Write the extra metadata both locally and to S3. These files are placed such that they will not be used by older versions of K3s that do not make use of them.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit 7464007037)
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-10-13 10:23:51 -07:00
Derek Nola
ad6579a43f [Release-1.26] Clear remove annotations on cluster reset (#8590)
* Use admin kubeconfig instead of supervisor for etcd snapshot CLI

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

* Skip creating CRDs and setting up event recorder for CLI controller context

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

* Don't export functions not needed outside the etcd package

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

* Reorganize Driver interface and etcd driver to avoid passing context and config into most calls

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

* Clear remove annotations on cluster reset; refuse to delete last member from cluster

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

---------

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
Co-authored-by: Brad Davidson <brad.davidson@rancher.com>
2023-10-11 16:17:44 -07:00
Derek Nola
8acba4a627 Server Token Rotation (#8577)
* Consolidate NewCertCommands
* Add support for user defined new token
* Add E2E testlets



* Ensure agent token also changes

Signed-off-by: Derek Nola <derek.nola@suse.com>
2023-10-10 09:45:36 -07:00
Manuel Buil
915120c1f0 ipFamilyPolicy:PreferDualStack for coredns and metrics-server
Signed-off-by: Manuel Buil <mbuil@suse.com>
2023-10-02 11:34:55 +02:00
Brad Davidson
e12600ea27 Add FilterCN function to prevent SAN Stuffing
Wire up a node watch to collect addresses of server nodes, to prevent adding unauthorized SANs to the dynamiclistener cert.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit aa76942d0f)
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-08-04 16:09:40 -07:00
Vitor Savian
e200425876 Adding cli to custom klipper helm image (#7682)
Adding cli to custom klipper helm image

Signed-off-by: Vitor Savian <vitor.savian@suse.com>
(cherry picked from commit 0809187cff)
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-07-07 16:28:01 -07:00
Derek Nola
87cb16e618 Fix rootless node password (#7901)
Signed-off-by: Derek Nola <derek.nola@suse.com>
2023-07-07 13:13:00 -07:00
Derek Nola
6afe691c62 Allow k3s to customize apiServerPort on helm-controller (#7874)
Signed-off-by: Daishan Peng <daishan@acorn.io>
Co-authored-by: Daishan Peng <daishan@acorn.io>
2023-07-05 14:50:06 -07:00
Brad Davidson
3592729e03 Improve error response logging
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit 5170bc5a04)
2023-06-12 10:53:09 -07:00
Brad Davidson
725aaa94e7 Soft-fail on node password verification if the secret cannot be created
Allows nodes to join the cluster during a webhook outage. This also
enhances auditability by creating Kubernetes events for the deferred
verification.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit 45d8c1a1a2)
2023-06-12 10:53:09 -07:00
Brad Davidson
1853afe124 Create new kubeconfig for supervisor use
Only actual admin actions should use the admin kubeconfig; everything done by the supervisor/deploy/helm controllers will now use a distinct account for audit purposes.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit 64a5f58f1e)
2023-06-12 10:53:09 -07:00
Brad Davidson
46631ef963 Use distinct clients for supervisor, deploy, and helm controllers
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit 8748813a61)
2023-06-12 10:53:09 -07:00
Brad Davidson
8312c171b2 Bump helm-controller version for repo auth/ca support
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-05-10 17:37:14 -07:00
Brad Davidson
03a3470193 Consistently use constant-time comparison of password hashes
As per https://github.com/golang/go/issues/47001 even subtle.ConstantTimeCompare should never be used with variable-length inputs, as it will return 0 if the lengths do not match. Switch to consistently using constant-time comparisons of hashes for password checks to avoid any possible side-channel leaks that could be combined with other vectors to discover password lengths.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit 239021e759)
2023-05-10 17:37:14 -07:00
Brad Davidson
82a403a0c2 Fix stack log on panic
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit f1b6a3549c)
2023-05-10 17:37:14 -07:00
Brad Davidson
bdd210e785 Fix race condition in tunnel server startup
Several places in the code used a 5-second retry loop to wait on
Runtime.Core to be set. This caused a race condition where OnChange
handlers could be added after the Wrangler shared informers were already
started. When this happened, the handlers were never called because the
shared informers they relied upon were not started.

Fix that by requiring anything that waits on Runtime.Core to run from a
cluster controller startup hook that is guaranteed to be called before
the shared informers are started, instead of just firing it off in a
goroutine that retries until it is set.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit c44d33d29b)
2023-05-10 17:37:14 -07:00
Brad Davidson
0861ce2d97 Create CRDs with schema
Fixes an issue where CRDs were being created without schema, allowing
resources with invalid content to be created, later stalling the
controller ListWatch event channel when the invalid resources could not
be deserialized.

This also requires moving Addon GVK tracking from a status field to
an annotation, as the GroupVersionKind type has special handling
internal to Kubernetes that prevents it from being serialized to the CRD
when schema validation is enabled.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit ad41fb8c96)
2023-05-10 17:37:14 -07:00
Brad Davidson
977a85559e Add support for cross-signing new certs during ca rotation
We need to send the full chain in order for cross-signing to work
properly during switchover to a new root.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-03-13 16:56:28 -07:00
Brad Davidson
0c302f4341 Fix etcd member deletion
Turns out etcd-only nodes were never running **any** of the controllers,
so allowing multiple controllers didn't really fix things.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-02-14 09:39:41 -08:00
Brad Davidson
3d146d2f1b Allow for multiple sets of leader-elected controllers
Addresses an issue where etcd controllers did not run on etcd-only nodes

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-02-10 10:46:48 -08:00
Brad Davidson
87f9c4ab11 Ensure that node exists when using node auth
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-02-07 14:55:04 -08:00
Brad Davidson
373df1c8b0 Add support for k3s token command
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-02-07 14:55:04 -08:00
Brad Davidson
215fb157ff Add certificate rotate-ca to write updated CA certs to datastore
This command must be run on a server while the service is running. After this command completes, all the servers in the cluster should be restarted to load the new CA files.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-02-06 15:09:31 -08:00
Manuel Buil
557fcd28d5 Change the priority of address types depending on flannel-external-ip
Signed-off-by: Manuel Buil <mbuil@suse.com>
2022-11-04 09:02:39 +01:00
Brad Davidson
8d28a38a18 Update helm-controller to pull in refactor
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2022-10-26 15:03:13 -07:00
Derek Nola
06d81cb936 Replace deprecated ioutil package (#6230)
* Replace ioutil package
* check integration test null pointer
* Remove rotate retries

Signed-off-by: Derek Nola <derek.nola@suse.com>
2022-10-07 17:36:57 -07:00
Brad Davidson
0b96ca92bc Move servicelb into cloudprovider LoadBalancer interface
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2022-09-30 08:17:20 -07:00
Brad Davidson
a15e7e8b68 Move DisableServiceLB/Rootless/ServiceLBNamespace into config.Control
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2022-09-30 08:17:20 -07:00
Vladimir Kochnev
13af0b1d88 Save agent token to /var/lib/rancher/k3s/server/agent-token
Having separate tokens for server and agent nodes is a nice feature.

However, passing server's plain `K3S_AGENT_TOKEN` value
to `k3s agent --token` without CA hash is insecure when CA is
self-signed, and k3s warns about it in the logs:

```
Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash.
Use the full token from the server's node-token file to enable Cluster CA validation.
```

Okay so I need CA hash but where should I get it?

This commit attempts to fix this issue by saving agent token value to
`agent-token` file with CA hash appended.

Signed-off-by: Vladimir Kochnev <hashtable@yandex.ru>
2022-08-01 14:11:50 -07:00
Brad Davidson
5eaa0a9422 Replace getLocalhostIP with Loopback helper method
Requires tweaking existing method signature to allow specifying whether or not IPv6 addresses should be return URL-safe.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2022-07-21 16:51:57 -07:00
Derek Nola
a9b5a1933f Delay service readiness until after startuphooks have finished (#5649)
* Move startup hooks wg into a runtime pointer, check before notifying systemd
* Switch default systemd notification to server
* Add 1 sec delay to allow etcd to write to disk
Signed-off-by: Derek Nola <derek.nola@suse.com>
2022-06-15 09:00:52 -07:00
Darren Shepherd
e6009b1edf Introduce servicelb-namespace parameter
This parameter controls which namespace the klipper-lb pods will be create.
It defaults to kube-system so that k3s does not by default create a new
namespace. It can be changed if users wish to isolate the pods and apply
some policy to them.

Signed-off-by: Darren Shepherd <darren@acorn.io>
2022-06-14 15:48:58 -07:00
Brad Davidson
ce5b9347c9 Replace DefaultProxyDialerFn dialer injection with EgressSelector support
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2022-04-29 17:54:36 -07:00
Brad Davidson
3d01ca1309 Make supervisor errors parsable by Kubernetes client libs
This gives nicer errors from Kubernetes components during startup, and
reduces LOC a bit by using the upstream responsewriters module instead
of writing the headers and body by hand.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2022-04-29 09:23:37 -07:00
Brad Davidson
b12cd62935 Move IPv4/v6 selection into helpers
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2022-04-15 01:02:42 -07:00
Brad Davidson
5b2c14b123 Print a helpful error when trying to join additional servers but etcd is not in use
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2022-04-15 01:02:42 -07:00
Brad Davidson
99851b0f84 Use core constants for cert user/group values
Also update cert gen to ensure leaf certs are regenerated if other key fields change.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2022-04-15 01:02:42 -07:00
Brad Davidson
f37e7565b8 Move the apiserver addresses controller into the etcd package
This controller only needs to run when using managed etcd, so move it in
with the rest of the etcd stuff. This change also modifies the
controller to only watch the Kubernetes service endpoint, instead of
watching all endpoints in the entire cluster.

Fixes an error message revealed by use of a newer grpc client in
Kubernetes 1.24, which logs an error when the Put to etcd failed because
kine doesn't support the etcd Put operation. The controller shouldn't
have been running without etcd in the first place.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2022-04-07 11:28:15 -07:00
Brad Davidson
49544e0d49 Allow agents to query non-apiserver supervisors for apiserver endpoints
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2022-04-06 13:03:14 -07:00
Roberto Bonafiglia
4afeb9c5c7 Merge pull request #5325 from rbrtbnfgl/fix-etcd-ipv6-url
Fixed etcd URL in case of IPv6 address
2022-04-05 09:55:42 +02:00
Roberto Bonafiglia
dda409b041 Updated localhost address on IPv6 only setup
Signed-off-by: Roberto Bonafiglia <roberto.bonafiglia@suse.com>
2022-03-29 09:35:54 +02:00
Brad Davidson
e811689df9 Fix etcd-only secrets encryption rotation
Improve feedback when running secrets-encrypt commands on etcd-only nodes, and
allow etcd-only nodes to properly restart when effecting rotation.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2022-03-25 10:40:58 -07:00
Brad Davidson
38706eeec0 Defer ensuring node passwords on etcd-only nodes during initial cluster bootstrap
This allows secondary etcd nodes to bootstrap the kubelet before an
apiserver joins the cluster. Rancher waits for all the etcd nodes to
come up before adding the control-plane nodes, so this needs to be
handled properly.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2022-03-18 10:58:37 -07:00
Brad Davidson
a93b9b6d53 Update helm-controller
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2022-03-16 23:49:14 -07:00