Commit Graph

29 Commits

Author SHA1 Message Date
Brad Davidson
6bbdf19a10 Fix the OTHER log message that prints the wrong variable
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit 1e663622d2)
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2024-01-11 16:30:56 -08:00
Brad Davidson
1aa0eac285 Print key instead of file path in snapshot metadata log message
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-11-21 14:49:38 -08:00
Brad Davidson
0473839f16 Don't request metadata when listing objects
While some implementations may support it, it appears that most don't,
and some may in fact return an error if it is requested.

We already stat the object to get the metadata anyway, so this was
unnecessary if harmless on most implementations.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-11-21 14:49:38 -08:00
Brad Davidson
6f489f08f9 Fix issue with snapshot metadata configmap
Omit snapshot list configmap entries for snapshots without extra metadata; reduce log level of warnings about missing s3 metadata files.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit 2088218c5f)
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-11-16 09:55:41 -08:00
Brad Davidson
b6f23014b2 Don't try to read token hash and cluster id during cluster-reset
These fields are only necessary when saving snapshots to S3, and will block restoration if attempted

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-10-27 15:06:50 -07:00
Brad Davidson
52c7d6c91d Add server token hash to CR and S3
This required pulling the token hash stuff out of the cluster package, into util.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit d885162967)
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-10-13 11:09:28 -07:00
Brad Davidson
e7014cf859 Switch to managing ETCDSnapshotFile resources
Reconcile snapshot CRs instead of ConfigMap; manage ConfigMap downstream from CR list

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit 550ab36ab7)
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-10-13 11:09:28 -07:00
Brad Davidson
22c0916abd Move snapshot delete into local/s3 functions
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit 5cd4f69bfa)
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-10-13 11:09:28 -07:00
Brad Davidson
df0fd0de49 Store extra metadata and cluster ID for snapshots
Write the extra metadata both locally and to S3. These files are placed such that they will not be used by older versions of K3s that do not make use of them.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit 7464007037)
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-10-13 11:09:28 -07:00
Brad Davidson
7375740ed4 Move s3 snapshot list functionality to s3.go
Also, don't list ONLY s3 snapshots if S3 is enabled.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit 80f909d0ca)
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-10-13 11:09:28 -07:00
Brad Davidson
487b3948a3 Consistently set snapshotFile timestamp
Attempt to use timestamp from creation or filename instead of file/object modification times

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit 8d47645312)
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-10-13 11:09:28 -07:00
Brad Davidson
d270e97825 Tidy s3 upload functions
Consistently refer to object keys as such, simplify error handling.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit f1afe153a3)
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-10-13 11:09:28 -07:00
Brad Davidson
7a0eecd601 Elide old snapshot data when apiserver rejects configmap with ErrRequestEntityTooLarge
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
(cherry picked from commit 2b0e2e8ada)
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2023-10-13 11:09:28 -07:00
Vitor Savian
e83b1ba4aa Fixed the etcd retention to delete orphaned snapshots based on the date (#8177)
* Fix retention using name instead of date

Signed-off-by: Vitor <vitor.savian@suse.com>
2023-08-14 18:48:59 -03:00
Vitor Savian
ca7aeed090 Etcd snapshots retention when node name changes (#8099)
Fixed the etcd retention to delete orphaned snapshots

Signed-off-by: Vitor <vitor.savian@suse.com>
2023-08-03 10:54:40 -03:00
Derek Nola
06d81cb936 Replace deprecated ioutil package (#6230)
* Replace ioutil package
* check integration test null pointer
* Remove rotate retries

Signed-off-by: Derek Nola <derek.nola@suse.com>
2022-10-07 17:36:57 -07:00
Brad Davidson
ffe72eecc4 Address issues with etcd snapshots
* Increase the default snapshot timeout. The timeout is not currently
  configurable from Rancher, and larger clusters are frequently seeing
  uploads fail at 30 seconds.
* Enable compression for scheduled snapshots if enabled on the
  command-line. The CLI flag was not being passed into the etcd config.
* Only set the S3 content-type to application/zip if the file is zipped.
* Don't run more than one snapshot at once, to prevent misconfigured
  etcd snapshot cron schedules from stacking up.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
2022-07-12 14:41:38 -07:00
Luther Monson
9a849b1bb7 [master] changing package to k3s-io (#4846)
* changing package to k3s-io

Signed-off-by: Luther Monson <luther.monson@gmail.com>

Co-authored-by: Derek Nola <derek.nola@suse.com>
2022-03-02 15:47:27 -08:00
Brian Downs
effcb15adb Adds the ability to compress etcd snapshots (#4866) 2022-01-14 10:31:22 -07:00
Chris Kim
ae4a1a144a etcd snapshot functionality enhancements (#4453)
Signed-off-by: Chris Kim <oats87g@gmail.com>
2021-11-29 10:30:04 -08:00
Brian Downs
0452f017c1 Add etcd s3 timeout (#4207) 2021-10-15 10:24:14 -07:00
Brian Downs
f4cea90cb9 set transport to skip verify if se skip flag passed (#4102) 2021-09-28 10:13:50 -07:00
Devin Buhl
a1ec43e0b7 feat: add option to disable s3 over https
Signed-off-by: Devin Buhl <devin.kray@gmail.com>
2021-09-05 12:03:49 -04:00
Derek Nola
b4eca61aeb Prevent snapshot commands from creating empty snapshot directory (#3783)
Signed-off-by: dereknola <derek.nola@suse.com>
2021-08-09 09:04:18 -07:00
Derek Nola
bba49ea447 Fix to allow prune to correctly cleanup custom named snapshots (#3649)
Signed-off-by: dereknola <derek.nola@suse.com>
2021-07-19 14:30:57 -07:00
Hussein Galal
f5fbb9a9a8 Export cli server flags and etcd restoration functions (#3527)
* Export cli server flags and etfd restoration functions

Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>

* export S3

Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>
2021-06-30 22:29:03 +02:00
Brian Downs
e998cd110d Add the ability to delete an etcd snapshot locally or from S3 (#3277)
* Add the ability to delete a given set of etcd snapshots from the CLI for locally stored and S3 store snapshots.
2021-05-07 16:10:04 -07:00
Brian Downs
c5ad71ce0b Collect and Store etcd Snapshots and Metadata (#3239)
* Add the ability to store local etcd snapshots and etcd snapshots stored in an S3 compatible object store in a ConfigMap.
2021-04-30 18:26:39 -07:00
Brian Downs
4d1f9eda9d Etcd Snapshot/Restore to/from S3 Compatible Backends (#2902)
* Add functionality for etcd snapshot/restore to and from S3 compatible backends.
* Update etcd restore functionality to extract and write certificates and configs from snapshot.
2021-03-03 11:14:12 -07:00