README.md 13.3 KB
Newer Older
Adam Harrison's avatar
Adam Harrison committed
1

Daniel Holbach's avatar
Daniel Holbach committed
2
3
# kured - Kubernetes Reboot Daemon

Adam Harrison's avatar
Adam Harrison committed
4
5
<img src="https://github.com/weaveworks/kured/raw/master/img/logo.png" align="right"/>

Adam Harrison's avatar
Adam Harrison committed
6
* [Introduction](#introduction)
Adam Harrison's avatar
Adam Harrison committed
7
8
* [Kubernetes & OS Compatibility](#kubernetes-&-os-compatibility)
* [Installation](#installation)
Adam Harrison's avatar
Adam Harrison committed
9
* [Configuration](#configuration)
Daniel Holbach's avatar
Daniel Holbach committed
10
11
12
13
14
15
16
  * [Reboot Sentinel File & Period](#reboot-sentinel-file-&-period)
  * [Setting a schedule](#setting-a-schedule)
  * [Blocking Reboots via Alerts](#blocking-reboots-via-alerts)
  * [Blocking Reboots via Pods](#blocking-reboots-via-pods)
  * [Prometheus Metrics](#prometheus-metrics)
  * [Slack Notifications](#slack-notifications)
  * [Overriding Lock Configuration](#overriding-lock-configuration)
Adam Harrison's avatar
Adam Harrison committed
17
* [Operation](#operation)
Daniel Holbach's avatar
Daniel Holbach committed
18
19
20
  * [Testing](#testing)
  * [Disabling Reboots](#disabling-reboots)
  * [Manual Unlock](#manual-unlock)
Michal Schott's avatar
Michal Schott committed
21
  * [Automatic Unlock](#automatic-unlock)
Adam Harrison's avatar
Adam Harrison committed
22
* [Building](#building)
Adam Harrison's avatar
Adam Harrison committed
23
* [Frequently Asked/Anticipated Questions](#frequently-askedanticipated-questions)
Adam Harrison's avatar
Adam Harrison committed
24
* [Getting Help](#getting-help)
Adam Harrison's avatar
Adam Harrison committed
25
26
27
28

## Introduction

Kured (KUbernetes REboot Daemon) is a Kubernetes daemonset that
Adam Harrison's avatar
Adam Harrison committed
29
30
performs safe automatic node reboots when the need to do so is
indicated by the package management system of the underlying OS.
Adam Harrison's avatar
Adam Harrison committed
31

32
* Watches for the presence of a reboot sentinel e.g. `/var/run/reboot-required`
Adam Harrison's avatar
Adam Harrison committed
33
34
* Utilises a lock in the API server to ensure only one node reboots at
  a time
35
* Optionally defers reboots in the presence of active Prometheus alerts or selected pods
Adam Harrison's avatar
Adam Harrison committed
36
37
* Cordons & drains worker nodes before reboot, uncordoning them after

Adam Harrison's avatar
Adam Harrison committed
38
39
## Kubernetes & OS Compatibility

Daniel Holbach's avatar
Daniel Holbach committed
40
41
42
43
44
The daemon image contains versions of `k8s.io/client-go` and
`k8s.io/kubectl` (the binary of `kubectl` in older releases) for the purposes of
maintaining the lock and draining worker nodes. Kubernetes aims to provide
forwards and backwards compatibility of one minor version between client and
server:
45
46
47

| kured  | kubectl | k8s.io/client-go | k8s.io/apimachinery | expected kubernetes compatibility |
|--------|---------|------------------|---------------------|-----------------------------------|
Daniel Holbach's avatar
Daniel Holbach committed
48
| master | 1.19.4  | v0.19.4          | v0.19.4             | 1.17.x, 1.18.x, 1.19.x            |
Daniel Holbach's avatar
Daniel Holbach committed
49
| 1.5.1  | 1.18.8  | v0.18.8          | v0.18.8             | 1.17.x, 1.18.x, 1.19.x            |
Daniel Holbach's avatar
Daniel Holbach committed
50
| 1.4.4  | 1.17.7  | v0.17.0          | v0.17.0             | 1.16.x, 1.17.x, 1.18.x            |
Daniel Holbach's avatar
Daniel Holbach committed
51
| 1.3.0  | 1.15.10 | v12.0.0          | release-1.15        | 1.15.x, 1.16.x, 1.17.x            |
Adam Harrison's avatar
Adam Harrison committed
52
| 1.2.0  | 1.13.6  | v10.0.0          | release-1.13        | 1.12.x, 1.13.x, 1.14.x            |
53
| 1.1.0  | 1.12.1  | v9.0.0           | release-1.12        | 1.11.x, 1.12.x, 1.13.x            |
Daniel Holbach's avatar
Daniel Holbach committed
54
| 1.0.0  | 1.7.6   | v4.0.0           | release-1.7         | 1.6.x, 1.7.x, 1.8.x               |
55
56
57
58

See the [release notes](https://github.com/weaveworks/kured/releases)
for specific version compatibility information, including which
combination have been formally tested.
Adam Harrison's avatar
Adam Harrison committed
59

Adam Harrison's avatar
Adam Harrison committed
60
61
Versions >=1.1.0 enter the host mount namespace to invoke
`systemctl reboot`, so should work on any systemd distribution.
Adam Harrison's avatar
Adam Harrison committed
62
63
64
65
66
67

## Installation

To obtain a default installation without Prometheus alerting interlock
or Slack notifications:

Daniel Holbach's avatar
Daniel Holbach committed
68
```console
69
latest=$(curl -s https://api.github.com/repos/weaveworks/kured/releases | jq -r .[0].tag_name)
70
kubectl apply -f "https://github.com/weaveworks/kured/releases/download/$latest/kured-$latest-dockerhub.yaml"
Adam Harrison's avatar
Adam Harrison committed
71
72
73
74
75
```

If you want to customise the installation, download the manifest and
edit it in accordance with the following section before application.

Adam Harrison's avatar
Adam Harrison committed
76
77
78
79
## Configuration

The following arguments can be passed to kured via the daemonset pod template:

Daniel Holbach's avatar
Daniel Holbach committed
80
```console
Adam Harrison's avatar
Adam Harrison committed
81
Flags:
82
83
84
85
      --alert-filter-regexp regexp.Regexp   alert names to ignore when checking for active alerts
      --blocking-pod-selector stringArray   label selector identifying pods whose presence should prevent reboots
      --ds-name string                      name of daemonset on which to place lock (default "kured")
      --ds-namespace string                 namespace containing daemonset on which to place lock (default "kube-system")
David Sauer's avatar
David Sauer committed
86
      --end-time string                     schedule reboot only before this time of day (default "23:59:59")
87
88
  -h, --help                                help for kured
      --lock-annotation string              annotation in which to record locking node (default "weave.works/kured-node-lock")
David Sauer's avatar
David Sauer committed
89
90
91
      --lock-ttl duration                   expire lock annotation after this duration (default: 0, disabled)
      --message-template-drain string       message template used to notify about a node being drained (default "Draining node %s")
      --message-template-reboot string      message template used to notify about a node being rebooted (default "Rebooting node %s")
92
      --period duration                     reboot check period (default 1h0m0s)
David Sauer's avatar
David Sauer committed
93
      --prefer-no-schedule-taint string     Taint name applied during pending node reboot (to prevent receiving additional pods from other rebooting nodes). Set to "" to disable tainting. (default "weave.works/kured-node-reboot")
94
      --prometheus-url string               Prometheus instance to probe for active alerts
David Sauer's avatar
David Sauer committed
95
      --reboot-days strings                 schedule reboot on these days (default [su,mo,tu,we,th,fr,sa])
96
      --reboot-sentinel string              path to file whose existence signals need to reboot (default "/var/run/reboot-required")
97
      --slack-channel string                slack channel for reboot notfications
98
99
      --slack-hook-url string               slack hook URL for reboot notfications
      --slack-username string               slack username for reboot notfications (default "kured")
David Sauer's avatar
David Sauer committed
100
101
102
      --start-time string                   schedule reboot only after this time of day (default "0:00")
      --time-zone string                    use this timezone for schedule inputs (default "UTC")

Adam Harrison's avatar
Adam Harrison committed
103
104
105
106
107
108
```

### Reboot Sentinel File & Period

By default kured checks for the existence of
`/var/run/reboot-required` every sixty minutes; you can override these
Adam Harrison's avatar
Adam Harrison committed
109
110
values with `--reboot-sentinel` and `--period`. Each replica of the
daemon uses a random offset derived from the period on startup so that
Adam Harrison's avatar
Adam Harrison committed
111
112
nodes don't all contend for the lock simultaneously.

113
114
115
116
117
118
119
120
121
### Setting a schedule

By default, kured will reboot any time it detects the sentinel, but this
may cause reboots during odd hours.  While service disruption does not
normally occur, anything is possible and operators may want to restrict
reboots to predictable schedules.  Use `--reboot-days`, `--start-time`,
`--end-time`, and `--time-zone` to set a schedule.  For example, business
hours on the west coast USA can be specified with:

Daniel Holbach's avatar
Daniel Holbach committed
122
123
124
125
126
```console
  --reboot-days mon,tue,wed,thu,fri
  --start-time 9am
  --end-time 5pm
  --time-zone America/Los_Angeles
127
128
129
130
131
132
133
134
135
```

Times can be formatted in numerous ways, including `5pm`, `5:00pm` `17:00`,
and `17`.  `--time-zone` represents a Go `time.Location`, and can be `UTC`,
`Local`, or any entry in the standard Linux tz database.

Note that when using smaller time windows, you should consider shortening
the sentinel check period (`--period`).

Adam Harrison's avatar
Adam Harrison committed
136
137
138
139
140
141
### Blocking Reboots via Alerts

You may find it desirable to block automatic node reboots when there
are active alerts - you can do so by providing the URL of your
Prometheus server:

Daniel Holbach's avatar
Daniel Holbach committed
142
```console
Adam Harrison's avatar
Adam Harrison committed
143
144
145
146
147
148
--prometheus-url=http://prometheus.monitoring.svc.cluster.local
```

By default the presence of *any* active (pending or firing) alerts
will block reboots, however you can ignore specific alerts:

Daniel Holbach's avatar
Daniel Holbach committed
149
```console
Adam Harrison's avatar
Adam Harrison committed
150
151
152
--alert-filter-regexp=^(RebootRequired|AnotherBenignAlert|...$
```

153
154
155
156
157
158
159
160
See the section on Prometheus metrics for an important application of this
filter.

### Blocking Reboots via Pods

You can also block reboots of an _individual node_ when specific pods
are scheduled on it:

Daniel Holbach's avatar
Daniel Holbach committed
161
```console
162
163
164
165
166
167
--blocking-pod-selector=runtime=long,cost=expensive
```

Since label selector strings use commas to express logical 'and', you can
specify this parameter multiple times for 'or':

Daniel Holbach's avatar
Daniel Holbach committed
168
```console
169
170
171
172
173
174
175
176
177
178
179
--blocking-pod-selector=runtime=long,cost=expensive
--blocking-pod-selector=name=temperamental
```

In this case, the presence of either an (appropriately labelled) expensive long
running job or a known temperamental pod on a node will stop it rebooting.

> Try not to abuse this mechanism - it's better to strive for
> restartability where possible. If you do use it, make sure you set
> up a RebootRequired alert as described in the next section so that
> you can intervene manually if reboots are blocked for too long.
Adam Harrison's avatar
Adam Harrison committed
180
181
182
183
184
185

### Prometheus Metrics

Each kured pod exposes a single gauge metric (`:8080/metrics`) that
indicates the presence of the sentinel file:

Daniel Holbach's avatar
Daniel Holbach committed
186
```console
Adam Harrison's avatar
Adam Harrison committed
187
188
189
# HELP kured_reboot_required OS requires reboot due to software updates.
# TYPE kured_reboot_required gauge
kured_reboot_required{node="ip-xxx-xxx-xxx-xxx.ec2.internal"} 0
Adam Harrison's avatar
Adam Harrison committed
190
191
```

Adam Harrison's avatar
Adam Harrison committed
192
193
194
195
The purpose of this metric is to power an alert which will summon an
operator if the cluster cannot reboot itself automatically for a
prolonged period:

Daniel Holbach's avatar
Daniel Holbach committed
196
```console
Adam Harrison's avatar
Adam Harrison committed
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
# Alert if a reboot is required for any machines. Acts as a failsafe for the
# reboot daemon, which will not reboot nodes if there are pending alerts save
# this one.
ALERT RebootRequired
  IF          max(kured_reboot_required) != 0
  FOR         24h
  LABELS      { severity="warning" }
  ANNOTATIONS {
    summary = "Machine(s) require being rebooted, and the reboot daemon has failed to do so for 24 hours",
    impact = "Cluster nodes more vulnerable to security exploits. Eventually, no disk space left.",
    description = "Machine(s) require being rebooted, probably due to kernel update.",
  }
```

If you choose to employ such an alert and have configured kured to
probe for active alerts before rebooting, be sure to specify
`--alert-filter-regexp=^RebootRequired$` to avoid deadlock!

### Slack Notifications

If you specify a Slack hook via `--slack-hook-url`, kured will notify
you immediately prior to rebooting a node:

Daniel Holbach's avatar
Daniel Holbach committed
220
![Notification](img/slack-notification.png)
Adam Harrison's avatar
Adam Harrison committed
221
222
223
224

We recommend setting `--slack-username` to be the name of the
environment, e.g. `dev` or `prod`.

225
226
227
228
Alternatively you can use the `--message-template-drain` and `--message-template-reboot` to customize the text of the message, e.g.
```
--message-template-drain="Draining node %s part of *my-cluster* in region *xyz*"
```
229

Adam Harrison's avatar
Adam Harrison committed
230
231
232
233
234
235
236
237
238
239
240
### Overriding Lock Configuration

The `--ds-name` and `--ds-namespace` arguments should match the name and
namespace of the daemonset used to deploy the reboot daemon - the locking is
implemented by means of an annotation on this resource. The defaults match
the daemonset YAML provided in the repository.

Similarly `--lock-annotation` can be used to change the name of the
annotation kured will use to store the lock, but the default is almost
certainly safe.

Adam Harrison's avatar
Adam Harrison committed
241
242
243
244
245
246
247
248
249
250
## Operation

The example commands in this section assume that you have not
overriden the default lock annotation, daemonset name or namespace;
if you have, you will have to adjust the commands accordingly.

### Testing

You can test your configuration by provoking a reboot on a node:

Daniel Holbach's avatar
Daniel Holbach committed
251
```console
Adam Harrison's avatar
Adam Harrison committed
252
253
254
255
256
257
258
259
sudo touch /var/run/reboot-required
```

### Disabling Reboots

If you need to temporarily stop kured from rebooting any nodes, you
can take the lock manually:

Daniel Holbach's avatar
Daniel Holbach committed
260
```console
Adam Harrison's avatar
Adam Harrison committed
261
262
263
264
265
266
267
268
269
270
271
kubectl -n kube-system annotate ds kured weave.works/kured-node-lock='{"nodeID":"manual"}'
```

Don't forget to release it afterwards!

### Manual Unlock

In exceptional circumstances, such as a node experiencing a permanent
failure whilst rebooting, manual intervention may be required to
remove the cluster lock:

Daniel Holbach's avatar
Daniel Holbach committed
272
```console
Adam Harrison's avatar
Adam Harrison committed
273
274
kubectl -n kube-system annotate ds kured weave.works/kured-node-lock-
```
Daniel Holbach's avatar
Daniel Holbach committed
275

Adam Harrison's avatar
Adam Harrison committed
276
277
278
> NB the `-` at the end of the command is important - it instructs
> `kubectl` to remove that annotation entirely.

Michal Schott's avatar
Michal Schott committed
279
280
281
282
283
### Automatic Unlock

In exceptional circumstances (especially when used with cluster-autoscaler) a node
which holds lock might be killed thus annotation will stay there for ever.

284
Using `--lock-ttl=30m` will allow other nodes to take over if TTL has expired (in this case 30min) and continue reboot process.
Michal Schott's avatar
Michal Schott committed
285

Adam Harrison's avatar
Adam Harrison committed
286
287
## Building

Jean-Philippe Evrard's avatar
Jean-Philippe Evrard committed
288
Kured now uses [Go
289
290
291
292
293
Modules](https://github.com/golang/go/wiki/Modules), so build
instructions vary depending on where you have checked out the
repository:

**Building outside $GOPATH:**
294

Daniel Holbach's avatar
Daniel Holbach committed
295
```console
296
make
Adam Harrison's avatar
Adam Harrison committed
297
```
298

299
300
**Building inside $GOPATH:**

Daniel Holbach's avatar
Daniel Holbach committed
301
```console
302
303
GO111MODULE=on make
```
304

305
306
You can find the current preferred version of Golang in the [go.mod file](go.mod).

Daniel Holbach's avatar
Daniel Holbach committed
307
308
309
310
311
If you are interested in contributing code to kured, please take a look at
our [development][development] docs.

[development]: DEVELOPMENT.md

Adam Harrison's avatar
Adam Harrison committed
312
313
## Frequently Asked/Anticipated Questions

Hidde Beydals's avatar
Hidde Beydals committed
314
### Why is there no `latest` tag on Docker Hub?
Adam Harrison's avatar
Adam Harrison committed
315
316
317
318
319
320
321

Use of `latest` for production deployments is bad practice - see
[here](https://kubernetes.io/docs/concepts/configuration/overview) for
details. The manifest on `master` refers to `latest` for local
development testing with minikube only; for production use choose a
versioned manifest from the [release page](https://github.com/weaveworks/kured/releases/).

Adam Harrison's avatar
Adam Harrison committed
322
## Getting Help
323
324
325

If you have any questions about, feedback for or problems with `kured`:

Daniel Holbach's avatar
Daniel Holbach committed
326
327
328
329
* Invite yourself to the <a href="https://slack.weave.works/" target="_blank">Weave Users Slack</a>.
* Ask a question on the [#kured](https://weave-community.slack.com/messages/kured/) slack channel.
* [File an issue](https://github.com/weaveworks/kured/issues/new).
* Join us in [our monthly meeting](https://docs.google.com/document/d/1bsHTjHhqaaZ7yJnXF6W8c89UB_yn-OoSZEmDnIP34n8/edit#),
330
  every fourth Wednesday of the month at 16:00 UTC.
331
332

Your feedback is always welcome!