Question:
I am using chaostoolkit and am able to run a chaos experiment successfully from command line. However, when i try to run the same as a job in k8s it is throwing up ‘connection refused’ error. What I find it strange is that at times the steady state hypothesis step runs successfully and returns 200 OK while failing in the terminate pod action, but a lot of times it is also failing at the hypothesis step itself (before the action to terminate pod). I am doing this in Google cloud by the way.During some runs i see that the hypothesis before action as well as the termination pod are successful but the hypothesis after action (termination) gets ‘connection refused’ error.
Any help/tip is appreciated.
Here is the error msg:
[2022-02-03 07:24:54 DEBUG] [caching:35] Cached 2 activities
[2022-02-03 07:24:54 INFO] [experiment:54] Validating the experiment's syntax
[2022-02-03 07:24:54 DEBUG] [configuration:47] Loading configuration...
[2022-02-03 07:24:54 DEBUG] [secret:74] Loading secrets...
[2022-02-03 07:24:54 DEBUG] [secret:89] Secrets loaded
[2022-02-03 07:25:12 INFO] [experiment:103] Experiment looks valid
[2022-02-03 07:25:12 DEBUG] [caching:42] Clearing activities cache
[2022-02-03 07:25:12 DEBUG] [caching:25] Building activity cache...
[2022-02-03 07:25:12 DEBUG] [caching:35] Cached 2 activities
[2022-02-03 07:25:12 INFO] [experiment:182] Running experiment: What happens if we terminate an instance of the application?
[2022-02-03 07:25:12 DEBUG] [configuration:47] Loading configuration...
[2022-02-03 07:25:12 DEBUG] [secret:74] Loading secrets...
[2022-02-03 07:25:12 DEBUG] [secret:89] Secrets loaded
[2022-02-03 07:25:12 DEBUG] [__init__:39] Initializing controls
[2022-02-03 07:25:12 DEBUG] [__init__:355] No controls to apply on 'experiment'
[2022-02-03 07:25:12 INFO] [hypothesis:184] Steady state hypothesis: The app is healthy
[2022-02-03 07:25:12 DEBUG] [__init__:355] No controls to apply on 'hypothesis'
[2022-02-03 07:25:12 DEBUG] [__init__:355] No controls to apply on 'activity'
[2022-02-03 07:25:12 INFO] [activity:160] Probe: app-responds-to-requests
[2022-02-03 07:25:12 DEBUG] [activity:233] Activity failed
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 156, in _new_conn
conn = connection.create_connection(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
health-http.yaml: |
version: 1.0.0
title: What happens if we terminate an instance of the application?
description: If an instance of the application is terminated, the applications as a whole should still be operational.
tags:
- k8s
- pod
steady-state-hypothesis:
title: The app is healthy
probes:
- name: app-responds-to-requests
type: probe
tolerance: 200
provider:
type: http
timeout: 10
verify_tls: false
url: http://newapp
headers:
Host: newapp.example.com
method:
- type: action
name: terminate-app-pod
provider:
type: python
module: chaosk8s.pod.actions
func: terminate_pods
arguments:
label_selector: app=newapp
rand: true
ns: default
pauses:
after: 2
Here is the experiment manifest:
apiVersion: batch/v1
kind: Job
metadata:
name: newapp-chaos
spec:
activeDeadlineSeconds: 600
backoffLimit: 0
template:
metadata:
labels:
app: newapp
annotations:
sidecar.istio.io/inject: "false"
spec:
serviceAccountName: newapp-chaos
restartPolicy: Never
containers:
- name: chaostoolkit
image: vfarcic/chaostoolkit:1.4.1-2
args:
- --verbose
- run
- /experiment/health-http.yaml
env:
- name: CHAOSTOOLKIT_IN_POD
value: "true"
volumeMounts:
- name: config
mountPath: /experiment
readOnly: true
resources:
limits:
cpu: 20m
memory: 64Mi
requests:
cpu: 20m
memory: 64Mi
volumes:
- name: config
configMap:
name: newapp-config
apiVersion: apps/v1
kind: Deployment
metadata:
name: newapp-v2
spec:
replicas: 1
selector:
matchLabels:
app: newapp
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: newapp
version: v2
spec:
containers:
- image: rstarmer/hostname:v2
imagePullPolicy: Always
name: newapp
restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
labels:
app: newapp
name: newapp
spec:
#externalTrafficPolicy: Cluster
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: newapp
sessionAffinity: None
[2022-02-03 09:43:22 INFO] [hypothesis:184] Steady state hypothesis: The app is healthy
[2022-02-03 09:43:22 DEBUG] [__init__:355] No controls to apply on 'hypothesis'
[2022-02-03 09:43:22 DEBUG] [__init__:355] No controls to apply on 'activity'
[2022-02-03 09:43:22 INFO] [activity:160] Probe: app-responds-to-requests
[2022-02-03 09:43:22 DEBUG] [activity:179] => succeeded with '{'status': 200, 'headers': {'Server': 'nginx/1.15.4', 'Date': 'Thu, 03 Feb 2022 09:43:22 GMT', 'Content-Type': 'text/html', 'Content-Length': '208', 'Last-Modified': 'Thu, 03 Feb 2022 07:21:47 GMT', 'Connection': 'keep-alive', 'ETag': '"61fb828b-d0"', 'Accept-Ranges': 'bytes'}, 'body': "<HTML>\n<HEAD>\n<TITLE>This page is on newapp-v2-866f8798cd-8s424 and is version v2</TITLE>\n</HEAD><BODY>\n<H1>THIS IS HOST newapp-v2-866f8798cd-8s424</H1>\n<H2>And we're running version: v2</H2>\n</BODY>\n</HTML>\n"}'
[2022-02-03 09:43:22 DEBUG] [__init__:355] No controls to apply on 'activity'
[2022-02-03 09:43:22 DEBUG] [hypothesis:212] allowed tolerance is 200
[2022-02-03 09:43:22 INFO] [hypothesis:222] Steady state hypothesis is met!
[2022-02-03 09:43:22 DEBUG] [__init__:355] No controls to apply on 'hypothesis'
[2022-02-03 09:43:22 DEBUG] [__init__:355] No controls to apply on 'method'
[2022-02-03 09:43:22 DEBUG] [__init__:355] No controls to apply on 'activity'
[2022-02-03 09:43:22 INFO] [activity:160] Action: terminate-app-pod
[2022-02-03 09:43:22 DEBUG] [python:34] Activity 'terminate-app-pod' loaded from '/usr/local/lib/python3.8/site-packages/chaosk8s/pod/actions.py'
[2022-02-03 09:43:23 DEBUG] [actions:193] Found 3 pods labelled 'app=newapp' in ns default
[2022-02-03 09:43:23 DEBUG] [activity:181] => succeeded without any result value
[2022-02-03 09:43:23 INFO] [activity:197] Pausing after activity for 2s...
[2022-02-03 09:43:25 DEBUG] [__init__:355] No controls to apply on 'activity'
[2022-02-03 09:43:25 DEBUG] [__init__:355] No controls to apply on 'method'
[2022-02-03 09:43:25 INFO] [hypothesis:184] Steady state hypothesis: The app is healthy
[2022-02-03 09:43:25 DEBUG] [__init__:355] No controls to apply on 'hypothesis'
[2022-02-03 09:43:25 DEBUG] [__init__:355] No controls to apply on 'activity'
[2022-02-03 09:43:25 INFO] [activity:160] Probe: app-responds-to-requests
[2022-02-03 09:43:25 DEBUG] [activity:233] Activity failed
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 156, in _new_conn
conn = connection.create_connection(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
Answer:
After struggling for over a day, I tried reinstalling Istio and it started working fine. Must be something wrong with Istio then.If you have better answer, please add a comment about this, thank you!
Source: Stackoverflow.com