Infrastructure Layer

Troubleshooting Problems with Gremlin on OpenShift

Gremlin network timeouts

This issue is most often seen with timeout errors in both Chao and Gremlin logs.

1error sending request for url (https://api.gremlin.com/v1/daemon/poll?multiple=1): operation timed out

This usually stems from network rules preventing Gremlin's access to the internet. It's important to figure out what the intended network behavior should be for Gremlin on your infrastructure with some questions:

What other services connect to the internet within your cluster?
Do services within your cluster rely on an HTTP proxy when connecting to the internet?

Proxy requirements

If any of your OpenShift pods require an HTTP proxy for connecting to the internet, and you plan to attack these pods within Gremlin, we recommend that you configure Gremlin to run behind the same proxy. Check out Gremlin's install instructions for configuring an HTTP proxy.

OpenShift Egress network policies

If you've reviewed the proxy requirements and determined that Gremlin does not need an HTTP proxy, but you are still unable to connect Gremlin to the internet, it's likely one or more OpenShift projects are preventing internet access with an EgressNetworkPolicy.

You can list such policies in any project with the following

shell

1oc -n $PROJECT get egressnetworkpolicies

1NAME   AGE
2test   20m

If you look at the details of such a policy, you can see if network access for api.gremlin.com is denied. Here's an example of a policy which denies api.gremlin.com, because it only allows specific IP address ranges and host names while denying everything else.

shell

1oc -n $PROJECT get egressnetworkpolicy test -o yaml

yaml

1apiVersion: network.openshift.io/v1
2kind: EgressNetworkPolicy
3metadata:
4  name: test
5  namespace: test
6spec:
7  egress:
8  - to:
9      cidrSelector: 1.2.3.0/24
10    type: Allow
11  - to:
12      dnsName: www.foo.com
13    type: Allow
14  - to:
15      cidrSelector: 0.0.0.0/0
16    type: Deny

Adding api.gremlin.com to such a EgressNetworkPolicy will fix this problem.

yaml

1apiVersion: network.openshift.io/v1
2kind: EgressNetworkPolicy
3metadata:
4  name: test
5  namespace: test
6spec:
7  egress:
8  - to:
9      cidrSelector: 1.2.3.0/24
10    type: Allow
11  - to:
12      dnsName: www.foo.com
13    type: Allow
14  - to:
15      dnsName: api.gremlin.com
16    type: Allow
17  - to:
18      cidrSelector: 0.0.0.0/0
19    type: Deny