A few of our users occasionally spin up pods that do a lot of number crunching. The front end is a web app that queries the pod and waits for a response.
Some of these queries exceed the default 30s timeout for the pod ingress. So, I added an annotation to the pod ingress to increase the timeout to 60s. Users still report occasional timeouts.
I asked how long they need the timeout to be. They requested 1 hour.
This seems excessive. My gut feeling is this will cause problems. However, I don’t know enough about ingress timeouts to know what will break. So, what is the worst case scenario of 3-10 pods having 1 hour ingress timeouts?
There’s a lot of smaller and bigger possible problems with that, but I think there’s only one way to find out if those can become actual problems.
Try it, and report back.