Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

set deadlines on outbound requests #2485

Merged
merged 6 commits into from
Feb 3, 2025
Merged

set deadlines on outbound requests #2485

merged 6 commits into from
Feb 3, 2025

Conversation

bjlaub
Copy link
Contributor

@bjlaub bjlaub commented Jan 30, 2025

Before this PR

No Expect-Within header is added to outbound requests

After this PR

Adds the deadlines-java library and utilizes it to set Expect-Within headers on outbound requests.

This change adds a DeadlineAdvertisementChannel which calls Deadlines.encodeToRequest to set an Expect-Within header equal to the configured socket read timeout for the client, or a remaining deadline read from a TraceLocal per the implementation in deadlines-java (whichever is smaller).

deadlines-java does not currently doe any enforcement on deadline expiration, and instead only reports metrics when a deadline is exceeded before sending an outbound request.

==COMMIT_MSG==
set deadlines on outbound requests
==COMMIT_MSG==

Possible downsides?

@changelog-app
Copy link

changelog-app bot commented Jan 30, 2025

Generate changelog in changelog/@unreleased

What do the change types mean?
  • feature: A new feature of the service.
  • improvement: An incremental improvement in the functionality or operation of the service.
  • fix: Remedies the incorrect behaviour of a component of the service in a backwards-compatible way.
  • break: Has the potential to break consumers of this service's API, inclusive of both Palantir services
    and external consumers of the service's API (e.g. customer-written software or integrations).
  • deprecation: Advertises the intention to remove service functionality without any change to the
    operation of the service itself.
  • manualTask: Requires the possibility of manual intervention (running a script, eyeballing configuration,
    performing database surgery, ...) at the time of upgrade for it to succeed.
  • migration: A fully automatic upgrade migration task with no engineer input required.

Note: only one type should be chosen.

How are new versions calculated?
  • ❗The break and manual task changelog types will result in a major release!
  • 🐛 The fix changelog type will result in a minor release in most cases, and a patch release version for patch branches. This behaviour is configurable in autorelease.
  • ✨ All others will result in a minor version release.

Type

  • Feature
  • Improvement
  • Fix
  • Break
  • Deprecation
  • Manual task
  • Migration

Description

set deadlines on outbound requests

Check the box to generate changelog(s)

  • Generate changelog entry

@bjlaub bjlaub changed the title WIP add support for soft deadlines set deadlines on outbound requests Jan 30, 2025
@bjlaub bjlaub marked this pull request as ready for review January 30, 2025 20:56
@@ -242,6 +242,8 @@ private static ImmutableList<LimitedChannel> createHostChannels(
channel = HostMetricsChannel.create(cf, channel, targetUri.uri());
channel =
new TraceEnrichingChannel(channel, DialogueTracing.tracingTags(cf, uriIndexForInstrumentation));
channel = new DeadlineAdvertisementChannel(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are there other places we need to create one of these?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There shouldn't be -- this should do the trick! This position looks great because executes after the queues and retrying channels so that it will be re-applied on each request with an updated Expect-With value

@@ -242,6 +242,8 @@ private static ImmutableList<LimitedChannel> createHostChannels(
channel = HostMetricsChannel.create(cf, channel, targetUri.uri());
channel =
new TraceEnrichingChannel(channel, DialogueTracing.tracingTags(cf, uriIndexForInstrumentation));
channel = new DeadlineAdvertisementChannel(
channel, cf.clientConf().readTimeout());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a couple places which set a read timeout value of zero expecting it to be infinite. We handle this case in the apache httpclient shim here:

if (socketTimeoutMillis == 0) {
// https://issues.apache.org/jira/browse/HTTPCLIENT-2099
log.debug(
"Working around HTTPCLIENT-2099 by using a 1 day socket "
+ "timeout instead of zero (unlimited). Client: {}",
SafeArg.of("client", clientName));
socketTimeoutMillis = Duration.ofDays(1).toMillis();
}

I think we'll need to handle that case in the DeadlineAdvertisementChannel

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 fixed!

@bulldozer-bot bulldozer-bot bot merged commit 3366176 into develop Feb 3, 2025
6 checks passed
@bulldozer-bot bulldozer-bot bot deleted the blaub/deadlines branch February 3, 2025 17:30
@autorelease3
Copy link

autorelease3 bot commented Feb 3, 2025

Released 5.1.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants