How to understand trace span tag peer.address?

[gxxxh]

I have deployed Jaeger on istio to monitor the bookinfo application. Then I want to analyze the trace data of Jaeger. However, I am confused by some span tags, such as, upstream_cluster and peer address. I found the meaning of peer.address from https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk_exporters/zipkin.md
, It says that peer.address means remote address. I think it means that when a client start a request and a server receive the request, the client's peer.address will be the server and the server's peer address will be the client.

Here is some information I get from one trace with 8 spans.

             spanID                                      operationName          podIP           serviceName               pid   refType   peer.address span.kind
0  0044bea01e572833  productpage.default.svc.cluster.local:9080/pro...  10.244.27.204  istio-ingressgateway                               192.168.0.49    client
1  a2bdd3593ab25d00  productpage.default.svc.cluster.local:9080/pro...  10.244.27.238   productpage.default  0044bea01e572833  CHILD_OF  10.244.27.204    server
2  626e09f9758da789           details.default.svc.cluster.local:9080/*  10.244.27.238   productpage.default  a2bdd3593ab25d00  CHILD_OF  10.244.27.238    client
3  5a76f143f0e5c301           details.default.svc.cluster.local:9080/*  10.244.27.247       details.default  626e09f9758da789  CHILD_OF  10.244.27.238    server
4  9c1aaf91ea5e5ce9           reviews.default.svc.cluster.local:9080/*  10.244.27.238   productpage.default  a2bdd3593ab25d00  CHILD_OF  10.244.27.238    client
5  923db83990a9e7e5           reviews.default.svc.cluster.local:9080/*  10.244.27.246       reviews.default  9c1aaf91ea5e5ce9  CHILD_OF  10.244.27.238    server
6  8e3dbd4dd0b70f85           ratings.default.svc.cluster.local:9080/*  10.244.27.246       reviews.default  923db83990a9e7e5  CHILD_OF  10.244.27.246    client
7  f227decb8d8b2c51           ratings.default.svc.cluster.local:9080/*  10.244.27.217       ratings.default  8e3dbd4dd0b70f85  CHILD_OF  10.244.27.246    server

However, the peer.address seems to be it's parent span's pod id, which is different from what I understand, and I don't know which is right.

Besides, I don't understand the meaning of upstream_cluster and downstream_cluster as below. Why all downstream_cluster is None, and what is the meaning of '9080' and 'inbound' in outstream_cluster?

             spanID                                      operationName          podIP  ...                                   upstream_cluster downstream_cluster span.kind
0  0044bea01e572833  productpage.default.svc.cluster.local:9080/pro...  10.244.27.204  ...  outbound|9080||productpage.default.svc.cluster...                  -    client
1  a2bdd3593ab25d00  productpage.default.svc.cluster.local:9080/pro...  10.244.27.238  ...                                     inbound|9080||                  -    server
2  626e09f9758da789           details.default.svc.cluster.local:9080/*  10.244.27.238  ...   outbound|9080||details.default.svc.cluster.local                  -    client
3  5a76f143f0e5c301           details.default.svc.cluster.local:9080/*  10.244.27.247  ...                                     inbound|9080||                  -    server
4  9c1aaf91ea5e5ce9           reviews.default.svc.cluster.local:9080/*  10.244.27.238  ...   outbound|9080||reviews.default.svc.cluster.local                  -    client
5  923db83990a9e7e5           reviews.default.svc.cluster.local:9080/*  10.244.27.246  ...                                     inbound|9080||                  -    server
6  8e3dbd4dd0b70f85           ratings.default.svc.cluster.local:9080/*  10.244.27.246  ...   outbound|9080||ratings.default.svc.cluster.local                  -    client
7  f227decb8d8b2c51           ratings.default.svc.cluster.local:9080/*  10.244.27.217  ...                                     inbound|9080||                  -    server

[Oberon00]

upstream_cluster and downstream_cluster are not OpenTelemetry-defined. You probably better ask the Istio/Envoy folks what these mean.

I found the meaning of peer.address from https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk_exporters/zipkin.md#remote-endpoint

As the table says, this comes from OpenTracing. The peer.address should be roughly the same as OTel's net.peer.ip https://github.com/open-telemetry/opentelemetry-specification/blob/v1.2.0/specification/trace/semantic_conventions/span-general.md#general-network-connection-attributes

the peer.address seems to be it's parent span's pod id

Assuming you mean "pod IP", then that seems to make sense to me: The parent POD calls the child pod, so from the child PODs perspective, it is the remote/peer address of the socket connection between the parent and the child.

[gxxxh]

Thank you for the answer, But I am still confused about it.

For example, consider the last four spans, the productpage(cliient) will establish a socket connection with reviews(server) to get books' review information. And reviews(client) needs to establish a socket connection with ratings(server) to get books' rating information. As you say:

it is the remote/peer address of the socket connection between the parent and the child

The 6th span says that reviews works as a server and it's peer.address is productpage's pod ip, the explanation makes sense. However, consider the 7th span, pod which runs the microservice reviews has two socket connections(productpage and ratings). but the 7th span's socket conncetion was with ratings, why it's peer address is productpage's pod ip instead of ratings' pod ip?

Here is pod info in kubernetes

NAME                              READY   STATUS     RESTARTS   AGE   IP              NODE       NOMINATED NODE   READINESS GATES
details-v1-79c697d759-s5kxw       2/2     Running    0          34h   10.244.27.247   ecs-8c0a   <none>           <none>
productpage-v1-65576bb7bf-s5kdv   2/2     Running    0          34h   10.244.27.238   ecs-8c0a   <none>           <none>
ratings-v1-7d99676f7f-78d9f       2/2     Running    0          34h   10.244.27.217   ecs-8c0a   <none>           <none>
reviews-v1-987d495c-xdzlz         2/2     Running    0          34h   10.244.27.245   ecs-8c0a   <none>           <none>
reviews-v2-6c5bf657cf-snmg5       2/2     Running    0          34h   10.244.27.211   ecs-8c0a   <none>           <none>
reviews-v3-5f7b9f4f77-vtgwn       2/2     Running    0          34h   10.244.27.246   ecs-8c0a   <none>           <none>