Orphaned subprocesses may cause command substitution to hang.

[hongyi-zhao]

On Ubuntu 20.04.3 LTS, I check the self-compiled git master version of xclip with the following bash script:

#! /bin/bash
myclip() {
xclip -selection clipbord -in <<< "hello world"
echo x
return
}
set -x
while true; do
y="$(myclip)"
done

It will be blocked there forever, as shown below. In other words, when running xclip in a function, it cannot return to the main script.

Please refer to the related discussions here and here.

Regards,
HZ

[mviereck]

The blocking only happens if the output of the function is captured with $(...).
It does not happen if the function is called without $(...) in the main loop:

#! /bin/bash
myclip() {
  xclip -selection clipbord -in <<< "hello world" 
  echo x
  return
}
set -x
while true; do
  myclip
done

However, I don't understand why the first example blocks.
I assume it is somehow related to xclip forking itself into background.
I have tried with nohup xclip ... &, but that does not help.

[hongyi-zhao]

The following script works, see the discussion here for more detailed information:

$ cat myclip.sh
#! /bin/bash
myclip()
{
printf '%s\n' 'hello world' |
xclip -selection clipboard -in > /dev/null
printf '%s\n' x
sleep 1
return
}
set -x
while true; do
y="$(myclip)"
done

$ bash myclip.sh
+ true
++ myclip
++ printf '%s\n' 'hello world'
++ xclip -selection clipboard -in
++ printf '%s\n' x
++ sleep 1
++ return
+ y=x
+ true
++ myclip
++ printf '%s\n' 'hello world'
++ xclip -selection clipboard -in
++ printf '%s\n' x
++ sleep 1
++ return
+ y=x
+ true
++ myclip
++ printf '%s\n' 'hello world'
++ xclip -selection clipboard -in
^C

[mviereck]

I have compared the code.
The essential difference is to redirect stdout of xclip to /dev/null:

  xclip -selection clipboard -in <<< "hello world" > /dev/null

That is quite surprising.
I wonder if that is a bash or an xclip issue.

[hongyi-zhao]

The following explanation comes from here given by Helmut Waitzmann <[email protected]>:

The various variants of your example I gave in the messages with
the Message‐Ids <[email protected]> and
<[email protected]> show: It's not the
function that causes the blocking. The script blocks if and only
if "xclip" is invoked without redirecting its standard output away
from feeding the command substitution "$( … )". Whether that
occurs inside of a function or just inline makes no difference.

So, what's going on, there?

When the shell performs a command substition

"$( some commands )",

it first creates and opens a pipe by means of the system call
"pipe()". (A pipe is a kernel object which can be written into at
one end and read from at the other end. See the manual pages
pipe(7) and pipe(2).)

Then the shell forks a process that will execute the subshell
environment: the command line inside the "$( )".

The forked subshell environment will write to the created pipe
while the outer shell, i. e. the shell that has forked the subshell
environment, will continue reading data from the created pipe and
collecting it until it reaches an end‐of‐file condition. When it
has reached the end‐of‐file condition, it will wait by means of one
of the "wait" family of system calls (see the manual page wait(2))
for the subshell environment to terminate.

It puts the collected data it has read from the pipe into the
parameter list of the command. In your example that command is the
variable assignment:

y="$( … )"

Therefore, the read data will get assigned to the shell variable
"y".

The exit status of the variable assignment will be the exit status
as obtained from the "wait" system call, i. e. it will be the exit
status of the command substitution.

The key to understanding why your example blocks is to understand
what an end‐of‐file condition in a pipe is:

When reading from a pipe, there are (at least) three conditions to
be distinguished:

(1) A writer has written something to the pipe. Then upon reading,
the reader will get that data, which will be removed from the pipe.

(2) The pipe is empty but the writing end is still open for
writing, i.e. nothing has been written to it since all data that
may have been written to it have been read. Then upon reading, the
reader will block and wait for further data to arrive as it does
not know whether the writer eventually will write further data.

(3) The writer may have written some (or may not have written any)
data to the pipe and then closes the writing end of the pipe. When
there is no writer left to write to the writing end of the pipe,
then upon reading, the reader will get the data (if there is one)
like in (1) and then it will get an end‐of‐file condition.

The end‐of‐file condition tells the reader (in your use case: the
outer shell) that there will no further data arrive in the pipe (as
no one has it open for writing any more).

In the use case of the command substitution the outer shell will
know that it has read all data to be used in the command
substitution. It will close its reading end of the pipe and then
wait for the subshell environment to terminate.

Now what's going on, when the subshell environment starts the
"xclip" command which will fork a subprocess?

Without redirection of its standard output, that is, the blocking
use cases, "xclip" will have its standard output file descriptor be
the writing end of the pipe. When "xclip" forks its subprocess, the
subprocess will of course inherit the writing end of the pipe. =>
There are two processes which might write to the pipe.

Now, if the "outer" "xclip" process terminates, its writing end of
the pipe will implicitly get closed, but the writing end of the pipe
hold by the forked "xclip" subprocess will remain open.

=> As written above in (2) and (3) the reader of the pipe won't get
an end‐of‐file condition as long as the forked "xclip" subprocess
neither closes its standard output (Of course it will not: No
program is expected to close its standard output.) nor terminates.

=> Therefore the outer shell will wait for the subshell environment
to send an end‐of‐file condition to the pipe until the cows come
home.

Now on the other hand, if "xclip" is called with its standard
output file descriptor redirected from the writing end of the pipe
to for example "/dev/null", then its forked subprocess will inherit
the redirected standard output file descriptor.

=> No dangling writing end of the pipe will stay open and the outer
shell will finally get its longed‐for end‐of‐file condition when
reading from the pipe.

That scenario is not specific to "xclip". Here is an example using
"sleep" in the background. Note: "sleep" won't write anything to
its standard output, and of course it won't even bother closing it:

(
while var="$(
sleep -- 2 &
date -- '+%Y-%m-%dT%T%z'
)"
do
printf 'command substitution ran at %s\n' "$var"
date -- '+command substitution ended at %Y-%m-%dT%T%z'
echo
sleep -- 1
done
)

The output shows that the command substitution lasts for 2 seconds
in spite of the "sleep" command being run in the background,
because the end‐of‐file condition for the reader is delayed until
that two‐seconds‐sleep terminates and implicitly gets it standard
output file descriptor, which refers to the writing end of the
pipe, closed.

If the standard output of the "sleep" command is redirected from
the writing end of the pipe to (for example) "/dev/null" then the
command substitution returns almost immediately, as can be seen
here:

(
while var="$(
sleep -- 2 > /dev/null &
date -- '+%Y-%m-%dT%T%z'
)"
do
printf 'command substitution ran at %s\n' "$var"
date -- '+command substitution ended at %Y-%m-%dT%T%z'
echo
sleep -- 1
done
)

May I add my refrain? Knowing linux or unix greatly helps
understanding the shell. Without that knowledge one will have a
hard job doing so.

The following is the relevant discussion Helmut Waitzmann sent me via private email on this issue. Because it is related to this issue, I also forward it here for reference:

The same "hanging" effect can be seen when using pipes in the
commandline: While in

( sleep -- 2 & date -- '+%Y-%m-%dT%T%z' ) | cat
date -- '+%Y-%m-%dT%T%z'

"cat" will wait for two seconds on the end‐of‐file condition, it
will almost return immediately in

( sleep -- 2 > /dev/null & date -- '+%Y-%m-%dT%T%z' ) | cat
date -- '+%Y-%m-%dT%T%z'

Kind regards

Helmut Waitzmann

[gklanderman]

OP mis-understands how xclip works.

When you run
% xclip` -selection clipboard -in <<< "hello world"

you immediately get your prompt back, but xclip remains running in the background (it forked itself) and will keep running so long as it owns the selection. When some other X11 application places data in the clipboard, only then it exits. And at that point, as described by Helmut in detail above, the writing end of the pipe will close, and your $(myclip) will return.

Try this: run your script, and in another window:

% xclip -o -selection clipboard
hello world
% xclip -o -selection clipboard
hello world
% xclip -o -selection clipboard
hello world

you can read from the clipboard multiple times, and the original xclip from your script keeps serving that (has not exited the $(myclip) yet).

but now do:

% xclip -selection clipbord -in <<< "replace"

and you'll see that the original xclip exits, myclip exits, and y is assigned.

Without knowing exactly what you're trying to achieve, it is hard to advise you, but you may want to look into the '-quiet' and '-loops' options.

'-quiet' will cause xclip not to fork (stays in foreground), and '-loops' will cause it to exit after a given number or readers have read the data it posted to the clipboard.

[hongyi-zhao]

% xclip -selection clipboard -in <<< "hello world"

This will put an additional \n into the clipboard, as described below:

$ man bash
**   Here Strings
       A variant of here documents, the format is:

              [n]<<<word

       The word undergoes tilde expansion, parameter and  variable  expansion,
       command  substitution,  arithmetic expansion, and quote removal.  Path‐
       name expansion and word splitting are not  performed.   The  result  is
       supplied as a single string, with a newline appended, to the command on
       its standard input (or file descriptor n if n is specified).
**

[gklanderman]

FWIW I went thru the open bugs and pull requests yesterday, and saw several that were proposing to have xclip -i close stdout when forking into the background.

So that would "fix" your example in the sense that myclip would no longer block, and instead return immediately, however, then you'll just be forking background xclip processes to replace themselves as fast as your shell can do so, which is almost certainly not what you want. What are you actually trying to accomplish?

[mviereck]

Originally the issue came up in my project x11docker.
The overall task to accomplish is to share the clipboard between two X servers.

A few days ago I tried a solution using xclip in a function where it reads from both X servers, compares the clips, sends the latest clip to both, and returns the latest clip. This was the point where I was stuck because the function was blocked by stdout of xclip and I did not understand why.

I don't think the behaviour of xclip should be changed as it is regular unix behaviour. Redirecting stdout already solves the issue as explained well above.

Meanwhile I have a new implementation that does not use a function.
FWIW, I have extracted my current implementation as a standalone script with Xephyr as a second X server:

#! /usr/bin/env bash

Hostdisplay=$DISPLAY
Hostxauthority=$XAUTHORITY
Newdisplay=:20
Xclientcookie=""

Xephyr $Newdisplay -ac &
sleep 1
DISPLAY=$Newdisplay xterm &

###############################
# share clipboard between X servers $Hostdisplay and $Newdisplay

X1auth[1]="DISPLAY=$Hostdisplay XAUTHORITY=$Hostxauthority"
X2auth[1]="DISPLAY=$Newdisplay XAUTHORITY=$Xclientcookie"
X1auth[2]="DISPLAY=$Newdisplay XAUTHORITY=$Xclientcookie"
X2auth[2]="DISPLAY=$Hostdisplay XAUTHORITY=$Hostxauthority"
Selection[1]=clipboard
Selection[2]=primary
OtherX[1]=2
OtherX[2]=1

while :; do
  for S in 1 2; do
    for X in 1 2; do
      export ${X1auth[$X]}
      Targets="$(xclip -out -selection ${Selection[$S]} -t TARGETS 2>/dev/null)"

      [ -n "$Targets" ] && {
        # check for image clip
        grep -q "image/png" <<< "$Targets" && Imagetarget="-t image/png" || Imagetarget=""

        # read content of ${Selection[$S]} of X server $X
        Clip[$X$S]="$(xclip -out -selection ${Selection[$S]} $Imagetarget | base64)"

        # check if ${Selection[$S]} of X server $X has changed
        [ -n "${Clip[$X$S]}" ] && [ "${Clipold[$X$S]}" != "${Clip[$X$S]}" ] && {
          Clipold[$X$S]="${Clip[$X$S]}"

          # send only to clipboard of other X server if it has different content. Important to keep highlighted text selection.
          [ "${Clip[${OtherX[$X]}$S]}" != "${Clip[$X$S]}" ] && {
            export ${X2auth[$X]}
            base64 -d <<< "${Clip[$X$S]}" | xclip -in -selection ${Selection[$S]} $Imagetarget
            #echo "SEND FROM $X (${Selection[$S]}): $(base64 -d <<< "${Clip[$X$S]}")" >&2
          }
        }
      }
    done
  done
  # sleep a bit to avoid high cpu usage
  sleep 0.3
done

################
A feature that would help me to improve this:
If xclip could optionally behave similar to xclip -in -quiet in a way that it waits for new content before returning (but doing nothing else), I could write a less CPU expensive loop.
I think of a new option like xclip -waitforchange that returns once the clipboard content changes.
xclip -in -quiet already behaves this way.
However, this leads to a special issue if it comes to -selection primary. The loop detects a new content with xclip -out -selection primary and comparing it with previous clipboard content.
But if I send this new content with xclip -in -selection primary -quiet to have a 'waiting xclip', the selected text looses its highlighting. (Because the application providing the clip has changed.)
This might sound a bit confusing. I'll add a script that helps to reproduce and understand the issue.

[gklanderman]

I don't think the behaviour of xclip should be changed as it is regular unix behaviour.
Redirecting stdout already solves the issue as explained well above.

since xclip -i will not write to stdout, I think it actually does make sense to close stdout, and I think that was the conclusion drawn in several bugs, but I did not read them at all carefully.

I think of a new option like xclip -waitforchange that returns once the clipboard
content changes.

I implemented something similar (locally) today, so that the -wait option works with -o, but not in exactly the way you want: what I implemented is that if no selection exists when you run xclip -o, it will wait up to the # of millisec specified by -wait, or indefinitely if -wait is negative. It should not be too hard to change that to do what you want (just bypass the "if no current selection" check).

this leads to a special issue if it comes to -selection primary

I think I see what you're getting at, but I can't think of a solution.

btw I just stumbled across this project the other day that sounds similar to what you're trying to do:

https://www.chiark.greenend.org.uk/~sgtatham/utils/xclipglue.html

have you seen it?

[mviereck]

since xclip -i will not write to stdout, I think it actually does make sense to close stdout

At least with option -filter it writes to stdout.

I implemented something similar (locally) today, so that the -wait option works with -o, but not in exactly the way you want: what I implemented is that if no selection exists when you run xclip -o, it will wait up to the # of millisec specified by -wait, or indefinitely if -wait is negative. It should not be too hard to change that to do what you want (just bypass the "if no current selection" check).

That sounds promising!
For my purpose -wait could be bound to -out as well; if I don't need the output, I could ignore it.

btw I just stumbled across this project the other day that sounds similar to what you're trying to do:

Thank you for the hint! xclipclue looks promising and I wasn't aware of it.
Unfortunately it has bugs. It does not sync the clipboards, instead it breaks the possibility to get context menus with right mouse click.

I think I see what you're getting at, but I can't think of a solution.

Basically I only need a waiting xclip other than with -in -quiet to avoid expensive loops. I cannot use -in for this because it removes the highlighting of text to copy in case of -selection primary.

I've rewritten the code above a bit to split it into 4 loops.
Some sort of xclip -out -waitforchange could replace the sleep command. The script would react immediately to clipboard changes without a sleep delay and it would not repeatedly compare the output of xclip -out at high cost to no avail.

#! /usr/bin/env bash

Hostdisplay=$DISPLAY
Hostxauthority=$XAUTHORITY
Newdisplay=:20
Xclientcookie=""

Xephyr $Newdisplay -ac &
sleep 1
DISPLAY=$Newdisplay xterm &

#############################

clipexchange() {
  # Send clipboard content of one X server to another X server and update on changes.
  # Run function twice with switched credentials to sync the clipboards.
  # $1  credentials to access 1. X server
  # $2  credentials to access 2. X server
  # $3  selection to use (primary or clipboard)
  # X credentials look like "DISPLAY=:0 XAUTHORITY=~/.Xauthority"

  local Selection
  Selection="${3:-}"

  while :; do
    Targets="$(env ${1:-} xclip -out -selection $Selection -t TARGETS 2>/dev/null)"

    # wait for the clipboard to have some content at all
    [ -z "$Targets" ] && env ${1:-} xclip -selection $Selection -in -verbose <<< ""

    [ -n "$Targets" ] && {
      # check for image clip
      grep -q "image/png" <<< "$Targets" && Imagetarget="-t image/png" || Imagetarget=""

      # read content of $Selection of X server $1
      Clip="$(env ${1:-} xclip -selection $Selection -out $Imagetarget | base64)"

      # check if $Selection of X server $1 has changed
      [ -n "$Clip" ] && [ "$Clip" != "$Clipold" ] && {
        Clipold="$Clip"

        # send only to clipboard of other X server if it has different content. Important to keep highlighted text selection.
        Clipold2="$(env ${2:-} xclip -selection $Selection -out $Imagetarget | base64)"
        [ "$Clipold2" != "$Clip" ] && {
          #echo "SEND $Selection TO $2"
          base64 -d <<< "$Clip" | env ${2:-} xclip -selection $Selection -in $Imagetarget
        }
      }

      # wait a bit to avoid high CPU usage
      sleep 0.3

      # exit if one X server is not accessible
      env ${1:-} xclip -o >/dev/null || break
      env ${2:-} xclip -o >/dev/null || break
    }
  done
}

X1auth="DISPLAY=$Hostdisplay XAUTHORITY=$Hostxauthority"
X2auth="DISPLAY=$Newdisplay XAUTHORITY=$Xclientcookie"

clipexchange "$X1auth" "$X2auth" clipboard &
clipexchange "$X2auth" "$X1auth" clipboard &
clipexchange "$X1auth" "$X2auth" primary &
clipexchange "$X2auth" "$X1auth" primary &

[gklanderman]

sorry xclipglue didn't work, it really seems like it should do exactly what you want, and should ideally be more robust than a shell script. you might reach out to the author, he last made a change ~1.5 years ago, and his email is at the bottom of this page:

https://www.chiark.greenend.org.uk/~sgtatham/utils/

re: your script, IIUC, you would be able to get rid of the sleep and the
xclip -selection $Selection -in -verbose <<< ""
with the waitforchange flag?

I have a couple hoops with my employer before I can submit a pull request for the wait feature I implemented. Once I submit that, the additional option you want should be pretty easy to add.

[mviereck]

re: your script, IIUC, you would be able to get rid of the sleep and the
xclip -selection $Selection -in -verbose <<< ""
with the waitforchange flag?

Yes, exactly.

I have a couple hoops with my employer before I can submit a pull request for the wait feature I implemented. Once I submit that, the additional option you want should be pretty easy to add.

That sounds great! No need to hurry.

[gklanderman]

Sounds good Martin, this week is busy and I'm on vacation next week, so while I might get to it this week, it could take a few weeks.

[mviereck]

@gklanderman Just want to kindly ask if you still have this project in mind?

[gklanderman]

Hi @mviereck sorry I have had no time at all unfortunately.. I will try to make some progress in the next few weeks, ping me again if you don't hear anything..

[gklanderman]

Hi @mviereck sorry for the very long delay; not sure if you're still interested in this, but I no longer have employer hoops to jump through, and could probably make some time in the next month or so..

[quartzjer]

hi @gklanderman! I'm helping maintain xclip and all PRs are welcome anytime 👍

[gklanderman]

Jer @quartzjer great to hear.. I have some changes I am using locally which I will try to clean up and send PRs for as a first step..

Then I can look at this issue and the related ones I mentioned above.

I see you're committing some changes lately.. do you have any plan for a release, and if so what's the timeline? Looks like the last release was 8 or so years ago..