[Bug] [Crash][Reproducible] EOFError: Ran out of input when import yapf with multiprocess

[whlook]

previous similar issue #1164

Problems

• When i use yapf (just import it) in multiprocess program, it will crash with 'EOFError: Ran out of inpu'，log info below:

Reproduction

1. yapf==0.40.2
2. rm -r ~/.cache/YAPF
3. run this python code:

 import multiprocessing
 
 def proc():
     import yapf
 
 if __name__ == "__main__":
     pp=[]
     for i in range(100):
         p = multiprocessing.Process(target=proc)
         pp.append(p)
 
     for p in pp:
         p.start()
 
     for p in pp:
         p.join()
                                                   

Reason

the problem code is here:

yapf/third_party/yapf_third_party/_ylib2to3/pgen2/driver.py

Line 247 in c0908d0

g = grammar.Grammar()

When in multi process enviroment,some process will create grammar cache(when ~/.cache/YAPF not exists), before the process write to the cache file, other process will see the file exist and load it, but the cache file is empty right now , so EOFError will raise and the program crashed.

How to fix it?

• I think we need to add try here(

  yapf/third_party/yapf_third_party/_ylib2to3/pgen2/driver.py

  Line 247 in c0908d0

  g = grammar.Grammar()

  ) like we had add try when we create the cache.
• Or we can add a multiprocess.Lock here
• Or we can do python -c 'import yapf' before we start our multiprocess program.(for users,trick)

[hartwork]

@whlook that's a great report, in particular the reproducer is very handy — thank you!

I have a pull request coming up for this — should be auto-linked below in a minute or two —, and I would like to share my extended version of your reproducer below:

import multiprocessing

def cleanup():
    import os
    import yapf
    from yapf_third_party._ylib2to3.pgen2.driver import _generate_pickle_name
    for i in ("Grammar.txt", "PatternGrammar.txt"):
        filename = _generate_pickle_name(i)
        print(f"  Removing {filename}...")
        os.remove(_generate_pickle_name(i))

def proc():
    import yapf

if __name__ == "__main__":
    max_parallelity = 30
    for parallelity in range(2, max_parallelity + 1):
        print("Cleaning up...")
        cleanup_p = multiprocessing.Process(target=cleanup)
        cleanup_p.start()
        cleanup_p.join()

        print(f"Testing for the race condition with {parallelity} processes...")
        pp = []
        for i in range(parallelity):
            p = multiprocessing.Process(target=proc)
            pp.append(p)

        for p in pp:
            p.start()

        for p in pp:
            p.join()

        if any(p.exitcode != 0 for p in pp):
            print(f"Done, race condition proven above, took {parallelity} processes.")
            break
    else:
        print("Done, no luck crashing this time.")