Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support creating hard links (in additional to symlinks) #301

Open
dxlr8r opened this issue Mar 3, 2023 · 7 comments
Open

Support creating hard links (in additional to symlinks) #301

dxlr8r opened this issue Mar 3, 2023 · 7 comments

Comments

@dxlr8r
Copy link

dxlr8r commented Mar 3, 2023

With for example, gitman.yml:

location: vendor/gitman
sources:
  - repo: https://github.com/jacebrowning/gitman.git
    name: gitman
    rev: main
    type: git
    links:
      - source: doc
        target: vendor/doc/gitman

And .gitignore:

vendor/gitman

vendor/doc/gitman will be a symlink, and content in vendor/gitman will be ignored.

For my project I want to keep copies of the libraries I use, and since git does not follow symlinks the approach above does not work.

I could of course remove the entry in .gitignore, but then I get a lot of unneeded resources.

Hard links seem to be the way to resolve this. Suggested implementation, gitman.yml:

location: vendor/gitman
sources:
  - repo: https://github.com/jacebrowning/gitman.git
    name: gitman
    rev: main
    type: git
    hardlinks:
      - source: doc
        target: vendor/doc/gitman

vendor/doc/gitman would be a regular directory, each file inside would be a hardlink, and any nesting directory would be recreated with the same name as a directory (not symlink). Like a copy (cp -r) but the files are hard linked instead of copies.

Of course hard links has some limitations, has to be on same file system for instance. So an option to fallback to a copy would perhaps required.

@jacebrowning
Copy link
Owner

Git does not follow symlinks

Can you expand on this a bit? If you attempt to use soft links for you use case, what happens?

Suggested implementation

Initially, my thought was something like this:

...
links:
      - source: doc
        target: vendor/doc/gitman
        type: hard

or this:

...
links:
      - source: doc
        target: vendor/doc/gitman
        symbolic: false

@dxlr8r
Copy link
Author

dxlr8r commented Mar 3, 2023

Git does not follow symlinks

Can you expand on this a bit? If you attempt to use soft links for you use case, what happens?

Git in general does not follow symlinks, for git, a symlink is a regular file. And if the directory/file it points to is in .gitignore, it will be copied in it's broken (pointing to a non existing directory/file) state.

Suggested implementation

Initially, my thought was something like this:

...
links:
      - source: doc
        target: vendor/doc/gitman
        type: hard

or this:

...
links:
      - source: doc
        target: vendor/doc/gitman
        symbolic: false

Seems good to me. The important part is the feature itself :)

@jacebrowning
Copy link
Owner

Most likely this function will need an additional parameter if someone wants to work on a proposal.

@jacebrowning jacebrowning changed the title Request for feature: hard links Support creating hard links (in additional to symlinks) Mar 7, 2023
@dxlr8r
Copy link
Author

dxlr8r commented May 12, 2023

Not an experience Python programmer myself, but I can take a look at it. First, how do I get started? I want to import shell.py to get access to functions mkdir, call etc. but I seem to be unable to do this. I could not find developer documentation either.

@dxlr8r
Copy link
Author

dxlr8r commented May 16, 2023

Trying to build following instructions https://gitman.readthedocs.io/en/latest/about/contributing/#installation , from my virtual environment:

(mygitman) sBook github/gitman % make all
poetry run isort gitman tests
('Configuration file exists at /Users/simen/Library/Application Support/pypoetry, reusing this directory.\n\nConsider moving TOML configuration files to /Users/simen/Library/Application Support/pypoetry, as support for the legacy directory will be removed in an upcoming release.',)
poetry run black gitman tests
('Configuration file exists at /Users/simen/Library/Application Support/pypoetry, reusing this directory.\n\nConsider moving TOML configuration files to /Users/simen/Library/Application Support/pypoetry, as support for the legacy directory will be removed in an upcoming release.',)

poetry run mypy gitman tests --config-file=.mypy.ini
('Configuration file exists at /Users/simen/Library/Application Support/pypoetry, reusing this directory.\n\nConsider moving TOML configuration files to /Users/simen/Library/Application Support/pypoetry, as support for the legacy directory will be removed in an upcoming release.',)
Success: no issues found in 34 source files
poetry run pylint gitman tests --rcfile=.pylint.ini
('Configuration file exists at /Users/simen/Library/Application Support/pypoetry, reusing this directory.\n\nConsider moving TOML configuration files to /Users/simen/Library/Application Support/pypoetry, as support for the legacy directory will be removed in an upcoming release.',)
Traceback (most recent call last):
  File "/Users/simen/projects/private/github/gitman/mygitman/bin/pylint", line 4, in <module>
    run_pylint()
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/__init__.py", line 22, in run_pylint
    PylintRun(sys.argv[1:])
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/lint/run.py", line 349, in __init__
    linter.check(args)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/lint/pylinter.py", line 862, in check
    self._check_files(
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/lint/pylinter.py", line 896, in _check_files
    self._check_file(get_ast, check_astroid_module, name, filepath, modname)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/lint/pylinter.py", line 922, in _check_file
    check_astroid_module(ast_node)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/lint/pylinter.py", line 1054, in check_astroid_module
    retval = self._check_astroid_module(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/lint/pylinter.py", line 1099, in _check_astroid_module
    walker.walk(ast_node)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/utils/ast_walker.py", line 75, in walk
    self.walk(child)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/utils/ast_walker.py", line 75, in walk
    self.walk(child)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/utils/ast_walker.py", line 75, in walk
    self.walk(child)
  [Previous line repeated 1 more time]
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/utils/ast_walker.py", line 72, in walk
    callback(astroid)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/checkers/base.py", line 782, in visit_call
    self._check_inferred_class_is_abstract(inferred, node)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/checkers/base.py", line 799, in _check_inferred_class_is_abstract
    abstract_methods = _has_abstract_methods(inferred)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/checkers/base.py", line 386, in _has_abstract_methods
    return len(utils.unimplemented_abstract_methods(node)) > 0
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/checkers/utils.py", line 825, in unimplemented_abstract_methods
    inferred = safe_infer(obj)
               ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/checkers/utils.py", line 1143, in safe_infer
    value = next(infer_gen)
            ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred
    yield next(generator)
          ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 96, in wrapped
    res = next(generator)
          ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/bases.py", line 136, in _infer_stmts
    for inferred in stmt.infer(context=context):
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/util.py", line 160, in limit_inference
    yield from islice(iterator, size)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/context.py", line 113, in cache_generator
    for result in generator:
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred
    yield next(generator)
          ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 96, in wrapped
    res = next(generator)
          ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/inference.py", line 227, in infer_call
    for callee in self.func.infer(context):
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/util.py", line 160, in limit_inference
    yield from islice(iterator, size)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/context.py", line 113, in cache_generator
    for result in generator:
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred
    yield next(generator)
          ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 96, in wrapped
    res = next(generator)
          ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/bases.py", line 136, in _infer_stmts
    for inferred in stmt.infer(context=context):
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/util.py", line 160, in limit_inference
    yield from islice(iterator, size)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/context.py", line 113, in cache_generator
    for result in generator:
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred
    yield next(generator)
          ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 96, in wrapped
    res = next(generator)
          ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/bases.py", line 136, in _infer_stmts
    for inferred in stmt.infer(context=context):
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/util.py", line 160, in limit_inference
    yield from islice(iterator, size)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/context.py", line 113, in cache_generator
    for result in generator:
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred
    yield next(generator)
          ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 93, in wrapped
    generator = _func(node, context, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/inference.py", line 273, in infer_import_from
    module = self.do_import_module()
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/mixins.py", line 99, in do_import_module
    return mymodule.import_module(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/scoped_nodes.py", line 642, in import_module
    return MANAGER.ast_from_module_name(absmodname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/manager.py", line 189, in ast_from_module_name
    return self.ast_from_file(found_spec.location, modname, fallback=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/manager.py", line 98, in ast_from_file
    return AstroidBuilder(self).file_build(filepath, modname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/builder.py", line 137, in file_build
    module = self._data_build(data, modname, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/builder.py", line 191, in _data_build
    module = builder.visit_module(node, modname, node_file, package)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/rebuilder.py", line 113, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/rebuilder.py", line 113, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/rebuilder.py", line 125, in visit
    return visit_method(node, parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/rebuilder.py", line 633, in visit_functiondef
    return self._visit_functiondef(nodes.FunctionDef, node, parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/rebuilder.py", line 623, in _visit_functiondef
    body=[self.visit(child, newnode) for child in node.body],
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/rebuilder.py", line 623, in <listcomp>
    body=[self.visit(child, newnode) for child in node.body],
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/rebuilder.py", line 123, in visit
    visit_method = getattr(self, visit_name)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'TreeRebuilder' object has no attribute 'visit_match'. Did you mean: 'visit_with'?
make: *** [check] Error 1
Exit: 2
(mygitman) sBook github/gitman %

@dxlr8r
Copy link
Author

dxlr8r commented Jun 10, 2023

I gave up using macOS, so tried using Debian Bookworm. Sadly got many of the same issues as I did with macOS. I found some references to that Python 3.11 was too new etc. so I installed pyenv in Debian, and found Python 3.8.17 working perfectly.

Now the make commands actually work.

And the code is done as well. I am not a Python developer, and could not wrap my head around the test framework though, but except for that everything should be in place and working.

@dxlr8r
Copy link
Author

dxlr8r commented Jun 10, 2023

#310

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants