nuclei fails to start when ~/.config is not writable

[ZhongRuoyu]

Nuclei version:

nuclei 2.9.2 (built from https://github.com/projectdiscovery/nuclei/archive/v2.9.2.tar.gz)

Current Behavior:

nuclei fails to start when the .config directory under home directory is not writable:

$ nuclei -target google.com -t test.yaml
[ERR] failed to create config directory at /Users/brew/.config/nuclei got: mkdir /Users/brew/.config/nuclei: operation not permitted
[ERR] failed to write config file at /Users/brew/.config/nuclei/.templates-config.json got: [:RUNTIME] could not create nuclei config directory at /Users/brew/.config/nuclei <- mkdir /Users/brew/.config/nuclei: operation not permitted

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.9.2

		projectdiscovery.io

[INF] nuclei-templates are not installed, installing...
[INF] nuclei-templates are not installed, installing...
[FTL] Could not create runner: [:RUNTIME] could not create config file <- open /Users/brew/.config/nuclei/reporting-config.yaml: no such file or directory

Expected Behavior:

The .config directory under home directory can be unwritable for some reason. For instance, when nuclei is being run in a sandboxed environment, or when the .config is deliberately made read-only.

nuclei should be able to accept a config directory other than the default one; or alternatively, it should not always attempt at creating the directory at startup.

It would probably help to have a flag -config-dir or an environment variable NUCLEI_CONFIG_DIR that allows overriding the default config directory.

Steps To Reproduce:

1. chmod -rx ~/.config
2. nuclei -target google.com -t test.yaml
3. See error.

Anything else:

This error was observed while packaging nuclei for Homebrew in Homebrew/homebrew-core#128819. Homebrew's tests are executed in a temporary sandboxed environment, where the HOME environment variable is changed to a temporary directory and everything outside is not writable. However, getDefaultConfigDir's implementation uses Go's os/user package to determine the home directory; os/user retrieves the home directory information from the system interface (pwd.h), which does not seem to respect the sandbox settings.

[ehsandeep]

@ZhongRuoyu, thanks for reporting this, there is already a flag for this specific case.

   -config-directory string       override the default config path ($home/.config)

[ehsandeep]

cc @tarunKoyalwar , NUCLEI_CONFIG_DIR could be a good addition for better UX

[ZhongRuoyu]

Thanks for your quick response @ehsandeep. May I know how it can be used? I am still able to reproduce the error with nuclei 2.9.2:

$ ./nuclei start -target google.com -config-directory /tmp
[ERR] failed to create config directory at /home/ruoyu/.config/nuclei got: mkdir /home/ruoyu/.config/nuclei: permission denied
[ERR] failed to write config file at /home/ruoyu/.config/nuclei/.templates-config.json got: [:RUNTIME] could not create nuclei config directory at /home/ruoyu/.config/nuclei <- mkdir /home/ruoyu/.config/nuclei: permission denied

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.9.2

		projectdiscovery.io

[INF] Your current nuclei-templates  are outdated. Latest is v9.4.2
[FTL] Could not create runner: [:RUNTIME] could not create config file <- open /home/ruoyu/.config/nuclei/reporting-config.yaml: permission denied

$ ./nuclei -version
[ERR] failed to create config directory at /home/ruoyu/.config/nuclei got: mkdir /home/ruoyu/.config/nuclei: permission denied
[ERR] failed to write config file at /home/ruoyu/.config/nuclei/.templates-config.json got: [:RUNTIME] could not create nuclei config directory at /home/ruoyu/.config/nuclei <- mkdir /home/ruoyu/.config/nuclei: permission denied
[INF] Nuclei Engine Version: v2.9.2

If I understand it correctly, this getDefaultConfigDir invocation in the init function makes it so that ~/.config/nuclei is always created unconditionally:

nuclei/v2/pkg/catalog/config/nucleiconfig.go

Lines 271 to 277 in e3ce33a

	func init() {
	ConfigDir := getDefaultConfigDir()
	if !fileutil.FolderExists(ConfigDir) {
	if err := fileutil.CreateFolder(ConfigDir); err != nil {
	gologger.Error().Msgf("failed to create config directory at %v got: %s", ConfigDir, err)
	}
	}

nuclei/v2/pkg/catalog/config/nucleiconfig.go

Lines 298 to 309 in e3ce33a

	func getDefaultConfigDir() string {
	// Review Needed: Earlier a dependency was used to locate home dir
	// i.e "github.com/mitchellh/go-homedir" not sure if it is needed
	// Even if such case exists it should be abstracted via below function call in utils/folder
	homedir := folderutil.HomeDirOrDefault("")
	// TBD: we should probably stick to specification and use config directories provided by distro
	// instead of manually creating one since $HOME/.config/ is config directory of Linux desktops
	// Ref: https://pkg.go.dev/os#UserConfigDir
	// some distros like NixOS or others have totally different config directories this causes issues for us (since we are not using os.UserConfigDir)
	userCfgDir := filepath.Join(homedir, ".config")
	return filepath.Join(userCfgDir, "nuclei")
	}

[tarunKoyalwar]

@ZhongRuoyu ,you can change userCfgDir to

userCfgDir , _ := os.UserConfigDIr()

[tarunKoyalwar]

however note that this is major change and any pre-existing configs will be discarded

[ZhongRuoyu]

Thanks @tarunKoyalwar. Yes, telling from https://cs.opensource.google/go/go/+/master:src/os/file.go;l=460, the change you proposed seems to be able to solve the issue, because $HOME is being considered. However, since (as you mentioned) this is a breaking change, can I expect and request it be finalised and committed (perhaps with proper error handling) before we proceed to ship the patch? I am requesting this because this is not good for end users' experience if it is only a temporary workaround.

[tarunKoyalwar]

@ZhongRuoyu yeah we will sort it out with proper migration in next release . the change also need to be done upstream in goflags .

[tarunKoyalwar]

TODO

• update config,Home dir logic in goflags
• when ^ is merged update same logic in nuclei

[S4lt5]

I have been running this with lambda + nuclearpond and @tarunKoyalwar 's change fixes the config problems, but note that the routine to find the nuclei-templates directory needs to be changed in the same way, I believe.

I think this can be changed in the same file in init -> DefaultConfig assignment and use os.UserHomeDir()

[S4lt5]

I'm going to open a PR just to start the dialog, but I've been working with nuclei in a lambda for a few months now and making fixes recommended in this thread worked great. I'd love to not have to use a custom fork of nuclei and just use the main branch.

[S4lt5]

I've gotta get my head around it, but the latest versions are back to not working with this change, I am starting to get template errors, so I really need to figure this out eventually. I think I am still stuck on 2.9.6 and am starting to get template errors

[tarunKoyalwar]

@S4lt5 since this is a breaking change we are hoping to release it with next major release . which will happen soon probably within a month

but the latest versions are back to not working with this change

is doing this change #3576 (comment) on a nuclei fork not resolving the issue ?? or are you perhaps facing other issue

[S4lt5]

The last time I checked, it failed in another area, but this is because my whole filesystem is read-only as I am running in a lambda function. Overall the experience is wonderful, i am able to scan 10s of thousands of sites in 10 minutes when running a single template, I just have a lot of backlog so I haven't really gotten around to fixing it yet.

[tarunKoyalwar]

@S4lt5 @ZhongRuoyu with this PR nuclei will use platform / sandbox specific config and cache directories and fixes this ~/.config not writeable issue

can you try nuclei from this PR and check if you are still facing any similar issue

[S4lt5]

Awesome, I will give it a shot as I am rebuilding in a new environment this week already.

[S4lt5]

This appears to have done the trick, doing a rather large scan and will see how my results turn up!

[S4lt5]

Seems great, thank you very much PD Team! I can drop my custom fork now, and rejoice!

[ZhongRuoyu]

Thanks @tarunKoyalwar and the Nuclei team! I gave the PR a try, and I can see that the issue is resolved. Closing now.