Project Evolution

[acrois]

Hello, I would like to propose a plan for the future of this project.

Purpose:

It will give the community of developers and users a common structure to collaborate as well as improve the quality and speed of archiving the web.

Scary issues out of the way first: We don't want to make any license changes, avoid major (breaking) refactors (requiring end-user changes), and maintain support for existing feature functionality. We should focus new development efforts on performance, stability, scalability, features, and pain points (a little of everything). We will want to encourage the adoption of the library and improve the user experience of the application. We will want to establish and grow the project's visibility by providing a tool to aspiring archivers (known by some as web preservationists) to use as well as give them a clear way to contribute to the massive work that needs to be done in order to preserve the web as we know it.

As a result, I will present to you an outline of the things I would like for us to agree and embark on with this project. I have roughly ordered them for (my perceived) importance to the project (vs complexity vs dependency). This is done considering what falls within the aforementioned categories of visibility, stability, performance, and features. This list and its ordering are subject to change as community prioritization process happens and we take this project to the next level. I would love to have a discussion in the comments if you are curious about any of this. After all, we need consensus to be considered a community. This is just a launch point for us to get started from.

I truly believe that all this is achievable, but it will require help from the community (you). Without any further feedback, I will be personally pitching in a lot on the code, as well as provide consultancy for PRs, but this project is bigger than me and I expect that we would like to encourage open collaboration on these issues. I am grateful to have the opportunity to not only use it, but also contribute to this wonderful project. I hope we can evolve this greater, together within the next 3-6 months!

Please review:

Project management
	Discover, define, estimate tasks
		GitHub Projects or other project management/tracking software
	Contribution guidelines
		Application versioning - Semantic versioning
		Git style
			Git flow?
		Code standards
			Linting
		Release creation documentation
	Application Package Roadmap(s)
		Encompassing all features and their releases in a timeline projected into the future few quarters.

Dockerization
	Document usage in README
	Follow best practices (conventions & security)
		Minimal layers
		Parameter pass-through
		User-space isolation
	Docker-compose/Helm/Kubernetes example(s)
	Test-suite harness
		Automated testing
	Daemon to spin up grab-clients?
		** Do not expose the docker socket!! :D
		Start crawl from dashboard
		Maybe custom resource definition for a grab-client operator in Kubernetes?
	Related:
		#93
		#182
		#149
		#176
		#175
	Browsing, searching the warc locally
		Visually browse/search warc (external project)
		browse @ http host (external project)
		Usage example and documentation

Dashboard improvements
	Re-organize front-end code
		It is kind of bloated and several files over 1k lines, mixed types of content in HTML files, etc.
	Include attempted crawls, connection error, etc. (re: #93), queued URLS (optionally)
		Queued URL logging would be less costly
			If we were able to put a cap and not max out to as much memory as I (you) have available to the browser.
	Ability to manage ignore sets and ignore rules while crawling
		Potentially modify any option
		Related: #3
	No log mode
		Display aggregate crawl stats only
	Authentication provider / access control
	TLS usage example

Server improvements
	Generalize log exporting layer
		Prometheus metric format exporter
			Allows usage by common libs and integration of system reporting services
		Globally & per-crawl

Client improvements
	Investigate and address random crawl hang issue
		May be able to improve the ratio of connections to request-per-second and archival throughput
		GC, database, expensive function calls, still need to do a perf. analysis on the app while crawling
		Related: #60
	More application-specific ignore set defaults to choose from
		Review to ensure top platforms are up to date
		Better support for forums (vBulletin, IPB, XenForo)
			Related: #178
			The defaults are good but not 100% for some versions of these softwares
				(vb tab pages, print views, smf sorting, etc. etc.)
				This has made a difference between a 500k crawl and a 5m crawl for me
	Ability to resume crawl
		Using ID
		Must define default behavior for when the directory exists once implemented
		Resync (recrawl/reindex)
			Delta WARC?
		Related:
			#57
			#58
			#185
	Dead URL / Dupe spotter false positives
		Optimize to avoid dead URLs
		Related: #43
	Windows CRLF/LF/CR adaptability
		Related: #48
	Detect when crawls have been limited, back off exponentially until crawl can resume
		Max retries, automatically adjust rate limiting until "sweet spot" is achieved where blocking does not occur

Documentation improvements
	Document resumption of a grab (specifically ID ("job_data.ident") field)
		Including more explicit docs on STOP and START process signal option (which already supported and code example)
		Different failure scenarios, also: when to just... start over!
	Some copy or quote to inspire people to become archivists
		Link to the ArchiveTeam wiki
	Systems documentation
		Internal concepts, implementation specific details, etc.
		Directory structure / descriptions of operational data files
		More in-depth documentation on gs-server and the role of it in the application architecture
			Functional dependencies between grab-site and gs-server, running grab-site standalone?
		Document parameters and live config update conventions
			Which parameters can be updated live, limitations, etc.
	Management
		Background Processing / Daemonization
		Scaling
		Logging
		Storage Management
		Resource Monitoring (IO, CPU, RAM, HDD)

Application packaging
	Deployment
		GitHub packages (docker, python)
		Docker Hub (docker)

Project website
	Statically generated website for landing, docs, etc.
	GitHub actions & GitHub sites?

[TheTechRobo]

I agree with most of what you said, but I don't like this:

no major (breaking) refactors (requiring end-user changes),

IMO while we should avoid them, there will be times when it is necessary to do it.

[acrois]

@TheTechRobo Thanks for the feedback

I totally agree with you. You have to sometimes. Sweeping statements/ideas like "don't make breaking changes" are never universally true or achievable. What I meant by that is if we build our next milestones in 1-2 quarters based on just the list, I feel strongly that it should be achievable without breaking things at all. I've revised the wording in the issue a bit to try and reflect that.

In the case of grab-site, I think getting a stable 2.x and then thinking about breaking things in a 3.x release version with all that stuff (and potentially upgraded/rewritten features) would be less volatile. I suppose it all just depends on the nature of change.

On that subject:
Maybe there are some other features that people want/need that might/will break things.
What do you think would be the best way to go about discovering and including those issues in creating a more concrete plan for this project?

[TheTechRobo]

What do you think would be the best way to go about discovering and including those issues in creating a more concrete plan for this project?

Probably create a guthib projectboard or milestone for "Look at later; requires refac" or something.

[TomLucidor]

Hope things turn out a bit better in the future