Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Develop #1542

Merged
merged 3,336 commits into from
Jan 19, 2025
Merged

Develop #1542

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3336 commits
Select commit Hold shift + click to select a range
a90fc0b
ci: trigger workflow
tikazyq Nov 19, 2024
cb301fa
ci: trigger workflow
tikazyq Nov 19, 2024
4141b47
fix: incorrect exec bin name
tikazyq Nov 19, 2024
e62b1c1
fix: incorrect exec bin name
tikazyq Nov 19, 2024
300aec8
ci: updated workflow
tikazyq Nov 19, 2024
16cd4aa
ci: updated workflow
tikazyq Nov 19, 2024
e38ef0d
ci: updated workflow
tikazyq Nov 19, 2024
0e8d127
ci: updated workflow
tikazyq Nov 19, 2024
6df52e7
fix: unable to connect to grpc server issue
tikazyq Nov 19, 2024
1a93b14
fix: unable to connect to grpc server issue
tikazyq Nov 19, 2024
4ed7fcf
feat: added health check for worker service
tikazyq Nov 19, 2024
dc9f62d
feat: added health check for worker service
tikazyq Nov 19, 2024
d5db15a
chore: added debug code
tikazyq Nov 19, 2024
bbeb752
chore: added debug code
tikazyq Nov 19, 2024
ff064b2
ci: updated dockerfiles
tikazyq Nov 20, 2024
1e2cad8
ci: updated dockerfiles
tikazyq Nov 20, 2024
0f09dcb
ci: updated base-image
tikazyq Nov 20, 2024
2203be9
ci: updated base-image
tikazyq Nov 20, 2024
a759b3d
ci: updated dockerfile
tikazyq Nov 20, 2024
874f02a
ci: updated dockerfile
tikazyq Nov 20, 2024
43b5e7e
ci: updated base-image
tikazyq Nov 20, 2024
6babe32
ci: updated base-image
tikazyq Nov 20, 2024
b67e7f0
ci: updated base-image
tikazyq Nov 20, 2024
46100eb
ci: updated base-image
tikazyq Nov 20, 2024
56655b3
fix: code warnings
tikazyq Nov 20, 2024
76aabc3
fix: code warnings
tikazyq Nov 20, 2024
aed1bdd
ci: updated base-image
tikazyq Nov 20, 2024
db4dce3
ci: updated base-image
tikazyq Nov 20, 2024
2448638
ci: updated base-image
tikazyq Nov 20, 2024
d407cc0
ci: updated base-image
tikazyq Nov 20, 2024
64bd5b4
ci: updated base-image
tikazyq Nov 20, 2024
a781d12
ci: updated base-image
tikazyq Nov 20, 2024
18f3a8e
ci: optimized base-image size
tikazyq Nov 20, 2024
1c618bd
ci: optimized base-image size
tikazyq Nov 20, 2024
db6a379
feat: added utils cli
tikazyq Nov 20, 2024
139d7de
feat: added utils cli
tikazyq Nov 20, 2024
36b7b6d
refactor: code cleanup
tikazyq Nov 20, 2024
b326134
refactor: code cleanup
tikazyq Nov 20, 2024
7d3082c
feat: added ipc to task runner
tikazyq Nov 21, 2024
d50383d
feat: added ipc to task runner
tikazyq Nov 21, 2024
4c0922c
fix: ci issue for base-image
tikazyq Nov 22, 2024
5f0ffa3
fix: ci issue for base-image
tikazyq Nov 22, 2024
be45478
chore: added debug code
tikazyq Nov 22, 2024
f64f407
chore: added debug code
tikazyq Nov 22, 2024
cf15e4f
test: fix test cases
tikazyq Nov 22, 2024
b12251d
test: fix test cases
tikazyq Nov 22, 2024
cc3da1d
ci: updated base-image
tikazyq Nov 22, 2024
76c6a48
ci: updated base-image
tikazyq Nov 22, 2024
f5965f2
test: fix test cases
tikazyq Nov 22, 2024
d00dd94
test: fix test cases
tikazyq Nov 22, 2024
5c94bfb
chore: added debug code
tikazyq Nov 22, 2024
53d3edd
chore: added debug code
tikazyq Nov 22, 2024
c558ef4
test: fix test cases
tikazyq Nov 22, 2024
da5c3b0
test: fix test cases
tikazyq Nov 22, 2024
739d796
test: fix test cases
tikazyq Nov 22, 2024
1100783
test: fix test cases
tikazyq Nov 22, 2024
6a65067
test: fix test cases
tikazyq Nov 22, 2024
aa1b01b
test: fix test cases
tikazyq Nov 22, 2024
23762cf
test: fix test cases
tikazyq Nov 22, 2024
eee10ea
test: fix test cases
tikazyq Nov 22, 2024
e5db1ce
fix: runner cancel issue
tikazyq Nov 22, 2024
8357dc6
fix: runner cancel issue
tikazyq Nov 22, 2024
0649dd7
fix: runner cancel issue
tikazyq Nov 22, 2024
36e7f94
fix: runner cancel issue
tikazyq Nov 22, 2024
3ae5c42
test: added more test cases
tikazyq Nov 22, 2024
9f1d7dd
test: added more test cases
tikazyq Nov 22, 2024
4d1bf18
test: fix test case issue
tikazyq Nov 22, 2024
a6e845a
test: fix test case issue
tikazyq Nov 22, 2024
7a97b70
refactor: optimized imports
tikazyq Nov 22, 2024
1a460e3
refactor: optimized imports
tikazyq Nov 22, 2024
331ddb4
ci: updated base-image
tikazyq Nov 22, 2024
ecd156a
ci: updated base-image
tikazyq Nov 22, 2024
b4862f9
test: fix test cases issues
tikazyq Nov 22, 2024
9560da6
test: fix test cases issues
tikazyq Nov 22, 2024
a43c118
chore: updated deps
tikazyq Nov 22, 2024
2b3473d
chore: updated deps
tikazyq Nov 22, 2024
c07a2db
ci: updated dockerfile
tikazyq Nov 22, 2024
c4adcd3
ci: updated dockerfile
tikazyq Nov 22, 2024
95971ed
ci: updated dockerfile
tikazyq Nov 22, 2024
c8c86e1
ci: updated dockerfile
tikazyq Nov 22, 2024
5ec6f46
ci: updated base-image
tikazyq Nov 22, 2024
92f4604
ci: updated base-image
tikazyq Nov 22, 2024
dd06ae0
chore: updated deps
tikazyq Nov 22, 2024
497f615
chore: updated deps
tikazyq Nov 22, 2024
a7fccb9
chore: updated deps
tikazyq Nov 22, 2024
5237f06
chore: updated deps
tikazyq Nov 22, 2024
fd74da6
ci: updated base-image
tikazyq Nov 22, 2024
c5e0ba8
ci: updated base-image
tikazyq Nov 22, 2024
6c22197
ci: updated workflow
tikazyq Nov 22, 2024
1984831
ci: updated workflow
tikazyq Nov 22, 2024
bfd2e21
ci: updated workflow
tikazyq Nov 22, 2024
cd2abe9
ci: updated workflow
tikazyq Nov 22, 2024
9df54c1
ci: updated workflow
tikazyq Nov 22, 2024
534f66e
ci: updated workflow
tikazyq Nov 22, 2024
5353c50
fix: unable to start api
tikazyq Nov 22, 2024
7a322ae
fix: unable to start api
tikazyq Nov 22, 2024
ed8ab6c
fix: unable to start api
tikazyq Nov 22, 2024
858e5c2
fix: unable to start api
tikazyq Nov 22, 2024
26dc9bb
chore: updated deps
tikazyq Nov 22, 2024
b98c72d
chore: updated deps
tikazyq Nov 22, 2024
fbc5e71
ci: updated workflow
tikazyq Nov 22, 2024
2a161d1
ci: updated workflow
tikazyq Nov 22, 2024
b4c233f
chore: updated deps
tikazyq Nov 22, 2024
a40bcee
chore: updated deps
tikazyq Nov 22, 2024
f016d27
ci: updated base-image
tikazyq Nov 23, 2024
b125ba3
ci: updated base-image
tikazyq Nov 23, 2024
5a7272e
fix: create index errors
tikazyq Nov 23, 2024
eed93ae
fix: create index errors
tikazyq Nov 23, 2024
728cbba
fix: index creation issue
tikazyq Nov 23, 2024
a127587
fix: index creation issue
tikazyq Nov 23, 2024
d3614cb
ci: updated base-image
tikazyq Nov 24, 2024
f9e7b16
ci: updated base-image
tikazyq Nov 24, 2024
90f576c
ci: fix issue
tikazyq Nov 24, 2024
f6a9964
ci: fix issue
tikazyq Nov 24, 2024
1396d33
ci: updated base-image
tikazyq Nov 24, 2024
a3b395e
ci: updated base-image
tikazyq Nov 24, 2024
5952fef
ci: fix issues
tikazyq Nov 24, 2024
af3749d
ci: fix issues
tikazyq Nov 24, 2024
dd090ab
ci: trigger workflow
tikazyq Nov 24, 2024
f1d0613
ci: trigger workflow
tikazyq Nov 24, 2024
40688ed
ci: updated base-image
tikazyq Nov 24, 2024
3a49f48
ci: updated base-image
tikazyq Nov 24, 2024
cd429d4
ci: updated dockerfile
tikazyq Nov 24, 2024
0912395
ci: updated dockerfile
tikazyq Nov 24, 2024
c337bf0
ci: updated dockerfile
tikazyq Nov 24, 2024
5e0fd1a
ci: updated dockerfile
tikazyq Nov 24, 2024
8df054e
ci: trigger workflow
tikazyq Nov 24, 2024
a6ac5b4
ci: trigger workflow
tikazyq Nov 24, 2024
04887c6
ci: updated docker-init.sh
tikazyq Nov 24, 2024
fd1ecd1
ci: updated docker-init.sh
tikazyq Nov 24, 2024
f24579d
ci: updated workflow
tikazyq Nov 24, 2024
1e65503
ci: updated workflow
tikazyq Nov 24, 2024
9281f44
ci: trigger workflow
tikazyq Nov 24, 2024
b9a2d03
ci: trigger workflow
tikazyq Nov 24, 2024
601db5a
refactor: optimized code
tikazyq Nov 24, 2024
24561bc
refactor: optimized code
tikazyq Nov 24, 2024
1937dc6
test: fix issues
tikazyq Nov 24, 2024
2764c43
test: fix issues
tikazyq Nov 24, 2024
fc036fd
ci: trigger workflow
tikazyq Nov 24, 2024
4bd28d5
ci: trigger workflow
tikazyq Nov 24, 2024
ebc469b
ci: trigger workflow
tikazyq Nov 29, 2024
8e398bd
ci: trigger workflow
tikazyq Nov 29, 2024
b359b74
ci: trigger workflow
tikazyq Nov 29, 2024
55b8b7c
ci: trigger workflow
tikazyq Nov 29, 2024
f9cc305
ci: updated workflow
tikazyq Nov 29, 2024
deccf4c
ci: updated workflow
tikazyq Nov 29, 2024
83e45df
fix: duplicated index recreation
tikazyq Dec 11, 2024
2a54762
fix: duplicated index recreation
tikazyq Dec 11, 2024
102b327
fix: duplicated index recreation
tikazyq Dec 11, 2024
19f1993
fix: duplicated index recreation
tikazyq Dec 11, 2024
a398313
ci: updated Dockerfile
tikazyq Dec 11, 2024
456fb4b
ci: updated Dockerfile
tikazyq Dec 11, 2024
baf611d
chore: updated deps
tikazyq Dec 11, 2024
2a7aa76
chore: updated deps
tikazyq Dec 11, 2024
a595315
feat: improve api token verification compatibility
tikazyq Dec 11, 2024
3076775
feat: improve api token verification compatibility
tikazyq Dec 11, 2024
d4b29a5
fix: added default fields
tikazyq Dec 11, 2024
6b78afe
fix: added default fields
tikazyq Dec 11, 2024
1f8f22e
fix: optimized node runners calculation
tikazyq Dec 11, 2024
1fe74fa
fix: optimized node runners calculation
tikazyq Dec 11, 2024
eaa834e
feat: allow set max runners for nodes
tikazyq Dec 11, 2024
272371d
feat: allow set max runners for nodes
tikazyq Dec 11, 2024
a08aa0c
chore: updated deps
tikazyq Dec 11, 2024
2b93407
chore: updated deps
tikazyq Dec 11, 2024
232b2b8
ci: trigger workflow
tikazyq Dec 11, 2024
5293207
ci: trigger workflow
tikazyq Dec 11, 2024
694cd40
feat: updated the way syncing pypi projects
tikazyq Dec 14, 2024
298420b
feat: updated the way syncing pypi projects
tikazyq Dec 14, 2024
89f93aa
feat: prepare for dependency lang install/setup
tikazyq Dec 14, 2024
25580b4
feat: prepare for dependency lang install/setup
tikazyq Dec 14, 2024
bb66362
chore: updated deps
tikazyq Dec 14, 2024
3b54a63
chore: updated deps
tikazyq Dec 14, 2024
0f034e9
feat: added dependency config setup (wip)
tikazyq Dec 15, 2024
c5c08df
feat: added dependency config setup (wip)
tikazyq Dec 15, 2024
53643da
feat: updated dependency config setup
tikazyq Dec 16, 2024
79c1d5d
feat: updated dependency config setup
tikazyq Dec 16, 2024
14dc8f3
feat: updated dependency config setup
tikazyq Dec 16, 2024
27a8f19
feat: updated dependency config setup
tikazyq Dec 16, 2024
2117c6b
chore: updated deps
tikazyq Dec 16, 2024
c945e09
chore: updated deps
tikazyq Dec 16, 2024
8842621
feat: updated dependency management
tikazyq Dec 17, 2024
e77d4cd
feat: updated dependency management
tikazyq Dec 17, 2024
8913cff
chore: updated deps
tikazyq Dec 17, 2024
f823c7f
chore: updated deps
tikazyq Dec 17, 2024
9071378
fix: ci issue
tikazyq Dec 17, 2024
19b9135
fix: ci issue
tikazyq Dec 17, 2024
5bdcd29
fix: getting stream error for dependency server
tikazyq Dec 18, 2024
f736b2c
fix: getting stream error for dependency server
tikazyq Dec 18, 2024
3636151
Merge branch 'develop' of github.com:crawlab-team/crawlab into develop
tikazyq Dec 19, 2024
e042555
ci: updated workflow
tikazyq Dec 20, 2024
5c4099c
ci: updated workflow
tikazyq Dec 20, 2024
be93f9d
feat: added retry for worker node start
tikazyq Dec 20, 2024
4d28421
chore: updated deps
tikazyq Dec 20, 2024
f5631f9
chore: updated deps
tikazyq Dec 20, 2024
83b81c8
chore: update Docker configuration and ignore files
tikazyq Dec 20, 2024
3932b29
chore: refine Docker build process and update context paths
tikazyq Dec 20, 2024
eed451e
ci: trigger workflow
tikazyq Dec 20, 2024
3cb74d7
feat: enhance gRPC client functionality and improve logging
tikazyq Dec 20, 2024
c897fb5
refactor: streamline error handling and improve HTTP request management
tikazyq Dec 21, 2024
b88b41a
feat: add API configuration options for port and path
tikazyq Dec 21, 2024
41e973a
fix: misaligned nodes when running tasks from a schedule through enha…
tikazyq Dec 21, 2024
75da4ff
refactor: update gRPC client initialization in model service tests
tikazyq Dec 21, 2024
29af5a3
feat: enhance gRPC client with state management and reconnection logic
tikazyq Dec 21, 2024
e44b416
feat: enhance model base service with BSON ID normalization
tikazyq Dec 21, 2024
afe21b7
feat: implement logging interfaces and service logger
tikazyq Dec 21, 2024
bdc347a
feat: add Fatalf logging method to Logger interface and ServiceLogger
Dec 23, 2024
ed8fb78
feat: expand Logger interface and implement additional logging method…
Dec 23, 2024
34455dc
feat: enhance DependencyConfig model with additional fields
Dec 23, 2024
c3f4c4a
feat: enhance gRPC client with structured logging and dependency actions
Dec 23, 2024
99ed439
refactor: improve logging messages and update configuration constants
Dec 23, 2024
e064889
refactor: replace apex/log with structured logger in master and worke…
tikazyq Dec 23, 2024
67165f5
feat: add zip package installation to base image dependencies
tikazyq Dec 23, 2024
3276083
refactor: replace apex/log with structured logger across multiple ser…
tikazyq Dec 24, 2024
a28ffbf
refactor: simplify interfaces and improve configuration handling
tikazyq Dec 24, 2024
dc59599
refactor: remove db module and update imports to core/mongo
Dec 25, 2024
e798226
ci: updated workflow
Dec 25, 2024
3243c2c
refactor: remove db module from Dockerfile and consolidate dependencies
Dec 25, 2024
a13893b
feat: add logo printing functionality and update logger usage
Dec 25, 2024
2a33bd4
chore: update dependencies in go.mod and go.sum files
Dec 25, 2024
9e67e50
fix: add new line after logo and welcome info prints
Dec 25, 2024
9db8193
fix: suppress nginx service start output in docker-init.sh
Dec 25, 2024
c4c7d25
fix: update pyenv installation script to use user-specific environmen…
Dec 25, 2024
4144271
feat: enhance Python installation script with command handling and ve…
Dec 25, 2024
f03b346
feat: enhance Python installation script with requirements handling a…
Dec 25, 2024
17f8917
chore: update dependencies and enhance gRPC services
Dec 25, 2024
a8bf115
fix: update default Python version in installation script to 3.12.8
Dec 25, 2024
5480097
feat: refactor system info retrieval and enhance logo output
tikazyq Dec 29, 2024
ef499a0
fix: improve logging in master and worker services
tikazyq Dec 29, 2024
d4d7714
feat: add setup command to Python installation script
Dec 30, 2024
9bdb0c9
refactor: remove default_version field from DependencyConfig
Dec 30, 2024
3438919
fix: update Python installation script to require version specification
Dec 30, 2024
fec6160
fix: update Python installation command to specify version 3.12 durin…
Dec 30, 2024
3786ef3
refactor: streamline dependency installation in Docker base image
tikazyq Dec 31, 2024
25fe273
refactor: improve logging in gRPC services by removing service prefixes
tikazyq Dec 31, 2024
8916194
feat: enhance Docker base image installation scripts
tikazyq Dec 31, 2024
7b6805a
feat: enhance task runner with improved logging and dependency support
tikazyq Dec 31, 2024
7e7ac62
fix: update default task log path for consistency across environments
tikazyq Jan 1, 2025
db2549e
fix: enhance error logging in file log driver and update default task…
tikazyq Jan 1, 2025
136daff
refactor: improve IPC handling and logging in task runner tests
tikazyq Jan 1, 2025
b056105
feat: add dependency installer service and enhance task runner with d…
tikazyq Jan 1, 2025
47094b8
refactor: update setting routes and enhance dependency management
tikazyq Jan 1, 2025
a585ab1
feat: enhance task runner with task status updates and process comman…
Jan 3, 2025
ff5cd32
refactor: streamline Node.js path configuration in task runner
Jan 3, 2025
37d77f7
refactor: enhance IPC handling in task runner tests
Jan 3, 2025
99c6f42
feat: update Python installation script and requirements
Jan 3, 2025
8aa801e
feat: add Go path configuration to task runner
Jan 6, 2025
f5d9ccf
feat: initialize configuration and enhance IPC handling in task runne…
Jan 6, 2025
8d8b47e
refactor: streamline file service retrieval and enhance spider templa…
Jan 6, 2025
c3c629a
feat: add ToKebabCase utility function for string formatting
tikazyq Jan 6, 2025
c937e0f
refactor: enhance Spider model and string utility functions
Jan 7, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 2 additions & 0 deletions .env.local
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
CRAWLAB_TAG=develop
CRAWLAB_PORT=8082
197 changes: 0 additions & 197 deletions .github/workflows/docker-crawlab-tencent.yml

This file was deleted.

Loading
Loading