Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: 2 disable host ip from docker compose command #1

Draft
wants to merge 162 commits into
base: master
Choose a base branch
from
Draft

WIP: 2 disable host ip from docker compose command #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
162 commits
Select commit Hold shift + click to select a range
5acf743
helloworld template
longshuicy Nov 4, 2019
ecc5c86
rabbitmq rpc
longshuicy Nov 5, 2019
de64d21
add screen name prompt in rabbitmq rpc mode
longshuicy Nov 5, 2019
3f79265
move all scripts to be rabbitmq
longshuicy Nov 5, 2019
38fa865
read in aws s3 credentials to read and write to S3 bucket
longshuicy Nov 5, 2019
c641964
put botometer_check_bot in docker
longshuicy Nov 5, 2019
6f30891
wrap screen_name_prompt to docker image
longshuicy Nov 5, 2019
645aebb
wrap check screen name to docker image
longshuicy Nov 5, 2019
cabe58a
fix bug; wrap collect time line into docker image
longshuicy Nov 5, 2019
ac516b8
add bulk comparison algorithm in docker
longshuicy Nov 8, 2019
bca2997
wramp get_sim_score and get_personality
longshuicy Nov 8, 2019
e50bf58
clean up
longshuicy Nov 8, 2019
9e624c7
add the requests in the requirement.txt
longshuicy Nov 8, 2019
9cc5675
initial docker compose script
longshuicy Nov 8, 2019
e41e191
the current host setting is working
longshuicy Nov 11, 2019
276a431
modify scripts to accept host name
longshuicy Nov 11, 2019
0068db7
using host-ip is working but now trying to see if i can use internal …
longshuicy Nov 11, 2019
d7c872e
switch to local network; add bae_server
longshuicy Nov 11, 2019
c793766
switch to environment variables
longshuicy Nov 12, 2019
cb55555
push images to docker hub hence modify docker compose script
longshuicy Nov 12, 2019
7cb136f
add volume
longshuicy Nov 15, 2019
2f794e6
relative path not allowed
longshuicy Nov 15, 2019
f03db28
comment out images that needs to interact with s3
longshuicy Nov 18, 2019
c66e216
bulk comparison change to read from local file as well
longshuicy Nov 18, 2019
5031668
construct the correct file path
longshuicy Nov 18, 2019
d8654c5
need to add volume mount to server too
longshuicy Nov 18, 2019
379c3f0
fix bug in bulk comparison and remove print
longshuicy Nov 19, 2019
0ddeea0
update docker-compose
longshuicy Nov 19, 2019
53c3a1b
update the yml file
longshuicy Nov 20, 2019
effbbe8
remove volume in yml file
longshuicy Dec 2, 2019
0a16742
reconfig S3 to connect to minIO
longshuicy Dec 3, 2019
dd9ae71
minIO is working
longshuicy Dec 6, 2019
3bf715a
create default bucket macroscope-bae on start
longshuicy Dec 9, 2019
42579cf
pass bucket name as an environment variable
longshuicy Dec 10, 2019
5fc5535
update yml file
longshuicy Dec 10, 2019
a99c579
remove credentials
longshuicy Dec 10, 2019
7e323d7
add a push script
longshuicy Dec 10, 2019
201baa3
add traceback
longshuicy Dec 13, 2019
1a8fd95
add traceback for bulk comparison, it's able to capture the custom va…
longshuicy Dec 13, 2019
a967ff7
update all algorithms with error reporting
longshuicy Dec 13, 2019
5e1be53
Merge remote-tracking branch 'origin/SMM-171-bae-script-refactoring-t…
longshuicy Dec 20, 2019
114e9c4
write docker script
longshuicy Jan 3, 2020
b87dbc8
Merge branch 'bugfix/SMM-179-error-logging-and-forwarding-in-dockeriz…
longshuicy Jan 3, 2020
8ccd7af
rewrite sentiment analysis rabbitmq handler
longshuicy Jan 6, 2020
35b7847
add requirement and dockerfile for sentiment analysis
longshuicy Jan 6, 2020
c3152ff
rewrite preprocessing
longshuicy Jan 6, 2020
c3eb8b6
rewrite network analysis
longshuicy Jan 6, 2020
bf7d051
rewrite histogram
longshuicy Jan 6, 2020
1d48ce8
rewrite classification
longshuicy Jan 6, 2020
3f6ba04
rewrite classification predict
longshuicy Jan 6, 2020
c4f4e8f
rewrite pipeline
longshuicy Jan 6, 2020
cd5a8b0
docker compose for smile
longshuicy Jan 6, 2020
ced6aa5
rewrite command
longshuicy Jan 7, 2020
96885d9
include screen name prompt
longshuicy Jan 8, 2020
b1bf9ba
update yml
longshuicy Jan 8, 2020
afafec7
edit requirement and dockerfile
longshuicy Jan 9, 2020
ad09977
typo
longshuicy Jan 9, 2020
aab9b03
create clowder collection
longshuicy Jan 9, 2020
4f1f179
add clowder lambda scripts to rabbitmq
longshuicy Jan 9, 2020
2f36e91
add list clowder
longshuicy Jan 9, 2020
dc907e5
include clowder_upload_file
longshuicy Jan 10, 2020
ae2c900
add clowder to rabbitmq
longshuicy Jan 10, 2020
2309714
add pika in the requirement
longshuicy Jan 10, 2020
045338c
temporarily modify how uploadfile works
longshuicy Jan 10, 2020
6eb7589
fix download error in clowder upload file
longshuicy Jan 10, 2020
1e23b14
update how to upload files to clowder
longshuicy Jan 13, 2020
fc31f16
fix clowder upload file and add metadata/tag/description
longshuicy Jan 13, 2020
77e4ddc
add topic modeling batch to rabbitmq
longshuicy Jan 13, 2020
9fbaaed
topic modeling working
longshuicy Jan 13, 2020
2565f67
move twitter ner to batch
longshuicy Jan 14, 2020
31090c3
wrap autophrase into docker
longshuicy Jan 14, 2020
27a3c9a
reuse same docker image for lambda and batch
longshuicy Jan 14, 2020
19f3e86
combine batch sentiment with normal sentiment analysis
longshuicy Jan 14, 2020
e1dd21f
add crimson hexagon monitors to docker
longshuicy Jan 14, 2020
220c6e7
add pika to crimson's monitor
longshuicy Jan 14, 2020
92b877d
bug fix
longshuicy Jan 14, 2020
2247713
rewrite rabbitmq handler to accept message and determine which platfo…
longshuicy Jan 22, 2020
87c8f1c
parse the message and read the platform variable
longshuicy Jan 22, 2020
f62e619
bug
longshuicy Jan 22, 2020
64cfc19
bug fix
longshuicy Jan 22, 2020
02b6fa8
write postToAWSLambda function
longshuicy Jan 23, 2020
2af935c
add batch invoke script
longshuicy Jan 23, 2020
1be9789
add build for the app server
longshuicy Jan 23, 2020
26ec418
replace old handler with new one
longshuicy Jan 23, 2020
6404b89
add functions
longshuicy Jan 23, 2020
350cb08
rewrite handler based on implemetation
longshuicy Jan 27, 2020
96e7f70
use explicit host ip instead of network name
longshuicy Jan 27, 2020
f42b274
queue name debug
longshuicy Jan 27, 2020
2320903
remove not used script
longshuicy Jan 27, 2020
b96b763
add image_crawler
longshuicy Jan 27, 2020
8250085
debug in image crawler
longshuicy Jan 27, 2020
c23a8ce
fix bug in reddit comment
longshuicy Jan 28, 2020
3e6f18f
remove tmp relative path
longshuicy Jan 28, 2020
2d74306
make sure the encoding and decoding of name entity recognition
longshuicy Jan 29, 2020
7e0a082
ner docker file need to download glove dataset
longshuicy Jan 29, 2020
cbc3b21
add email password environment variable to network analysis, sentimen…
longshuicy Jan 29, 2020
45fa4b2
debugging classification train
longshuicy Jan 29, 2020
8e25159
not using heartbeat
longshuicy Jan 30, 2020
952d17a
make sure only lambda use two way rabbitmq communication
longshuicy Jan 30, 2020
b795279
remove heartbeat restriction since it's important
longshuicy Jan 30, 2020
891dcc5
add ways to invoke remote aws job submission for sentiment analysis u…
longshuicy Jan 31, 2020
b0f4d32
modify aws lambda preprocessing to read local minio datasets
longshuicy Feb 3, 2020
c27a655
fix bug
longshuicy Feb 3, 2020
301e0f1
refactoring aws network lambda to accept local minio dataset
longshuicy Feb 3, 2020
afd72c2
refactoring classification split aws lambda to use local object
longshuicy Feb 3, 2020
d40d1a9
refactoring classification train aws lambda to work with local minio …
longshuicy Feb 3, 2020
885149a
refactoring aws lambda classification predict to be able to use local…
longshuicy Feb 3, 2020
3facda3
refactoring topic modeling batch to be able to work with local minio …
longshuicy Feb 3, 2020
626e65c
fix the bug in topic modeling that records the credentials
longshuicy Feb 3, 2020
0fd7f7d
update NER batch to use local minio
longshuicy Feb 3, 2020
0662edc
batch networkx
longshuicy Feb 3, 2020
440e1e0
refactoring pipline, preprocessing, sentiment, authorphase to able to…
longshuicy Feb 3, 2020
05bb435
refactoring lambda handler to remove credentials in config.json
longshuicy Feb 4, 2020
52411bc
mount tag directory so tag can persist through different docker-compo…
longshuicy Feb 4, 2020
a2be375
add cronjob to clean /tmp folders
longshuicy Feb 7, 2020
e00d020
add cron job to clean /tmp directory
longshuicy Feb 7, 2020
619ff88
rewrite image_crawler so it works for batch with local minio dataset
longshuicy Feb 10, 2020
a51e418
make reddit comment collection work with local minio dataset
longshuicy Feb 10, 2020
4232d58
refactor yml file
longshuicy Feb 28, 2020
dc7ed29
modify gitignore
longshuicy Apr 2, 2020
00dccad
yml
longshuicy Apr 6, 2020
9f20709
add volume mount to mongo
longshuicy Sep 8, 2020
6134a4b
commented out build
longshuicy Sep 8, 2020
4852165
update docker push script
longshuicy Sep 9, 2020
488f5dc
fix bug
longshuicy Sep 10, 2020
b893df0
fix nltk version to be 3.2.5
longshuicy Sep 16, 2020
fb3250d
writeToS3 should from autophrase repo
longshuicy Nov 23, 2020
feb6b7b
read email setting as environment variable
longshuicy Dec 10, 2020
ef8f6f8
specify python to 3.7 in dockerfile
longshuicy Mar 31, 2021
f9e9f34
change sentiment dockerfile to load pre-trained model
longshuicy Mar 31, 2021
1b53a33
add a build script
longshuicy Apr 1, 2021
39a96c6
fix sentiment analysis
longshuicy Apr 16, 2021
78d5ecc
extend heartbeat
longshuicy Apr 16, 2021
c402046
fix sentiment
longshuicy Apr 16, 2021
c410bf0
update bae yml file
longshuicy Apr 27, 2021
ab6a10c
update botometer
longshuicy Apr 28, 2021
0137d75
update rapidapi key to the correct image
longshuicy Apr 28, 2021
7df0554
pass additional environment variables
longshuicy May 24, 2021
3bf8668
update check screen name container
longshuicy May 24, 2021
da93e7d
update get personality container
longshuicy May 24, 2021
7a21166
upgrade requirement.txt
longshuicy May 24, 2021
d095463
Merge branch 'master' into feature/SMM-180-wrap-smile-into-docker-con…
longshuicy May 24, 2021
6655ef5
initialize utku
longshuicy May 24, 2021
2283987
wrote queue name
longshuicy May 24, 2021
67a17fc
add utku brand personality container
longshuicy May 25, 2021
9e4eba6
rewrite dockerfile for utku
longshuicy May 25, 2021
d98a700
add gitignore
longshuicy May 25, 2021
5937597
need to update collect timeline
longshuicy May 25, 2021
0004952
fix bug in collect timeline
longshuicy May 25, 2021
1a1911b
update get_personality
longshuicy May 25, 2021
3339b14
update bulk comparison
longshuicy May 25, 2021
8231874
increase heartbeat timeout; remove default email password/host
longshuicy May 25, 2021
41fedb6
update yml file
longshuicy Jun 1, 2021
b7ebb8e
remove clowder and crimson in docker compose since we no longer suppo…
longshuicy Jan 12, 2022
218505a
need to set port for mongodb and redis
longshuicy Jan 18, 2022
d2b008b
hostname minio works
longshuicy Jan 3, 2023
02599f5
put nginx infront of minio
longshuicy Jan 3, 2023
d769a07
resume
longshuicy Jan 3, 2023
b00782a
nginx minio console works
longshuicy Jan 3, 2023
72ffb60
still cannot connect to localhost 9000
longshuicy Jan 3, 2023
b6a2533
do not set minio root user and password; use access key and secret in…
longshuicy Jan 3, 2023
89ba684
rename minio1 to minio
longshuicy Jan 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
5 changes: 4 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@ email_password.txt
__pycache__
glove.twitter.27B.*
numberbatch-en-17.04b.*
bae_docker.sh
password.txt
docker-command-smile.sh
docker-command-bae.sh
test_orca
test_orca
checkpoint-17315-epoch-5
159 changes: 0 additions & 159 deletions batch/batch_classification_predict.py
View file
Open in desktop

This file was deleted.

177 changes: 0 additions & 177 deletions batch/batch_classification_split.py
View file
Open in desktop

This file was deleted.

Loading