Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compare latest branch #298

Open
wants to merge 443 commits into
base: master
from
Open

compare latest branch #298

Changes from all commits
Commits
Show all changes
443 commits
Select commit Hold shift + click to select a range
2605389
minor: update interface for plan and visualize
lf-zhao Jan 13, 2025
366298d
update visualization of transition graph to only show fluents (change…
lf-zhao Jan 14, 2025
25b0beb
update visualization, group by objects
lf-zhao Jan 14, 2025
5750b91
major update to mock spot/robot perceiver; still in progress
lf-zhao Jan 14, 2025
1fac3b7
update console log to show fluents; update other viz stuff
lf-zhao Jan 14, 2025
3fe70c7
move mock env tests
lf-zhao Jan 14, 2025
dd6d4c6
major update: traverse all possible states and visualize the transiti…
lf-zhao Jan 14, 2025
1f951fa
show state sequence
lf-zhao Jan 14, 2025
27ee421
major update: plot graph of transition system with all possible state…
lf-zhao Jan 15, 2025
9b0456c
major fix: remove invalid edges, update graph plotting to show edges/…
lf-zhao Jan 15, 2025
64c57ba
update test for belief space to be more clear
lf-zhao Jan 15, 2025
03a4800
double check filtering of invalid transitions
lf-zhao Jan 15, 2025
4c3c5b0
update visualization options to make easier to see; track self loops …
lf-zhao Jan 15, 2025
b09d330
update plotting
lf-zhao Jan 15, 2025
d3a952e
add utils for getting transitions and edges, update plotting
lf-zhao Jan 15, 2025
8f8084d
add a complete new test for debugging graph transition edges; still n…
lf-zhao Jan 15, 2025
0a7518c
minor fix
lf-zhao Jan 15, 2025
89645d6
update save name for these
lf-zhao Jan 15, 2025
04204f3
fix
lf-zhao Jan 15, 2025
e05906d
separate out transition graph generation and visualization
lf-zhao Jan 16, 2025
c13a2ef
major update: add using `cytoscape.js` to visualize graph data intera…
lf-zhao Jan 16, 2025
45a1fea
update: now can load from local json for saved graph data
lf-zhao Jan 16, 2025
d68bbb5
major update: lots of fixes to make it look better, fix visualization…
lf-zhao Jan 17, 2025
a7eaa86
fix node color issue
lf-zhao Jan 17, 2025
9973725
fix: showing start and goal states in correct color, other fixes and …
lf-zhao Jan 17, 2025
6ded696
test for comparing agent transition and env transition for empty cup …
lf-zhao Jan 17, 2025
83dd9e3
major update: replace with cytoscape.js library for graph viz
lf-zhao Jan 17, 2025
69c4fc9
update html saving
lf-zhao Jan 17, 2025
624d6e2
add html template for graph saving
lf-zhao Jan 17, 2025
1c2550f
fix saving names
lf-zhao Jan 17, 2025
51182ec
major update to graph viz: fix visualizing nodes with correct colors,…
lf-zhao Jan 18, 2025
b8e255c
add new operators for drawer opening task, for agent + env view
lf-zhao Jan 18, 2025
e86e5ad
new test: transition system for cleaning up drawer with 1 object, inc…
lf-zhao Jan 18, 2025
9916f7e
update for belief predicates
lf-zhao Jan 19, 2025
75814bf
minor fix computing paths
lf-zhao Jan 19, 2025
314b90c
add info panel for edge information for operator/transition
lf-zhao Jan 19, 2025
a02bff1
update flag for belief ops
lf-zhao Jan 19, 2025
348ae2e
minor update
lf-zhao Jan 19, 2025
b9660ed
fix test for opening drawer, after fixing operators
lf-zhao Jan 19, 2025
c28fc1a
update
lf-zhao Jan 19, 2025
8320984
update plotting style, from 'dagre' to 'breadthfirst' with curved edges
lf-zhao Jan 19, 2025
a38fa63
update opendrawer ops
lf-zhao Jan 19, 2025
9385828
fix operators for move to hand view + hand observe
lf-zhao Jan 19, 2025
0801eff
fix test for observe cup emptiness
lf-zhao Jan 19, 2025
44fdca9
update parameters
lf-zhao Jan 20, 2025
b54e341
integrate new task as subclass of mock env
lf-zhao Jan 20, 2025
5dcce0b
update task class
lf-zhao Jan 20, 2025
007de78
update
lf-zhao Jan 20, 2025
f51a3ac
major: add new test for creating task using task subclass, building t…
lf-zhao Jan 20, 2025
f869f70
pass env instance in
lf-zhao Jan 20, 2025
7714a5b
name
lf-zhao Jan 20, 2025
b1ca18e
minor
lf-zhao Jan 20, 2025
6c24b63
[in progress] major update to mock env: add VLM predicates, add obser…
lf-zhao Jan 20, 2025
663ca46
[in progress] major update to perceiver: clean up handling of predica…
lf-zhao Jan 20, 2025
38a937d
add note for mock perceiver
lf-zhao Jan 20, 2025
d3a2a03
fix arg type
lf-zhao Jan 21, 2025
8ded816
add unposed image dataclass
lf-zhao Jan 21, 2025
ca205e9
add UnposedImageWithContext
lf-zhao Jan 21, 2025
51dee2a
major update, now collector can complete collection and running: upda…
lf-zhao Jan 21, 2025
aa8a1c6
major update to mock env: update Mock Obs classes for saving and for …
lf-zhao Jan 21, 2025
8c8382c
move mock obs into separate file to avoid cyclic import
lf-zhao Jan 21, 2025
d695b8b
the manual creator class is basically dummy now
lf-zhao Jan 21, 2025
7a1683d
minor
lf-zhao Jan 21, 2025
3812630
new docs: guide of testing planning using mock robot env
lf-zhao Jan 21, 2025
cc4e0c2
update to perceiver
lf-zhao Jan 21, 2025
915bf81
test on loading
lf-zhao Jan 21, 2025
f5981f6
major update to test manual collected env: fully working
lf-zhao Jan 21, 2025
4c910cf
update another test - this may be outdated
lf-zhao Jan 21, 2025
c845572
update docs on perceiver
lf-zhao Jan 21, 2025
6a169f3
[in progress] major update to mock perceiver
lf-zhao Jan 21, 2025
40bf20c
major update to mock spot perceiver: correctly handle predicate value…
lf-zhao Jan 22, 2025
b683ec6
minor
lf-zhao Jan 22, 2025
a5424af
minor
lf-zhao Jan 22, 2025
ea93041
major update: add consistency check for unknown/known, make sure upda…
lf-zhao Jan 22, 2025
43fbd44
working on updating the ground atoms saving logic
lf-zhao Jan 22, 2025
e4af8e4
update
lf-zhao Jan 22, 2025
aa55d28
add new field for saving atom and new GroundTruthPredicate class for …
lf-zhao Jan 22, 2025
a5980c8
major update: clean up state storage logic, separate state graph buil…
lf-zhao Jan 22, 2025
b0a396a
update the test for saving env with images
lf-zhao Jan 22, 2025
2f57972
update previous test
lf-zhao Jan 22, 2025
fee4c9f
update test
lf-zhao Jan 22, 2025
6fec032
update; not fully working with latest ver
lf-zhao Jan 22, 2025
d82f218
update test for new init
lf-zhao Jan 22, 2025
e6e3975
new utils for computing ground atom combinations without state
lf-zhao Jan 22, 2025
facabb9
major update to env: use GroundTruePredicate that directly loads from…
lf-zhao Jan 22, 2025
125a1c4
fixng saving and loading
lf-zhao Jan 22, 2025
1f99a49
udpate sacing loading logic fro perceiver
lf-zhao Jan 22, 2025
a3e2a3a
major update: loading environment state/transition system, lots of cl…
lf-zhao Jan 22, 2025
590b24e
test new perciever with the env
lf-zhao Jan 22, 2025
76295fe
minor
lf-zhao Jan 22, 2025
f414e3b
major update: wrote logsic for saving spot images [in progress]
lf-zhao Jan 22, 2025
b2a6e00
update saving paths
lf-zhao Jan 22, 2025
5026803
update test for planning
lf-zhao Jan 22, 2025
67d567a
update docs for perceiver
lf-zhao Jan 22, 2025
adcef75
note on creating new mock env tasks
lf-zhao Jan 22, 2025
8ec0416
update flag name to use VLM predicates
lf-zhao Jan 23, 2025
f2bec9f
fix saving yaml
lf-zhao Jan 23, 2025
4662c53
upate flag
lf-zhao Jan 23, 2025
f2bfddf
update docs for action wrapping for operators
lf-zhao Jan 23, 2025
4a4da9a
update fixing VLM pred evaluation
lf-zhao Jan 23, 2025
3cd3bb6
fix for vlm pred
lf-zhao Jan 23, 2025
9f7c4a9
fix for objects
lf-zhao Jan 23, 2025
2b2f0c6
update for env input
lf-zhao Jan 23, 2025
e1f39bd
update
lf-zhao Jan 23, 2025
5123b7c
update mock perceiver test, fixed vlm predicate stuf
lf-zhao Jan 23, 2025
d2ee262
major update to testing planning with perceiver on mock env
lf-zhao Jan 23, 2025
32fb086
fix timer
lf-zhao Jan 23, 2025
7be5698
fix saving non-VLM atoms, move predicate definition to the file
lf-zhao Jan 23, 2025
823fda8
moved predicate definitions
lf-zhao Jan 23, 2025
4ff16d9
update test for mock perceiver
lf-zhao Jan 23, 2025
1999b4a
update struct
lf-zhao Jan 23, 2025
7a4c5bb
minor fix
lf-zhao Jan 23, 2025
0a8f981
fix objects passing in; fix missing env names; fix task init; fix som…
lf-zhao Jan 23, 2025
604ca82
fix: add new env to factory
lf-zhao Jan 23, 2025
4944137
fix object info missing; add logging with rich table for vlm pred
lf-zhao Jan 23, 2025
81b4f2a
[in progress] test planning, need to feed right images
lf-zhao Jan 23, 2025
17ce627
update to process HEIC images from iphone
lf-zhao Jan 23, 2025
7117789
update test
lf-zhao Jan 23, 2025
c19262f
add a new "test" file for 2 cup pick place with images from iphone
lf-zhao Jan 23, 2025
c1d7a28
minor fixes and check
lf-zhao Jan 23, 2025
40ec0a7
minor bugs on action space and overrides
lf-zhao Jan 23, 2025
1596900
one mock env task for debug: for pick place 2 cups
lf-zhao Jan 23, 2025
b58697f
update openai
lf-zhao Jan 23, 2025
b66e7f8
minor
lf-zhao Jan 23, 2025
13f4632
add logging for environment side transitions
lf-zhao Jan 24, 2025
92c0c49
add information of objects into action info
lf-zhao Jan 24, 2025
8add8c0
add a flag for llm verbose
lf-zhao Jan 24, 2025
0a0606a
add test source images
lf-zhao Jan 24, 2025
d5c0172
add implementation note for mock spot env
lf-zhao Jan 24, 2025
1e64726
add implementation note for mock env creator
lf-zhao Jan 24, 2025
b7dce7f
add implementation note and docs for belief operators and predicates …
lf-zhao Jan 24, 2025
b2446a1
add more logging for llm planner
lf-zhao Jan 24, 2025
0436088
Merge remote-tracking branch 'origin/master' into lis-spot/202501-mer…
lf-zhao Jan 24, 2025
5062db6
deal with more merge+edits for VLM predicate + labeling related: my v…
lf-zhao Jan 24, 2025
209a552
update code missed in merging
lf-zhao Jan 24, 2025
0efb1ea
minor fixes
lf-zhao Jan 24, 2025
1513a2a
move creator functions to model interface, avoiding circular imports
lf-zhao Jan 24, 2025
6d354e0
add record for cost for openai models
lf-zhao Jan 24, 2025
8d5ace6
minor fix
lf-zhao Jan 24, 2025
4ec5f4b
fix vlm open loop planner
lf-zhao Jan 24, 2025
5301a44
fix wrongly init train task
lf-zhao Jan 24, 2025
026ae58
fix; no offline training for LLM learning based method - we only want…
lf-zhao Jan 24, 2025
3b6985e
update
lf-zhao Jan 24, 2025
8c435bb
fix cyclic imports for vlm creation
lf-zhao Jan 24, 2025
2e334bc
save number of actual online execution steps
lf-zhao Jan 25, 2025
ba8e3f8
update
lf-zhao Jan 25, 2025
77b3d3c
debug for random options approach
lf-zhao Jan 25, 2025
d774eed
add back
lf-zhao Jan 25, 2025
3192b50
update for saving results
lf-zhao Jan 25, 2025
1a6c59a
logging plans and replanned plans for debugging
lf-zhao Jan 25, 2025
92aa9b6
add some debug information for logging
lf-zhao Jan 25, 2025
675b772
[to check] add a vlm bilevel planning approach, not sure if correct; …
lf-zhao Jan 25, 2025
cbcaae5
[wip] add saving optimal plan for checking
lf-zhao Jan 25, 2025
3639f8e
update log
lf-zhao Jan 25, 2025
d5b3835
add note for foundation model planning approaches
lf-zhao Jan 25, 2025
6bc69ba
llm planner update v1
lf-zhao Jan 25, 2025
33afc8e
update llm planner to give oracle NSRTs for fair comparison
lf-zhao Jan 25, 2025
f8c9c50
also update vlm planner to include oracle NSRTS with pre/eff in prompt
lf-zhao Jan 25, 2025
7417818
update saving
lf-zhao Jan 25, 2025
c9ebb91
add new llm zero shot prompt
lf-zhao Jan 25, 2025
9b68d7e
update args for llm vlm planning
lf-zhao Jan 25, 2025
7cb15e3
major update: fix LLM open loop planner approach - based on latest VL…
lf-zhao Jan 27, 2025
2760952
update factory to include new task
lf-zhao Jan 27, 2025
15e37be
add new task for drawer cleaning
lf-zhao Jan 27, 2025
aa79a95
update name for pick place
lf-zhao Jan 27, 2025
997ec28
minor
lf-zhao Jan 27, 2025
8f27ac7
add a script for running all approaches on 2 cup pick place example
lf-zhao Jan 27, 2025
a7b25c7
add vlm and llm approaches for using expected atom check, otherwise t…
lf-zhao Jan 28, 2025
7e60d13
fix creation for 2 cup pick place
lf-zhao Jan 28, 2025
718b1bc
fix for cyclic import
lf-zhao Jan 28, 2025
af0fcf0
update script
lf-zhao Jan 28, 2025
b154742
major update: add CLI mock env creator, iterating over canonical pred…
lf-zhao Jan 28, 2025
3737e4b
major update: support iterating over canonical states; major update t…
lf-zhao Jan 28, 2025
e595ca4
add manual test for building drawer cleaning task - now we don't need…
lf-zhao Jan 28, 2025
6904b80
update for drawer cleaning task
lf-zhao Jan 28, 2025
2954984
upload images
lf-zhao Jan 28, 2025
d2d93ef
source images for drawer cleaning task
lf-zhao Jan 28, 2025
8d6b73a
add saving for objects in view - useful for PO tasks!
lf-zhao Jan 28, 2025
44d1287
minor fix
lf-zhao Jan 29, 2025
d442018
fix for random
lf-zhao Jan 29, 2025
e53cb1e
fix
lf-zhao Jan 29, 2025
6f51cab
minor
lf-zhao Jan 29, 2025
c234434
update/fix for llm open loop - now include state atoms in prompt
lf-zhao Jan 29, 2025
c80bd33
also the prompt
lf-zhao Jan 29, 2025
442a49d
major, WIP: add VLM perceiver and LLM text planner; add prompt
lf-zhao Jan 29, 2025
2d039a5
Merge branch 'lis-spot/update-fm-baselines-experiments' of https://gi…
lf-zhao Jan 29, 2025
cc62d86
not sure why but this was probably not committed
lf-zhao Jan 29, 2025
7472ad3
push fix when visible objects are none
lf-zhao Jan 29, 2025
506e24e
update - this is not vlm, deprecate
lf-zhao Jan 29, 2025
4cd046c
add photos for task of removing empty boxes with scale
lf-zhao Jan 29, 2025
485a4f2
add action history to prompts
lf-zhao Jan 29, 2025
9b2f02c
include action history
lf-zhao Jan 29, 2025
9b4c9e3
fix action history; for LLM approach first
lf-zhao Jan 29, 2025
6ed9f04
major, finish and fix: vlm planner with action history and observatio…
lf-zhao Jan 29, 2025
ff3ede0
update prompts
lf-zhao Jan 29, 2025
8be4cae
minor update for history
lf-zhao Jan 29, 2025
116bcbf
test for vlm perceiver
lf-zhao Jan 29, 2025
a68d3de
updates
aidan-curtis Jan 29, 2025
5f2ed0b
baseline implemented
aidan-curtis Jan 29, 2025
9ea1ae1
updates to captioning baseline
aidan-curtis Jan 29, 2025
c4a2271
fix perceiver for saving image+action history
lf-zhao Jan 29, 2025
f7672cd
fix in reset as well
lf-zhao Jan 29, 2025
d39dbd1
Merge branch 'lis-spot/update-fm-baselines-experiments' of https://gi…
lf-zhao Jan 29, 2025
68a14a1
searate action history maintain
lf-zhao Jan 29, 2025
2d416ee
some small updates
aidan-curtis Jan 29, 2025
e09808e
updates
aidan-curtis Jan 29, 2025
8c9b552
fix camera image history for vlm perceiver
lf-zhao Jan 29, 2025
72c9a30
we shouldn't need this anymore
lf-zhao Jan 29, 2025
abc5b4d
update vlm perceiver for image history
lf-zhao Jan 29, 2025
ade21c5
updates to experiments
aidan-curtis Jan 29, 2025
4c1231d
monir
lf-zhao Jan 30, 2025
b848f06
fix: now only use active predicates in active operators
lf-zhao Jan 30, 2025
51e2e6e
fix image load
lf-zhao Jan 30, 2025
d989d2f
major update: fix VLM belief predicates, update predicate exports, ge…
lf-zhao Jan 30, 2025
fbf9e11
Merge branch 'lis-spot/update-fm-baselines-experiments' of https://gi…
lf-zhao Jan 30, 2025
b699acd
mock modifications
aidan-curtis Jan 30, 2025
0808c6f
updates
aidan-curtis Jan 30, 2025
18c1498
minor update
lf-zhao Jan 30, 2025
67feab4
update hand view
lf-zhao Jan 30, 2025
d3afb5d
[major update spot env] update predicates and operators, update task
lf-zhao Jan 30, 2025
9360c4c
Merge branch 'lis-spot/update-fm-baselines-experiments' of https://gi…
lf-zhao Jan 30, 2025
6bccea5
updates
aidan-curtis Jan 30, 2025
7832600
fix bug
aidan-curtis Jan 30, 2025
99963ac
updates
aidan-curtis Jan 30, 2025
83f17cf
uncomment all
aidan-curtis Jan 30, 2025
a433ada
updates
aidan-curtis Jan 30, 2025
e232df8
updates
aidan-curtis Jan 31, 2025
a841411
save
lf-zhao Jan 31, 2025
c854a14
Merge branch 'lis-spot/update-fm-baselines-experiments' of https://gi…
lf-zhao Jan 31, 2025
e31356b
minor fixes/updates
lf-zhao Jan 31, 2025
45bf321
test
lf-zhao Jan 31, 2025
755a8cd
or atoms untested
aidan-curtis Jan 31, 2025
ea80d60
good transition graph
aidan-curtis Jan 31, 2025
e598250
move this to utils
lf-zhao Jan 31, 2025
b88e064
minor update
lf-zhao Jan 31, 2025
826938b
fix
lf-zhao Jan 31, 2025
bec0f34
updates
aidan-curtis Jan 31, 2025
11f117b
Merge remote-tracking branch 'origin/lis-spot/update-fm-baselines-exp…
lf-zhao Jan 31, 2025
dc74e15
Merge branch 'lis-spot/update-fm-baselines-experiments' of https://gi…
lf-zhao Jan 31, 2025
55cc006
update perception vlm prompting
lf-zhao Jan 31, 2025
051cd0a
push new 2 cup 1 empty removal task
lf-zhao Feb 1, 2025
878c722
add new class for the task and also test file
lf-zhao Feb 1, 2025
1d94d27
update nsrts for it
lf-zhao Feb 1, 2025
ae5bbf0
update max token
lf-zhao Feb 1, 2025
f44e61d
update nsrts
lf-zhao Feb 6, 2025
0624708
update some goals
lf-zhao Feb 6, 2025
603a08d
update metadata
lf-zhao Feb 6, 2025
a215ccd
cleanup
lf-zhao Feb 6, 2025
c448c70
[in progress] update detection; fixing molmo sam detection
lf-zhao Feb 6, 2025
56b33de
update spot env: VLM only evaluate active VLM predicates; update VLM …
lf-zhao Feb 6, 2025

Sorry, this diff is taking too long to generate.

It may be too large to display on GitHub.