New cid summary api and simplified cid info #730

asutula · 2020-12-16T00:27:22Z

The existing Data.CidInfo API is just too much data, different types of data, in a list, and it gets too big to consume efficiently. This creates a "summary" api that will return the most basic information about all the cids you have stored in powergate, then you can call Data.CidInfo with any one of those cids to still get detailed and varied information about a single cid.

Ended up merging a couple other PRs into this one. For completeness, the are:

Proposed CLI package structure #728 which creates a package structure for cli code
List storage jobs query API #748 which adds a new API for querying storage jobs

asutula · 2020-12-16T00:29:07Z

api/server/user/data.go

-// CidInfo returns information about cids managed by the FFS instance.
-func (s *Service) CidInfo(ctx context.Context, req *userPb.CidInfoRequest) (*userPb.CidInfoResponse, error) {
+// CidSummary gives a summary of the storage and jobs state of the specified cid.
+func (s *Service) CidSummary(ctx context.Context, req *userPb.CidSummaryRequest) (*userPb.CidSummaryResponse, error) {


This method is surprisingly slow, and I need to figure out why. When I call it against our powergate instance with about 700 cids stored, it takes 10s of seconds to return the data. The amount of data is small, so it must be some processing in here that is taking a while.

I'm not sure when you test that, but today there were some heavy processes in the Atlas db.
Maybe you got unlucky? Mentioning it since was a special day today.
Dismiss if isn't the case... might be a real prob.

Tested before the downtime. Will have to dig in and see what's up.

asutula · 2020-12-16T00:31:00Z

proto/powergate/user/v1/user.proto

  repeated string cids = 1;
 }

+message CidSummaryResponse {
+  repeated CidSummary cid_summary = 1;
+}


The new request/response type.

asutula · 2020-12-16T00:31:28Z

proto/powergate/user/v1/user.proto

 message CidInfoResponse {
-  repeated CidInfo cid_infos = 1;
+  CidInfo cid_info = 1;


CidInfo request/response is now just for a single cid.

asutula · 2020-12-16T00:31:44Z

proto/powergate/user/v1/user.proto

+  bool stored = 2;
+  repeated string queued_jobs = 3;
+  string executing_job = 4;
+}


Summary returns some simple inflation about a cid.

jsign

LGTM

jsign · 2020-12-16T03:28:51Z

api/server/user/data.go

-// CidInfo returns information about cids managed by the FFS instance.
-func (s *Service) CidInfo(ctx context.Context, req *userPb.CidInfoRequest) (*userPb.CidInfoResponse, error) {
+// CidSummary gives a summary of the storage and jobs state of the specified cid.
+func (s *Service) CidSummary(ctx context.Context, req *userPb.CidSummaryRequest) (*userPb.CidSummaryResponse, error) {


I'm not sure when you test that, but today there were some heavy processes in the Atlas db.
Maybe you got unlucky? Mentioning it since was a special day today.
Dismiss if isn't the case... might be a real prob.

api/server/user/data.go

api/server/user/util.go

cmd/pow/cmd/data_summary.go

asutula · 2021-01-16T03:57:49Z

Ok all rebased against develop here and changed the target of the PR to develop.

Signed-off-by: Aaron Sutula <[email protected]>

* move data cli commands to packages, refactor common code Signed-off-by: Aaron Sutula <[email protected]> * add data summary command that was forgotten Signed-off-by: Aaron Sutula <[email protected]> * apply cli org to the rest Signed-off-by: Aaron Sutula <[email protected]> * update docs Signed-off-by: Aaron Sutula <[email protected]> * Data summary TUI (#729) * basic tui for pow data summary Signed-off-by: Aaron Sutula <[email protected]> * pr feedback Signed-off-by: Aaron Sutula <[email protected]>

Signed-off-by: Aaron Sutula <[email protected]>

* first cut a a list implementation for storage jobs Signed-off-by: Aaron Sutula <[email protected]> * bubble up list storage jobs api through scheduler, ffs, and apis Signed-off-by: Aaron Sutula <[email protected]> * client for list, better name for next page token Signed-off-by: Aaron Sutula <[email protected]> * list cli Signed-off-by: Aaron Sutula <[email protected]> * fix selector filter Signed-off-by: Aaron Sutula <[email protected]> * remove old/redundant code Signed-off-by: Aaron Sutula <[email protected]> * better alias Signed-off-by: Aaron Sutula <[email protected]> * pr feedback Signed-off-by: Aaron Sutula <[email protected]> * remove counts Signed-off-by: Aaron Sutula <[email protected]> * migration Signed-off-by: Aaron Sutula <[email protected]> * good tests for sjstore.List Signed-off-by: Aaron Sutula <[email protected]>

Signed-off-by: Aaron Sutula <[email protected]>

Signed-off-by: Ignacio Hagopian <[email protected]>

Signed-off-by: Aaron Sutula <[email protected]>

* new summary api and simplified cid info Signed-off-by: Aaron Sutula <[email protected]> * pr feedback Signed-off-by: Aaron Sutula <[email protected]> * Proposed CLI package structure (#728) * move data cli commands to packages, refactor common code Signed-off-by: Aaron Sutula <[email protected]> * add data summary command that was forgotten Signed-off-by: Aaron Sutula <[email protected]> * apply cli org to the rest Signed-off-by: Aaron Sutula <[email protected]> * update docs Signed-off-by: Aaron Sutula <[email protected]> * Data summary TUI (#729) * basic tui for pow data summary Signed-off-by: Aaron Sutula <[email protected]> * pr feedback Signed-off-by: Aaron Sutula <[email protected]> * update docs Signed-off-by: Aaron Sutula <[email protected]> * List storage jobs query API (#748) * first cut a a list implementation for storage jobs Signed-off-by: Aaron Sutula <[email protected]> * bubble up list storage jobs api through scheduler, ffs, and apis Signed-off-by: Aaron Sutula <[email protected]> * client for list, better name for next page token Signed-off-by: Aaron Sutula <[email protected]> * list cli Signed-off-by: Aaron Sutula <[email protected]> * fix selector filter Signed-off-by: Aaron Sutula <[email protected]> * remove old/redundant code Signed-off-by: Aaron Sutula <[email protected]> * better alias Signed-off-by: Aaron Sutula <[email protected]> * pr feedback Signed-off-by: Aaron Sutula <[email protected]> * remove counts Signed-off-by: Aaron Sutula <[email protected]> * migration Signed-off-by: Aaron Sutula <[email protected]> * good tests for sjstore.List Signed-off-by: Aaron Sutula <[email protected]> * update pow docs Signed-off-by: Aaron Sutula <[email protected]> * fix compilation problems Signed-off-by: Aaron Sutula <[email protected]> * update docs Signed-off-by: Aaron Sutula <[email protected]> * comment exported func Signed-off-by: Aaron Sutula <[email protected]> * fix docs Signed-off-by: Ignacio Hagopian <[email protected]> * add missin cmd to cli, better name for admin jobs cli Signed-off-by: Aaron Sutula <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]>

asutula self-assigned this Dec 16, 2020

asutula requested a review from jsign December 16, 2020 00:27

asutula commented Dec 16, 2020

View reviewed changes

jsign approved these changes Dec 16, 2020

View reviewed changes

asutula force-pushed the asutula/cid-info-summary-split branch from 3965f92 to 067c664 Compare January 16, 2021 03:56

asutula changed the base branch from master to develop January 16, 2021 03:57

asutula added 9 commits January 16, 2021 14:00

new summary api and simplified cid info

cd8de19

Signed-off-by: Aaron Sutula <[email protected]>

pr feedback

113c5b3

Signed-off-by: Aaron Sutula <[email protected]>

update docs

e88f105

Signed-off-by: Aaron Sutula <[email protected]>

update pow docs

416931c

Signed-off-by: Aaron Sutula <[email protected]>

fix compilation problems

4a5b145

Signed-off-by: Aaron Sutula <[email protected]>

update docs

99513bd

Signed-off-by: Aaron Sutula <[email protected]>

comment exported func

86358b5

Signed-off-by: Aaron Sutula <[email protected]>

asutula force-pushed the asutula/cid-info-summary-split branch from 1b973e7 to 86358b5 Compare January 16, 2021 20:05

fix docs

b0277b4

Signed-off-by: Ignacio Hagopian <[email protected]>

asutula added feature major labels Jan 18, 2021

add missin cmd to cli, better name for admin jobs cli

66d73bb

Signed-off-by: Aaron Sutula <[email protected]>

asutula merged commit 71ed460 into develop Jan 18, 2021

asutula deleted the asutula/cid-info-summary-split branch January 18, 2021 19:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New cid summary api and simplified cid info #730

New cid summary api and simplified cid info #730

asutula commented Dec 16, 2020 •

edited

Loading

asutula Dec 16, 2020

jsign Dec 16, 2020

asutula Dec 16, 2020

asutula Dec 16, 2020

asutula Dec 16, 2020

asutula Dec 16, 2020

jsign left a comment

jsign Dec 16, 2020

asutula commented Jan 16, 2021

New cid summary api and simplified cid info #730

New cid summary api and simplified cid info #730

Conversation

asutula commented Dec 16, 2020 • edited Loading

asutula Dec 16, 2020

Choose a reason for hiding this comment

jsign Dec 16, 2020

Choose a reason for hiding this comment

asutula Dec 16, 2020

Choose a reason for hiding this comment

asutula Dec 16, 2020

Choose a reason for hiding this comment

asutula Dec 16, 2020

Choose a reason for hiding this comment

asutula Dec 16, 2020

Choose a reason for hiding this comment

jsign left a comment

Choose a reason for hiding this comment

jsign Dec 16, 2020

Choose a reason for hiding this comment

asutula commented Jan 16, 2021

asutula commented Dec 16, 2020 •

edited

Loading