-
-
Notifications
You must be signed in to change notification settings - Fork 372
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
getDocsBatch for bulk processing #2371
base: master
Are you sure you want to change the base?
getDocsBatch for bulk processing #2371
Conversation
ab947a3
to
38463da
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
1308553
to
1d65815
Compare
This comment has been minimized.
This comment has been minimized.
1d65815
to
4353c0d
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
`getDocsBatch` cuurently (& before) used only for single name retrieval function. Use of it is in `Documentation` module `getDocumentationTryGhc` where it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with 1 entry & unsafely "lookups" doc in it. This work is to supply the proper single name retrieval-optimized version to stop that `getDocsBatch` there. & further ideally `getDocumentationTryGhc` uses single-retrieval & `getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along the lines of: haskell#2371
`getDocsBatch` cuurently (& before) used only for single name retrieval function. Use of it is in `Documentation` module `getDocumentationTryGhc` where it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with 1 entry & unsafely "lookups" doc in it. This work is to supply the proper single name retrieval-optimized version to stop that `getDocsBatch` there. & further ideally `getDocumentationTryGhc` uses single-retrieval & `getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along the lines of: haskell#2371
`getDocsBatch` cuurently (& before) used only for single name retrieval function. Use of it is in `Documentation` module `getDocumentationTryGhc` where it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with 1 entry & unsafely "lookups" doc in it. This work is to supply the proper single name retrieval-optimized version to stop that `getDocsBatch` there. & further ideally `getDocumentationTryGhc` uses single-retrieval & `getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along the lines of: haskell#2371
`getDocsBatch` cuurently (& before) used only for single name retrieval function. Use of it is in `Documentation` module `getDocumentationTryGhc` where it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with 1 entry & unsafely "lookups" doc in it. This work is to supply the proper single name retrieval-optimized version to stop that `getDocsBatch` there. & further ideally `getDocumentationTryGhc` uses single-retrieval & `getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along the lines of: haskell#2371
`getDocsBatch` cuurently (& before) used only for single name retrieval function. Use of it is in `Documentation` module `getDocumentationTryGhc` where it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with 1 entry & unsafely "lookups" doc in it. This work is to supply the proper single name retrieval-optimized version to stop that `getDocsBatch` there. & further ideally `getDocumentationTryGhc` uses single-retrieval & `getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along the lines of: haskell#2371
where | ||
throwErrors = liftIO . throwIO . mkSrcErr | ||
compiled n = | ||
((_warns,errs), res) <- initTc hsc_env HsSrcFile False _mod fakeSpan $ Map.fromList <$> traverse undefined undefined |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From https://gitlab.haskell.org/ghc/ghc/-/issues/20831#note_398227:
I looked at the HLS code and you should probably rather use:
initIfaceLoad hsc_env (loadSysInterface "docs" module)
to load the interface rather than callinginitTc
.
Then that will cache the loaded interface so you don't load it multiple times from disk.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(code currently is created so that it loads & opens interfaces once, but if I understand it right - the mentioned method caches interfaces between function runs).
`getDocsBatch` cuurently (& before) used only for single name retrieval function. Use of it is in `Documentation` module `getDocumentationTryGhc` where it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with 1 entry & unsafely "lookups" doc in it. This work is to supply the proper single name retrieval-optimized version to stop that `getDocsBatch` there. & further ideally `getDocumentationTryGhc` uses single-retrieval & `getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along the lines of: haskell#2371
4353c0d
to
4413a0f
Compare
`getDocsBatch` cuurently (& before) used only for single name retrieval function. Use of it is in `Documentation` module `getDocumentationTryGhc` where it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with 1 entry & unsafely "lookups" doc in it. This work is to supply the proper single name retrieval-optimized version to stop that `getDocsBatch` there. & further ideally `getDocumentationTryGhc` uses single-retrieval & `getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along the lines of: haskell#2371
`getDocsBatch` cuurently (& before) used only for single name retrieval function. Use of it is in `Documentation` module `getDocumentationTryGhc` where it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with 1 entry & unsafely "lookups" doc in it. This work is to supply the proper single name retrieval-optimized version to stop that `getDocsBatch` there. & further ideally `getDocumentationTryGhc` uses single-retrieval & `getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along the lines of: haskell#2371
`getDocsBatch` cuurently (& before) used only for single name retrieval function. Use of it is in `Documentation` module `getDocumentationTryGhc` where it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with 1 entry & unsafely "lookups" doc in it. This work is to supply the proper single name retrieval-optimized version to stop that `getDocsBatch` there. & further ideally `getDocumentationTryGhc` uses single-retrieval & `getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along the lines of: haskell#2371
`getDocsBatch` cuurently (& before) used only for single name retrieval function. Use of it is in `Documentation` module `getDocumentationTryGhc` where it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with 1 entry & unsafely "lookups" doc in it. This work is to supply the proper single name retrieval-optimized version to stop that `getDocsBatch` there. & further ideally `getDocumentationTryGhc` uses single-retrieval & `getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along the lines of: haskell#2371
`getDocsBatch` cuurently (& before) used only for single name retrieval function. Use of it is in `Documentation` module `getDocumentationTryGhc` where it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with 1 entry & unsafely "lookups" doc in it. This work is to supply the proper single name retrieval-optimized version to stop that `getDocsBatch` there. & further ideally `getDocumentationTryGhc` uses single-retrieval & `getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along the lines of: haskell#2371
`getDocsBatch` cuurently (& before) used only for single name retrieval function. Use of it is in `Documentation` module `getDocumentationTryGhc` where it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with 1 entry & unsafely "lookups" doc in it. This work is to supply the proper single name retrieval-optimized version to stop that `getDocsBatch` there. & further ideally `getDocumentationTryGhc` uses single-retrieval & `getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along the lines of: haskell#2371
`getDocsBatch` cuurently (& before) used only for single name retrieval function. Use of it is in `Documentation` module `getDocumentationTryGhc` where it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with 1 entry & unsafely "lookups" doc in it. This work is to supply the proper single name retrieval-optimized version to stop that `getDocsBatch` there. & further ideally `getDocumentationTryGhc` uses single-retrieval & `getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along the lines of: haskell#2371
`getDocsBatch` cuurently (& before) used only for single name retrieval function. Use of it is in `Documentation` module `getDocumentationTryGhc` where it wraps arg into singleton & gives to `getDocsBatch` & then recieves a Map with 1 entry & unsafely "lookups" doc in it. This work is to supply the proper single name retrieval-optimized version to stop that `getDocsBatch` there. & further ideally `getDocumentationTryGhc` uses single-retrieval & `getDocumentationsTryGhc` uses a batch mode & batch mode gets optimized along the lines of: haskell#2371
A branch from: #2349
Holds a relevant
getDocsBatch
code to make bulk requests.TODO:
error
-> exception handling codeMap
s be Strictfirst
s can be flattened)Besides that - can not find the better-optimized version, I hope that
swap ... swap
& tuple passing et al. optimizes/fuses away by GHC (which should).Also - after functor laws are applied - count the number of traverses through the
[]
& weigh it for a number of batch requests.Currently in
master
primary use ofgetDocsBatch
is:getDocumentationTryGhc
that runs singleton element doc on:getDocumentationsTryGhc
that runsgetDocsBatch
on singleton.getDocsBatch
internally has a modified (with removed interactive check) copy of GHCsgetDocs
. If that internal 1-elem non-interactive function is needed - it probably needs to begetDocsNonInteractive
& so be module scope.And then
getDocsBatch
is designed for what it is intended - a bulk[Name]
requests.Then I went further: if
getDocsBatch
would be used really in batch mode - why is to dogetDocsBatch
ongetDocsNonInteractive
- that searches & loads module interface on each element:So, currently I do in
getDocsBatch
:Load
[Name]
.For each
Name
get module it says being part of.Then group all
Names
by accordingModule
:[(Module, [Name])]
.Then load module interface once & process through it in bulk [Name]s that are in that module.
For current way function is used (singleton retrieval) that would introduce
+ C
steps for retrieval (that may be specialized-away by the compiler).If HLS starts bulking retrieval from GHC interfaces based on [Name] - that would be optimal way.