Skip to content

Commit

Permalink
add instruction for using alluxiocommon
Browse files Browse the repository at this point in the history
  • Loading branch information
lucyge2022 committed Jun 4, 2024
1 parent 1400b91 commit a166df5
Show file tree
Hide file tree
Showing 2 changed files with 20 additions and 2 deletions.
18 changes: 18 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,24 @@ ds2.count()
# End of Python example
```

#### Enable alluxiocommon enhancement module

alluxiocommon package is a native enhancement module for alluxiofs based on PyO3 rust bindings.
Currently it enhances big reads (multi-page reads from alluxio) by issuing multi-threaded requests to alluxio.

to enable it, first install alluxiocommon package:
```
pip install alluxiocommon
```
and when start the Alluxio fsspec instance, add an additional option flag:
```
alluxio_options = {"alluxio.common.extension.enable" : "True"}
alluxio_fs = fsspec.filesystem(
"alluxiofs", etcd_hosts="localhost", target_protocol="s3",
options=alluxio_options
)
```

### Running examples with Pyarrow

```
Expand Down
4 changes: 2 additions & 2 deletions alluxiofs/client/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -205,8 +205,8 @@ def __init__(
ALLUXIO_COMMON_EXTENSION_ENABLE in options
and options[ALLUXIO_COMMON_EXTENSION_ENABLE].lower() == "true"
):
print("Using alluxiocommon extension..")
logger.debug("alluxiocommon extension enabled.")
print("alluxiocommon extension enabled.")
logger.info("alluxiocommon extension enabled.")
ondemand_pool_disabled = (
ALLUXIO_COMMON_ONDEMANDPOOL_DISABLE in options
and options[ALLUXIO_COMMON_ONDEMANDPOOL_DISABLE].lower()
Expand Down

0 comments on commit a166df5

Please sign in to comment.