Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Polars pivot_longer implementation #1355

Merged
merged 134 commits into from
Jun 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
134 commits
Select commit Hold shift + click to select a range
862c7dd
add make_clean_names function that can be applied to polars
Apr 19, 2024
01531cc
add examples for make_clean_names
Apr 20, 2024
0fb440e
changelog
Apr 20, 2024
5e944b2
limit import location for polars
Apr 20, 2024
501d9c6
limit import location for polars
Apr 20, 2024
9506832
fix polars in environment-dev.yml
Apr 20, 2024
1ae8edd
install polars in doctest
Apr 20, 2024
3b1829b
limit polars imports - user should have polars already installed
Apr 20, 2024
52fd80c
use subprocess.run
Apr 20, 2024
2dce78b
add subprocess.devnull to docstrings
Apr 20, 2024
37b3feb
add subprocess.devnull to docstrings
Apr 20, 2024
0953f2d
add subprocess.devnull to docstrings
Apr 20, 2024
d7c71b6
add subprocess.devnull to docstrings
Apr 20, 2024
40b8502
add os.devnull
Apr 20, 2024
4f11d09
add polars as requirement for docs
Apr 20, 2024
54b179c
add polars to tests requirements
Apr 20, 2024
25b39b9
delete irrelevant folder
Apr 20, 2024
a09f34b
changelog
Apr 20, 2024
1b375f8
create submodule for polars
Apr 21, 2024
799532f
fix doctests
Apr 21, 2024
dbce4b9
fix tests; add polars to documentation
Apr 21, 2024
1c642e6
fix tests; add polars to documentation
Apr 21, 2024
407d21b
import janitor.polars
Apr 21, 2024
aedfc65
control docs output for polars submodule
Apr 21, 2024
db9b486
exclude functions in docs rendering
Apr 21, 2024
6a91e67
exclude functions in docs rendering
Apr 21, 2024
7a88078
show_submodules=true
Apr 21, 2024
6d7885e
fix docstring rendering for polars
Apr 21, 2024
944fa02
Expression -> expression
Apr 21, 2024
b9aefaa
Merge dev into samukweku/polars_clean_names
ericmjl Apr 23, 2024
e9c370a
rename functions.py
Apr 23, 2024
ee66d2a
pivot_longer implemented for polars
Apr 29, 2024
959b082
changelog
Apr 30, 2024
3177503
keep changes related only to pivot_longer
Apr 30, 2024
ee899b2
pd -> pl
Apr 30, 2024
8ea9b71
pd -> pl
Apr 30, 2024
d12ae1a
df.pivot_longer -> df.janitor.pivot_longer
Apr 30, 2024
652f3e3
df.pivot_longer -> df.janitor.pivot_longer
Apr 30, 2024
9b9c1a9
pd -> pl
Apr 30, 2024
69c273f
pd -> pl
Apr 30, 2024
b3391e8
add >>> df
Apr 30, 2024
4ffaac5
add >>> df
Apr 30, 2024
1de57bb
keep changes related only to polars pivot_longer
Apr 30, 2024
8097bb4
code update
May 3, 2024
1bcd1b4
names_transform if there is not dot value
May 4, 2024
a492b85
add support for lazyframe
May 4, 2024
193fdc7
fix column order in docs
May 4, 2024
cd5baeb
fix column order in docs
May 4, 2024
1ba2650
fix column order in docs
May 4, 2024
908d71d
fix column order in docs
May 4, 2024
0cc6a93
separate namespaces for lazyframe and eager dataframe
May 5, 2024
0d41284
separate namespaces for lazyframe and eager dataframe
May 5, 2024
e8e6baf
improve perf if no dot_value and values_to is a str
May 5, 2024
2886ea1
fix docs
May 6, 2024
beec9e2
handle deprecation warning for pd.unique
May 6, 2024
a8d97ff
Merge dev into samukweku/polars_pivot_longer
ericmjl May 6, 2024
302a39b
Merge branch 'dev' into samukweku/polars_pivot_longer
samukweku May 10, 2024
7e4cc00
added pivot_longer_spec
May 12, 2024
a2df26f
update docs
May 12, 2024
3635639
update docs
May 12, 2024
d799ef5
update docs
May 12, 2024
198886d
update docs
May 12, 2024
c8a89eb
update docs
May 12, 2024
f733cfd
update docs
May 12, 2024
765837f
update docs
May 12, 2024
0fbe05a
update docs
May 12, 2024
b51f803
reduce repetitiveness in function calls
May 12, 2024
4798b09
update docs
May 12, 2024
30a290b
update docs
May 12, 2024
0755e91
update docs
May 13, 2024
2eeba86
update docs
May 13, 2024
1efdc96
refactor code - create spec then unpivot
May 13, 2024
41f450d
refactor code - create spec then unpivot
May 13, 2024
e03c11a
update docs
May 13, 2024
ba6df7d
update docs
May 13, 2024
0aac3cb
update docs
May 13, 2024
7f689a7
update docs
May 13, 2024
56bca98
add check for spec dataframe
May 17, 2024
65ee3b8
fix doc fail
May 17, 2024
7a69b54
fix doc fail
May 17, 2024
3365f01
exclude non polars code
May 17, 2024
04a8444
Merge dev into samukweku/polars_pivot_longer
ericmjl May 19, 2024
e20513e
Merge dev into samukweku/polars_pivot_longer
ericmjl May 23, 2024
c04f017
change column selection
May 25, 2024
2ac6ecd
Merge dev into samukweku/polars_pivot_longer
ericmjl May 27, 2024
2abc798
call join only if necessary
May 29, 2024
32bf56c
rechunk
May 29, 2024
0198e78
names_transform should be a pl.Expr
May 29, 2024
97c77fc
adjust tests
May 29, 2024
a5942d8
__future__
May 29, 2024
cd03949
changelog
May 29, 2024
5db1808
compute only when necessary
May 29, 2024
67a7724
improve performance
May 31, 2024
69a17ee
fix docs
May 31, 2024
d8cdc66
fix docs
May 31, 2024
face78a
non-equi join rewrite numba (#1341)
samukweku Jun 2, 2024
d47073f
Implement clean_names for polars (#1351)
samukweku Jun 2, 2024
1765cf8
add make_clean_names function that can be applied to polars
Apr 19, 2024
86aa80d
add examples for make_clean_names
Apr 20, 2024
70fa448
changelog
Apr 20, 2024
1114c4f
limit import location for polars
Apr 20, 2024
b3f1d36
limit polars imports - user should have polars already installed
Apr 20, 2024
12b7dce
add subprocess.devnull to docstrings
Apr 20, 2024
cf1aee7
add polars to tests requirements
Apr 20, 2024
219a8ed
changelog
Apr 20, 2024
5d54b9a
create submodule for polars
Apr 21, 2024
30d00b5
fix doctests
Apr 21, 2024
e2b6d40
fix tests; add polars to documentation
Apr 21, 2024
d16ad9c
control docs output for polars submodule
Apr 21, 2024
5bc6216
fix docstring rendering for polars
Apr 21, 2024
bc15280
Expression -> expression
Apr 21, 2024
8c3ec6b
rename functions.py
Apr 23, 2024
a790b4c
pivot_longer implemented for polars
Apr 29, 2024
9627586
changelog
Apr 30, 2024
272e16f
keep changes related only to pivot_longer
Apr 30, 2024
0611f74
add support for lazyframe
May 4, 2024
9f49999
separate namespaces for lazyframe and eager dataframe
May 5, 2024
b992cc4
added pivot_longer_spec
May 12, 2024
785b9be
update docs
May 12, 2024
b0bc268
reduce repetitiveness in function calls
May 12, 2024
532f661
add check for spec dataframe
May 17, 2024
db2afb2
names_transform should be a pl.Expr
May 29, 2024
d3b9e87
__future__
May 29, 2024
c942b60
changelog
May 29, 2024
bcb95f3
updates
Jun 3, 2024
49dafe7
fix conflicts
Jun 3, 2024
6ed2095
non-equi join rewrite numba (#1341)
samukweku Jun 2, 2024
e2ab8df
Add polars support for `janitor.io.xlsx_table` (#1357)
samukweku Jun 3, 2024
bc132a7
Add support for polars in `xlsx_cells` (#1358)
samukweku Jun 3, 2024
a0e5571
Merge branch 'dev' into samukweku/polars_pivot_longer
samukweku Jun 3, 2024
6140c23
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 3, 2024
16bf8fd
fix conflicts
Jun 3, 2024
22337b3
fix changelog
Jun 3, 2024
b5ceef3
Merge dev into samukweku/polars_pivot_longer
ericmjl Jun 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
555 changes: 278 additions & 277 deletions CHANGELOG.md

Large diffs are not rendered by default.

41 changes: 19 additions & 22 deletions janitor/functions/clean_names.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
from __future__ import annotations

import unicodedata
from typing import Optional, Union

import pandas as pd
import pandas_flavor as pf
Expand All @@ -18,9 +17,9 @@
@deprecated_alias(preserve_original_columns="preserve_original_labels")
def clean_names(
df: pd.DataFrame,
axis: Union[str, None] = "columns",
column_names: Union[str, list] = None,
strip_underscores: Optional[Union[str, bool]] = None,
axis: str = "columns",
column_names: str | list = None,
strip_underscores: str | bool = None,
case_type: str = "lower",
remove_special: bool = False,
strip_accents: bool = True,
Expand Down Expand Up @@ -170,14 +169,14 @@ def clean_names(


def _clean_names(
obj: Union[pd.Index, pd.Series],
strip_underscores: Optional[Union[str, bool]] = None,
case_type: str = "lower",
remove_special: bool = False,
strip_accents: bool = False,
enforce_string: bool = False,
truncate_limit: int = None,
) -> Union[pd.Index, pd.Series]:
obj: pd.Index | pd.Series,
strip_underscores: str | bool,
case_type: str,
remove_special: bool,
strip_accents: bool,
enforce_string: bool,
truncate_limit: int,
) -> pd.Index | pd.Series:
"""
Generic function to clean labels in a pandas object.
"""
Expand All @@ -202,9 +201,9 @@ def _clean_names(


def _change_case(
obj: Union[pd.Index, pd.Series],
obj: pd.Index | pd.Series,
case_type: str,
) -> Union[pd.Index, pd.Series]:
) -> pd.Index | pd.Series:
"""Change case of labels in obj."""
case_types = {"preserve", "upper", "lower", "snake"}
case_type = case_type.lower()
Expand All @@ -226,9 +225,7 @@ def _change_case(
)


def _normalize_1(
obj: Union[pd.Index, pd.Series]
) -> Union[pd.Index, pd.Series]:
def _normalize_1(obj: pd.Index | pd.Series) -> pd.Index | pd.Series:
"""Perform normalization of labels in obj."""
FIXES = [(r"[ /:,?()\.-]", "_"), (r"['’]", ""), (r"[\xa0]", "_")]
for search, replace in FIXES:
Expand All @@ -238,8 +235,8 @@ def _normalize_1(


def _strip_accents(
obj: Union[pd.Index, pd.Series],
) -> Union[pd.Index, pd.Series]:
obj: pd.Index | pd.Series,
) -> pd.Index | pd.Series:
"""Remove accents from a label.

Inspired from [StackOverflow][so].
Expand All @@ -258,9 +255,9 @@ def _strip_accents(


def _strip_underscores_func(
obj: Union[pd.Index, pd.Series],
strip_underscores: Union[str, bool] = None,
) -> Union[pd.Index, pd.Series]:
obj: pd.Index | pd.Series,
strip_underscores: str | bool = None,
) -> pd.Index | pd.Series:
"""Strip underscores."""
underscore_options = {None, "left", "right", "both", "l", "r", True}
if strip_underscores not in underscore_options:
Expand Down
14 changes: 2 additions & 12 deletions janitor/functions/pivot.py
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ def pivot_longer(
6 setosa Petal.Width 0.2
7 virginica Petal.Width 1.8

Split the column labels into parts:
Split the column labels into individual columns:
>>> df.pivot_longer(
... index = 'Species',
... names_to = ('part', 'dimension'),
Expand Down Expand Up @@ -167,7 +167,7 @@ def pivot_longer(
value int64
dtype: object

Use multiple `.value` to reshape dataframe:
Use multiple `.value` to reshape the dataframe:
>>> df = pd.DataFrame(
... [
... {
Expand Down Expand Up @@ -265,16 +265,6 @@ def pivot_longer(
... "Gin": [16, 200, 34],
... "Vodka": [20, 33, 18],
... },
... columns=[
... "City",
... "State",
... "Name",
... "Mango",
... "Orange",
... "Watermelon",
... "Gin",
... "Vodka",
... ],
... )
>>> df
City State Name Mango Orange Watermelon Gin Vodka
Expand Down
Loading
Loading