Generate raw data #54
Conversation
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>
Co-authored-by: Helber Belmiro <[email protected]>
| transactions_before["transaction_timestamp"] < transactions_before["created_x"] | ||
| ] | ||
| transactions_before["days_between_transactions"] = ( | ||
| transactions_before["transaction_timestamp"] - transactions_before["created_x"] |
There was a problem hiding this comment.
| transactions_before["transaction_timestamp"] - transactions_before["created_x"] | |
| abs(transactions_before["transaction_timestamp"] - transactions_before["created_x"]) |
There was a problem hiding this comment.
This currently generates negative values, causing the "days_since_last_transaction" and "days_since_first_transaction" to get mixed up.
| df = pd.concat([train, test, valid], axis=0).reset_index(drop=True) | ||
|
|
||
| df["user_id"] = [f"user_{i}" for i in range(df.shape[0])] | ||
| df["transaction_id"] = [f"txn_{i}" for i in range(df.shape[0])] |
There was a problem hiding this comment.
This transaction_id seems to not be used again later and is not part of the output parquets.
Should it be added as a column in transactions_list in generate_random_transactions?
|
|
||
| print("generating transaction level data...") | ||
| user_purchase_history = generate_random_transactions( | ||
| users_df=df[df["repeat_retailer"] == 1].reset_index(drop=True), |
There was a problem hiding this comment.
Im curious, why we only are using data from when they have purchased multiple times from the same retailer?
| days_since_first_transaction=("days_between_transactions", "max"), | ||
| ) | ||
| .reset_index() | ||
| .fillna(0) |
There was a problem hiding this comment.
| .fillna(0) |
There was a problem hiding this comment.
This still leaves NaNs in the dataframe, applying fillna(0) on the final_df fixes this issue
| .reset_index(drop=True) | ||
| .drop("created_x", axis=1) | ||
| ) | ||
|
|
There was a problem hiding this comment.
| final_df = final_df.fillna(0) |
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main() |
There was a problem hiding this comment.
Great PR!
Just so I understand this correctly - do you think this should come in as its own data prep section, or that the parquet files this code produces should exist ahead of time and just be used during training/inference?
Resolves #53