Skip to content

Commit

Permalink
First code cleaning
Browse files Browse the repository at this point in the history
  • Loading branch information
@viniciusandrade01
viniciusandrade01 committed Sep 15, 2023
1 parent 99cbd5b commit b802fc3
Show file tree
Hide file tree
Showing 11 changed files with 69 additions and 27 deletions.
16 changes: 5 additions & 11 deletions Log_20230915.log
View file
Original file line number Diff line number Diff line change
@@ -1,11 +1,5 @@
2023-09-15 10:27:31 | INFO | SALVANDO ARQUIVO XLSX, DO ZIP, REFERENTE AOS RELAT�RIOS MENSAIS DED�VIDA.
2023-09-15 10:28:59 | INFO | DataFrame salvo como R_Mensal_Divida_20230701 em formato CSV.
2023-09-15 10:28:59 | INFO | DOCUMENTO CRIADO COM SUCESSO!
2023-09-15 10:31:33 | INFO | SALVANDO ARQUIVO XLSX, DO ZIP, REFERENTE AOS RELAT�RIOS MENSAIS DED�VIDA.
2023-09-15 10:41:40 | INFO | SALVANDO ARQUIVO XLSX, DO ZIP, REFERENTE AOS RELAT�RIOS MENSAIS DED�VIDA.
2023-09-15 10:44:55 | INFO | SALVANDO ARQUIVO XLSX, DO ZIP, REFERENTE AOS RELAT�RIOS MENSAIS DED�VIDA.
2023-09-15 10:55:06 | INFO | DataFrame salvo como R_Mensal_Divida_20230301 em formato CSV.
2023-09-15 10:55:33 | INFO | DOCUMENTO CRIADO COM SUCESSO!
2023-09-15 10:57:06 | INFO | SALVANDO ARQUIVO XLSX, DO ZIP, REFERENTE AOS RELAT�RIOS MENSAIS DED�VIDA.
2023-09-15 11:00:03 | INFO | DataFrame salvo como R_Mensal_Divida_20220701 em formato CSV.
2023-09-15 11:00:03 | INFO | DOCUMENTO CRIADO COM SUCESSO!
2023-09-15 17:07:22 | INFO | OCORREU UM ERRO AO TENTAR SUBIR OBJETO AO BUCKET: cannot access local variable 'e' where it is not associated with a value
2023-09-15 17:07:22 | INFO | SALVANDO ARQUIVO XLSX, DO ZIP, REFERENTE AOS RELAT�RIOS MENSAIS DED�VIDA.
2023-09-15 17:31:13 | INFO | DataFrame salvo como R_Mensal_Divida_20221201 em formato CSV.
2023-09-15 17:31:13 | INFO | DOCUMENTO CRIADO COM SUCESSO!
2023-09-15 17:31:14 | INFO | OCORREU UM ERRO AO TENTAR SUBIR OBJETO AO BUCKET: cannot access local variable 'e' where it is not associated with a value
2 changes: 1 addition & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
- O Relatório Mensal da Dívida Pública Federal (RMD) apresenta informações sobre emissões, resgates, estoque, composição, programa Tesouro Direto, perfil de vencimentos e custo médio, dentre outras, da Dívida Pública Federal - DPF, nela incluídas as dívidas interna e externa de responsabilidade do Tesouro Nacional em mercado. O documento reúne ainda informações sobre a reserva de liquidez e as garantias honradas pela União.
Fonte: Tesouro Nacional Transparente

# - OBS.: ESSE PROJETO NÃO É UMA RECOMENDAÇÃO DE INVESTIMENTO, MAS UMA INICIATIVA DE FORNECER ALGUMAS INFORMAÇÕES E DADOS ÚTEIS - E PÚBLICOS - AOS INVESTIDORES.
- OBS.: ESSE PROJETO NÃO É UMA RECOMENDAÇÃO DE INVESTIMENTO, MAS UMA INICIATIVA DE FORNECER ALGUMAS INFORMAÇÕES E DADOS ÚTEIS - E PÚBLICOS - AOS INVESTIDORES.

---------------------------------------------------------------------------------------------
# PROJETO
Expand Down
Binary file added RMD_20221201/Anexo_RMD_Julho_23.xlsx
View file
Binary file not shown.
52 changes: 52 additions & 0 deletions RMD_20221201/R_Mensal_Divida_20221201.csv
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
SERIE TITULO DATA_VENCIMENTO DATA_REF DATA_CAPTURA FINANCEIRO (R$ BI) QUANTIDADE (MIL) COD_REF
BRSTNCLTN7D3 'LTN' 2023-01-01 2023-07-01 2022-12-01 146.33795656 146337.95656 'BRSTNCLTN7D3'
BRSTNCLTN7X1 'LTN' 2023-04-01 2023-07-01 2022-12-01 25.4324635415615 26237.273 'BRSTNCLTN7X1'
BRSTNCLTN7Q5 'LTN' 2023-07-01 2023-07-01 2022-12-01 191.08811237619298 198321.204 'BRSTNCLTN7Q5'
BRSTNCLTN814 'LTN' 2023-10-01 2023-07-01 2022-12-01 9.02414681778142 9918.356 'BRSTNCLTN814'
BRSTNCLTN7S1 'LTN' 2024-01-01 2023-07-01 2022-12-01 266.0246868719 280908.663 'BRSTNCLTN7S1'
BRSTNCLTN7Y9 'LTN' 2024-04-01 2023-07-01 2022-12-01 65.3437170110938 75589.425 'BRSTNCLTN7Y9'
BRSTNCLTN7W3 'LTN' 2024-07-01 2023-07-01 2022-12-01 83.09567534637286 92859.07435 'BRSTNCLTN7W3'
BRSTNCLTN822 'LTN' 2024-10-01 2023-07-01 2022-12-01 31.8970824678511 39179.516 'BRSTNCLTN822'
BRSTNCLTN7N2 'LTN' 2025-01-01 2023-07-01 2022-12-01 71.09534936131405 85879.88111 'BRSTNCLTN7N2'
BRSTNCLTN7Z6 'LTN' 2025-07-01 2023-07-01 2022-12-01 86.3014706457653 114409.794 'BRSTNCLTN7Z6'
BRSTNCLTN7U7 'LTN' 2026-01-01 2023-07-01 2022-12-01 125.83542266699985 176512.29572999998 'BRSTNCLTN7U7'
BRSTNCLTN806 'LTN' 2029-01-01 2023-07-01 2022-12-01 1.01614235621514 2059.85428 'BRSTNCLTN806'
BRSTNCNTF147 'NTN-F' 2023-01-01 2023-07-01 2022-12-01 130.05322625658158 124000.88562999999 'BRSTNCNTF147'
BRSTNCNTF170 'NTN-F' 2025-01-01 2023-07-01 2022-12-01 108.43543880576354 104942.80642000001 'BRSTNCNTF170'
BRSTNCNTF1P8 'NTN-F' 2027-01-01 2023-07-01 2022-12-01 114.59290586419537 110293.48211 'BRSTNCNTF1P8'
BRSTNCNTF1Q6 'NTN-F' 2029-01-01 2023-07-01 2022-12-01 85.14605978670622 77623.61601000001 'BRSTNCNTF1Q6'
BRSTNCNTF204 'NTN-F' 2031-01-01 2023-07-01 2022-12-01 44.06809994428643 38949.86563 'BRSTNCNTF204'
BRSTNCNTF212 'NTN-F' 2033-01-01 2023-07-01 2022-12-01 14.08092836640031 15020.80838 'BRSTNCNTF212'
BRSTNCLF1R82 'LFT' 2023-03-01 2023-07-01 2022-12-01 181.88144006465564 14402.966970000001 'BRSTNCLF1R82'
BRSTNCLF1R90 'LFT' 2023-09-01 2023-07-01 2022-12-01 292.23466837959 23153.056 'BRSTNCLF1R90'
BRSTNCLF1RA8 'LFT' 2024-03-01 2023-07-01 2022-12-01 147.308421083892 11666.054 'BRSTNCLF1RA8'
BRSTNCLF0008 'LFT' 2024-09-01 2023-07-01 2022-12-01 200.0237965546861 15845.46066 'BRSTNCLF0008'
BRSTNCLF1RC4 'LFT' 2025-03-01 2023-07-01 2022-12-01 148.7287383573071 11785.42059 'BRSTNCLF1RC4'
BRSTNCLF1RD2 'LFT' 2025-09-01 2023-07-01 2022-12-01 182.327535660861 14445.717 'BRSTNCLF1RD2'
BRSTNCLF1RE0 'LFT' 2026-03-01 2023-07-01 2022-12-01 181.751678872712 14402.804 'BRSTNCLF1RE0'
BRSTNCLF1RF7 'LFT' 2026-09-01 2023-07-01 2022-12-01 50.1632217564877 3978.106 'BRSTNCLF1RF7'
BRSTNCLF1RG5 'LFT' 2027-03-01 2023-07-01 2022-12-01 219.8561120368935 17646.20265 'BRSTNCLF1RG5'
BRSTNCLF1RH3 'LFT' 2027-09-01 2023-07-01 2022-12-01 257.102833338839 20644.379 'BRSTNCLF1RH3'
BRSTNCLF1RI1 'LFT' 2028-03-01 2023-07-01 2022-12-01 166.355603279871 13331.895 'BRSTNCLF1RI1'
BRSTNCLF1RK7 'LFT' 2028-09-01 2023-07-01 2022-12-01 175.13288346151998 14006.48 'BRSTNCLF1RK7'
BRSTNCLF1RL5 'LFT' 2029-03-01 2023-07-01 2022-12-01 69.7051373551037 5579.181 'BRSTNCLF1RL5'
BRSTNCNTB054 'NTN-B' 2023-03-15 2023-07-01 2022-12-01 0.02202721467273 5.414 'BRSTNCNTB054'
BRSTNCNTB4O9 'NTN-B' 2023-05-15 2023-07-01 2022-12-01 185.916970143623 45767.11 'BRSTNCNTB4O9'
BRSTNCNTB096 'NTN-B' 2024-08-15 2023-07-01 2022-12-01 216.04041294789127 51759.3359 'BRSTNCNTB096'
BRSTNCNTB633 'NTN-B' 2025-05-15 2023-07-01 2022-12-01 142.526010171475 33523.057 'BRSTNCNTB633'
BRSTNCNTB4U6 'NTN-B' 2026-08-15 2023-07-01 2022-12-01 147.00422729214006 34317.784060000005 'BRSTNCNTB4U6'
BRSTNCNTB682 'NTN-B' 2027-05-15 2023-07-01 2022-12-01 83.73470947007179 20553.08 'BRSTNCNTB682'
BRSTNCNTB4X0 'NTN-B' 2028-08-15 2023-07-01 2022-12-01 92.5278532833915 20643.713 'BRSTNCNTB4X0'
BRSTNCNTB3B8 'NTN-B' 2030-08-15 2023-07-01 2022-12-01 124.16396004376469 28066.729440000003 'BRSTNCNTB3B8'
BRSTNCNTB674 'NTN-B' 2032-08-15 2023-07-01 2022-12-01 37.492323437061685 9046.79088 'BRSTNCNTB674'
BRSTNCNTB0O7 'NTN-B' 2035-05-15 2023-07-01 2022-12-01 127.2543027332329 32000.80374 'BRSTNCNTB0O7'
BRSTNCNTB3C6 'NTN-B' 2040-08-15 2023-07-01 2022-12-01 79.87603671353997 17702.75866 'BRSTNCNTB3C6'
BRSTNCNTB0A6 'NTN-B' 2045-05-15 2023-07-01 2022-12-01 100.44020817643981 27064.99005 'BRSTNCNTB0A6'
BRSTNCNTB3D4 'NTN-B' 2050-08-15 2023-07-01 2022-12-01 182.339056162318 41175.9465 'BRSTNCNTB3D4'
BRSTNCNTB4Q4 'NTN-B' 2055-05-15 2023-07-01 2022-12-01 126.46859992448975 26882.39055 'BRSTNCNTB4Q4'
BRSTNCNTB690 'NTN-B' 2060-08-15 2023-07-01 2022-12-01 21.940939267061502 5302.349 'BRSTNCNTB690'
BRSTNCNTB0K5 'NTN-B Principal' 2024-08-15 2023-07-01 2022-12-01 11.61582969694007 3155.62224 'BRSTNCNTB0K5'
BRSTNCNTB4W2 'NTN-B Principal' 2026-08-15 2023-07-01 2022-12-01 14.0557709565606 4109.37559 'BRSTNCNTB4W2'
BRSTNCNTB3E2 'NTN-B Principal' 2035-05-15 2023-07-01 2022-12-01 12.438188872702145 5864.20066 'BRSTNCNTB3E2'
BRSTNCNTB2U0 'NTN-B Principal' 2045-05-15 2023-07-01 2022-12-01 4.53103349484539 3337.99668 'BRSTNCNTB2U0'
BRSTNCNTC0K4 'NTN-C' 2031-01-01 2023-07-01 2022-12-01 80.28159981344272 11041.88923 'BRSTNCNTC0K4'
Binary file modified methods/extractors/__pycache__/webPageDataScrapers.cpython-311.pyc
View file
Binary file not shown.
20 changes: 10 additions & 10 deletions methods/loaders/filesSave.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,52 +36,52 @@ def creatingFinalDataFrame(self, df: pd.DataFrame, data: str, fileName, sep, nam
if file_type == 'csv':
novo_df.to_csv(f"{diretorio}.csv", sep=f"{sep}",
columns=['SERIE', 'TITULO', 'DATA_VENCIMENTO', 'DATA_REF', 'DATA_CAPTURA', 'FINANCEIRO (R$ BI)', 'QUANTIDADE (MIL)', 'COD_REF'], index=False)
logging.info(f"DataFrame salvo como {fileName} em formato CSV.")
logging.info(f"DATAFRAME SALVO COMO {fileName} EM FORMATO CSV.")
return fileName, file_type

elif file_type == 'excel':
novo_df.to_excel(f"{diretorio}.xlsx", index=False)
logging.info(f"DataFrame salvo como {fileName} em formato Excel.")
logging.info(f"DATAFRAME SALVO COMO {fileName} EM FORMATO EXCEL.")
return fileName, file_type

elif file_type == 'json':
novo_df.to_json(f"{diretorio}.json", orient='records')
logging.info(f"DataFrame salvo como {fileName} em formato JSON.")
logging.info(f"DATAFRAME SALVO COMO {fileName} EM FORMATO JSON.")
return fileName, file_type

elif file_type == 'parquet':
novo_df.to_parquet(f"{diretorio}.parquet", index=False)
logging.info(f"DataFrame salvo como {fileName} em formato Parquet.")
logging.info(f"DATAFRAME SALVO COMO {fileName} EM FORMATO PARQUET.")
return fileName, file_type

elif file_type == 'hdf':
novo_df.to_hdf(f"{diretorio}.h5", key='data')
logging.info(f"DataFrame salvo como {fileName} em formato HDF5/H5.")
logging.info(f"DATAFRAME SALVO COMO {fileName} EM FORMATO HDF5/H5.")
return fileName, file_type

elif file_type == 'pickle':
novo_df.to_pickle(f"{diretorio}.pkl")
logging.info(f"DataFrame salvo como {fileName} em formato Pickle.")
logging.info(f"DATAFRAME SALVO COMO {fileName} EM FORMATO PICKLE.")
return fileName, file_type

elif file_type == 'feather':
novo_df.to_feather(f"{diretorio}.feather")
logging.info(f"DataFrame salvo como {fileName} em formato Feather.")
logging.info(f"DATAFRAME SALVO COMO {fileName} EM FORMATO FEATHER.")
return fileName, file_type

elif file_type == 'avro':
with open(f"{diretorio}.avro", 'wb') as out_avro:
fastavro.writer(out_avro, novo_df.to_dict(orient='records'))
logging.info(f"DataFrame salvo como {fileName} em formato Avro.")
logging.info(f"DATAFRAME SALVO COMO {fileName} EM FORMATO AVRO.")
return fileName, file_type

elif file_type == 'html':
novo_df.to_html(f"{diretorio}.html", index=False)
logging.info(f"DataFrame salvo como {fileName} em formato HTML.")
logging.info(f"DATAFRAME SALVO COMO {fileName} EM FORMATO HTML.")
return fileName, file_type

else:
logging.info("Tipo de arquivo não suportado. Por favor, escolha um formato válido.")
logging.info("TIPO DE ARQUIVO NÃO SUPORTADO. ESCOLHA UM FORMATO VÁLIDO.")

def concatDataFrame(self, df: pd.DataFrame, dictionary: dict, index: int):
try:
Expand Down
Binary file modified methods/transformers/__pycache__/transformData.cpython-311.pyc
View file
Binary file not shown.
Binary file modified utils/__pycache__/aws.cpython-311.pyc
View file
Binary file not shown.
Binary file modified utils/__pycache__/logger_config.cpython-311.pyc
View file
Binary file not shown.
Binary file modified utils/__pycache__/tools.cpython-311.pyc
View file
Binary file not shown.
6 changes: 1 addition & 5 deletions utils/aws.py
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
import os
import boto3
import time
import utils.logger_config as logger_config
Expand All @@ -8,20 +7,17 @@
generalTools = GeneralTools()
logger_config.setup_logger(time.strftime("%Y-%m-%d %H:%M:%S"))


class AboutAWS:
def __init__(self):
self.jsonData = generalTools.openJson()

def createClient(self, service: str):
try:
# ACESSAR OS DADOS PELO DATA.JSON, aws_account
client = boto3.client(
service_name=service,
aws_access_key_id=self.jsonData['aws_account']['access_key'],
aws_secret_access_key=self.jsonData['aws_account']['secret_key'],
region_name=self.jsonData['aws_account']['region']
# Podendo usar qualquer região
)
return client
except Exception as e:
Expand All @@ -31,7 +27,7 @@ def createBucket(self, client, bucketname: str):
try:
client.create_bucket(
ACL='private',
Bucket=bucketname, # altere para um nome qualquer
Bucket=bucketname,
CreateBucketConfiguration={
'LocationConstraint': self.jsonData['aws_account']['region']
},
Expand Down

0 comments on commit b802fc3

Please sign in to comment.