With Brazilian and Portuguese nationality (EU citizen), I currently work as a data engineer, with experiences in business intelligence and data science as well. In my academic and professional experiences, I could improve my autodidact and analytical skills and my profile as ”problem solver”.
During my experiences I was able to work on different projects covering the following topics:
- Development, maintenance and monitoring of data pipelines mainly using Python, SQL, Spark for data processing, cloud AWS, bash/shell, relational and non‐relational databases, Git for version control and Airflow as a orchestrator;
- Integrations mainly using Python with APIs;
- Development of Flask APIs;
- Data visualization using mainly Metabase and Tableau tools, as well as some Python libraries like Plotly, Seaborn, Matplotlib.
Python
AWS
Azure
• Development, maintenance, monitoring and orchestration of data pipelines on AWS using Python, Pentaho Data Integration and DBT
• Data Modeling in Redshift Data Warehouse / S3 Data Lake
• Data pipeline orchestration and CI/CD using Rundeck
• Productization of Data Science team models
• Development of app using Streamlit
• Data Architecture
• Responsible for the development, maintenance and monitoring of data pipelines for stakeholders in Asia using mainly Azure resources like Azure Databricks (Spark) for data processing, Blob Storage, SQL Server
• Data pipeline orchestration using Azure Data Factory, Airflow and Prefect
• Development of Flask APIs, which are the core of the main client application
• Database versioning using Liquibase
• CI/CD using Azure DevOps/Pipelines
• Development, maintenance and monitoring of data pipelines mainly using the programming languages Python and SQL, data processing using Apache Hive, Apache Hadoop and Apache Spark (PySpark), cloud services in AWS (S3, RDS, EMR and EC2), bash/shell, relational and non‑relational databases, Git for version control and Airflow as a orchestrator.
• Integrations mainly using Python with APIs
• Development of Flask APIs
• Web Scrapper and Crawler development using Scrapy and Requests (Python)
• Data streaming using Apache Kafka
Using Python, Shell Script, AWS, Xplenty, Redshift, Postgres, SQL and Metabase for maintenance of the data structure, automation and optimization of internal processes, project development with the Data Visualization team, ETL and data enrichment.