From the very beginning, one of the core ideas behind GRID has been that spreadsheets — and spreadsheet models in particular — can be made easier to share and consume by bringing them to the web with beautiful narration and engaging interactivity. For the interactivity we knew that we’d need…

The upcoming version of Python is set to be twice as fast as the previous versions. — While Python is one of the most widely used computer languages in finance, it is more commonly utilized for specific activities such as data analysis than for actual market-making software. …

Python is well known for its simplicity and many resources that can help you. Not only do we have many different resources in our community but we also have a lot of helpful resources inside Python standard libraries. One that is really useful for example is a decorator named “cache”…

Data Science projects that will help you land your dream job — Data science is a hot field right now, and there is no better way to show that you are qualified for the job than by doing your own data science projects. In this blog post, we will look at 7 data science projects that you can do with your free…

How to automatically build your Docker images; a case study. — Jenkins is a self-contained, open-source automation server with packages for Windows, Mac OS X, and other Unix-like operating systems. If you visit the project’s landing page, it informs you that Jenkins is the leading open-source automation server, with hundreds of plugins to support building, deploying, and automating any project. This…

A Curated List of the Best Kaggle Notebooks from 2021 — Be sure to subscribe here or to my exclusive newsletter to never miss another article on data science guides, tricks and tips, life lessons, and more! Introduction I love Kaggle. When I started my journey learning data science, Kaggle was one of the main resources that accelerated my learning. …

I’ve recently been developing and operating a lot more on the Airflow platform. While doing this development, there was always the challenge of communicating the parts of your DAG effectively without the developer or operator having to open the DAG code to understand how it functions. Whilst trying to find…

Use Official Docker Images as Base Image Try to use Official Docker Images as Base Image, this will made your Dockerfile cleaner and let you use a verified image and already built with best practices. Use Specific Docker Image Versions “ From node ” it will use the latest tag “ From node:latest ” and this is very bad !!!, you might…

Increase your breadth of knowledge with these major engineering topics — I was pretty late to the non-fiction party. I started reading books relating to software engineering when I became a senior. By no means was it the end of the world, and I learnt a lot from online sources instead, but there are incredible amounts to be learnt from books…

A framework proposal for Apache Airflow — This is the second article about why Data Engineers shouldn’t write Airflow DAGs. In this new article, we are going to introduce a framework proposal for Apache Airflow. This article aims to shed some light on how building a framework can help you solve some of the problems related to…

In the blog post, we will see some best practices for authoring DAGs. Let’s start. DAG as configuration file The Airflow scheduler scans and compiles DAG files at each heartbeat. If DAG files are heavy and a lot of top-level codes are present in them, the scheduler will consume a…

Testing is an integral part of any software system to build confidence and increase the reliability of the system. Recently, I joined Grab and here at Grab, we are using Airflow to create and manage pipelines. But, we were facing issues with Airflow. …