First caveat: this isn’t a post just for women. This is a post to anyone and everyone who’s asking, wondering, or thinking. Second caveat: I’m speaking from the perspective of a woman and a data engineer, because that’s what I know. However, the sentiment could certainly apply to endless other…

Early on, software engineers are taught: develop locally, test in staging, deploy to production. What does this mean for analytics? Development on many analytics teams This might sound familiar: “Developing locally” meaning running queries in the one data warehouse in hopes of not stalling jobs powering BI reporting; “testing in staging” meaning performing ad-hoc analysis…

Use Grafana on top of the official Apache Airflow image to monitor queue health and much more. — An unsettling yet likely familiar situation: you deployed Airflow successfully, but find yourself constantly refreshing the webserver UI to make sure everything is running smoothly. You rely on certain alerting tasks to execute upon upstream failures, but if the queue is full and tasks are stalling, how will you be…