Isolation forest or “iForest” is an astoundingly beautiful and elegantly simple algorithm that identifies anomalies with few parameters. The original paper is accessible to a broad audience and contains minimal math. …

Hint: It’s not programming skills or familiarity with algorithms — The most important aspect of data science is communication. Algorithms, coding languages and software are important to know but these things are easily and quickly looked up when details become shrouded in the dust of time. …

Demonstrating where to download JARs and how to install them on AWS EMR clusters for access from EMR Notebooks — I have yet to see a straightforward and comprehensive guide on how to get JAR files onto every worker node of an EMR cluster and yet this is a critically important, common need. This article addresses those needs. …

Communicating a coherent, data-driven story is the most important skill for today’s data scientist yet the least developed. Better tools can help — learn about a new one today. — Over the years, I have seen many PhD-holding data scientists spend weeks or months building highly effective machine learning pipelines that (theoretically) will deliver real-world value. Unfortunately, these fruits of labor can die on the vine if they fail to effectively communicate the value of their work, a misfortune I…

A quick guide with code (i.e. my rough notes for replication purposes) Motivation There are a lot of interesting applications for packet capture data. I will refrain from stating them for corporate privacy reasons. Instructions (Part 1: Wireshark GUI ) This part is straightforward and useful for starting off. In part 2, I show you how to…

Setting up your Amazon Web Services (AWS) Elastic MapReduce (EMR) Cluster with XGBoost — Introduction This article assumes you are already familiar with what XGBoost/CatBoost/etc. do and that you are here to actually get them to work. Installing packages on a local machine/single node is easy. Doing the same for a cluster environment in order to work with big data is less so and the…

Because AWS documentation is out-of-date, wrong, verbose yet not specific enough or requires you to read 5–10 different link trees of pages of documentation. Download the latest Stable installation of PuTTY (e.g. putty-64bit-0.74-installer.msi). The installation should also install needed utilities like puttygen and pageant. 2. Create an EMR instance (guide…

Produce website-worthy visualizations Introduction Flourish is a simple browser-based point-and-click, drag-and-drop data visualization creator suitable for well-structured, tabular data in the form of .csv or Excel files. Introduced in March 2016¹, it is a relatively new tool compared to entrenched competitors like Tableau (founded in 2003²) and aims at an audience…

I had many managers and held multiple leadership roles during my life. This is a living list of notes based on those experiences. While the following concepts are widely applicable, I write with the data science team in mind. When given headcount, hiring should be your top priority. You can…