Career & Skills Progression

---
config:
  look: handDrawn
---

flowchart LR
  id[****FINANCIAL MODELING****]

  classDef bluedrawn fill:#c0e5fe,stroke:#612200,color:#000000;
  class id bluedrawn;

---
config:
  look: handDrawn
---

flowchart LR
  id2[****DATA SCIENCE****]

  classDef peachdrawn fill:#ffddd6,stroke:#612200,color:#000000;
  class id2 peachdrawn;

---
config:
  look: handDrawn
---

flowchart LR
  A[EY] --> B[PG&E]
  B --> C[KPMG]
  C --> D{UCI's<br>MSBA<br>Program}
  D --> E[Centene]
  E --> F[Bloomreach]
  F --> G[Centene]

  classDef whitedrawn fill:#ffffff,stroke:#612200,color:#000000;
  class A,B,C,D,E,F,G whitedrawn;

Career & Skills Progression

---
config:
  look: handDrawn
---

flowchart LR
  id[**FINANCIAL MODELING**]

  classDef bluedrawn fill:#c0e5fe,stroke:#612200,color:#000000;
  class id bluedrawn;

---
config:
  look: handDrawn
---

flowchart LR
  id2[**DATA SCIENCE**]

  classDef peachdrawn fill:#ffddd6,stroke:#612200,color:#000000;
  class id2 peachdrawn;

---
config:
  look: handDrawn
---

flowchart LR
  A[EY] --> B[PG&E]
  B --> C[KPMG]
  C --> D{UCI's<br>MSBA<br>Program}
  D --> E[Centene]
  E --> F[Bloomreach]
  F --> G[Centene]

  classDef bluedrawn fill:#c0e5fe,stroke:#612200,color:#000000;
  classDef whitedrawn fill:#ffffff,stroke:#612200,color:#000000;
  class A,B,C bluedrawn;
  class D,E,F,G whitedrawn;

Career & Skills Progression

---
config:
  look: handDrawn
---

flowchart LR
  id[**FINANCIAL MODELING**]

  classDef bluedrawn fill:#c0e5fe,stroke:#612200,color:#000000;
  class id bluedrawn;

---
config:
  look: handDrawn
---

flowchart LR
  id2[**DATA SCIENCE**]

  classDef peachdrawn fill:#ffddd6,stroke:#612200,color:#000000;
  class id2 peachdrawn;

---
config:
  look: handDrawn
---

flowchart LR
  A[EY] --> B[PG&E]
  B --> C[KPMG]
  C --> D{UCI's<br>MSBA<br>Program}
  D --> E[Centene]
  E --> F[Bloomreach]
  F --> G[Centene]

  classDef bluedrawn fill:#c0e5fe,stroke:#612200,color:#000000;
  classDef peachdrawn fill:#ffddd6,stroke:#612200,color:#000000;
  classDef whitedrawn fill:#ffffff,stroke:#612200,color:#000000;
  class A,B,C bluedrawn;
  class D whitedrawn;
  class E,F,G peachdrawn;

Career & Skills Progression

---
config:
  look: handDrawn
---

flowchart LR
  id[**FINANCIAL MODELING**]

  classDef bluedrawn fill:#c0e5fe,stroke:#612200,color:#000000;
  class id bluedrawn;

---
config:
  look: handDrawn
---

flowchart LR
  id2[**DATA SCIENCE**]

  classDef peachdrawn fill:#ffddd6,stroke:#612200,color:#000000;
  class id2 peachdrawn;

---
config:
  look: handDrawn
---

flowchart LR
  A[EY] --> B[PG&E]
  B --> C[KPMG]
  C --> D{UCI's<br>MSBA<br>Program}
  D --> E[Centene]
  E --> F[Bloomreach]
  F --> G[Centene]
  
  classDef bluedrawn fill:#c0e5fe,stroke:#612200,color:#000000;
  classDef peachdrawn fill:#ffddd6,stroke:#612200,color:#000000;
  classDef whitedrawn fill:#ffffff,stroke:#612200,color:#000000;
  class A,B,C bluedrawn;
  class D whitedrawn;
  class E,F,G peachdrawn;

---
config:
  look: handDrawn
---

flowchart LR
  A[Excel, PowerPoint, Access, SQL] --> B{UCI's<br>MSBA<br>Program}
  B --> C[Automating Manual Excel Work]
  C --> D[Developing R Packages & Web Apps]
  B --> E[Code-based Analytics]
  E --> F[Machine Learning & AI]
  F --> G[R, Linux, git, Posit tools, Databricks]

  classDef bluedrawn fill:#c0e5fe,stroke:#612200,color:#000000;
  classDef peachdrawn fill:#ffddd6,stroke:#612200,color:#000000;
  classDef whitedrawn fill:#ffffff,stroke:#612200,color:#000000;
  class A,C bluedrawn;
  class D,E,F,G peachdrawn;
  class B whitedrawn;

Standing Out with Shiny

Shiny Demo as a Resume Accessory

• Most corporate dashboards feel clunky…

• They’re laggy, very boxy, e.g., Power BI, Tableau, MicroStrategy

• Dashboards typically struggle with fast, real-time calculations, search, and data manipulation capabilities

• For hiring teams, you can use Shiny as a resume accessory to demonstrate how easy it is to develop and deploy a web app styled with the company’s brand aesthetics

Shiny Demo as a Resume Accessory

Bloomreach’s website:

Shiny Demo as a Resume Accessory

Javier’s Shiny demo:

Pro Tip! 💡 Shiny Assistant

• The new Shiny Assistant… 🤯… An LLM-powered tool that builds functional Python and R Shiny apps in minutes!
  
  
• Shiny Assistant builds apps using Shinylive, a web interface made possible by WebAssembly (or “wasm”)
  
  
• The LLM’s training data is a few months old as of this writing so it won’t know about the latest Shiny features

My Professional Data Science Workflow

My Professional Data Science Toolkit

My Professional Data Science Toolkit

My Professional Data Science Toolkit

Typical Python Tools for Data Science

Typical Python Tools for Data Science

In Review

• RStudio IDE’s “always on” panes welcomed R users seeking a data-analysis-first experience

• For Python users, RStudio felt too R-centric and other tools worked just fine including VS Code, Jupyter Notebooks, PyCharm, etc.

• There are many programming languages that can be used for data analysis, but Python and R are the de facto standards for data science

Positron, it looks familiar!

About Positron

• What is Positron? From Posit’s getting started docs:

• A next-generation data science IDE built by Posit PBC
• An extensible, polyglot tool for writing code and exploring data
• A familiar environment for reproducible authoring and publishing

• Positron is a tailor-made IDE for data science built on top of Code OSS that can be used with any combination of programming languages

VS Code OSS w/ RStudio panes!

Data Explorer Overview

Learn More about Positron

Installing Arrow, DuckDB, and Polars

# Arrow env vars for AWS S3 & GCP support
import os
os.environ["LIBARROW_MINIMAL"] = "FALSE"
os.environ["ARROW_S3"] = "ON"
os.environ["ARROW_GCS"] = "ON"

# Install packages
!pip install pyarrow duckdb polars

# Arrow env vars for AWS S3 & GCP support
Sys.setenv(LIBARROW_MINIMAL = "false")
Sys.setenv(ARROW_S3 = "ON")
Sys.setenv(ARROW_GCS = "ON")

# Enable Polars install with pre-built
# Rust library binary 
Sys.setenv(NOT_CRAN = "true")

# Install packages
install.packages(c("arrow", "duckdb"))
install.packages("polars", repos = "https://community.r-multiverse.org")
install.packages("polarssql", repos = "https://rpolars.r-universe.dev")

Back to fast… How fast?

• My personal laptop is a MacBook Air with 24 GB RAM
• To test Arrow’s capabilities, I read a 40 GB dataset with more than 1.1 billion rows and 24 columns
• The .parquet dataset was partitioned by Year and Month (120 files)
• Important to note that my laptop would not be able to load this object entirely into memory as a data.frame or tibble given my laptop’s limited RAM

Benchmarking Read Times

R

library(ggplot2)
library(bench)

# Benchmark Read Times
bnch <- bench::mark(
  min_iterations = 1000,
  arrow = arrow::open_dataset(here::here("data/nyc-taxi"))
)

autoplot(bnch)

Benchmarking Read Times

• Once “locally” available, the 40GB, 1.1 billion row dataset (benchmarked 1,000 times) can be read on average in 17ms

Benchmarking Data Manipulation

• Using R’s dplyr, we can perform ETL on an arrow table and collect() the results back to a df

# Open Arrow connection to dataset (40 GB)
nyc_taxi <- open_dataset(here::here("data/nyc-taxi"))

# Benchmark dplyr pipeline
bnch <- bench::mark(
  min_iterations = 100,
  arrow = nyc_taxi |> 
    filter(payment_type %in% c("Credit card", "Cash")) |> 
    group_by(payment_type) |> 
    summarise(mean_fare = mean(fare_amount, na.rm = T),
              mean_tip = mean(tip_amount, na.rm = T)) |> 
    ungroup() |> 
    dplyr::collect()
)

autoplot(bnch)

Benchmarking ETL

• The results show the ETL pipeline on average summarized 1.1 billion rows in 9.5s, benchmarked over 100 iterations

Note for R’s Tidyverse Users

• Many functions from the tidyverse collections of packages have 1:1 compatibility with Arrow tables
• However, sometimes you’ll encounter a breaking point
• Take this stringr::str_replace_na() example:
  
  

nyc_taxi |> 
  mutate(vendor_name = str_replace_na(vendor_name, "No vendor"))
#> Error: Expression str_replace_na(vendor_name, "No vendor") 
#> not supported in Arrow

User Defined Functions

• Arrow allows users to create and register User Defined Functions (or “UDFs”) to the Arrow engine
• Almost any function can be made Arrow-friendly by registering custom UDFs to the Arrow kernel
• Let’s learn how to register stringr::str_replace_na() with the Arrow kernel

Registering UDFs

• First, run arrow::schema() on your Arrow table to review the field name and data type pairs

arrow::schema(nyc_taxi)
#> Schema
#> vendor_name: string
#> pickup_datetime: timestamp[ms]
#> dropoff_datetime: timestamp[ms]
#> passenger_count: int64
#> trip_distance: double
#> pickup_longitude: double
#> pickup_latitude: double
#> ...

• Since I want to mutate the vendor_name field, I know I’ll be working with an Arrow string() data type

Registering UDFs

• Next, we’ll use arrow::register_scalar_function()
• Name your UDF replace_arrow_nas
• If you’re registering a tidyverse function, set auto_convert = TRUE

arrow::register_scalar_function(
  name = "replace_arrow_nas",
  # Note: The first argument must always be context
  function(context, x, replacement) {
    stringr::str_replace_na(x, replacement)
  },
  in_type = schema(
    x = string(),
    replacement = string()
  ),
  out_type = string(),
  auto_convert = TRUE
)

Trying the UDF

• Let’s see if the registered replace_arrow_nas() Arrow UDF works…

nyc_taxi |> 
  mutate(vendor_name = replace_arrow_nas(vendor_name, "No vendor")) |> 
  distinct(vendor_name) |> 
  arrange(vendor_name) |> 
  collect()

Trying the UDF

• Let’s see if the registered replace_arrow_nas() Arrow UDF works…

nyc_taxi |> 
  mutate(vendor_name = replace_arrow_nas(vendor_name, "No vendor")) |> 
  distinct(vendor_name) |> 
  arrange(vendor_name) |> 
  collect()

#> # A tibble: 3 × 1
#>   vendor_name
#>   <chr>      
#> 1 CMT
#> 2 No vendor
#> 3 VTS

DuckDB Usage Basics

import duckdb

# Connect to an in-memory DuckDB instance and scan
# the Parquet data set to make a temp View
con = duckdb.connect()

con.execute("""
    CREATE VIEW nyc_taxi AS
    SELECT * FROM read_parquet('data/nyc-taxi/**/*.parquet', hive_partitioning=true)
""")

# Run your SQL query
df = con.execute("""
    SELECT 
        payment_type,
        AVG(fare_amount) AS mean_fare,
        AVG(tip_amount)  AS mean_tip
    FROM nyc_taxi
    WHERE payment_type IN ('Credit card', 'Cash')
    GROUP BY payment_type
""").df()

print(df)

library(duckdb)

# Connect to an in-memory DuckDB instance and scan
# the Parquet data set to make a temp View
con <- dbConnect(duckdb())

dbExecute(con, "
  CREATE VIEW nyc_taxi AS 
  SELECT * FROM read_parquet('data/nyc-taxi/**/*.parquet', hive_partitioning = true)"
)

# Run your SQL query
df <- dbGetQuery(con, "
    SELECT 
        payment_type,
        AVG(fare_amount) AS mean_fare,
        AVG(tip_amount)  AS mean_tip
    FROM nyc_taxi
    WHERE payment_type IN ('Credit card', 'Cash')
    GROUP BY payment_type
")

print(df)

DuckDB Streaming with Python

# DuckDB via Python
# Open dataset using year,month folder partition
nyc = ds.dataset('nyc-taxi/', partitioning=["year", "month"])

# Get database connection
con = duckdb.connect()

# Run query that selects part of the data
query = con.execute("SELECT total_amount, passenger_count,year FROM nyc where total_amount > 100 and year > 2014")

# Create Record Batch Reader from Query Result.
# "fetch_record_batch()" also accepts an extra parameter related to the desired produced chunk size.
record_batch_reader = query.fetch_record_batch()

# Retrieve all batch chunks
chunk = record_batch_reader.read_next_batch()
while len(chunk) > 0:
    chunk = record_batch_reader.read_next_batch()

# We must exclude one of the columns of the NYC dataset due to an unimplemented cast in Arrow
working_columns = ["vendor_id","pickup_at","dropoff_at","passenger_count","trip_distance","pickup_longitude",
    "pickup_latitude","store_and_fwd_flag","dropoff_longitude","dropoff_latitude","payment_type",
    "fare_amount","extra","mta_tax","tip_amount","tolls_amount","total_amount","year", "month"]

# Open dataset using year,month folder partition
nyc_dataset = ds.dataset(dir, partitioning=["year", "month"])
# Generate a scanner to skip problematic column
dataset_scanner = nyc_dataset.scanner(columns=working_columns)

# Materialize dataset to an Arrow Table
nyc_table = dataset_scanner.to_table()

# Generate Dataframe from Arow Table
nyc_df = nyc_table.to_pandas()

# Apply Filter
filtered_df = nyc_df[
    (nyc_df.total_amount > 100) &
    (nyc_df.year >2014)]

# Apply Projection
res = filtered_df[["total_amount", "passenger_count","year"]]

# Transform Result back to an Arrow Table
new_table = pa.Table.from_pandas(res)

For R users: duckplyr

• duckplyr, from DuckDB Labs, offers 1:1 compatibility with dplyr functions but there are some caveats:
  • Factor columns, nested lists, and nested tibbles are not yet supported
  • To group data, use dplyr’s summarize() function with the .by argument as group_by() is not be supported

Polars

Polars

• While Arrow is primarily a memory format and data transfer standard, Polars is specifically designed for data analysis with lazy evaluation, DataFrame manipulations, and a consistent API across languages
  
  
• DuckDB is a full database system that excels at SQL queries and can integrate with both Arrow and Polars, but Polars focuses on in-memory processing and programmatic data manipulation

Benchmarking Analysis

• Arrow and DuckDB really stood out for fast manipulation of data using dplyr syntax
• The SQL below shows the basic transformation done to the data using dplyr, arrow, duckdb, duckplyr, and polars

SELECT 
    payment_type,
    AVG(fare_amount) AS mean_fare,
    AVG(tip_amount)  AS mean_tip
FROM nyc_taxi
WHERE payment_type IN ('Credit card', 'Cash')
GROUP BY payment_type;

Benchmark: 1 million rows

Benchmark: 10 million rows

Benchmark: 100 million rows

Benchmark: 500 million rows

Benchmark: 1.1 billion rows

Quarto Presentations

• The following sections were copied almost entirely from Quarto’s Reveal.js demo documentation

• The next few slides cover what you can do with Quarto and Reveal.js including:

  • Presenting code and LaTeX equations
  • Rendering code chunk computations in slide output
  • Fancy transitions, animations, and code windows

Pretty Code

• Over 20 syntax highlighting themes available
• Default theme optimized for accessibility

# Define a server for the Shiny app
function(input, output) {
  
  # Fill in the spot we created for a plot
  output$phonePlot <- renderPlot({
    # Render a barplot
  })
}

Code Animations

• Over 20 syntax highlighting themes available
• Default theme optimized for accessibility

# Define a server for the Shiny app
function(input, output) {
  
  # Fill in the spot we created for a plot
  output$phonePlot <- renderPlot({
    # Render a barplot
    barplot(WorldPhones[,input$region]*1000, 
            main=input$region,
            ylab="Number of Telephones",
            xlab="Year")
  })
}

Line Highlighting

• Highlight specific lines for emphasis
• Incrementally highlight additional lines

import numpy as np
import matplotlib.pyplot as plt

r = np.arange(0, 2, 0.01)
theta = 2 * np.pi * r
fig, ax = plt.subplots(subplot_kw={'projection': 'polar'})
ax.plot(theta, r)
ax.set_rticks([0.5, 1, 1.5, 2])
ax.grid(True)
plt.show()

LaTeX Equations

• To include a LaTeX equation in Quarto, you would use the double dollar sign delimiters ($$) for a display equation on a separate line, like this:

• $$x = \frac{-b \pm \sqrt{(b^2 - 4ac)}}{2a}$$

\[x = \frac{-b \pm \sqrt{(b^2 - 4ac)}}{2a}\]

Counters

Great for training purposes or when you’ve got a time limit

# YOUR TURN! Complete the below 
# dplyr pipeline to group by and
# summarize the data set

library(dplyr)
data(package = "ggplot2", "diamonds")

diamonds |> 
  <your code here>

library(countdown)
countdown(
  minutes = 0, 
  seconds = 10,
  play_sound = TRUE,
  color_border = "#1A4064",
  color_text = "#76AADB",
  font_size = "3em"
)