DataFrames

Julia has a general representation for Tables with different implementations. The most used package is DataFrames.jl. It takes advantage of Julia syntax to define certain operation.

Define a DataFrame

using DataFrames
import Random

Random.seed!(11)
n = 100
data = DataFrame(id = 1:n, x = rand(n), y = randn(n))

Subsetting

Let's select a column without doing a copy:

data.id

100-element Vector{Int64}:
   1
   2
   3
   4
   5
   6
   7
   8
   9
  10
   ⋮
  92
  93
  94
  95
  96
  97
  98
  99
 100

data[!, :id]

100-element Vector{Int64}:
   1
   2
   3
   4
   5
   6
   7
   8
   9
  10
   ⋮
  92
  93
  94
  95
  96
  97
  98
  99
 100

Modifications on these columns will be reflected in the original data.

xaux = data.x

100-element Vector{Float64}:
 0.49843398493613866
 0.3897206442798764
 0.26454039637865223
 0.7194237171475113
 0.6766024102635798
 0.1840786120276101
 0.4697903183183523
 0.5680018585914868
 0.5108399240545426
 0.3503150251639139
 ⋮
 0.5055020246950553
 0.20199332879192944
 0.4411645679023518
 0.5312642809575425
 0.0985509072108206
 0.6776076972596555
 0.10551392206669663
 0.2581188423585902
 0.8888936999885325

xaux[1] = 100

100

first(data, 2)

A copy of a column is created as folows:

xaux = data[:, :x]

100-element Vector{Float64}:
 100.0
   0.3897206442798764
   0.26454039637865223
   0.7194237171475113
   0.6766024102635798
   0.1840786120276101
   0.4697903183183523
   0.5680018585914868
   0.5108399240545426
   0.3503150251639139
   ⋮
   0.5055020246950553
   0.20199332879192944
   0.4411645679023518
   0.5312642809575425
   0.0985509072108206
   0.6776076972596555
   0.10551392206669663
   0.2581188423585902
   0.8888936999885325

xaux[1] = 10

10

first(data, 2)

More specific subsetting can be done with:

data.id[1:10]

10-element Vector{Int64}:
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10

data.id[90:end]

11-element Vector{Int64}:
  90
  91
  92
  93
  94
  95
  96
  97
  98
  99
 100

data[1:10, 1:2]

You can also create a view that references to you DataFrame:

subdata = view(data, 1:10, 1:3)
subdata.x[1] = rand()
first(data, 2)

Transform variables

To perform operations over columns, we use the Pair syntax:

transform(data, :x => (z -> z .^ 2))

typeof(:x => (z -> z .^ 2))

Pair{Symbol, Main.var"Main".var"#3#4"}

We can explicitly provide the output name:

transform(data, :x => (z -> z .^ 2) => :x2)

We can also vectorize any function with ByRow:

transform(data, :x => ByRow(sqrt))

Notice that the previous operations did not modify the original dataset. You can modify your original dataset using the in-place function transform!:

transform!(data, :x => (z -> z .^ 2) => :x2)

data

The function select works in a similar way, but only includes the desired variables:

select(data, :x => (z -> z .^ 2) => :x2)

select(data, :x, :y)

typeof(r"^x")

Regex

select(data, r"^x")

Let's add new columns:

insertcols!(data, :z => rand(100))

first(data, 5)

Let's remove columns:

select!(data, Not(:x2))

first(data, 5)

Another simple operation is to rename columns:

rename(data, :x => :xnew, :z => :znew)

rename(data, [:x, :z] .=> [:xnew, :znew])

rename(var -> var * "_new", data)