R, Python, Julia の速度比較 2021/12
実験環境
macOS Monterey バージョン 12.1
Mac mini (M1, 2020)
チップ Apple M1
メモリ 8GB
R M1 チップ対応
version 4.2.0
platform aarch64-apple-darwin21.1.0
arch aarch64
os darwin21.1.0
system aarch64, darwin21.1.0
Python M1 チップ対応
Python 3.10.0
Julia M1 チップ対応
Version 1.7.0 (2021-11-30)
使用するデータ
> n = 100000
> m = 1500
> x = data.frame(round(matrix(rnorm(n*m, 50, 10), n, m), 2))
> system.time({write.csv(x, file="test.csv", row.names=FALSE)})
user system elapsed
59.049 1.063 60.135
データフレームとして読み込み
R の場合
> system.time({df = read.csv("test.csv", colClasses="numeric")})
user system elapsed
12.105 0.906 13.627
ちなみに，data.table パッケージの fread だと更に速い。
> library(data.table)
> system.time({df = fread("test.csv")})
user system elapsed
1.723 0.258 3.058
しかし，fread で読み込まれるのは data.table なので，data.frame と若干の違いがあるので注意。
以下は，read.csv で読み込んだ data.frame を使う。
Python の場合
>>> from time import time
>>> import pandas as pd
>>> start = time(); df = pd.read_csv('test.csv'); print(time() - start)
6.904027938842773
Julia の場合
using CSV, DataFrames
@time df = CSV.read("test.csv", DataFrame); # 4.313486 seconds
@time df = CSV.read("test.csv", DataFrame, type=Float64); # 3.268237 seconds
単変量関数
合計
> system.time({s = colSums(df)})
user system elapsed
0.368 0.318 0.743
>>> start = time(); s = df.sum(); print(time() - start)
2.547268867492676
@time s = sum.(eachcol(df)); # 0.044548 seconds
算術平均
> system.time({m = colMeans(df)})
user system elapsed
0.353 0.084 0.436
>>> start = time(); m = df.mean(); print(time() - start)
0.2523488998413086
@time m = mean.(eachcol(df)); # 0.041935 seconds
不偏分散
> system.time({v = sapply(df, var)})
user system elapsed
0.487 0.002 0.491
>>> start = time(); v = df.var(ddof=1); print(time() - start)
0.6684608459472656
@time v = var.(eachcol(df)); # 0.080120 seconds
標準偏差
> system.time({sd = sapply(df, sd)})
user system elapsed
0.496 0.003 0.497
>>> start = time(); sd = df.std(ddof=1); print(time() - start)
0.7022347450256348
@time sd = std.(eachcol(df)); # 0.076600 seconds
中央値
median() はデータをソートする必要があるので，処理時間は長くなる。
> system.time({sapply(df, median)})
user system elapsed
2.086 0.364 2.691
>>> start = time(); s = df.median(); print(time() - start)
1.9600980281829834
@time med = median.(eachcol(df)); # 1.667501 seconds
二変量関数
ピアソンの積率相関係数
> system.time({cor(df[,1], df[,2])})
user system elapsed
0.005 0.001 0.005
>>> import numpy as np
>>> start = time(); r = np.corrcoef(df['X1'], df['X2'])[0,1]; print(time() - start)
0.0030248165130615234
>>> from scipy.stats import pearsonr
>>> start = time(); r = pearsonr(df['X1'], df['X2'])[0]; print(time() - start)
0.0011849403381347656
@time cor(df.X1, df.X2) # 0.000154 seconds
多変数関数
ピアソンの積率相関係数行列
> system.time({r = cor(df)})
user system elapsed
106.154 0.377 106.509
>>> start = time(); r = df.corr(); print(time() - start)
377.98001408576965
@time r = cor(Matrix(df)); # 3.985498 seconds
スピアマンの順位相関係数行列
> system.time({rs = cor(df[1:200], method="spearman")})
user system elapsed
4.698 0.152 4.858
>>> start = time(); rs = df.iloc[:, 0:200].corr(method='spearman'); print(time() - start)
23.894737243652344
@time rs = corspearman(Matrix(df[:, 1:200])); # 217.821656 seconds
ケンドールの順位相関係数行列
> system.time({rk = cor(df[1:5], method="kendall")})
user system elapsed
327.977 0.465 328.402
探索していて見つけたライブラリ
https://rdrr.io/cran/pcaPP/man/cor.fk.html
library(pcaPP)
> system.time({cor.fk(df[1:5])})
user system elapsed
0.106 0.002 0.109
> system.time({cor.fk(df[1:50])})
user system elapsed
10.330 0.262 10.560
>>> start = time(); rk = df.iloc[:, 0:5].corr(method='kendall'); print(time() - start)
0.2503321170806885
>>> start = time(); rk = df.iloc[:, 0:50].corr(method='kendall'); print(time() - start)
26.7141010761261
@time rk = corkendall(Matrix(df[:, 1:5])) # 0.134137 seconds
@time rk = corkendall(Matrix(df[:, 1:50])) # 11.786005 seconds;
分散共分散行列
> system.time({v = var(df)})
user system elapsed
106.169 0.688 106.949
>>> start = time(); v = df.cov(); print(time() - start)
3.9224328994750977
@time v = cov(Matrix(df)); # 6.487346 seconds
重回帰分析
> system.time({ans = lm(X1 ~ X2+X3+X4+X5+X6+X7+X8+X9+X10, data=df)
+ summary.ans = summary(ans)
+ coeff = ans$coeff
+ R2 = summary(ans)$r.squared})
user system elapsed
0.046 0.028 0.136
>>> from sklearn.linear_model import LinearRegression
>>> start = time();
>>> clf = LinearRegression()
>>> y = df['X1']
>>> x = df[['X2', 'X3', 'X4', 'X5', 'X6', 'X7', 'X8', 'X9', 'X10']]
>>> ans = clf.fit(x, y)
>>> coef = ans.coef_
>>> intercept = ans.intercept_
>>> print(time() - start)
0.031842947006225586
using GLM
@time begin
result = lm(@formula(X1 ~ X2+X3+X4+X5+X6+X7+X8+X9+X10), df)
coefs = coef(result);
R2 = r2(result)
end # 0.025926 seconds
独立変数の個数が非常に多い場合
> system.time({
+ ans = lm(X1 ~ ., data=df)
+ summary.ans = summary(ans)
+ coeff = ans$coeff
+ R2 = summary(ans)$r.squared})
user system elapsed
168.687 2.262 172.028
>>> start = time();
>>> y = df['X1']
>>> x = df.drop('X1', axis=1)
>>> ans = clf.fit(x, y)
>>> coef = ans.coef_
>>> intercept = ans.intercept_
>>> R2 = ans.score(x, y)
>>> print(time() - start)
11.533896923065186
using GLM
@time fit(LinearModel, hcat(ones(size(df, 1)), Matrix(select(df, Not(:X1)))), df.X1); # 5.237764 seconds
