redes neuronales - introduciondonos en la inteligencia...

Redes NeuronalesIntroduciondonos en la Inteligencia Artificial

Xavi Barber

Centro de Investigación Operativa.Universidad Miguel Hernández de Elche

Xavi Barber (Cio-UMH) Redes Neuronales 1 / 48

1 Revisando Conceptos

2 Tipos de Redes Neuronales

3 Deep learning taxonomy

4 Implementando ANN y Deep Leraning con R


Revisando Conceptos

Revisando Conceptos


Revisando Conceptos

La importancia de las capas ocultas

Sunila Gullina. Practical Machine Learning. 2016


Tipos de Redes Neuronales




Jordan networksSon redes parciamente recurrentes. Estas redes se diferencias por la entradarecurrente en la primera capa oculta desde la última capa oculta.



Elman networksSon como el caso anterior, pero ahora reciben lainformación recurrente desde laprimera cpa oculta, pero como si de un nuevo input se tratase.



Radial Bias Function (RBF) networksEstas redes tienen un comportamiento especial, pues utilizan una distribucionsGausian (Normal) en las capas ocultas para el comportamiento de la “distancia”entre el valor “input” y el “centro”.



Hopfield networks

La capa oculta le aporta información a otra neurona buscando el equilibrio de la red.



Métodos de propagación

Backpropagation algorithm (Gradient descent algorithm)Softmax regression technique (multinomial logistic regression)


Deep learning taxonomy




Separando información con Rendes Neuronales

En la acualidad existen diferentes lenguajes o programas para realizar este tipo detaxonomia:

Theano (Python library)Torch (un lenguaje de progtramación en Lua)Deeplearning4J (open source Java, que funciona en Spark y Hadoop)Caffe (programado en C++)



Convolutional neural networks (CNN/ConvNets)

Como se puede observar, las redes convolucionales se convierten en 3D.



Recurrent Neural Networks (RNNs)



Deep Boltzmann Machines (DBMs)DBMs restringen las conexiones entre las variables ocultas y las variables “input”utilizando datos sin etiquetar para entrenar al modelo. Los datos etiquetados seusan para “reajustar” y “afinar” el modelo.



AutoencodersEs un tipo de Red Neuronal que tíne una única capa. Aplica “backpropagation” yaprendizaje no-supervisado. Los autoencoders empiezan con una aunción que elvalor de la variable respuesta es el mismo que el valor de la variable “input”, esdecir, x = y .


Implementando ANN y Deep Leraning con R




Indicaciones previas

No todos los comandos que aquí se muestran funcionan sin la instalaciónprevia de las herramientas necesiarias.Lo que aquí se va a mostrar es simplemente a modo de ejemplo para que elalumno pueda visualizar la potencia de el deep Learning.Para ampliar conocimeintos recomiendo la lectura del libro:

R Deep Learning Essentials Joshua F. Wiley Copyright © 2016 PacktPublishing BN 978-1-78528-058-0



R packages for deep learning

Paquetes Generales:



Paquetes para ANN y Deep Learning## neural networkslibrary(nnet)library(neuralnet)library(RSNNS) # The Stuttgart Neural Network Simulator (SNNS)

## deep learninglibrary(deepnet)# Implement some deep learning architectures and neural network algorithms,# including BP,RBM,DBN,Deep autoencoder and so on.

library(darch)# This package is for generating neural networks with many layers (deep# architectures) and train them with the method introduced by the# publications 'A fast learning algorithm for deep belief nets'



library(h2o)# R scripting functionality for H2O, the open source math engine for big# data that computes parallel distributed machine learning algorithms such# as generalized linear models, gradient boosting machines, random forests,# and neural networks (deep learning) within various cluster environments.

# Se necesita tenerr instalado un JDK de JAVA, por ejemplo JAVA SE# Development Kit. http://www.oracle.com/technetwork/java/javase/downloads

# ----------------------------------------------------------------------# Your next step is to start H2O and get a connection object (named# 'localH2O', for example): > localH2O = h2o.init() For H2O package# documentation, ask for help: > ??h2o After starting H2O, you can use the# Web UI at http://localhost:54321 For more information visit# http://docs.0xdata.com# ----------------------------------------------------------------------



http://localhost:54321


http://localhost:54321


Paquetes para modelizar## Paquetes para optimizar tiempo de cálculo

library(parallel)# It builds on the work done for CRAN packages multicore (Urbanek,# 2009–present) and snow (Tierney et al., 2003–present) and provides drop-# in replacements for most of the functionality of those packages, with# integrated handling of random-number generation.

library(foreach)# Support for the foreach looping construct. Foreach is an idiom that allows# for iterating over elements in a collection, without the use of an# explicit loop counter.

library(doSNOW)# Provides a parallel backend for the %dopar% function using Luke Tierney's# snow package.



Paquetes para modelizar

library(glmnet)# Extremely efficient procedures for fitting the entire lasso or elastic-net# regularization path for linear regression, logistic and multinomial# regression models, Poisson regression and the Cox model. Two recent# additions are the multiple-response Gaussian, and the grouped multinomial.# The algorithm uses cyclical coordinate descent in a path-wise fashion

library(data.table)# Fast aggregation of large data (e.g. 100GB in RAM), fast ordered joins,# fast add/modify/delete of columns by group using no copies at all, list# columns and a fast file reader (fread). Offers a natural and flexible# syntax, for faster development.

library(gridExtra)# Provides a number of user-level functions to work with 'grid' graphics,# notably to arrange multiple grid-based plots on a page, and draw tables.

library(mgcv)# GAMs, GAMMs and other generalized ridge regression with multiple smoothing# parameter estimation by GCV, REML or UBRE/AIC. Includes a gam() function,# a wide variety of smoothers, JAGS support and distributions beyond the# exponential family.



Training an auto-encoder in R: H2O

source("checkpoint.R")options(width = 70, digits = 2)## data and H2O setupdigits.train <- read.csv("train.csv")digits.train$label <- factor(digits.train$label, levels = 0:9)cl <- h2o.init(max_mem_size = "20G", nthreads = 10)h2odigits <- as.h2o(digits.train, destination_frame = "h2odigits")i <- 1:20000h2odigits.train <- h2odigits[i, -1]itest <- 20001:30000h2odigits.test <- h2odigits[itest, -1]xnames <- colnames(h2odigits.train)



Para realizar el análisis vamos a utilizar h2o.deeplearning().Utilizaremos computción en Paralelo (see Hogwild!: A Lock-Free Approach toParallelizing Stochastic Gradient Descent by Niu, F., Recht, B., Ré, C., and Wright,S. J. (2011) athttps://www.eecs.berkeley.edu/~brecht/papers/hogwildTR.pdf).


https://www.eecs.berkeley.edu/~brecht/papers/hogwildTR.pdf


m1 <- h2o.deeplearning(x = xnames, training_frame = h2odigits.train, validation_frame = h2odigits.test,activation = "Tanh", autoencoder = TRUE, hidden = c(50), epochs = 20, sparsity_beta = 0,input_dropout_ratio = 0, hidden_dropout_ratios = c(0), l1 = 0, l2 = 0)



# m1 Training Set Metrics: ===================== MSE: (Extract with# 'h2o.mse') 0.014 H2OAutoEncoderMetrics: deeplearning ** Reported on# validation data. ** Validation Set Metrics: ===================== MSE:# (Extract with 'h2o.mse') 0.014

error1 <- as.data.frame(h2o.anomaly(m1, h2odigits.train))



Proponiendo diferentes modelos y viendo el error que cometemos:



Estudio de Casos: Información de HeartRate

Veamos un ejemplo de los datos que los móviles recogen sobre la actividad físisca de sus usuariuosy el estado de su corazón.Los datos incluyen “actimetría” de un gran número de usuarios: sentados, de pie, acostado,caminando, subir y bajar escalones. Nuestro objetivo es identificar valores anómalos o valores quepueden ser “aberrantes” o extrañamente inusuales; con el objetivo de mejorar el comportamientodel acelerómetro y la aplicación HeartRate.

use.train.x <- read.table("UCI HAR Dataset/train/X_train.txt")use.test.x <- read.table("UCI HAR Dataset/test/X_test.txt")use.train.y <- read.table("UCI HAR Dataset/train/y_train.txt")[[1]]use.test.y <- read.table("UCI HAR Dataset/test/y_test.txt")[[1]]

use.labels <- read.table("UCI HAR Dataset/activity_labels.txt")

h2oactivity.train <- as.h2o(use.train.x, destination_frame = "h2oactivitytrain")

h2oactivity.test <- as.h2o(use.test.x, destination_frame = "h2oactivitytest")



El modelo

mu1 <- h2o.deeplearning(x = colnames(h2oactivity.train), training_frame = h2oactivity.train,validation_frame = h2oactivity.test, activation = "Tanh", autoencoder = TRUE,hidden = c(100, 100), epochs = 30, sparsity_beta = 0, input_dropout_ratio = 0,hidden_dropout_ratios = c(0, 0), l1 = 0, l2 = 0)



No existe una gran diferncia entre el modelo ajustado y el modelos con los datos devalidación:

# mu1 Training Set Metrics: ===================== MSE: (Extract with# `h2o.mse`) 0.001 H2OAutoEncoderMetrics: deeplearning ** Reported on# validation data. ** Validation Set Metrics: ===================== MSE:# (Extract with `h2o.mse`) 0.0011



erroru1 <- as.data.frame(h2o.anomaly(mu1, h2oactivity.train))pue1 <- ggplot(erroru1, aes(Reconstruction.MSE)) + geom_histogram(binwidth = 0.001,

fill = "grey50") + geom_vline(xintercept = quantile(erroru1[[1]], probs = 0.99),linetype = 2) + theme_bw()

print(pue1)



Y ahora veamos el resultado:



training a deep neural network for automatic classification

download.file("http://archive.ics.uci.edu/ml/machine-learningdatabases/\n00203/YearPredictionMSD.txt.zip",destfile = "YearPredictionMSD.txt.zip")

unzip("YearPredictionMSD.txt.zip")

# Now we can read data into R using fread() from the data.table package. The# fread() function is preferable to read.csv() here because it can be# orders-of-magnitude faster,and it still took 30 seconds on a high-end# desktop with a solid state hard drive:

d <- fread("YearPredictionMSD.txt", sep = ",")

# First we can take a quick look at the distribution of the outcome, the# year of release.

p.hist <- ggplot(d[, .(V1)], aes(V1)) + geom_histogram(binwidth = 1) + theme_classic() +xlab("Year of Release")

print(p.hist)



quantile(d$V1, probs = c(0.005, 0.995))# 0.5% 100% 1957 2010



d.train <- d[1:463715][V1 >= 1957 & V1 <= 2010]d.test <- d[463716:515345][V1 >= 1957 & V1 <= 2010]

h2omsd.train <- as.h2o(d.train, destination_frame = "h2omsdtrain")

h2omsd.test <- as.h2o(d.test, destination_frame = "h2omsdtest")

summary(m0 <- lm(V1 ~ ., data = d.train))$r.squared# [1] 0.24

cor(d.test$V1, predict(m0, newdata = d.test))^2# [1] 0.23

Obteniéndose unos resultados relativamente pobres para la regresion lineal.



Hagamos pues una red neuronal H2O.

# m1 <- h2o.deeplearning(x = colnames(d)[-1], y = 'V1',training_frame=# h2omsd.train, validation_frame = h2omsd.test,activation =# 'RectifierWithDropout', hidden = c(50), epochs = 100,input_dropout_ratio =# 0, hidden_dropout_ratios = c(0), score_training_samples = 0,# score_validation_samples = 0, diagnostics = TRUE,# export_weights_and_biases = TRUE, variable_importances = TRUE )



# m1# Model Details:# ==============# H2ORegressionModel: deeplearning# Model ID: DeepLearning_model_R_1451972322936_5# Status of Neuron Layers: predicting V1, regression, gaussian distribution,# Quadratic loss, 4,601 weights/biases, 72.5 KB, 13,702,476 training samples,# mini-batch size 1# layer units type dropout l1 l2 mean_rate# 1 1 90 Input 0.00 %# 2 2 50 RectifierDropout 0.00 % 0.000000 0.000000 0.009403# 3 3 1 Linear 0.000000 0.000000 0.000218# rate_RMS momentum mean_weight weight_RMS mean_bias bias_RMS# 1# 2 0.007939 0.000000 -0.018219 0.598229 -2.199141 2.245173# 3 0.000202 0.000000 -0.042807 0.103305 -0.767868 0.000000# H2ORegressionMetrics: deeplearning# ** Reported on training data. **# Description: Metrics reported on full training frame# MSE: 76# R2 : 0.37# Mean Residual Deviance : 76# H2ORegressionMetrics: deeplearning# ** Reported on validation data. **# Description: Metrics reported on temporary (load-balanced) validation frame# MSE: 80# R2 : 0.32# Mean Residual Deviance : 80



El R2 no ha mejorado mucho, deberemos mejorar el modelo de la Red que hemosutilizando, para ello

# m2 <- h2o.deeplearning( x = colnames(d)[-1], y = 'V1', training_frame=# h2omsd.train, validation_frame = h2omsd.test, activation =# 'RectifierWithDropout', # tres capas ocultas con esas neuronas hidden =# c(200, 200, 400), epochs = 100, input_dropout_ratio = 0,# hidden_dropout_ratios = c(.2, .2, .2), score_training_samples = 0,# score_validation_samples = 0, diagnostics = TRUE,# export_weights_and_biases = TRUE, variable_importances = TRUE )



# m2# Model Details:# ==============# H2ORegressionModel: deeplearning# Model ID: DeepLearning_model_R_1452031055473_5# Status of Neuron Layers: predicting V1, regression, gaussian distribution,# Quadratic loss, 139,201 weights/biases, 1.6 MB, 22,695,351 training# samples, mini-batch size 1# layer units type dropout l1 l2 mean_rate# 1 1 90 Input 0.00 %# 2 2 200 RectifierDropout 20.00 % 0.000000 0.000000 0.011513# 3 3 200 RectifierDropout 20.00 % 0.000000 0.000000 0.014861# 4 4 400 RectifierDropout 20.00 % 0.000000 0.000000 0.054338# 5 5 1 Linear 0.000000 0.000000 0.001258# rate_RMS momentum mean_weight weight_RMS mean_bias bias_RMS# 1# 2 0.004978 0.000000 0.000848 0.207373 -0.254659 0.321144# 3 0.012359 0.000000 -0.032566 0.104347 1.017329 0.341556# 4 0.036596 0.000000 -0.031768 0.072171 0.651546 0.292565# 5 0.000505 0.000000 0.001421 0.020867 -0.596303 0.000000# H2ORegressionMetrics: deeplearning# ** Reported on training data. **# Description: Metrics reported on full training frame# MSE: 66# R2 : 0.40# Mean Residual Deviance : 66# H2ORegressionMetrics: deeplearning# ** Reported on validation data. **# Description: Metrics reported on temporary (load-balanced) validation frame# MSE: 70# R2 : 0.35# Mean Residual Deviance : 70Xavi Barber (Cio-UMH) Redes Neuronales 44 / 48


Podemos probar otros ajustes con más neuronas en las capas ocultas:m3 con > hidden = c(500, 500, 1000),m2b con > checkpoint = “DeepLearning_model_R_1452031055473_5”“,Obteniendo un R2 para el modelo m3 de 0.28 y para el modelo m2b de 0.43



Los errores de predicción del modelo m1 los podemos observar:

yhat <- as.data.frame(h2o.predict(m1, h2omsd.train))yhat <- cbind(as.data.frame(h2omsd.train[["V1"]]), yhat)

p.resid <- ggplot(yhat, aes(factor(V1), predict - V1)) +geom_boxplot() +geom_hline(yintercept = 0) +theme_classic() +theme(axis.text.x = element_text(

angle = 90, vjust = 0.5, hjust = 0)) +xlab("Year of Release") +ylab("Predicted Year of Release")

print(p.resid)



Como conclusión, desde R y utilizando las “conexiones adecuadas”, se puedenmover grandes conjuntos de datos y obtener análisis de los datos. Siempre dentrode lo que nos permita el servidor dónde se albergue la información.


redes neuronales - introduciondonos en la inteligencia...

Documents