Wednesday, June 11, 2014

Improving MNIST results

After getting the system set up, I set out to experiment with Theano by taking the included MLP that classifies MNIST digits and seing how much I can improve it by different means, without reading up anything else on Theano.

First, running mlp.py took 94 minutes. It trained a simple 784-500-10 MLP for 1000 epochs. The resulting error rate on the test set was %1.65 (on epoch 947). I figure a 10% improvement should count as a success, so my target was %1.50.

This is what I did:

  1. Used rectified linear units instead of tanh units. This already improved performance, at least in terms of convergence speed.
  2. For the fun of it, I added a few more layers. I created a 784-1000-400-150-10 network for this.
  3. Adaptive Learning Rate. Well technically not adaptive. I just made an ad-hoc schedule for the learning rate. Each epoch lr(t+1) = lr(t) - lr(t)^2 + 0.000002. I would have been nicer to just use 0.1 for first 10 epochs, 0.05 for the next 20, 0.01 for the next 300, 0.005 for the next 300 and 0.002 for the rest. Also would have been better to use rprop. In any case, having a high learning rate at the beginning really helped convergence speed a lot.
I had lots of ideas, but didn't want to spend too much time implementing, testing or tweaking. I'll list here some of the things I would have tried if I had the time.
  1. Rprop - to do away with that pesky learning rate
  2. Complex-valued Neural networks. This would be more for fun. I tried setting the Theano types to Complex, but the gradient doesn't really work on those. It would be easy to create a network that simulates complex values, but I didn't think it was worth it. I haven't heard anything good about Complex Neural Networks ever. Actually I haven't heard anything for that matter, so the jury is out on whether this would be useful.
  3. Adaptive Learning rate 
  4. Adding noise to the inputs
  5. Augmenting the dataset by stretching, translating, scaling etc 
  6. Tune batch size
  7. Tune Learning rate 
  8. Use DropConnect
  9. Tune regularization. I used an L2 regularizatio coefficient of 0.000001, 2 order s of magnitude smaller than the default, in order to accomodate the larger number of parameters. Also interesting when using no regularization.
  10. Adaptive regularization. Intuitively (read I have nothing to back this up), regularization parameter that increases over time should allow exploration early on and exploitation later.
  11. (Adaptive) Momentum
  12. Randomized batches. 
  13. Auto encoders for pretraining.

Running this model ( which can be found at https://github.com/lucianionita/dhlv/blob/master/src/my3mlp.py ) resulted in a test error of %1.43, obtained on epoch 117. After epoch 150, you could see the validation error going up, showing overfitting. I'm sure this result can be improved significantly by the methods I've listed above, and if someone tries, I'd be happy to hear about it. But for me, I think it's time I moved on to convolution neural networks. I'll be looking at the pseudo-LeNet that's included in the turorial to see if I can further improve my result. I'll be shooting for less than %1.00 error, but if I reach %1.20  I'll consider it a success.





Bonus notes:

1. Apparently the Nvidia Grid K520, which powers the cg2.x2large amazon instance is not the bees' knees after all. In benchmark testing, it fares below the GTX 560, GTX 660 and GTX 760. Something a little bit better, ending in 70 or 80 will blow the K520 out of the water.

2. Amazon also charges for EBS I/O requests, which seem  to add up. After 44 hours of cg2.x2large instances, my bill is about $7.00, out of which 1 cent is from EBS access (which exceed the free tier 2M requestions). 1M I/O requests cost $0.05. I'm not sure how that will add up over time, but if it looks prohibitive, I might move everything on to a ramdisk, the instance has 15GB of RAM anyway.

Sunday, June 8, 2014

Installing Theano on Ubuntu 12.04

Finally cracked the nut. I know there might be some things that can be improved, but right now if it all works, I'm fine with it. If you would like to propose improvements, please feel free to post them as long as they have been tested.

Here's the process:

Go to EC2 management console, request a new spot instance, go to Community AMI, type "Ubuntu 12.04" in the search bar and use this AMI:
ubuntu/images/hvm/ubuntu-precise-12.04-amd64-server-20140408 - ami-26e08b16
Be sure to get the HVM virtualization, not the paravirtual one. On the next page select the g2.xlarge instance. On the next, select Spot Instance. Right now the spot price is about $0.08, eight times cheaper than the On-Demand price. I usually chose a maximum price of $0.20 to be sure it doesn't terminate unexpectedly if the price goes up. That's about it for the configuration.
Once you can connect to the machine, get this script and run it:
https://raw.githubusercontent.com/lucianionita/dhlv/master/install.sh
The red headders that appear when installing are just to vaguely inform of the section of the script that is running. If the script doesn not suddenly exit, then all is well.
That's all. Now you should have a fully functioning environment with theano installed. The last part of the script runs logistic_sgd.py first on CPU then on GPU. Enjoy.

The next challenge on this road to learning deep learning is creating and training a model that performs better than the MLP included in the deeplearning.net code. 



Wednesday, June 4, 2014

Step 1: Instalation

Installing everything proved harder than I expected. I found 4 very good resources (links at the bottom) that I'm going to go through in order to find the best installation procedure. The point of this exercise is to have a fully functioning EC2 GPU-optimized instance that can run Theano with good CPU and GPU acceleration. Many little issues along the way slowed me down: cuda wants gcc 4.6 or older; the blas library from ubuntu is not well optimized; people are skipping some of their own steps when posting their solution; using the nvidia ppa for ubuntu seems to always complain about some unmet dependencies; nvidia ppa for ubuntu always wants to install cuda-6; cuda 6 didn't work so far for me with theano - this might change in the future.

Step 1 will be done when I can just launch an EC2 instance, run a script, then compare the logistic regression from deeplearning theano tutorial on CPU and GPU and I should have significantly faster training than the times listed in the tutorial. Also I aim for 80-100% CPU usage at least when running on CPU.

An interesting thing I found out: EC2 spot instances currently cost $0.082/hour. That's peanuts. In my opinion, it's a lot more effective than buying your own nvidia card, at least for the purpose of learning to use theano.

Until next time, keep your fingers crossed!




The links relevant to installation are:

http://www.deeplearning.net/software/theano/install_ubuntu.html#install-ubuntu
http://www.kurtsp.com/deep-learning-in-python-with-pylearn2-and-amazon-ec2.html
http://blog.sbos.in/post/37828521741/setting-up-gpu-cluster-instance-on-amazon-ec2-for-a
https://theano.readthedocs.org/en/rel-0.6rc3/install.html#gpu-linux

Friday, May 30, 2014

First post




What's this?

A blog about learning deep neural networks. I'll update the description as the purpose/scope gets more refined.

Right now the plan is for me to learn deep neural networks and share my findings so that other interested people would have an easier time than myself doing it.

I'm going to start with Amazon EC2 GPU instances running Ubuntu. Also, I'm going to have a bias towards python, so I'm starting with theano.