First, running mlp.py took 94 minutes. It trained a simple 784-500-10 MLP for 1000 epochs. The resulting error rate on the test set was %1.65 (on epoch 947). I figure a 10% improvement should count as a success, so my target was %1.50.
This is what I did:
- Used rectified linear units instead of tanh units. This already improved performance, at least in terms of convergence speed.
- For the fun of it, I added a few more layers. I created a 784-1000-400-150-10 network for this.
- Adaptive Learning Rate. Well technically not adaptive. I just made an ad-hoc schedule for the learning rate. Each epoch lr(t+1) = lr(t) - lr(t)^2 + 0.000002. I would have been nicer to just use 0.1 for first 10 epochs, 0.05 for the next 20, 0.01 for the next 300, 0.005 for the next 300 and 0.002 for the rest. Also would have been better to use rprop. In any case, having a high learning rate at the beginning really helped convergence speed a lot.
I had lots of ideas, but didn't want to spend too much time implementing, testing or tweaking. I'll list here some of the things I would have tried if I had the time.
- Rprop - to do away with that pesky learning rate
- Complex-valued Neural networks. This would be more for fun. I tried setting the Theano types to Complex, but the gradient doesn't really work on those. It would be easy to create a network that simulates complex values, but I didn't think it was worth it. I haven't heard anything good about Complex Neural Networks ever. Actually I haven't heard anything for that matter, so the jury is out on whether this would be useful.
- Adaptive Learning rate
- Adding noise to the inputs
- Augmenting the dataset by stretching, translating, scaling etc
- Tune batch size
- Tune Learning rate
- Use DropConnect
- Tune regularization. I used an L2 regularizatio coefficient of 0.000001, 2 order s of magnitude smaller than the default, in order to accomodate the larger number of parameters. Also interesting when using no regularization.
- Adaptive regularization. Intuitively (read I have nothing to back this up), regularization parameter that increases over time should allow exploration early on and exploitation later.
- (Adaptive) Momentum
- Randomized batches.
- Auto encoders for pretraining.
Running this model ( which can be found at https://github.com/lucianionita/dhlv/blob/master/src/my3mlp.py ) resulted in a test error of %1.43, obtained on epoch 117. After epoch 150, you could see the validation error going up, showing overfitting. I'm sure this result can be improved significantly by the methods I've listed above, and if someone tries, I'd be happy to hear about it. But for me, I think it's time I moved on to convolution neural networks. I'll be looking at the pseudo-LeNet that's included in the turorial to see if I can further improve my result. I'll be shooting for less than %1.00 error, but if I reach %1.20 I'll consider it a success.
Bonus notes:
1. Apparently the Nvidia Grid K520, which powers the cg2.x2large amazon instance is not the bees' knees after all. In benchmark testing, it fares below the GTX 560, GTX 660 and GTX 760. Something a little bit better, ending in 70 or 80 will blow the K520 out of the water.
2. Amazon also charges for EBS I/O requests, which seem to add up. After 44 hours of cg2.x2large instances, my bill is about $7.00, out of which 1 cent is from EBS access (which exceed the free tier 2M requestions). 1M I/O requests cost $0.05. I'm not sure how that will add up over time, but if it looks prohibitive, I might move everything on to a ramdisk, the instance has 15GB of RAM anyway.
No comments:
Post a Comment