Download RSS Media Files using Python

Today I wanted to download a course from an RSS feed. The media files (mp3) were attached to the RSS entries. So instead of downloading tens of files one by one, I wrote a script!

With the help of Alvin's post to parse RSS with Python, I attached a download method and the script was ready to do the work for me.

First, here is the script:
[rss_downloader.py]

#!/usr/bin/python

import feedparser
import sys
import urllib2

#
# Takes a url and a directory for saving the file. Directory must exist.
#
def download(url, dir_name):
    file_name = url.split('/')[-1]
    u = urllib2.urlopen(url)
    f = open(dir_name+'/'+file_name, 'wb')
    meta = u.info()
    file_size = int(meta.getheaders("Content-Length")[0])
    print "Downloading File: %s (Size: %s Bytes)" % (file_name, file_size)

    file_size_dl = 0
    block_sz = 8192
    while True:
        buffer = u.read(block_sz)
        if not buffer:
            break

        file_size_dl += len(buffer)
        f.write(buffer)
        status = r"%10d  [%3.2f%%]" % (file_size_dl, file_size_dl * 100. / file_size)
        status = status + chr(8)*(len(status)+1)
        print status,

    f.close()

#
# Take url and directory parameters from user call
#
url = sys.argv[1]
dir_name = sys.argv[2]

#
# Get the feed data from the url
#
feed = feedparser.parse(url)

#
# Collect urls to download
#
urls_to_download = []
for entry in feed.entries:
    links = entry.links
    for link in links:
        if link.type == u'audio/mpeg':
            urls_to_download.append(link.href)

print("Files count: %s" % (len(urls_to_download)))

#
# Download files
#
for url in urls_to_download:
    download(url, dir_name)
    # print(url)

You just call the script, pass RSS url, and the directory to save the files.

python rss_downloader.py http://rss.dw.com/xml/DKpodcast_dwn1_en /home/madly/DeutschWarumNicht/serie1

This will parse the RSS feed, print the available links with type "audio/mpeg" (you can change this to be a value passed by the user), and download them with a progress display

This it. But there is more...

After doing this, I realized that I could simply print the output, copy the links all together to my download manager as a batch download, and enjoy the features of my download manager. After all, I wanted to download the files, not make a full application. Dummy me :D

However you choose to go with the script, I hope you find it useful.

CSE 321 Project - Part 2 - Install Ruby, Rails, PostgreSQL, MySQL and others

This is one post in a series of posts related to CSE 321 - Software Engineering project for ASU CSE 2017 students. The series starts from setting up the environment, to planning the work on the website project, to implementation

After Installing Lubuntu/Ubuntu in a VM in part 1, we want to prepare it for development.

Thinking about the required project, this is what I want to install.

• Ruby
• Rails
• PostgresSQL (or MySQL)
• Node.js
• Git
• Gitg (a graphical tool to use with Git)
• Atom editor (or you can install Sublime if you like)

Plus enabling copy & paste between the VM and the computer, and enabling flexible display resolution to suit your taste. And I will start with this.

Install Guest Additions

Now we need to install some additional packages to add more power to our VM. This will make our job easier in scaling the VM window (and keeping the aspect ratio), and enables copy&paste between the VM and the host computer to easily follow the next instructions.

If you have a toolbar in the outer VM window, go to Devices -> Insert Guest Additions CD image.... If you cannot find the toolbar, press Right Ctrl + Home buttons and the menu should appear. (if you still fail, press Right Ctrl + C and search for the toolbar again)

Once you do this, a virtual CD will be mounted (inserted) into the machine to install some additional packages.

In case of Ubuntu, the process should start immediately once you click Run in the box that will appear.

In case of Lubuntu, you will have something like this:

Click OK. Then run autorun.sh in the window that will appear (this is the content of the virtual CD).

Then choose Execute.

Authorize the script to have additional privileges by entering your password and pressing OK.

The script will run for a few seconds, installing some packages. Once finished you will have to press Enter to exit.

Extra: In case of Lubuntu, you need to install another package. Open the terminal by pressing Ctrl + Alt + T or clicking the start menu -> System Tools -> XTerm. Then type (you cannot copy at this point):

sudo apt-get install --yes virtualbox-guest-dkms

Then enter your password when it asks for it (it will download about 60MB).

Now shutdown the machine, go to this VM Settings -> General -> Advanced tab, and set Shared Clipboard to Bidirectional. This way you can copy instructions from your host OS to the VM terminal, and copy any error message you face from the VM terminal to -for example- your browser in the host OS.

Now let's start the VM again.

This time, try to copy any text (for example, a url) from your host OS. Then search for Firefox browser in the VM and paste it. It works!

One more thing has happened. If you resize the VM window, it will scale and keeps the aspect ratio. If you find the screen is getting 'compressed' and losing the correct ratio, press Right Ctrl + C.

Now we have a clean and helpful environment to work with.
Let's start with the development installations. Open the terminal and get ready!

Install Ruby

(references used: DigitalOcean, GoRails)

First there are some packages to install before Ruby.

Update out package list

sudo apt-get update

Install git, curl, and some other packages for compilation (~90MB)

sudo apt-get install git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev python-software-properties libffi-dev

Now install rbenv (the tool for manging Ruby versions, better than installing Ruby directly)

cd
git clone git://github.com/sstephenson/rbenv.git .rbenv
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(rbenv init -)"' >> ~/.bashrc
git clone git://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build
echo 'export PATH="$HOME/.rbenv/plugins/ruby-build/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

Next, install Ruby and set the installed version to be the default. (this will take a while to download and compile)

rbenv install -v 2.2.3
rbenv global 2.2.3

Now if you type

ruby -v

you will see the default ruby version.

Let's add one more step, disable documentation installation. We do not need them.

echo "gem: --no-document" > ~/.gemrc

Install Bundler Gem

This is the first gem to install. It is responsible for handling later gems in Rails projects.

gem install bundler

Install Rails

gem install rails -v 4.2.5

When it finishes, if you type

rails -v

you will see the default rails version.

Install Node.js

(reference: https://github.com/nodesource/distributions)
Node.js is a platform for development using Javascript. Rails uses Node.js in a certain step when joining multiple Javascript files together for better performance. You may not need this at first, but let's setup the environment the right way for future use and future reference.

curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -
sudo apt-get install -y nodejs

Now type

node -v

to check the installed version.

Install PostgreSQL

PostgreSQL is the biggest competitor to MySQL database. It is getting more popular, so it is nice to stay up-to-date and give it a try. But don't be scared. at first, it will not matter if you use MySQL or PostgreSQL because most of your interactions will be through Rails. (You can skip it and install MySQL)

sudo sh -c "echo 'deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main' > /etc/apt/sources.list.d/pgdg.list"
wget --quiet -O - http://apt.postgresql.org/pub/repos/apt/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
sudo apt-get install -y postgresql-common
sudo apt-get install -y postgresql-9.4 libpq-dev

Create PostgreSQL User

The last step is to create a new db user for later.

sudo -u postgres createuser cse -s

If you would like to set a password for the user, you can do the following

sudo -u postgres psql
postgres=# \password cse 

then enter the password you like.

Install MySQL

Use this command to install MySQL server and client.

sudo apt-get install mysql-server mysql-client libmysqlclient-dev

When prompted, you can choose to enter a password for the root user of MySQL or leave it blank. Whatever you choose, then use the down arrow to go to OK line and hit enter.

The root user has access to all databases. It is not safe to use it in production environment. It is better to create a new user for each project and give access to project's database only. But since this is a development environment, we can use root user.

Install Atom Editor

Now we need a nice editor to help us with developmen. Recently, I switched from Sublime Text to Atom Editor. So I will continue with using Atom. Feel free to use whatever you want.
You can install Atom by downloading the installer (*.deb) from their home page, or install it from the terminal. To avoid explaining how to share folders between VM and host, I will install it from the terminal. You can follow my steps or download the installer from the VM browser.

sudo add-apt-repository ppa:webupd8team/atom
sudo apt-get update
sudo apt-get install atom

Install Gitg

The last optional step is to install Gitg. It is a graphical interface to use Git. It make things a bit easier when it comes to committing changes, reviewing history, and other operations you may need.

sudo apt-get install gitg

That's all folks! Your machine is ready. You can compress the machine and share it with others or keep it as a backup as a ready-to-use machine.

CSE 321 Project - Part 1 - Install Lubuntu on VirtualBox

This is one post in a series of posts related to CSE 321 - Software Engineering project for ASU CSE 2017 students. The series starts from setting up the environment, to planning the work on the website project, to implementation

In this tutorial, we will start a virtual machine and install Lubuntu or Ubuntu.

Install VirtualBox

The first thing is downloading and installing VirtualBox:
https://www.virtualbox.org/wiki/Downloads

And assuming that you are working on Linux, then you'll go for the first link. The download link is for both 32-bit and 64-bit computers.
(note: x86 means 32-bit, amd64 means 64-bit)

Create a New Lubuntu Machine

Let's create a new machine by clicking the New icon.

Enter a suitable name for your machine. Any name to remind you what this machine does.

Select the amount of memory this machine is allowed to use. If you have plenty of RAM, give it 2GB. No worries, this value can be changed later.

Leave the default option to create a new virtual hard disk.

Leave the VDI option as it is.

Leave the dynamic size as it is. This option allows the virtual hard disk (the file created) to start small then expand later to a max limit. The other option will create the full sized file from the beginning.

Choose a different name for the virtual hard disk or leave it with the same name as the virtual machine name. And you can leave the size at it is for now. Actually 8GB is more than enough.

Now the VM is ready.

Install Lubuntu

Before starting, let's mount the Lubuntu iso image as a CD to use it for installation.

Select the machine and click Settings.

Go to Storage, and select the empty CD.

Click on the small CD icon on the far right. And choose Choose Virtual Optical Disk file.
Then browse your folders and select the iso file you downloaded (Ubuntu iso , Lubuntu iso) then click Open.

Click OK. When done.

Now we are ready to start the machine. Select the machine and click Start or double-click it.

Choose English.

Choose Install Lubuntu.

Wait for a few seconds until it boots the installer.

Choose English.

Click Continue.

Now for the critical part of formatting if it was a real installation. Leave the first option as it is. After all, this is a virtual disk created for Lubuntu. Not your actual disk. Click Install Now. Then click Continue.

Choose the city of Cairo or any other city you prefer, and click Continue.

Leave the keyboard layout as it is and press Continue.

Enter your name and choose the username and password. Click Continue.

Now leave it for a few minutes to install and pull only some basic data from the Internet.

When finished, press Restart Now.

The machine will eject the virtual CD it used for installation and asks you to press Enter. Do it.

The fresh Lubuntu OS will boot.

The login dialog will appear. Enter your password.

Welcome to your new machine. Now you can shut it down and start it again whenever you want.

That's all folk! :)

In the next tutorial, we will work on installing:

• Ruby
• Rails
• Node.js
• PostgresSQL
• Atom Editor
• Git
• Gitg (a graphical tool to use with Git)
• Enable copy and paste between the VM and the computer, and enable flexible display resolution.

Take your time and try and fail without worries. Virtual machines are created and deleted all the time. If you mess it up, delete it and try again.

[update: Part 2 is available now]

Custom 404 Page in Slim Framework 3

In the past month, I've been learning Slim 3 PHP framework through a website as a real experiment of how powerful Slim Framework is. In this post, I'm sharing a code snippet for rendering 404 page.

Between creating a Slim Container and creating the Slim App, I override the default notFoundHandler.

$container = new Slim\Container;

...

// Override the default Slim Not Found Handler
$container['notFoundHandler'] = function ($c) {
    return function ($request, $response) use ($c) {
        return $c['view']->render($response, '404.html', [])->withStatus(404);
    };
};

...

// Initialize the app
$app = new Slim\App($container);

In my case, I use Twig template engine for views. If your case is different, change the render line.

File System Caching in Slim Framework 3

In the past month, I've been learning Slim 3 PHP framework through a website as a real experiment of how powerful Slim Framework is. In this post, I'm sharing a code snippet of how I implemented the Filesystem caching.

The summary is:
- Add a middleware for caching successful responses (200 OK) by saving the response in a file. To avoid long file names or deep paths, I hash the response URL.
- When receiving a request, check its hashed path in the cached files. If found, return the content. Else, continue with initializing the app, routes, and everything else. This way, there is no initialization overhead for cached requests.

Here is the beginning of my index.php:

require './lib/initializer.php';

function getCachePath($uri) {
    $webpage_handle = md5($uri);
    return ROOT."/public/cache/".$webpage_handle;
}

/* Check for cached file here before loading anything or opening DB */
if(CACHING_ENABLED) {
    $requestedURI = "http://$_SERVER[HTTP_HOST]$_SERVER[REQUEST_URI]";
    $cachedFilePath = getCachePath($requestedURI);
    if (file_exists($cachedFilePath) && (filemtime($cachedFilePath) > (time() - CACHE_SECONDS ))) {
        $cacheFile = file_get_contents($cachedFilePath);
        die($cacheFile);
    }
}

$cacheMiddleware = function ($request, $response, $next) {
    /* process the request */
    $response = $next($request, $response);

    /* cache response */
    if(CACHING_ENABLED && ($response->getStatusCode()==200) && ($request->getMethod() == 'GET')) {
        $cachedFilePath = getCachePath($request->getUri());
        $file = fopen($cachedFilePath,"w");
        fwrite($file, $response->getBody());
        fclose($file);
    }

    return $response;
};

First, I require a file where I initialize some configs and constants, like CACHE_SECONDS.

Then, I create a function for deciding the cache path for requests. To use it in both reads and writes.

Then, -if caching is enabled- I check for the existence of the requested file and the creation date. This way I override the file if it was older. (*check the end of the post for another better way for expiry)

Next, you can see the caching middleware I use with some routes I wish to cache. Here is an example:

$app->get('/', function ($request, $response, $args) {
    // some app-related code
})->setName('home')->add($cacheMiddleware);

And that's it. The file system is now implemented with minimal code. Of course there are many changes and upgrades depending on your case.

----------------------------------
Notes:
*Better than just checking for file creation date (and leaving the files to increase), you can remove the date-check part and add a cron job for removing files older than a specific duration. Here is how my cron job script looks like:

CACHE_DIR=$1;
cd $CACHE_DIR && find * -type f -cmin +5 -exec rm {} \;

It takes the path to clean in the cron job command. And removes any file older than 5 minutes.

The cron job looks like:

*/5 * * * * bash ~/path/to/cache/expiry/script.sh ~/path/to/cache/folder/ 

Arabic-friendly Exeption reporting in Slim Framework 2 & 3

Today I've been upgrading my Slim PHP project from version 2 to version to version 3 beta. And one of the issues I had was reporting my Arabic exceptions from some models I have. I did this before in version 2, to end up with moving from this scrambled message:

to this one:

In Slim Framework 2, I had to edit Middleware / PrettyExceptions.php to add a meta tag for content UTF-8 encoding:

<meta http-equiv='Content-Type' content='text/html; charset=utf-8'>

Now, when I moved to Slim Framework 3.0.0-beta2, I had to do this again but in Handlers/Error.php.

I hope that this trick helps someone on the planet. And I think I may create a pull request for this small hack.

Update: pull request has been merged. (#1470)

[Node JS] Test specific files with Mocha

This is a handy command line I use for testing specific files while developing a certain module for a faster test. I just call Mocha, (optionally) specify my preferred terminal reporter, then I call the files to go through:

mocha --reporter spec ./test/modules/abc.js ./test/modules/def.js

That's it. It is that simple. This way I can run tests on specific files and save some time.

How to detect jQuery mmenu close event

A few days ago, I was trying jQuery mmenu. I had a close button (called #mmenu-btn) that changed shape between open and close states and I wanted to make sure that it will never get stuck in a state different than the desired one.

    

After some reading in the mmenu docs about events and configurations, and after playing a lot with the button and mmenu on desktop and mobile, I had two states to cover:
1- An event is triggered by code (mmenu API): In this case, it is my code, so I can handle it.
2- The user clicks/touches the part outside the mmenu. In this case, the mmenu does not tell me that it is closing now.

For the second case, I found a nice solution that depends on how mmenu works. When mmenu opens, there is a created overlay called #mm-blocker that covers the part outside the menu. When this overlay is clicked, the mmenu is closed. So what I did was to bind the click event (and other similar events) of this #mm-blocker and do whatever I want to change the state of my close button.

// if user clicks outside the mmenu, change the state of mmenu-btn
$('#mm-blocker').on('click mousedown touchstart', function() {
  $('#mmenu-btn').removeClass('close');
});

This was my little trick. I hope it helps.

Nginx + Slim Framework + phpMyAdmin on Ubuntu

Yesterday I was giving Slim Framework a test ride on my Ubuntu machine. But the machine was not ready with Nginx, or phpMyAdmin (I needed it to check the output of an ORM called RedBean). So I'm writing what I did exactly for my reference and for other's benefit.

First, in case you don't have any of the following packages, go ahead and do it

Installations:

Install MySQL

sudo apt-get install mysql-server php5-mysql

and remember your root password you will enter in the installation process.

Install Nginx

sudo apt-get install nginx

Install PHP-FPM (PHP FastCGI Process Manager)

sudo apt-get install php5-fpm

Install phpMyAdmin

sudo apt-get install phpmyadmin

During installation, you will be asked to choose your web server (apache2/lighttpd). But since we will use Nginx, just press the TAB key then Enter. Then you will be asked if you want to proceed with phpMyAdmin database configurations, choose yes and continue then enter your MySQL root password when asked to.

Configurations:

FPM

sudo nano /etc/php5/fpm/php.ini

or as I prefer

sudo gedit /etc/php5/fpm/php.ini

Find the line, cgi.fix_pathinfo=1. Uncomment it and change the 1 to 0.

Then

sudo gedit /etc/php5/fpm/pool.d/www.conf

and find the value given to listen =, to be used later in Nginx configuration.
It should be /var/run/php5-fpm.sock or 127.0.0.1:9000. 

Now restart fpm

sudo service php5-fpm restart

Nginx

Now let's configure Nginx for both phpMyAdmin and Slim Framework project.

sudo gedit /etc/nginx/sites-enabled/default

And add the configurations for both phpMyAdmin and the Slim Framework project.

server {
    listen 80;
    server_name phpmyadmin.local;
    root /usr/share/phpmyadmin;
    index index.php;
    location / {
        try_files $uri $uri/ /index.html;
    }
    # pass the PHP scripts to FastCGI server listening on the php-fpm socket
    location ~ \.php$ {
        try_files $uri =404;
        include fastcgi_params;
        fastcgi_pass unix:/var/run/php5-fpm.sock;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;       
    }
}

server {
    listen 80;
    server_name slimtest.local;
    root /home/madly/slim;
    try_files $uri /index.php;
    # pass the PHP scripts to FastCGI server listening on the php-fpm socket
    location /index.php {
        fastcgi_connect_timeout 3s;     # default of 60s is just too long
        fastcgi_read_timeout 10s;       # default of 60s is just too long
        include fastcgi_params;
        fastcgi_pass unix:/var/run/php5-fpm.sock;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;      
    }
}

Now let's restart Nginx server to apply the changes

sudo service nginx restart

Notes on these server configs:

1- The fastcgi_pass value is set to unix:/var/run/php5-fpm.sock because of the value I mentioned earlier in fpm configs. It may be 127.0.0.1:9000.

2- I chose to listen to port 80 but with different server names. So I must edit my hosts file

sudo gedit /etc/hosts

and add these two lines

127.0.0.1 phpmyadmin.local
127.0.0.1 slimtest.local 

3- During my tests with these configs, I had the css/img files accessible in my project with paths like

<link rel="stylesheet" type="text/css" href="/css/main.css">

by having a project structure like this

A problem I had:
The following day I had a problem with a project giving me the page "no input file specified". After inspecting the "/var/log/nginx/error.log" file, I found "(13: Permission denied)" errors. So I had to add extra permissions to the project:

chmod -R 755 ./

SyntaxHighlighter Fix for Blogger Dynamic View

After moving from old theme to the new Blogger Dynamic View theme, I hade my biggest issue: How to enable my SyntaxHighlighter again? I'd prefer find a solution that reformat ALL my blog posts throughout the years.

One solution I found was adding a Javascript snippet to all my posts (http://kevin-junghans.blogspot.com/2013/01/adding-syntaxhighlighter-to-blogger.html). But this will require A LOT of work too! So what I did was a bit of nasty yet effective solution:

In the head code I already have this code:

<!--SYNTAX HIGHLIGHTER BEGINS-->
<link href='http://alexgorbatchev.com/pub/sh/current/styles/shCore.css' rel='stylesheet' type='text/css'/>
<link href='http://alexgorbatchev.com/pub/sh/current/styles/shThemeDefault.css' rel='stylesheet' type='text/css'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shCore.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCpp.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCSharp.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJava.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPython.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushSql.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushXml.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJScript.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCss.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPhp.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushRuby.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPlain.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushBash.js' type='text/javascript'/>

<script language='javascript'>
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.config.clipboardSwf = &#39;http://alexgorbatchev.com/pub/sh/current/scripts/clipboard.swf&#39;;
SyntaxHighlighter.all();
</script>
<!--SYNTAX HIGHLIGHTER ENDS-->

So I added this interval code after the last javascript line:

setInterval(function(){
  SyntaxHighlighter.highlight(); 
}, 5000);

This way I will ensure that syntax highlighting is always enforced, and that it will work with infinite scrolling too.

Now I can enjoy my syntax highlighting with the new dynamic view theme.