Linux News | Top Links

WordPress on Nginx, Part 2: vhost, MySQL & APC Configurations

Latest news from Linux for you magazine - Mon, 02/13/2012 - 18:17

Last time around we made our Debian VPS ready with the LEMP recipe. Let’s now configure the stack and migrate over the old WP website.

What good a website with a “Welcome to nginx” note? That’s where we left last time.

My primary reference for this Apache to Nginx migration was this article — in fact, my configs are more or less a copy-paste from this guide.

For your convenience I’ll just repeat the steps here…

Configuring the Nginx vhost

Since it’s always nice to save a backup of the original default config files before we make any changes — because it’s easy to roll back to the reference point and troubleshoot when something goes wrong — we move the original nginx.conf file as follows:

# mv /etc/nginx/nginx.conf /etc/nginx/nginx.conf-org

Then create a new /etc/nginx/nginx.conf file and insert the following text in it:

user www-data; worker_processes 1; pid /var/run/nginx.pid; events { worker_connections 1024; # multi_accept on; } http { sendfile on; tcp_nopush on; tcp_nodelay on; server_tokens off; include mime.types; default_type application/octet-stream; index index.php index.htm index.html redirect.php; #Gzip gzip on; gzip_vary on; gzip_proxied any; gzip_comp_level 6; gzip_buffers 16 8k; gzip_http_version 1.1; gzip_disable "MSIE [1-6].(?!.*SV1)"; gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript; #FastCGI fastcgi_intercept_errors on; fastcgi_ignore_client_abort on; fastcgi_buffers 8 16k; fastcgi_buffer_size 32k; fastcgi_read_timeout 120; fastcgi_index index.php; limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s; ## # Virtual Host Configs ## include /etc/nginx/conf.d/*.conf; include /etc/nginx/sites-enabled/*; #Our individual site vhost server files will live here }

The worker_processes 1 directive above is of special importance here. I have the worker_processes set to 1 simply because the VPS I’m using currently offers me one CPU. It’s usually safe to set the worker_processes number to the number of processor cores you have. Like, for example, under a Rackspace Cloud Server install, I’ll have it set to 4, simply because I’m offered that many virtual cores.

Second, if you notice, the nginx.conf above doesn’t have any WordPress specific configs yet. However, if you look at the last statement, it basically tells Nginx to refer to /etc/nginx/sites-enabled/ directory for these configs.

If you do an ls inside of /etc/nginx/ the default DotDeb install gives your two directories that we all must take note of /etc/nginx/sites-enabled/ as mentioned in our /etc/nginx.conf file, and more importantly /etc/nginx/sites-available/ directory.

This second directory is where we’ll have our vhost configs, while in the former we’ll simply have individual vhost configs files’ symlinks. This gives us the option to disable a site while keeping the original config files intact by simply deleting the symlinks — and Nginx thinks the site is gone.

Do an ls inside /etc/nginx/sites-available/, and you’ll notice a default file already present. This same file is also simlinked inside /etc/nginx/sites-enabled/default. Open this file and you notice a location of a file: /usr/share/nginx/www/index.html. The content of this file follows:

<html> <head> <title>Welcome to nginx!</title> </head> <body bgcolor="white" text="black"> <center><h1>Welcome to nginx!</h1></center> </body> </html>

Remember the “Welcome to nginx!” message from the concluding screenshot of part 1? This /etc/nginx/sites-enabled/default file is a useless location, yet an excellent starting point to understand Nginx vhost configurations.

Let us now create our WordPress vhost config — in my case it’s /etc/nginx/sites-available/linuxforu.com — and insert the following text:

server { listen 80; server_name linuxforu.com www.linuxforu.com; root /srv/www/linuxforu.com/public; access_log /srv/www/linuxforu.com/logs/access.log; error_log /srv/www/linuxforu.com/logs/error.log; client_max_body_size 8M; client_body_buffer_size 128k; #The section below contains your WordPress rewrite rules location / { try_files $uri $uri/ /index.php?q=$uri&$args; } location /search { limit_req zone=one burst=3 nodelay; rewrite ^ /index.php; } fastcgi_intercept_errors off; location ~* \.(ico|css|js|gif|jpe?g|png)$ { expires max; add_header Pragma public; add_header Cache-Control "public, must-revalidate, proxy-revalidate"; } #sample 301 redirect # location /2011/06/26/permalink/ { # rewrite //2011/06/26/permalink/ http://example.com/2011/06/27/permalink_redirecting_to/ permanent; # } #Send the php files to upstream to PHP-FPM #This can also be added to separate file and added with an include location ~ \.php$ { try_files $uri =404; #This line closes a big security hole #see: http://forum.nginx.org/read.php?2,88845,page=3 fastcgi_param QUERY_STRING $query_string; fastcgi_param REQUEST_METHOD $request_method; fastcgi_param CONTENT_TYPE $content_type; fastcgi_param CONTENT_LENGTH $content_length; fastcgi_param SCRIPT_NAME $fastcgi_script_name; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_param REQUEST_URI $request_uri; fastcgi_param DOCUMENT_URI $document_uri; fastcgi_param DOCUMENT_ROOT $document_root; fastcgi_param SERVER_PROTOCOL $server_protocol; fastcgi_param GATEWAY_INTERFACE CGI/1.1; fastcgi_param SERVER_SOFTWARE nginx; fastcgi_param REMOTE_ADDR $remote_addr; fastcgi_param REMOTE_PORT $remote_port; fastcgi_param SERVER_ADDR $server_addr; fastcgi_param SERVER_PORT $server_port; fastcgi_param SERVER_NAME $server_name; fastcgi_pass 127.0.0.1:9000; } location /wp-admin { auth_basic "Administrator Login"; auth_basic_user_file /srv/www/linuxforu.com/.htpasswd; } #!!! IMPORTANT !!! We need to hide the password file from prying eyes # This will deny access to any hidden file (beginning with a .period) location ~ /\. { deny all; } #Once you have your w3-total.conf file ready uncomment out the line below include w3-total.conf; }

—————–
Now, let’s quickly go through the essential portions of this file.

In line #4, we give Nginx the domain name of our website.
In line #5, we tell Nginx which directory to load our website from — the webroot.
In line #6 and 7, we tell it where to save the access and error logs.
Line #12 to 21 hold the rewrite rules. Remember, we use mod_rewrite in Apache to take care of URL rewriting for WordPress permalinks? In Apache we do it using .htaccess files. Nginx doesn’t support .htaccess, so we write the rewrites in the vhost config itself, so that permalinks work out of the box once we’ve migrated the WordPress directory over from our old Apache host.
Line #31 holds a very important statement, to protect our website from arbitrary PHP code execution. (A lot of WordPress-based Nginx configs available on the Internet miss out on this.)
Line #57 to 60 hold the directive for password protecting the /wp-admin/ folder with a Web server-level password. This is just an additional layer of security before the WP authentication.
Line #64 hides any . (dot) files from prying eyes. Although our .htpasswd file is not inside the webroot, since we’ll be migrating the files from an Apache server we’d obviously like to hide all the .htaccess files that comes along with it that are hidden inside many subdirectories.
Finally in line #67, we define where W3 Total Cache should save its configurations. It usually does it in the .htaccess file of webroot automatically under Apache. But again since Nginx doesn’t support .htaccess as we’ve discussed, we define it specifically in in this vhost config file.

With that done. Let us create our webroot folder structure. Note that we’re not configuring it inside /usr/share/nginx/www/, but under /srv/. Under a fresh Debian install, this directory is empty.

# mkdir -p /srv/www/linuxforu.com/{public,logs}

So, now we have /srv/www/linuxforu.com/public/, which will be our webroot for linuxforu.com, and /srv/www/linuxforu.com/logs/, where Nginx saves the access and error logs.

Finally let’s create the /etc/nginx/w3-total.conf file that we’ll need later (as we defined in the vhost config above).

# touch /etc/nginx/w3-total.confMigration time

Time to migrate the wordpress files from the Apache server…

Note that if you’re configuring the Nginx server on the same host, you simply need to point to the correct WordPress install directory in the vhost config file above in line #5 and 6.

Since, for me, it was from MediaTemple to this VPS, and also since MediaTemple gives me shell access with rsync facility, it was a simple recursive rysnc for me.

In case you don’t have shell access to your old server, or your host doesn’t provide you with rsync facility, the the job becomes a bit tiresome — that is, having to download all the files over FTP. I somehow don’t like FTP, so I don’t touch shared hostings facilities that don’t provide me with rsync.

rysnc --progress -ruvpa <apache-server-username>@linuxforu.com:path/to/webroot/ /srv/www/linuxforu.com/public/

Maybe you can take a latest mysqldump of database before running the rsync command above — a plugin like WP-DBManager is a handy resoruce for the same. So, what happens in this case is, you rsync over the latest DB dump along with the other files.

Refer back to the /etc/nginx/nginx.conf file — note that the nginx daemon runs as www-data:www-data. However, our webroot — /srv/www/linuxforu.com/public — is root:root. Let’s correct that so that Nginx/WordPress has no issues writing to the webroot.

chown -R www-data:www-data /srv/www/linuxforu.com/public

Remeber, W3 Total Cache also needs to write its configurations, so we change the ownership of this file too:

chown www-data:www-data /etc/nginx/w3-total.conf

All good.

Now that the WordPress files are in place, type to generate the password to secure the /wp-admin/ directory. The easier way is to simply install the htpadded utility, part of the apache2-utils package

# apt-get install apache2-utils

No need to panic, this package doesn’t pull the main Apache server as a dependency.

Generate the htpasswd:

htpasswd -c /srv/www/linuxforu.com/.htpasswd <htpassds-username>

The location of the .htpasswd file could be anywhere as long as it corresponds with the location mentioned in the vhost file.

Database import and MySQL tuning

Although you can drive the website without altering any settings in MySQL, the following tunings are handy if you have a 2GB RAM VPS.

Let’s first save a backup of the default /etc/mysql/my.cnf file:

mv /etc/mysql/my.cnf /etc/mysql/my.cnf-bak

Now create an empty /etc/mysql/my.cnf file and append the following directives:

[client] port = 3306 socket = /var/run/mysqld/mysqld.sock [mysqld_safe] socket = /var/run/mysqld/mysqld.sock nice = 0 [mysqld] user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock port = 3306 basedir = /usr datadir = /var/lib/mysql tmpdir = /tmp language = /usr/share/mysql/english skip-external-locking key_buffer = 16M max_allowed_packet = 16M thread_stack = 192K thread_cache_size = 16 myisam-recover = BACKUP max_connections = 100 table_cache = 256 thread_concurrency = 16 query_cache_limit = 4M query_cache_size = 128M general_log_file = /var/log/mysql/mysql.log general_log = 1 log_slow_queries = /var/log/mysql/mysql-slow.log long_query_time = 2 log-queries-not-using-indexes expire_logs_days = 10 max_binlog_size = 100M [mysqldump] quick quote-names max_allowed_packet = 16M [isamchk] key_buffer = 16M # * IMPORTANT: Additional settings that can override those from this file! # The files must end with '.cnf', otherwise they'll be ignored. # !includedir /etc/mysql/conf.d/

We’ve just ended up supplying some good default memory power to our MySQL server.

Time to import the database — but first, we need to create the database to import the data into:

# mysql -u root -p Enter password:

Enter the root password that you’ve set when you installed the mysql-server-5.5 packages in the earlier article. Once you get the mysql> prompt after authentication, create a new MySQL user and a database on which this user has rights:

mysql> CREATE DATABASE lfydb; Query OK, 1 row affected (0.00 sec) mysql> GRANT ALL PRIVILEGES ON lfydb.* TO "lfy_user_name"@"localhost" IDENTIFIED BY "password"; Query OK, 0 rows affected (0.00 sec) mysql> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.01 sec) mysql> EXIT

Now import the current database — if you used a plugin like WP-DBManager with it’s default settings, the location should be something like this: /srv/www/linuxforu.com/wp-content/backup-db/1328724761_-_databasename.sql

Import this dump into your freshly created database as follows:

mysql -u lfy_user_name -p -h localhost lfydb < /srv/www/linuxforu.com/public/wp-content/backup-db/1328724761_-_databasename.sql

It will ask for lfy_user_name’s password. It will take some time before returning you the shell prompt — depends on the size of the database.

Finally, open your /srv/www/linuxforu.com/public/wp-config.php file, and make sure the database name, user, password and hostname correspondent to the new DB we created and imported the dump onto just now:

// ** MySQL settings - You can get this info from your web host ** // /** The name of the database for WordPress */ define('DB_NAME', 'lfydb'); /** MySQL database username */ define('DB_USER', 'lfy_user_name'); /** MySQL database password */ define('DB_PASSWORD', 'password'); /** MySQL hostname */ define('DB_HOST', 'localhost');

All good — reload mysql service:

# /etc/init.d/mysql reloadBack on WordPress to make final changes

Let us enable our vhost first — remember we need to create the correct symlink:

ln -s /etc/nginx/sites-available/linuxforu.com /etc/nginx/sites-enabled/linuxforu.com

Reload nginx service with the new config:

/etc/init.d/nginx reload

Time to now check the site! However, before making changes permanent to your domain’s DNS service, we will edit our local machine’s (not the VPS — but the desktop or laptop we’re using) /etc/hosts file to cheat the browser into bypassing a DNS check to open linuxforu.com:

<Nginx-VPS-IP-ADDRESS> linuxforu.com www.linuxforu.com

Save and close… run a ping test on your local machine to confirm that linuxforu.com resolves to the IP address of the Nginx VPS.

Time to finally launch your browser, and login to the wordpress backend — upon entering the www.your-domain-name-here.com/wp-admin/ you should be greeted by the htpasswd authentication first :-)

Web server directory access authentication for accessing /wp-admin/

Validate, and you get your WordPress login page. Once you authenticate, first thing to do is add the nginx Compatibility plugin. Quoting the plugin’s page will make it clear why this is necessary:

The plugin solves two problems:

When WordPress detects that FastCGI PHP SAPI is in use, it disregards the redirect status code passed to wp_redirect. Thus, all 301 redirects become 302 redirects which may not be good for SEO. The plugin overrides wp_redirect when it detects that nginx is used.
When WordPress detects that mod_rewrite is not loaded (which is the case for nginx as it does not load any Apache modules) it falls back to PATHINFO permalinks in Permalink Settings page. nginx itself has built-in support for URL rewriting and does not need PATHINFO permalinks. Thus, when the plugin detects that nginx is used, it makes WordPress think that mod_rewrite is loaded and it is OK to use pretty permalinks.

Good things is, this is a zero-setup plugin — and takes care of the above two point after simple activation. However, it drops in two plugins, and by default activates “nginx Compatibility (PHP4)”. Since, we’re using PHP5, deactivate that and active “nginx Compatibility (PHP5)” as you can see in the following screenshot:

Make sure to activate the currect nginx Compatibility plugin

Finally, access your W3 Total Cache settings… and you should be greeted by a lot of red warnings. Ignore them and scroll down to the bottom of the page.

W3 Total Cache automatically detects that the server is Nginx, and since it can’t have the easy way around of writing configs to .htaccess anymore, it will present you with a box to fill in the location of “Nginx server configuration file path”. Since we defined the location as /etc/nginx/w3-total.confin the vhost file, we enter the same full path here — as you can see in the screenshot below.

Enter the full path for W3 Total Cache config file here

Save settings.

Scroll to the top and click “auto-install” buttons on all those red warning boxes. Unfortunately, I don’t have a screenshot handy for that — but it basically implies that the specified settings are not available in the text file for it to refer to. (You don’t lose any of this plugin-specific custom settings because it also has the settings saved in the DB, and that’s why the prompt — the plugin is smart!) Clicking auto-install simply dumps the details, which it otherwise saves on a .htaccess in Apache, to our /etc/nginx/w3-total.conf location.

The best part: if you’re coming from a server environment where APC was not available, make W3 Total Cache’s opcode and db cache dropdowns to APC :-)

Go back to VPS SSH terminal, open /etc/php5/conf.d/apc.ini, and append the following settings (taken as it is from the tutorial I referred at the beginning of the article):

; configuration for php apc module extension = apc.so apc.enabled = 1 apc.shm_segments = 1 apc.shm_size = 512M apc.optimization = 0 apc.num_files_hint = 2700 apc.user_entries_hint = 2700 apc.ttl = 7200 apc.user_ttl = 3600 apc.gc_ttl = 600 apc.cache_by_default = 1 apc.slam_defense = 1 apc.use_request_time = 1 apc.mmap_file_mask = /dev/zero apc.file_update_protection = 2 apc.enable_cli = 0 apc.max_file_size = 2M apc.stat = 1 apc.write_lock = 1 apc.report_autofilter = 0 apc.include_once_override = 0 apc.rfc1867 = 0 apc.rfc1867_prefix = "upload_" apc.rfc1867_name = "APC_UPLOAD_PROGRESS" apc.rfc1867_freq = 0 apc.localcache = 1 apc.localcache.size = 1350 apc.coredump_unmap = 0 apc.stat_ctime = 0

Reload the php5-fpm service

# /etc/init.d/php5-fpm reload

Hopefully, you shouldn’t encounter any errors. Go back to your browser — to the settings page of W3 Total Cache and clear all cache.

View your site! We’re done here!

Finally, open your CDN administrator settings and enter the IP address of your VPS — I had to since we use MaxCDN as an Origin Pull CDN. And then go to your domain’s DNS server admin area and enter the new IP — since linuxforu.com is propagated by CloudFlare it took less than 15 minutes before I could remove the /etc/hosts file hack from my local system and was accessing the website from the new VPS — pages served by Nginx :-)

BTW, here’s some benchmark test (using the ab utility that also comes as part of the apache2-utils package that we’d installed for htpassed utility earlier) — for 500 simultaneous connections 10,000 times:

$ ab -n 10000 -c 500 http://www.linuxforu.com/ This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking www.linuxforu.com (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed 10000 requests Finished 10000 requests Server Software: nginx Server Hostname: www.linuxforu.com Server Port: 80 Document Path: / Document Length: 75493 bytes Concurrency Level: 500 Time taken for tests: 3.086 seconds Complete requests: 10000 Failed requests: 0 Write errors: 0 Total transferred: 758990000 bytes HTML transferred: 754930000 bytes Requests per second: 3240.81 [#/sec] (mean) Time per request: 154.282 [ms] (mean) Time per request: 0.309 [ms] (mean, across all concurrent requests) Transfer rate: 240209.24 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 1 5.6 0 30 Processing: 28 43 15.3 39 268 Waiting: 20 42 14.9 39 268 Total: 35 44 18.3 40 268 Percentage of the requests served within a certain time (ms) 50% 40 66% 40 75% 40 80% 41 90% 41 95% 69 98% 122 99% 131 100% 268 (longest request)

That’s quite impressive on a standard 2GB VPS with one cpu core :-) Try to replicate the same results on Apache.

Dealing with logs

One final step is set correct server paths so that logroate can also compress the logs for our Nginx “access” and “error” logs located at the non-standard location /srv/www/linuxforu.com/logs/. Open /etc/logrotate.d/nginx, and make sure it reads like this:

/srv/www/linuxforu.com/logs/*.log { daily missingok rotate 52 compress delaycompress notifempty create 0640 www-data adm sharedscripts prerotate if [ -d /etc/logrotate.d/httpd-prerotate ]; then \ run-parts /etc/logrotate.d/httpd-prerotate; \ fi; \ endscript postrotate [ ! -f /var/run/nginx.pid ] || kill -USR1 `cat /var/run/nginx.pid` endscript }

That’s all folks!Related Posts:

Tags: Apache, cache, Debian, FastCGI, IP address, LEMP, Linux hosting, MySQL, nginx, PHP FPM, Rackspace, RAM, rsync, server software, vhost, VPS, W3 Total Cache, Web server, WordPress, wordpress plugins, WP-DBManager

Categories: Electronics News, Linux News, News

WordPress on Nginx, Part 1: Preparing VPS the Debian Way

Latest news from Linux for you magazine - Sat, 02/11/2012 - 16:13

In this first part, we deal with the LEMP (Linux, Nginx, MySQL, PHP) stack recipe and set up the basic server after installing the required packages.

It started off thus:

The theme developer pushed out point update that came with no info on what all files have changed so that I could have done a quick drag and drop file replacement for patch-up work.

However, what looked like a very boring workload to proceed with, turned out to be quite interesting. Well, updating the theme, and then including my customisations wasn’t at all interesting… what was instead, was changing the architecture of the website — from LAMP to LEMP.

Since November, linuxforu.com was hosted on the grid hosting facility of MediaTemple (gs), with a dedicated MySQL container to give it some much-needed “dedicated” boost. Caching was the responsibility of W3 Total Cache — and thus serving in excess of 100,000 pages a month since then has had become a child’s play. (The elders might remember the performance issues with the website before December 2011!) The reason for going with MediaTemple’s gs hosting was what comes as a given with shared-server type of hosting environments — security is the headache of the hosting company.

However, once you move to a VPS/Cloud environment, OS hardening all of a sudden becomes the webmaster’s responsibility — stinks for me because I wear that cap since November last year.

Well, anyhow — a couple of weeks back I did switch this website to a 512MB Rackspace Cloud Server for a little more than 24 hours for stress testing, and actually saw a performance boost (specifically the backend — which I need to use most of the time unlike our visitors). That was of course a LEMP setup (notice the E?).

Why LEMP? Well, it’s something new for me dabble with… Besides WordPress.com runs on it. And Boudhayan, long-time LFY author and my good friend, wrote a quick tutorial on switching from LAMP to LEMP in December, which sort of gave me the push to check it out for myself.

So, my recipe for those 24 hours was this:

Debian 6.0 64-bit OS on a 512MB Rackspace Cloud Server
Nginx 1.0.x (from the dotdeb repo) for Web Server
MariaDB 5.3 (from official mariadb channel) for WP’s DB needs
PHP-FPM (from the dotdeb repo) for Nginx’s PHP-FastCGI needs
W3 Total Cache with APC taking care of object cache — for this website’s caching needs

And, fortunately, it ran smoothly enough before I made the switch back to MediaTemple’s Apache. Apart from Nginx, where I actually had to set up the virtual hosts… the rest of the pieces — MariaDB, APC, PHP-FPM — were really vanilla configs with no config changes atop what came out of the box.

Thus, the idea was planted in my (bone)head. Taking a cue from the Rackspace experience my plan was to setup a DR (disaster recovery) image ready on Rack and dump it on its CloudFiles storage.

But I had to run the website for a month to figure out any issues if we were to proceed with a LEMP stack for our future deployments. (Well, I didn’t say we’re not switching back to MT, did I? The files are lying there. I just need to import the MySQL dump, run rsync to sync wp-content/uploads/ folder between the two servers, and change the DNS — all this hardly takes an hour.)

It started off with GoDaddy offering 98% discount on their 1GB cloud offerings for the first month. Cool. Rs 54 for a month is literally free of cost. Time to setup.

What? No Debian.

Fine, let’s go with Ubuntu 10.04 LTS.

Too many repos to setup just to fetch the latest versions of Nginx, PHP-FPM and APC. All done, but too much effort. Delete. Fedora and CentOS as servers were out of question — not because they are bad, but due to being less familiar with those environments.

Myself being a cheapstear — I mean, who wants to spend a bomb just for PoCs? ;-) — I headed over to Hetzner.de to acquire a 2GB VPS (yeah, not cloud, but what the heck!).

So, what’s my new recipe this time?

Debian 6.0 32-bit on a Hetzner 2GB VPS (only downside is this one offers a single core compared to Rackspace’s 4 virtual core offerings)
Nginx 1.0.x (from the dotdeb repo) for Web Server
MySQL 5.5 (from dotdeb repo) for WP’s DB needs
PHP-FPM (from the dotdeb repo) for Nginx’s PHP-FastCGI needs
W3 Total Cache with APC taking care of object and database cache — for this website’s caching needs
Varnish 3.0 (from official Varnish repo) as the HTTP accelerator. (It’s disabled for now, because I was unable to provide a streamlined and separate mobile theme experience — the home page was desktop, but the articles were WPTouch — weird, not an expert, no clue, too lazy to carry it forward. Dumped.)

Just FYI: Our DNS layer is CloudFlare and W3 Total Cache with page and minify on disk (DB and object cache disabled), and a CDN to deliver static content (initially from Amazon CloudFront, and now from MaxCDN) were already being used with the MediaTemple setup too. Besides, the list of WP plugins also hasn’t changed.

The reason for dumping CloudFront by end of December was simple — the CSS and the JS files that were delivered by the CDN were not gzipped. I gathered text file gzipping is not offered by Amazon CloudFront — at least my pea-brain couldn’t identify any such setting on their dashboard. Being a sucker for Page Speed scores (our current score is 94/100), I made the switch to MaxCDN since enabling text file gzipping is as simple as checking a checkbox. (Note that MaxCDN has lesser Edge servers compared to CloudFlare, but they are literally half the price of CloudFront.)

Well, with that introduction over. Let’s get down to the actual configurations — based on tutorials from all over the Web.

Debian on the VPS — hardening it a bit

I chose a Debian 32-bit minimal. Hetzner took more than 12 hours to provision the VPS. Since by this time I had already given up on GoDaddy’s Ubuntu thanks to the headaches of setting up a LEMP stack on 10.04 LTS, I thought of setting up the LEMP templates on Rackspace Cloud Server — they provision the server, because of the advantages that come with Cloud offerings, in less than 5 minutes.

The idea was to get up the LEMP stack with WP support, and rsync the configurations over to the Hetzner VPS once I get access. Rackspace bills only for the hours I use their infra, so I keep it till I get the other VPS, and then dump the backup onto their CloudFiles storage, and delete the server instance. This exercise actually gave me the idea of a DR entity that I could manage if the LFY management decides to go with Rackspace and WP atop LEMP for the long haul.

The first thing was to create a new user account:

# useradd -m <user-name> # passwd <user-name>

Now, open the /etc/ssh/sshd_conf file, find the following directive:

PermitRootLogin yes

…and change this to read:

PermitRootLogin no

This means, SSH root logins are now disabled. So, from now on, we login as a regular user and su - to do admin stuff.

Before we setup a firewall, login over SSH as the <user-name> we created earlier to test everything is working. If successful, reload the changed SSH config into memory so that root logins are rejected by the SSH server henceforth:

/etc/init.d/ssh restart

Next up, the firewall:

# apt-get update # apt-get install ufw

UFW (Uncomplicated Firewall) is an amazingly easy-to-setup firewall software that I had never heard of before, but liked the fact that by using it I don’t need to dabble into the world of IPTABLES commands anymore.

The only ports that should be a available over the www is 22 for SSH, and 80 for accessing web server. So, run:

# ufw allow www # ufw allow ssh # ufw default deny # ufw enable

Running the last command will warn you that your SSH session could be disrupted — since we’ve already successfully tested SSH connections for <user-name> after disabling root logins, it’s safe to press y to confirm. It, however, didn’t interrupt my SSH session — so much for panic-driven false warnings ;-)

Run the following command to check the firewall status:

# ufw status Status: active To Action From -- ------ ---- 22 ALLOW Anywhere 80 ALLOW Anywhere

Good enough. You might wanna read this excellent UFW guide to customise the firewall settings further to your liking.

Time to harden the OS a bit more against some of the wild attacks that the server might be subjected to.

Taking a cue from this excellent nixCraft article, open your /etc/sysctl.conf, check what all things already exist there and then enter the rest of the following directives in that file:

# Avoid a smurf attack net.ipv4.icmp_echo_ignore_broadcasts = 1 # Turn on protection for bad icmp error messages net.ipv4.icmp_ignore_bogus_error_responses = 1 # Turn on syncookies for SYN flood attack protection net.ipv4.tcp_syncookies = 1 # Turn on and log spoofed, source routed, and redirect packets net.ipv4.conf.all.log_martians = 1 net.ipv4.conf.default.log_martians = 1 # No source routed packets here net.ipv4.conf.all.accept_source_route = 0 net.ipv4.conf.default.accept_source_route = 0 # Turn on reverse path filtering net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.default.rp_filter = 1 # Make sure no one can alter the routing tables net.ipv4.conf.all.accept_redirects = 0 net.ipv4.conf.default.accept_redirects = 0 net.ipv4.conf.all.secure_redirects = 0 net.ipv4.conf.default.secure_redirects = 0 # Don't act as a router net.ipv4.ip_forward = 0 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.send_redirects = 0 # Turn on execshild kernel.exec-shield = 1 kernel.randomize_va_space = 1 # Tuen IPv6 net.ipv6.conf.default.router_solicitations = 0 net.ipv6.conf.default.accept_ra_rtr_pref = 0 net.ipv6.conf.default.accept_ra_pinfo = 0 net.ipv6.conf.default.accept_ra_defrtr = 0 net.ipv6.conf.default.autoconf = 0 net.ipv6.conf.default.dad_transmits = 0 net.ipv6.conf.default.max_addresses = 1 # Optimization for port usefor LBs # Increase system file descriptor limit fs.file-max = 65535 # Allow for more PIDs (to reduce rollover problems); may break some programs 32768 kernel.pid_max = 65536 # Increase system IP port limits net.ipv4.ip_local_port_range = 2000 65000 # Increase TCP max buffer size setable using setsockopt() net.ipv4.tcp_rmem = 4096 87380 8388608 net.ipv4.tcp_wmem = 4096 87380 8388608 # Increase Linux auto tuning TCP buffer limits # min, default, and max number of bytes to use # set max to at least 4MB, or higher if you use very high BDP paths # Tcp Windows etc net.core.rmem_max = 8388608 net.core.wmem_max = 8388608 net.core.netdev_max_backlog = 5000 net.ipv4.tcp_window_scaling = 1

Load the directives to memory using the following command:

# sysctl -p

Check the output for any error messages — correct those, reload — and done.

The next phase is getting the server ready to install the LEMP stack

Fetching Nginx, PHP-FPM, APC & MySQL

The default Debian repo contain older versions of Nginx and MySQL. The excellent maintainer at DotDeb offers a Debian 6.0-compatible source with newer packages of these servers.

Open your /etc/apt/sources.list file, and append the following statements to setup the repo:

#Dotdeb repo deb http://packages.dotdeb.org stable all deb-src http://packages.dotdeb.org stable all

Next up, import the signing key to apt as follows:

# wget http://www.dotdeb.org/dotdeb.gpg # cat dotdeb.gpg | apt-key add -

Time to install Nginx, MySQL, PHP-FPM and other essential packages:

# apt-get install nginx-full # apt-get install mysql-server-5.5 # apt-get php5-fpm php5 php5-mysql php5-apc php5-mysql \ php5-xsl php5-xmlrpc php5-sqlite php5-snmp php5-curl php5-gd \ php5-imagick php5-tidy php5-memcache php5-imap

While you’re on an apt-get mood, go ahead and install the following two utilities as well:

# apt-get install rsync htop

rsync will help us later to fetch the WordPress files from the current server, while htop is a personal choice over the top command — just looks a bit better :-)

htop screenshot — note the memory consumption

A memory consumption of 114MB after setting up the WordPress site, and right after reloading nginx, php-fpm and mysql services (more about that later) is impressive.

By the way, we haven’t set the server timezone yet, have we? Run the following to set it to your local time:

# dpkg-reconfigure tzdata

The ncurses-based interface makes setting up server time a no-brainer

I, of course, have set it to IST (Asia/Kolkata — UTC + 0530Hrs).

That’s it for today. In the second part, which hopefully should be up by Monday, we’ll cover how to setup a Nginx vhost for WordPress, move the files from the old server to the new one, import the MySQL database, and finally make necessary changes to the CDN facility and CloudFlare’s DNS.

For the time being, enter your VPS IP address in your browser and you should see the Nginx server there, for now :-)

Nginx server is listening…

Tags: Amazon, Caching, CDN, CDN facility, CentOS, Cloud Server, CloudFlare, CloudFront, DNS, Fedora, firewall, GoDaddy, Hetzner, LEMP, Linux, MaxCDN, MediaTemple, MySQL, nginx, os hardening, PHP, PHP FPM, Rackspace, stress testing, ubuntu, W3 Total Cache, Web server, WordPress

Categories: Electronics News, Linux News, News

Proprietary Solutions Can’t Milk Money from these Cooperatives!

Latest news from Linux for you magazine - Tue, 01/31/2012 - 22:29

The dairy cooperative societies in Kerala are gradually switching to open source in an attempt to cut costs and stay secure.

While people are still contemplating whether to use open source in their mission-critical operations, the Kerala Dairy Cooperative Societies (DCS) has set an example by using open source in its day-to-day activities, to cut costs.

In cooperative societies, a group of dairy farmers come together and are assisted by the government to set up a society. Each district has an average of 100 societies, which could each comprise 200 to 2,000 dairy farmers. The farmers bring their produce to the societies, where it is measured in terms of quantity and quality, and this data is recorded on a daily basis.

Earlier, their process involved manually entering data into registers, ledgers, schedules and forms. This caused a lot of difficulty in verifying and tracking the status of each activity, and settling accounts with dairy farmers and customers.

The initial exercise, prior to adopting open source, included discussions with the dairy societies to identify their complete requirements and problems. Their main need was automation of office procedures like data entry and the generation of reports.

A computerised information system was developed with the help of National Informatics Centre (NIC) in 2001. Called Lypsaa, it was developed on a Microsoft platform and used Visual Basic for development, Crystal Reports to generate reports, and the Windows 98 operating system.

“Though this introduced transparency into the system, proprietary solutions meant a high cost in the form of licence fees for at least 70 systems and servers,” shares Khalid C, solutions architect of Lypsaa and a freelance consultant. Moreover, Microsoft stopped supporting some of the software after they launched newer versions of the Windows OS, and there were issues with compatibility.

A software development committee created jointly by DCS and NIC was against the hidden costs of proprietary software. They evaluated Lypsaa and submitted a proposal to migrate it to an open source solution named OpenLypsaa. The committee started its efforts to develop free software using hired resources in 2007.

Open source for cost-cutting

The most prominent benefit to the societies is related to cost. “As a result of using free and open source software at every possible step, the Kerala DCSs collectively saved about Rs 25 lakh,” states Simon E K, District Informatics Officer, NIC.

“If we went ahead with Windows-based PCs and proprietary databases, it would have cost about Rs 5,000 per society. We are assured that our total investment of Rs 5-6 lakhs was a good one. We also have a hold on the total cost of ownership including development, installing and using software for the duration of its life,” he adds.

The DCSs did not have to make additional investments for new computer systems as they could use their regular systems with Intel architecture, 512 MB or higher RAM, and 80 GB hard disk drives.

Ease of maintenance is another advantage that Jose Emmanuel, deputy director, Dairy Development, Government of Kerala, noted. He states, “With Microsoft, we would often get calls from the societies due to viruses on their machines — at least one call per month. Our experience with open source proves that it is much more secure, as we receive no complaints about viruses, reformatting and system crashes.”

Going the open source way

To begin with, Ubuntu was introduced on all the DCS systems. OpenLypsaa was developed and implemented at 30 societies by 2008. It is a complete software solution developed using the LAMP platform. OpenLypsaa is used in functions like generating milk-bills for farmers and taking care of their daily accounting. It is used to register members, keep milk procurement details, manage financial accounting, prepare milk bills, register any service connections, and maintain the accounts and details of various customers.

“Milk bills include details like the quantity of produce brought, its quality, the price at which it was sold, etc. When data was entered manually, it was impossible to avoid numerical errors. Mistakes are almost negligible now. Moreover, there was room for malpractice when it came to creating a voters’ list for elections within the societies. This system ensures fair play by filtering out farmers who have not contributed a minimum amount of milk to the society,” Khalid confidently states.

The system maintains registers for sales, producers’ personal details, cattle feed sales, advance payments and financial accounts, and also helps in generating crucial reports on voters, pensioners and scheme beneficiaries.

“For instance, the state government sometimes gives benefits to farmers who have contributed regularly for two months, or have given extra produce. It was earlier a cumbersome task to look through our registers and fish through all the entries. With OpenLypsaa, the beneficiaries are a click away. We can also generate a periodical report on farmers based on these entries,” adds Khalid.

OpenLypsaa used TCPDF — a PHP class to generate PDF documents, “Business Intelligence and Reporting Tool” (BIRT) for report generation, and HighCharts for charting MIS reports. HighCharts is a charting library written in pure JavaScript, which offers an easy way to add interactive charts to a website or Web application.

Khalid says, “All data and configuration comes in JSON format. It fits well with PHP, without having to write tons of code or use some third-party library. The json_encode function is very handy when passing options from PHP to HighCharts.”

The committee, along with NIC, also developed a solution called “System for Automating Milk Procurement” (SAMP). It was developed using Java and SQLite. With the previous system, farmers would contribute their produce in the morning and a sample would be sent to the lab to test for quality details like fat content, protein and water content. Based on this test, the price for each farmer’s produce was decided, and the farmer would be informed about the price only in the evening.

“With the introduction of SAMP, the quality and content of the milk can be measured when the farmer brings his produce, and the price is set right away,” Khalid reveals.

About a year ago, an SMS response system was also created by the team for auto-answering farmers’ SMS queries using a Kannel SMS Gateway. Farmers can type in key words like “Milk rate”, “Monthly payment”, etc., to the number provided, and the database generates the reply in the form of an SMS. The committee is also developing a dairy portal, through which farmers can access information about periodical reports and statements, statistical reports, government orders and circulars in Malayalam. This is due to be launched early next year.

Not a smooth journey

Despite the advantages, migrating to open source was not an easy task. One of the main challenges was training the employees at each society. “The staff had to be trained on many aspects like file operations, office applications, taking back-ups and restoring databases on Ubuntu Linux. For this, our support team visited each society and trained the staff themselves,” explains Khalid. Finding manpower for this support has also been a constant challenge.

Khalid adds, “With NIC’s support, the DCS formed a formal committee, but it has not been set up well yet. Their salary comes from the cooperative societies’ contributions, and we still find it difficult to get employees with expertise in PHP and Linux to stay here for a long tenure.”

Some of the other challenges faced in the initial stages were architecture compatibility of open source tools, finding open source solutions for various requirements, and the selection of the right open source tools. For this, the software development team and NIC interacted regularly with the online open source communities to overcome the issues.

Just around the corner

The DCSs are still looking for people who can add to their support team. The committee members still feel that the challenges they have faced are minuscule compared to the benefits they have received. At present, 50 societies are still running on Lypsaa, and they gradually plan to migrate to OpenLypsaa over the next two years, until the DCSs completely run on open source.Related Posts:

Tags: computerised information system, cooperative societies, daily accounting, Dairy, dairy farmers, dairy portal, data entry, farmers, financial accounting, Free Software, Intel, Java, JavaScript, Jose Emmanuel, JSON, Kerala, Kerala Dairy Cooperative Societies, Khalid C, LFY January 2012, licence fees, Linux, Microsoft, Milk, National Informatics Centre, open source tools, PHP, proprietary software, SMS, ubuntu, Visual Basic, web applications, Windows 98

Categories: Electronics News, Linux News, News

Voices Across the Digital Divide — Using Audio Portals to Connect Communities

Latest news from Linux for you magazine - Tue, 01/31/2012 - 17:38

Human beings are the only species on earth with the ability to communicate complex ideas through language. Speaking and listening have been the basis of human society since people started living in communities. In fact, the words “community” and “communication” share a common etymology.

Democracy, as a system, is completely dependent on communication, to the extent that when communication breaks down, so does the democratic process. In order for a group of people to participate equally in democracy, they must necessarily share a communication platform, where they can share not just facts, but also views and opinions. Small wonder then, that free speech is prized and cherished by all democracies, and coveted by citizens of almost all countries that are yet to become democracies.

One of the fundamental requirements of free speech and participation in democracy is the availability of a free, open medium and platform of communication that is equally accessible by all members of the democratic community. Almost every culture in the world has a concept of a central community gathering place, where people gather after a day’s work, to talk and share information.

In India, this is typically the village chaupal, in West Kalimantan (erstwhile Borneo), Indonesia, it’s called a ruai. In Afghanistan, it may be called a chaikhana. These community structures have traditionally provided the common platform and free medium for communication.

This type of platform is structured like a circle, and the free medium is air. In a circular structure, everyone has an equal say, because everyone has equal access to the medium and equal reach to every other member of the platform. No special equipment is required to use this medium; ears and a mouth will typically suffice. These structures provided a way for people to voice their opinion, share their concerns and find solutions to conflict through dialogue.

After the industrial revolution and the dawn of the corporation, mass media began to play this role in people’s lives. Newspapers, radio and television became the new media that people used. These media had a much wider reach and they seemed like the perfect democratic tool. However, these media have a structural problem that prevents them from being truly democratic. By virtue of corporate and editorial hierarchy, these media are structured like a triangle (Figure 1).

Figure 1: Media communication

News, in this model, travels downwards from an elite minority that determines what content is “newsworthy” to the community. The community typically cannot relate the incoming news to their own lives, and either becomes disenfranchised by virtue of lack of representation, or assumes the media version of facts to be true, and that they themselves are an anomaly. At the very least, this influences their participation in democracy, and at worst, they are rendered voiceless in that most fundamental democratic process — debate.

This hierarchical model of modern commercial media requires profits for the media organisation to continue to run. This means that news needs to sell. If a newspaper cannot generate advertising revenue, it will soon shut down. Obviously, with profit as the first imperative, relevance of the content to the community and their feedback must become secondary. Moreover, there is an incentive in preventing communication technology from reaching its true potential. For example, if community radio became fully deregulated, would commercial radio or, for that matter, television, stand a chance?

This skewed set of incentives, and the resulting policies and actions, has led to several communities across the world, particularly in the developing world, becoming alienated and disenfranchised with mainstream society. These communities are particularly susceptible to coercion and this might partly explain the escalating violence in the world today.

This conundrum should be quite familiar to open source enthusiasts, since the basic principles involved are much the same as the ones in the open source vs closed source software debate. To draw a parallel from The Cathedral And The Bazaar, mainstream media follows the cathedral model, while community platforms are more like bazaars. Both paradigms have their value and importance in the structure of society at large. However, in the context of media, the cathedral or top-down model appears to have reached its limits of effectiveness — and, in my opinion, has passed the point of diminishing returns.

The growth of user-generated content on the Internet over the last decade is a clear indicator that as connectivity improves, people are increasingly eager to directly voice their opinions and concerns without the need of mainstream media as an intermediary, particularly since in the real world, no intermediary is perfectly impartial.

The developing world

In the developing world, this uprising of citizen media has been stunted by the uneven distribution of resources, such as infrastructure, connectivity and literacy. While connectivity in the developed world has allowed the blogosphere to become a political force to contend with, most developing countries have an Internet penetration of less than 10 per cent, typically concentrated in urban areas.

Even where connectivity exists, the vast majority of users are only just starting to view the Internet as anything more than email and instant messaging. In many of these countries, even as economies have opened up and globalisation has settled in, entire communities are still disconnected from the rest of the world, primarily because they do not represent a market segment worthy of media representation.

Mainstream media in these countries typically focus on urban issues that relate to economic and political decision makers, rather than the vox populi.

In several of these countries, however, innovation is now taking place to bridge this gap by other means. While Internet penetration remains low, the use of mobile phones is a different story altogether. Most of the developing world has far outpaced the developed world in terms of mobile phone adoption and versatility of usage. Even in places where people earn less than a dollar a day, cell phones are ubiquitous. A medium that uses voice, the oldest mode of communication known to man, amplified by several orders of magnitude, so as to cover unimaginable distances, is as irresistible to a Gond tribal in Chhattisgarh, India, as it is to a street food vendor in Jakarta, Indonesia.

Recognising the potential of this medium, several groups are now actively engaged in developing technology to allow people to use their voice to connect themselves and their communities to the rest of the world. One of the first tools of this new age of innovation is the audio portal.

An audio portal?

An audio portal (Figure 2) is essentially a website with a lot of audio content that can be accessed both through the Web as well as by phone.

Figure 2: An audio portal

While the Web interface is usually like a blog, the phone interface is an IVR (Interactive Voice Response) system, where users press keys to navigate through menus and content. In more advanced IVR systems, voice recognition may be used, though this is still limited to the well-documented accents of the English language. The Web interface is very similar to a blog, and several audio portals do use the blog layout.

Behind the scenes, the platform will also provide an interface to manage posts. Early implementations of audio portals tended to rely on specialised moderation consoles, which have media-previewing capabilities as well as functionality for moderators to add metadata, such as a summary and title, to the content to make it friendlier to users on the Web.

Users will typically call the IVR interface to record and listen to content using their cell phones, while Web users will access the website interface to listen to the audio posts using a browser, and leave comments in text, which then may or may not be converted to audio using a text-to-speech system.

People who own the latest Android or iPhone may find the idea of an IVR interface to browse content somewhat counter-intuitive, since it makes no sense to call in and scroll through a set of menus, particularly with an irritatingly monotonic voice rattling out instructions all the time, when you can simply open the Web page on your cell phone’s browser, and read.

The graph in Figure 3 may help clarify why a purely visual interface is not adequate to reach the majority of the world.

Figure 3: Cell phone and broadband users

The percentage of Internet users, even among the mobile phone users of the world, is a fraction of the percentage of people using their phones purely for voice and SMS. While mobile Internet use is, and will continue to be, on the rise, the bulk of the world will continue to be on voice for some time to come.

This is also historically consistent, since most societies have far stronger oral traditions than written ones. Voice captures much more than simply language. Tone, quality, emotion are all interwoven in the spoken word. If a picture is worth a thousand written words, then a spoken word counts for at least a few hundred… not to mention that drawing an attractive picture takes considerably more skill than speaking!

What makes mobile phones particularly attractive as a medium, though, is the two-way nature of the medium. With radio and television, though the reach may be much wider than mobile phones, the ability to respond immediately to what you hear or see — on the same platform, at the same level as the source, which is extremely valuable in fostering dialogue — is missing.

The audio portal concept caters to every cell phone, whether mass-market or smartphone equally, which works very well to level the platform. Most importantly, audio portals use technology, skills and other resources that are available now, as opposed to those that require extensive “capacity building” exercises. This is probably the reason why audio portals, as a tool, find more favour with grassroots workers and members of the community, rather than with technology evangelists and academia.

The technology

Audio portals utilise relatively simple technology, most of which has been around in the open source world for some time. An audio portal will typically consist of a phone interface (either fixed-line or mobile), connected to a content-management system (usually a database) and a Web front-end, via an IVR running on a soft switch or software PBX system. Two examples of audio portal platforms are Swara and FreedomFone.

Swara

Swara is an open source project, originally written as a research project by students and professors at MIT to augment the outreach and activities of CGNet, a people’s discussion group working with indigenous communities in central India. CGNet was started by veteran journalist Shubhranshu Choudhary, who returned to Central India, where he grew up, to find it torn by violence. Probing to find the reason for the conflict, he quickly realised that open, accessible community media would be a key component of any solution to the conflict. Given that Internet penetration in the region is less than 1 per cent, and community radio is limited by regulation, the next best medium for a community platform was the mobile phone.

The first pilot of Swara was deployed in Bengaluru for use by indigenous communities in Chhattisgarh and neighbouring states in February 2010. Today, the pilot receives over 300 calls a day, and the team is working on building the platform out as an open source project for deployment in other locations. The first replica of the project went live in Indonesia in December 2011.

Swara uses a combination of the Asterisk PBX system in combination with the LoudBlog audio blogging platform, with the integration written in Python. The tested interfaces are GSM gateways (Topex Mobilink, etc) and fixed lines (PRI/BRI) using a Digium telephony card.

FreedomFone

FreedomFone was developed by Alberto Escudero Pascual and Louise Berthilson of IT46, a Swedish IT consultancy, for the Kubatana Trust in Zimbabwe. It was created for many of the same reasons as Swara was developed in India, i.e., lack of impartial and open commercial media, and the need for local and community-level reporting. The FreedomFone pilot, a weekly audio magazine called Inzwa, has been running in Zimbabwe since July 2009, and received over 2,500 calls between July and September 2009. FreedomFone’s team is also working on developing the platform as a user-friendly DIY IVR kit, and is keen on replicating the model in other areas.

FreedomFone uses the FreeSWITCH soft switch to interface with telephony devices such as the Mobigater and Office Router GSM gateways. The content management system is written in CakePHP, and FreedomFone additionally uses the Cepstral speech synthesis system for text-to-speech conversions. The stated objective is to create a purely phone-accessible platform.

Deployment 101

Both platforms have an almost identical design, as would most audio portal software. This is almost analogous to how traditional websites are built, with the choice of platform being similar to the choice between different Web frameworks. Just as you will find lots of different opinions and preferences for Web platforms among Web designers, you will find that the few implementers of audio portals are just as varied in their preferences for platforms. This usually depends on which platform the implementer is most familiar with — and if you are implementing your own, one is essentially as good as the other.

The key question, irrespective of which platform you use, is one of deployment strategy. At present, most implementations of audio portals as community media platforms are centralised instances deployed by a single organisation or group, with a specific agenda (such as news, healthcare or governance).

Centralised function-oriented deployment

Centralised, function-oriented deployments require content of a certain quality and, as a result, must usually be moderated. Speech-recognition technology, particularly in the area of automatic transcription, is still a far cry from being very accurate. As a result, moderating a function-specific audio portal is still a manual job, for the most part.

Typically, audio portal moderators will need to listen to each message and summarise and/or transcribe it. Beyond transcription, there may be more work to do to improve the quality of the content for the specific purpose of the deployment, like sound quality clean-ups and edits, fact verification (if journalism is the function, for example) and categorisation. All of this work is further exacerbated in a centralised deployment, since all incoming calls come to the same central hub (see Figure 4).

Figure 4: Centralised deployments

In India, and other countries where long-distance call charges are higher than local call charges, centralised platforms also suffer from an added cost element, since all callers must call the central number, regardless of their own locations.

Hyperlocal deployments

An alternative model is a hyperlocal community-oriented one. In this model, an instance of the platform is deployed at the community level and maintained by community members. Such community-level audio portals could be used as voice-based bulletin boards. By managing the size of the user base, and ensuring a manageable user adoption rate by limiting publicity to word of mouth, communities could eliminate the need for moderation by making sure everyone on the platform was known by the others and therefore accountable to the community.

Several communities can then choose to link their platforms, either by sharing content, or by simply listening to each other. This will eventually lead to an organically expanding network, where people can choose which deployments they want to subscribe to, much in the same way as Internet users subscribe to different forums and websites. This would also ease the burden on centralised deployments already in existence, since they could then simply trawl the community bulletin boards for usable content, rather than filter out unusable content on their own incoming stream. As you can see from Figure 5, the hyperlocal model offers more avenues for collaboration and the cross-fertilisation of ideas between communities than the centralised model.

Figure 5: The advantage of hyperlocal deployments

A word of caution: This approach is still experimental, and needs several more deployments before it can be considered a best practice. However, for communities interested in improving their information access and level of participation in mainstream society, this is a very worthwhile experiment to take on. Both systems described here can be installed on a mid-range notebook computer.

The software is all open source and free for non-commercial use. Mobile interfaces like GSM gateways and mobile ATAs are relatively cheap — a Matrix SETU ATA 211G would cost roughly US$ 120, and the Mobigater is priced at about US$ 50. The total cost of setting up a local IVR installation and running it through a year, including the cost of connectivity, is typically less than US$ 200 a year.

Of course, the most important thing to remember while setting up an alternative communication platform is that while technology will certainly provide the tools, the key to success is to build a strong community around your platform, and quickly demonstrate value to the community from participating. This is where most of the hard work lies.

It would be interesting to see how well the open source community in India takes to these projects and how quickly the hyperlocal model can be tested with several more installations.

References

Tags: audio portal, broadband, Cathedral And The Bazaar, cell phones, CGNet, citizen media, communication technology, content management system, Democracy, Developing country, Digital Divide, free speech, FreedomFone, GitHub, GSM, healthcare, India, Interactive Voice Response, Internet penetration, Internet users, IVR, journalists, LFY January 2012, mainstream media, mass media, MIT, mobile phones, open medium, PBX system, SMS, Speech-recognition technology, Swara, text-to-speech system, voice recognition, Web frameworks, Web interface, Web platforms

Categories: Electronics News, Linux News, News

Learn to Animate with Blender, Part 2: Creating Animation Controls

Latest news from Linux for you magazine - Tue, 01/31/2012 - 17:03

This part of the series [Part 1 & 2] deals with creating controls for the fish and animating it.

Blender is great software for animation and is particularly suited for character animation. Following a process of modelling and creating vertex groups for complex models, and then auto-assigning bones using the same nomenclature as for the vertex groups, makes it very easy to set up a working character rig. The Armature tool in Blender is very advanced, yet quite simple to use even for novices. Setting up controls is also quite easy in Blender, compared to software like Maya or Max.

Create controls and add IK

Shift to the orthographic side view by pressing 3 and 5 on the numeric pad. Click just in front of the mouth of the fish. Press the space bar and Add-Mesh-Cube. Scale down the cube as shown in Figures 1 and 2.

Figure 1: Adding a mesh cube

Figure 2: Scaling down the cube

You can create IK in Blender in various ways. The easiest is to first shift to Pose Mode. Select the control cube you just created. Shift-select the head bone. Press Ctrl+I. A pop-up asks you whether you want to assign IK to the active object. Click OK. Go down to the Armature panel and set the chain length to 1.

Add similar cubes to create controls for the fins and the tail. Add IK for all the controls. Shift to the Animation UI and click the render boxes next to the cube icons in the outliner (Restrict/Allow renderability). This will disable rendering of the cubes (since in our renders we want only the goldfish to be displayed and not the controls). Figures 3 to 6 show the process of adding controls and IK, and Figure 7 shows how to toggle rendering of the cube.

Figure 3: Adding IK control to the head

Figure 7: Renaming Ctrls Toggle Render

Testing the controls

Next, it is time to test the controls. While in object mode, move around the cubes to test the deformations on the fish. If you find these are not as expected, shift to the weight painting mode and try to get rid of overlapping weights — remove extra weights and add weight where necessary. On the weight painting slider, the lower part is for lower values and the upper part for higher values.

Creating keyframes for animation

Shift to the Animation UI through the drop-down menu. Select the whole armature at Frame 1. Press ‘I’ for a pop-up menu to key in various attributes. Key the location, rotation and scale on the first frame. Selecting the armature and controls, move the goldfish forward in the viewport.

Shift to frame 250 and key the frame (press ‘I’ and key the location, rotation and scale). Shifting back to Frame 1, test your animation — press Alt+A to play it. You will observe that the goldfish moves forward and the animation loops. Figures 8 and 9 show the insertion of keyframes at 1 and 250. Figure 10 shows the goldfish playing as a loop in the camera view.

Path animation: a good way to animate simple objects

The easiest and the most convincing method of animating simple objects is path animation. It can create both the simplest and the most complex animation, depending on the type of path you create. In Blender, you first create your object, and then your path, after which you parent the object to the path using the Follow Path command, which could be done in various ways. Once your object starts to follow the path, the animation can be fine-tuned in various ways (modify the path’s attributes, or modify the IPO curves in the Curve Editor).

Creating circular motion

While at the Animation UI, press Add to add a nurbs circle. Press F9 to go to the buttons for nurbs and surface attributes. Set the path length to 250. Make it a 3D object. Also enable Curve Path and Curve Follow. Now select the armature, the controls and shift-select the nurbs circle.

Press Ctrl+P. From the pop-up menu, select Follow Path. This makes your fish follow the path defined by the nurbs circle — a circular motion. Play your animation (Alt+A). If you find the fish is not aligned to the path properly, you can fix this by manually rotating and aligning the fish on the nurbs circle. Figures 11 and 12 show how the nurbs circle is used to make a circular path for the goldfish.

Figure 11: Add nurbs circle

Figure 12: Curve and surface parameters

Creating random motion using a hand-drawn path

Create a path. Shift to edit mode. Select the end-point and start to extrude it using E. Make a random path. Once you are done with path creation, select the armature with the fish controls and the parent to the path (Follow Path from the pop-up menu).

Play the animation (Alt+A). Using Shift+D, you could make multiple copies of the same animation and place them randomly before the camera. When you play the animation, it gives the impression of a whole school of fish swimming randomly. This is by far the best way of creating animation using the Animate along path method.

Once you practice creating such animation you could fine-tune it to make it appear more realistic, perhaps by using the cube controls to move the fish fins back and forth, move the tail from side to side, open and close the mouth of the fish as it moves, and so on.

There is no limit to the innumerable ways you can make your animation more realistic. Figure 13 shows you the goldfish moving along randomly created paths.

Figure 13: Random animation using the path

Tags: 2d backgrounds, 3D computer graphics, 3D graphics software, animation, Animation software, Arts, Blender, Blender Animation Series, character animation, Curve Editor, goldfish, graphics, Inverse kinematics, LFY January 2012, Media Production

Categories: Electronics News, Linux News, News

Device Drivers, Part 14: A Dive Inside the Hard Disk for Understanding Partitions

Latest news from Linux for you magazine - Tue, 01/31/2012 - 16:17

This article, which is part of the series on Linux device drivers, takes you on a tour inside a hard disk.

“Doesn’t it sound like a mechanical engineering subject: The design of the hard disk?” questioned Shweta. “Yes, it does. But understanding it gives us an insight into its programming aspect,” reasoned Pugs, while waiting for the commencement of the seminar on storage systems.

The seminar started with a few hard disks in the presenter’s hand and then a dive into her system, showing the output of fdisk -l (Figure 1).

Figure 1: Partition listing by fdisk

The first line shows the hard disk size in human-friendly format and in bytes. The second line mentions the number of logical heads, logical sectors per track, and the actual number of cylinders on the disk — together known as the geometry of the disk.

The 255 heads indicate the number of platters or disks, as one read-write head is needed per disk. Let’s number them, say D1, D2, D255. Now, each disk would have the same number of concentric circular tracks, starting from the outside to the inside. In the above case, there are 60,801 such tracks per disk. Let’s number them, say T1, T2, T60801. And a particular track number from all the disks forms a cylinder of the same number. For example, tracks T2 from D1, D2, D255 will together form the cylinder C2. Now, each track has the same number of logical sectors — 63 in our case, say S1, S2, S63. And each sector is typically 512 bytes. Given this data, one can actually compute the total usable hard disk size, using the following formula:

Usable hard disk size in bytes = (Number of heads or disks) * (Number of tracks per disk) * (Number of sectors per track) * (Number of bytes per sector, i.e. sector size)

For the disk under consideration, it would be: 255 * 60801 * 63 * 512 bytes = 500105249280 bytes.

Note that this number may be slightly less than the actual hard disk (500107862016 bytes, in our case). The reason is that the formula doesn’t consider the bytes in the last partial or incomplete cylinder. The primary reason for that is the difference between today’s technology of organising the actual physical disk geometry and the traditional geometry representation using heads, cylinders and sectors.

Note that in the fdisk output, we referred to the heads and sectors per track as logical not physical. One may ask that if today’s disks don’t have such physical geometry concepts, then why still maintain them and represent them in a logical form? The main reason is to be able to continue with the same concepts of partitioning, and be able to maintain the same partition table formats, especially for the most prevalent DOS-type partition tables, which heavily depend on this simplistic geometry. Note the computation of cylinder size (255 heads * 63 sectors / track * 512 bytes / sector = 8225280 bytes) in the third line and then the demarcation of partitions in units of complete cylinders.

DOS-type partition tables

This brings us to the next important topic: understanding DOS-type partition tables. But first, what is a partition, and why should we partition? A hard disk can be divided into one or more logical disks, each of which is called a partition. This is useful for organising different types of data separately, for example, different operating system data, user data, temporary data, etc.

So, partitions are basically logical divisions and need to be maintained by metadata, which is the partition table. A DOS-type partition table contains four partition entries, each a 16-byte entry. Each of these entries can be depicted by the following ‘C’ structure:

typedef struct { unsigned char boot_type; // 0x00 - Inactive; 0x80 - Active (Bootable) unsigned char start_head; unsigned char start_sec:6; unsigned char start_cyl_hi:2; unsigned char start_cyl; unsigned char part_type; unsigned char end_head; unsigned char end_sec:6; unsigned char end_cyl_hi:2; unsigned char end_cyl; unsigned long abs_start_sec; unsigned long sec_in_part; } PartEntry;

This partition table, followed by the two-byte signature 0xAA55, resides at the end of the disk’s first sector, commonly known as the Master Boot Record (MBR). Hence, the starting offset of this partition table within the MBR is 512 - (4 * 16 + 2) = 446. Also, a 4-byte disk signature is placed at offset 440.

The remaining top 440 bytes of the MBR are typically used to place the first piece of boot code, that is loaded by the BIOS to boot the system from the disk. The part_info.c listing contains these various definitions, along with code for parsing and printing a formatted output of the partition table.

From the partition table entry structure, it could be noted that the start and end cylinder fields are only 10 bits long, thus allowing a maximum of 1023 cylinders only. However, for today’s huge hard disks, this is in no way sufficient. Hence, in overflow cases, the corresponding <head, cylinder, sector> triplet in the partition table entry is set to the maximum value, and the actual value is computed using the last two fields: the absolute start sector number (abs_start_sec) and the number of sectors in this partition (sec_in_part).

The code for this too is in part_info.c:

#include <stdio.h> #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> #include <unistd.h> #define SECTOR_SIZE 512 #define MBR_SIZE SECTOR_SIZE #define MBR_DISK_SIGNATURE_OFFSET 440 #define MBR_DISK_SIGNATURE_SIZE 4 #define PARTITION_TABLE_OFFSET 446 #define PARTITION_ENTRY_SIZE 16 // sizeof(PartEntry) #define PARTITION_TABLE_SIZE 64 // sizeof(PartTable) #define MBR_SIGNATURE_OFFSET 510 #define MBR_SIGNATURE_SIZE 2 #define MBR_SIGNATURE 0xAA55 #define BR_SIZE SECTOR_SIZE #define BR_SIGNATURE_OFFSET 510 #define BR_SIGNATURE_SIZE 2 #define BR_SIGNATURE 0xAA55 typedef struct { unsigned char boot_type; // 0x00 - Inactive; 0x80 - Active (Bootable) unsigned char start_head; unsigned char start_sec:6; unsigned char start_cyl_hi:2; unsigned char start_cyl; unsigned char part_type; unsigned char end_head; unsigned char end_sec:6; unsigned char end_cyl_hi:2; unsigned char end_cyl; unsigned long abs_start_sec; unsigned long sec_in_part; } PartEntry; typedef struct { unsigned char boot_code[MBR_DISK_SIGNATURE_OFFSET]; unsigned long disk_signature; unsigned short pad; unsigned char pt[PARTITION_TABLE_SIZE]; unsigned short signature; } MBR; void print_computed(unsigned long sector) { unsigned long heads, cyls, tracks, sectors; sectors = sector % 63 + 1 /* As indexed from 1 */; tracks = sector / 63; cyls = tracks / 255 + 1 /* As indexed from 1 */; heads = tracks % 255; printf("(%3d/%5d/%1d)", heads, cyls, sectors); } int main(int argc, char *argv[]) { char *dev_file = "/dev/sda"; int fd, i, rd_val; MBR m; PartEntry *p = (PartEntry *)(m.pt); if (argc == 2) { dev_file = argv[1]; } if ((fd = open(dev_file, O_RDONLY)) == -1) { fprintf(stderr, "Failed opening %s: ", dev_file); perror(""); return 1; } if ((rd_val = read(fd, &m, sizeof(m))) != sizeof(m)) { fprintf(stderr, "Failed reading %s: ", dev_file); perror(""); close(fd); return 2; } close(fd); printf("\nDOS type Partition Table of %s:\n", dev_file); printf(" B Start (H/C/S) End (H/C/S) Type StartSec TotSec\n"); for (i = 0; i < 4; i++) { printf("%d:%d (%3d/%4d/%2d) (%3d/%4d/%2d) %02X %10d %9d\n", i + 1, !!(p[i].boot_type & 0x80), p[i].start_head, 1 + ((p[i].start_cyl_hi << 8) | p[i].start_cyl), p[i].start_sec, p[i].end_head, 1 + ((p[i].end_cyl_hi << 8) | p[i].end_cyl), p[i].end_sec, p[i].part_type, p[i].abs_start_sec, p[i].sec_in_part); } printf("\nRe-computed Partition Table of %s:\n", dev_file); printf(" B Start (H/C/S) End (H/C/S) Type StartSec TotSec\n"); for (i = 0; i < 4; i++) { printf("%d:%d ", i + 1, !!(p[i].boot_type & 0x80)); print_computed(p[i].abs_start_sec); printf(" "); print_computed(p[i].abs_start_sec + p[i].sec_in_part - 1); printf(" %02X %10d %9d\n", p[i].part_type, p[i].abs_start_sec, p[i].sec_in_part); } printf("\n"); return 0; }

As the above is an application, compile it with gcc part_info.c -o part_info, and then run ./part_info /dev/sda to check out your primary partitioning information on /dev/sda. Figure 2 shows the output of ./part_info on the presenter’s system. Compare it with the fdisk output in Figure 1.

Figure 2: Output of ./part_info

Partition types and boot records

Now, as this partition table is hard-coded to have four entries, that’s the maximum number of partitions you can have. These are called primary partitions, each having an associated type in the corresponding partition table entry. These types are typically coined by various OS vendors, and hence sort of map to various OSs like DOS, Minix, Linux, Solaris, BSD, FreeBSD, QNX, W95, Novell Netware, etc., to be used for/with the particular OS. However, this is more a formality than a real requirement.

Besides this, one of the four primary partitions can be labelled as something called an extended partition, which has a special significance. As the name suggests, it is used to further extend hard disk division, i.e., to have more partitions. These are called logical partitions and are created within the extended partition. The metadata of these is maintained in a linked-list format, allowing an unlimited number of logical partitions (at least theoretically).

For that, the first sector of the extended partition, commonly called the Boot Record (BR), is used like the MBR to store (the linked-list head of) the partition table for the logical partitions. Subsequent linked-list nodes are stored in the first sector of the subsequent logical partitions, referred to as the Logical Boot Record (LBR). Each linked-list node is a complete 4-entry partition table, though only the first two entries are used — the first for the linked-list data, namely, information about the immediate logical partition, and the second as the linked list’s next pointer, pointing to the list of remaining logical partitions.

To compare and understand the primary partitioning details on your system’s hard disk, follow the steps (as the root user — hence with care) given below:

./part_info /dev/sda ## Displays the partition table on /dev/sda fdisk -l /dev/sda ## To display and compare the partition table entries with the above

In case you have multiple hard disks (/dev/sdb, …), hard disk device files with other names (/dev/hda, …), or an extended partition, you may try ./part_info <device_file_name> on them as well. Trying on an extended partition would give you the information about the starting partition table of the logical partitions.

Right now, we have carefully and selectively played (read-only) with the system’s hard disk. Why carefully? Since otherwise, we may render our system non-bootable. But no learning is complete without a total exploration. Hence, in our next session, we will create a dummy disk in RAM and do destructive exploration on it.Related Posts:

Tags: BIOS, boot, device drivers, DOS, fdisk, Hard disk drive, hard disks, kernel, kernel programming, LFY January 2012, linked-list head, Linux, linux device drivers, Linux Device Drivers Series, MBR, mechanical engineering, Minix, operating system, Partition table, partitions, platters, sectors, storage systems

Categories: Electronics News, Linux News, News

Lisp: Tears of Joy, Part 8

Latest news from Linux for you magazine - Tue, 01/31/2012 - 13:03

Lisp has been hailed as the world’s most powerful programming language. But only the top percentile of programmers use it because of its cryptic syntax and academic reputation. This is rather unfortunate, since Lisp isn’t that hard to grasp. If you want to be among the crème de la crème, this series is for you. This is the eighth article in the series that began in June 2011.

In rare moments of self-reflection, when I allow myself to doubt my skills as a Lisp evangelist, I sometimes wonder if I have left behind some of my fellow programmers who favour the object-oriented style of programming. Just because I have been focusing on Lisp as a functional programming paradigm, it doesn’t mean we don’t have a role for you in our plans of world domination. Read on to know where you fit in.

Functional vs object-oriented (OO) programming

With an OO approach, programmers write code that describes in exacting detail the steps that the computer must take to accomplish the goal. They focus on how to perform tasks, and how to track changes in state. They would use loops, conditions and method calls as their primary flow control, and instances of structures or classes as primary manipulation units. OO tries to control state behind object interfaces.

In contrast, functional programming (FP) involves composing the problem as a set of functions to be executed. FP programmers focus on what information is desired and what transformations are required, by carefully defining the input to each function and what each function returns. They would use function calls, including recursion, as their primary flow control, functions as first-class objects and data collections as primary manipulation units. FP tries to minimise state by using pure functions as much as possible.

According to Michael Feathers: “OO makes code understandable by encapsulating moving parts. FP makes code understandable by minimising moving parts.”

Conard Barski points out that the critics of the OO programming style may complain that object-oriented techniques force data to be hidden away in a lot of disparate places by requiring them to live inside many different objects. Having data located in disparate places can make programs difficult to understand, especially if that data changes over time.

Therefore, many Lispers prefer to use functional techniques over object-oriented techniques, though the two can often be used together — with some care. Nonetheless, there are still many domains in which object-oriented techniques are invaluable, such as in user interface programming or simulation programming.

On the other hand, James Hague, in his assessment of functional programming argues that, “100 per cent pure functional programming doesn’t work. Even 98 per cent pure functional programming doesn’t work. But if the slider between functional purity and 1980s BASIC-style imperative messiness is kicked down a few notches — say to 85 per cent — then it really does work. You get all the advantages of functional programming, but without the extreme mental effort and un-maintainability that increases as you get closer and closer to perfectly pure.”

CLOS

If OO is what gets you going, Common Lisp offers the most sophisticated object-oriented programming framework of any major programming language. It’s called Common Lisp Object System (CLOS). It is customisable at a fundamental level, using the Meta-Object Protocol (MOP). It has been claimed that there’s really nothing like it anywhere else in programming. It lets you control incredibly complex software without losing control over the code.

Let the tears of joy flow…

Object-oriented programming in Common LispWhat is CLOS?#1: It is a layered system designed for flexibility.

One of the design goals of CLOS is to provide a set of layers that separate different programming language concerns from one another. The first level of the Object System provides a programmatic interface to object-oriented programming. This level is designed to meet the needs of most serious users, and to provide a syntax that is crisp and understandable.

The second level provides a functional interface into the heart of the Object System. This level is intended for programmers who are writing very complex software or a programming environment. The first level is written in terms of this second level.

The third level provides the tools for programmers who are writing their own object-oriented language. It allows access to the primitive objects and operators of the Object System. It is this level at which the implementation of the Object System itself is based.

The layered design of CLOS is founded on the meta-object protocol, a protocol that is used to define the characteristics of an object-oriented system. Using the meta-object protocol, other functional or programmatic interfaces to the Object System, as well as other object systems, can be written.

#2: It is based on the concept of generic functions rather than on message-passing.

This choice is made for two reasons:

there are some problems with message-passing in operations of more than one argument;
the concept of generic functions is a generalisation of the concept of ordinary Lisp functions.

A key concept in object-oriented systems is that given an operation and a tuple of objects on which to apply the operation, the code that is most appropriate to perform the operation is selected, based on the classes of the objects.

In most message-passing systems, operations are essentially properties of classes, and this selection is made by packaging a message that specifies the operation and the objects to which it applies, before sending that message to a suitable object. That object then takes responsibility for selecting the appropriate piece of code. These pieces of code are called methods.

#3: It is a multiple inheritance system.

Another key concept in object-oriented programming is the definition of structure and behaviour on the basis of the class of an object. Classes thus impose a type system — the code that is used to execute operations on objects depends on the classes of the objects. The sub-class mechanism allows classes to be defined that share the structure and the behaviour of other classes. This sub-classing is a tool for modularisation of programs.

#4: It provides a powerful method combination facility.

Method combination is used to define how the methods that are applicable to a set of arguments can be combined to provide the values of a generic function. In many object-oriented systems, the most specific applicable method is invoked, and that method may invoke other, less specific methods.

When this happens, there is often a combination strategy at work, but that strategy is distributed throughout the methods as local control structure. Method combination brings the notion of a combination strategy to the surface, and provides a mechanism for expressing that strategy.

#5: The primary entities of the system are all first-class objects.

In the Common Lisp Object System, generic functions and classes are first-class objects with no intrinsic names. It is possible and useful to create and manipulate anonymous generic functions and classes. The concept of “first-class” is important in Lisp-like languages. A first-class object is one that can be explicitly made and manipulated; it can be stored in any location that can hold general objects.

What CLOS is not

It does not make for a great pickup conversation at the bar. I tried. It did not work!

It also does not attempt to solve problems of encapsulation. The inherited structure of a class depends on the names of the internal parts of the classes from which it inherits. CLOS does not support subtractive inheritance. Within Common Lisp, there is a primitive module system that can be used to help create separate internal namespaces.

Classes

The defclass macro is used to define a new class. The definition of a class consists of its name, a list of its direct super-classes, a set of slot specifiers and a set of class options. The direct super-classes of a class are those from which the new class inherits structure and behaviour. When a class is defined, the order in which its direct super-classes are mentioned in the defclass form defines a local precedence order on the class and those super-classes. The local precedence order is represented as a list consisting of the class, followed by its direct super-classes, in the order mentioned in the defclass form. The following two classes define a representation of a point in space. The x-y-position class is a sub-class of the position class:

> (defclass position () ()) > (defclass x-y-position (position) ((x :initform 0) (y :initform 0)) (:accessor-prefix position-))

The position class is useful if we want to create other sorts of representations for spatial positions. The x and y coordinates are initialised to 0 in all instances, unless explicit values are supplied for them. To refer to the x coordinate of an instance of x-y-position, you would write:

> (position-x position)

To alter the x coordinate of that instance, you would write:

(setf (position-x position) new-x)

The macro defclass is part of the Object System programmatic interface and, as such, is on the first of the three levels of the Object System.

Generic functions

The class-specific operations of the Common Lisp Object System are provided by generic functions and methods. A generic function is one whose behaviour depends on the classes or identities of the arguments supplied to it. The methods associated with the generic function define the class-specific operations of the generic function.

Like an ordinary Lisp function, a generic function takes arguments, performs a series of operations and returns values. An ordinary function has a single body of code that is always executed when the function is called. A generic function is able to perform different series of operations and to combine the results of the operations in different ways, depending on the class or identity of one or more of its arguments.

Generic functions are defined by means of the defgeneric-options and defmethod macros. The defgeneric-options macro is designed to allow for the specification of properties that pertain to the generic function as a whole, and not just to individual methods. The defmethod form is used to define a method. If there is no generic function of the given name, however, it automatically creates a generic function with default values for the argument precedence order (left-to-right, as defined by the lambda-list), the generic function class (the class standard-generic-function), the method class (the class standard-method) and the method combination type (standard-method combination).

Methods

The class-specific operations provided by generic functions are themselves defined and implemented by methods. The class or identity of each argument to the generic function indicates which method or methods are eligible to be invoked.

A method object contains a method function, an ordered set of parameter specialisers that specify when the given method is applicable, and an ordered set of qualifiers that are used by the method combination facility to distinguish between methods.

The defmethod macro is used to create a method object. A defmethod form contains the code that is to be run when the arguments to the generic function cause the method that it defines, to be selected. If a defmethod form is evaluated, and a method object corresponding to the given generic function name, parameter specialisers and qualifiers already exists, then the new definition replaces the old.

Generic functions can be used to implement a layer of abstraction on top of a set of classes. For example, the x-y-position class can be viewed as containing information in polar coordinates.

Two methods have been defined — position-rho and position-theta, that calculate the ρ and Θ coordinates given an instance of x-y-position:

> (defmethod position-rho ((pos x-y-position)) (let ((x (position-x pos)) (y (position-y pos))) (sqrt (+ (* x x) (* y y))))) > (defmethod position-theta ((pos x-y-position)) (atan (position-y pos) (position-x pos)))

It is also possible to write methods that update the “virtual slots” position-rho and position-theta:

> (defmethod-setf position-rho ((pos x-y-position)) (rho) (let* ((r (position-rho pos)) (ratio (/ rho r))) (setf (position-x pos) (* ratio (position-x pos))) (setf (position-y pos) (* ratio (position-y pos))))) > (defmethod-setf position-theta ((pos x-y-position)) (theta) (let ((rho (position-rho pos))) (setf (position-x pos) (* rho (cos theta))) (setf (position-y pos) (* rho (sin theta)))))

To update the ρ-coordinate you may write:

> (setf (position-rho pos) new-rho)

This is precisely the same syntax that would be used if the positions were explicitly stored as polar coordinates.

Class redefinition

The Common Lisp Object System provides a powerful class-redefinition facility.

When a defclass form is evaluated, and a class with the given name already exists, the existing class is redefined. Redefining a class modifies the existing class object to reflect the new class definition.

You may define methods on the generic function class-changed to control the class redefinition process. This generic function is invoked automatically by the system after defclass has been used to redefine an existing class; for example, suppose it becomes apparent that the application that requires representing positions uses polar coordinates more than it uses rectangular coordinates. It might make sense to define a sub-class of position that uses polar coordinates:

> (defclass rho-theta-position (position) ((rho :initform 0) (theta :initform 0)) (:accessor-prefix position-))

The instances of x-y-position can be automatically updated by defining a class-changed method:

> (defmethod class-changed ((old x-y-position) (new rho-theta-position)) ;; Copy the position information from old to new to make new ;; be a rho-theta-position at the same position as old. (let ((x (position-x old)) (y (position-y old))) (setf (position-rho new) (sqrt (+ (* x x) (* y y))) (position-theta new) (atan y x))))

At this point, we can change an instance of the class x-y-position, p1, to be an instance of rho-theta-position by using change-class:

> (change-class p1 'rho-theta-position)Inheritance

Inheritance is the key to program modularity within CLOS. A typical object-oriented program consists of several classes, each of which defines some aspect of behaviour. New classes are defined by including the appropriate classes as super-classes, thus gathering the desired aspects of behaviour into one class.

In general, slot descriptions are inherited by sub-classes. That is, slots defined by a class are usually slots implicitly defined by any sub-class of that class, unless the sub-class explicitly shadows the slot definition. A class can also shadow some of the slot options declared in the defclass form of one of its super-classes by providing its own description for that slot.

A sub-class inherits methods in the sense that any method applicable to an instance of a class is also applicable to instances of any sub-class of that class (all other arguments to the method being the same).

The inheritance of methods acts the same way regardless of whether the method was created by using defmethod or by using one of the defclass options that cause methods to be generated automatically.

I hope with this article I have managed to convince OO programmers that Lisp is generous enough to cater to your style of thinking. Stick with me, and I promise that you won’t be disappointed. So far we’ve seen how to fit the nuts and bolts into the engine. Next month, we’ll learn how to paint it a nice shiny red… I am referring to Graphical Programming in Lisp!

References

Let Over Lambda, Doug Hoyte
CLOS: Integrating Object-Oriented and Functional programming, Richard P. Gabriel, Jon L White, Daniel G. Bobrow

Tags: CLOS, Common Lisp, Common Lisp Object System, Conard Barski, control functions, Daniel G. Bobrow, Doug Hoyte, flow control, functional programming, James Hague, Jon L White, LFY January 2012, Lisp, Lisp: Tears of Joy series, Michael Feathers, object interfaces, object-oriented programming, oo approach, oo programming, programming, programming style, recursion, Richard P Gabriel

Categories: Electronics News, Linux News, News

Developing Applications on QT, Part 1

Latest news from Linux for you magazine - Tue, 01/31/2012 - 11:56

This article introduces application development using the Qt GUI framework.

There was a time when all desktop applications were developed from scratch. Then came the concept of code reuse. Static and shared libraries were created for use in application development, but developers didn’t stop at that; they came up with software frameworks. Though libraries and frameworks seem to functionally be the same, there are some major differences between them, which should be understood:

Libraries offer reusability of functionality, whereas frameworks offer reusability of behaviour. For example, a library may provide classes for TCP and UDP sockets, while a framework will provide a class for an abstract socket.
A library may provide functions used for signals/events, but the framework function is how they interact with the system and other components.
Libraries are called from application code, but the framework calls application code — or, you can say, provides services to the code.

Desktop applications

Desktop applications are very platform-specific. An application compiled for Linux cannot be directly executed on another OS without some sort of emulation, due to differences in system calls and libraries between OSs. A big question was how to write platform-independent applications.

One method was to develop libraries for each platform, keeping application code the same and recompiling for each target platform. This does make life easier for developers — but then everything changed with the revolutionary concept of virtual machines from Sun Microsystems, which gave the world the platform-independent Java programming language. Java applications run on the Java Virtual Machine (JVM). Code developed once can be deployed to every platform that has a JVM for it.

But everything comes with a price. In Java’s case, the price was application performance. The performance of Java applications is not as good as of C/C++ applications compiled to platform-native code. Another problem is the large memory footprint. No doubt Java is still a leading language, but these days we have some really big data applications (for example, in biotech) with very high performance requirements. So again, the need is to develop applications compilable to native code. Application development using C is time-consuming, while C++ is a better option. We have some frameworks that support C++ for application development. The Qt framework is one of them.

Introducing Qt

Qt is a cross-platform application framework developed by Trolltech and presently owned by Nokia. Its APIs are for C++. Qt has been extensively used by application developers to develop cross-platform applications.

Qt can help with graphical application development, network applications, database and multimedia applications, handling XML and 3D, painting, drawing and Web access. As far as platform support is concerned, it supports Linux, Mac OS, Windows, Meego, Embedded Linux and Symbian.

Qt architecture

In Figure 1, you can see a minimal architecture diagram.

Figure 1: Qt architecture

The top layer is C++ program code. Below that are Qt classes for GUI, WebKit, databases, etc., and then an OS-specific support layer. Earlier, Qt also supported Java; its Java-based version was called Jambie. As Qt development progressed, it was getting difficult to support both C++ and Java, so the decision was made to support only C++.

Qt installation

Installation methods differ by OS. On Ubuntu 10.04 LTS, I installed Qt via the Synaptic package manager — install the qtcreator package; dependency resolution will install Qt Assistant, Qt Designer, Qt Linguist and Qt Creator.

If you want to try out the latest Qt release, you can download the offline installer from their official website and install it. Just chmod the installer file to make it executable and run it, and then follow the prompts.

A ‘Hello World’ program (non-GUI)

Open a terminal. Create a directory (first) and create a simple “Hello World” program with the following code:

#include<QtCore> int main(){ qDebug() << "Hello world\n"; }

The included header file QtCore contains declarations for classes that do not use a GUI. We will look at these in detail in later articles. For now, just use the qDebug class, which outputs debugging messages to the console and is equivalent to cout in traditional C++ programming.

So how does one compile it? The Trolltech people came out with an easy solution to support cross-platform compilation. First, a project file should be created, and then a Makefile is created using it. Then, just run make to compile the program. When you install QT Creator, a utility called qmake is also installed. This is a cross-platform Makefile generator for Qt. Check out its man pages for more. Run qmake -project to create a project file (.pro extension, with its name that of the containing directory, i.e., first).

$ cat first.pro ###################################################################### # Automatically generated by qmake (2.01a) Mon Nov 28 05:49:29 2011 ###################################################################### TEMPLATE = app TARGET = DEPENDPATH += . INCLUDEPATH += . # Input SOURCES += main.cpp

You can create both applications and libraries; the value of TEMPLATE is app, indicating this is an application. I will cover library development in later articles. The SOURCES entry lists source files in the project, about which more details appear later in this article series.

Now, let us generate the Makefile with qmake. It’s a long file — it may contain up to 200 lines. Run make to create the executable file:

$ make g++ -c -pipe -O2 -Wall -W -D_REENTRANT -DQT_NO_DEBUG -DQT_GUI_LIB -DQT_CORE_LIB -DQT_SHARED -I/usr/share/qt4/mkspecs/linux-g++ -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4/QtGui -I/usr/include/qt4 -I. -I. -o main.o main.cpp make: Circular all <- first dependency dropped. g++ -Wl,-O1 -o first main.o -L/usr/lib -lQtGui -lQtCore -lpthread

In the last line, you can see that our sample program uses the POSIX thread library too. Run the file with ./first to see the output “Hello world”. It’s done! We have successfully compiled our first program.

Sample GUI program

Qt has an option to create UI files using a drag-and-drop method, which we will explore in the next article. For now, let us hand-code a sample simple GUI program:

#include<QApplication> #include<QLabel> int main(int argc, char *argv[]){ QApplication a(argc, argv); QLabel label; label.setText("Hello World"); label.show(); a.exec(); }

In the first example, there was no GUI. The program terminates when main returns. However, in GUI programs, we can’t do that, or the application will not be usable. We want the GUI to run until the user closes the window. To achieve this, run your program in a loop till this happens, so you use the event-loop-based class QApplication. When you create an object of this class and call its exec() function, main never returns. The application keeps on waiting for user input events.

The second header file contains the class QLabel, a simple widget used to display text. Instantiate it and set its text to “Hello World”. When a widget’s show() function is called, then it becomes a window. So the widget label will be seen like a window. In Figure 2, you can see the resultant label window with title bar. You can resize the output window simply with the help of the mouse.

Figure 2: Widget label

In the next articles, we will cover the core classes of Qt. In the meanwhile, I suggest you go through the official documentation.

Feature image courtesy: nokia_fan. Reused under the terms of CC-BY-NC 2.0 License.Related Posts:

Tags: application development, application performance, C/C++, cpp, desktop applications, GUI, Java, LFY January 2012, Microsoft Windows, POSIX, Qt, Qt applications, Qt Creator, Qt Designer, Qt development, Qt GUI applications, Qt Linguist, software frameworks, Sun Microsystems, Symbian, Trolltech, xml

Categories: Electronics News, Linux News, News

Working with MTD Devices

Latest news from Linux for you magazine - Tue, 01/31/2012 - 10:58

This article shows how kernel and application developers (in C) can make use of MTD in Linux.

MTD (Memory Technology Devices) are NAND/NOR-based flash memory chips used for storing non-volatile data like boot images and configurations. Readers are cautioned not to get confused with USB sticks, SD cards, etc., which are also called flash devices, but are not MTD devices. The latter are generally found on development boards, used to store boot loaders, an OS, etc.

Even though MTD devices are for data storage, they differ from hard disks and RAM in several aspects. The biggest difference is that while hard disk sectors are rewritable, MTD device sectors must be erased before rewriting — which is why they are more commonly called erase-blocks. Second, hard disk sectors can be rewritten several times without wearing out the hardware, but MTD device sectors have a limited life and are not usable after about 10^3-10^5 erase operations. The worn out erase-blocks are called bad blocks and the software must take care not to use such blocks.

Like hard disks, MTD devices can be partitioned and can therefore act as independent devices. On a system with one or more MTD devices, device and partition information can be obtained from the /proc/mtd file. A typical /proc/mtd file is as follows:

cat /proc/mtd dev: size erasesize name mtd0: 000a0000 00020000 "misc" mtd1: 00420000 00020000 "recovery" mtd2: 002c0000 00020000 "boot" mtd3: 0fa00000 00020000 "system" mtd4: 02800000 00020000 "cache" mtd5: 0af20000 00020000 "userdata"

A partitioned MTD device can be depicted as in Figure 1, which shows the relation between an MTD device, a partition and a sector.

Figure 1: An MTD device

As already said, MTD write operations are different from usual storage devices. Therefore, before we move further, let’s understand how write operations take place on MTD devices. Figure 2 shows a typical write case.

Figure 2: An MTD write operation

The left-most part shows a sector that has some data at the end. The rest of the sector has not been written since the last erase. A user wants to write “new data 1″ to this sector at offset 0. Since this part of the sector has already been erased, it is ready to be written and so “new data 1″ can be directly written to the sector. Later, the user may want to write “new data 2″, again at offset 0. To do this, the sector must be erased. Since the sector needs to be erased in entirety, the “old data” must be backed up in a temporary buffer. After erasing the complete sector, the “new data 2″ and “old data” must be written at appropriate offsets.

This procedure is the reason there are specific file systems for MTD devices, like JFFS2 and YAFFFS, and flash translation layers (FTL) like NFTL, INFTL, etc. These FTLs and file systems take special care of MTD device properties to hide complexity from the user.

In the first section that follows, we will look at how to access, read/write and erase MTD devices from Linux applications. The second section describes the same things in kernel space, so that this article can be useful to both application as well as kernel developers.

Accessing MTDs from applications

The user must know the device partition to work upon, which can be found from /proc/mtd as shown earlier. Assuming users want to work on the “userdata” partition, they must use the /dev/mtd5 device.

The first thing to do is to get information about the MTD device. Use the MEMGETINFO ioctl command, as follows:

#include <stdio.h> #include <fcntl.h> #include <sys/ioctl.h> #include <mtd/mtd-user.h> int main() { mtd_info_t mtd_info; int fd = open("/dev/mtd5", O_RDWR); ioctl(fd, MEMGETINFO, &mtd_info); printf("MTD type: %u\n", mtd_info.type); printf("MTD total size : %u bytes\n", mtd_info.size); printf("MTD erase size : %u bytes\n", mtd_info.erasesize); return 0; }

Error handling has been omitted for brevity. The mtd_info_t structure is used with the MEMGETINFO command. The MTD type can be MTD_ABSENT, MTD_RAM, MTD_ROM, MTD_NAND, MTD_NOR, etc., which are defined in the mtd/mtd-abi.h header file. The mtd_info.size indicates the size of the whole device (i.e., the partition, in this case). Finally, mtd_info.erasesize indicates the sector size. During an erase operation, this is the minimum size that can be erased, as we’ll see later.

Reading MTD devices is similar to ordinary devices:

/* read something from last sector */ unsigned char buf[64]; lseek(fd, -mtd_info.erasesize, SEEK_END); read(fd, buf, sizeof(buf));

A write operation can be performed in the same way, provided the sector has been erased previously. Finally, we come to the erase operation. Here is an example of erasing a partition, sector by sector:

void erase_partition(mtd_info_t *mtd_info, int fd) { erase_info_t ei; ei.length = mtd_info->erasesize; for(ei.start = 0; ei.start < mtd_info->size; ei.start += mtd_info->erasesize) { ioctl(fd, MEMUNLOCK, &ei); ioctl(fd, MEMERASE, &ei); } }

All sectors of the device are writeable after this erase operation. Notice the use of MEMUNLOCK before MEMERASE, which is essential to allow the erase operation.

Accessing MTDs from kernel space

This section will repeat the functions explained in the previous section, but in kernel space. This needs a separate section since the erase operation is more complex here – the erase operation may sleep and therefore the kernel programmer has to wait until the operation is completed. This is the case for applications too, but the sleep is transparently taken care of by the scheduler.

As explained earlier, the first MTD information is the mtd_info structure. This is retrieved by iterating through all registered MTD devices:

#include <linux/kernel.h> #include <linux/mtd/mtd.h> #include <linux/err.h> static struct mtd_info *mtd_info = NULL; int init_module(void) { int num; for(num = 0; num < 64; num++) { mtd_info = get_mtd_device(NULL, num); if(IS_ERR(mtd_info)) { printk("No device for num %d\n", num); continue; } if(mtd_info->type == MTD_ABSENT) { put_mtd_device(mtd_info); continue; } if(strcmp(mtd_info->name, "userdata")) { put_mtd_device(mtd_info); continue; } printk("MTD type: %u\n", mtd_info->type); printk("MTD total size : %u bytes\n", mtd_info->size); printk("MTD erase size : %u bytes\n", mtd_info->erasesize); return 0; } mtd_info = NULL; return 0; } void cleanup_module(void) { if(mtd_info) put_mtd_device(mtd_info); }

The above kernel module searches for the “userdata” partition. The function get_mtd_device(), when invoked with the first argument NULL, returns the MTD device associated with the minor number specified in the second argument. On a successful search, it increments the reference count of the device. That’s why, before exiting, a call to put_mtd_device() must be made to release (decrement) the reference count.

Additionally, the module uses the flag MTD_ABSENT (which is available to applications too). This check is required to function correctly with some probing device drivers used to allocate placeholder MTD devices on systems that have socketed or removable media.

Having retrieved the mtd_info structure, reading is relatively simple:

/* read something from last sector */ u_char buf[64]; mtd_info->read(mtd_info, mtd_info.size-mtd_info.erasesize, sizeof(buf), buf);

The second argument of the read function specifies the read offset, and the third the length to read. Note that the read operation too may sleep and, therefore, it must not be performed in an interrupt context. The write operation can be performed as follows (assuming the sector has been previously erased):

/* write something to last sector */ mtd_info->write(mtd_info, mtd_info.size-mtd_info.erasesize, sizeof(buf), buf);

As mentioned before, the read, write and erase operations may sleep. Therefore, kernel code must wait for the operation to finish. Here is an example of erasing the partition and waiting to finish the operation:

#include <linux/sched.h> void erase_partition(struct mtd_info *mtd_info) { unsigned int start; for(start = 0; start < mtd_info->size; start += mtd_info->erasesize) erase_sector(mtd_info, start, mtd_info->erasesize); } void erase_sector(struct mtd_info *mtd_info, unsigned int start, unsigned int len) { int ret; struct erase_info ei = {0}; wait_queue_head_t waitq; DECLARE_WAITQUEUE(wait, current); init_waitqueue_head(&waitq); ei.addr = start; ei.len = mtd_info->erasesize; ei.mtd = mtd_info; ei.callback = erase_callback; ei.priv = (unsigned long)&waitq; ret = mtd_info->erase(mtd_info, &ei); if(!ret) { set_current_state(TASK_UNINTERRUPTIBLE); add_wait_queue(&waitq, &wait); if (ei.state != MTD_ERASE_DONE && ei.state != MTD_ERASE_FAILED) schedule(); remove_wait_queue(&waitq, &wait); set_current_state(TASK_RUNNING); ret = (ei.state == MTD_ERASE_FAILED)?-EIO:0; } } void erase_callback (struct erase_info *instr) { wake_up((wait_queue_head_t *)instr->priv); }

The erase_partition() function iterates over all sectors, and erases them with erase_sector(). At the core of erase_sector() is the mtd_info->erase call, which (as mentioned previously) may sleep. Therefore, erase_sector() prepares a wait queue and a wait queue head.

After a call to mtd_info->erase, the function prepares itself to relinquish the CPU (presuming that mtd_info->erase will sleep) by changing task state to TASK_UNINTERRUPTIBLE and adding itself to the wait queue head. Before relinquishing the CPU, it checks if erase is done, through the ei.state flag. If erase is done successfully, this flag will be set to MTD_ERASE_DONE.

If the erase operation is not complete, the task relinquishes the CPU by calling schedule(). Later, when the erase operation is complete, the driver calls the callback function provided in ei.callback. Here the task wakes up to itself, then removes itself from the wait queue, changes the task state to TASK_RUNNING and finally, the erase_sector() function returns.

MTD devices have many more features that can be used by application programmers. ECC (error correction codes) and OOB (out of band) data are some of them. The MTD framework is integrated into the Linux kernel — therefore it makes working with MTD devices very simple, as we have seen in this article.Related Posts:

Tags: application developers, bad blocks, C, Data storage device, Disk sector, driver, file systems, flash devices, flash memory, flash memory chips, Hard disk drive, kernel, kernel programmer, LFY January 2012, Memory Technology, MTD, MTD Devices, NAND/NOR-based flash, partitions, removable media, scheduler, sd cards, USB flash drive, volatile data

Categories: Electronics News, Linux News, News

Building Image Processing Embedded Systems using Python, Part 1

Latest news from Linux for you magazine - Mon, 01/30/2012 - 23:18

The first part of this three-part series gives a brief overview of the embedded vision and the various components required to make it work. It also covers the installation procedure for the OpenCV library.

Modern life is incomplete without gadgets, smartphones, automated appliances, et al. These electronic devices aide us in our daily grind, making our otherwise mundane/hectic life a bit easier. So what controls these devices? In layman’s language, it’s a small circuit with preprogrammed human logic, called an embedded system. Listed below are some useful definitions.

Embedded system (ES): An embedded system is some combination of computer hardware and software, either fixed in capability or programmable, that is specifically designed for a particular function. Industrial machines, automobiles, medical equipment, cameras, household appliances, air-planes, vending machines and toys (as well as the more obvious cellular phone and PDA) are among the myriad possible hosts of an embedded system.
Image processing: In electrical engineering and computer science, image processing is any form of signal processing for which the input is an image, such as a photograph or video frame. The output of image processing may be either an image, or a set of characteristics or parameters related to the image. Most image-processing techniques involve treating the image as a two-dimensional signal, and applying standard signal-processing techniques to it. In other words, it is basically the transformation of data from a still or video camera into either a decision or a new representation. All such transformations are done to achieve some particular goal. The input data may be a live video feed, the decision may be that a face has been detected, and a new representation may be conversion of a colour image into a greyscale image.
Embedded vision: Embedded vision is the merging of two technologies — embedded systems and image-processing/computer vision (also sometimes referred to as machine vision). Due to the emergence of very powerful, low-cost and energy-efficient processors, it has become possible to incorporate vision capabilities into a wide range of embedded systems. One successful example is the Microsoft Kinect video game controller, which uses embedded vision to track the movements of Xbox 360 users, avoiding the need for handheld controllers. Microsoft sold eight million Kinect units in the first two months after its introduction.

Moving on, the five basic components required to build an embedded system using Python are discussed below, in brief.

The operating system (OS)

The OS is the heart of an embedded vision system. Many different dependencies that are required to run software are provided by the OS, which is the interface between the hardware and the software. Of the many OSs in the market (such as Windows, Linux, etc.) there are reasons why Linux is preferable:

It is open source, i.e., it is free of cost and free, as in “freedom”.
It is virtually virus-free.
It has pretty good support — bug trackers, documentation, forums, mailing lists, IRC, etc.
It is easy on systems resources — that is, more resources can be dedicated to applications.

There are various Linux distributions like Ubuntu, Fedora, etc., to choose from. I personally prefer Ubuntu, because of its easy learning curve. Now, size is the main constraint for an embedded system. Using a PC OS won’t do. A much more portable solution is required, such as the PandaBoardshown in Figure 1.

Figure 1: PandaBoard

Check out its specs:

OMAP4 (Cortex-A9) CPU-based open development platform
OMAP4430 Application processor
1 GB low-power DDR2 RAM
Display HDMI v1.3 Connector (Type A) to drive HD displays, DVI-D Connector
8.89cm audio in/out and HDMI Audio out
Full-size SD/MMC card
Built-in 802.11 and Bluetooth v2.1+EDR
Onboard 10/100 Ethernet
Expansion: 1xUSB OTG, 2xUSB HS host ports, general-purpose expansion header

PandaBoard is such a powerful mobile computing platform that one can port various OSs like Android and Ubuntu 10.04 to it!

Image-Processing (IP) software

There are many software and libraries available for image processing, like MATLAB, OpenCV, etc. Licensing costs for MATLAB are very high, but OpenCV is preferable since:

It is free.
It is fast.
Has good documentation, tutorials, user groups, forums, etc.
There are a lot of prebuilt functions and algorithms to get a head start.
There is active development on interfaces for other languages like Ruby, Python, MATLAB, etc.

OpenCV grew out of an Intel Research initiative to advance CPU-intensive applications. The intent behind OpenCV was to provide a platform that a student could readily use for developing applications, instead of reinventing basic functions from scratch. Figure 2 is a block diagram showing the different modules in OpenCV. Check out the documentation here.

Figure 2: OpenCV modules

Arduino

To make any changes in the external environment, we need a hardware circuit with a “brain”. This role can be filled by any microcontroller or microprocessor. Various companies such as Texas Instruments, NXP, Maxim, Atmel, etc., make microcontrollers.

A microcontroller cannot be directly used in a circuit, as is. A full development circuit board has to be made, providing access to different pins of the microcontroller. This process is not easy, as it involves designing the circuit, soldering, etc. All changes are permanent. Also, a program loader has to be made to load the code from the computer into the microcontroller. This can be quite cumbersome, so Arduino is used, due to the following reasons:

It is open source, so all schematics are available and you can design the same board on your own.
It is based on the concept of breadboard prototyping, which encourages the use of hook-up wires and breadboard, to make a circuit which can be easily modified later. This avoids the hassles of soldering.
There are a number of tutorials, tons of code, forums, etc., to help beginners.
The design of the board and IDE is so simple that one does not have to be an engineer to use it. Even a middle school student can use it.
It is pretty cheap compared to other available development boards.

There are various versions of Arduino to choose from, such as Uno, Mega, Lilypad, etc. I personally prefer the Arduino Mega (Figure 3), which is a microcontroller board based on the ATmega1280.

Figure 3: Arduino Mega

It has 54 digital input/output pins, of which 14 can be used as PWM (Pulse-width modulation) outputs, 16 analogue inputs, 4 UARTs (hardware serial ports), a 16 MHz crystal oscillator, a USB connection, a power jack, an ICSP header, and a reset button. It contains everything needed to support the microcontroller; simply connect it to a computer with a USB cable, or power it with an AC-to-DC adaptor or battery to get started.

Pyserial

This is a library that provides Python support for serial connections (RS-232) over a variety of different devices: old-style serial ports, Bluetooth dongles, infra-red ports, and so on. It also supports remote serial ports via RFC 2217 (since V2.5). We use this for communication with the external hardware (Arduino). This library is easy to use, and has good documentation.

Python

Python is a general-purpose, high-level programming language whose design philosophy emphasises code readability. Python claims to combine remarkable power with very clear syntax, and its standard library is large and comprehensive.

OpenCV installation

Let us now proceed to OpenCV installation. All commands are meant to be run in a terminal.

Install all prerequisites as shown below:sudo apt-get install build-essential sudo apt-get install cmake sudo apt-get install pkg-config sudo apt-get install libpng12-0 libpng12-dev libpng++-dev libpng3 sudo apt-get install libpnglite-dev libpngwriter0-dev libpngwriter0c2 sudo apt-get install zlib1g-dbg zlib1g zlib1g-dev sudo apt-get install libjasper-dev libjasper-runtime libjasper1 sudo apt-get install pngtools libtiff4-dev libtiff4 libtiffxx0c2 libtiff-tools sudo apt-get install libjpeg8 libjpeg8-dev libjpeg8-dbg libjpeg-prog sudo apt-get install ffmpeg libavcodec-dev libavcodec52 libavformat52 libavformat-dev sudo apt-get install libgstreamer0.10-0-dbg libgstreamer0.10-0 libgstreamer0.10-dev sudo apt-get install libxine1-ffmpeg libxine-dev libxine1-bin sudo apt-get install libunicap2 libunicap2-dev sudo apt-get install libdc1394-22-dev libdc1394-22 libdc1394-utils sudo apt-get install swig sudo apt-get install libv4l-0 libv4l-dev sudo apt-get install python-numpy sudo apt-get install build-essential libgtk2.0-dev libjpeg62-dev libtiff4-dev libjasper-dev \ libopenexr-dev cmake python-dev python-numpy libtbb-dev libeigen2-dev yasm libfaac-dev \ libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev
Install Python development headers with sudo apt-get install python-dev.
Download the OpenCV source code. It is recommended that you move the downloaded OpenCV package to your /home/<user> directory. With that as your current directory, extract the archive, as follows:tar -xvf OpenCV-2.3.1a.tar.bz2 cd OpenCV-2.3.1/
Make a new directory called build and cd into it (mkdir build; cd build).
Run Cmake:cmake -D WITH_TBB=ON -D BUILD_NEW_PYTHON_SUPPORT=ON -D WITH_V4L=OFF-D INSTALL_C_EXAMPLES=ON -D INSTALL_PYTHON_EXAMPLES=ON -D BUILD_EXAMPLES=ON ..
Follow this with make and sudo make install (to install the library).
Next, configure the system to use the new OpenCV shared libraries. Edit a configuration file with sudo gedit /etc/ld.so.conf.d/opencv.conf. Add the following line at the end of the file (it may be an empty file, which is okay) and then save it:/usr/local/lib
Close the file and run sudo ldconfig.
Open your system bashrc file (for me, that is sudo gedit /etc/bash.bashrc) and add the following code at the end of the file:PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig export PKG_CONFIG_PATH
Save and close the file, then log out and log in again, or reboot the system.

In the second part of this series, I plan to cover the basics and functions of OpenCV, and then how to develop image-processing programs based on it. In the third (and final) part, I will cover Arduino and Pyserial programming, as well as how to integrate all five components to finally build an embedded system for image processing.

Acknowledgement: I would really like to thank my colleagues Sohil Patel and Samarth Shah for their help and support. I would also like to thank Prof N.P Gajjar and Prof Dhaval Shah for their guidance.Feature image courtesy: Christian Scholz. Reused under the terms of CC-BY-NC 2.0 License.Related Posts:

Tags: Android, Arduino, C, embedded systems, embedded vision, Image processing, LFY January 2012, matlab, OMAP4430, OpenCV, PandaBoard, portable solution, Pyserial, python, signal processing

Categories: Electronics News, Linux News, News

IPsec VPN Penetration Testing with BackTrack Tools

Latest news from Linux for you magazine - Mon, 01/30/2012 - 17:12

This article outlines the value of penetration-testing VPN gateways for known vulnerabilities and also shows you how to prevent a breach into the internal network.

IPsec is the most commonly used technology for both gateway-to-gateway (LAN-to-LAN) and host to gateway (remote access) enterprise VPN solutions. It offers complete data protection for tunnelled traffic, with confidentiality, integrity, data origin authentication and anti-replay services. IPsec uses a lot of components to achieve high-level security. The major protocols that IPsec uses are:

ESP (Encapsulation Security Payload): ESP can provide data confidentiality and integrity, but cannot protect the IP header. The IP protocol number of ESP is 50.
AH (Authentication Header): AH can provide the integrity service to the data packet, but cannot offer confidentiality to data packets like ESP. The IP protocol number of AH is 51.
IKE (Internet Key Exchange): IKE provides support for the negotiation of parameters between end points or VPN peers and thus establishes, maintains and terminates security associations (SA). The SA termination can be based on time (seconds) or transfer (kilobytes) rate. Actually, IKE is a type of ISAKMP (Internet Security Association Key Management Protocol) implementation, which is a framework for authentication and key exchange. IKE establishes the security association (SA) between two endpoints through a three-phase process.
- IKE Phase 1: IKE Phase 1 sets up a secure channel between two IPsec endpoints by the negotiation of parameters like the encryption algorithm, integrity algorithm, authentication type, key distribution mechanism, life time, etc. IKE Phase 1 can either use the main mode or aggressive mode to establish the bidirectional security association. Main mode negotiates SA through three pairs of messages, while aggressive mode offers faster operations through the exchange of three messages.
- IKE Phase 2: IKE Phase 2 is used for data protection. The VPN peers negotiate the IPsec parameters needed for data security with ESP and AH. Finally, a unidirectional SA is built between pairs with a special mode known as Quick Mode. The establishment of the Phase 2 security association can use an entirely different algorithm from what’s used by Phase 1 — the Diffie Hellman Algorithm — for more security. This concept is known as Perfect Forward Secrecy (PFS).
- IKE Phase 1.5: IKE Phase 1.5 or the Extended Authentication Phase is an optional phase and is commonly used in remote access VPN solutions. IKE Phase 1.5 will enhance security by adding end-user-level authentication.

Commercial VPN gateways from different manufacturers like Cisco, Checkpoint, Juniper, Microsoft, etc., are readily available. Some of those vendors offer both hardware- and software-based solutions for IPsec implementations. Quite a few robust open source solutions like Openswan, StrongVPN, etc., can also be used for IPsec implementations.

Does your IPsec VPN solution offer complete protection?

VPN penetration testing will help the organisation to baseline (identify the loopholes that exist in the present implementation and modify the configuration accordingly to protect itself from known problems) its current VPN security posture, identify threats and weaknesses, and implement a new security policy that will mitigate risks.

Setting up the test lab for VPN pen-testing

GNS3 is a great tool for simulating Cisco devices (and other vendor devices like Juniper too). There are many tutorials on the Internet for IPsec remote access and site-to-VPN configurations using GNS3. One such tutorial is available here.

By using a PC with a Core 2 Duo or higher processor with 2 GB RAM, you can completely simulate the test lab. Distros like Ubuntu now have GNS3 in their repository. Command-line installation in Ubuntu is as simple as sudo apt-get install gns3. A complete coverage of the GNS3 setup is beyond the scope of this article. So those who are new to GNS3, consult the project documentation.

For this test lab, you need to simulate a router with IPsec support with two interfaces. Do not forget to add the BackTrack PC to the simulated Internet region of your test lab setup (external interface of router), as shown in the sample topology (Figure 1). All tests will be performed from the BackTrack PC.

Figure 1: Sample topology

The same test lab setup can also be arranged with other solutions like Checkpoint SPLAT (Secure Platform or SPLAT is a software-based gateway solution from Checkpoint Software), Microsoft Server 2003 or 2008 (configured as an IPsec VPN gateway), etc.

Penetration testing an IPsec VPN

Penetration testing an IPsec VPN includes several phases like:

Scanning or identifying the VPN gateway.
Fingerprinting the VPN gateway for guessing implementation.
PSK mode assessment and PSK sniffing.
Offline PSK cracking.
Checking for default user accounts.
Testing the VPN gateway for vendor specific vulnerabilities.

Scanning or identifying the VPN gateway

To determine the presence of an IPsec VPN gateway, the penetration tester needs to port-scan the target. Most IPsec implementations will be ISAKMP-based. ISAKMP is an application layer key-exchange protocol that provides mechanisms to establish, negotiate, modify and delete Security Associations. ISAKMP uses UDP port 500, so a direct UDP port-scan on the suspected VPN gateway may give you the results. You can use Nmap or Ike-scan for this.

Scanning with Nmap

A direct port-scan on the VPN gateway with this powerful open source scanner provides supplemental information on the presence of the VPN gateway. Nmap can later be used at the fingerprinting phase for version or OS identification.

root@bt:~# nmap -sU -p 500 172.16.21.200 Starting Nmap 5.51 (http://nmap.org) at 2011-11-26 10:56 IST Nmap scan report for 172.16.21.200 Host is up (0.00036s latency). PORT STATE SERVICE 500/udp open isakmp MAC Address: 00:1B:D5:54:4D:E4 (Cisco Systems) Nmap done: 1 IP address (1 host up) scanned in 0.17 seconds

The options used were -sU for UDP scan, and -p to only scan the specified port. The scan output shows the ISAKMP port (UDP port 500) open.

Ike-scan

Ike-scan is a simple but powerful command-line tool that is used to find and fingerprint VPN gateways. It sends specially crafted IKE packets to target gateways and enlists any IKE responses that are received. By default, Ike-scan works in main mode, and sends a packet to the gateway with an ISAKMP header and a single proposal with eight transforms inside it.

Each transform contains a number of attributes like DES or 3DES as the encryption algorithm, SHA or MD5 as the integrity algorithm, a pre-shared key as the authentication type, Diffie-Hellman 1 or 2 as the key distribution algorithm and 28800 seconds as the lifetime.

Initial IPsec VPN discovery with Ike-scan is as shown below:

root@bt:~# ike-scan -M 172.16.21.200 Starting ike-scan 1.9 with 1 hosts (http://www.nta-monitor.com/tools/ike-scan/) 172.16.21.200 Main Mode Handshake returned HDR=(CKY-R=d90bf054d6b76401) SA=(Enc=3DES Hash=SHA1 Group=2:modp1024 Auth=PSK LifeType=Seconds LifeDuration=28800) VID=4048b7d56ebce88525e7de7f00d6c2d3c0000000 (IKE Fragmentation) Ending ike-scan 1.9: 1 hosts scanned in 0.015 seconds (65.58 hosts/sec). 1 returned handshake; 0 returned notify

The -M shows each payload in a line, so that the output will be neat and easy to understand. The output can be any of the following:

0 returned handshake; 0 returned notify: This means the target is not an IPsec gateway.
1 returned handshake; 0 returned notify: This means the target is configured for IPsec and is willing to perform IKE negotiation, and either one or more of the transforms you proposed are acceptable.
0 returned handshake; 1 returned notify: VPN gateways respond with a notify message when none of the transforms are acceptable (though some gateways do not, in which case further analysis and a revised proposal should be tried).

In the example shown, the VPN gateway replies with one returned handshake and the acceptable transform set has these parameters:

Enc=3DES Hash=SHA1 Group=2:modp1024 Auth=PSK LifeType=Seconds LifeDuration=28800 Custom transform sets can be tried against the target with the "--trans" switch: --trans=(1=1,2=2,3=1,4=2)

…where 1=Encryption Algorithm, 2=Hash Algorithm, 3=Authentication Method, 4=Group Description, and 5=Group Type.

Kindly refer to RFC 2409 Appendix A for a complete understanding of transform set values. There are a number of other tools like ipsectrace, ipsecscan, etc., available for IPsec scanning, but undoubtedly Ike-scan is one of the best and a frequently updated tool.

Vulnerability assessment tools like Nessus, Nexpose, etc, can be used to identify the vulnerabilities of VPN implementations. A full security audit on the target gateway with such types of tools will generate a detailed report with all identified problems and the mitigation steps available.

Fingerprinting the VPN gateway for guessing implementation

Vendor identification and software detection of the gateway is achieved in the fingerprinting phase. To proceed with fingerprinting, you need to get a handshake message from the gateway, containing the acceptable transform set details. As the default IKE doesn’t offer reliability for transmitted packets, VPN gateway vendors will use their own back-off algorithm to deal with the “lost in transit” traffic.

The attacker sends an initial IKE proposal to the VPN gateway with an acceptable transform set. The attacker doesn’t reply and carefully analyses the server response messages for some time. (The default time Ike-scan waits for back-off fingerprinting is 60 seconds.) By analysing the time difference between the received messages from the server and the matching response pattern, the pen tester can successfully fingerprint the VPN gateway vendor.

Some VPN servers will use the optional Vendor ID (VID) payload with IKE to carry some proprietary extensions. This will really make fingerprinting easy for the attacker. Most of the time, VID is a hashed text string. Ike-scan can use the --vendor switch to add the VID payload to outbound packets. The received VID payload can be displayed by Ike-scan directly, as shown below:

root@bt:~# ike-scan -M --showbackoff 172.16.21.200 Starting ike-scan 1.9 with 1 hosts (http://www.nta-monitor.com/tools/ike-scan/) 172.16.21.200 Main Mode Handshake returned HDR=(CKY-R=4f3ec84731e2214a) SA=(Enc=3DES Hash=SHA1 Group=2:modp1024 Auth=PSK LifeType=Seconds LifeDuration=28800) VID=4048b7d56ebce88525e7de7f00d6c2d3c0000000 (IKE Fragmentation) IKE Backoff Patterns: IP Address No. Recv time Delta Time 172.16.21.200 1 1322286031.744904 0.000000 172.16.21.200 2 1322286039.745081 8.000177 172.16.21.200 3 1322286047.745989 8.000908 172.16.21.200 4 1322286055.746972 8.000983 172.16.21.200 Implementation guess: Cisco VPN Concentrator Ending ike-scan 1.9: 1 hosts scanned in 84.080 seconds (0.01 hosts/sec). 1 returned handshake; 0 returned notify

Note that the Ike-scan fingerprinting can be done without even using the --trans option, but adding it will make the process faster.

So you have been successful in fingerprinting the vendor of the VPN gateway; in this case, it is a Cisco VPN Server like ASA or PIX.

PSK mode assessment and PSK sniffing

The aggressive mode of IPsec does not use a key distribution algorithm like Diffie-Hellman to protect the authentication data exchange. This makes it possible for the attacker to capture the authentication data. A server that works with aggressive mode will send the authentication hash in clear-text mode, which can be captured and cracked offline by tools like ike-crack. In the following example, the penetration tester sniffs the PSK hash and saves it into a file for offline cracking:

root@bt:~# ike-scan --pskcrack --aggressive --id=peer 172.16.21.200 > psk.txt

Ike-probe or Ike-scan can be used to capture authentication data, as the following example shows:

root@bt:~# ike-scan --pskcrack --aggressive --id=peer 172.16.21.200 Starting ike-scan 1.9 with 1 hosts (http://www.nta-monitor.com/tools/ike-scan/) 172.16.21.200 Aggressive Mode Handshake returned HDR=(CKY-R=7eb59f437bbc5445) SA=(Enc=3DES Hash=SHA1 Group=2:modp1024 Auth=PSK LifeType=Seconds LifeDuration=28800) KeyExchange(128 bytes) Nonce(20 bytes) ID(Type=ID_IPV4_ADDR, Value=172.16.21.200) Hash(20 bytes) VID=12f5f28c457168a9702d9fe274cc0100 (Cisco Unity) VID=09002689dfd6b712 (XAUTH) VID=4048b7d56ebce88525e7de7f00d6c2d3c0000000 (IKE Fragmentation) VID=1f07f70eaa6514d3b0fa96542a500100 (Cisco VPN Concentrator) IKE PSK parameters (g_xr:g_xi:cky_r:cky_i:sai_b:idir_b:ni_b:nr_b:hash_r): 41391d84dd47367e7f3182b07ccf3bcf48e0d8c917452ac071bce3673c4352583759e5086a9806ab7c5531944273c25a8722c259c76e5e393a2e48c36bf205d571cfd0eba36c573fe4b94939b867ec4ecf197c23930ed496a73df4a149ea6220029c6658e401de40f7f4fa098606a70ab9483c0eb2ac54258a06dd572ae2cd32:88bf0e2a5a07bd19924583ccef6523cb8f4fa56cd7ce65d015b61b2feeb700f37265de794c51af0a749e29339ee0f581870b7c515279c1672e827c6a686fe70d6cc0d6945ac73f1187764a0ebc333d8dd00c0a4e0ba29a0fc276277bbfdfc2e0b84e71881b5dde8869a57600141b939c1139afa865df52911e6ef866e6319eaf:7eb59f437bbc5445:059885068c28a7c4:00000001000000010000009801010004030000240101000080010005800200028003000180040002800b0001000c000400007080030000240201000080010005800200018003000180040002800b0001000c000400007080030000240301000080010001800200028003000180040002800b0001000c000400007080000000240401000080010001800200018003000180040002800b0001000c000400007080:01110000ac1015c8:bb3c0d7f23234a70d4e125def19bf249cdb299d7:68aeca96d276fba861756a48d79e11cca2623843:229f9468990c4887d2b13e73160c2288e51ff6c9 Ending ike-scan 1.9: 1 hosts scanned in 0.018 seconds (55.19 hosts/sec). 1 returned handshake; 0 returned notifyOffline PSK cracking

Before cracking the captured hashed authentication string offline, edit the output file to include only the hash value. (It should only include 9 colon-separated values.) The offline cracking in Backtrack is done with psk-crack, which supports the dictionary, brute-force and hybrid mode cracking.

There are a number of other tools like Cain and Abel available for the offline PSK hash cracking. The following example shows the dictionary mode of psk-crack:

root@bt:~# psk-crack -d /usr/local/share/ike-scan/psk-crack-dictionary psk.txt Starting psk-crack [ike-scan 1.9] (http://www.nta-monitor.com/tools/ike-scan/) Running in dictionary cracking mode key "ADMIN" matches SHA1 hash c1dc52bbb88d4b434c1050a6e77e923f03afbc82 Ending psk-crack: 136 iterations in 0.001 seconds (153153.15 iterations/sec)

So the VPN gateway is configured with a simple pre-shared key ADMIN!

Checking for default user accounts

Most VPN solutions have the end-user-level authentication Xauth (Phase 1.5 of IKE) or Extended Authentication enabled by default. So with the psk-crack output alone, it will not be possible to get into the internal network. After the initial peer authentication, Xauth is required before the VPN gateway grants access.

Xauth login credentials can be captured by using fiked, a command-line tool that impersonates the VPN gateway’s IKE responder and sniffs the authentication data by intercepting the IKE traffic. You need to redirect IKE traffic to fiked for sniffing, which can be done with the help of ARP spoofing.

Given below is a simple example of fiked. The -g switch specifies the IP address of the gateway, captured data is written to a file with the -l switch, -d is used to run it in daemon mode, and -k is for “group id: shared key” representation:

root@bt:~# fiked -g 192.168.1.50 -k testgroup:secretkey -l output.txt -d

In some cases, the VPN gateway will have default user accounts, which the pen-tester can use for Xauth. If not, extensive social engineering or information gathering will do the trick. You may use a proper IPsec VPN client like the Cisco EasyVPN client for the final verification.

Testing the VPN gateway for vendor specific vulnerabilities

There are hundreds of known IPsec/IKE vulnerabilities. Exploitation of these can cause disruption of VPN gateway services, so testing the gateway for these is very important. The following three websites are very useful; they list the known vulnerabilities of different VPN solutions:

Sophisticated vulnerability assessment tools like MetaSploit Framework Pro, Qualys, Core Impact, etc., can be used to test the VPN gateway against known vulnerabilities, along with custom-created exploit scripts. Security patches, OS upgradation or additional configuration may be needed to mitigate these threats, as guided by the vendor.

By compromising a VPN gateway server, the attacker can gain access to valuable internal resources, so organisations need to make sure that VPN gateways are hardened against these threats. Vulnerability assessment and penetration-testing of the VPN gateway, along with periodic reviews of configured security policies, can help organisations to tighten up overall security.

Feature image courtesy: Walter Logeman. Reused under the terms of CC-BY-NC-ND 2.0 License.Related Posts:

Tags: ARP spoofing, authentication header, authentication type, Backtrack, Cisco, data packets, encryption algorithm, enterprise VPN, IKE, Internet Key Exchange, IP addresses, IP header, IPsec, isakmp, Juniper, LFY January 2012, MAC address, Metasploit, metasploit framework, Nessus, NMap, origin authentication, Penetration testing, Perfect Forward Secrecy, protocol implementation, secure channel, VPN

Categories: Electronics News, Linux News, News

Guard Your Network with IPCop, Part 2: Add-ons

Latest news from Linux for you magazine - Mon, 01/30/2012 - 15:47

A basic installation of IPCop provides minimum firewall functionality such as a proxy, compatibility with various Internet connections, port forwarding, IPSec VPN, etc. To convert it into a complete Unified Threat Management (UTM) box, various additional features are required, which are available as add-ons. In this article, we explore how to install four important add-ons.

IPCop add-ons like Advanced Proxy, URL Filter, Update Accelerator and Calamaris are not officially part of the IPCop distro, but provide excellent additional functions such as advanced proxy, enabling network-based access control and authentication; URL filtering, with automatic blacklist updates; OpenVPN server; blocking outgoing traffic based on ports, etc. The important details, features and download links of our four add-ons are summarised in the following table.

Add-onCurrent VersionFunction & featuresAdvanced Proxy3.0.6Provides various additional functionality over and above the basic proxy:

Seamless GUI integration for advanced Web proxy configuration
Local user authentication including group-based user management
identd (RFC 1413) authentication
LDAP authentication including Active Directory, eDirectory and OpenLDAP
Windows authentication including native Windows domains and Samba
RADIUS authentication
Extended cache management
Web access control by IP and MAC addresses
Download throttling
Time-based access restrictions
Classroom extensions for supervising Web access by classrooms
MIME type filter
Blocking of unauthorised browsers or client software
Automatic client configuration support (PAC and WPAD)

URL Filter1.9.3Block websites just by selecting the unwanted category. Available for both, IPCop and SmoothWall, it’s ready to use — download, install, and run. Following are some of its features:

Seamless GUI integration for configuration and log viewer
Very flexible, block categories are not hardcoded
Custom block categories can be included
Works with all squidGuard-compatible blacklists
Automatic blacklist updates on a scheduled basis
Time, category and client-based constraints (IPCop only)
No reboot required, for installation/removal, nor during operation

Update Accelerator2.1.3

Caching various operating system patches and anti-virus updates
Increases download speed up to a factor of 1.500 for a 64kBit/s ISDN connection.
Guaranteed delivery from local cache, even if the Web Proxy cache has been cleared.
The Update Accelerator cache can be transferred from one IPCop to another for offline preloading.

Calamaris Report Generator2.1.2This one is for generation of categorised proxy reports. Following are some of its features:

Request method (GET, HEAD, …)
Incoming requests (TCP and UDP)
Outgoing requests
Requested first- and second-level domains
Protocol report (http, gopher, ftp, …)
Requested content-types and file extensions
Size based distribution of objects
Performance in defined time ranges

Note: Advanced Proxy is a prerequisite for URL Filter and Update Accelerator.

The setup assumes IPCop (green) IP as 192.168.51.1, the IPCop Web access port to be 445 and the IPCop SSH access port of 222, with SSH access enabled (System –> SSH Access in the IPCop Web GUI). To copy add-on binaries, you need to use SCP, and for installation you need direct console access or SSH access from another system. Linux users can use the scp and ssh utilities. Windows users can download and install WinSCP and Putty for these purposes. That done, download the various add-on binaries from the links provided in the table above to your desktop. Secure-copy (SCP) them to /root on the IPCop box. Get a command prompt on the IPCop box via SSH (or Putty).

Extract each of the tarballs with tar -xzf <tarballname>. Change to each of the extracted folders in turn (ipcop-advproxy, ipcop-urlfilter, ipcop-updatexlrator, ipcop-calamaris), and run ./install in each of them to install that add-on.

Subsequent configuration is via the IPCop Web GUI being prerequisite for various other add-ons, install ADV Proxy first followed by others.

AdvProxy

In the Web GUI, go to Services –> Advanced Proxy. The important settings under various categories are given below.

Common settings:
1. Enable this add-on on the Green network (and any others if needed).
2. Whether or not to use Transparent mode (no change in client browser connection settings is required, though the browser must be configured to use the IPCop proxy port).
3. Proxy port (default TCP 800).
Other settings include:
1. Upstream Proxy: To be used if the Internet connection is via a proxy server. Here, the username and password can also be provided.
2. Log settings: To enable/disable proxy logs.
3. Cache management: Define cache size.
4. Network-based access control: Allows you to control Internet access only to the defined subnets (or IP addresses). For example, 192.168.51.0/27 will allow Internet access to 192.168.51.1 to 192.168.51.30. (Some clients can be banned by entering their IPs under ‘Banned IP addresses’.)
5. Time restrictions: Internet access can be allowed only during certain time periods.
6. Authentication methods: IPCop supports user authentication methods such as Local (IPCop username and passwords), LDAP/RADIUS (external LDAP server), Windows (Windows Domain Controller), etc.

Figure 1: Advanced Proxy settings

URL Filter

To enable URL Filter, go to the bottom of Services –> Advanced Proxy and select Enable URL Filter. Go to Services –> URL Filter for more settings. Various configuration items and features are listed below:

Block Categories — Choose the unwanted category to block corresponding websites. The blacklist database can be scheduled to be updated daily/weekly or monthly. The default list has only a few block categories. Once updated, you will see a detailed list to block from.
Black list and White list — If a blacklisted website is to be accessible, add it to Custom Whitelist; to ban an accessible website, add it to Custom Blacklist.
Custom Expressions list — Add words to be blocked. For an example, add cricket, score and scores under this to block sites featuring these words.
File Extension Blocking — Block executable, compressed or Audio/Video file downloads by selecting the corresponding check-box.
Network-based access control — Lets some users browse the Web unrestricted, and can block others from using the Web at all.
Block Page settings — The message a user receives when trying to access a blocked website.
Log — Enable it to track who is trying to access blocked websites.
URL Filter Maintenance — Blacklist update settings, configure a daily/weekly/monthly update schedule and choose from four sources.
Backup URL Filter — Backup settings and complete blacklist, which can be restored later, or on a new IPCop installation.

Figure 2: URL Filter block categories

URL Filter allows three categories of Internet access based on the IP address — filtered access, unrestricted access and no access (banned). One very important provision is that all sites from the custom whitelist can be accessed by banned IP addresses if you enable “Allow custom whitelist for banned clients”. This can be very helpful if all users need to access some websites.

Update Accelerator

Enable it at the bottom of Services –> Advanced Proxy and go to Services –> Update Accelerator. This requires only a few settings. Select Enable Log, Enable Passive Mode and Lower CPU priority for downloads. You may also define a maximum download rate.

This is very useful; it caches various large downloads like updates for anti-virus and OS patches, etc. Repeat requests are supplied from local cache, saving bandwidth and increasing download speed tremendously. To clear cache, either manually delete individual files, or set it to automatically delete unused files, via the Maintenance button in Services –> Update Accelerator.

Calamaris report generator

This add-on requires no configuration. Go to Logs –> Proxy Reportsto access the report generator. Calamaris can generate reports based on parameters like Domain, Performance, Contents, Requester, etc. The time needed for report generation may vary based upon the CPU, hard disk and log size. Reports can be viewed on-screen or exported to text files (see Figures 3 and 4).

Figure 3: Proxy reports options

Figure 4: Sample proxy report

So folks, this adds four important add-ons to the vanilla IPCop. Watch out for further details!Related Posts:

Tags: Advanced Proxy, cache management, Caching, Calamaris Report Generator, firewall, Internet connections, IPCop, IPCop add-ons, ISDN, LFY January 2012, outgoing traffic, proxy server, putty, SmoothWall, ssh, ssh utilities, UDP, Unified Threat Management, Update Accelerator, URL Filter, VPN

Categories: Electronics News, Linux News, News

Linux on POWER

Latest news from Linux for you magazine - Fri, 01/27/2012 - 23:42

Anyone familiar with GNU/Linux will not be surprised by the fact that this operating system runs on almost all known processors. However, very few people are aware that mere support just might not be enough. You’ll also need to keep an up-to-date repository of code. This is especially true when it comes to serious hardware such as POWER.

I guess that almost all desktop systems in the office or home are derived from one prototype, the IBM PC (personal computer), which was built around the concept of an open architecture that IBM introduced 30 years ago. It’s true that the company itself didn’t quite understand how revolutionary open hardware architecture would be a few years later, in the mid-80s. Whatever the case, the fact remains that the PC became popular because many independent manufacturers could produce a wide range of compatible peripherals. This, in turn, allowed a competitive market to develop, eventually making desktop PCs available to everyone.

The open development model of Linux first embraced the desktop market of x86 computers, and then proceeded to “capture” other architectures as well (now more than 20), ranging from x86, ARM and MIPS, to Alpha, SPARC, PowerPC, IA64 and S390.

While “desktop” platforms (x86 and ARM) have many different Linux distributions and other operating systems that can run on them, the “pure” server architectures like IA64, S390 and Power cannot boast of this — the number of distributions is much, much smaller. This fact is understandable. Quality and the cost of server hardware for industrial applications is immeasurably higher than for the usual PC. Moreover, not every Linux distribution vendor can find the resources to build and provide long-term support for major server hardware architecture — even if they are willing to.

Some time ago, I got my hands on an IBM IntelliStation POWER 285 workstation, considered the closest model to large industrial servers. The main brain of the station is a dual-core POWER 5+ processor.

Although announced back in 2005, the system is serious stuff even now. Judge for yourself: 4 GB RAM (max capacity 32 GB), with a disk subsystem that has 2×146 GB SCSI drives (max disk storage 1.2 TB). It has a self-diagnostic system and memory “self-healing” technology known as ChipKill, a remote HMC (Hardware Management Console) and a pair of NICs working at 1 Gbit. This system is unbeatable for an entry-level enterprise server and simply forces you to take another respectful look at IBM — Big Blue creates innovative and striking products.

Well, that’s enough of an introduction. Let’s learn Linux installation and operation on IBM hardware. And later decide how it is possible to effectively exploit Linux on Power.

Remote access

IBM produces equipment that has OpenFirmware (OF) inside. This is analogous to the traditional PC BIOS used on x86 systems. With OF’s help, you can assign which source should be used to start an OS, perform a number of diagnostic procedures, etc. OF itself uses a ServiceProcessor — a special unit embedded into the big tower. When you plug in the power cord, first the ServiceProcessor starts. That initially sets up a remote access option, and then awaits the user’s response.

The user has to press the hardware Start button on the front panel. So, when plugged in, IntelliStation is never completely turned off — even with the main CPUs off, the ServiceProcessor doesn’t sleep. The scheme is reminiscent of a similar product — iLO, developed by Compaq/HP. iLO also lets one start and control a server remotely.

It’s worth noting that IBM also has additional functions run by HMC (Hardware Management Console), among which are planning and allocation of physical resources to logical partitions (LPAR/DLPAR), splitting main CPU performance onto logical CPUs (Capacity On Demand) and related actions that provide virtualisation management on POWER-based processors. Unfortunately, I can only mention these capabilities, and not go any deeper into the subject, because that would be beyond the scope of this review.

So, in order to establish an access link to the ServiceProcessor and later install Linux, you will need an RS232(F)<->RS232(F) cable and a dedicated workstation, from which you can directly control your IntelliStation 285. As a console, any PC with an installed command-line tool like minicom or surecom (a GUI alternative) will suit. Set the communication speed to 19,200 bit/s, the standard default rate for IBM hardware, plug in the power cord and wait for the login prompt after running minicom -D /dev/ttyS0 -s 19200 -o. See Figure 1.

Figure 1: Control Power hardware via RS232 cable

To start the whole system and gain access to OF, from where you can select an OS source, choose: 1. Power/Restart Control -> 1. Power On/Off System -> 8. Power on. After that, in approximately two minutes, you will hear the noise of the fans — the big system has started initialisation and a minute later the hardware will be completely initialised, signalled via two speaker beeps.

At the same time, on the console, you should see a message from SMS (System Management Services) as in Figure 2, where you choose 1 to enter the system OF menu and then select 1. Select Install/Boot Device to choose the boot device (which you will run the installer from).

Figure 2: SMS greeting before entering OpenFirmware menu

You should see something like Figure 3, where I chose 4. IDE, because the built-in CD-ROM is connected to the IDE connector. In your version, it might be 3. CD/DVD.

Figure 3: Choose installation source

In the subsequent menu (Figure 4), I chose 2. Normal Mode Boot, which is suitable for installation, and confirmed exit from SMS. My SLES 11′s Linux kernel correctly identified the IntelliStation 285.

Figure 4: Choose boot mode

The installation process will proceed as follows:

Welcome to SuSE Linux Enterprise 11! Type "install" to start the YaST installer on this CD/DVD Type "slp" to start the YaST install via network Type "rescue" to start the rescue system on this CD/DVD Welcome to yaboot version r22.8-r1190.SuSE booted from '/pci@800000020000003/pci@2,3/ide@1/disk@0:1,\suseboot\yaboot.ibm' running with firmware 'IBM,SF240_403' on model 'IBM,9111-285', serial 'IBM,0206ABB30', partition '0' Enter "help" to get some basic usage information boot: Please wait, loading kernel... Allocated 01800000 bytes for executable @ 03000000 Elf32 kernel loaded... SuSE Linux zImage starting: loaded at 03000000-0469cd30 (4000000/0/02039a68; sp: 02c9fd50) uncompressing ELF header done. (00000100 bytes) Allocated 0164c090 bytes for kernel @ 04800000 Leave 0122a5ba bytes for initrd @ 034668bb uncompressing kernel done. (00f5a920 bytes) entering kernel at 04810000(34668bb/122a5ba/02039a68)

The installation procedure reminds me of what happens on the x86, with just one difference — in our case, control is via a serial line in text VT102 mode, so some characters may not be displayed properly as you can see in Figure 5.

Figure 5: SUSE Linux installation could be fun

Several menu items, such as the network address, may require you to return to re-edit these fields. We can use Tab/Alt+Tab to move between text interface fields. These keyboard shortcuts are functional even in this very minimal mode.

Note: Installing SLES 10 involved no difficulty in navigation or in selecting the text buttons. However, with SLES 11, there was the problem of having to check the Internet connection (Test Connection to the Internet Via Ethernet Controller). The problem is that there is no systems setting to exclude or interrupt this operation when you are in text mode via a serial line. In this case, simply restart the system and assign the necessary IP address after a reboot (via serial connection and minicom). After that, you are encouraged to connect via SSH over the Ethernet connection.Embedded Linux on ServiceProcessor

In addition to control via a serial line, IntelliStation (and in general, all IBM products) offers management via an ordinary Web browser. This Web control mechanism (Advanced System Management — see Figure 6) is provided by a tiny computer within the IntelliStation, with Linux and BusyBox utilities.

Figure 6: Management via Web interface

This ServiceProcessor also uses Power technology — a PowerPC 405 CPU from IBM [Datasheet PDF] but with a much smaller form factor. This uses a low-power processor FSP (Flexible Service Processor) running at about 200 MHz (see Figure 7).

Figure 7: Embedded Linux runs on PowerPC 405 from IBM

ASM provides the same controls as the serial-line SMS but in a more usable manner (Figure 8). It lets you manage the system remotely — start/stop it, view logs, allocate resources (like Capacity on Demand parameters), etc.

Figure 8: ASM's user-friendly controls presentation

Let’s briefly see how this subsystem is launched. When you connect the mains power to the box, the ServiceProcessor automatically starts the embedded Linux. After initially testing some components of a larger block, ServiceProcessor displays on a front panel the readiness status for full system start. When you hit the “Power On” button, OpenFirmware loads and ultimately passes control to your Linux boot loader, yaboot.

Like with the large system, this small subsystem has two independent Ethernet controllers, referred to as HMC 1 and HMC 2. It makes sense to connect them to separate LANs/VLANs for a special management network. First, it helps access restriction and security. Second, you can avoid a network traffic overload on a single LAN.

IP addresses for these ports can be supplied by a DHCP server on a management network. If one isn’t found, default addresses are automatically assigned: 192.168.2.147 (port C7-T1, on HMC1) and 192.168.3.147 (port C7-T2, HMC2). You can log in via telnet, with the default credentials admin/admin.

$ mount /dev/mtdblock/0/3 on / type cramfs (rw) none on /dev type devfs (rw) proc on /proc type proc (rw) /dev/mtdblock/0/1 on /opt/extucode type jffs2 (rw) /dev/mtdblock/0/1 on /etc type jffs2 (rw) /dev/mtdblock/tmpram/tmpram on /tmp type jffs2 (rw) none on /var type tmpfs (rw) /dev/mtdblock/pcscore/pcscore on /core type jffs2 (rw) /dev/mtdblock/pcsram/pcsram on /opt/p1 type jffs2 (rw) shm on /var/shm type shm (rw) /dev/mtdblock/1/1 on /alt/extucode type jffs2 (rw) /dev/mtdblock/2/3 on /opt/fips type cramfs (rw) /dev/mtdblock/nvram/nvram on /opt/p2 type zofs (rw,sync) $ uptime 8:49am up 1 day, 14:53, load average: 0.52, 0.20, 0.07 $ uname -a Linux fsp 2.4.18-rc4 #1 Wed Mar 25 15:01:22 UTC 2009 ppc unknown $ cat /proc/version Linux version 2.4.18-rc4 (root@mcpbuild6) (gcc version 3.2.3 20030401 (prerelease)) #1 Wed Mar 25 15:01:22 UTC 2009

As we see, Embedded Linux uses JFFS/CRAMFS filesystems, which are common in tiny products. An interesting feature is NVRAM memory mounted as storage (/dev/mtdblock/nvram/nvram). Reading data from its files gives the status of different subsystems. You can also change some values using commands from /opt/fips/bin — particularly, you can even try to launch several subsystems (or maybe the full system box) with /opt/fips/bin/powr_fsp_spcn, which checks the current system status, and accordingly launches /bin/powr_L4 or /bin/powr_sqh. The ASM Web server (named webs), when you click the button Save settings and power on does the same.

Available distros for POWER

Originally, I planned to use Red Hat Enterprise Linux as the main OS on this system. However, neither RHEL 5.7 nor 6.2 Beta, nor even 6.2 started correctly. RHEL 6.2 Beta and a more recent version 6.2 spews a kernel trap error during initial startup. Yes, the IntelliStation 285 wasn’t in the supported hardware list, but neither was it for SLES 10/11. Still, SLES 10 and 11 support it unofficially. Let us see how the installation process goes, and whether we can bring the system to production mode.

In addition to RHEL and SLES, numerous systems based on the POWER processor even supported Ubuntu 10.04 (LTS edition). However, my experience proved that this model wasn’t suitable for Ubuntu — it cannot start properly. Moreover, newer versions of Ubuntu come without any POWER/PowerPC architecture support at all, unfortunately.

I checked if the most recent release (6.0.3) of Ubuntu’s parent distribution, Debian, could boot. It wasn’t able to start — it drops directly into OpenFirmware. Another failure! Well, let’s not despair. Quite possibly, the Debian community will fix it.

Oddly enough, a completely amateurish project called Crux PPC has a fully working distribution — Crux PPC 2.7a, which not only loads on Power hardware, but also provides video output via correctly-set framebuffer mode. None of the other distributions mentioned can boast of that! All provided only a serial-line text mode. In general, Crux PPC runs as a LiveCD and has no installer, so I advise professionals to use it. Again, this topic is out of the scope of this article, so I will leave it for later.

Performance

As I mentioned a bit earlier, IBM products are one level, or a “head above” their competitors. Comparing the very different Power and x86 architectures is hard, because there are no objective criteria for such a comparison. However, from an ordinary user’s perspective, we can try the 7Zip archiver in its benchmark mode. This program is completely open source, so anyone can compile it.

Let us compare IntelliStation based on the POWER5+ processor with 2 cores and 4 GB RAM, manufactured in 2005, with an Intel Core 2 Duo (2 cores with hyper-threading) and 4 GB RAM, but released three years later (2008). First, the IntelliStation:

linux:~/src/p7zip_9.20.1/bin # ./7za b 7-Zip (A) 9.20 Copyright (c) 1999-2010 Igor Pavlov 2010-11-18 p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,2 CPUs) RAM size: 3632 MB, # CPU hardware threads: 2 RAM usage: 425 MB, # Benchmark threads: 2 Dict Compressing | Decompressing Speed Usage R/U Rating | Speed Usage R/U Rating KB/s % MIPS MIPS | KB/s % MIPS MIPS 22: 1931 99 1891 1878 | 24309 100 2198 2194 23: 1871 100 1916 1907 | 24132 100 2212 2209 24: 1816 99 1964 1953 | 23917 100 2221 2219 25: 1780 99 2043 2032 | 23525 100 2213 2212 ---------------------------------------------------------------- Avr: 99 1954 1942 100 2211 2209 Tot: 100 2082 2076

Next, let us test the Core 2 Duo system:

Intel(R) Core(TM)2 Duo CPU E6550 @ 2.33GHz 7-Zip 9.04 beta Copyright (c) 1999-2009 Igor Pavlov 2009-05-30 p7zip Version 9.04 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,2 CPUs) RAM size: 4006 MB, # CPU hardware threads: 2 RAM usage: 425 MB, # Benchmark threads: 2 Dict Compressing | Decompressing Speed Usage R/U Rating | Speed Usage R/U Rating KB/s % MIPS MIPS | KB/s % MIPS MIPS 22: 2588 138 1831 2518 | 33730 165 1850 3045 23: 2532 136 1903 2580 | 32265 159 1853 2954 24: 2588 142 1958 2783 | 32828 165 1843 3046 25: 2540 144 2011 2900 | 30299 155 1838 2849 ---------------------------------------------------------------- Avr: 140 1925 2695 161 1846 2974 Tot: 150 1886 2834

As you can see, the performance doesn’t differ much. Even a slightly outdated IBM station can compete with a relatively new x86 product.

Despite the fact that equipment based on the Power processor is marketed by IBM as a corporate power-horse for commercial operating systems like AIX and IBM i (produced by IBM itself), we have seen that it can also be used with Linux. Thus, virtually the entire software stack that successfully works on x86 can also be run on Power — Apache, Java, WebSphere, MySQL, etc.

Taking into account the fact that IBM hardware usually comes with unique technologies like the hardware management console and the hardware virtualisation hypervisor, you can easily create an effective solution by combining the best features from both the hardware and software sides — the corporate and the public world.Related Posts:

Tags: ELF, Flexible Service Processor, GNU/Linux, hardware architecture, IBM, IBM Hardware Management Console, IBM IntelliStation, IBM PC, Intel, Java, LFY January 2012, Linux, Linux distributions, open architecture, POWER, PowerPC, PPC, s390, server hardware, SUSE Enterprise Linux, ubuntu

Categories: Electronics News, Linux News, News

Postgres-XC — A PostgreSQL Clustering Solution

Latest news from Linux for you magazine - Fri, 01/27/2012 - 18:39

What started with a simple relational database system, is expanding its horizons by developing new technology that satiates the ever-increasing need for more data storage, greater transaction throughput and higher availability. Using a cluster to solve these scalability problems is a present trend. This article talks about Postgres-XC, a clustering solution based on the popular PostgreSQL RDBMS.

A cluster is a collection of commodity components that provide scalability and availability at a low cost to the consumer. A database cluster is a collection of database servers that store and process data using commodity hardware, satisfying the need for more data storage, higher throughput and providing high availability. Postgres-XC is such a database cluster system; it is based on PostgreSQL, and follows the same open source model.

The Postgres-XC project began in 2009, through a collaboration between NTT and EnterpriseDB. The goal was to build an open source clustering solution based on PostgreSQL with 100 per cent compatible client APIs. Having PostgreSQL-compatible APIs allows existing PostgreSQL applications to use Postgres-XC with little (or no) change. The licensing terms of this project are the same as that of PostgreSQL.

Postgres-XC architecture

Postgres-XC is a write-scalable, synchronous multi-master, transparent PostgreSQL clustering solution based on shared-nothing architecture. It is a collection of tightly coupled database components, which can be installed on one or more physical or virtual machines. The components do not share any resources such as disk, cache or memory.

Write-scalability means that Postgres-XC can be configured with as many database servers as needed; Postgres-XC is able to handle more writes than a single PostgreSQL server.
Multi-master implies that clients can connect to multiple database servers, and that each database server provides a single, consistent, cluster-wide view of the database.
Synchronous means that a write from any of the masters is immediately visible to other transactions running on other masters.
Transparent means that applications do not have to worry about how the data is stored in multiple database servers, internally.

Figure 1: Postgres-XC architecture

Figure 1 gives the architectural overview of Postgres-XC with its three main components:

Global Transaction Manager (GTM) gathers and manages information about transactional activities in Postgres-XC, issues global transaction identifiers to transactions (to maintain a consistent view of the database on all nodes), and provides ACID properties. It provides support for other global data, such as sequences and timestamps. It stores no user data, except control information.
Coordinators (masters) provide a point of contact for the application/client. They are responsible for parsing and executing queries from the clients, and returning the results (if needed). They do not store any user data themselves, but gather the data from datanodes, with the help of SQL queries fired through a PostgreSQL-native interface. The coordinators also process the data if required and even manage the two-phase commit. Although coordinators do not store user data, they use the catalogue data to parse queries, resolve symbols, plan queries, locate data, etc.
Datanodes store user data and catalogues. The datanodes execute the queries received from the coordinator and return results to the coordinator.

Distribution of data and scalability

Postgres-XC allows two ways of storing the tables on the datanodes:

Distributed tables: A table is distributed on a given set of datanodes using strategies like hash, round-robin, or modulo partitioning. Every row of a distributed table resides on a single datanode. Multiple rows can be modified or written in parallel to various datanodes; we can also read the rows from various datanodes in parallel. Performance is greatly improved by parallel writes and reads from different datanodes.
Replicated tables: A table is replicated on a given set of datanodes using statement-level replication, which performs better than log-based replication, since the size of the logs that must be shipped is much greater than the size of the statement. In the case of a replicated table, a row in the table resides on each datanode on which the table is replicated. Any modifications to the row must be duplicated to each replicated copy. Since all the data in the table is available on a single datanode, the coordinator can gather all the data from a single node and in some cases, act as a proxy between the client application and the datanode. This allows multiple queries on the same table to be directed to different datanodes, thus balancing the load and increasing the read throughput.

Figures 2 and 3 depict the read and write concepts for distributed and replicated tables, respectively.

Figure 2: Distributed tables

Figure 3: Replicated tables

High availability

To achieve high availability, one needs data redundancy, component redundancy and automatic failover. In Postgres-XC, data redundancy can be achieved by using the PostgreSQL native replication with Hot Standby for datanodes. Since each coordinator is a master (capable of writing data) and is capable of reading writes performed by any other coordinator instantaneously, every coordinator is capable of replacing any other, should that coordinator fail. GTM-standby acts as a redundant component for GTM. However, third-party tools are required for automatic failover of all the three types of components.

Performance evaluation

Initial transaction throughput measurements carried out using the DBT-1 benchmark have shown significant throughput scalability, as shown in Figure 4.

Figure 4: Performance evaluation

The figure plots the Scaling Factor versus the Number of Servers in Postgres-XC. The Scaling Factor is the ratio of the number of transactions completed per unit time by Postgres-XC, to that completed by PostgreSQL. A Server comprised of a coordinator and a datanode run on single machine. This benchmark demonstrated an improvement in throughput of approximately 6 times, when using 10 servers.

Release management & development processes

The Postgres-XC project is hosted on SourceForge. The Postgres-XC team tries to release a minor version of Postgres-XC every three to four months to ensure that the latest Postgres-XC features are available to users. The team also tries to make the latest PostgreSQL features available in Postgres-XC by doing frequent merges with the latest stable release of PostgreSQL.

The last release (0.9.6) of Postgres-XC supports most of the SQL syntax and features of PostgreSQL 9.1. The team is currently working on the first major release of Postgres-XC 1.0, due in 2012, with maximum PostgreSQL compliance. Some of the other features like the dynamic addition and removal of components, global deadlock detection, global constraint support, etc, will be targeted for major release after 1.0.

The development team follows the open source development model, where the issues, features or any other development related items are discussed on the public mailing list: postgres-xc-developers@lists.sourceforge.net. The postgres-xc-general@lists.sourceforge.net mailing list is used to discuss other Postgres-XC matters and to solicit help about Postgres-XC.

Wish to contribute?

The Postgres-XC team needs help with feature development, bug fixing, creating installers and distribution packages, testing, and evaluation of Postgres-XC on real applications. To be part of the Postgres-XC community, feel free to contact the Postgres-XC team at the appropriate mailing list.Related Posts:

Tags: cluster system, clustering solution, data storage, database, database clusters, database servers, disk cache, EnterpriseDB, high availability, LFY January 2012, Postgres-XC, PostgreSQL, relational database system, scalability, scalability problems, SQL

Categories: Electronics News, Linux News, News

Partitioning in PostgreSQL

Latest news from Linux for you magazine - Fri, 01/27/2012 - 15:37

Partitioning refers to splitting a large table into smaller pieces. This article covers the basics of partitioning in PostgreSQL.

Currently, PostgreSQL supports range and list partitioning via table inheritance. Basically, you have to create each partition as a child table of the master table. Generally, if you want to split data into specific ranges, then use range partitioning.

Date and timestamp values are good examples for this. For example, you may want to split yearly data into quarterly partitions. Similarly, if you want to split data based on a specific list of values, then use list partitioning. For this, you can consider creating partitions for each state of a country. Generally, in range partitioning, you would be adding more partitions over time, whereas in list partitioning the data will be continuously growing within each partition. This design decision is very important from the partition maintenance point of view, which most people ignore.

Although the syntax for creating partitioned tables (just like the one in a commercial RDBMS) is not available in PostgreSQL, you may wonder exactly how to set up partitioning in PostgreSQL. You will be glad to know that PostgreSQL uses the concept of inheritance, triggers and constraint exclusion to support partitioning. Additionally, you can also use rules instead of triggers.

Now, inheritance and constraint exclusion come with their own pros and cons. There are various scenarios that you need to take care of. Sharing details on them is the main goal of this article. Further, once that has been addressed, the next goal should be to automate partition maintenance because you would most certainly like to automate the creation and maintenance part of partitioning. As you will realise by the end of this article, partition maintenance is another important task, apart from setting up the partitions.

Partitioning using inheritance

Before we begin, you should note that there are various ways to set up partitioning. The scheme that I am going to show here is just one of many. You would need to tweak it as per the actual requirements.

Before we start, make sure that the constraint_exclusion configuration option is set to either ON or PARTITION in the postgresql.conf file. This is absolutely necessary to get the partitioning scheme working.

First, let us create an empty master table without any primary key. If you are wondering about the missing primary key, then you should know that this is an empty master table, and that we need to set up a primary key on each of the partitions separately (more on this in the latter part of the article):

CREATE TABLE orders ( id INT NOT NULL, address TEXT NOT NULL, order_date TIMESTAMP NOT NULL );

Now, create some partitions using non-overlapping CHECK constraints:

CREATE TABLE orders_part_2011 ( CHECK (order_date >= DATE '2011-01-01' AND order_date < DATE '2012-01-01') ) INHERITS (orders); CREATE TABLE orders_part_2010 ( CHECK (order_date < DATE '2011-01-01') ) INHERITS (orders);

Here, the partition orders_part_2011 will hold all data from 2011 onwards, whereas orders_part_2010 will hold all previous data. Note that I am not using any primary key clause, as order_date cannot be a primary key in this particular setup. Further, always make sure that you match the data-type of values used with that of the column referred to in the CHECK constraints.

Optionally, you may create indexes on the partitioning key columns for each of the partitions:

CREATE INDEX orders_part_2011_idx ON orders_part_2011(order_date); CREATE INDEX orders_part_2010_idx ON orders_part_2010(order_date);

The partitioning scheme is not yet ready. What I really want is that the data inserted into the master table should be redirected to the appropriate partition. This is where I need a trigger or rule. Triggers and rules have their own pros and cons. Here I use triggers, with trigger functions.

Again, a trigger function can be written in many ways, depending on the requirements. I will show two ways of writing one, along with their advantages and disadvantages. Let’s create a trigger function as shown below:

CREATE OR REPLACE FUNCTION orders_insert_simple() RETURNS TRIGGER AS $$ DECLARE vsql Text; BEGIN vsql := 'INSERT INTO orders_part_'|| to_char(NEW.order_date, 'YYYY' )|| ' VALUES ('||NEW.id||','||quote_litera(NEW.address)||','||quote_literal(NEW.order_date)||')'; RETURN NULL; END; $$ LANGUAGE plpgsql;

Now we need to attach this trigger function to a trigger, to handle the INSERT on the master table:

CREATE TRIGGER orders_insert_trigger BEFORE INSERT ON orders FOR EACH ROW EXECUTE PROCEDURE orders_insert_simple();

There are quite a few interesting things used in the trigger function that you need to closely look at. It is important to note that the partition name is computed on the fly. This works for our particular case, as all the required partitions were created beforehand. This function would not require any updates, even if you add new partitions for the next year, and so on.

Looks easy? Then take a close look again, at the part where I have used quote_literal to quote the input values. Essentially, you need to quote each and every possible value that is going to be inserted into the partitions.

Similarly, you would need to take care of NULL values. This is the most cumbersome part of using this easy-to-write trigger function, so let’s try out another way of writing the trigger function:

CREATE OR REPLACE FUNCTION orders_insert() RETURNS TRIGGER AS $$ BEGIN IF (NEW.order_date >= DATE '2011-01-01' AND NEW.order_date < DATE '2012-01-01') THEN INSERT INTO orders_part_2011 VALUES (NEW.*); ELSIF (NEW.order_date < DATE '2011-01-01') THEN INSERT INTO orders_part_2010 VALUES (NEW.*); ELSE RAISE EXCEPTION 'Date out of range. check orders_insert() function!'; END IF; RETURN NULL; END; $$ LANGUAGE plpgsql; CREATE TRIGGER orders_insert_trigger BEFORE INSERT ON orders FOR EACH ROW EXECUTE PROCEDURE orders_insert();

As you can see, we are now explicitly checking order_date to redirect data into the appropriate partitions. This doesn’t need handling of quote or NULL values — at least, for non-partition-key values. I have simply used NEW.* instead of referring to individual column values.

In case you are not able to create all partitions beforehand, you would need to update the above trigger function with the logic to create partitions on-the-fly. Generally, we use separate triggers for partition creation and data redirection. Obviously, the partition creation trigger should always be fired before the redirection trigger. You should take care to avoid race conditions between those two triggers.

Now that the trigger is in place, let’s insert some records and verify the setup:

pg=# INSERT INTO orders VALUES(1, 'pune', '2011-08-22'); INSERT 0 0 pg=# INSERT INTO orders VALUES(2, 'pune', '2010-02-22'); INSERT 0 0 pg=# UPDATE orders SET address = 'bengaluru' WHERE id = 2; UPDATE 1

Do not panic on seeing the INSERT statement messages — they simply convey that zero records were inserted in the base (master) table. Actually, the records were transparently inserted into the related partitions.

Another thing to note about the UPDATE statement is that you are not supposed to update the partitioning key. Basically, any change to the partitioning key value might result in the movement of that record to another partition, which is termed as the row movement.

A simple way to handle queries that do update the partitioning key is to capture the UPDATE query, delete the related record from the partition, and then fire an INSERT on the base table. Then the partitioning mechanism will kick in and redirect the record to the appropriate partition. This is not handled in the current setup, as I am updating the address value, which is a non-partition-key column.

Let’s now check where the records are stored:

PostgreSQL knows about the child tables of the orders table, so it assumes that the user wants all the data, from the parent as well as all the children. The partitions are normal tables, and you can query them as usual:

You can also check if the master table is really empty, by using the ONLY clause, which restricts the lookup to only the table specified in the statement:

SELECT * FROM ONLY orders; id | address | order_date ----+---------+------------ (0 rows)

Querying over partitions
Use the EXPLAIN feature to check the plan for querying over partitions:

EXPLAIN SELECT * FROM orders WHERE order_date = '02-JAN-11'; QUERY PLAN ----------------------------------------------------------------- Result (cost=0.00..26.01 rows=7 width=40) -> Append (cost=0.00..26.01 rows=7 width=40) -> Seq Scan on orders (cost=0.00..23.75 rows=6 width=44) Filter: (order_date = '02-JAN-11 00:00:00'::timestamp without time zone) -> Seq Scan on orders_part_2011 orders (cost=0.00..2.26 rows=1 width=18) Filter: (order_date = '02-JAN-11 00:00:00'::timestamp without time zone) (6 rows)

In the above output, you see that only one partition was scanned, based on the WHERE clause conditions. Let’s look at another example:

EXPLAIN SELECT * FROM orders WHERE order_date = now(); QUERY PLAN ----------------------------------------------------------------- Result (cost=0.00..30.03 rows=8 width=41) -> Append (cost=0.00..30.03 rows=8 width=41) -> Seq Scan on orders (cost=0.00..26.50 rows=6 width=44) Filter: (order_date = now()) -> Seq Scan on orders_part_2011 orders (cost=0.00..2.51 rows=1 width=18) Filter: (order_date = now()) -> Seq Scan on orders_part_2010 orders (cost=0.00..1.01 rows=1 width=44) Filter: (order_date = now()) (8 rows)

Here all the partitions are scanned — definitely not what we wanted! You should be aware of the fact that the planner analyses the query before the values from the parameters or stored procedures are substituted. As the planner does not know the exact value of now() during the planning phase, it cannot prune partitions, and so scans all the partitions. You need to look out for such cases where constant values are expected. In case you are planning to use functions in the WHERE clause, do make sure to understand the various types of functions that can be created in PostgreSQL.

Let us now go through the finer details of the three features used in PostgreSQL partitioning.

Constraint exclusion

Constraint exclusion works with only range or equality check constraints. It might not work for constraints like the following:

ALTER TABLE product_items_j ADD CONSTRAINT chk_item_name CHECK (item_name LIKE 'P%');

More importantly, the WHERE condition should be similar to the CHECK constraints. For example, if you have the following CHECK constraint:

ALTER TABLE product_items_j ADD CONSTRAINT chk_item_name CHECK (item_name BETWEEN 'P' AND 'PB'');

Then constraint exclusion would not help for the following query, since the WHERE condition is not similar to the CHECK constraint:

SELECT item_name FROM product_items WHERE item_name LIKE 'Q%' ;

To get the constraint exclusion working, you need this form of query:

SELECT item_name FROM product_items WHERE item_name = 'Pen';

From the above examples, it should be clear that the user needs to take extra effort while handling the WHERE clause predicates, in order to hit the CHECK constraints. The other major problem with this setup is that there is no automatic way to verify if all the CHECK constraints are mutually exclusive. You need to be extra careful while setting them up.

The EXPLAIN feature comes in very handy to tackle these issues. Basically, any change to the query or table, even the slightest one, should be rigorously followed by looking at the EXPLAIN output. If you do not see the expected plan, then either the WHERE clause or the CHECK constraints need to be looked into.

Inheritance

The important rule that you should always be aware of is that child tables inherit column DEFAULT values, not NULL and CHECK constraints only. Any other constraints, like UNIQUE, PRIMARY and FOREIGN key, will not be inherited. Also, indexes, ownership and permissions will not be inherited.

One has to set up the non-inherited constraints on all the child tables. However, there are various ways of setting up these constraints, which we will discuss in the very next section. The child tables cannot rename inherited columns, but can add new columns — this is the biggest advantage of using inheritance.

You should also note that one can enable or disable inheritance on child tables by using the ALTER TABLE command. This is, in fact, very useful when you want to remove or move partitions from the base table.

Using inheritance, you can have multiple levels of partitioning. You can always create a great-great-grandchild of a partition, but you might experience a performance loss with this kind of setup, since more base tables are involved, directly or indirectly. So generally, 1 or 2 levels for partitions and sub-partitions should be good. If you are going any deeper, then most probably you need to rethink the partitioning scheme employed.

Uniqueness

PostgreSQL uses indexes for supporting uniqueness. However, there are no multi-table indexes, which means that an index cannot span over more than one table. This is the reason for not having any primary key in our setup. In general, you might want to concentrate on indexing the partitioning key columns. For this, create non-overlapping CHECK constraints and then create a unique index on partitioning key columns over each partition. This is almost like having unique values over all partitions.

For indexing the non-partitioning-key columns, you create unique indexes over each partition. However, for verifying uniqueness, you need custom functions to scan all partitions, which would hopefully use the related indexes.

As you must have realised, this scanning of all partitions was not what we wanted, to begin with. Also, due to the lack of multi-table indexes, maintaining a primary key or a foreign key is technically not possible.

Consider that you created a primary key on each of the order_id columns of all child tables. Now it may happen that both orders_part_2011 and order_part_2010 have an order_id value of 1. Even though the uniqueness constraint within each child table is upheld, the base table orders now has duplicate order_id values.

To overcome this, you can create an additional lookup table, with primary keys from the base as well as child tables. Basically, you need to use triggers to keep the lookup table updated as you insert, update or delete records from the related partitioned table. Now all the primary and foreign key constraints can be mapped to this lookup table.

That’s it! These are basic setup issues that you must be able to get through now. You should now be ready to face the next challenge — performance tuning for partitions.

What’s around the corner?

The PostgreSQL community does have several discussions on this feature. For more details, you can visit this wiki entry on partitioning.Related Posts:

Tags: constraints, database partition, database trigger, design decision, inheritance, LFY January 2012, partition maintenance, partitions, PostgreSQL, Tables, timestamp values

Categories: Electronics News, Linux News, News

FOSS Solutions for Call Centres that’s 10 Times Cheaper

Latest news from Linux for you magazine - Fri, 01/27/2012 - 15:05

Team OrisysIndia with Mervin Alexander, CEO, Technopark (standing fifth from left)

OrisysIndia, a Kerala-based IT solutions provider, is using open source to deliver advanced communication solutions to call centres in Kerala at a mere 10 per cent of the cost of similar proprietary solutions.

OrisysIndia is an incubatee firm at Trivandrum’s Technopark, and is into providing IT consultancy and solutions related to domains such as Web applications, website design and development, e-commerce services, telephony solutions, and many others. You may well wonder what’s new about it since there is no dearth of IT consultants and solutions providers in the market.

What makes Orisys an interesting case to evaluate is the fact that the company has been able to create a niche for itself by developing a sophisticated yet drastically low-cost communications solution for call centres, by leveraging Asterisk, a free and open source software that transforms a computer into a communications server.

Telephony solution with a difference

Arun Raj, CEO, OrisysIndia, shares more details about the Asterisk telephony solution, which he believes is revolutionary in many ways: “One of our clients (name kept anonymous on request) is running a call centre with 10 telecallers. The firm was using a typical PBX box and commonly-used analogue phones. The traffic mainly related to outbound calls, with an average of 60 to 65 calls per user, per day.

“This proved very cumbersome, as it included a process of dialling numbers and then manually posting call feedback entries and sorting data. Replacing this, we have now implemented a CRM-integrated Asterisk-based customer care solution. Asterisk provides a complete set of tools that can be used to route calls, set up a conference bridge, VoIP gateway, IVR system, etc. It is one of the most powerful platforms, which offers full voice call control to programmers.

“If you use conventional PBX hardware like what is provided by vendors like Cisco and Avaya, it costs approximately Rs 20-30 lakhs. But our solution costs around Rs 2-5 lakhs, depending upon the nature of the complete solution required by a customer, which may involve CRM integration, developing the IVR tree, solution customisation, and so on. I think this solution has the potential to trigger a revolution in the telecom industry.”

In the new system, calls are initiated from the computer itself at a single mouse click. “The productivity of agents has increased to 125 to 135 calls per day, which is double their earlier daily average. The process of data sorting and reports generation, too, has been automated,” he adds.

Working of the Asterisk-based call centre

Apart from developing Asterisk-based telephony solutions, the company works extensively on a number of open technology platforms, frameworks and libraries such as PHP, CodeIgniter, Symfony, Zend, MySQL, Drupal, Magento WordPress, Joomla, jQuery, and so on, to suit the other specific needs of its clients.

Switching to the OSS stack

During the initial stages of their firm’s journey, the company also faced a resource crunch. It had neither an office, nor enough money to set up the infrastructure to build solutions. The company started out with virtually zero investment.

The startup storyThe seeds for the firm were sown in 2008, when the company’s founding members, who were mostly independent freelancers simultaneously pursuing their studies in college, came together and pooled their resources for a common project that aimed at providing cost-effective IT solutions to firms.

Raj recapitulates: “After our initial monetary success, we decided to continue this collaborative pursuit within the framework of an organisation. Hence, OrisysIndia was incorporated in 2009 as a limited-liability partnership firm.” Initially, Orisys was a three-member team, comprising Arun Raj (CEO, OrisysIndia), Amrutha Raj (creative head) and Manodh Mohan (chief marketing officer). They were later joined by Joseph Alexander (project manager).

It was during this phase that the Orisys team members spotted a few big projects, for which they believed they had the expertise to build a solution. Interestingly, they also discovered that some of the solution providers who had pitched for the projects had given very high cost estimates.

Hence, prior to coming up with an offer, the team did a little research into the offerings of the different vendors and found that most of the solutions were priced so high because they were built on proprietary licensed technologies.

“While there wasn’t a conscious effort put into building Orisys as an open source solutions provider, it so happened that in the pursuit of building a firm that provides the most apt solutions at the most competitive price, the company realised that OSS fitted the bill, perfectly. It hence became the business strategy of Orisys to leverage the open source stack and provide turnkey end-to-end solutions with long maintenance contracts, so that customers are no longer worried about the support services for the solutions rendered to them,” avers Raj.

The business took a leap forward when the team began building solutions using all its OSS knowledge resources. Success followed sooner than expected, thereafter.

The ‘open’ strategy that worked

Working with OSS proved to be a huge advantage for Orisys. Raj adds: “Our client base has increased phenomenally, as we have been successful in building a cost-effective open source solutions portfolio. This has been the core reason for our business’ growth.”

Today, the company has many prestigious clients. It provides technology support to ICFOSS (International Centre for Free and Open Source Software), Government of Kerala. It also provides specialised services to companies and NGOs like Make A Difference (M.A.D.) and Mirakle.

The Orisys team vouches for the usefulness of the OSS stack to bootstrap IT businesses, especially start-ups. Raj reasons: “The start-ups get access to the entire source code from the OSS community, and can customise those features with respect to the requirements of different clients. If we start from scratch, we may not be able to complete the project in time. However, using OSS, we are able to deliver solutions to clients in time.

“Apart from this, open source tools and development kits are easy to customise, and there is a very large community behind these projects, which is always a great help if you get stuck during the process of product development.”

What’s in the offing?

Orisys is planning to expand its operations by setting up an office in the US. The company also plans to hire 200 employees in the next fiscal year, of which 70 per cent would be OSS professionals. Currently, Orisys is a nine-member team, seven of whom are OSS professionals, whom the company has roped in from events such as tech meets and bar camps. Raj feels that barcampkerala.org is one of the perfect head-hunting grounds for OSS professionals.

On the solutions improvisation and innovation front, Orisys is currently doing research on building a 3G video gateway to facilitate video calls, video broadcasting, and interactive video and voice-response systems. Raj divulges more details: “3G video calls are new to the market, and such value-added services can play a key role in the telecom sector. Some of the application areas include on-demand video streaming channels, live cricket broadcasts, access to on-the-go healthcare facilities, Interactive Voice and Video Response (IVVR) for product demos and online support, etc.”

Orisys owes a lot of its success to open source, and contrary to popular perception, Raj feels it is easy to make money through OSS. To young OSS professionals who have a business idea up their sleeves, he suggests: “Make a product with a good GUI and navigation features. Approach a problem with a solution; resolve it in your mind first, interact and collaborate with the open source community, then choose the right tools from the open source stack and you will be amazed to see the results.” And we concur!Related Posts:

Tags: analogue phones, Arun Raj, Asterisk, Avaya, call centre, Cisco, communication solutions, communications server, CRM, customer care solution, e-commerce services, Joseph Alexander, Kerala, LFY January 2012, Manodh Mohan, MySQL, Orisys, OrisysIndia, PBX, proprietary solutions, telephony solutions, Trivandrum, United States, VOIP, VoIP gateway, web applications

Categories: Electronics News, Linux News, News

Aseema CEO: High Time Non-telecom Device Engineers Latch on to Android Skills

Latest news from Linux for you magazine - Fri, 01/27/2012 - 13:58

Samartha Raghava Nagabhushanam, CEO, Aseema Softnet Technologies

Android has always been associated with the telecom industry, being an OS for mobile phones and tablets. Few are aware of the use of Android in other areas of the electronics sector. Aseema Softnet Technologies is a Bengaluru-based organisation that focuses on product development services for mobile phones, and has been using Android since its original release through Cupcake, Donut, Eclair, Froyo and Gingerbread, to Honeycomb. LINUX For You caught up with Samartha Raghava Nagabhushanam, CEO of Aseema, who has been associated with the telecom industry for over 18 years, to know more about the young firm’s association with Android, the value it can add to the process of project development, and more…

Could you elaborate on Aseema’s open source offerings?

Aseema provides device engineering and ERP/CRM solutions for mobiles. A core area we are focusing on is segment-centric tablets that can be used in the retail, educational, medical and other specialised areas. For these tablets, we do the device engineering and the required applications according to the segment we are focusing on.

Android is often associated with mobile phones. Is there an increase in the use of Android on other platforms?

Android is seeping into many devices in sectors other than telecom. The doors in the electronics device segment are opening up for Android. It is used not only in phones and tablets, but even the electronics sector. However, Android is comparatively a new phenomenon and the industry started waking up to it only about two years back. Hence, the sudden increase in demand. Not many people have worked on this platform, and as a result, you see a rapid need for Android expertise.

Now, many organisations in India are building an extensive Android excellence centre. There is an acute scarcity of Android expertise globally in the non-telecom market and this is felt especially in India, because the global market looks to India for such solutions.

What are the sectors driving this demand?

The requirement for people with Android expertise is highest in the telecom sector, closely followed by the electronics industry. Backseat entertainment, retail dashboard activities, smart grid to optimise utilities, home entertainment, set-top boxes, television, etc, are also moving onto Android. It may take a while, but Android will also soon be available on home appliances.

Do you feel using Android can add value to the process of project development? If so, please state a few advantages of using Android platforms or technology tools?

Customers prefer to be connected with their devices all the time and Android is an apt operating system to connect to other devices over the Internet. Moreover, Android is associated with cost-effectiveness, and its real value lies in an extremely user-friendly growth. It is also reassuring to technologists that Google is a hidden parent for Android and that its framework does not allow fragmentation.

Is cost a significant reason for companies to adopt Android on their devices?

Yes, Android devices earlier cost $200 but are now available with the same features for $80. Using the Android OS reduces the cost of the device; Windows and Apple devices prove to be much more expensive. Moreover, proprietary operating systems are not flexible, as the source code is not available to developers.

Another reason why customers choose Android is because security is an important aspect of the device, and a lot of developers are building security around it in the form of apps. We have seen Android penetrate into defense organisations as well.

How many employees do you have at Aseema? Do you plan to hire Android experts in the near future?

Currently, we have about 25 employees, and around half of them have been trained to work on Android. We plan to hire many more in the next two years, to work on 20 devices concurrently. We will require about 10 to 20 people to work on designing each device, so the approximate number we are looking at hiring is 300.

What do you look for while choosing employees?

In our current team, we looked only for core programming skills in C language, as we knew people with Android skills were not available easily. We also look for analytical skills, so our tests are based on these two skills plus the candidate’s attitude in terms of eagerness to learn. We take the responsibility to hire, mentor and develop Android skills in the person.

How do you train your developers to work on Android?

Though we do not train employees, we allot even freshers to work on new devices and provide a mentor for them to learn from. An employee of good calibre takes approximately three months to learn the basics, and about six to ten months to be productive.

Do you involve developers from the community or take any help from them during project development?

Until now, we have been working on designing device solutions and so we did not get involved with the online community, because we were device-centric. We are also working on applications for our tablets and we plan to participate in the community by sharing some of our work and contributing code.

What is your message to the community and developer fraternity?

It is high time that non-telecom device engineers latch on to Android skills. In fact, this is the right time to be skilled in the Android domain, as there is a dearth of skills in this area and Android penetration is very rapid. It will be like Windows for PCs, which some people were slow to adopt, but eventually was embraced by all. Android will soon be an inevitable part of designing devices.Related Posts:

Tags: Android, Apple, Aseema, Aseema Softnet Technologies, Bengaluru, device engineering, electronics device segment, electronics industry, electronics sector, Google, Handhelds, India, LFY January 2012, mobile phones, retail, Samartha Raghava Nagabhushanam, telecom market

Categories: Electronics News, Linux News, News

Some Glimpses of the ‘Dark’ Internet in Protest of SOPA/PIPA

Latest news from Linux for you magazine - Wed, 01/18/2012 - 13:50

Well, the time has come — January 18, 2012 — and many of the Free/Open Source Software project/advocacy sites, some popular news/discussion sites, as well as our favorite encyclopedia Wikipedia’s English version has gone dark in protest of the pending US Senate and House bills PIPA and SOPA.

Quoting EFF on why making our collective voices heard against these bills is important:

The Internet blacklist legislation — known as PROTECT IP Act (PIPA) in the Senate and Stop Online Piracy Act (SOPA) in the House — invites Internet security risks, threatens online speech, and hampers Internet innovation. Urge your members of Congress to reject this Internet blacklist campaign in both its forms!

To make the most impact right now, we’re asking YOU to do two things. Today, use our handy tool to send an email to your representatives, letting them know you oppose these bills and they should too. Then, on January 23, when the Senate is back in session (and scheduled to vote on PIPA on January 24), call your Senator and tell him or her that it’s time to stand with the Internet and against the Internet blacklists!

Since LINUX For You is based out of India, unfortunately we can’t write to the US Senators and Representatives in protest. Thus, instead, we thought of documenting how some of the major websites of the world took a stand against the bill — because, if bills like these pass, the dark days of Internet won’t be too far fetched.

Well, imagine a world without our favorite reference website

The official website of Free Software Foundation, along with all its other portals, display a common homepage for 24 hours

linuxforu.com uses the WordPress engine for managing it's content. And it's heartening to see the project has taken its stand.

news.opensuse.org has taken a stand by redirecting all traffic to sopastrike.com

boingboing.net, one of our favourite source of 'free' news, has gone dark for 24 hours

Mozilla, the foundation and project that is 'dedicated to keeping the Web free, open and accessible to all', will join the cause for 12 hours soon

michaelmoore.com, the home of award-winning writer and documentary film maker, is off for 24 hours in protest

reddit.com discussion forum will go dark for 12 hours — and their clock is ticking

O'Reilly, one of our favourite publishers, will go dark for 12 hours. And it's not just oreilly.com, but all of its other web properties too. Ironically, bills like SOPA and PIPA claim that they are for the well-being of publishers.

status.net official blog reports they will take down identi.ca for 12 hours

Well, the reason for documenting this stand by some of the major websites is to point out that although these bills are US bases, however, if passed they will pose a major threat to the “open Web” as we know it.

Already our very own [Indian] government is taking matters to court in order to put restrictions on our “free speech”. What’s worse is the judge has even come up with threats like — here’s quoting a report by Glyn Moody:

“Like China, we will block all such websites,” Justice Suresh Kait said while asking counsel for Facebook and Google India to develop a mechanism to keep a check and remove “offensive and objectionable” material from their web pages.

And we, like naïves, thought “free speech” was our constitutional right? Wake up, the “BIG BROTHER is watching!” They will refuse to bow down to our “rights”.

Well, in the spirit of Free Software, where the word “free” means — not free as in free beer, but free speech — we must stand up for our rights.

In the words of great Bob Marley, let us “get up, stand up: stand up for your rights!”, before it’s too late.

Read more about SOPA/PIPA and why bills like these are harmful here:

How PIPA and SOPA Violate White House Principles Supporting Free Speech and Innovation
Tim O’Reilly’s comments — well, because a publisher’s comments matter, because he feels “SOPA and PIPA are bad industrial policy“
EFF’s Deeplinks Blog

Update (1613 Hrs IST): While, visiting google.com from outsite the US only shows a “Tell Congress: Please don’t censor the web!” message right under the search box, the US version of the site (if you’re from outside the US set your browser to use a US proxy server from proxyblind.org before visiting google.com) appears somewhat like this:

Google censors its logo

Bold move! If only the search service was also off for a day…

Update (2005 Hrs IST): reddit.com, oreilly.com, identi.ca, and mozilla.org screen grabs:

reddit.com is now black

So is identi.ca

So is oreilly.com…

And finally, mozilla.org

Tags: Boing Boing, China, Congress, Creative Commons, Electronic Frontier Foundation, Facebook, FOSS, Free Software Foundation, FSF, Google, Identi.ca, India, Internet blacklist legislation, Internet innovation, Michael Moore, mozilla, O'Reilly, online speech, Open Web, openSUSE, PIPA, Reddit, Senate, SOPA, status.net, Stop Online Piracy Act, Tim O'Reilly, United States, White House, Wikipedia, WordPress

Categories: Electronics News, Linux News, News

Adoption of Linux/FOSS: Challenges & Opportunities

Latest news from Linux for you magazine - Fri, 12/30/2011 - 19:43

The hurdles faced in the adoption of open source by enterprises and the Indian government can easily be overcome if a planned, six-step approach is followed.

Open source has opened minds and provided a great amount of freedom of choice not just to organisations but to our government as well. In my view, open source has brought about a change in the way we view and adapt to technology. We are seeing a paradigm shift from packaged software to open source standards not just within organisations, but also at the government level. A significant amount of government administration processes have been simplified by employing various open source tools.

In the last five years, there has been a sudden rise in open source developers being hired. There is a huge untapped potential for developers in the open source domain. However, it remains to be seen what measures the government is taking at the central and the state levels to implement this technology and how it is addressing the challenges associated with migrating to open source.

Both at the central and state levels, governments have formulated policies promoting the adoption of Linux and other open source software for several years now. Initiatives such as the National Resource Centre for Free and Open Source Software (NRCFOSS) and BOSS Linux from Centre for the Development of Advanced Computing (C-DAC) point to the governments level of involvement in spreading awareness about and driving the adoption of Free and Open Source Software (FOSS). Individual bodies such as C-DAC and the National Informatics Center (NIC) have made enormous contributions to the adoption and migration of operations on the open source platform.

The governments emphasis on open standards such as the Open Document Format (ODF) has also helped create a favourable climate for the adoption of FOSS within the government. Setting up independent bodies to execute OSS adoption/migration projects is evidence of the high interest levels in this sector. For example, the State government of Assam insists that all office documents be shared only in the Open Document Format to maintain compatibility and this is a statutory message that appears in all mails.

Similarly, the IT policy of the Government of Kerala is also very supportive of open source. In recent years, the hardware and software procurement policies of our government have been made largely vendor-neutral, creating a level playing field for Linux and open source software. Several massive projects such as the state service gateways, the National Knowledge Network, etc., are in the development phases now and for most of them the entire infrastructure is new, so Linux and open source software have been adopted.

“The policy on Open Standards for e-governance” notified by the Government of India in November, 2010 mandates adoption of royalty free open standards for all e-governance projects. This brings in a level playing field for Open Source software in the multi-billion dollar e-governance market in India.

Similarly, private organisations are realising the importance of migrating to OSS. Sectors like BFSI and telecom currently derive the maximum benefit of OSS. They are realising that OSS eventually yields better results in terms of quality, performance and pricing.

Adoption challenges in enterprises

The adoption of OSS at the desktop in enterprises faces several challenges because of the legacy of poorly written client-server applications, and Web applications “optimised” for Internet Explorer 5.0, etc. Some of the other key hurdles are discussed below:

Applications — the biggest bottleneck for Linux adoption

Some time ago, a major financial institution in India was evaluating Linux as an option for its desktops. It had more than 50 custom applications, most of which were Web applications that had been developed over the last decade. The Web applications were supposedly platform-agnostic, but when they were tested on a Linux desktop, it was discovered that just 10 per cent of the applications worked flawlessly.

The rest had many compatibility issues, especially with Active-X controls, IE-specific Java script code, the use of IE-specific code, invocation of locally installed Windows applications from within the Web applications, etc. The company found that the only way it could successfully migrate to Linux on the desktop was by first making all its applications truly platform agnostic at the client level.

An interesting side effect of migration is that many organisations suddenly realise the importance of having access to the source code. When an organisation wants to transform its legacy apps to cross-platform apps, it often finds that it does not have access to the source code to most of the custom developed applications.

It is almost impossible to make any changes to the applications, as the firms do not have any ongoing maintenance contract with the original vendor that developed the software, nor do they have access to the source code.

Major ISVs are not enthusiastic

This is a classic chicken-or-egg scenario. The packaged software supplied by major ISVs poses a big challenge in any large scale migration initiative. Independent Software Vendors (ISVs) claim that they would support Linux better if there is demand from the market. Enterprises insist that they would migrate when there is better support for Linux from the ISVs.

Organisational practices

When a large media organisation in India decided to migrate to Linux as the OS platform, the biggest hindrance it faced was that it had dozens of MS-Office (MS-Excel) macros that had been developed and, in most cases, its partner organisations also built their workflows around these macros. Now, for migration, the organisation had to make an inventory of all its macros, document and rationalise their use, and get them translated into Open Office Macros with a significant investment of money and time, before it could adopt Linux and OpenOffice.org.

Solutions and opportunities

These hurdles can be easily overcome with a six-step approach. Here are the six essential steps to open source migration:

Discovery
Analysis
Design
Planning
Implementation
Maintenance

Quality time and effort spent in the Discovery and Analysis phase, will lead to an optimal solution design. Implementation itself is generally done in multiple phases for an easy roll out. The trick for successful implementation is to prioritise the applications and attack those that are most compatible with the system, and then deal with the non-compatible applications.

Enterprises can exercise freedom of choice on the OS front by taking the determined stand “If it is not cross-platform, it is not in”, and embarking on a drive to make current applications platform agnostic. This will not only make the Linux migration more effective and sustainable, but will also have the additional benefit of being ready for any future devices.

To summarise, there is an increased awareness about the benefits of the open source model and a realisation that these benefits extend far beyond the cost advantages. There is also a demand for tools and products developed under a truly open source model supported by enterprises and government.Related Posts:

Tags: Assam, C-DAC, client-server applications, e-governance, government, government of india, India, Indian government, Internet Explorer, ISVs, Java, Kerala, LFY December 2011, Linux, Microsoft Windows, National Informatics Center, National Informatics Centre, non-compatible applications, NRC-FOSS, open source software, open standards, OpenOffice.org, packaged software, web applications

Categories: Electronics News, Linux News, News

Fedora Scientific: Open Source Scientific Computing

Latest news from Linux for you magazine - Fri, 12/30/2011 - 19:18

Fedora Scientific Spin is a Fedora Linux spin that aims to showcase the open source tools for scientific and numerical computing. It was first released officially with Fedora 16 in November 2011, and is targeted at current and future Linux users in the domain of scientific computing.

I started work on this spin with a simple goal, which I will put into perspective here. My work and play involves programming, writing articles, visualisation and analysis of numerical data — and hence, I use a lot of open source libraries and tools to assist me. If you haven’t guessed already, I end up installing these tools every single time I do a fresh install of Linux. This motivated the need for a Linux distribution with these tools preinstalled, specifically targeted at users like me, who use Linux for scientific work. The Fedora community’s popularity and the insanely simple process of creating spins provided an ideal launch pad.

Now let us take a brief look at the applications and libraries currently available in Fedora Scientific.

Applications in Fedora Scientific

The current set of applications shipped in Fedora Scientific are broadly classified into the following categories:

Scientific computing tools and environments: The numerical computing package GNU Octave, the computer algebra system Maxima, with its front-end wxMaxima, the Python scientific libraries SciPy, NumPy and Spyder (a Python environment for scientific computing) are some of the software included in this category. A development environment for R, the statistical computing environment, is also included, and so are the ROOT tools for analysing large amounts of data.
Generic programming and development tools and libraries: Software in this category includes the GNU C/C++ and FORTRAN compilers, the OpenJDK Java development tools, and the IDEs NetBeans and Eclipse. Also included are autotools, flex, bison, ddd and valgrind.
Parallel and distributed programming tools/libraries: Software tools and libraries included in this category include the popular parallel programming libraries OpenMPI, PVM, and the shared-memory programming library OpenMP. Also included is the Torque resource manager to enable you to set up a batch-processing system.
Editing, drawing and visualisation tools: So you have simulated your grand experiments, and need to visualise the data, plot graphs, and create publication-quality articles and figures. The tools included to help you in this include LaTex compilers and the Texmaker and Kile editors, plotting and visualisation tools Gnuplot, xfig, MayaVi, Dia and Ggobi , and the vector graphics tool Inkscape.
Version control, backup tools and document managers: Version control and backup tools are included to help you manage your data and documents better — Subversion, Git and Mercurial are available, along with the backup tool backintime. Also included is a bibliography manager, BibTool.

Besides these four main categories, some of the other miscellaneous utilities include: hevea (the awesome LaTex-to-HTML converter), GNU Screen and IPython. You can find the complete list of all the additional packages included in the spin here.

Fedora Spins

Now that we have taken a look at Fedora Scientific, let us explore the enabler behind it. What made Fedora Scientific possible is the Fedora Spins effort. Quoting from the website: “Fedora Spins are alternate versions of Fedora, tailored for various types of users, via hand-picked application sets or customisations.” As of the Fedora 16 release, there are nine custom spins — six of them highly customised for niche audiences like security professionals, designers, kids, researchers and robotics enthusiasts.

Creating a custom Fedora Spin is really easy. And unlike a lot of things in life, it is as easy to do it as it is to talk about it. A tool called livecd-creator is used to create a custom Fedora Spin. A configuration file called a kickstart file needs to be created first, where you specify the list of packages that you want to be installed. You can also specify various other custom configurations, such as the desktop icons, launchers, etc. You can take a look at the kickstart files for all the Fedora Spins over here. If you have an idea for a custom spin, start by taking a look at one of these spins and then creating a kickstart file for yourself.

Once you have a kickstart file ready, you can use livecd-creator to create a customised Live ISO using the following code:

# livecd-creator --config=fedora-livecd-custom.ks --fslabel=Fedora-Live-Custom-CD -- cache=/var/cache/live

This will start the spinning process for your shiny new Fedora Spin. Once you have the ISO, you can write it to a USB stick using the dd command.

Where next?

First, if your interest in this article was to know more about Fedora Scientific, then head to its website to download an ISO and try it for yourself. While you are at it, you may direct your queries and comments to the SciTech SIG mailing list, or use the other forms of communication listed in the support tab.

However, if your interest was to know more about Fedora Spins in general, head over to the Fedora Spins page, and learn more about the Fedora Spins process. Happy spinning!Related Posts:

Tags: backup tools, batch-processing system, configuration file, Fedora, Fedora Spin, Fedora Spins, GNU, GNU Octave, html, Java, kickstart, LaTeX, LFY December 2011, Linux, miscellaneous utilities, python, science, scientific tools, vector graphics, visualisation tools

Categories: Electronics News, Linux News, News

Recent comments

Navigation

User login

Active forum topics

Recent blog posts

Syndicate

Who's online

Who's new

Games News

Lifestyle News

latest Mobile News

WordPress on Nginx, Part 2: vhost, MySQL & APC Configurations

WordPress on Nginx, Part 1: Preparing VPS the Debian Way

Proprietary Solutions Can’t Milk Money from these Cooperatives!

Voices Across the Digital Divide — Using Audio Portals to Connect Communities

Learn to Animate with Blender, Part 2: Creating Animation Controls

Device Drivers, Part 14: A Dive Inside the Hard Disk for Understanding Partitions

Lisp: Tears of Joy, Part 8

Developing Applications on QT, Part 1

Working with MTD Devices

Building Image Processing Embedded Systems using Python, Part 1

IPsec VPN Penetration Testing with BackTrack Tools

Guard Your Network with IPCop, Part 2: Add-ons

Linux on POWER

Postgres-XC — A PostgreSQL Clustering Solution

Partitioning in PostgreSQL

FOSS Solutions for Call Centres that’s 10 Times Cheaper

Aseema CEO: High Time Non-telecom Device Engineers Latch on to Android Skills

Some Glimpses of the ‘Dark’ Internet in Protest of SOPA/PIPA

Adoption of Linux/FOSS: Challenges & Opportunities

Fedora Scientific: Open Source Scientific Computing

New forum topics

Feedback