Get Hands Dirty with NoSQL + Cloud Data Storage - Part2 - Programming in NoSQL

Hello Readers,

Last week, I was talking about NoSQL Cloud Data Storage, MongoDB in the "Get Hands Dirty with NoSQL + Cloud Data Storage" post. (it shows the details about how to create a free 512MB, MongoDB NoSQL database in https://mongolab.com/signup/ also introduce few industry buzz words.)

I hope, you might have signed up in mongolab.com and created a test MongoDB for your expriment.

Database without programmatic interface is like a watching a Rajinikanth movie without Rajinkanth fans croud. What a boaring thing to do. Rajinikanth movies should be always enjoyed with fans croud in a normal big movie theatres (Not alone in home or in HiFi multiplexs) .

MongoDB provides lots of programming language integrations. Technically, the piece of software which helps your application to talk to Database is called " Database Driver".

Here is the list of MongoDB drivers for major programming languages like C, C++, Java, Perl, PHP, Python, etc...

As I have promised in the Part1, In this post, I will show you, How to connect to MongoDB which is hosted in clouds via Perl programming language.

Why Perl?

Hmm. Where do I start?, I started my career with C programming language, I also wrote applications in Java. I got introduced to Perl when I was working in Hewlett-Packard, initially Perl looks weird but when I was learning more and more, I realized its power. I was still working on Java but then I moved to Nokia, Where I got chance to learn Python mainly for PyS60 (Python for Symbian) project.

@work, I designed and developed multiple SW test and Build framework, Currently I am working in NXP Semiconductor, Where I got chance to design and develop a highly secure, centralized, cryptography key management, cryptography operations SOAP web services using Perl + SOAP::Lite + Apache + Gtk2.

So with these limited knowledge, I can say, Perl is super powerful for most of the application development areas.

I chose Perl to show the MongoDB database programming interface but the MongoDB API and programming flow is almost similar for all other programming languages.

Preparation

To get start, we need a PC with Perl + MongoDB driver installation+ MongoDB in the cloud. I am on Fedora 20 Linux PC. Almost all the linux distributions comes with pre installed Perl.

Note: For Windows PC, I highly recommend to install ActiveState perl and use activestate perl's nice feature called PPM to install the Perl Modules. I shall write a separate post on how to install perl modules in Linux and windows with different approaches.

There are multiple perl drivers for MongoDB. Officially  "MongoDB" Perl driver is supported by MongoDB development team. This Perl driver is a wrapper over C MongoDB APIs, So this driver gives good performance and mostly support latest MongoDB APIs.

You can install perl modules using CPAN but you need a local installation of MongoDB and its development package (header files and libs). Since we are talking about cloud, we dont have anything in local machine, So I use the yum tool to install perl-MongoDB driver.

sudo yum install perl-MongoDB

Once the installation is complete, You can check the status like below,

perl -MMongoDB -e ' print $MongoDB::VERSION'

For me, the above command shows,

0.702.2

Ok, Good, Driver is Sane and working.

Now, we need to login to our MongoDB cloud database in https://mongolab.com/ and get few details,

1. MongoDB database host name and port - Get it in the "Database" page. Ex: ds063140.mongolab.com:63140
2. MongoDB database name - Get it in the "Database" page. Ex: npointsolutionsmdb
3. MongoDB collection name - Get it in the "Database" page. Ex: userDetailsCollection 
4. MongoDB user name and password - Get it in the "Users" page. Ex: username=user1, password=user123

Here is my fully working Perl code to show how to connect to MongoDB running in the google cloud. I name this perl script as a mongoDB_test.pl

#!perl
use strict;
use warnings;
use Try::Tiny;
use MongoDB;
use Data::Dumper;

#Global Variables

$|=1; #Auto flush STDOUT

my $dbHost = 'ds063140.mongolab.com:63140';
my $dbUser = 'user1';
my $dbPassword = 'user123';
my $dbName = 'npointsolutionsmdb';
my $dbCollectionName = 'userDetailsCollection';
my $dbClientObj = '';
my $dbObj = '';
my $userDetailsCollectionObj = '';
my $dbCursor = '';
my $userDetailsRecordHashRef = '';

local $,= ' , '; #Make array prints nicer
my $now = '';

#1. Connect to MongoDB 
try
{
 $dbClientObj = MongoDB::MongoClient->new(host=>$dbHost,username=>$dbUser,password=>$dbPassword,
                                          db_name=>$dbName,query_timeout => 10000);
 print "\n INFO: MongoDB in $dbHost is connected.";
}
catch
{
 die 'ERROR: Can not create MongoDB client Object: ', $_;
};

#2. Get the Mongo DB object
$dbObj = $dbClientObj->get_database($dbName);

#3. Get all Collection Names
print "\n All Collections: ", $dbObj->collection_names();

#4. Get the Collection Object
$userDetailsCollectionObj = $dbObj->get_collection($dbCollectionName);

#5. Get all the documents / records from the collection
$dbCursor = $userDetailsCollectionObj->find();
$dbCursor->immortal(1);

#Loop through all the user documents
print "\n INFO: All documents in the Collection $dbCollectionName \n";
while ($userDetailsRecordHashRef = $dbCursor->next()) 
{
 #MongoDB Cursor retrives all the fields in the Document as a name-value pair hash reference
    print "\n", Dumper (\$userDetailsRecordHashRef);
}

#6. Adding a new user document into collection
$now = scalar localtime();
try
{
    print "\n INSERT ID: ", $userDetailsCollectionObj->insert(
                             { 
                              userName => 'testPerlUser'.int(rand(100)), 
                              OS => 'Fedora 20 Linux',
                              LastLogin => $now,
                              DeviceArch => "x86_64",
                              Score => int(rand(500))
                             }, {safe => 1});
}
catch
{
 print "\n ERROR: Can not insert a record into Collection. $_";
};

#7. Query our document is in collection or not
print "\n INFO: Find all perl users";
$dbCursor = $userDetailsCollectionObj->find({userName => qr/^testPerlUser/i}); #All the userName starts with testPerlUser
$dbCursor->immortal(1);

#Loop through all the user documents
while ($userDetailsRecordHashRef = $dbCursor->next()) 
{
    #MongoDB Cursor retrives all the fields in the Document as a name-value pair hash reference
    print "\n", Dumper (\$userDetailsRecordHashRef);
}

#8. All is well, CLose connection
undef $dbClientObj;
print "\n INFO: Done. Quit";
exit (0);

Here is the output from my PC when I ran the above code,

 INFO: MongoDB in ds063140.mongolab.com:63140 is connected.
 All Collections:  , system.indexes , userDetailsCollection , objectlabs-system , objectlabs-system.admin.collections
 INFO: All documents in the Collection userDetailsCollection 

 , $VAR1 = \{
            '_id' => bless( {
                            'value' => '54846970e4b0d2e1a43f8654'
                          }, 'MongoDB::OID' ),
            'LastLogin' => 'Dec 07, 2014',
            'DeviceArch' => 'x86_64',
            'Score' => 103,
            'OS' => 'Linux Fedora 20',
            'userName' => 'test1'
          };

 , $VAR1 = \{
            'DeviceArch' => 'x86_64',
            'LastLogin' => 'Dec 06, 2014',
            '_id' => bless( {
                            'value' => '548469bee4b0d2e1a43f8656'
                          }, 'MongoDB::OID' ),
            'userName' => 'test2',
            'OS' => 'Windows 7',
            'Score' => 54
          };

 , $VAR1 = \{
            'DeviceArch' => 'x86_64',
            'LastLogin' => 'Sun Dec 14 20:58:56 2014',
            '_id' => bless( {
                            'value' => '548dacb85c3cc20b44000000'
                          }, 'MongoDB::OID' ),
            'OS' => 'Fedora 20 Linux',
            'Score' => 480,
            'userName' => 'testPerlUser55'
          };

 , $VAR1 = \{
            'DeviceArch' => 'x86_64',
            'LastLogin' => 'Sun Dec 14 21:06:45 2014',
            '_id' => bless( {
                            'value' => '548dae8d98e64d8c45000000'
                          }, 'MongoDB::OID' ),
            'userName' => 'testPerlUser41',
            'OS' => 'Fedora 20 Linux',
            'Score' => 424
          };

 , $VAR1 = \{
            'OS' => 'Fedora 20 Linux',
            'Score' => 108,
            'userName' => 'testPerlUser67',
            'DeviceArch' => 'x86_64',
            'LastLogin' => 'Sun Dec 14 21:07:00 2014',
            '_id' => bless( {
                            'value' => '548dae9c2d09dc9845000000'
                          }, 'MongoDB::OID' )
          };

 , $VAR1 = \{
            '_id' => bless( {
                            'value' => '548db07a0f9cd12946000000'
                          }, 'MongoDB::OID' ),
            'LastLogin' => 'Sun Dec 14 21:14:58 2014',
            'DeviceArch' => 'x86_64',
            'Score' => 316,
            'OS' => 'Fedora 20 Linux',
            'userName' => 'testPerlUser57'
          };

 , $VAR1 = \{
            'DeviceArch' => 'x86_64',
            'LastLogin' => 'Sun Dec 14 21:15:30 2014',
            '_id' => bless( {
                            'value' => '548db09ae3528e9546000000'
                          }, 'MongoDB::OID' ),
            'OS' => 'Fedora 20 Linux',
            'Score' => 211,
            'userName' => 'testPerlUser57'
          };

 INSERT ID:  , 548db2c5cb53ab1b47000000
 INFO: Find all perl users
 , $VAR1 = \{
            '_id' => bless( {
                            'value' => '548dae8d98e64d8c45000000'
                          }, 'MongoDB::OID' ),
            'LastLogin' => 'Sun Dec 14 21:06:45 2014',
            'DeviceArch' => 'x86_64',
            'userName' => 'testPerlUser41',
            'Score' => 424,
            'OS' => 'Fedora 20 Linux'
          };

 , $VAR1 = \{
            'DeviceArch' => 'x86_64',
            '_id' => bless( {
                            'value' => '548dacb85c3cc20b44000000'
                          }, 'MongoDB::OID' ),
            'LastLogin' => 'Sun Dec 14 20:58:56 2014',
            'userName' => 'testPerlUser55',
            'OS' => 'Fedora 20 Linux',
            'Score' => 480
          };

 , $VAR1 = \{
            'userName' => 'testPerlUser57',
            'Score' => 211,
            'OS' => 'Fedora 20 Linux',
            '_id' => bless( {
                            'value' => '548db09ae3528e9546000000'
                          }, 'MongoDB::OID' ),
            'LastLogin' => 'Sun Dec 14 21:15:30 2014',
            'DeviceArch' => 'x86_64'
          };

 , $VAR1 = \{
            'userName' => 'testPerlUser57',
            'Score' => 316,
            'OS' => 'Fedora 20 Linux',
            'LastLogin' => 'Sun Dec 14 21:14:58 2014',
            '_id' => bless( {
                            'value' => '548db07a0f9cd12946000000'
                          }, 'MongoDB::OID' ),
            'DeviceArch' => 'x86_64'
          };

 , $VAR1 = \{
            'DeviceArch' => 'x86_64',
            '_id' => bless( {
                            'value' => '548dae9c2d09dc9845000000'
                          }, 'MongoDB::OID' ),
            'LastLogin' => 'Sun Dec 14 21:07:00 2014',
            'userName' => 'testPerlUser67',
            'OS' => 'Fedora 20 Linux',
            'Score' => 108
          };

 , $VAR1 = \{
            'Score' => 191,
            'OS' => 'Fedora 20 Linux',
            'userName' => 'testPerlUser75',
            '_id' => bless( {
                            'value' => '548db2c5cb53ab1b47000000'
                          }, 'MongoDB::OID' ),
            'LastLogin' => 'Sun Dec 14 21:24:45 2014',
            'DeviceArch' => 'x86_64'
          };

 INFO: Done. Quit

Assuming that you know basic of perl, I hope the mongoDB_test.pl code + comments is self explanatory.

Note: I have shown my database host, user name and password. So you can just copy paste and run the code. Please Do not abuse the credentials.

Let me know via comments if you have any questions.

Refer:

1. https://metacpan.org/pod/distribution/MongoDB/lib/MongoDB/Tutorial.pod

2. For more query examples -> https://metacpan.org/pod/distribution/MongoDB/lib/MongoDB/Examples.pod
 
Christmas is very near, Get ready for the festival and holidays... Merry Christmas... 

Get Hands Dirty with NoSQL + Cloud Data Storage

NoSQL, Big Data, Cloud Data Storage, Scalability, Shards etc .. are buzzwords in the Data management industry. I thought to get my hands wet with these.

Let me quickly introduce them, (Ya very quick because you will find good intro in the Wikipedia)

• NoSQL - Data management without traditional Table format, Without traditional SQL queries. 


• Big Data - Huge Data Sets. Like weather details of the whole century.


• Cloud Data Storage - Get the data storage from online cloud vendors instead of storing the data in your own servers / machines.


• Scalability - When the Data grows, Easily upgrade the computing capacity so data base operations like read , search and writes are remain faster.


• Shards - To achieve Scalability, Instead of increasing computing power (RAM, HDD Space etc..) on the single server, Split the Data base into smaller chunks and place them in a smaller servers / machines. These chunks are calles Shards and collection of Shards forms a logical single database.

Data storage and retrial needs changes over the time. Now, we are in the world where the application users are very impatience, Can not wait for 30 sec to get the data.

The motivation behind the NoSQL is for high performance, High availability, Simple to store / search / get data and easy administration.

For example, If you are developing a mobile game and you want to store few details like userid, gamescore, devicename in the database. The data itself is very simple, creating SQL schema, creating table, querying data in SQL way is too over kill for this usecase. What you need is a online connected data storage, easy to store and super raw speed.

No offence to SQL guys, SQL is too big for us.

Hey, Good usecase right, Let me use this usecase and create online, high available, cloud provided, NoSQL database (hubbbaa...)..

When we are talking about the NoSQL, there are multiple thought process and Databases types are available in the industry. Each type stores the data in different format but the goals of the the NoSQL DBs are same. (You might need to read NoSQL in Wikipedia, if you haven't done yet).

OK. Stop talking, This post suppose to be a hands on ...

The reason why I was talking so much because I am afraid to read the manual, download, install the Db engine, configure and the use ... I am too lazy.

This is a cloud era, So lets create a online, high available, cloud provided, NoSQL database without installing anything in your machine. (I can see smile and continue to read face :) )

I chose MongoDB NoSQL database.

• MongoDB - Open Source NoSQL Database. Uses JSON like documents for storage. Gives super speeeeeeeeeeeeeeeed.

Multiple Cloud vendors are providing MongoDB as a hosted service. Most of them provide 512 MB , Non production grade MongoDB space for free to tryout. Awesome!! Lets start,

1. Go to https://mongolab.com/signup/ and Sign up. Mongolab offers consolidated dashboard and tools for various data base hosting service provides (Like google cloud, Microsoft Azure, Amazon cloud etc..) 

 2. Fill the data and sign up. Once you Create account, You will get a mail in your provided mail address from Mongolab for Email verification. Check your mail Box and complete the Email verification process.

3. Go to https://mongolab.com/home  - This is your Database Dashboard

4. Click "Create New" button

5. Choose your favourite cloud computing provider, I Chose Amazon.

Note: Do not change the "Location" Its only for production, Paid databases. 

6. Under "Plan" choose Single-node, Then Choose "Sandbox" for free 512MB Mongo DB. 

7. Provide Database name (Mandatory for free databases)

8. Click "Create new MongoDB deployment"

9. Once its created, You can see the database in your home / dashboard view.

Note: You can create multiple MongoDBs from Multiple Cloud providers. All of them can be managed with Dashboard page. Each database comes with 512MB data limit. But its from non production area not suitable for real bussines.

10. Good. Now you have your own database.

11. Lets add some data. In NoSQL database, as I have mentioned above there is no scheme and no need to create a table / data storage structure. In MongoDB terms,

• Collection - Like SQL Table. Collection of documents.
• Document - Like SQL table row data. A data record.

12.  Click your newly created MongoDB database in the dashboard. My database name is npointsolutionsmdb.

13. mongolab will take you to the Database dashboard page. Click "Add collection" button and provide collection (SQL Table) name. I gave "userDetailsCollection" as a collection name to demonstrate my above mobile game usecase.

14. Now, collection might have created. In the Dashboard, Click the just created collection name so we can add the data into collection.

15. Once you click the collection, mongolab will take you to the collection administration page, Click "Add document" button

16. You will see a Create Document page. In MongoDB, Data is represented as a JSON style documents. We can add any vaild JSON document in the Create Document page. I have added following JSON data,

{
        "userName": "test1",
        "OS": "Linux Fedora 20",
        "DeviceArch": "x86_64",
        "LastLogin": "Dec 07, 2014",
        "Score": 103
}

17. If you like my data format, Just copy paste the above data into "Create document" area and Click "Create and go back" button.

18. You can add any number of documents (one by one, by following step #15, #17) into your database. But maximum size limit for whole database (Non productional, No backup) is 512MB.

19. You can check the Database size in the Top Dashboard https://mongolab.com/home

20. If you are adding your own Data, Check its JSON syntax validity in http://jsonlint.com/ before you paste in "Create document".

21.  You can see all the documents from the collections page

22. From the collections page, You can edit / delete each document.

23. Try edit one of the document, Using "Edit" button right most side of each document

Note: You can see "_id" field is added. This is unique object ID added for each document by the MongoDB. "_id" acts like a Primary Key for the document.

24. Click "Cancel and go back" button.

25. Lets try to create index for our collection, Index will make the query run faster.

26. Click on the "Indexes" tab parallel to "Documents" tab

27. In the Add new index, Add following code (almost all the data representaion in the MongoDB is a JSON format)

{
        "userName": 1,
        "Score": 1
        
}

28. Click "Create in background" button. MongoDB also supports creating index as a foreground operation but it will block all the database operations. Usually forground index is very fast but it will take long time if you have large data set. So always create index in background.

29. Go to your collections by clicking db: "<your db name>" link.

30. Click User tab parallel to Collections tab

31. Lets add Database users so they can access our data. Click "Add database user" button

32. Provide user details, I have created user name "user1" with password "user123"

33. READ ONLY? shows that the user can only read the documents or even update / insert documents. false indicates the user can update and insert the documents into collections.

34. Goto home / dashboard , https://mongolab.com/home

35. You can see your MongoDB database size is grown up because we have added few documents into the collections, created users and indexes.


Congrats !!! , Now you have your own online, high available, cloud provided, NoSQL database. 

Keep your  MongoDB read, I will show you how to access the Online, cloud provided, MongoDB via application program code (mostly Perl). Stay tune ...

Refer:

1. http://www.tutorialspoint.com/mongodb/mongodb_overview.htm

2. http://www.mongodb.org/

3. And google is your best friend.