ExpressJS Snippets

1)
If next is called then it function will not return after execution of request.

app.get(/dishes/:dishId, function(req, res, next) {
res.end(‘sending it to server’+ req.params.dishId);
});

2)
Get the request data
req.body.name; req.body.description

3)
res.write();
res.end();

4) Express Routers

var dishRouter = express.Router();dishRouter.use(bodyParser.json());

dishRouter.route(‘/’)
.all(function(req,res,next){
res.writeHead(200, {‘Content-type’:’text/plain’});
next();
})
.get(function(req,res, next) {
res.end(‘retrieving the request’);
});

app.use(‘/dishes’, dishRouter);

Web Services and REST (Representational State Transfer)

What is a web service ?

This is a system designed to support interoperability among systems that are connected over the network.

Web services is service oriented architecture (SOA).

Somebody provides a service, somebody else consumes the service provided by the other person. So this is standardized way of integrating web based applications using open standards. So that clients can communicate with servers, there is a standardized way of exchanging information between the two sites.

There are two common approaches that people take for implementing web services.

  1. One approach, called a Simple object access protocol, or SOAP, uses something called the Web Services Description Language (WSDL).

    SOAP uses XML as the presentation approach.

  2. Other approach REST, or representational state transfer.

    This uses web standards.

    The data exchange can happen either using XML or JSON.

    REST is simpler compared to SOAP and WSDL.

SOAP and WSDL have a lot more features that are more suitable for distributive operation across the Internet. But for typical web applications communicating with the server, REST seems to work just fine.


What is REST ?

It is a style of software architecture for distributed hypermedia systems, like the World Wide Web.

So, REST is a more general concept than what we use in the World Wide Web.

REST is a collection of network architecture principles, which specify how you can identify the sources, how you can access them, how you can address your sources, and how you can perform operations on those resources.

Four basic design principles for REST-
1) Use HTTP methods explicitly
2) Be stateless
3) Expose directory structure like URI’s.
4) Transfer using XML or JSON or both

The motivation for REST was to capture the characteristics of the Web that made the Web successful-
1) URI (Uniform Resource Identifier) addressable resources.2) HTTP Protocol
3) Make request – receive response – display response

Exploits the use of HTTP Protocol beyong HTTP GET and POST-
1) uses HTTP PUT and DELETE.
2) Preserve Idempotence.

REST Grammar:
1) Nouns:
These are the resources represented by URI’s.
2) Verbs:
Verbs are the HTTP request GET (read), POST (create), PUT (update) and DELETE (delete).
3) Representation: It is the format in which data is represented, it is either JSON or XML.

REST is stateless on server side, this helps in scalability of server.
Client side should track its state using cookies, client-side database etc. Client side should include sufficient information for server to serve the request.

Java Swing Samples

Download a file in Worker(Background) Thread:

 

public static class DownloadResource extends SwingWorker<Void, Void> {
	
	@Override
	public Void doInBackground() {
		
				
		SearchResultModel model = resultList;			
		ListSelectionModel selectionModel = checkListManager.getSelectionModel();			
		
		for(int i=0; i<model.getSize(); i++) {
			
			if(selectionModel.isSelectedIndex(i) == true) {					
				
				String downloadURL = model.getElementAt(i).address;				    
				if(downloadURL.contains("abc")) {
					downloadFile(model.getElementAt(i).address);
				} else {
					System.out.println("Download url does not contain abc");
				}
			}
		}
		return null;
	}
}

public static void downloadFile(String downloadURL){
	try {
		
		URL url=new URL(downloadURL);
		HttpURLConnection connection = (HttpURLConnection) url.openConnection();
		int filesize = connection.getContentLength();
		float totalDataRead=0;
		java.io.BufferedInputStream in = new java.io.BufferedInputStream(connection.getInputStream());
		java.io.FileOutputStream fos = new java.io.FileOutputStream("C:/Users/Public/Downloads"+getFileName(downloadURL));
		java.io.BufferedOutputStream bout = new BufferedOutputStream(fos,1024);
		byte[] data = new byte[1024];
		int i=0;
		while( (i=in.read(data,0,1024) ) >= 0 )
		{
			//totalDataRead=totalDataRead+i;
			bout.write(data,0,i);
			//float Percent=(totalDataRead*100)/filesize;
			//current.setValue((int)Percent);
		}	
		bout.close();
		in.close();
	}
	catch(Exception e) {
		 javax.swing.JOptionPane.showConfirmDialog((java.awt.Component) null,e.getMessage(), "Error",
				 javax.swing.JOptionPane.DEFAULT_OPTION);
	}
}

Java Swing custom list with checkbox

public class CheckListCellRenderer extends JPanel implements ListCellRenderer { 
	
    private ListCellRenderer delegate; 
    private ListSelectionModel selectionModel; 
    private JCheckBox checkBox = new JCheckBox(); 
 
    public CheckListCellRenderer(ListCellRenderer renderer, ListSelectionModel selectionModel){ 
        this.delegate = renderer; 
        this.selectionModel = selectionModel; 
        setLayout(new BorderLayout()); 
        setOpaque(false); 
        checkBox.setOpaque(false); 
    } 
 
    public Component getListCellRendererComponent(JList list, Object value, int index, boolean isSelected, boolean cellHasFocus) { 
    	
        Component renderer = delegate.getListCellRendererComponent(list, value, index, isSelected, cellHasFocus); 
        checkBox.setSelected(selectionModel.isSelectedIndex(index)); 
        removeAll(); 
        add(checkBox, BorderLayout.WEST); 
        add(renderer, BorderLayout.CENTER); 
        return this;
    } 
}


public class CheckListManager extends MouseAdapter implements ListSelectionListener, ActionListener{ 
    
	private ListSelectionModel selectionModel = new DefaultListSelectionModel(); 
    private JList list = new JList(); 
    int hotspot = new JCheckBox().getPreferredSize().width; 
 
    public CheckListManager(JList list){ 
        this.list = list; 
        
        list.setCellRenderer(new CheckListCellRenderer(list.getCellRenderer(), selectionModel)); 
        
        list.registerKeyboardAction(this, KeyStroke.getKeyStroke(KeyEvent.VK_SPACE, 0), JComponent.WHEN_FOCUSED); 
        list.addMouseListener(this); 
        selectionModel.addListSelectionListener(this); 
    } 
 
    public ListSelectionModel getSelectionModel(){ 
        return selectionModel; 
    } 
 
    private void toggleSelection(int index){ 
        if(index<0) 
            return; 
 
        if(selectionModel.isSelectedIndex(index)) 
            selectionModel.removeSelectionInterval(index, index); 
        else 
            selectionModel.addSelectionInterval(index, index); 
    } 
 
    public void mouseClicked(MouseEvent me){ 
        int index = list.locationToIndex(me.getPoint()); 
        if(indexlist.getCellBounds(index, index).x+hotspot) 
            return; 
        toggleSelection(index); 
    } 
 
    public void valueChanged(ListSelectionEvent e){ 
        list.repaint(list.getCellBounds(e.getFirstIndex(), e.getLastIndex())); 
        System.out.println("list selected index->"+e.getFirstIndex());
        System.out.println("value selected -->"+list.getSelectedValue());
        
        System.out.println("value selected -->"+list.getSelectedValuesList());
        
    } 
 
    public void actionPerformed(ActionEvent e){ 
        toggleSelection(list.getSelectedIndex()); 
    } 
} 

Usage:

CheckListManager checkListManager;
checkListManager = new CheckListManager(list);
ListSelectionModel selectionModel = checkListManager.getSelectionModel();

if(selectionModel.isSelectedIndex(i) == true)

NodeJS/ExpressJS Middlewares

NodeJS/ExpressJS Middleware:

The order in which middleware are defined in a file tells the order in which they would be applied to an incoming request.

To install a middleware :: npm install middleware-name –save

1) morgan for logging.
app.use(morgan(‘dev’));

2) express.static for serving static content
app.use(express.static(__dirname + ‘/public’));

Static files requested can only be sent using the GET request not the POST request. Create a folder named ‘public’ with the static content inside.

3) body-parser for parsing the json or other content in body of request

eg. var bodyParser = require(‘body-parser’);
app.use(bodyParser.json());

4) Express Generator: Quick scaffolding tool to generate an Express application skeleton.
npm install express-generator -g

Go into the folder in which you want to create the node express application skeleton:
express app-name (generates the application skeleton).

Now install the dependencies from package.json
npm install

Now you can start the express server with :
npm start

Express Generator: http://expressjs.com/en/starter/generator.html

 

Using MongoDB with NodeJS

http://blog.modulus.io/mongodb-tutorial

Install modules:
npm install mongodb –save
npm install assert –save

var MongoClient = require('mongodb').MongoClient,
assert = require('assert');

// Connection URL
var url = 'mongodb://localhost:27017/conFusion';

// Use connect method to connect to the Server
MongoClient.connect(url, function (err, db) {

    assert.equal(err,null);
    console.log("Connected correctly to server");
	
	var collection = db.collection("dishes");
	collection.insertOne({name: "Uthapizza", description: "test"}, function(err,result){
		assert.equal(err,null);
		console.log("After Insert:");
		console.log(result.ops);
			collection.find({}).toArray(function(err,docs){
				assert.equal(err,null);
				console.log("Found:");
				console.log(docs);
				db.dropCollection("dishes", function(err, result){
				   assert.equal(err,null);
				   db.close();
				});
			});
	});
});

MongoDB Concepts

Basics of MongoDB:

Database:
Is a physical container for collections.
Each database gets its own set of files on the file system.
A single MongoDB server typically has multiple databases.

Collection:
Is a group of MongoDB documents.
It is the equivalent of an RDBMS table.
A collection exists within a single database.
Collections do not enforce a schema.
Documents within a collection can have different fields.
Typically, all documents in a collection are of similar or related purpose.

Document:
Is effectively a JSON document with some additional features.
Is a set of key-value pairs.
Documents have dynamic schema.
Dynamic schema means that documents in the same collection do not need to have the same set of fields or structure, and common fields in a collection’s documents may hold different types of data.

MongoDB stores the documents in BSON (Binary JSON) format
– Supports length prefix on each value (Easy to skip over a field)
– Information about the type of a field value
– Additional primitives types not suppported by raw JSON like UTC date time, raw binary, and ObjectId


The data structure that defines a MongoDB document is called BSON (Binary JSON).
BSON is binary representation of JSON and also supports data types such as Date, which is not supported in plain JSON format.

MongoDB internally converts JSON to BSON and vice versa for performance benefits, although the user can save, query and retrieve as JSON.

MongoDB ObjectId
• Every document in Mongo must have an “_id” field that is unique
• Default ObjectId created by Mongo when you insert a document
• Example:
{
“_id” : ObjectId(“56ce74c0b02806eff4558f1f”),
“name” : “Uthapizza”,
“description” : “Test”
}

ObjectId is a 12 byte field:
• id.getTimestamp() returns the timestamp in ISO Date format.

var id = new ObjectId();
id.getTimestamp();

MongoDB Useful Commands

1) db.stats()
2) db.help()

Databases:
Create a new database: use name_of_db
Show all existing database: show dbs
Drop currently selected database: db.dropDatabase()
Insert in a database: db.name_of_db.insert(“name”:”abc”)

Collections:
To create a collection: db.createCollection(name, options)
In mongodb you don’t need to create collection. MongoDB creates collection automatically, when you insert some document.

Drop a collection: db.COLLECTION_NAME.drop()

Documents:
Insert Document:
we can insert a document with a insert or save command.
db.COLLECTION_NAME.insert(document)
db.COLLECTION_NAME.save(document)
If collection does not exist a new one is created.

Query Document: To query data from MongoDB collection, you need to use MongoDB’s find() method.
db.COLLECTION_NAME.find()

To get results in formatted way: db.mycol.find().pretty()
To get a single result in output file: db.mycol.findOne()

OR Query:
db.mycol.find(
{
$or: [
{key1: value1}, {key2:value2}
]
}
).pretty()

AND Query:
db.mycol.find({key1:value1, key2:value2}).pretty()


Update Document:
with update() and save() method
With update() method:
db.COLLECTION_NAME.update(SELECTIOIN_CRITERIA, UPDATED_DATA)

With save() method:
db.COLLECTION_NAME.save({_id:ObjectId(),NEW_DATA})

Remove Document:
db.COLLECTION_NAME.remove(DELLETION_CRITTERIA)

Remove only one document:
db.COLLECTION_NAME.remove(DELETION_CRITERIA, 1)

Remove all documents:
db.COLLECTION_NAME.remove()

 

Query Projection:
db.COLLECTION_NAME.find({},{KEY:1})

 

Limit the records:
db.COLLECTION_NAME.find().limit(NUMBER)

Skip the documents:
db.COLLECTION_NAME.find().limit(NUMBER).skip(NUMBER)

 

Sort the documents: 1 is used for ascending order while -1 is used for descending order.
db.COLLECTION_NAME.find().sort({KEY:1})

MongoDB Getting Started

To install mongodb on windows: https://code.msdn.microsoft.com/Mongo-Database-setup-on-6963f46f

Steps:
1) download the windows installer from mongodb.
2) install the msi file.
3) setup the path environment variable to the bin directory of the installation of mongodb.
4) run the command:

mongod --dbpath PATH_TO_DIRECTORY_WHERE_YOU_WANT_DB

Mongodb server will start waiting for a connection.
5) Now in an another cmd prompt:

mongo
db.test.save({ a: 1 })
db.test.find()

Now we have inserted values in the db.

CKAN, Proxy setting, Virtual box and etc

Set the proxy on the shell:
export ftp_proxy=proxy_url
export http_proxy=proxy_url
export https_proxy=proxy_url

Set proxy on git installation:
git config –global http.proxy  proxy_url
git config –global https.proxy proxy_url

Set proxy for pip:
pip install –proxy proxy_url -r /usr/lib/ckan/default/src/ckan/requirements.txt
sudo /usr/lib/ckan/datapusher/bin/pip install –proxy proxy_url -r requirements.txt

=====================================================================
File Store Backup and Restore

Backup :
paster db dump -c /etc/ckan/default/production.ini my_database_dump.sql

Restore:
paster db clean -c /etc/ckan/default/production.ini
paster db load -c /etc/ckan/default/production.ini my_database_dump.sql

paster db init -c /etc/ckan/default/development.ini

Create Admin:
paster sysadmin add user_name-c /etc/ckan/default/production.ini

Create test data:
paster create-test-data -c /etc/ckan/default/production.ini

tar -czvf /home/user/Desktop/filestore_backup.tar.gz /var/lib/ckan/default
rm -rf /var/lib/ckan/default/*
tar xvf /home/user/Desktop/filestore_backup.tar.gz

cp -r var/lib/ckan/default/* /var/lib/ckan/default/
chmod -R 777 /var/lib/ckan/default/
sudo chown –R apache /var/lib/ckan/default
==================================================================
/var/lib/pgsql

su – postgres
pg_dump -U postgres -W -F p ckan_default > dump_ckan_default.sql

dropdb -U ckan_default -W ckan_default

=================================================================
Mounting in VirtualBox  :
http://serverfault.com/questions/674974/how-to-mount-a-virtualbox-shared-folder/674978

mkdir share
sudo mount -t vboxsf wd ~/share/

================================================================

Datastore

Insert in datastore
curl -X POST http://127.0.0.1:5000/api/3/action/datastore_create -H “Authorization: db125a88-ece7-4f27-9e6b-8a1bbd13faf6” -d ‘{“resource”: {“package_id”: “d0136066-6daf-4791-83b8-3dbbc0513a31”}, “fields”: [ {“id”: “a”}, {“id”: “b”} ], “records”: [ { “a”: 1, “b”: “45”}, {“a”: 2, “b”: “78”}, {“a”: 3, “b”: “8”}
, {“a”: 4, “b”: “38”}, {“a”: 5, “b”: “118”} ]}’

curl -X POST http://127.0.0.1:5000/api/3/action/datastore_create -H “Authorization: 03c076ad-e1a1-41c8-bf9d-2a35678dfd3b” -d ‘{“resource”: {“package_id”: “e00a3583-ae57-4155-bbd1-ae14d54ca4f3”}, “fields”: [ {“id”: “a”}, {“id”: “b”} ], “records”: [ { “a”: 1, “b”: “aaa”}, {“a”: 2, “b”: “bbb”} ]}’

delete a resource
curl -X POST http://127.0.0.1:5000/api/3/action/datastore_delete -H “Authorization: 03c076ad-e1a1-41c8-bf9d-2a35678dfd3b” -d ‘{“resource_id”: “9016e26a-3b56-4ea3-9da3-9283a003c01d”}’

View a resource:
http://127.0.0.1:5000/api/3/action/datastore_search?resource_id=RESOURCE_ID

http://127.0.0.1:5000/api/3/action/datastore_search?resource_id=4cc335d5-62f2-47dc-9894-4b8dad818862

DataPusher Installation:
http://docs.ckan.org/projects/datapusher/en/latest/

#create a virtualenv for datapusher
sudo virtualenv /usr/lib/ckan/datapusher

#create a source directory and switch to it
sudo mkdir /usr/lib/ckan/datapusher/src
cd /usr/lib/ckan/datapusher/src

#clone the source (always target the stable branch)
sudo git clone -b stable https://github.com/ckan/datapusher.git

#install the DataPusher and its requirements
cd datapusher
sudo /usr/lib/ckan/datapusher/bin/pip install -r requirements.txt
sudo /usr/lib/ckan/datapusher/bin/python setup.py develop

#copy the standard Apache config file
# (use deployment/datapusher.apache2-4.conf if you are running under Apache 2.4)
sudo cp deployment/datapusher.conf /etc/httpd/conf.d/datapusher.conf

#copy the standard DataPusher wsgi file
#(see note below if you are not using the default CKAN install location)
sudo cp deployment/datapusher.wsgi /etc/ckan/

#copy the standard DataPusher settings.
sudo cp deployment/datapusher_settings.py /etc/ckan/

#open up port 8800 on Apache where the DataPusher accepts connections.
#make sure you only run these 2 functions once otherwise you will need
#to manually edit /etc/apache2/ports.conf.
sudo sh -c ‘echo “NameVirtualHost *:8800” >> /etc/httpd/ports.conf’
sudo sh -c ‘echo “Listen 8800” >> /etc/httpd/ports.conf’

#enable DataPusher Apache site
sudo a2ensite datapusher