The solution Homework 2.3 M101JS: MongoDB for Node.js Developers

You should see four files in the ‘blog’ directory: app.js, users.js, posts.js, and sessions.js. There is also a ‘views’ directory which contains the templates for the project and a ‘routes’ directory which contains our express routes.

If everything is working properly, you should be able to start the blog by typing:

npm install
node app.js

Note that this requires Node.js to be correctly installed on your computer.
After you run the blog, you should see the message:

Express server listening on port 8082

If you go to http://localhost:8082 you should see the front page of the blog. Here are some URLs that must work when you are done.

http://localhost:8082/signup
http://localhost:8082/login
http://localhost:8082/logout

When you login or sign-up, the blog will redirect to http://localhost:8082/welcome and that must work properly, welcoming the user by username.

We have removed parts of the code that uses the Node.js driver to query MongoDB from users.js and marked the area where you need to work with “TODO: hw2.3”. You should not need to touch any other code. The database calls that you are going to add will add a new user upon sign-up and validate a login by retrieving the right user document.

The blog stores its data in the blog database in two collections, users and sessions. Here are two example docs for a username search with password salty. You can insert these if you like, but you don’t need to.

> use blog
switched to db blog
> db.users.find()
{ "_id" : "sverch", "password" : "$2a$10$wl4bNB/5CqwWx4bB66PoQ.lmYvxUHigM1ehljyWQBupen3uCcldoW" }
> db.sessions.find()
{ "username" : "sverch", "_id" : "8d25917b27e4dc170d32491c6247aabba7598533" }
>

Once you have the the project working, the following steps should work:

go to http://localhost:8082/signup
create a user

It should redirect you to the welcome page and say: welcome username, where username is the user you signed up with. Now:

    Goto http://localhost:8082/logout
    Now login http://localhost:8082/login

Ok, now it’s time to validate you got it all working.

From the top of this page, there was one additional program that should have been downloaded:mongoProc.

With it, you can test your code and look at the Feedback section. When it says “user creation successful” and “user login successful”, you can Turn in your assignment.

Solution:

  • Open user.js
  • See changes below //TODO HW 2.3. Changes are done in two places.
var bcrypt = require('bcrypt-nodejs');
/* The UsersDAO must be constructed with a connected database object */
function UsersDAO(db) {
    "use strict";
    /* If this constructor is called without the "new" operator, "this" points
     * to the global object. Log a warning and call it correctly. */
    if (false === (this instanceof UsersDAO)) {
        console.log('Warning: UsersDAO constructor called without "new" operator');
        return new UsersDAO(db);
    }
    var users = db.collection("users");
    this.addUser = function(username, password, email, callback) {
        "use strict";
        // Generate password hash
        var salt = bcrypt.genSaltSync();
        var password_hash = bcrypt.hashSync(password, salt);
        // Create user document
        var user = {'_id': username, 'password': password_hash};
        // Add email if set
        if (email != "") {
            user['email'] = email;
        }
        // TODO: hw2.3
        users.insert(user, function (err, result) {
            "use strict";
            callback(err, result[0]);
        });
        //callback(Error("addUser Not Yet Implemented!"), null);
    }
    this.validateLogin = function(username, password, callback) {
        "use strict";
        // Callback to pass to MongoDB that validates a user document
        function validateUserDoc(err, user) {
            "use strict";

            if (err) return callback(err, null);

            if (user) {
                if (bcrypt.compareSync(password, user.password)) {
                    callback(null, user);
                }
                else {
                    var invalid_password_error = new Error("Invalid password");
                    // Set an extra field so we can distinguish this from a db error
                    invalid_password_error.invalid_password = true;
                    callback(invalid_password_error, null);
                }
            }
            else {
                var no_such_user_error = new Error("User: " + user + " does not exist");
                // Set an extra field so we can distinguish this from a db error
                no_such_user_error.no_such_user = true;
                callback(no_such_user_error, null);
            }
        }
        // TODO: hw2.3
        users.findOne({ '_id' : username}, function(err, user){
            if(err) throw err;
            validateUserDoc(err, user);
        });
       // callback(Error("validateLogin Not Yet Implemented!"), null);
    }
}
module.exports.UsersDAO = UsersDAO;
  • Save and run node js and perform signup and login in the browser if all are OK then open mongoProc and perform “Test” and if all seems OK then “Turn in”. While turning in I see the following screenshot.
m101js-homework_2.3

Let me know if get any problem so that we can solve each other help. As well as if you got any easy way to do it then please let me know.

The solution HW 2.2 M101JS: MongoDB for Node.js Developers

Write a program that finds the document with the highest recorded temperature for each state, and adds a “month_high” field for that document, setting its value to true. Use the weather dataset that you imported in HW 2.1.

From the top of this page, there was one additional program that should have been downloaded:mongoProc.

With it, you can test your code and look at the Feedback section. When it says “Correct amount of data documents.”, “Correct amount of month_high documents.” and “Correct month_high documents.”, you can Turn in your assignment.

Solution:

  • Create app.js
  • Paste the following code
var MongoClient = require('mongodb').MongoClient;
MongoClient.connect('mongodb://localhost:27017/weather', function(err, db) {
    if(err) throw err;
    db.collection('data').find().sort({"State" : 1, "Temperature" : -1}).toArray(function(err, docs) {
        if(err) throw err;
        var newstate = "";
        var query = {};
        docs.forEach(function (doc) {
            console.log("Doc : " + doc);
            if(newstate != doc.State) {
                // update row
                newstate = doc.Statdbe;
                query['_id'] = doc['_id'];
                var operator = { '$set' : { 'month_high' : true } };
                db.collection('data').update(query, operator, function(err, updated) {
                    if(err) throw err;
                    console.dir("Successfully updated " + updated + " document!");
                    return db.close();
                });
            }
        });
        db.close();
    });
});
  • Run with nodejs as node app.js
  • Check the result in mongo shell with query as db.data.find({“month_high”:true}); and result will be following:
{ "_id" : ObjectId("5627f4216f0eb671e142efbd"), "Day" : 23, "Time" : 1654, "State" : "Vermont", "Airport" : "BTV", "Temperature" : 57, "Humidity" : 78, "Wind Sp
eed" : 13, "Wind Direction" : 170, "Station Pressure" : 29.1, "Sea Level Pressure" : 982, "month_high" : true }
{ "_id" : ObjectId("5627f4216f0eb671e142f180"), "Day" : 11, "Time" : 1150, "State" : "California", "Airport" : "LAX", "Temperature" : 81, "Humidity" : 32, "Wind
 Speed" : 5, "Wind Direction" : 50, "Station Pressure" : 29.63, "Sea Level Pressure" : 151, "month_high" : true }
{ "_id" : ObjectId("5627f4216f0eb671e142f41d"), "Day" : 8, "Time" : 1453, "State" : "Florida", "Airport" : "ORL", "Temperature" : 83, "Humidity" : 55, "Wind Spe
ed" : 5, "Wind Direction" : 190, "Station Pressure" : 29.97, "Sea Level Pressure" : 195, "month_high" : true }
{ "_id" : ObjectId("5627f4226f0eb671e142f746"), "Day" : 11, "Time" : 1453, "State" : "New Mexico", "Airport" : "SAF", "Temperature" : 57, "Humidity" : 32, "Wind
 Speed" : 11, "Wind Direction" : 340, "Station Pressure" : 23.87, "Sea Level Pressure" : 174, "month_high" : true }
  • Then I run mongoProc and click “Test” button which shows data are fine then I click “Turn In” which shows the following screenshot.
m101js-homework_2.2

Let me know if get any problem so that we can solve each other help. As well as if you got any easy way to do it then please let me know.

The solution to Homework 6.5 M101JS: MongoDB for Node.js Developers

Homework: Homework 6.5

In this homework, you will build a small replica set on your own computer. We will check that it works with validate.js, which you should download from the Download Handout link.

Create three directories for the three mongod processes.

On UNIX or mac, this could be done as follows:

mkdir -p /data/rs1 /data/rs2 /data/rs3

Now start three mongo instances as follows. Note that are three commands. The browser is probably wrapping them visually.

mongod --replSet m101 --logpath "1.log" --dbpath /data/rs1 --port 27017 --smallfiles --oplogSize 64 --fork 
mongod --replSet m101 --logpath "2.log" --dbpath /data/rs2 --port 27018 --smallfiles --oplogSize 64 --fork
mongod --replSet m101 --logpath "3.log" --dbpath /data/rs3 --port 27019 --smallfiles --oplogSize 64 --fork

Windows users: Omit -p from mkdir. Also omit –fork and use start mongod with Windows compatible paths (i.e. backslashes “\”) for the –dbpath argument (e.g;C:\data\rs1).

Now connect to a mongo shell and make sure it comes up

mongo --port 27017

Now you will create the replica set. Type the following commands into the mongo shell:

config = { _id: "m101", members:[
          { _id : 0, host : "localhost:27017"},
          { _id : 1, host : "localhost:27018"},
          { _id : 2, host : "localhost:27019"} ]
         };
rs.initiate(config);

At this point, the replica set should be coming up. You can type

rs.status()

to see the state of replication.

Now install and run validate.js to confirm that it works.

npm install
node validate.js

Validate connects to your local replica set and checks that it has three nodes. Type the validation code below.

Solution: XdYfY6aqjqS3ik35qS6v   My validation code is above, but I hope it may differ for you so be careful before submitting.   How i achieve that validation code? These are my own steps so may differ for you:

  • Gone to C: drive data/ folder and created rs1/, rs2 and rs3 folders
  • Open command prompt and path to C:/data/rs1 and run following command: mongod –replSet m101 –logpath “1.log” –dbpath C:/data/rs1 –port 27017 –smallfiles –oplogSize 64
  • Then opened another command prompt and path to C:/data/rs2 and fun following command: mongod –replSet m101 –logpath “2.log” –dbpath C:/data/rs2 –port 27018 –smallfiles –oplogSize 64
  • Again opened another command prompt and path to C:/data/rs3 and run following command: mongod –replSet m101 –logpath “3.log” –dbpath C:/data/rs3 –port 27019 –smallfiles –oplogSize 64
  • Again open another command prompt and run following command: mongo –port 27017
  • Then mongo database shell get opened on which run following command: config = { _id: “m101”, members:[ { _id : 0, host : “localhost:27017”}, { _id : 1, host : “localhost:27018”}, { _id : 2, host : “localhost:27019”} ] }; rs.initiate(config); You will see following message if other database ports are not opened:
    {“ok” : 0, “errmsg” : “server is not running with –replSet” }
    If all ports databases are running then you will get following success message
    {
    “info” : “Config now saved locally. Should come online in about a minute.”,
    “ok” : 1
    }
  • Now check status of the replicated databases with following command: > rs.status() { “set” : “m101”, “date” : ISODate(“2015-11-22T20:24:21Z”), “myState” : 1, “members” : [ { “_id” : 0, “name” : “localhost:27017”, “health” : 1, “state” : 1, “stateStr” : “PRIMARY”, “uptime” : 225, “optime” : Timestamp(1448223786, 1), “optimeDate” : ISODate(“2015-11-22T20:23:06Z”), “electionTime” : Timestamp(1448223795, 1), “electionDate” : ISODate(“2015-11-22T20:23:15Z”), “self” : true }, { “_id” : 1, “name” : “localhost:27018”, “health” : 1, “state” : 2, “stateStr” : “SECONDARY”, “uptime” : 74, “optime” : Timestamp(1448223786, 1), “optimeDate” : ISODate(“2015-11-22T20:23:06Z”), “lastHeartbeat” : ISODate(“2015-11-22T20:24:19Z”), “lastHeartbeatRecv” : ISODate(“2015-11-22T20:24:19Z”), “pingMs” : 0, “syncingTo” : “localhost:27017” }, { “_id” : 2, “name” : “localhost:27019”, “health” : 1, “state” : 2, “stateStr” : “SECONDARY”, “uptime” : 74, “optime” : Timestamp(1448223786, 1), “optimeDate” : ISODate(“2015-11-22T20:23:06Z”), “lastHeartbeat” : ISODate(“2015-11-22T20:24:19Z”), “lastHeartbeatRecv” : ISODate(“2015-11-22T20:24:20Z”), “pingMs” : 0, “syncingTo” : “localhost:27017” } ], “ok” : 1 } m101:PRIMARY> It shows all are OK
  • Now download the handout provided for the assignment, then unzip it.
  • Run another command prompt again and go to the path where you unzip your assignment and to hw6-5/ folder and run the following command: npm install node validate.js
  • You will see validation code as following in command line:

This way I have achieved my validation code, I hope it will be the same for you.
Please let me know if there is any way that I can run multiple command prompt instead of Opening multiple open.
Likewise, let me know if there is any simpler way.

The solution to Homework 6.4 M101JS: MongoDB for Node.js Developers

Homework: Homework 6.4

You have a sharded system with three shards and have sharded the collections “students” in the “school” database across those shards. The output of sh.status() when connected to Mongos looks like this:

mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
	"_id" : 1,
	"minCompatibleVersion" : 5,
	"currentVersion" : 6,
	"clusterId" : ObjectId("5531512ac723271f602db407")
}
  shards:
	{  "_id" : "s0",  "host" : "s0/localhost:37017,localhost:37018,localhost:37019" }
	{  "_id" : "s1",  "host" : "s1/localhost:47017,localhost:47018,localhost:47019" }
	{  "_id" : "s2",  "host" : "s2/localhost:57017,localhost:57018,localhost:57019" }
  balancer:
	Currently enabled:  yes
	Currently running:  yes
		Balancer lock taken at Fri Apr 17 2015 14:32:02 GMT-0400 (EDT) by education-iMac-2.local:27017:1429295401:16807:Balancer:1622650073
	Collections with active migrations: 
		school.students started at Fri Apr 17 2015 14:32:03 GMT-0400 (EDT)
	Failed balancer rounds in last 5 attempts:  0
	Migration Results for the last 24 hours: 
		2 : Success
		1 : Failed with error 'migration already in progress', from s0 to s1
  databases:
	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
	{  "_id" : "school",  "partitioned" : true,  "primary" : "s0" }
		school.students
			shard key: { "student_id" : 1 }
			chunks:
				s0	1
				s1	3
				s2	1
			{ "student_id" : { "$minKey" : 1 } } -->> { "student_id" : 0 } on : s2 Timestamp(3, 0) 
			{ "student_id" : 0 } -->> { "student_id" : 2 } on : s0 Timestamp(3, 1) 
			{ "student_id" : 2 } -->> { "student_id" : 3497 } on : s1 Timestamp(3, 2) 
			{ "student_id" : 3497 } -->> { "student_id" : 7778 } on : s1 Timestamp(3, 3) 
			{ "student_id" : 7778 } -->> { "student_id" : { "$maxKey" : 1 } } on : s1 Timestamp(3, 4) 


If you ran the query

use school
db.students.find({'student_id':2000})

Which shards would be involved in answering the query?  
Answer: s1

hw6.4answer

The solution to Homework 6.3 M101JS: MongoDB for Node.js Developers

Homework: Homework 6.3

Which of the following statements are true about choosing and using a shard key? MongoDB can not enforce unique indexes on a sharded collection other than the shard key itself, or indexes prefixed by the shard key. There must be an index on the collection that starts with the shard key. You can change the shard key on a collection if you desire. Any update that does not contain the shard key will be sent to all shards. The shared key must be unique

hw6.3answer

The solution to Homework 6.2 M101JS: MongoDB for Node.js Developers

Homework: Homework 6.2

Let’s suppose you have a five-member replica set and want to assure that writes are committed to the journal and are acknowledged by at least 3 nodes before you proceed forward. What would be the appropriate settings for w and j?  

Solution: w=”majority”

hw6.2answer

The solution to Homework 6.1 M101JS: MongoDB for Node.js Developers

Homework: Homework 6.1

Which of the following statements are true about replication in MongoDB? Check all that apply.

hw6.1answer

The minimum sensible number of voting nodes to a replica set is three.
MongoDB replication is synchronous.
By default, using the new MongoClient connection class, w=1 and j=1.
The oplog utilizes a capped collection.

The solution to Homework 5.4 M101JS: MongoDB for Node.js Developers

The solution to Homework 5.4 M101JS: MongoDB for Node.js Developers Homework: Homework 5.4

Removing Rural Residents

In this problem, you will calculate the number of people who live in a zip code in the US where the city starts with a digit. We will take that to mean they don’t really live in a city. Once again, you will be using the zip code collection, which you will find in the ‘handouts’ link on this page. Import it into your MongoDB using the following command from the command line:

> mongoimport -d test -c zips --drop zips.json

If you imported it correctly, you can go to the test database in the mongo shell and confirm that

> db.zips.count()

yields 29,467 documents.

The project operator can extract the first digit from any field. For example, to extract the first digit from the city field, you could write this query:

db.zips.aggregate([
    {$project: 
     {
	first_char: {$substr : ["$city",0,1]},
     }	 
   }
])

Using the aggregation framework, calculate the sum total of people who are living in a zip code where the city starts with a digit. Choose the answer below.

You will need to probably change your projection to send more info through than just that first character. Also, you will need a filtering step to get rid of all documents where the city does not start with a digital (0-9).

Note: When you mongoimport the data, you will probably see a few duplicate key errors; this is to be expected, and will not prevent the mongoimport from working. There is also an issue with some versions of MongoDB 3.0 where it claims that 0 documents were mongoimported, when in fact there were 29,467 documents imported. You can verify this for yourself by going into the shell and counting the documents in the “test.zips” collection.

Solution:

Run the following queries:

db.zips.aggregate([
    { $project: { _id: 0, city: 1, pop: 1 } },
    { $match: { city: /^\d.*/ } },
    { $group: { _id: null, pop: { $sum: "$pop" } } },
    { $sort: { city: 1} }
])
hw5.4answer2

Seeing the above result I submit the 298015 and it is correct for me:

hw5.4answer

The solution to Homework 5.3 (Hands-On) M101JS: MongoDB for Node.js Developers

The solution to Homework 5.3 (Hands-On) M101JS: MongoDB for Node.js DevelopersWho’s the easiest grader on campus?
A set of grades are loaded into the grades collection. The documents look like this:

{
	"_id" : ObjectId("50b59cd75bed76f46522c392"),
	"student_id" : 10,
	"class_id" : 5,
	"scores" : [
		{
			"type" : "exam",
			"score" : 69.17634380939022
		},
		{
			"type" : "quiz",
			"score" : 61.20182926719762
		},
		{
			"type" : "homework",
			"score" : 73.3293624199466
		},
		{
			"type" : "homework",
			"score" : 15.206314042622903
		},
		{
			"type" : "homework",
			"score" : 36.75297723087603
		},
		{
			"type" : "homework",
			"score" : 64.42913107330241
		}
	]
}

There are documents for each student (student_id) across a variety of classes (class_id). Note that not all students in the same class have the same exact number of assessments. Some students have three homework assignments, etc.

Your task is to calculate the class with the best average student performance. This involves calculating an average for each student in each class of all non-quiz assessments and then averaging those numbers to get a class average. To be clear, each student’s average includes only exams and homework grades. Don’t include their quiz scores in the calculation.

What is the class_id which has the highest average student performance?

Hint/Strategy: You need to group twice to solve this problem. You must figure out the GPA that each student has achieved in a class and then average those numbers to get a class average. After that, you just need to sort. The class with the lowest average is the class with class_id=2. Those students achieved a class average of 37.6

If you prefer, you may download the handout and perform your analysis on your machine with

 mongoimport -d test -c grades --drop grades.json

Solution:

Run the following SQL and the first id that you will get be the solution.

db.grades.aggregate([
    { $unwind: "$scores" },
    { $match: { $or: [ {"scores.type": "homework"}, {"scores.type":"exam"} ] } },
    { $group: { _id: { 'student_id': "$student_id", 'class_id': "$class_id" }, avg: { $avg: "$scores.score" } } },
    { $group: { _id: "$_id.class_id", class_avg: { $avg: "$avg" } } },
    { $sort: { 'class_avg': -1 } }
])

Following is the result that I got:

hw5.3answer2

So the answer is:

hw5.3answer

The solution to Homework 5.2 (Hands-On) M101JS: MongoDB for Node.js Developers

The solution to Homework 5.2 (Hands-On) M101JS: MongoDB for Node.js Developers

Crunching the Zipcode dataset
Please calculate the average population of cities in California (abbreviation CA) and New York (NY) (taken together) with populations over 25,000. For this problem, assume that a city name that appears in more than one state represents two separate cities. Please round the answer to a whole number.
Hint: The answer for CT and NJ (using this data set) is 38177.

Please note:

  • Different states might have the same city name.
  • A city might have multiple zip codes.

For purposes of keeping the Hands-On shell quick, we have used a subset of the data you previously used in zips.json, not the full set. This is why there are only 200 documents (and 200 zip codes), and all of them are in New York, Connecticut, New Jersey, and California.

If you prefer, you may download the handout and perform your analysis on your machine with

> mongoimport -d test -c zips --drop small_zips.json

Solution:

Run the following query and find the answer:

db.zips.aggregate([
    { $match: {$or: [ {state: "CA"}, {state: "NY"} ] } },
    { $group: { _id: { city: "$city" }, pop: { $sum: "$pop" } } },
    { $match: { "pop": { $gt: 25000 } } },
    { $group: { _id: null, avg_pop_of_city: { $avg: "$pop" } } }
])

I got the following answer and submit and it is correct for me. let me know if you got any other results and an easy way to proceed through it.

hw5.2answer