Home Blog Page 23

%type: !message in %function (line %line of %file) Drupal error

I keep on getting this error logging at watchdog table in the database while loading front end or home page: %type: !message in %function (line %line of %file) Drupal error I spend around 20 min, keep on clicking the “Clear All Caches” button at Performance section. I checked the code and found the following:

function watchdog_exception($type, Exception $exception, $message = NULL, $variables = array(), $severity = WATCHDOG_ERROR, $link = NULL) {

 // Use a default value if $message is not set.
 if (empty($message)) {
 // The exception message is run through check_plain() by _drupal_decode_exception().
 $message = '%type: !message in %function (line %line of %file).';
 }
 // $variables must be an array so that we can add the exception information.
 if (!is_array($variables)) {
 $variables = array();
 }

 require_once DRUPAL_ROOT . '/includes/errors.inc';
 $variables += _drupal_decode_exception($exception);
 watchdog($type, $message, $variables, $severity, $link);
}

If the message is not set then the given above log is set.

Instantly I cleared browser cache and here you go the page is loading well.

Maybe this is the only case for me, but keep on posting if I find any other solution.

The solution Homework 3.1 M101JS: MongoDB for Node.js Developers

Write a program in the language of your choice that will remove the lowest homework score for each student. Since there is a single document for each student containing an array of scores, you will need to update the scores array and remove the homework.

Remember, just remove a homework score. Don’t remove a quiz or an exam!

Hint/spoiler: With the new schema, this problem is a lot harder and that is sort of the point. One way is to find the lowest homework in code and then update the scores array with the low homework pruned.

To confirm you are on the right track, here are some queries to run after you process the data with the correct answer shown:

Let us count the number of students we have:

  1. use school
  2. DB.students.count()

The answer will be 200.

Let’s see what Tamika Schildgen’s record looks like:

  1. db.students.find( { _id : 137 } ).pretty( )

This should be the output:

{
	"_id" : 137,
	"name" : "Tamika Schildgen",
	"scores" : [
		{
			"type" : "exam",
			"score" : 4.433956226109692
		},
		{
			"type" : "quiz",
			"score" : 65.50313785402548
		},
		{
			"type" : "homework",
			"score" : 89.5950384993947
		}
	]
}

To verify that you have completed this task correctly, provide the identity (in the form of their _id) of the student with the highest average in the class with following query that uses the aggregation framework. The answer will appear in the _id field of the resulting document.

db.students.aggregate( { ‘$unwind’ : ‘$scores’ } , { ‘$group’ : { ‘_id’ : ‘$_id’ , ‘average’ : { $avg : ‘$scores.score’ } } } , { ‘$sort’ : { ‘average’ : -1 } } , { ‘$limit’ : 1 } )

Solution:

13

Enter 13 and submit.

Create a file app.js and insert the following code:

var MongoClient = require('mongodb').MongoClient;

MongoClient.connect('mongodb://localhost:27017/school', function(err, db){
  if(err) throw err;
  var query = {};

  db.collection('students').find(query).toArray(function(err, docs) {
    if(err) throw err;

    var min;
    var doc;
    var value;
    for (var i = 0; i < docs.length; i++){
        doc = docs[i];
        min = -1;
        for(var j = 0; j < doc.scores.length; j++){
           value = doc.scores[j];
           if(value.type === "homework"){
             if(min === -1 || value.score < doc.scores[min].score ){
               min = j;
             }
           }
        }
        console.log("Ant length :" + doc.scores.length + " min : " + min);
        // important: Array.splice need the number of elements to delete...  splice(index, )
        doc.scores.splice(min, 1);
        console.log("Pos length :" + doc.scores.length);
        db.collection('students').save(doc, function(err, saved) {
          if(err) throw err;
          console.log("Doc saved: " + saved);

        });
    }
    //console.dir("Result" + docs);
    console.dir("End");
    return db.close();
  });

});

Now run it with node app.js

Check your result with the following query:

db.students.find( { _id : 13 } ).pretty( )

Result:

{
        "_id" : 13,
        "name" : "Jessika Dagenais",
        "scores" : [
                {
                        "type" : "exam",
                        "score" : 90.47179954427436
                },
                {
                        "type" : "quiz",
                        "score" : 90.3001402468489
                },
                {
                        "type" : "homework",
                        "score" : 95.17753772405909
                }
        ]
}

You can see the lowest homework scores are removed from all documents.

Now again run

db.students.aggregate( { '$unwind' : '$scores' } , { '$group' : { '_id' : '$_id' , 'average' : { $avg : '$scores.score' } } } , { '$sort' : { 'average' : -1 }, { '$limit' : 1 } )

The result will be:

{ "_id" : 13, "average" : 91.98315917172745 }

I have submitted “13” and the answer is correct but if you have any great solution then please let me know as well as if any problem then let me know so that we can solve together.

Final: Question 1 M101JS: MongoDB for Node.js Developers

Final: Question 1

Please download the Enron email dataset enron.zip, unzip it and then restore it using mongorestore. It should restore to a collection called “messages” in a database called “enron”. Note that this is an abbreviated version of the full corpus. There should be 120,477 documents after restore.

Inspect a few of the documents to get a basic understanding of the structure. Enron was an American corporation that engaged in widespread accounting fraud and subsequently failed.

In this dataset, each document is an email message. Like all Email messages, there is one sender but there can be multiple recipients.

Construct a query to calculate the number of messages sent by Andrew Fastow, CFO, to Jeff Skilling, the president. Andrew Fastow’s email address was andrew.fastow@enron.com. Jeff Skilling’s email was jeff.skilling@enron.com.

For reference, the number of email messages from Andrew Fastow to John Lavorato (john.lavorato@enron.com) was 1.Solution: 3  

  • Download the  enron.zip and extract it
  • Now run “mongod”
  • Now import the extracted database mongorestore –drop –db enron dump/enron
  • Now run “mongo”
  • See if Enron database is imported with “show databases”
  • In the list, if you see enron then you are ready to go.
  • Now run “use enron”
  • Then “show collections”
  • You will see messages collection
  • Check your data with “db.messages.findOne()” { “_id” : ObjectId(“4f16fc97d1e2d32371003f02”), “body” : “COURTYARD\n\nMESQUITE\n2300 HWY 67\nMESQUITE, TX 75150\ntel: 972-681-3300\nfax: 972-681-3324\n\nHotel Information: http://courtyard.com/DALCM \n\n\nARRIVAL CONFIRMATION:\n Confirmation Number:84029698\nGuests in Room: 2\nNAME: MR ERIC BASS \nGuest Phone: 7138530977\nNumber of Rooms:1\nArrive: Oct 6 2 001\nDepart: Oct 7 2001\nRoom Type: ROOM – QUALITY\nGuarantee Method:\n Credit card guarantee\nCANCELLATION PERMITTED-BEFORE 1800 DAY OF ARRIVAL\n\nRATE INFORMA TION:\nRate(s) Quoted in: US DOLLAR\nArrival Date: Oct 6 2001\nRoom Rate: 62.10 per night. Plus tax when applicable\nRate Program: AAA AMERICAN AUTO ASSN\n\nSP ECIAL REQUEST:\n NON-SMOKING ROOM, GUARANTEED\n \n\n\nPLEASE DO NOT REPLY TO THIS EMAIL \nAny Inquiries Please call 1-800-321-2211 or your local\ninternationa l toll free number.\n \nConfirmation Sent: Mon Jul 30 18:19:39 2001\n\nLegal Disclaimer:\nThis confirmation notice has been transmitted to you by electronic\nma il for your convenience. Marriott’s record of this confirmation\nnotice is the official record of this reservation. Subsequent\nalterations to this electronic m essage after its transmission\nwill be disregarded.\n\nMarriott is pleased to announce that High Speed Internet Access is\nbeing rolled out in all Marriott hote l brands around the world.\nTo learn more or to find out whether your hotel has the service\navailable, please visit Marriott.com.\n\nEarn points toward free va cations, or frequent flyer miles\nfor every stay you make! Just provide your Marriott Rewards\nmembership number at check in. Not yet a member? Join for free at\nhttps://member.marriottrewards.com/Enrollments/enroll.asp?source=MCRE\n\n”, “filename” : “2.”, “headers” : { “Content-Transfer-Encoding” : “7bit”, “Content-Type” : “text/plain; charset=us-ascii”, “Date” : ISODate(“2001-07-30T22:19:40Z”), “From” : “reservations@marriott.com”, “Message-ID” : “<32788362.1075840323896.JavaMail.evans@thyme>”, “Mime-Version” : “1.0”, “Subject” : “84029698 Marriott Reservation Confirmation Number”, “To” : [ “ebass@enron.com” ], “X-FileName” : “eric bass 6-25-02.PST”, “X-Folder” : “\\ExMerge – Bass, Eric\\Personal”, “X-From” : “Reservations@Marriott.com”, “X-Origin” : “BASS-E”, “X-To” : “EBASS@ENRON.COM”, “X-bcc” : “”, “X-cc” : “” }, “mailbox” : “bass-e”, “subFolder” : “personal” }
  • Now run following query to get the output
    • db.messages.find({“headers.From”:”andrew.fastow@enron.com”,”headers.To”:”jeff.skilling@enron.com”}).count()
  • Confirm you the result, I got “3” as an answer.
final_exam_question_1

The solution Homework 2.3 M101JS: MongoDB for Node.js Developers

You should see four files in the ‘blog’ directory: app.js, users.js, posts.js, and sessions.js. There is also a ‘views’ directory which contains the templates for the project and a ‘routes’ directory which contains our express routes.

If everything is working properly, you should be able to start the blog by typing:

npm install
node app.js

Note that this requires Node.js to be correctly installed on your computer.
After you run the blog, you should see the message:

Express server listening on port 8082

If you go to http://localhost:8082 you should see the front page of the blog. Here are some URLs that must work when you are done.

http://localhost:8082/signup
http://localhost:8082/login
http://localhost:8082/logout

When you login or sign-up, the blog will redirect to http://localhost:8082/welcome and that must work properly, welcoming the user by username.

We have removed parts of the code that uses the Node.js driver to query MongoDB from users.js and marked the area where you need to work with “TODO: hw2.3”. You should not need to touch any other code. The database calls that you are going to add will add a new user upon sign-up and validate a login by retrieving the right user document.

The blog stores its data in the blog database in two collections, users and sessions. Here are two example docs for a username search with password salty. You can insert these if you like, but you don’t need to.

> use blog
switched to db blog
> db.users.find()
{ "_id" : "sverch", "password" : "$2a$10$wl4bNB/5CqwWx4bB66PoQ.lmYvxUHigM1ehljyWQBupen3uCcldoW" }
> db.sessions.find()
{ "username" : "sverch", "_id" : "8d25917b27e4dc170d32491c6247aabba7598533" }
>

Once you have the the project working, the following steps should work:

go to http://localhost:8082/signup
create a user

It should redirect you to the welcome page and say: welcome username, where username is the user you signed up with. Now:

    Goto http://localhost:8082/logout
    Now login http://localhost:8082/login

Ok, now it’s time to validate you got it all working.

From the top of this page, there was one additional program that should have been downloaded:mongoProc.

With it, you can test your code and look at the Feedback section. When it says “user creation successful” and “user login successful”, you can Turn in your assignment.

Solution:

  • Open user.js
  • See changes below //TODO HW 2.3. Changes are done in two places.
var bcrypt = require('bcrypt-nodejs');
/* The UsersDAO must be constructed with a connected database object */
function UsersDAO(db) {
    "use strict";
    /* If this constructor is called without the "new" operator, "this" points
     * to the global object. Log a warning and call it correctly. */
    if (false === (this instanceof UsersDAO)) {
        console.log('Warning: UsersDAO constructor called without "new" operator');
        return new UsersDAO(db);
    }
    var users = db.collection("users");
    this.addUser = function(username, password, email, callback) {
        "use strict";
        // Generate password hash
        var salt = bcrypt.genSaltSync();
        var password_hash = bcrypt.hashSync(password, salt);
        // Create user document
        var user = {'_id': username, 'password': password_hash};
        // Add email if set
        if (email != "") {
            user['email'] = email;
        }
        // TODO: hw2.3
        users.insert(user, function (err, result) {
            "use strict";
            callback(err, result[0]);
        });
        //callback(Error("addUser Not Yet Implemented!"), null);
    }
    this.validateLogin = function(username, password, callback) {
        "use strict";
        // Callback to pass to MongoDB that validates a user document
        function validateUserDoc(err, user) {
            "use strict";

            if (err) return callback(err, null);

            if (user) {
                if (bcrypt.compareSync(password, user.password)) {
                    callback(null, user);
                }
                else {
                    var invalid_password_error = new Error("Invalid password");
                    // Set an extra field so we can distinguish this from a db error
                    invalid_password_error.invalid_password = true;
                    callback(invalid_password_error, null);
                }
            }
            else {
                var no_such_user_error = new Error("User: " + user + " does not exist");
                // Set an extra field so we can distinguish this from a db error
                no_such_user_error.no_such_user = true;
                callback(no_such_user_error, null);
            }
        }
        // TODO: hw2.3
        users.findOne({ '_id' : username}, function(err, user){
            if(err) throw err;
            validateUserDoc(err, user);
        });
       // callback(Error("validateLogin Not Yet Implemented!"), null);
    }
}
module.exports.UsersDAO = UsersDAO;
  • Save and run node js and perform signup and login in the browser if all are OK then open mongoProc and perform “Test” and if all seems OK then “Turn in”. While turning in I see the following screenshot.
m101js-homework_2.3

Let me know if get any problem so that we can solve each other help. As well as if you got any easy way to do it then please let me know.

The solution HW 2.2 M101JS: MongoDB for Node.js Developers

Write a program that finds the document with the highest recorded temperature for each state, and adds a “month_high” field for that document, setting its value to true. Use the weather dataset that you imported in HW 2.1.

From the top of this page, there was one additional program that should have been downloaded:mongoProc.

With it, you can test your code and look at the Feedback section. When it says “Correct amount of data documents.”, “Correct amount of month_high documents.” and “Correct month_high documents.”, you can Turn in your assignment.

Solution:

  • Create app.js
  • Paste the following code
var MongoClient = require('mongodb').MongoClient;
MongoClient.connect('mongodb://localhost:27017/weather', function(err, db) {
    if(err) throw err;
    db.collection('data').find().sort({"State" : 1, "Temperature" : -1}).toArray(function(err, docs) {
        if(err) throw err;
        var newstate = "";
        var query = {};
        docs.forEach(function (doc) {
            console.log("Doc : " + doc);
            if(newstate != doc.State) {
                // update row
                newstate = doc.Statdbe;
                query['_id'] = doc['_id'];
                var operator = { '$set' : { 'month_high' : true } };
                db.collection('data').update(query, operator, function(err, updated) {
                    if(err) throw err;
                    console.dir("Successfully updated " + updated + " document!");
                    return db.close();
                });
            }
        });
        db.close();
    });
});
  • Run with nodejs as node app.js
  • Check the result in mongo shell with query as db.data.find({“month_high”:true}); and result will be following:
{ "_id" : ObjectId("5627f4216f0eb671e142efbd"), "Day" : 23, "Time" : 1654, "State" : "Vermont", "Airport" : "BTV", "Temperature" : 57, "Humidity" : 78, "Wind Sp
eed" : 13, "Wind Direction" : 170, "Station Pressure" : 29.1, "Sea Level Pressure" : 982, "month_high" : true }
{ "_id" : ObjectId("5627f4216f0eb671e142f180"), "Day" : 11, "Time" : 1150, "State" : "California", "Airport" : "LAX", "Temperature" : 81, "Humidity" : 32, "Wind
 Speed" : 5, "Wind Direction" : 50, "Station Pressure" : 29.63, "Sea Level Pressure" : 151, "month_high" : true }
{ "_id" : ObjectId("5627f4216f0eb671e142f41d"), "Day" : 8, "Time" : 1453, "State" : "Florida", "Airport" : "ORL", "Temperature" : 83, "Humidity" : 55, "Wind Spe
ed" : 5, "Wind Direction" : 190, "Station Pressure" : 29.97, "Sea Level Pressure" : 195, "month_high" : true }
{ "_id" : ObjectId("5627f4226f0eb671e142f746"), "Day" : 11, "Time" : 1453, "State" : "New Mexico", "Airport" : "SAF", "Temperature" : 57, "Humidity" : 32, "Wind
 Speed" : 11, "Wind Direction" : 340, "Station Pressure" : 23.87, "Sea Level Pressure" : 174, "month_high" : true }
  • Then I run mongoProc and click “Test” button which shows data are fine then I click “Turn In” which shows the following screenshot.
m101js-homework_2.2

Let me know if get any problem so that we can solve each other help. As well as if you got any easy way to do it then please let me know.

The solution to Homework 6.5 M101JS: MongoDB for Node.js Developers

Homework: Homework 6.5

In this homework, you will build a small replica set on your own computer. We will check that it works with validate.js, which you should download from the Download Handout link.

Create three directories for the three mongod processes.

On UNIX or mac, this could be done as follows:

mkdir -p /data/rs1 /data/rs2 /data/rs3

Now start three mongo instances as follows. Note that are three commands. The browser is probably wrapping them visually.

mongod --replSet m101 --logpath "1.log" --dbpath /data/rs1 --port 27017 --smallfiles --oplogSize 64 --fork 
mongod --replSet m101 --logpath "2.log" --dbpath /data/rs2 --port 27018 --smallfiles --oplogSize 64 --fork
mongod --replSet m101 --logpath "3.log" --dbpath /data/rs3 --port 27019 --smallfiles --oplogSize 64 --fork

Windows users: Omit -p from mkdir. Also omit –fork and use start mongod with Windows compatible paths (i.e. backslashes “\”) for the –dbpath argument (e.g;C:\data\rs1).

Now connect to a mongo shell and make sure it comes up

mongo --port 27017

Now you will create the replica set. Type the following commands into the mongo shell:

config = { _id: "m101", members:[
          { _id : 0, host : "localhost:27017"},
          { _id : 1, host : "localhost:27018"},
          { _id : 2, host : "localhost:27019"} ]
         };
rs.initiate(config);

At this point, the replica set should be coming up. You can type

rs.status()

to see the state of replication.

Now install and run validate.js to confirm that it works.

npm install
node validate.js

Validate connects to your local replica set and checks that it has three nodes. Type the validation code below.

Solution: XdYfY6aqjqS3ik35qS6v   My validation code is above, but I hope it may differ for you so be careful before submitting.   How i achieve that validation code? These are my own steps so may differ for you:

  • Gone to C: drive data/ folder and created rs1/, rs2 and rs3 folders
  • Open command prompt and path to C:/data/rs1 and run following command: mongod –replSet m101 –logpath “1.log” –dbpath C:/data/rs1 –port 27017 –smallfiles –oplogSize 64
  • Then opened another command prompt and path to C:/data/rs2 and fun following command: mongod –replSet m101 –logpath “2.log” –dbpath C:/data/rs2 –port 27018 –smallfiles –oplogSize 64
  • Again opened another command prompt and path to C:/data/rs3 and run following command: mongod –replSet m101 –logpath “3.log” –dbpath C:/data/rs3 –port 27019 –smallfiles –oplogSize 64
  • Again open another command prompt and run following command: mongo –port 27017
  • Then mongo database shell get opened on which run following command: config = { _id: “m101”, members:[ { _id : 0, host : “localhost:27017”}, { _id : 1, host : “localhost:27018”}, { _id : 2, host : “localhost:27019”} ] }; rs.initiate(config); You will see following message if other database ports are not opened:
    {“ok” : 0, “errmsg” : “server is not running with –replSet” }
    If all ports databases are running then you will get following success message
    {
    “info” : “Config now saved locally. Should come online in about a minute.”,
    “ok” : 1
    }
  • Now check status of the replicated databases with following command: > rs.status() { “set” : “m101”, “date” : ISODate(“2015-11-22T20:24:21Z”), “myState” : 1, “members” : [ { “_id” : 0, “name” : “localhost:27017”, “health” : 1, “state” : 1, “stateStr” : “PRIMARY”, “uptime” : 225, “optime” : Timestamp(1448223786, 1), “optimeDate” : ISODate(“2015-11-22T20:23:06Z”), “electionTime” : Timestamp(1448223795, 1), “electionDate” : ISODate(“2015-11-22T20:23:15Z”), “self” : true }, { “_id” : 1, “name” : “localhost:27018”, “health” : 1, “state” : 2, “stateStr” : “SECONDARY”, “uptime” : 74, “optime” : Timestamp(1448223786, 1), “optimeDate” : ISODate(“2015-11-22T20:23:06Z”), “lastHeartbeat” : ISODate(“2015-11-22T20:24:19Z”), “lastHeartbeatRecv” : ISODate(“2015-11-22T20:24:19Z”), “pingMs” : 0, “syncingTo” : “localhost:27017” }, { “_id” : 2, “name” : “localhost:27019”, “health” : 1, “state” : 2, “stateStr” : “SECONDARY”, “uptime” : 74, “optime” : Timestamp(1448223786, 1), “optimeDate” : ISODate(“2015-11-22T20:23:06Z”), “lastHeartbeat” : ISODate(“2015-11-22T20:24:19Z”), “lastHeartbeatRecv” : ISODate(“2015-11-22T20:24:20Z”), “pingMs” : 0, “syncingTo” : “localhost:27017” } ], “ok” : 1 } m101:PRIMARY> It shows all are OK
  • Now download the handout provided for the assignment, then unzip it.
  • Run another command prompt again and go to the path where you unzip your assignment and to hw6-5/ folder and run the following command: npm install node validate.js
  • You will see validation code as following in command line:

This way I have achieved my validation code, I hope it will be the same for you.
Please let me know if there is any way that I can run multiple command prompt instead of Opening multiple open.
Likewise, let me know if there is any simpler way.

The solution to Homework 6.4 M101JS: MongoDB for Node.js Developers

Homework: Homework 6.4

You have a sharded system with three shards and have sharded the collections “students” in the “school” database across those shards. The output of sh.status() when connected to Mongos looks like this:

mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
	"_id" : 1,
	"minCompatibleVersion" : 5,
	"currentVersion" : 6,
	"clusterId" : ObjectId("5531512ac723271f602db407")
}
  shards:
	{  "_id" : "s0",  "host" : "s0/localhost:37017,localhost:37018,localhost:37019" }
	{  "_id" : "s1",  "host" : "s1/localhost:47017,localhost:47018,localhost:47019" }
	{  "_id" : "s2",  "host" : "s2/localhost:57017,localhost:57018,localhost:57019" }
  balancer:
	Currently enabled:  yes
	Currently running:  yes
		Balancer lock taken at Fri Apr 17 2015 14:32:02 GMT-0400 (EDT) by education-iMac-2.local:27017:1429295401:16807:Balancer:1622650073
	Collections with active migrations: 
		school.students started at Fri Apr 17 2015 14:32:03 GMT-0400 (EDT)
	Failed balancer rounds in last 5 attempts:  0
	Migration Results for the last 24 hours: 
		2 : Success
		1 : Failed with error 'migration already in progress', from s0 to s1
  databases:
	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
	{  "_id" : "school",  "partitioned" : true,  "primary" : "s0" }
		school.students
			shard key: { "student_id" : 1 }
			chunks:
				s0	1
				s1	3
				s2	1
			{ "student_id" : { "$minKey" : 1 } } -->> { "student_id" : 0 } on : s2 Timestamp(3, 0) 
			{ "student_id" : 0 } -->> { "student_id" : 2 } on : s0 Timestamp(3, 1) 
			{ "student_id" : 2 } -->> { "student_id" : 3497 } on : s1 Timestamp(3, 2) 
			{ "student_id" : 3497 } -->> { "student_id" : 7778 } on : s1 Timestamp(3, 3) 
			{ "student_id" : 7778 } -->> { "student_id" : { "$maxKey" : 1 } } on : s1 Timestamp(3, 4) 


If you ran the query

use school
db.students.find({'student_id':2000})

Which shards would be involved in answering the query?  
Answer: s1

hw6.4answer

The solution to Homework 6.3 M101JS: MongoDB for Node.js Developers

Homework: Homework 6.3

Which of the following statements are true about choosing and using a shard key? MongoDB can not enforce unique indexes on a sharded collection other than the shard key itself, or indexes prefixed by the shard key. There must be an index on the collection that starts with the shard key. You can change the shard key on a collection if you desire. Any update that does not contain the shard key will be sent to all shards. The shared key must be unique

hw6.3answer

The solution to Homework 6.2 M101JS: MongoDB for Node.js Developers

Homework: Homework 6.2

Let’s suppose you have a five-member replica set and want to assure that writes are committed to the journal and are acknowledged by at least 3 nodes before you proceed forward. What would be the appropriate settings for w and j?  

Solution: w=”majority”

hw6.2answer

The solution to Homework 6.1 M101JS: MongoDB for Node.js Developers

Homework: Homework 6.1

Which of the following statements are true about replication in MongoDB? Check all that apply.

hw6.1answer

The minimum sensible number of voting nodes to a replica set is three.
MongoDB replication is synchronous.
By default, using the new MongoClient connection class, w=1 and j=1.
The oplog utilizes a capped collection.

The solution to Homework 5.4 M101JS: MongoDB for Node.js Developers

The solution to Homework 5.4 M101JS: MongoDB for Node.js Developers Homework: Homework 5.4

Removing Rural Residents

In this problem, you will calculate the number of people who live in a zip code in the US where the city starts with a digit. We will take that to mean they don’t really live in a city. Once again, you will be using the zip code collection, which you will find in the ‘handouts’ link on this page. Import it into your MongoDB using the following command from the command line:

> mongoimport -d test -c zips --drop zips.json

If you imported it correctly, you can go to the test database in the mongo shell and confirm that

> db.zips.count()

yields 29,467 documents.

The project operator can extract the first digit from any field. For example, to extract the first digit from the city field, you could write this query:

db.zips.aggregate([
    {$project: 
     {
	first_char: {$substr : ["$city",0,1]},
     }	 
   }
])

Using the aggregation framework, calculate the sum total of people who are living in a zip code where the city starts with a digit. Choose the answer below.

You will need to probably change your projection to send more info through than just that first character. Also, you will need a filtering step to get rid of all documents where the city does not start with a digital (0-9).

Note: When you mongoimport the data, you will probably see a few duplicate key errors; this is to be expected, and will not prevent the mongoimport from working. There is also an issue with some versions of MongoDB 3.0 where it claims that 0 documents were mongoimported, when in fact there were 29,467 documents imported. You can verify this for yourself by going into the shell and counting the documents in the “test.zips” collection.

Solution:

Run the following queries:

db.zips.aggregate([
    { $project: { _id: 0, city: 1, pop: 1 } },
    { $match: { city: /^\d.*/ } },
    { $group: { _id: null, pop: { $sum: "$pop" } } },
    { $sort: { city: 1} }
])
hw5.4answer2

Seeing the above result I submit the 298015 and it is correct for me:

hw5.4answer

The solution to Homework 5.3 (Hands-On) M101JS: MongoDB for Node.js Developers

The solution to Homework 5.3 (Hands-On) M101JS: MongoDB for Node.js DevelopersWho’s the easiest grader on campus?
A set of grades are loaded into the grades collection. The documents look like this:

{
	"_id" : ObjectId("50b59cd75bed76f46522c392"),
	"student_id" : 10,
	"class_id" : 5,
	"scores" : [
		{
			"type" : "exam",
			"score" : 69.17634380939022
		},
		{
			"type" : "quiz",
			"score" : 61.20182926719762
		},
		{
			"type" : "homework",
			"score" : 73.3293624199466
		},
		{
			"type" : "homework",
			"score" : 15.206314042622903
		},
		{
			"type" : "homework",
			"score" : 36.75297723087603
		},
		{
			"type" : "homework",
			"score" : 64.42913107330241
		}
	]
}

There are documents for each student (student_id) across a variety of classes (class_id). Note that not all students in the same class have the same exact number of assessments. Some students have three homework assignments, etc.

Your task is to calculate the class with the best average student performance. This involves calculating an average for each student in each class of all non-quiz assessments and then averaging those numbers to get a class average. To be clear, each student’s average includes only exams and homework grades. Don’t include their quiz scores in the calculation.

What is the class_id which has the highest average student performance?

Hint/Strategy: You need to group twice to solve this problem. You must figure out the GPA that each student has achieved in a class and then average those numbers to get a class average. After that, you just need to sort. The class with the lowest average is the class with class_id=2. Those students achieved a class average of 37.6

If you prefer, you may download the handout and perform your analysis on your machine with

 mongoimport -d test -c grades --drop grades.json

Solution:

Run the following SQL and the first id that you will get be the solution.

db.grades.aggregate([
    { $unwind: "$scores" },
    { $match: { $or: [ {"scores.type": "homework"}, {"scores.type":"exam"} ] } },
    { $group: { _id: { 'student_id': "$student_id", 'class_id': "$class_id" }, avg: { $avg: "$scores.score" } } },
    { $group: { _id: "$_id.class_id", class_avg: { $avg: "$avg" } } },
    { $sort: { 'class_avg': -1 } }
])

Following is the result that I got:

hw5.3answer2

So the answer is:

hw5.3answer

The solution to Homework 5.2 (Hands-On) M101JS: MongoDB for Node.js Developers

The solution to Homework 5.2 (Hands-On) M101JS: MongoDB for Node.js Developers

Crunching the Zipcode dataset
Please calculate the average population of cities in California (abbreviation CA) and New York (NY) (taken together) with populations over 25,000. For this problem, assume that a city name that appears in more than one state represents two separate cities. Please round the answer to a whole number.
Hint: The answer for CT and NJ (using this data set) is 38177.

Please note:

  • Different states might have the same city name.
  • A city might have multiple zip codes.

For purposes of keeping the Hands-On shell quick, we have used a subset of the data you previously used in zips.json, not the full set. This is why there are only 200 documents (and 200 zip codes), and all of them are in New York, Connecticut, New Jersey, and California.

If you prefer, you may download the handout and perform your analysis on your machine with

> mongoimport -d test -c zips --drop small_zips.json

Solution:

Run the following query and find the answer:

db.zips.aggregate([
    { $match: {$or: [ {state: "CA"}, {state: "NY"} ] } },
    { $group: { _id: { city: "$city" }, pop: { $sum: "$pop" } } },
    { $match: { "pop": { $gt: 25000 } } },
    { $group: { _id: null, avg_pop_of_city: { $avg: "$pop" } } }
])

I got the following answer and submit and it is correct for me. let me know if you got any other results and an easy way to proceed through it.

hw5.2answer

The solution to Homework 5.1 (Hands-On) M101JS: MongoDB for Node.js Developers

Homework: Homework 5.1 (Hands-On)

Finding the most frequent author of comments on your blog

In this assignment, you will use the aggregation framework to find the most frequent author of comments on your blog. We will be using the same basic dataset as last week, with posts and comments shortened considerably, and with many fewer documents in the collection in order to streamline the operations of the Hands-On web shell.

Use the aggregation framework in the web shell to calculate the author with the greatest number of comments.

Just to clarify, the data set for this week is not available for download.

To help you verify your work before submitting, the author with the fewest comments is Cody Strouth and he commented 68 times.

Once you’ve found the correct answer with your query, please choose your answer below for the most prolific comment author.

Note: this data set is relatively large. Due to some quirks of the shell, the entire result set gets pulled into the browser on find(), so if you want to see the document schema, we recommend either using db.posts.findOne(), db.posts.find().limit(1), or that you plan on waiting for a bit after you hit enter. We also recommend that the last phase of your aggregation pipeline is {$limit: 1} (or some single-digit number)

Solution:

Run the following query and you will get the answers.

db.posts.aggregate([
    { $unwind: "$comments" },
    { $group: { _id: "$comments.author", count: { $sum: 1 } } },
    { $sort: { count: -1 } },
    { $limit: 1 }
])
hw5.1answer

The solution to HOMEWORK 4.3 M101JS: MongoDB for Node.js Developers

HOMEWORK: HOMEWORK 4.3 Solution to HOMEWORK 4.3 M101JS: MongoDB for Node.js Developers

NOTE there is a bug (TOOLS-939) affecting some versions of mongoimport and mongorestore that causes mongoimport -d blog -c posts < posts.json to fail. As a workaround, you can usemongoimport -d blog -c posts < posts.json --batchSize 1.If you have any difficulty using MongoProc, here are 2 video lectures showing how to set it up.

You will need to download one of the following versions of the MongoProc client, which will verify on your local machine (port 8082) that the signup and login pages of the blog work properly.

Download MongoProc

Making the Blog fast to get started, please download hw4-3.zip from the Download Handout link and unpack the file to your computer. This assignment requires Mongo 2.2 or above.

In this homework assignment, you will be adding some indexes to the post-collection to make the blog fast.

We have provided the full code for the blog application and you don’t need to make any changes, or even run the blog. But you can, for fun.

We are also providing a patriotic (if you are an American) data set for the blog. There are 1000 entries with lots of comments and tags. You must load this dataset to complete the problem.

# from the mongo shell
use blog
db.posts.drop()
# from the a mac or PC terminal window
mongoimport -d blog -c posts < posts.json

The blog has been enhanced so that it can also display the top 10 most recent posts by tag. There are hyperlinks from the post tags to the page that displays the 10 most recent blog entries for that tag. (run the blog and it will be obvious)

Your assignment is to make the following blog pages fast:

  • The blog home page
  • The page that displays blog posts by tag (http://localhost:8082/tag/whatever)
  • The page that displays a blog entry by permalink (http://localhost:8082/post/permalink)

By fast, we mean that indexes should be in place to satisfy these queries such that we only need to scan the number of documents we are going to return.

To figure out what queries you need to optimize, you can read the code in posts.js and see what queries it is doing to return the data needed for the relevant pages. Isolate those queries and use explain to explore.

Once you have added the indexes to make those pages fast, just validate your work with MongoProc. Correct You have used 1 of 3 submissions.


Solutions:
Run the following queries one by one and submit it.

db.posts.ensureIndex({ date : -1})
db.posts.find({date : -1}).limit(10).explain()
db.posts.ensureIndex({ tags : 1, date : -1})
db.posts.find({tags : 'vietnam'}).sort({ date : -1}).explain()
db.posts.ensureIndex({ permalink : 1})
db.posts.find({permalink: "xnafqemtzzcyyzjdegkj"}).explain()
hw4.3answer

The solution to HOMEWORK 4.1 M101JS: MongoDB for Node.js Developers

The solution to HOMEWORK 4.1 M101JS: MongoDB FOR NODE.JS DEVELOPERS

Suppose you have a collection with the following indexes:

> db.products.getIndexes()
[
	{
		"v" : 1,
		"key" : {
			"_id" : 1
		},
		"ns" : "store.products",
		"name" : "_id_"
	},
	{
		"v" : 1,
		"key" : {
			"sku" : 1
		},
                "unique" : true,
		"ns" : "store.products",
		"name" : "sku_1"
	},
	{
		"v" : 1,
		"key" : {
			"price" : -1
		},
		"ns" : "store.products",
		"name" : "price_-1"
	},
	{
		"v" : 1,
		"key" : {
			"description" : 1
		},
		"ns" : "store.products",
		"name" : "description_1"
	},
	{
		"v" : 1,
		"key" : {
			"category" : 1,
			"brand" : 1
		},
		"ns" : "store.products",
		"name" : "category_1_brand_1"
	},
	{
		"v" : 1,
		"key" : {
			"reviews.author" : 1
		},
		"ns" : "store.products",
		"name" : "reviews.author_1"
	}

Which of the following queries can utilize at least one index to find all matching documents or to sort? Check all that apply.

hw4.1answer

Solutions:
db.products.find( { ‘brand’ : “GE” } ).sort( { price : 1 } )
db.products.find( { $and : [ { price : { $gt : 30 } },{ price : { $lt : 50 } } ] } ).sort( { brand : 1 } )  

I have submitted those two answers and found it to be correct.
Let me know if you have any problems.

Final: Question 6 Solution M101JS: MongoDB for Node.js Developers

Final: Question 6

Suppose you have a collection of students of the following form:

{
	"_id" : ObjectId("50c598f582094fb5f92efb96"),
	"first_name" : "John",
	"last_name" : "Doe",
	"date_of_admission" : ISODate("2010-02-21T05:00:00Z"),
	"residence_hall" : "Fairweather",
	"has_car" : true,
	"student_id" : "2348023902",
	"current_classes" : [
		"His343",
		"Math234",
		"Phy123",
		"Art232"
	]
}

Now suppose that basic inserts into the collection, which only include the last name, first name, and student_id, are too slow (we can’t do enough of them per second from our program). What could potentially improve the speed of inserts? Check all that apply.

Solution: I selected as in the image, please let me know if there are any correct answer.

final_exam_question_6

Final: Question 4 Solution M101JS: MongoDB for Node.js Developers

Final: Question 4

Enhancing the Blog to support viewers liking certain comments
In this problem, you will be enhancing the blog project to support users liking certain comments and the like counts showing up in the permalink page. Start by downloading Final4.zip and posts.json from the Download Handout link and loading up the blog dataset posts.json. The user interface has already been implemented for you. It’s not fancy. The /post URL shows the like counts next to each comment and displays a Like button that you can click on. Like button POSTS to the /like URL on the blog, makes the necessary changes to the database state (you are implementing this), and then redirect the browser back to the permalink page.

This full round trip and redisplay of the entire web page is not how you would implement liking in a modern web app, but it makes it easier for us to reason about, so we will go with it.

Your job is to search the code for the string “TODO: Final exam question – Increment the number of likes” and make any necessary changes. You can choose whatever schema you want, but you should note that the entry_template makes some assumptions about how the like value will be encoded and if you go with a different convention than it assumes, you will need to make some adjustments.

The validation script does not look at the database. It looks at the blog.

The validation script, final4-validate.js, will fetch your blog, go to the first post’s permalink page and attempt to increment the vote count. You run it as follows:

node final4-validate.js

Remember that the blog needs to be running as well as Mongo. The validation script takes some options if you want to run outside of the localhost.
After you have gotten it working, enter the validation string below.Solution: VQ3jedFjG5VmElLTYKqS   How do I achieve the result?

  • Open blog.js file and find following lines: // TODO: Final exam question – Increment the number of likes callback(Error(“incrementLikes NYI”), null);
  • Replace above lines with following lines var selector = {}; selector[‘comments.’ + comment_ordinal + ‘.num_likes’] = 1; posts.update({‘permalink’: permalink}, {‘$inc’: selector}, function(err, numModified) { if (err) return callback(err, null); callback(err, numModified); }); //callback(Error(“incrementLikes NYI”), null);
  • Now check it by running blog with “node app.js” and go to blog’s detail and click like button
  • Run validation by command line “node final4-validate.js” Trying to fetch blog homepage for url http://localhost:3000/ Trying to grab the number of likes for url http://localhost:3000/post/mxwnnnqaflufnqwlekfd Trying to increment the number of likes for post: /post/mxwnnnqaflufnqwlekfd Trying to grab the number of likes for url http://localhost:3000/post/mxwnnnqaflufnqwlekfd Successfully clicked like Blog validated successfully! Your validation code is: VQ3jedFjG5VmElLTYKqS
final_exam_question_4
  • Validation code that I found is VQ3jedFjG5VmElLTYKqS

Answer may vary for you so please verify for yourself.

Thanks

Final: Question 3 Solution M101JS: MongoDB for Node.js Developers

Final: Question 3

In this problem, you will update a document in the Enron dataset to illustrate your mastery of updating documents from the shell.

Please add the email address “mrpotatohead@mongodb.com” to the list of addresses in the “headers. To” array for the document with “headers.Message-ID” of “<8147308.1075851042335.JavaMail.evans@thyme>”

After you have completed that task, please download final3.zip from the Download Handout link and run final3-validate.js to get the validation code and put it in the box below without any extra spaces. The validation script assumes that it is connecting to a simple mongo instance on the standard port on the localhost.

Solution: vOnRg05kwcqyEFSve96R  

How to achieve it?

  • Extract the downloaded handout
  • In the first command prompt, go to the Final3 folder where you will see final3-validate.js and package.json files.
  • Now run “npm install”
  • Now open your mongo shell with mongod in second command prompt and again in third command prompt mongo
  • Run command “show databases” and see whether Enron is there or not if install from final question 2 nor run query “use enron”
  • Run the following query:
    db.messages.update({“headers.Message-ID”:”<8147308.1075851042335.JavaMail.evans@thyme>”}, {$addToSet: {“headers.To”: “mrpotatohead@mongodb.com”}}, {multi: 1})
  • Now run the command “node final3-validate.js”  in first command prompt
  • You will see something like below:

    Welcome to the Final Exam Q3 Checker. My job is to make sure you correctly updated the document
    Final Exam Q3 Validated successfully!
    Your validation code is: vOnRg05kwcqyEFSve96R
  • Thus my answer is “vOnRg05kwcqyEFSve96R”
  • It may differ to yours so please confirm before submitting.

Thanks

Final: Question 2 Solution M101JS: MongoDB for Node.js Developers

Final: Question 2

Please use the Enron dataset you imported for the previous problem. For this question, you will use the aggregation framework to figure out pairs of people that tend to communicate a lot. To do this, you will need to unwind the To list for each message.

This problem is a little tricky because a recipient may appear more than once in the To list for a message. You will need to fix that in a stage of the aggregation before doing your grouping and counting of (sender, recipient) pairs.

Which pair of people have the greatest number of messages in the dataset?

Solution:   Query to run as per the question is below as per my understanding:

db.messages.aggregate([
	{$project: {
		from: "$headers.From",
		to: "$headers.To"
	}},
	{$unwind: "$to"},
	{$project: {
		pair: {
			from: "$from",
			to: "$to"
		},
		count: {$add: [1]}
	}},
	{$group: {
		_id: "$pair",
		count: {$sum: 1}
	}},
	{$sort: {
		count: -1
	}},
	{$limit: 2},
	{$skip: 1}
])

Results that I found is below:

{ "_id" : { "from" : "susan.mara@enron.com", "to" : "richard.shapiro@enron.com" }, "count" : 974 }
final_exam_question_and-answer-2

Let me know if you guys find extra so that we can discuss it.

Thanks