Text Analysis – Facebook Post Comments

Text Analysis – Facebook Post Comments

Folks,

In this blog we will learn how to analyze the comments of Public Facebook post using Facebook Graph API Explorer & R!

Facebook developer account required to get started with this Facebook Graph API .

If you don’t have Facebook developer account, you can upgrade your personal Facebook account to a Facebook Developer account from here this link.

After registering as Facebook Developer, go to “Tools & Support”->”Graph API Explorer”

To explore Graph API – Token & Permissions are required, so just click on the “Get Token”.

1.png

~ Courtesy Facebook Developer

As public profiles included by default in permissions, just click on “Get Access Token”.

4.png

Below is my access token, which will expire after some time. As shown in token info.

5.png
Graph Explorer

Now we have token,  let’s explore now.


Extracting Comments from the Public Facebook Post.

First thing you required is the Post Id of the post. See below steps to get the post Id.

Suppose below is the post, we want to analyze. Click on the Post Date Time. See below highlighted box. ~ Post Courtesy Facebook

3.png

Copy below Id. This is the post Id.

2.png

Go to the Graph Explorer.

Type “Post_id/comments” in below box & click on Submit.

6.pngYou can also give limit for number of comments to return, like this

Post_id/comments?limit=”.

My Input:

7.png

Output: Below are post comments in the JSON format.

If you want more comments click on “next” for next page of comments.

7.png

Click on “Get Code” to get the cURL code. Copy this URL, we will use this URL in R.

8.pngText Analysis in R 

R Packages required: 

install.packages(“RCurl “): It allows us to compose general HTTP requests and provides convenient functions to fetch data.

install.packages(“rjson”): It allows us to converts JSON object into R objects and vice-versa.

install.packages(“tm”):  A Mining Package for text mining applications within R. It offers a number of transformations that ease the tedium of cleaning data.

R Commands:

111

url used in above image is copied from cURL code from Graph Explorer.

Output: So the first page give me 25 comments only. We will analyze here only 1st page i.e. 25 comments only.

output

Cleaning & Analyzing Data:

Creating corpus & removing extra spaces, special characters & other unwanted things.

cleaning.png

Creating Term Document Matrix:

1111.png

Here is the 760 extracted words with frequency.

1112.png

Creating Wordcloud: 

install.packages(“wordcloud”): For plotting a word cloud

lastt.png

In this Word Cloud we are taking only 100 words with minimum frequency of 2.

Output:

lasttttttt.png

Graph API Reference ~Facebook Developer  For more details, please read this Link.

Feedback and suggestions are most welcome. If you have any feedback, suggestions or questions please comment.


Thanks!

Happy Learning!

Geocoding in R!

Folks,

In this blog we will explore the basics of Geocoding using R,  using Google Maps Geocoding API & R!

Geocoding: Process of transforming a description of a location (address, name of a place, or coordinates) to a location on the earth’s surface.

Forward Geocoding: Converting address/name of a place into geographic coordinates.

Reverse Geocoding: Converting geographic coordinates into readable address.

R Package required: 

install.packages(“RCurl ”): It allows us to compose general HTTP requests and provides convenient functions to fetch data.

install.packages(“rjson”): It allows us to converts JSON object into R objects and vice-versa.


Forward Geocoding using R

Google Maps Geocoding API:

11

Where output may be either of JSON/XML & parameters = address or components.

Example for JSON:

https://maps.googleapis.com/maps/api/geocode/json?address=India+Gate,+New+Delhi,+India&components=country:IN

Where address = “India Gate, New Delhi, India”, components = Country “IN” for India.

Component Country: country name or a two letter country code. For more components like locality & postal code etc. details read this link.

R Commands:

url<- “”

jdata<- getURL(url) # Fetching JSON data.

rdata<- fromJSON(jdata) # Convert a JSON object into an R object.

Output:

1

Geographic coordinates present in  rdata results.

2

Extracting geographic coordinates from the rdata.

3.png

PS: These are approx geographic coordinates, as it depends on the address provided in URL.


Reverse Geocoding using R

Google Maps Reverse Geocoding API:

12

Where output may be either of JSON/XML. Parameters:  ,

R Commands:

url<- “”

jdata<- getURL(url) # Fetching JSON data.

rdata<- fromJSON(jdata) # Convert a JSON object into an R object.

For Example taking latitude & longitude 28.632455, 77.217218 – Courtesy: Google Maps

4.png

Output:

7.png

5

For more details please read this google link.


Thanks!

Happy Learning!

Accessing Database (MS SQL Server/Oracle/MySQL) from R!

Folks,

In this blog we will learn how to access Oracle, MySQL & SQL Server databases from R!


Accessing Oracle Database using RJDBC Package

Below is my database hosted on AWS Cloud. See here how to setup Oracle Db on cloud (free tier).

11

R Packages required:-  install.packages(“RJDBC”)

Also download Oracle JDBC Drivers from here Link. After downloading, copy file location.

location.png

Commands:


driver <- JDBC("oracle.jdbc.driver.OracleDriver"," <paste JDBC Driver file location>")

dbConnect(driver , "jdbc:oracle:thin:@Host:Port:ServiceName or SID", "<username>", "<password>")

code1

R users have few more choices to access their Oracle Database: Using RODBC and ROracle packages.


MySQL Database

Here is my database hosted on AWS Cloud. See here how to setup MySQL Db on cloud (free tier).

1

a) Accessing MySQl Database using ODBC:

R Packages required:-  install.packages(“RODBC”)

For Windows: Go to start, search ODBC. Follow below steps of setup.

1

Click on “Add”.

2

Select MySQL ODBC Driver. If not present, then download using this link.

3

Provide your server details, username & password. Test your connection. Note down Data Source name.

4

Commands:


my_conn <- odbcConnect("<ODBC Data Source Name >")

sqlQuery(my_conn, paste("<Query>"))

code

b) Accessing MySQL DB using RMySQL Package:

R Packages required:-  install.packages(“RMySQL”)

Commands:


my_conn <- dbConnect(MySQL(), user="<username>", password="<password>", database= "<Database name>", host="<your host>")

result<- dbSendQuery(my_conn, "<Query>")

dbFetch(result)

code 2.png


Accessing SQL Server Database using ODBC

Here is my SQL Server database hosted locally.

server.png

a) Accessing SQL Server using ODBC:

For Windows: Go to start, search ODBC.

1

Follow below steps to setup ODBC datasource.

output_KUL4JC.gif

Commands:


my_conn <- odbcConnect("<Data Source Name>", uid="<username>", pwd="<password>")

sqlQuery(my_conn, "<Query>")

code2.png

b) Accessing SQL Server DB using RJDBC Package:

Suppose SQL Server database hosted on cloud. We can use below commands to connect.

R Packages required:-  install.packages(“RJDBC”) & Download SQL JDBC Drivers file from internet.


driver <- JDBC("com.microsoft.sqlserver.jdbc.SQLServerDriver",
"<location of SQL JDBC Drivers file on computer>")

conn <- dbConnect(driver , "jdbc:sqlserver://<serverName>", "<userID>", "<password>")

result <- dbGetQuery(conn, "<Query>")


Thanks!

Happy Learning!

Exploring Instagram API using R!

Folks,

In this blog we will explore the basics of Instagram API using R.

“instaR” Package in R: Provides an interface to the Instagram API , which allows R users to access public users’ profile data.

Install “instaR” package from CRAN : install.packages(“instaR”)

Install “RCurl” package from CRAN : install.packages(“RCurl”)


Step 1: Registering an Application with Instagram.

If you already have an account with Instagram, go to Instagram Developer and register.

1.png

Click “Register your application” button. After you register as a Instagram developer, you can go to  a Manage clients & register a new client.

Note down your App Name, Client ID & Client Secret

2


Step 2: Create OAuth token to Instagram R session.

instaOAuth creates an OAuth access token that enables R to make authenticated calls to the Instagram API.

instaOAuth( client_id, client_secret, scope = “basic”)

Scope is related to the access permissions. Read Login Permissions (Scopes) Link

my_app_client_id  <- “”
my_app_client_secret <- “”

code 1

Now run the  instaOAuth command in R. See below format.

code 3

For setting the ‘redirect_uri’. Go to Manage Clients.

Copy and paste http://localhost:1410/ into ‘redirect_uri’ on Instagram App Settings.

2.png

After setting the redirect uri press enter on R command window. Automatic browser will open below page & ask for the Authorization.

r 1

After Authorizing the app. Below message will come.

r 2

See below message “Authentication Complete”. Now the connection is done with the Instagram API & R.

comp.PNG

Extracting the Token from OAuth. Just type below command.

code 5

The token can be saved as a file in disk to be re-used in future sessions.

saving auth

Now we have token = my_access_token. So here we goes with some of the basic end points of Instagram API.

Instagram End Points Links: Here


1) Getting the information from Instagram for token owner.

End Points: https://api.instagram.com/v1/users/self/?access_token=ACCESS-TOKEN

Using R

json1.PNG

Output:

11111111111.png

Using Postman API Client

End Points: https://api.instagram.com/v1/users/self/?access_token=ACCESS-TOKEN

Output: JSON/XML

pstman.png

2) Get the most recent media published on Instagram by token owner.

https://api.instagram.com/v1/users/self/media/recent/?access_token=ACCESS-TOKEN

Using Postman API Client

Output:

json3.png

Actual Image:

sunburn.png


Checking the details of any public image.

Hit this endpoint: https://api.instagram.com/oembed/?url=YOUR URL

See below gif for explanation. I’m using postman client for API testing.

output_wlPtEf.gif

You can get/post API request in your own way! Explore more End Points here: Link

PS: In Sandbox mode we cannot use extended permissions. Read Login Permissions (Scopes) here Link


Thanks!

Happy Learning!

Mining Facebook Data Using R & Facebook API!

Mining Facebook Data Using R & Facebook API!

Folks,

In this blog we will learn the basics of extracting Facebook data using R & Facebook API.

Rfacebook Package: Provides an interface to the Facebook API.

Rfacebook package in R provides functions that allow R to access Facebook’s API to get information about posts, comments, likes, group that mention specific keywords & much more.

Install “Rfacebook” package from CRAN : install.packages(“Rfacebook”) 


Step 1: Registering an Application with Facebook.

If you already have an account with Facebook, go to FacebookDeveloper and register.

Click “Register Now” button. After you register as a Facebook developer, you can register a new application.

Register a new application

From  FacebookDeveloper click on Apps at the top of the page to go to the application dashboard.

Click the fb-create-new-app-button button near the top. Once you are done with the verification process, your application is created. Note down the App Id & App Secret.

1
Application Dashboard

Site URL on Facebook App Settings: http://localhost:1410 

6.PNG
App Settings

Step 2: Create OAuth token to Facebook R session.

fbOAuth creates a long-lived OAuth access token that enables R to make authenticated calls to the Facebook API.

fbOAuth(app_id=””,app_secret=”YourAppSecret”)

4

5

Saving “my_oauth”as a file to be re-used in future sessions, which can be used as token in functions.

7.PNG

Now the connection is done with the Facebook API & R. So here we goes with some of the basic functions.

If you are getting below error while calling function, please go the bottom of this blog post for fix.

Error in callAPI(query, token) :
An active access token must be used to query information about the current user.

function getLikes

getLikes(user, n = n , token): Extract list of liked pages of a Facebook user with page id.

Arguments: user: user name/ID , n: Number of liked pages to return for user.

Use below command in R to get likes.

8.PNG

Here is the my result data set “my_likes”. With three variables: Id, names & website of pages.

view.PNG

9.PNG
Pages Liked

function getPage 

getPage(page , token, n = n): Extract list of posts from a public Facebook page.

Arguments: page: Page ID or page name, n : Number of posts to return for page.

Example: Extracting 10 posts of Facebook Page “Narendra Modi”. 

Facebook Page Id of this page you can get from this link: http://findmyfbid.com/ . See below the gif.

output_LJA4Du.gif

After getting the page id use below command in R to get posts.

getpage.PNG

Here is my result data set “getpagedata”. With 10 Observation & 10 Variables. Variables like Post with likes_count, share_count & comments_count etc.

padess.PNG

1a

1b


function search_groups

searchGroup(“text”,token, n = n): Find any group with its privacy status & Facebook ID.

Arguments: text: text string, n : Number of groups to return.

Use below command in R to search groups.

groups.PNG

Here is my result data set “search_groups”. With many observation & 3 Variables.

11.png
Groups


function getGroup

getGroup(ID, token, n = n): Extract list of posts from a public Facebook page. Whose privacy is open.

Arguments: ID: Group ID , n: Number of posts to return for group.

Example: We have to extract 10 posts from 7th group “Web Scraping and Data mining” present in above image groups.

Use below command in R to get groups.

12

Here is my result data set. With 10 Observation & 3 Variables.

postttttttttt

12a


function searchPages

searchPages(, token, n = n): It Search pages that having a string/keyword.

Arguments: string: any string , n: Number of pages to return

Example: We have to search 10 pages that mention a string “Sports”.

Use below command in R to search pages.

sports.PNG

Here is my result data set with 10 Observation & 16 Variables.

s1

s2


function updateStatus

updateStatus(“text”, token)

Arguments: text: any string , token

Use below command in R to update Facebook status.

status1.PNG

Result:

status2


 

Fix for Error in callAPI:

If you are getting below error, please follow below steps to fix this issue.

Error in callAPI(query, token) :
An active access token must be used to query information about the current user.

Run Below Function First :-

fbOAuth <- function(app_id, app_secret, extended_permissions=FALSE, legacy_permissions=FALSE, scope=NULL)
{
  ## getting callback URL
  full_url <- oauth_callback()
  full_url <- gsub("(.*localhost:[0-9]{1,5}/).*", x=full_url, replacement="\\1")
  message <- paste("Copy and paste into Site URL on Facebook App Settings:",
                   full_url, "\nWhen done, press any key to continue...")
  ## prompting user to introduce callback URL in app page
  invisible(readline(message))
  ## a simplified version of the example in httr package
  facebook <- oauth_endpoint(
    authorize = "https://www.facebook.com/dialog/oauth",
    access = "https://graph.facebook.com/oauth/access_token") 
  myapp <- oauth_app("facebook", app_id, app_secret)
  if (is.null(scope)) {
    if (extended_permissions==TRUE){
      scope <- c("user_birthday", "user_hometown", "user_location", "user_relationships",
                 "publish_actions","user_status","user_likes")
    }
    else { scope <- c("public_profile", "user_friends")}
  
    if (legacy_permissions==TRUE) {
      scope <- c(scope, "read_stream")
    }
  }

  if (packageVersion('httr') < "1.2"){
    stop("Rfacebook requires httr version 1.2.0 or greater")
  }

  ## with early httr versions
  if (packageVersion('httr') <= "0.2"){
    facebook_token <- oauth2.0_token(facebook, myapp,
                                     scope=scope)
    fb_oauth <- sign_oauth2.0(facebook_token$access_token)
    if (GET("https://graph.facebook.com/me", config=fb_oauth)$status==200){
      message("Authentication successful.")
    }
  }

  ## less early httr versions
  if (packageVersion('httr') > "0.2" & packageVersion('httr') <= "0.6.1"){
    fb_oauth <- oauth2.0_token(facebook, myapp,
                               scope=scope, cache=FALSE) 
    if (GET("https://graph.facebook.com/me", config(token=fb_oauth))$status==200){
      message("Authentication successful.")
    } 
  }

  ## httr version from 0.6 to 1.1
  if (packageVersion('httr') > "0.6.1" & packageVersion('httr') < "1.2"){
    Sys.setenv("HTTR_SERVER_PORT" = "1410/")
    fb_oauth <- oauth2.0_token(facebook, myapp,
                               scope=scope, cache=FALSE) 
    if (GET("https://graph.facebook.com/me", config(token=fb_oauth))$status==200){
      message("Authentication successful.")
    } 
  }

  ## httr version after 1.2
  if (packageVersion('httr') >= "1.2"){
    fb_oauth <- oauth2.0_token(facebook, myapp,
                               scope=scope, cache=FALSE) 
    if (GET("https://graph.facebook.com/me", config(token=fb_oauth))$status==200){
      message("Authentication successful.")
    } 
  }

  ## identifying API version of token
  error <- tryCatch(callAPI('https://graph.facebook.com/pablobarbera', fb_oauth),
                    error = function(e) e)
  if (inherits(error, 'error')){
    class(fb_oauth)[4] <- 'v2'
  }
  if (!inherits(error, 'error')){
    class(fb_oauth)[4] <- 'v1'
  }

  return(fb_oauth)
}

111

script.png

Fix Credit goes to (www.listendata.com/) Visit this link for more information.


Thanks!

Happy Learning!

Connecting MySQL DB Instance present on AWS Cloud

Folks,

For creation of any Amazon RDS DB Instance (Free Tier), kindly visit this blog: Link

In this blog we will learn how to connect MySQL Database present on AWS Cloud.

Here is my MySQL DB Instance in “available” Status. Please note the End Point, which is basically the HOST.

1
RDS Dashboard

Step for adding new Inbound Rule for allowing connection on port 3306.

Select MySQL Instance & Click on “Instance Actions“->”See Details“.

2

After that click on “Security Groups“.

3

Now click on the “Inbound Rules” in Security Groups page.

4
Security Groups Page

Create a Inbound Rule for Type: MYSQL, Protocol: TCP , Port: 3306 Source: Anywhere. So that you can connect this instance from any machine. After creation of rule click on Save.

Security Group is basically allowing the connections on the port.

5

After Saving Inbound Rule. Go to RDS Dashboard.

1
RDS Dashboard

Now MySQL Instance is ready to connect & use it.


 

Steps for connecting MySQL Database: Using CMD

Install MySQL Server on your machine. Copy the location on bin folder in program files.

Also check your firewall settings for port 3306.

7

Also check the MySQL service is running properly or not.

6

Open Command Prompt: Use this command: cd <bin folder location> then Press Enter.

Example:  cd C:\Program Files\MySQL\MySQL Server 5.1\bin

Now you are in bin folder.

8

Use below command in cmd & it will ask for password.

mysql – h <host> -P <port> – u <username> – p

Where <host>: Your End Point, <post>:3306, <username>: Master username

9

Now we are connected to the MySQL DB Instance.

10


Steps for connecting MySQL Database: Using MySQL GUI

I’m using SQLyog GUI. Please provide below details.

01

Provide your password & test the connection.

02

Click on connect to connect the database.

03

Now we are connected to the MySQL DB Instance.


Thanks!

Happy Learning!

AWS RDS Instance Setup: Oracle DB on Cloud (Free Tier)

Folks,

AWS Free Tier (Link), the Amazon RDS Free Tier helps us to get started with a database instance in the cloud. Free for 12 months starting with the date on which we create our AWS account.

We can use this to develop new applications or simply gain hands-on experience with Cloud Computing. It is easy to set up, operate, and scale DB.

Database Available:- MySQL, MariaDB, PostgreSQL, Oracle & SQL Server.

Steps for creating your own: Oracle DB Instance on Cloud (Free Tier)

Login AWS with your amazon account. Link

0
Amazon Web Services Login Page

After login AWS redirect to this below Home Page.

Select “RDS” in the Database Section. RDS –  Relational Database Service.

1
AWS Home Page

After selecting “RDS” in the Database Section. Below RDS Dashboard comes up.

Click on “Launch a DB Instance” in this screen.

2
AWS RDS Dashboard

Now from here the main setup starts…


 Step 1: Select Engine: Select the DB Engine:

I’m selecting the Oracle EE Database Engine.

3

Step 2: Production Or Free Tier

For Free Tier: Select Option Second: “Dev/Test”

4

After that click on Next Step.

Step 3: Specify DB Details:

Select check box for options available for free tier RDS.

5

Provide DB details here like master username & password. Please note down these details for future reference.

6

After that click on Next Step.

Step 4:  Configure Advanced Settings:

Use default setting here for free tier service.

7

Now we can launch the Instance.

8

After launching: Db instance is being created.

9

Go to the RDS Dashboard: Instance Tab:  You can check your instance here.

10
DB Instance Creation is in process

 

After some time…

Oracle DB Instance is ready. Note down your Endpoint, which is basically the HOST.

11
Db Instance is available now

My End Point: shobhitdbdemo.cbxoxihdzrvi.us-west-2.rds.amazonaws.com:1521


 

Steps for Connecting Oracle Database: Windows

Install Oracle Client on machine. Also check your firewall settings for port 1521. Open Command Prompt & use below command to connect DB.

sqlplus username/password@host:port/service 

Where username: Your Database Master Username in Step 3

Example:

12

12

Now you can create your own database & other database objects.


 

Thanks!

Happy Learning!