stevenmortimer.com

salesforcer 0.2.2 - Relationship Queries and the Reports API

Steven M. Mortimer — Sat, 12 Sep 2020 15:35:33 +0000

The latest version of the {salesforcer} package (v0.2.2) is now available on CRAN and is ready to help you have better access to data in your Salesforce Org. Along with a host of bug fixes this release has three big features:

Experimental Functions for the Reports and Dashboards REST API (jump to section) - Now you have programmatic access to executing and managing reports, dashboards, and analytics notifications in your Org. You can familiarize yourself with the Salesforce documentation HERE. Not all functions have been implemented yet, but your questions, comments, and feedback are welcome!
Support for Bulk 2.0 API Queries (jump to section) - In Salesforce version 47.0 (Winter ’20) query jobs in the Bulk API 2.0 were added. Now you can leverage this resource from the {salesforcer} package in addition to the queries via the REST, SOAP, and Bulk 1.0 APIs.
Support for Relationship Queries (jump to section) - In previous versions of the package the child-to-parent and nested parent-to-child queries did not work or returned a jumbled mess of results that were not parsed correctly. This version fixes those issues in both the REST and SOAP APIs with better test coverage on a variety of query types.

For a complete list of updates, please review the release notes from v0.2.0 onwards listed on the {salesforcer} pkgdown site here: https://stevenmmortimer.github.io/salesforcer/news/index.html.

Experimental Functions for Reports and Dashboards REST API

Salesforce has rich support for Reports in your Salesforce Org. Sometimes reports are a better way to collaborate with other users because they can create reports in the GUI or you can create one for them so they always have access to the most current recordset meeting your report criteria. The challenge comes when trying to access this data programmatically. Fortunately, Salesforce provides the Reports and Dashboards REST API as a means to not only execute Reports, but to also manage them in your Org.

In Salesforce there is a dedicated page to displaying the list of reports in your Org. It typically follows the pattern: https://na1.salesforce.com/00O/o (replace na1 with your server instance). When you click on a report in the GUI you should see the report’s results. Below is a screenshot of how a report may look in your Org (note the report Id in the URL bar):

The report Id above ("00O3s000006tE7zEAE") is the only information needed to pull those same results from an R session, like so:

my_report_id <- "00O3s000006tE7zEAE"
results <- sf_run_report(my_report_id)
results
#> # A tibble: 14 x 8
#>   `Contact ID` `First Name` `test number` `Contact Owner` `Account ID`
#>   <chr>        <chr>                <dbl> <chr>           <chr>       
#> 1 0036A000002… Edna                    NA Steven Mortimer 0016A000003…
#> 2 0036A000002… Liz                     NA Steven Mortimer 0016A000003…
#> 3 0036A000002… Tom                     NA Steven Mortimer 0016A000003…
#> 4 0036A000002… Sean                    NA Steven Mortimer 0016A000003…
#> 5 0036A000002… Rose                    NA Steven Mortimer 0016A000003…
#> # … with 9 more rows, and 3 more variables: `Account Name` <chr>, `Billing
#> #   City` <chr>, `Account Owner` <chr>

Currently, all of the report related functionality in the Reports and Dashboards REST API has been ported into the {salesforcer} package and you can do some pretty neat stuff like on-the-fly filtering and sorting:

# filter records that was created before this month
filter1 <- list(column = "CREATED_DATE",
                        operator = "lessThan", 
                        value = "THIS_MONTH")

# filter records where the account billing address city is not empty
filter2 <-  list(column = "ACCOUNT.ADDRESS1_CITY",
                        operator = "notEqual", 
                        value = "")

# combine filter1 and filter2 using 'AND' which means that records must meet both filters
results_using_AND <- sf_run_report(my_report_id, 
                                   report_boolean_logic = "1 AND 2",
                                   report_filters = list(filter1, filter2))
results_using_AND
#> # A tibble: 14 x 8
#>   `Contact ID` `First Name` `test number` `Contact Owner` `Account ID`
#>   <chr>        <chr>                <dbl> <chr>           <chr>       
#> 1 0036A000002… Rose                    NA Steven Mortimer 0016A000003…
#> 2 0036A000002… Sean                    NA Steven Mortimer 0016A000003…
#> 3 0036A000002… Jack                    99 Steven Mortimer 0016A000003…
#> 4 0036A000002… Tim                     NA Steven Mortimer 0016A000003…
#> 5 0036A000002… John                    23 Steven Mortimer 0016A000003…
#> # … with 9 more rows, and 3 more variables: `Account Name` <chr>, `Billing
#> #   City` <chr>, `Account Owner` <chr>

# combine filter1 and filter2 using 'OR' which means that records must meet one 
# of the filters but also throw in a row limit based on a specific sort order
results_using_OR <- sf_run_report(my_report_id, 
                                  report_boolean_logic = "1 OR 2",
                                  report_filters = list(filter1, filter2), 
                                  sort_by = "Contact.test_number__c", 
                                  decreasing = TRUE, 
                                  top_n = 3)
results_using_OR
#> # A tibble: 3 x 8
#>   `Contact ID` `First Name` `test number` `Contact Owner` `Account ID`
#>   <chr>        <chr>                <dbl> <chr>           <chr>       
#> 1 0033s000012… KEEP                  1000 Steven Mortimer 0013s00000z…
#> 2 0033s000012… KEEP                  1000 Steven Mortimer 0013s00000z…
#> 3 0033s000012… KEEP                  1000 Steven Mortimer 0013s00000z…
#> # … with 3 more variables: `Account Name` <chr>, `Billing City` <chr>, `Account
#> #   Owner` <chr>

For more detail on how to take advantage of this new functionality please see the pkgdown website https://stevenmmortimer.github.io/salesforcer and, more specifically, the Working with Reports vignette which provides a soft introduction to these concepts. Finally, keep an eye out as more dashboard and analytics notifications functionality is also added.

Support for Bulk 2.0 API Queries

In Salesforce version 47.0 (Winter ’20) query functionality added to the Bulk 2.0 API. In the overview of this feature Salesforce emphasizes the consistency with the REST APIs and the ease of use (e.g. “Automatic File Batching”), but does not mention any claims in terms of speed compared to the Bulk 1.0 query functionality. In {salesforcer 0.2.2} the default API when using sf_run_bulk_query() or sf_query_bulk() is now the Bulk 2.0 API, assuming it is better than the Bulk 1.0 API. However, You can easily switch between the APIs just as you did before in previous {salesforcer} releases by specifying it in the api_type argument. Please note that, because of additional standardization on the column ordering and arguments to guess types, the queries below will all return the same exact format of results. For example we prioritize the following fields in queries alphabetically within this prioritization waterfall:

First, the sObject field (indicates the record’s object if multiple objects returned in the results)
Second, the Id field (Id, id, sf__Id)
Third, record success status (Success, success, sf_Success)
Fourth, record created status (Created, created, sf__Created)
Fifth, record error(s) status (Error, error, errors, errors.statusCode, errors.fields, errors.message, sf__Error)
Sixth, all other fields from the target object (e.g. Name, Phone, etc.)
Seventh, relationship fields (fields from a parent or child of the target). For example, anything typically containing a dot like Account.Id, Owner.Name, etc.

In short, Bulk 2.0 now has query functionality and it is consistent with the other API’s query functionality. I recommend checking to see for yourself which API works well. Below is a simple example comparing a single run of the REST, Bulk 1.0, and Bulk 2.0 APIs. Consider using the {microbenchmark} package to run more precise performance tests.

soql <- "SELECT Id, LastName, Account.Id, Account.Name, Owner.Id
         FROM Contact
         LIMIT 1000"
system.time(sf_query(soql, api_type = "REST"))
#>    user  system elapsed 
#>   4.033   0.032   5.065
system.time(sf_query(soql, object_name = "Contact", api_type = "Bulk 1.0"))
#>    user  system elapsed 
#>   0.186   0.018   8.345
system.time(sf_query(soql, api_type = "Bulk 2.0"))
#>    user  system elapsed 
#>   0.046   0.009   7.541

Support for Relationship Queries

One upgrade for {salesforcer 0.2.2} is better support for relationship queries, both child-to-parent lookups using the dot notation and parent-to-child nested queries. In prior releases the results were not parsed consistently and presented themselves in a variety of hard to debug issues on GitHub that were brought up in #19, #35, #38, and #54. This release finally aims to address some of those bugs through more consistent parsing methods for both the XML returned by the SOAP API and the JSON returned by the REST API. However, I would strongly recommend testing in your Org with your own queries to see the impact before deploying to a production environment. If any unexpected behavior crops up, then please file an issue on GitHub using the query issue template so we can get it resolved.

Old Nested Query Behavior (v0.1.4 or earlier)

sf_query("SELECT Name, (SELECT LastName FROM Contacts) FROM Account", api_type="SOAP")
#> # A tibble: 24 x 4
#>    Id    Name                        Contacts         LastName
#>    <lgl> <chr>                       <list>           <chr>   
#>  1 NA    GenePoint                   <named list [4]> <NA>    
#>  2 NA    <NA>                        <NULL>           Frank   
#>  3 NA    United Oil & Gas, UK        <named list [4]> <NA>    
#>  4 NA    <NA>                        <NULL>           James   
#>  5 NA    United Oil & Gas, Singapore <named list [5]> <NA> 
#> # … with 19 more rows

New Query Behavior (v0.2.2)

sf_query("SELECT Name, (SELECT LastName FROM Contacts) FROM Account", api_type="SOAP")
#> # A tibble: 16 x 4
#>    Name                        Contact.LastName
#>    <chr>                       <chr>
#>  1 GenePoint                   Frank
#>  2 United Oil & Gas, UK        James
#>  3 United Oil & Gas, Singapore D'Cruz
#>  4 United Oil & Gas, Singapore Ripley
#>  5 Edge Communications         Forbes
#> # … with 11 more rows

A new vignette has been included with this release that covers the types of queries currently supported by the package and is available here: Supported Queries. I highly recommend reviewing for guidance and inspiration on how to what types of queries are possible running against your Org.

Additional Information

For a complete listing of all changes made in recent releases of {salesforcer} please view the Changelog (aka NEWS.md) file. Bug reports and feature requests are welcome on GitHub in the repository issues section.

Thank you for your continued support!

Running GitHub Actions Sequentially

Steven M. Mortimer — Sun, 26 Jul 2020 20:05:58 +0000

TL;DR: If you need sequential execution in GitHub Actions consider these solutions:

Sequential steps: Steps within a job are always executed sequentially!
Sequential jobs: Set max-parallel: 1 within the jobs.strategy element of the workflow.
Sequential workflows: Use a repository_dispatch API call at the end of the workflow to trigger the next workflow (code available in the An Example section below).

The issue

Ever since the generally available release of GitHub Actions in November (2019), it seems like many R packages developed on GitHub have switched from Travis CI or another continuous integration service to now using GitHub Actions (GHA). It seems like a great service to have integrated as closely as possible to your codebase, but the product is still under active development. There is a dedicated tag in the GitHub Support Community to ask questions and browse answers. However, in a few different questions (here, here, and here) it seems like folks are still grappling with GitHub’s design decision to execute all workflows and jobs in a repository in parallel, myself included.

As background, it is important to note here the basic lingo of GitHub Actions. A repository can contain one or more ‘workflows’ defined by YAML files located in the .github/workflows folder in the top level of the repository. Each ‘workflow’ can contain one or more ‘jobs’ that execute a series of ‘steps’. By default, all steps in a single job execute sequentially. If you’re trying to limit the number of parallel ‘jobs’ then you you can set a limit of 1 for the workflow by setting max-parallel: 1 within the jobs.strategy element of the workflow YAML. However, the issue with multiple jobs in a single workflow is that if one job fails out of 10 jobs in your workflow, then you’ll have to re-run all 10 jobs for the workflow status to be a success. For this reason, I have decided to split jobs across different workflows. This way, I can re-run one individually, if needed. I will refer to ‘jobs’ and ‘workflows’ interchangeably throughout this article because you can have workflows that execute one job and one job only.

Why do I need sequential workflows/jobs?

The reason I need this functionality is that my workflows interact with a 3rd-party service (Salesforce) and one workflow might affect the results of another workflow if accessing the service simultaneously. I also want to prevent other workflows from executing if I can’t get the first one to succeed since the issue could occur across all workflows. This allows me to reduce the total number of API calls to Salesforce which are capped in a 24-hour period by catching issues early before other workflows execute.

An Example

You can setup sequential workflows using a repository_dispatch action in 4 easy steps:

For context, a required element in every workflow the name of the GitHub event that triggers the workflow. For example, on: pull_request means “execute this workflow every time a pull request is opened”. If you want to run workflows sequentially, then you just need to issue a specific event type that lets the next workflow know when to begin. You could try to write your own solution that uses the GitHub Actions APIs to list the workflows, jobs, or check-suites and find out which ones have failed or not, but the easy alternative is to use a repository_dispatch event.

“You can use this endpoint to trigger a webhook event called repository_dispatch when you want activity that happens outside of GitHub to trigger a GitHub Actions workflow or GitHub App webhook.”

^{https://developer.github.com/v3/repos/#create-a-repository-dispatch-event}

Workflows aren’t aware of other workflows, so this event webhook is perfect to trigger, or “daisy-chain”, separate workflows. You could execute the event via a curl command from the shell in a new job step, but I recommend using the Repository Dispatch action that Peter Evans has released in the GitHub Marketplace, which makes it dirt simple to execute a repository dispatch event from a workflow.

Step 1 - Create a Personal Access Token (PAT)

Follow GitHub’s instructions here and when it comes time to select the scopes, or permissions, you’d like to grant the token then check "repo" if you’re on a private repository or "public_repo" if you’re on a public one.

Step 2 - Add the PAT as an actions secret in the repository

Follow GitHub’s instructions here. I recommend naming the secret REPO_GHA_PAT.

Step 3 - Add the `repository_dispatch` event to Workflow 1

This step is where you update Workflow 1’s YAML file. For this example, consider “Workflow 1” as the workflow, and the job(s) contained within it, as what you’d like to execute first. Think of “Workflow 2” as the workflow you’d like to execute after “Workflow 1”. Add the following as the last step in the workflow YAML file:

- name: Trigger next workflow
  if: success()
  uses: peter-evans/repository-dispatch@v1
  with:
    token: ${{ secrets.REPO_GHA_PAT }}
    repository: ${{ github.repository }}
    event-type: trigger-workflow-2
    client-payload: '{"ref": "${{ github.ref }}", "sha": "${{ github.sha }}"}'

In this example above, you’ll notice the line if: success(). This means that, only if all the prior steps in the workflow were successful, we should run this step that triggers Workflow 2. Also you should notice the line which passes data from Workflow 1 to Workflow 2:

client-payload: '{"ref": "${{ github.ref }}", "sha": "${{ github.sha }}"}'

In this case it is telling Workflow 2 the branch and commit hash to checkout and use so that we know Workflows 1 and 2 are using the same exact code. Remember, it’s possible that you have a couple different Workflow 1’s running because you’ve pushed code or triggered them in some way and you want to make sure each one triggers the same code in Workflow 2.

Step 4 - Add the `repository_dispatch` event as trigger in Workflow 2 YAML

This step is where you update Workflow 2’s YAML file. First, add your event name as the type of repository dispatch that should trigger Workflow 2. This name must match exactly as what you specified the event-type we covered in Step 3 (the last step of Workflow 1). In our case we called the event to trigger Workflow 2 as event-type: trigger-workflow-2. This could be called anything you wish. The important part is using the same name in the types key of the Workflow 2 YAML file. It should be included within square brackets and without quotes as shown below:

name: Workflow 2

on:
  repository_dispatch:
    types: [trigger-workflow-2]

Second, use the client payload data from the event to checkout the same code. You can so this by modifying the checkout step, usually one of the first step in your job.

    steps:
      - uses: actions/checkout@v2
        with:
          ref: ${{ github.event.client_payload.sha }}
      
      ... (other steps)
      
      - uses: r-lib/actions/setup-r@master
      
      - uses: r-lib/actions/setup-pandoc@master

That’s it! If you have more than 2 workflows, then simply add a unique trigger as the last step in each workflow that calls the next. For the first workflow, I typically trigger based on a push to a certain branch like this:

name: Workflow 1

on:
  push:
    branches: main

If you would like to see a complete example in action, then feel free to browse the .github/workflows folder for the {salesforcer} package.

Considerations

If you need your jobs to execute sequentially but you want them to all still run, even if some fail, then just change the if: statement mentioned above in Step 3 to if: always(). The only reason I did not do this is that even if the following workflow is successful, it will get triggered again when we re-run the failed workflow, which I didn’t want. In order to achieve that you may need to use some more advanced tricks/hacks to only execute if the next workflow has the same sha and the latest run does not have a ‘completed’ status.

Another consideration is the cost to execute GitHub actions in private repositories. It’s true that many projects will not likely need to enforce sequential workflows because the tests, examples, checks or dependencies do not affect other workflows. However, GitHub Actions is only free for public respositories. You may have private repositories and want to limit the amount of processing time so you can stay within the free tier (less than 2,000 minutes per month). Sequential execution can prevent all the workflows from executing if upstream workflows fail.

References

You may find the following resources helpful when setting up on your own repository:

5 steps to change GitHub default branch from master to main

Steven M. Mortimer — Thu, 23 Jul 2020 12:37:37 +0000

The R community is a diverse and inclusive group. About a month ago (June 2020) GitHub announced it would start to remove references to ‘master’ across its sites and I’m surprised the R community hasn’t jumped on the opportunity to be on the bleeding edge of this change. I think we can all take a very quick and very small step forward in removing divisive language in tech by changing the default branch name in R package repositories to ‘main’ instead of ‘master’. There are a wealth of other posts/tutorials that others have written. Here is a small sampling listed in order of publishing date:

Easily rename your Git default branch from master to main by Scott Hanselman
Git: Renaming the “master” branch by Richard Walters
How To: Safely Rename master Branch on GitHub by Jake Jarvis

Eugene Shen even made a small web app to automate the entire process for you that is available at: (https://eyqs.ca/tools/rename/). I thought I’d share the 5 simple steps that I tested and used to make the change in under 1 minute. Yes, you can do this in under a minute, so what’s stopping you? If you are concerned about compatibility, GitHub is providing updates on their renaming efforts here: https://github.com/github/renaming.

Below are screenshots to walk you through each step. In addition, there is a block of code with all Terminal commands in one place at the end of this post here and GitHub repo here: https://github.com/StevenMMortimer/master-to-main.

Steps

Step 1 - Move the ‘master’ branch to ‘main’

Run the following command which creates a branch called ‘main’ using the history from ‘master’. Using the argument -m will transfer all of the commit history on the ‘master’ branch to your new ‘main’ branch so nothing gets lost.

git branch -m master main

Step 2 - Push ‘main’ to remote repo

Remember that git is version control software on your local machine and GitHub is the remote server that stores your code. For this reason, you’ll have to push your new ‘main’ branch up to GitHub and tell the local branch to start tracking the remote branch with the same name.

git push -u origin main

Step 3 - Point HEAD to ‘main’ branch

At this stage if ‘master’ was your default branch you cannot remove it without first changing HEAD, the pointer to the current branch reference. This following command will point it to ‘main’.

git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/main

All three steps are shown in the screenshot below:

At this point you can take a breather. You’re on the home stretch! If you want to check that things are going as planned then you’re welcome to run the following that should show the HEAD is pointing to main which now frees you up to delete ‘master’. Note: When you enter this command in your Terminal you will have to type :q to exit it. Not CTRL+C, ESC, etc.

git branch -a

Step 4 - Change default branch to ‘main’ on GitHub site

At this point you’ve succesfully transitioned everything to the ‘main’ branch, but you can’t delete the ‘master’ branch without changing the default branch in GitHub to something other than ‘master’. This is the only step that requires you to leave the Terminal and navigate in your browser to your GitHub repository. From there, click "Settings" -> "Branches" on the left rail and change the default branch to ‘main’. I’ve included some screenshots below and GitHub provides instructions for doing this here: https://docs.github.com/en/github/administering-a-repository/setting-the-default-branch.

Step 5 - Delete ‘master’ branch on the remote repo

Now that the local ‘master’ is gone and there’s nothing pointing to it on Github you can delete it with the following command:

git push origin --delete master

That’s it! You’re done! You should no longer see “master” or “remotes/origin/master” locally or on your GitHub repository. If you want to check, you can always run the following:

git branch -a

Feel free to let me know if this worked for you or if you had issues! Note: I take no responsibility for messing up your git repo. In case this wasn’t clear enough, all instructions are available in its own repo here: https://github.com/StevenMMortimer/master-to-main.

All commands

# Step 1 
# create main branch locally, taking the history from master
git branch -m master main

# Step 2 
# push the new local main branch to the remote repo (GitHub) 
git push -u origin main

# Step 3
# switch the current HEAD to the main branch
git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/main

# Step 4
# change the default branch on GitHub to main
# https://docs.github.com/en/github/administering-a-repository/setting-the-default-branch

# Step 5
# delete the master branch on the remote
git push origin --delete master

Setting up VS Code for Python Development like RStudio

Steven M. Mortimer — Tue, 14 Jul 2020 18:45:00 +0000

In this article I will highlight the features of VS Code that match RStudio exactly, such as the “interactive notebook window” (called the Console in R) or the “variable explorer” (like running View() on a data frame in RStudio). At the bottom of this post I will provide two JSON files (settings.json and keybindings.json) and a block of code to install from the command line a list of extensions that I recommend. By using these files as a guide you can configure your VS Code installation to do a pretty good job at mimicking RStudio.

First, why try to write Python like you write R code in RStudio??

RStudio is a great all around IDE for data analysis. A few years ago I was transitioning from writing a lot of R code to more Python code at work. I initially chose PyCharm as my Python IDE for a variety of reasons outlined in another blog post of mine: An R User Chooses a Python IDE. However, as of last summer (June 2019), I switched to VS Code as a Python IDE and never looked back. In hindsight, PyCharm just seems too clunky with an over-engineered GUI of buttons to click and not really be sure what’s going on. VS Code is making great strides towards becoming an IDE that works well for the REPL (read–eval–print loop) style of coding that RStudio excels at supporting. I love how lightweight VS Code feels and how the configurations are portable via JSON files making it easier to share a common config with team members. I’ll keep writing R code in RStudio, but VS Code is quckly becoming a second home for me to write Python code.

How to Configure VS Code to run like RStudio

Aesthetics (Textmate theme and margin column ruler)

You may know that the default theme in RStudio is “Textmate”. This is found in Tools -> Global Options -> Appearance. To get the color scheme in VS Code you can install this with the Textmate theme extension.

I tend to not change much beyond the default RStudio installation just so that I don’t need to maintain a unique and unfamiliar setup compared to other colleagues who are also using RStudio. The only customization I recommend is adding a vertical line at 80 characters. This is a must-have feature to keep your code readable and you can do the same with VS Code by specifying "editor.rulers" in your user settings. More detail on how to configure that is in the section of this post entitled Settings JSON File.

Running Code

As far as running code in RStudio, it is fairly common to write code in the “Source” pane (normally above the console), then send the code to the “Console” pane to run using CMD+ENTER (CRTL+ENTER if on Windows - please assume anywhere I refer to CMD in this article it is CTRL if you use Windows). In VS Code you can think of the Editor pane as having the exact same purpose (writing a script), but instead of sourcing lines to the “Console” you use the same command (CMD+ENTER) to run the code in the Python Interactive Window. More specifically, the keyboard shortcut you need to set in VS Code is for the command "python.datascience.execSelectionInteractive".

This is a game changer when writing Python code for analysis because you no longer need to code in a Jupyter Notebook to execute your analysis. Simply write your code in a .py file and press CMD+ENTER to execute line-by-line in the Python Interactive Window. Repeat this process as you run code, explore, and build out your analysis. Note that you can also type Python directly into the Interactive Window just like you can type directly in R’s Console as well to execute code.

https://code.visualstudio.com/docs/python/jupyter-support-py#_python-interactive-window

Variable Explorer (“Environment” tab in R)

Notebooks always seem clunky in terms of executing single lines of code and reviewing variables in the global environment, but VS Code has a variable explorer just like the “Environment” tab in R.

https://code.visualstudio.com/docs/python/jupyter-support-py#_variable-explorer-and-data-viewer

Data Viewer (Like running `View()` in R)

Once you’ve created a pandas DataFrame you can explore a dynamic view of the data frame with basic filtering and sorting. I have found this be exactly like the View() function in RStudio. The only difference is it does seem to struggle a bit when viewing data frames over 1 million rows. It is laggy or crashes VS Code, but I don’t find that too much of a problem because you can always save a sample of your dataset into a variable or aggregate your data prior to viewing.

https://code.visualstudio.com/docs/python/jupyter-support-py#_variable-explorer-and-data-viewer

Plot Viewer (“Plots” tab in R)

You can flip through prior plots, save them, resize, and interact with plots the same way as you would in the “Plots” pane of RStudio. Nothing different here.

https://code.visualstudio.com/docs/python/jupyter-support-py#_plot-viewer

Version Control (“Git” tab in R)

Just like RStudio you can stage files, write a commit message, and push to a git repository. Instead of ticking a checkbox to stage files, you will have to press the + sign, which is about the only difference. Everything else is the similar as far as clicking on the files to see “diffs”, writing the commit message, and pushing code to a remote repository.

https://code.visualstudio.com/docs/editor/versioncontrol#_git-support

All of these git features can be made even better with Eric Amodio’s GitLens extension, which I highly recommend. It makes it easier to navigate branches, see when files were changed and by whom.

What’s missing or different in VS Code?

R’s Document Outline (aka Minimap in VS Code)

RStudio creates a nice outline of your scripts based on your functions and comments that start with:

# A new section ----
code here...

This table of contents style view is helpful. As a somewhat suitable alternative I have become accustomed to annotating Python code blocks with region folding syntax which creates collapsible code sections in the VS Code editor.

# region ----

code here...

# endregion

Debugging

RStudio has a nice feature that will ask you if you want to run a piece of code with debug mode on if it initially errors. VS Code’s Python Interactive Window also prints out error messages, but will not let you debug with the click of a button. You can however set breakpoints and run your script in debug mode, which is a familiar experience, just not as seamless as RStudio.

There are obviously many other features, like having a built-in terminal, remote connections, app development via Shiny, etc. that make R/RStudio and Python/VS Code different tools, but if you’re just running analysis the two can provide very similar workflows, right down to the keyboard shortcuts. The only challenge you may have left is figuring out how to make your pandas code as legible and well organized as dplyr code… but that is for another day. (Hint: It’s possible with a strong commitment to method chaining).

Setup Files and Scripts

Settings JSON File

In order to update your settings.json file, open the Command Palette with CMD+SHIFT+P and select "Preferences: Open Settings (JSON)" to edit the JSON file where your settings are held. You can copy/paste the entire block of JSON below or just individual lines.

Note: These settings files were automatically generated from my VS Code installation using Shan Khan’s Settings Sync extension.

settings.json (User Settings, as opposed to workspace/project-specific settings)

{
    "window.zoomLevel": 1,
    "explorer.confirmDelete": false,
    "explorer.confirmDragAndDrop": false,
    "files.associations": {},
    "files.autoSave": "off",
    "files.exclude": {
        "**/.git": true,
        "**/.svn": true,
        "**/.hg": true,
        "**/CVS": true,
        "**/.DS_Store": true,
        "**/.history/**": true,
        "**/History_**": true
    },
    "workbench.colorTheme": "textmate",
    "workbench.editor.enablePreview": false,
    "workbench.startupEditor": "welcomePageInEmptyWorkbench",
    "terminal.integrated.fontSize": 14,
    "terminal.integrated.cursorStyle": "line",
    "terminal.integrated.copyOnSelection": true,
    "terminal.integrated.confirmOnExit": false,
    "editor.largeFileOptimizations": false,
    "editor.suggest.shareSuggestSelections": true,
    "editor.suggestSelection": "first",
    "editor.minimap.enabled": false,
    "editor.foldingStrategy": "indentation",
    "editor.showFoldingControls": "always",
    "editor.rulers": [
        80
    ],
    "editor.formatOnSave": true,
    "python.linting.enabled": true,
    "python.linting.flake8Enabled": true,
    "python.linting.flake8Args": [
        "--ignore=E203, E266, E501, W503",
        "--max-line-length=79",
        "--select=B,C,E,F,W,T4,B9",
        "--max-complexity=18"
    ],
    "python.formatting.provider": "autopep8",
    "python.formatting.autopep8Args": [
        "--ignore=E501,E402"
    ],
    "sonarlint.rules": {
        "python:S3776": {
            "level": "off"
        }
    },
    "python.languageServer": "Microsoft",
    "python.dataScience.askForKernelRestart": false,
    "python.dataScience.runStartupCommands": "%load_ext autoreload\\n%autoreload 2",
    "python.dataScience.sendSelectionToInteractiveWindow": true,
    "python.dataScience.useNotebookEditor": false,
    "git.autofetch": true,
    "git.confirmSync": false,
    "gitlens.views.repositories.location": "gitlens",
    "gitlens.views.repositories.files.layout": "list",
    "gitlens.views.fileHistory.location": "gitlens",
    "gitlens.views.lineHistory.enabled": false,
    "gitlens.views.compare.location": "gitlens",
    "gitlens.views.search.location": "gitlens",
    "gitlens.mode.statusBar.enabled": true,
    "gitlens.mode.statusBar.alignment": "left",
    "gitlens.currentLine.enabled": false,
    "gitlens.hovers.enabled": false,
    "vsintellicode.modify.editor.suggestSelection": "automaticallyOverrodeDefaultValue",
    "tabnine.experimentalAutoImports": true,
    "indentRainbow.errorColor": "rgba(255,255,255,0.0)",
    "todo-tree.tree.flat": true,
    "todo-tree.highlights.defaultHighlight": {
        "type": "text-and-comment",
        "foreground": "grey"
    },
    "better-comments.tags": [
        {
            "tag": "?",
            "color": "#8f5785",
            "strikethrough": false,
            "backgroundColor": "transparent"
        }
        {
            "tag": "*",
            "color": "#69a800",
            "strikethrough": false,
            "backgroundColor": "transparent"
        }
    ],
    "bracket-pair-colorizer-2.colors": [
        "#992e24",
        "#ffb997",
        "#263859"
    ],
    "cSpell.userWords": [
        "YYYYMMDD",
        "groupby",
        "idxmax",
        "inplace",
        "itertools",
        "multindex",
        "rfind",
        "strptime"
    ]
}

Keyboard Shortcuts JSON File

Updating the keyboard shortcuts JSON file is similar to editing settings.json. Open the Command Palette with CMD+SHIFT+P and select "Preferences: Open Keyboard Shortcuts (JSON)" to edit the JSON file where your settings are held. You can copy/paste the entire block of JSON below or just individual lines.

In addition to setting the keyboard shortcut CMD+ENTER to execute lines in the Interactive Window I have set a few other two keyboard shortcuts to work exactly like RStudio. For example, clearing the console and restarting the R session:

CRTL+L - “Clear Console” (in RStudio) => Clear Cells (in Python Interactive Window)
CMD+SHIFT+F10 - “Restart R” (in RStudio) => Restart Kernel (in Python Interactive Window)

Of course there are other shortcuts that you can configure like R’s block comment command (Code -> Comment/Uncomment Lines) (CMD+SHIFT+C). Simply set the VS Code command "editor.action.commentLine" to that shortcut.

Another example is R’s command (Code -> Reflow Comment) which hard wraps code to 80 characters using CMD+SHIFT+/. VS Code can do the same after installing the extension called “rewrap” and then giving its command "rewrap.rewrapComment" the same keyboard shortcut. It is really up to you to configure the shortcuts that you use most often.

keybindings.json

// Place your key bindings in this file to override the defaultsauto[]
[
    {
        "key": "cmd+enter",
        "command": "python.datascience.execSelectionInteractive",
        "when": "editorTextFocus && python.datascience.featureenabled && python.datascience.ownsSelection && !findInputFocussed && !notebookEditorFocused && !replaceInputFocussed && editorLangId == 'python'"
    },
    {
        "key": "shift+enter",
        "command": "-python.datascience.execSelectionInteractive",
        "when": "editorTextFocus && python.datascience.featureenabled && python.datascience.ownsSelection && !findInputFocussed && !notebookEditorFocused && !replaceInputFocussed && editorLangId == 'python'"
    },
    {
        "key": "cmd+enter",
        "command": "python.datascience.runcurrentcell",
        "when": "editorTextFocus && python.datascience.featureenabled && python.datascience.hascodecells && !editorHasSelection && !notebookEditorFocused"
    },
    {
        "key": "ctrl+enter",
        "command": "-python.datascience.runcurrentcell",
        "when": "editorTextFocus && python.datascience.featureenabled && python.datascience.hascodecells && !editorHasSelection && !notebookEditorFocused"
    },    
    {
        "key": "cmd+enter",
        "command": "notebook.cell.executeAndSelectBelow",
        "when": "notebookEditorFocused && activeEditor == 'workbench.editor.notebook'"
    },
    {
        "key": "shift+enter",
        "command": "-notebook.cell.executeAndSelectBelow",
        "when": "notebookEditorFocused && activeEditor == 'workbench.editor.notebook'"
    },    
    {
        "key": "ctrl+l",
        "command": "python.datascience.removeallcells",
        "when": "python.datascience.featureenabled && !terminalFocus"
    },
    {
        "key": "ctrl+l",
        "command": "workbench.action.terminal.clear",
        "when": "terminalFocus"
    },    
    {
        "key": "cmd+shift+f10",
        "command": "python.datascience.restartkernel"
    },
    {
        "key": "cmd+shift+c",
        "command": "editor.action.commentLine",
        "when": "editorTextFocus && !editorReadonly"
    },
    {
        "key": "cmd+/",
        "command": "-editor.action.commentLine",
        "when": "editorTextFocus && !editorReadonly"
    },
    {
        "key": "cmd+i",
        "command": "python.datascience.showhistorypane"
    },
    {
        "key": "ctrl+shift+/",
        "command": "rewrap.rewrapComment",
        "when": "editorTextFocus"
    },
    {
        "key": "alt+q",
        "command": "-rewrap.rewrapComment",
        "when": "editorTextFocus"
    },
    {
        "key": "ctrl+shift+s",
        "command": "extension.updateSettings"
    },
    {
        "key": "shift+alt+u",
        "command": "-extension.updateSettings"
    }
]

Extensions Installation Script

You can run each of the lines below in your Terminal or other Command Prompt after installing VS Code in order to install the extensions. NOTE: This does require having the code command in your PATH. On Mac use the following instructions to install ‘code’ command in PATH: https://code.visualstudio.com/docs/setup/mac#_launching-from-the-command-line. Alternatively, you can search for extensions in the VS Code Marketplace and install from there (more detail is available here: https://code.visualstudio.com/docs/editor/extension-gallery#_browse-for-extensions).

# editor visual aids
code --install-extension CoenraadS.bracket-pair-colorizer-2
code --install-extension oderwat.indent-rainbow
code --install-extension gerane.Theme-textmate

# development aids
code --install-extension eamodio.gitlens
code --install-extension aaron-bond.better-comments
code --install-extension Gruntfuggly.todo-tree
code --install-extension mikestead.dotenv
code --install-extension streetsidesoftware.code-spell-checker

# code formatting
code --install-extension ms-python.python
code --install-extension VisualStudioExptTeam.vscodeintellicode
code --install-extension TabNine.tabnine-vscode
code --install-extension christian-kohler.path-intellisense
code --install-extension stkb.rewrap
code --install-extension SonarSource.sonarlint-vscode

# file viewer tools
code --install-extension DotJoshJohnson.xml
code --install-extension jithurjacob.nbpreviewer
code --install-extension GrapeCity.gc-excelviewer

# remote development tools
code --install-extension ms-azuretools.vscode-docker
code --install-extension ms-vscode-remote.remote-containers
code --install-extension ms-vscode-remote.remote-ssh
code --install-extension ms-vscode-remote.remote-ssh-edit
code --install-extension ms-vscode-remote.remote-wsl
code --install-extension ms-vscode-remote.vscode-remote-extensionpack

# tool to save your settings as a gist
code --install-extension Shan.code-settings-sync

salesforcer 0.1.3 - Features for Better CRM Data Management

Steven M. Mortimer — Mon, 10 Jun 2019 00:00:00 +0000

The latest version of the salesforcer package (v0.1.3) is now available on CRAN and it is ready to help you better manage your Salesforce data. Along with a host of bug fixes this release has three big features:

Pull lists of deleted and updated records by timeframe and be able to undelete them or permanently delete them from the Recycle Bin to free up space in your org
Search for duplicates using the duplicate rules in your org and merge records
Control API call behaviors to perform “All or None” inserts, bypass duplicate rules, configure the transfer of related records when changing a record’s owner, and more

Pulling Lists of Delete and Updated Records

Salesforce is an organizational tool and with many hands touching the system sometimes you are called in to do some detective work to figure out which records have been deleted or updated in your org. Luckily, Salesforce has your back. Their SOAP API has two methods that will return the deleted or updated records within a timeframe. Those two methods have been implemented in salesforcer 0.1.3 as sf_get_deleted() and sf_get_updated(). Here is an example pulling out anything deleted yesterday:

library(tidyverse)
library(lubridate)

# if no timezone is provided UTC is assumed
deleted_recs <- sf_get_deleted("Contact", start = Sys.Date() - 1, end = Sys.Date())
# here is how to pull deleted using yesterday according to EDT with help from lubridate
deleted_recs <- sf_get_deleted("Contact", 
                               start = ymd(Sys.Date()-1, tz="America/New_York"), 
                               end = ymd(Sys.Date(), tz="America/New_York"))
deleted_recs
#> # A tibble: 125 x 2
#>   deletedDate         id                
#>   <dttm>              <chr>             
#> 1 2020-09-11 19:08:40 0033s000014AyIXAA0
#> 2 2020-09-11 19:08:40 0033s000014AyIbAAK
#> 3 2020-09-11 19:08:46 0033s000014AyIgAAK
#> 4 2020-09-11 19:11:08 0033s000014AyLFAA0
#> 5 2020-09-11 19:11:08 0033s000014AyLGAA0
#> # … with 120 more rows

Now if you see records in there that you may want back, now it’s as simple as calling the sf_undelete() function with the ids of the records you want.

# undelete the first three records from the list of records deleted yesterday 
undeleted <- sf_undelete(deleted_recs$id[1:3])
undeleted
#> # A tibble: 3 x 2
#>   id                 success
#>   <chr>              <lgl>  
#> 1 0033s000014AyIXAA0 TRUE   
#> 2 0033s000014AyIbAAK TRUE   
#> 3 0033s000014AyIgAAK TRUE

You can double check that the records are actually no longer deleted by running a query for them and checking the IsDeleted field like this:

# check that the first three records are no longer deleted 
is_not_deleted <- sf_query(sprintf("SELECT Id, IsDeleted 
                                    FROM Contact 
                                    WHERE Id IN ('%s')", 
                                   paste0(undeleted$id, collapse="','")))
is_not_deleted
#> # A tibble: 3 x 2
#>   Id                 IsDeleted
#>   <chr>              <lgl>    
#> 1 0033s000014AyIXAA0 FALSE    
#> 2 0033s000014AyIbAAK FALSE    
#> 3 0033s000014AyIgAAK FALSE

Now let’s say you really do want them deleted and by that, I mean permanently from your org so they do not count against your space quota. Now you can empty items in your Recycle Bin. Here is an example where we delete the records we just undeleted and then empty them from the Recycle Bin.

# delete the records to move them to the Recycle Bin, them empty them from it
deleted <- sf_delete(is_not_deleted$Id)
really_deleted <- sf_empty_recycle_bin(deleted$id)
really_deleted
#> # A tibble: 3 x 2
#>   id                 success
#>   <chr>              <lgl>  
#> 1 0033s000014AyIXAA0 TRUE   
#> 2 0033s000014AyIbAAK TRUE   
#> 3 0033s000014AyIgAAK TRUE

If you want to double-check the field values of a deleted record, just make sure to set queryall = TRUE in your query so that it returns deleted or archived records in the resultset. Note that after records are deleted from the Recycle Bin using this call, they can be queried using queryall = TRUE for some time. Typically this time is 24 hours, but may be shorter or longer.

# permanently deleted records not there in regular query
not_visible <- sf_query(sprintf("SELECT Id, IsDeleted 
                                 FROM Contact 
                                 WHERE Id IN ('%s')", 
                                paste0(really_deleted$id, collapse="','")))
not_visible
#> # A tibble: 0 x 0

# but still there if using queryall for ~24 hrs
still_visible <- sf_query(sprintf("SELECT Id, IsDeleted 
                                   FROM Contact 
                                   WHERE Id IN ('%s')", 
                                  paste0(really_deleted$id, collapse="','")), 
                          queryall = TRUE)
still_visible
#> # A tibble: 3 x 2
#>   Id                 IsDeleted
#>   <chr>              <lgl>    
#> 1 0033s000014AyIXAA0 TRUE     
#> 2 0033s000014AyIbAAK TRUE     
#> 3 0033s000014AyIgAAK TRUE

Searching for Duplicates and Merging Them

Part of having a Salesforce instance means maintaining a high-level of data integrity. That means not allowing duplicates to crop up and, if they do, merging and deleting them quickly so that any activity with a record can be tracked in a single place. In salesforcer 0.1.3 you can find duplicates using the Duplicate Rules already in place in your org. In the example below I will create three records with the same email address. I’ve already set up a Duplicate Rule in my org to recognize and prevent records from being creating if they have the same email address as another record already in Salesforce. When I run sf_find_duplicates() it finds all three records.

# create the records 
new_contacts <- tibble(FirstName = rep("Rick", 3), 
                       LastName = rep("James", 3), 
                       Email = rep("rick.james@gmail.com", 3),
                       Description = c("Musician", "Funk Legend", NA))
new_contacts
#> # A tibble: 3 x 4
#>   FirstName LastName Email                Description
#>   <chr>     <chr>    <chr>                <chr>      
#> 1 Rick      James    rick.james@gmail.com Musician   
#> 2 Rick      James    rick.james@gmail.com Funk Legend
#> 3 Rick      James    rick.james@gmail.com <NA>

# insert the records
new_recs <- sf_create(new_contacts, "Contact", 
                      DuplicateRuleHeader = list(allowSave = TRUE,
                                                 includeRecordDetails = FALSE,
                                                 runAsCurrentUser = TRUE))
new_recs
#> # A tibble: 3 x 2
#>   id                 success
#>   <chr>              <lgl>  
#> 1 0033s000014B3IZAA0 TRUE   
#> 2 0033s000014B3IaAAK TRUE   
#> 3 0033s000014B3IbAAK TRUE

# find them as dupes
dupes_found <- sf_find_duplicates(search_criteria = list(Email = "rick.james@gmail.com"),
                                  object_name = "Contact")
#> Using Contact rules:
#>   - Standard_Rule_for_Contacts_with_Duplicate_Leads
#>   - Contact_Dupe_Email
dupes_found
#> # A tibble: 6 x 2
#>   sObject Id                
#>   <chr>   <chr>             
#> 1 Contact 0033s000013Xt92AAC
#> 2 Contact 0033s000014B35QAAS
#> 3 Contact 0033s000014B35RAAS
#> 4 Contact 0033s000014B3IZAA0
#> 5 Contact 0033s000014B3IaAAK
#> # … with 1 more row

The recordset from sf_find_duplicates() shows the object that the records were found in and their Ids. You will notice that those are the same Ids as what we received back when creating the records to verify that we found those same three records based on the duplicated email address. You will also notice in the call that there is a message showing which rules were executed when searching for duplicates, one of them being a custom matching rule I created called Contact_Dupe_Email. In order for this function to work you must have Duplicate Rules configured in your system. There is a lot of good documentation online about how to set up these rules. Here is a link to one Salesforce Trailhead article that I like: Resolve and Prevent Duplicate Data.

Now if I’d like to merge those three duplicates records into one I can do so using the new sf_merge() function. I need to pick one of the records as the master record and supply its Id to the function. The other records’ Ids will be the victim_ids. All of the fields on the master record will supercede the others, but if you would like to override some of those fields, then just specify them in the master_fields argument. In this example, we are merging the second and third record into the first record. However, I like the description “Funk Legend” from the second record to be on the master record, so I just specify it in the master_fields argument.

merge_summary <- sf_merge(master_id = dupes_found$Id[1], 
                          victim_ids = dupes_found$Id[2:3], 
                          object_name = "Contact",
                          master_fields = c("Description" = "Funk Legend"))
merge_summary
#> # A tibble: 1 x 5
#>   id                 success mergedRecordIds updatedRelatedIds errors
#>   <chr>              <lgl>   <list>          <lgl>             <lgl> 
#> 1 0033s000013Xt92AAC TRUE    <chr [2]>       NA                NA

You can check that the other two records that were merged into the master record have now been deleted and that they are tagged in the MasterRecordId field with the Id of the master record they were merged into.

merge_confirm <- sf_query(sprintf("SELECT Id, MasterRecordId, IsDeleted 
                                   FROM Contact 
                                   WHERE Id IN ('%s')", 
                                  paste0(dupes_found$Id, collapse="','")), 
                          queryall = TRUE)
merge_confirm
#> # A tibble: 6 x 3
#>   Id                 IsDeleted MasterRecordId    
#>   <chr>              <lgl>     <chr>             
#> 1 0033s000013Xt92AAC FALSE     <NA>              
#> 2 0033s000014B35QAAS TRUE      0033s000013Xt92AAC
#> 3 0033s000014B35RAAS TRUE      0033s000013Xt92AAC
#> 4 0033s000014B3IZAA0 FALSE     <NA>              
#> 5 0033s000014B3IaAAK FALSE     <NA>              
#> # … with 1 more row

Controlling API Call Behavior

You may have noticed an argument called all_or_none in the previous CRAN release of salesforcer. This argument is put into the header of the API call so that when records are created/updated/deleted it will only complete if all records were successful, otherwise all of the record changes are rolled back. This “All or None” behavior is just one of over a dozen headers that the SOAP API accepts. The REST, Bulk, and Metadata APIs also have their own headers which can be set. You can pass these controls into salesforcer functions directly or by setting the control argument that now appears in many functions. For example, if you are inserting records and would like to bypass the Duplicate Rules in your org, then set the DuplicateRuleHeader) like this:

new_contacts <- tibble(FirstName = rep("Test", 2),
                       LastName = rep("Contact", 2), 
                       Email = rep("testcontact@gmail.com", 2))

# add control to allow the creation of records that violate a duplicate email rule
new_recs <- sf_create(new_contacts, object_name = "Contact", 
                      DuplicateRuleHeader = list(allowSave = TRUE,
                                                 includeRecordDetails = FALSE,
                                                 runAsCurrentUser = TRUE))
new_recs
#> # A tibble: 2 x 2
#>   id                 success
#>   <chr>              <lgl>  
#> 1 0033s000014B3IjAAK TRUE   
#> 2 0033s000014B3IkAAK TRUE

In the documentation you may notice that the control argument in the sf_create() function definition looks like this control = list(...). This allows you to include the control headers as an argument right in the function or put them inside the sf_control function. An example of using the sf_control() function looks like this:

new_recs <- sf_create(new_contacts, object_name = "Contact", 
                      control=sf_control(DuplicateRuleHeader=list(allowSave=TRUE,
                                                                  includeRecordDetails=FALSE,
                                                                  runAsCurrentUser=TRUE)))
new_recs
#> # A tibble: 2 x 2
#>   id                 success
#>   <chr>              <lgl>  
#> 1 0033s000014B3HhAAK TRUE   
#> 2 0033s000014B3HiAAK TRUE

The sf_control() function works exactly like glm.control() when using the glm() function in the stats package. It will accept any of the extra arguments in the function call and validate them to see if they are accepted by the API type and operation that the function is performing. As mentioned before, there are over a dozen control parameters that are now possible across the various APIs. Those are documented on the pkgdown website’s documentation for sf_control(). There is also a vignette which describes all of this in more detail at Passing Control Args.

Links to GitHub

For a complete listing of all changes made in the 0.1.3 release of salesforcer please view the NEWS.md file on the pkgdown site. Bug reports and feature requests are welcome in the GitHub issues.

Thank you for your continued support!

One R Package a Day

Steven M. Mortimer — Wed, 27 Jun 2018 06:00:00 +0000

Discover one R package a day by following the @RLangPackage account on Twitter! Inspired by One R Tip Day (@RLangTip) I created a Twitter bot that will feature one package with its description and GitHub URL each day. The R programming language (referred to as #rstats on Twitter) now has over 12,600 packages. Tweeting one each day would take more than 34 years to get through them all and that doesn’t even consider the rapid growth of new packages as shown in this post.

The purpose of the @RLangPackage Twitter account is to feature the diversity of packages that R has to offer. Most users have heard of ggplot2 and dplyr, but there are thousands of other great packages waiting to be discovered. In order to highlight those I took a list of all packages on CRAN and filtered it down to ones with anywhere between 5,000 and 1,000,000 CRAN downloads. I then took a subset of that with anywhere between 10 and 1,000 stars on GitHub. I decided to focus on these mildly popular packages with source code hosted on GitHub so that I can embed the link in the tweet and promote packages with code that people have already started to explore. After following those subsetting rules, one package is selected at random after narrowing to a list of packages that haven’t already been tweeted. I am hosting the bot on Heroku and using Heroku Scheduler to send out a tweet each day at 10:30am UTC or 6:30am Eastern Time. Below the credits and resources is the sloppily written Python that’s currently being hosted on Heroku and executed.

Credits

I was found a couple different blog posts on creating a Twitter bot with R, namely here and here, but they didn’t involve deploying on Heroku or another cloud service. However, there were plenty of Python based Twitter bot tutorials, so I took full advantage of them and went the Python route. Below is the host of resources I considered while figuring out how to deploy the app, what Twitter package to use, and some basic Python syntax which, embarrassingly, I should know by now. I think most users will gravitate towards the tweepy Python library but I found that it throws up an error when using it with Conda 3.6. The issue is documented on GitHub https://github.com/tweepy/tweepy/issues/894.

Resources

Used as a general guide to deploying: http://briancaffey.github.io/2016/04/05/twitter-bot-tutorial.html
Setting up credentials as config vars: https://devcenter.heroku.com/articles/config-vars
How to load environment vars from file: https://robinislam.me/blog/reading-environment-variables-in-python/
Timeline dump code: https://github.com/geduldig/TwitterAPI/blob/master/examples/dump_timeline.py

Full Script

# script.py
from os import environ
from os.path import join, dirname
from dotenv import load_dotenv
from re import sub
import pandas
from TwitterAPI import TwitterAPI, TwitterPager

# create .env file path
dotenv_path = join(dirname(__file__), '.env')

# load file from the path
load_dotenv(dotenv_path)

if __name__ == "__main__":

    # connect to api
    api = TwitterAPI(consumer_key=environ['TWITTER_CONSUMER_KEY'],
                     consumer_secret=environ['TWITTER_CONSUMER_SECRET'],
                     access_token_key=environ['TWITTER_ACCESS_TOKEN'],
                     access_token_secret=environ['TWITTER_ACCESS_TOKEN_SECRET'])

    # scrape all prior tweets to check which packages I've already tweeted about
    SCREEN_NAME = 'RLangPackage'
    pager = TwitterPager(api,
                         'statuses/user_timeline',
                         {'screen_name': SCREEN_NAME, 'count': 100})

    # parse out the package name that occurs before the hyphen at the beginning
    previous_pks = []
    for item in pager.get_iterator(wait=3.5):
        if 'text' in item:
            this_pkg = sub("^(\w+) - (.*)", "\\1", item['text'])
            previous_pks.append(this_pkg)

    # convert the package names to a dataframe
    prev_df = pandas.DataFrame({'name': previous_pks})
    prev_df.set_index('name')

    # load the data I've compiled on R packages
    url = "https://raw.githubusercontent.com/StevenMMortimer/one-r-package-a-day/master/r-package-star-download-data.csv"
    all_df = pandas.read_csv(url)
    all_df.set_index('name')

    # do an "anti join" to throw away previously tweeted rows
    all_df = pandas.merge(all_df, prev_df, how='outer', indicator=True)
    all_df = all_df[all_df['_merge'] == 'left_only']

    # focus on packages in middle ground of downloads and stars
    filtered_df = all_df[all_df['stars'].notnull()]
    filtered_df = filtered_df[filtered_df['stars'].between(10,1000)]
    filtered_df = filtered_df[filtered_df['downloads'].notnull()]
    filtered_df = filtered_df[filtered_df['downloads'].between(5000, 1000000)]

    # randomly select one of the remaining rows
    selected_pkg = filtered_df.sample(1)

    # pull out the name and description to see if we need to 
    # truncate because of Twitters 280 character limit
    prepped_name = selected_pkg.iloc[0]['name']
    prepped_desc = sub('\s+', ' ', selected_pkg.iloc[0]['description'])

    name_len = len(prepped_name)
    desc_len = len(prepped_desc)

    # 280 minus 3 for " - ", then minus 23 because links are counted as such,
    # then minus 9 for the " #rstats " hashtag
    if desc_len <= (280-3-23-9-name_len):
        prepped_desc = prepped_desc[0:desc_len]
    else:
        prepped_desc = prepped_desc[0:(280-6-23-9-name_len)] + "..."

    # cobble together the tweet text
    TWEET_TEXT = prepped_name + " - " + prepped_desc + " #rstats " + selected_pkg.iloc[0]['github_url']
    print(TWEET_TEXT)

    # tweet it out to the world!
    r = api.request('statuses/update', {'status': TWEET_TEXT})
    print('SUCCESS' if r.status_code == 200 else 'PROBLEM: ' + r.text)

Scraping Responsibly with R

Steven M. Mortimer — Thu, 21 Jun 2018 00:00:00 +0000

I recently wrote a blog post here comparing the number of CRAN downloads an R package gets relative to its number of stars on GitHub. What I didn’t really think about during my analysis was whether or not scraping CRAN was a violation of its Terms and Conditions. I simply copy and pasted some code from R-bloggers that seemed to work and went on my merry way. In hindsight, it would have been better to check whether or not the scraping was allowed and maybe find a better way to get the information I needed. Of course, there was a much easier way to get the CRAN package metadata using the function tools::CRAN_package_db() thanks to a hint from Maëlle Salmon provided in this tweet:

Cool work! I see you scraped the data, you could get a table with all pkgs and an URL column via tools::CRAN_package_db() which is gentler for CRAN website I think.

Not sure their website has a robots.txt but in general https://t.co/20TtlsGBOf is useful when webscraping 🙂
— Maëlle Salmon (@ma_salmon) June 20, 2018

How to Check if Scraping is Permitted

Also provided by Maëlle’s tweet was the recommendation for using the robotstxt package (currently having 27 Stars + one Star that I just added!). It doesn’t seem to be well known as it only has 6,571 total downloads. I’m hoping this post will help spread the word. It’s easy to use! In this case I’ll check whether or not CRAN permits bots on specific resources of the domain.

My other blog post analysis originally started with trying to get a list of all current R packages on CRAN by parsing the HTML from https://cran.rstudio.com/src/contrib. The page looks like this:

The question is whether or not scraping this page is permitted according to the robots.txt file on the cran.rstudio.com domain. This is where the robotstxt package can help us out. We can check simply by supplying the domain and path that is used to form the full link we are interested in scraping. If the paths_allowed() function returns TRUE then we should be allowed to scrape, if it returns FALSE then we are not permitted to scrape.

library(robotstxt)

paths_allowed(
  paths  = "/src/contrib", 
  domain = "cran.rstudio.com", 
  bot    = "*"
)
#> [1] TRUE

In this case the value that is returned is TRUE meaning that bots are allowed to scrape that particular path. This was how I originally scraped the list of current R packages, even though you don’t really need to do that since there is the wonderful function tools::CRAN_package_db().

After retrieving the list of packages I decided to scrape details from the DESCRIPTION file of each package. Here is where things get interesting. CRAN’s robots.txt file shows that scraping the DESCRIPTION file of each package is not allowed. Furthermore, you can verify this using the robotstxt package:

paths_allowed(
  paths = "/web/packages/ggplot2/DESCRIPTION",
  domain = "cran.r-project.org", 
  bot    = "*"
)
#> [1] FALSE

However, when I decided to scrape the package metadata I did it by parsing the HTML from the canonical package link that resolves to the index.html page for the package. For example, https://cran.r-project.org/package=ggplot2 resolves to https://cran.r-project.org/web/packages/ggplot2/index.html. If you check whether scraping is allowed on this page, the robotstxt package says that it is permitted.

paths_allowed(
  paths = "/web/packages/ggplot2/index.html",
  domain = "cran.r-project.org", 
  bot    = "*"
)
#> [1] TRUE

paths_allowed(
  paths = "/web/packages/ggplot2",
  domain = "cran.r-project.org", 
  bot    = "*"
)
#> [1] TRUE

This is a tricky situation because I can access the same information that is in the DESCRIPTION file just by going to the index.html page for the package where scraping seems to be allowed. In the spirit of respecting CRAN it logically follows that I should not be scraping the package index pages if the individual DESCRIPTION files are off-limits. This is despite there being no formal instruction from the robots.txt file about package index pages. All in all, it was an interesting bit of work and glad that I was able to learn about the robotstxt package so I can have it in my toolkit going forward.

Remember to Always Scrape Responsibly!

DISCLAIMER: I only have a basic understanding of how robots.txt files work based on allowing or disallowing specified paths. I believe in this case CRAN’s robots.txt broadly permitted scraping, but too narrowly disallowed just the DESCRIPTION files. Perhaps this goes back to an older time where those DESCRIPTION files really were the best place for people to start scraping so it made sense to disallow them. Or the reason could be something else entirely.

UPDATE: David G. Johnston messaged me and brought up an important point that the robots.txt file primarily is used to tell search engines whether or not to crawl certain portions of a domain. The “Terms of Use” might be a better place to begin looking for any limitations on the usage of the data for the type of project you are creating.

Most Starred R Packages on GitHub

Steven M. Mortimer — Mon, 18 Jun 2018 00:00:00 +0000

It seems like all the best R packages proudly use GitHub and have a README adorned with badges across the top. The recent Microsoft acquisition of GitHub got me wondering: What proportion of current R packages use GitHub? Or at least refer to it in the URL of the package description. Also, what is the relationship between the number of CRAN downloads and the number of stars on a repository? My curiosity got the best of me so I hastily wrote a script to pull the data. Click here to go straight to the full script and data included at the bottom of this post. I acknowledge there are more elegant ways to have coded this, but let’s press on.

Pulling List of Packages & their Details

CORRECTION: I orginally posted this article with code that scraped CRAN to get the list of packages and their metadata. This could violate CRAN Terms and Conditions and generally isn’t responsible. Thanks to Maëlle Salmon (@ma_salmon) I was reminded of how to scrape responsibly and that there is a function, tools::CRAN_package_db(), that returns all the metadata for the current packages on CRAN. I have written another blog post entitled Scraping Responsibly with R that details how to use the robotstxt package from rOpenSci to check a domain’s directive on bots. Without scraping CRAN here is how you get the package metadata:

pkgs <- tools::CRAN_package_db()
# remove duplicate MD5sum column since tibbles can't handle duplicate column names
pkgs <- pkgs[,unique(names(pkgs))]
pkgs %>%
  select(Package, Version, Author, BugReports, URL) %>%
  as.tibble()
#> Warning: `as.tibble()` is deprecated as of tibble 2.0.0.
#> Please use `as_tibble()` instead.
#> The signature and semantics have changed, see `?as_tibble`.
#> This warning is displayed once every 8 hours.
#> Call `lifecycle::last_warnings()` to see where this warning was generated.
#> # A tibble: 16,070 x 5
#>    Package  Version Author                  BugReports          URL             
#>    <chr>    <chr>   <chr>                   <chr>               <chr>           
#>  1 A3       1.0.0   "Scott Fortmann-Roe"    <NA>                <NA>            
#>  2 aaSEA    1.1.0   "Raja Sekhara Reddy D.… <NA>                <NA>            
#>  3 AATtools 0.0.1   "Sercan Kahveci [aut, … https://github.com… <NA>            
#>  4 ABACUS   1.0.0   "Mintu Nath [aut, cre]" <NA>                https://shiny.a…
#>  5 abbyyR   0.5.5   "Gaurav Sood [aut, cre… http://github.com/… http://github.c…
#>  6 abc      2.1     "Csillery Katalin [aut… <NA>                <NA>            
#>  7 abc.data 1.0     "Csillery Katalin [aut… <NA>                <NA>            
#>  8 ABC.RAP  0.9.0   "Abdulmonem Alsaleh [c… <NA>                <NA>            
#>  9 abcADM   1.0     "Zongjun Liu [aut],\n … <NA>                <NA>            
#> 10 ABCanal… 1.2.1   "Michael Thrun, Jorn L… <NA>                https://www.uni…
#> # … with 16,060 more rows

From there I looked at the package description fields "URL" and "BugReports" to see if either contained “github.com”. It turns out that 3,718 of the packages (29.3% of the total) referenced GitHub. After retrieving the package metadata I pinged the GitHub API to see if I could get the number of stars for the repository. Currently, GitHub allows 5,000 authenticated requests per hour (link), but out of all the packages only 3,718 referenced GitHub, so I could make all the requests at once. Here is the function I used to take a cleaned up version of the package’s URL then form a request to the GitHub API to get star counts:

# get the star count from a clean version of the package's URL
gh_star_count <- function(url){
  stars  <- tryCatch({
    this_url <- gsub("https://github.com/", "https://api.github.com/repos/", url)
    req <- GET(this_url, gtoken)
    stop_for_status(req)
    cont <- content(req)
    cont$stargazers_count  
  }, error = function(e){
    return(NA_integer_)
  })
  return(stars)
}

Analyzing the Data

Once I had all the package detail data, I found that R packages, on average, have 35.7 GitHub stars, but the median number of stars is only 6! ggplot2 has the most stars with 3,174. In my analysis I removed the xgboost, h2o, and feather packages which point to the repository of their implementations in many languages, not just R.

What I really found interesting was comparing CRAN downloads to GitHub repo stars. Using the cranlogs package I was able to get the total package downloads dating back to January 1, 2014. In contrast with the low star counts, the median downloads for R packages is 8,975. Combining stars and downloads data I found that the median R package has 903 downloads per star. Only 38.7% of packages had more than 10 stars, which shows how hard stars are to get even if you’ve written a great package. I’m not sure what proportion of R users frequently reference and contribute to GitHub, but it would be interesting to compare that with the high ratios of downloads to stars.

There are some real outliers in the data. For example, the Rcpp package, perhaps the most downloaded package of all-time, has 15.8M downloads and only 377 stars. Similarly, Hadley’s scales package has 9.4M downloads and only 115 stars. These support/helper packages just don’t get the same star love as the headliners like ggplot2, shiny, and dplyr.

Of course, I could not help but check out the stats for some of the most prolific package authors. After parsing out individuals with the ["aut", "cre"] roles I came to the not so surprising conclusion that Hadley has the most stars of any author with 12,408 stars. In contrast, Dirk Eddelbuettel had one of the lowest star-to-download ratios. For every ~38K downloads Dirk’s repositories will receive one star. Pay no attention to the man behind the curtain since his Rcpp package underpins a whole host of packages without all the GitHub fanfare. Here is a list of popular R package authors and their stats:

Author	Notable Packages	Downloads	Stars	Downloads Per Star
Hadley Wickham	ggplot2, dplyr, httr	113,160,314	12,408	9,119.9
Dirk Eddelbuettel	Rcpp, BH	28,433,586	745	38,165.9
Yihui Xie	knitr, rmarkdown, bookdown	42,472,860	6,315	6,725.7
Winston Chang	R6, shiny	17,161,005	4,027	4,261.5
Jennifer Bryan	readxl, gapminder, googlesheets	6,055,774	1,714	3,533.1
JJ Allaire	rstudioapi, reticulate, tensorflow	8,882,553	2,798	3,174.6
Jeroen Ooms	jsonlite, curl, openssl	25,907,868	1,483	17,469.9
Scott Chamberlain	geojsonio, taxize	1,770,664	2,528	700.4
Jim Hester	devtools, memoise, readr	22,867,071	4,332	5,278.6
Kirill Müller	tibble, DBI	36,159,009	1,077	33,573.8

I’m sure you could create mixed models to determine the unique download to star relationship for individuals. Also, you could use other package attributes to predict stars or downloads, but I’ll leave that to another curious soul. I will include tables below regarding the top 10 most downloaded, most starred, most and least downloaded per star.

Appendix

Top 10 Most Starred Packages

Name	Author	Downloads	Stars	Downloads Per Star
ggplot2	Hadley Wickham	13,001,703	3,174	4,096.3
shiny	Winston Chang	4,571,794	2,902	1,575.4
dplyr	Hadley Wickham	8,276,844	2,408	3,437.2
devtools	Jim Hester	5,536,730	1,645	3,365.8
knitr	Yihui Xie	7,131,564	1,581	4,510.8
data.table	Matt Dowle	6,005,795	1,457	4,122.0
plotly	Carson Sievert	1,195,880	1,255	952.9
rmarkdown	Yihui Xie	5,432,495	1,160	4,683.2
tensorflow	JJ Allaire	94,856	1,033	91.8
bookdown	Yihui Xie	126,586	1,009	125.5

Top 10 Most Downloaded Packages with Stars

Name	Author	Downloads	Stars	Downloads Per Star
Rcpp	Dirk Eddelbuettel	15,824,781	377	41,975.5
ggplot2	Hadley Wickham	13,001,703	3,174	4,096.3
stringr	Hadley Wickham	11,547,828	268	43,088.9
stringi	Marek Gagolewski	11,310,113	122	92,705.8
digest	Dirk Eddelbuettel with contributions by Antoine Lucas	11,233,244	42	267,458.2
plyr	Hadley Wickham	10,340,396	470	22,000.8
R6	Winston Chang	9,993,128	212	47,137.4
reshape2	Hadley Wickham	9,582,245	173	55,388.7
scales	Hadley Wickham	9,380,757	115	81,571.8
jsonlite	Jeroen Ooms	9,112,790	176	51,777.2

Top 10 Packages by Stars per Download (frequently starred)

Name	Author	Downloads	Stars	Downloads Per Star
r2d3	Javier Luraschi	416	235	1.77
workflowr	John Blischak	448	169	2.65
goodpractice	Hannah Frick	523	192	2.72
xtensor	Johan Mabille	2,057	664	3.10
scico	Thomas Lin Pedersen	185	59	3.14
shinytest	Winston Chang	418	113	3.70
furrr	Davis Vaughan	724	171	4.23
pkgdown	Hadley Wickham	1,589	332	4.79
rtika	Sasha Goodman	168	32	5.25
mindr	Peng Zhao	2,051	368	5.57

Bottom 10 Packages by Stars per Download (infrequently starred)

Name	Author	Downloads	Stars	Downloads Per Star
mime	Yihui Xie	7,398,765	12	616,563.8
pkgmaker	Renaud Gaujoux	1,228,173	2	614,086.5
rngtools	Renaud Gaujoux	1,224,959	2	612,479.5
magic	Robin K. S. Hankin	344,741	1	344,741.0
gsubfn	G. Grothendieck	675,056	2	337,528.0
bindrcpp	Kirill Müller	2,996,452	10	299,645.2
plogr	Kirill Müller	3,343,099	12	278,591.6
digest	Dirk Eddelbuettel with contributions by Antoine Lucas	11,233,244	42	267,458.2
munsell	Charlotte Wickham	7,778,712	31	250,926.2
proto	Hadley Wickham	2,593,246	11	235,749.6

Full Script

Below and available via gist with data at: https://gist.github.com/StevenMMortimer/1b4b626d3d91240a77f969ae04b37114

# load packages & custom functions ---------------------------------------------

library(tidyverse)
library(httr)
library(cranlogs)
library(ggrepel)
library(scales)
library(knitr)
library(stringr)

gh_from_url <- function(x){
  if(!grepl(',', x)){
    x <- strsplit(x, " ")[[1]]
    x <- trimws(x[min(which(grepl(pattern='http://github.com|https://github.com|http://www.github.com', x, ignore.case=TRUE)))])    
  } else {
    x <- strsplit(x, ",")[[1]]
    x <- trimws(x[min(which(grepl(pattern='http://github.com|https://github.com|http://www.github.com', x, ignore.case=TRUE)))])    
  }
  x <- gsub("http://", "https://", tolower(x))
  x <- gsub("www\\.github\\.com", "github.com", x)
  x <- gsub("/$", "", x)
  x <- gsub("^github.com", "https://github.com", x)
  x <- gsub("/issues", "", x)
  x <- gsub("\\.git", "", x)
  return(x)
}

aut_maintainer_from_details <- function(x){
  x <- gsub("'|\"", "", x)
  if(grepl(',', x)){
    x <- strsplit(x, "\\],")[[1]]
    aut_cre_ind <- grepl(pattern='\\[aut, cre|\\[cre, aut|\\[cre', x, ignore.case=TRUE)
    if(any(aut_cre_ind)){
      x <- x[min(which(aut_cre_ind))]
      x <- gsub("\\[aut, cre|\\[cre, aut|\\[cre", "", x)
    }
    x <- strsplit(x, ",")[[1]][1]
    x <- trimws(gsub("\\]", "", x))
    x <- trimws(gsub(" \\[aut", "", x))
  }
  return(x)
}

gh_star_count <- function(url){
  stars  <- tryCatch({
    this_url <- gsub("https://github.com/", "https://api.github.com/repos/", url)
    req <- GET(this_url, gtoken)
    stop_for_status(req)
    cont <- content(req)
    cont$stargazers_count  
  }, error = function(e){
    return(NA_integer_)
  })
  return(stars)
}

# authenticate to github -------------------------------------------------------
# use Hadley's key and secret
myapp <- oauth_app("github",
                   key = "56b637a5baffac62cad9",
                   secret = "8e107541ae1791259e9987d544ca568633da2ebf")
github_token <- oauth2.0_token(oauth_endpoints("github"), myapp)
gtoken <- config(token = github_token)

# pull list of packages --------------------------------------------------------

# get list of currently available packages on CRAN
pkgs <- tools::CRAN_package_db()

# remove duplicate MD5sum column since tibbles can't handle duplicate column names
pkgs <- pkgs[,unique(names(pkgs))]

# filter out lines any duplicates
pkgs <- pkgs %>% 
  rename(Name = Package) %>%
  distinct(Name, .keep_all = TRUE)

# get details for each package -------------------------------------------------

all_pkg_details <- NULL

# old fashioned looping! 
# WARNING: This takes awhile to complete
for(i in 1:nrow(pkgs)){
  
  if(i %% 100 == 0){
    message(sprintf("Processing package #%s out of %s", i, nrow(pkgs)))
  }
  
  this_url <- pkgs[i,]$URL
  on_github <- FALSE
  this_github_url <- NA_character_
  gh_stars <- NA_integer_
  if(!is.null(this_url)){
    on_github <- grepl('http://github.com|https://github.com|http://www.github.com', this_url)
    if(on_github){
      this_github_url <- gh_from_url(this_url)
      gh_stars <- gh_star_count(this_github_url)
    } else {
      # check the BugReports URL as a backup (e.g. shiny package references GitHub this way)
      issues_on_github <- grepl('http://github.com|https://github.com|http://www.github.com', pkgs[i,]$BugReports)
      if(length(issues_on_github) == 0 || !issues_on_github){
        this_github_url <- NA_character_
      } else {
        this_github_url <- gh_from_url(pkgs[i,]$BugReports)
        gh_stars <- gh_star_count(this_github_url)  
        on_github <- TRUE
      }
    }
  } else {
    this_url <- NA_character_
  }
  
  downloads <- cran_downloads(pkgs[i,]$Name, from = "2014-01-01", to = "2018-06-15")
  
  all_pkg_details <- rbind(all_pkg_details, 
                           tibble(name = pkgs[i,]$Name, 
                                  published = pkgs[i,]$Published,
                                  author = aut_maintainer_from_details(pkgs[i,]$Author),
                                  url = this_url,
                                  github_ind = on_github, 
                                  github_url = this_github_url,
                                  downloads = sum(downloads$count),
                                  stars = gh_stars
                           )
  )
}

# basic summary stats ----------------------------------------------------------

# remove observations where the GitHub URL refers to a repository that 
# is not specific to R and therefore might have an inflated star count
all_pkg_details_clean <- all_pkg_details %>% 
  filter(!(name %in% c('xgboost', 'h2o', 'feather'))) %>% 
  mutate(downloads_per_star = downloads / stars, 
         downloads_per_star = ifelse(!is.finite(downloads_per_star), NA_real_, downloads_per_star))

# proportion of all packages listing github
sum(all_pkg_details$github_ind)
mean(all_pkg_details$github_ind)
# proportion of packages with stars
mean(!is.na(all_pkg_details$stars))

# typical number of stars per package
mean(all_pkg_details_clean$stars, na.rm=TRUE)
median(all_pkg_details_clean$stars, na.rm=TRUE)
max(all_pkg_details_clean$stars, na.rm=TRUE)

# typical number of downloads per package
mean(all_pkg_details_clean$downloads, na.rm=TRUE)
median(all_pkg_details_clean$downloads, na.rm=TRUE)

# percent of packages over 10 stars
mean(all_pkg_details_clean$stars > 10, na.rm=TRUE)

mean(all_pkg_details_clean$downloads_per_star, na.rm=TRUE)
median(all_pkg_details_clean$downloads_per_star, na.rm=TRUE)


# stars histogram --------------------------------------------------------------
ggplot(data=all_pkg_details_clean, mapping=aes(stars)) +
  geom_histogram(aes(fill=..count..), bins=60) +
  scale_x_continuous(trans = "log1p", breaks=c(0,1,2,3,10,100,1000,3000)) + 
  labs(x = "Stars", 
       y = "Count", 
       fill = "Count", 
       caption = "Source: api.github.com as of 6/16/18") + 
  ggtitle("Distribution of GitHub Stars on R Packages") + 
  theme_bw() + 
  theme(panel.grid.minor = element_blank(), 
        plot.caption=element_text(hjust = 0))

# stars to downloads scatterplot -----------------------------------------------
plot_dat <- all_pkg_details_clean
idx_label <- which(with(plot_dat, downloads > 10000000 | stars > 1000))
plot_dat$name2 <- plot_dat$name
plot_dat$name <- ""
plot_dat$name[idx_label] <- plot_dat$name2[idx_label]

ggplot(data=plot_dat, aes(stars, downloads, label = name)) +
  geom_point(color = ifelse(plot_dat$name == "", "grey50", "red")) +
  geom_text_repel(box.padding = .5) + 
  scale_y_continuous(labels = comma) + 
  scale_x_continuous(labels = comma) + 
  labs(x = "GitHub Stars", 
       y = "CRAN Downloads", 
       caption = "Sources:\napi.github.com as of 6/16/18\ncranlogs as of 1/1/14 - 6/15/18") + 
  ggtitle("Relationship Between CRAN Downloads and GitHub Stars") + 
  theme_bw() + 
  theme(plot.caption=element_text(hjust = 0))

# author stats -----------------------------------------------------------------

# summary by author
authors_detail <- all_pkg_details_clean %>% 
  group_by(author) %>% 
  summarize(downloads = sum(downloads, na.rm=TRUE), 
            stars = sum(stars, na.rm=TRUE)) %>% 
  mutate(downloads_per_star = downloads / stars, 
         downloads_per_star = ifelse(!is.finite(downloads_per_star), NA_real_, downloads_per_star)) %>% 
  arrange(desc(downloads))

# popular authors
pop_authors <- tibble(author = c('Hadley Wickham', 
                                 'Dirk Eddelbuettel', 
                                 'Yihui Xie', 
                                 'Winston Chang', 
                                 'Jennifer Bryan', 
                                 'JJ Allaire',
                                 'Jeroen Ooms', 
                                 'Scott Chamberlain', 
                                 'Jim Hester', 
                                 'Kirill Müller'), 
                      notable_packages = c('ggplot2, dplyr, httr',
                                           'Rcpp, BH', 
                                           'knitr, rmarkdown, bookdown', 
                                           'R6, shiny', 
                                           'readxl, gapminder, googlesheets', 
                                           'rstudioapi, reticulate, tensorflow', 
                                           'jsonlite, curl, openssl', 
                                           'geojsonio, taxize', 
                                           'devtools, memoise, readr', 
                                           'tibble, DBI')
)
author_stats <- pop_authors %>% 
  inner_join(., authors_detail, by='author') %>%
  select(author, notable_packages, downloads, stars, downloads_per_star) %>% 
  mutate(downloads_per_star = round(downloads_per_star, 1)) %>%
  rename_all(. %>% gsub("_", " ", .) %>% str_to_title)

# single author 
#all_pkg_details_clean %>% filter(author == 'Dirk Eddelbuettel') %>% arrange(desc(downloads))  

# top 10 lists -----------------------------------------------------------------

# Top 10 Most Starred Packages

top_starred <- all_pkg_details_clean %>% 
  select(name, author, downloads, stars, downloads_per_star) %>%
  arrange(desc(stars)) %>%
  slice(1:10) %>% 
  mutate(downloads_per_star = round(downloads_per_star, 1)) %>%
  rename_all(. %>% gsub("_", " ", .) %>% str_to_title)

# Top 10 Most Downloaded Packages with stars

top_downloaded <- all_pkg_details_clean %>% 
  filter(!is.na(stars)) %>%
  select(name, author, downloads, stars, downloads_per_star) %>%
  arrange(desc(downloads)) %>%
  slice(1:10) %>% 
  mutate(downloads_per_star = round(downloads_per_star, 1)) %>%
  rename_all(. %>% gsub("_", " ", .) %>% str_to_title)

# Bottom 10 Packages by Downloads per Star (frequently starred) 

frequently_starred <- all_pkg_details_clean %>% 
  filter(downloads > 100) %>% 
  select(name, author, downloads, stars, downloads_per_star) %>%
  arrange(downloads_per_star) %>%
  slice(1:10) %>% 
  mutate(downloads_per_star = round(downloads_per_star, 2)) %>%
  rename_all(. %>% gsub("_", " ", .) %>% str_to_title)

# Top 10 Packages by Downloads per Star (infrequently starred) 

infrequently_starred <- all_pkg_details_clean %>% 
  select(name, author, downloads, stars, downloads_per_star) %>%
  arrange(desc(downloads_per_star)) %>%
  slice(1:10) %>% 
  rename_all(. %>% gsub("_", " ", .) %>% str_to_title)

Analyzing Customer Data from Square

Steven M. Mortimer — Fri, 18 May 2018 00:00:00 +0000

The Square Data Model

Whether you own your own business or consult for a business using Square to capture payment data, Square can offer some amazing opportunities to gain insights by leveraging their Connect v1 & v2 APIs. The Square data backend operates much like a CRM system that holds information about transactions that customers make when purchasing items offered at a location. Naturally, there are API endpoints for each of the object types italicized above (Locations, Customers, Transactions, and Items) and many more endpoints for administrative tasks (employees, roles, timecards, refunds, etc.).

Authenticating

While working with Square data I decided to develop an R package squareupr that makes it easier to retrieve Square data from R so that you can focus on the analysis. After installing the squareupr package you must authenticate by supplying a personal access token (PAT) or using an OAuth 2.0 flow. You can find your PAT by logging into the Square Dashboard -> Apps -> My Apps. Create an app or click “Manage App” if you’ve already created one and there you should see your personal access token:

# The squareupr package is not yet available on CRAN so you must install from GitHub
# install.packages("devtools")
devtools::install_github("StevenMMortimer/squareupr")

library(tidyverse)
library(squareupr)

# authenticate using your Personal Access Token (PAT)
sq_auth(personal_access_token = "sq-Th1s1sMyPers0nalAcessT0ken")

The package also offers OAuth 2.0 authentication. More information is available <a target="_blank" href-“https://github.com/StevenMMortimer/squareupr#authenticate”>here.

Pulling Transaction Data

As mentioned above there are endpoints for every major type of data stored by Square. The API documentation does a very good job at laying out how requests should be made to each of those endpoints. One thing to note is that it is important to first pull down the location details for your business because the location is often required when searching for things like transactions and items.

our_locations <- sq_list_locations()
our_locations$name <- "{HIDDEN}"
our_locations %>% select(id, name, address, timezone, 
                        capabilities, status, created_at)
#> # A tibble: 5 x 7
#>   id            name     address  timezone  capabilities status created_at
#>   <chr>         <chr>    <list>   <chr>     <list>       <chr>  <chr>     
#> 1 46FYN9N9RQS54 {HIDDEN} <list [… America/… <list [1]>   ACTIVE 2017-04-2…
#> 2 DRDCJ2X8E2PMV {HIDDEN} <list [… America/… <list [1]>   ACTIVE 2016-09-2…
#> 3 8T1TYXE840S00 {HIDDEN} <list [… America/… <list [1]>   ACTIVE 2016-09-2…
#> 4 1AWPRVVVFWGQF {HIDDEN} <list [… America/… <list [1]>   ACTIVE 2017-04-1…
#> 5 50X1GNAWEC8V0 {HIDDEN} <list [… America/… <list [1]>   ACTIVE 2017-03-0…

Now that you have the location ids if you would like to pull all of the transactions during a given timeframe, you would use the function sq_list_transactions().

# list all transactions for our 2nd location on May 11, 2018
# by default, if a date is provided with no time, then the time component is set to midnight
our_transactions <- sq_list_transactions(location = our_locations$id[2], 
                                         begin_time = as.Date('2018-05-11'), 
                                         end_time = as.Date('2018-05-12'))
our_transactions
#> # A tibble: 245 x 6
#>    id          location_id  created_at    tenders product client_id       
#>    <chr>       <chr>        <chr>         <list>  <chr>   <chr>           
#>  1 bUjFGVjBvN… DRDCJ2X8E2P… 2018-05-12T0… <list … REGIST… D5528FBA-E5DE-4…
#>  2 5PZP31N5Zs… DRDCJ2X8E2P… 2018-05-11T2… <list … REGIST… A3A1FF51-325A-4…
#>  3 BTrGydD6he… DRDCJ2X8E2P… 2018-05-11T2… <list … REGIST… 2B3D32EB-8E58-4…
#>  4 XsqOAHl68z… DRDCJ2X8E2P… 2018-05-11T2… <list … REGIST… C50AF3D7-BE32-4…
#>  5 vmLRzrwByS… DRDCJ2X8E2P… 2018-05-11T2… <list … REGIST… 52E40E1B-2333-4…
#>  6 pTbzQApZW7… DRDCJ2X8E2P… 2018-05-11T2… <list … REGIST… 962766FF-1436-4…
#>  7 lnE20zklpP… DRDCJ2X8E2P… 2018-05-11T2… <list … REGIST… A02191CC-9AC9-4…
#>  8 DSumrqQW0L… DRDCJ2X8E2P… 2018-05-11T2… <list … REGIST… 1135FF4F-9B89-4…
#>  9 tPwFXetIwe… DRDCJ2X8E2P… 2018-05-11T2… <list … REGIST… 0D95E79D-B44C-4…
#> 10 bqUuFrzH71… DRDCJ2X8E2P… 2018-05-11T2… <list … REGIST… 48FD6A49-80A9-4…
#> # ... with 235 more rows

At first glance there does not appear to be very much detail on the transaction record. However, the tender field represents a method of payment used in a Square transaction so it contains information regarding the amount of money paid in total, in Square fees, and tip. The tender field even contains information regarding the customer_id and credit card information. In the following I will loop through the transactions in April 2018 and determine the total spend for each customer in the dataset. Note: The call to sq_list_transactions may take a couple minutes to complete if you are pulling thousands of transactions.

april_transactions <- sq_list_transactions(location = our_locations$id[2], 
                                           begin_time = as.Date('2018-04-01'), 
                                           end_time = as.Date('2018-05-01'))

In order to extract the customer ID and money spent I create a function that checks for the tender object on the transaction and if it exists tries to extract the data into a tbl_df. When I supply this function as map_df(extract_cust_info_func) %>% I get the data from each transaction stacked into a single tbl_df that’s ready to analyze.

# create a function that will extract out just the customer id and money spent
extract_cust_info_func <- function(x){
  if(!is.null(x$tender)){
    tibble(customer_id = sq_null_to_na(x$tender[[1]]$customer_id),
           money_spent = sq_null_to_na(x$tender[[1]]$amount_money$amount))  
  } else {
    tibble(customer_id = NA_character_, 
           money_spent = NA_integer_)
  }
}

april_customer_spend <- april_transactions %>% 
  transpose() %>%
  # pull out just the information we want from each transaction
  map_df(extract_cust_info_func) %>%
  group_by(customer_id) %>%
  summarize(total_spend = sum(money_spent, na.rm=TRUE))

april_customer_spend
#> # A tibble: 208 x 2
#>    customer_id                total_spend
#>    <chr>                            <int>
#>  1 064HFDQG0N52AHDBSG00C1BAC8        1700
#>  2 07VNWH1V4S6W4W2EJ4AN7SEJNR           0
#>  3 08M453QNJ97BCT97SM09TN7QK4           0
#>  4 08XE43X8FS0MPX8P2W4N0DEQY0         350
#>  5 0CZ78CVRW12V7AZET8S3S82GGW         675
#>  6 0G1J81148H42GGMTMQKRWSJHGC           0
#>  7 0V1Y1BX23WYRK889ERVBE2T0KM         900
#>  8 13HHFBFZTD33RX0RSJNAZQKV5M           0
#>  9 1BFCHB9MK91GQ39HTD7QK6R7ZR           0
#> 10 1DR0AK5GKX57H9ER9SG1JF01P0        2900
#> # ... with 198 more rows

The amounts in april_customer_spend may seem large, but the Square APIs return money as integers that represent whole cents. If you divide by 100, then you will have the money amounts in dollars.

Spend by Customer Group

Square has this concept of “groups” that customers belong to. These groups can be fashioned to do marketing campaigns complete with email blasts. In our analysis let’s further determine which groups these customers belong to. The Square API has an endpoint to retrieve one customer at a time; however, with large lists you may get subjected to rate limiting. Rate limiting is errors on your requests because too many are coming from the same application or access token. I would recommend pulling down the entire list of customers with sq_list_customers() and then matching them up in R.

cust_groups <- sq_list_customers() %>%
  select(id, groups) %>%
  sq_extract_cust_groups() %>%
  # filter to only the groups designated automatically by Square
  filter(grepl("^CQ689YH4KCJMY", groups.id))

cust_groups
#> # A tibble: 13,444 x 3
#>    id                         groups.id                 groups.name       
#>    <chr>                      <chr>                     <chr>             
#>  1 M1RBDFRK7S1Q1EP6EZFJFV3CBW CQ689YH4KCJMY.LOYALTY_ALL Loyalty Participa…
#>  2 58MK9F1HQ5447D1QZDX60NHTP4 CQ689YH4KCJMY.CHURN_RISK  Lapsed            
#>  3 58MK9F1HQ5447D1QZDX60NHTP4 CQ689YH4KCJMY.REACHABLE   Reachable         
#>  4 MBSJA4QV4WX6N2XV8WV9VJJTG8 CQ689YH4KCJMY.LOYALTY_ALL Loyalty Participa…
#>  5 MBSJA4QV4WX6N2XV8WV9VJJTG8 CQ689YH4KCJMY.REACHABLE   Reachable         
#>  6 ZCBZJ234217KTV812WX4DP2404 CQ689YH4KCJMY.REACHABLE   Reachable         
#>  7 FKEMR8KZCN3BH98RV78PKHKQ1R CQ689YH4KCJMY.LOYALTY_ALL Loyalty Participa…
#>  8 FKEMR8KZCN3BH98RV78PKHKQ1R CQ689YH4KCJMY.LOYAL       Regulars          
#>  9 78VMJPJNK959AHH0ZQPXDXEG3C CQ689YH4KCJMY.LOYALTY_ALL Loyalty Participa…
#> 10 QASM1G54VX0QN2S15YS6KHEFCC CQ689YH4KCJMY.LOYAL       Regulars          
#> # ... with 13,434 more rows

Now that we know the customer memberships, let’s join back with the transaction data to determine the average total spend across the different membership groups.

cust_groups %>% 
  # bring in the spend data
  inner_join(., april_customer_spend, by=c('id'='customer_id')) %>%
  # group by the customer groups to find average spend per group
  group_by(groups.name) %>%
  summarize(avg_spend = mean(total_spend / 100, na.rm=TRUE))
#> # A tibble: 6 x 2
#>   groups.name              avg_spend
#>   <chr>                        <dbl>
#> 1 Cards on File                569. 
#> 2 Lapsed                        20.3
#> 3 Loyalty Participants          13.5
#> 4 Nearing a Loyalty Reward      15.3
#> 5 Reachable                     51.0
#> 6 Regulars                      27.4

Issues with the APIs

What is great about this analysis is that we can use the Square APIs to quickly and reliably pull down transaction data, match it to customer information and see how certain customer groupings or campaigns are performing. However, I did find some quirks and drawbacks that I wish the Square team would consider:

Cannot Request Specific Fields

First, it would be nice to implement, as part of the query parameters, the ability to only return certain fields (e.g. the id and groups fields from the Customer object). This would help for two reasons: 1) It would improve the speed since only the required data would be passed across and 2) For analysts who do not want to deal with personally identifiable information (even in RAM) you would not be forced to pull information like phone number, email, names, and even credit card information.

Cannot Update Customer Groups Programmatically

Second, the API will not allow you to update the customer groups programmatically. This means that you are stuck using the dashboard to create and assign customers to the groups you want to analyze. It is a drag when you want to create a reproducible research workflow to assign customer groups but the API hinders your ability to do so.

Customer ID Not On Transactions

Finally, it appears that the customer ID associated to a transaction is not reliably captured on every transaction’s tender object. This means that you cannot get a complete picture of all the transactions. I believe part of this is an issue with the merchant capturing the customer information at the point of sale, but also partly Square’s fault since they seem to be able to piece everything together in their dashboard. However, this is not the case with the APIs. Overall, the Square APIs are a rich data resource for helping run a business and they should only get better with time as development progresses.

Tips for Making Professional Shiny Apps with R

Steven M. Mortimer — Tue, 09 Jan 2018 00:00:00 +0000

If you’ve built a Shiny app with R you’ll know how exciting it feels to get your first app up and running. Shiny is an amazing tool for R users, but how do you take it to the next level? This post with outline five easy ways to make Shiny apps more professional.

Add a Progress Wheel

I’ll start with the progress wheel because it is the easiest and most impactful change to make. Oftentimes, Shiny apps involve calculations that take more than one second to execute. It can be a frustrating experience for the user to interact with an application and wait without knowing what is happening. Is the application busy doing some calculation? Did the user select the correct button? Is the server down? You have likely been in this situation yourself as a user of the web. There are a few different ways to implement a progress indicator, but I have found the simplest and most effective solution already in the shinysky package. The busyIndicator() function allows you to specify a custom message and wait time before the progress wheel appears.

busyIndicator(text = "Calculation in progress ... ", wait = 0)

Create a www Folder

Most web applications contain extra resources or libraries to enhance their appearance and functionality. Shiny apps are no different. If you would like to include images, styling, or even JavaScript, then all of these files should be contained inside a single folder entitled www that is located in the same directory as the ui.R and server.R files. The www folder is the main location for all web resources in your application. If you would like to enforce a global standard of styling, then create a .css file containing the CSS stylesheet language for formatting. If you would like to add images or GIFs to your Shiny app, then place them in the www folder as well. Shiny supports most image file formats such as .jpg, .png, and .gif. If you would like to add JavaScript logic, then create a .js file containing any functions or variables that you would like to execute. Again, place this file inside the www folder. Note that you must explicitly reference these files inside the <head> tag of your app. The best way to add to your app is by using tags$head() and tags$link() like this:

tags$head(
  tags$link(rel = "stylesheet", type = "text/css", 
            href = "style.css")
)

You can even give your app a bookmark icon using the link tag instead of the head tag!

tags$link(rel = "shortcut icon", type = "image/x-icon", 
          href = "my-shortcut-icon.ico")

Style with CSS

If you are deploying your Shiny apps in a professional setting, it is a best practice to format and style your apps using rules defined by the CSS stylesheet language. This will reduce the amount of time and effort to style new applications and provide a clean and consistent experience so that users quickly become familiar with how elements of your apps function based on a consistent appearance. Using CSS in Shiny apps is no different from styling a traditional web page, so any resource teaching CSS will help, such as, the W3Schools CSS Tutorial available at http://www.w3schools.com/css/. A Shiny CSS guide is also available at http://shiny.rstudio.com/articles/css.html

Use a global.R file

It is good practice when coding software to place commonly used pieces of logic in a central location. You can put this concept into practice by creating a global.R file for your Shiny app. The global.R file is not a requirement for a Shiny app, but it will reduce redundant code and increase the app’s speed by re-using logic and variables from a global scope among concurrent app users. The global.R file should be located in the same directory as the ui.R and server.R files. Whenever a Shiny app is first used, the global.R file makes its logic available to the ui.R and server.R files. This speeds up your app because it is only runs once (when the app is first launched). Typically, global.R is a good place for loading libraries, sourcing R functions or variables needed by your app, and setting options like stringsAsFactors = FALSE.

Encourage Data Downloading

A big part of Shiny’s value proposition in the enterprise is promoting data democracy and the free sharing of information for use to further benefit the company. Shiny app users may prefer other tools, such as Excel or Tableau or Python. Empower your users and allow them the flexibility to intergrate into their preferred tool. The best way to facilitate that is to allow downloads of the data. The best package by far for displaying data and allowing for downloads is the datatable package. The following code snippet creates a row for your download button and then the datatable. I recommend pulling the button to the right of the screen using float: right; in your CSS so the button is naturally placed at the top right of your table and ready for the user to click.

fluidRow(
  column(12, downloadButton("downloadDataFromTable", "Download Table Data"))
)
fluidRow(
  column(12, DT::dataTableOutput("campaign_table", width = "100%"))
)

Building a Corporate R Package

Steven M. Mortimer — Fri, 03 Nov 2017 00:00:00 +0000

Analytical Functions

Creating functions is a programming best practice. They are easier to monitor, test, and teach to others. If your team performs routine calculations, establish a standard by adding functions to the shared library and encourage everyone to use those functions. If your company works a lot with customer phone numbers, then write a function to standardize the formatting. See the example below for how to create and document these functions.

#' Format A Column of Phone Numbers
#' 
#' This function takes a dataframe and formats a column
#' to match a 10-digit phone number (XXX) XXX-XXXX
#'
#' @usage cleanPhoneNumbers(dat, phoneColumnName, returnExtColumn=FALSE)
#' @keywords cleaning transformation etl salesforce
#' @param dat a data.frame containing a column for formatting
#' @param phoneColumnName a character string defining the target data.frame column to
#' format as (XXX) XXX-XXXX
#' @param returnExtColumn a logical specifying whether to return digits seemingly formatted as
#' an extension to the main phone number
#' @return A \code{data.frame} with a clean phone number column in the format (XXX) XXX-XXXX
#' @examples
#' \dontrun{
#' my_data <- data.frame(Phone=c('2349325339 x7779', '8888-888-888', 
#'                               '2343-234-2341', '(832) 234-3494'))
#' my_data %>% cleanPhoneNumbers('Phone')
#' }
#' @export
cleanPhoneNumbers <- function(dat, phoneColumnName, returnExtColumn=FALSE){
  
  if(!(phoneColumnName %in% names(dat))){
    
    message(paste(phoneColumnName, 'was not in the dataset, no cleaning performed.'))
    return(dat)
    
  } else {
    
    original_values <- dat[,phoneColumnName]
    
    ext_part <- gsub('(.*)(extension|ext|ex|x)(.*)', '\\3', dat[,phoneColumnName])
    ext_part <- ifelse(ext_part == dat[,phoneColumnName], '', ext_part)
    ext_part <- ifelse(((is.na(ext_part)) | 
                          (ext_part == '')), '', 
                       gsub("[^\\d]+", "", ext_part, perl=TRUE))
    
    dat[,phoneColumnName] <- gsub('(.*)(extension|ext|ex|x)(.*)', '\\1', 
                                  dat[,phoneColumnName])
    
    dat[,phoneColumnName] <- ifelse(((is.na(dat[,phoneColumnName])) | 
                                        (dat[,phoneColumnName] == '')), '', 
                                     gsub("[^\\d]+", "", dat[,phoneColumnName], perl=TRUE))
    
    # convert 11 digit numbers starting with 1 to 10 because we assume US country code
    dat[,phoneColumnName] <- ifelse(((nchar(dat[,phoneColumnName]) == 11) & 
                                       (grepl('^1', dat[,phoneColumnName]))), 
                                     substring(dat[,phoneColumnName], 2), 
                                    dat[,phoneColumnName])
    
    # if not 10 digits, then set to ''
    dat[,phoneColumnName] <- ifelse((nchar(dat[,phoneColumnName]) == 10), 
                                    dat[,phoneColumnName], '')
    
    # if weird then set to ''
    wierd_vals <- c('0000000000', '1111111111', '1234567789', '1231231234', 
                    '1234567890', '8888888888', '9999999999')
    
    dat[,phoneColumnName] <- ifelse((dat[,phoneColumnName] %in% wierd_vals), '', 
                                    dat[,phoneColumnName])
    
    dat[,phoneColumnName] <- gsub('([0-9]{3})([0-9]{3})([0-9]{4})', '(\\1) \\2-\\3', 
                                  dat[,phoneColumnName])
    
    if(returnExtColumn){
      dat[,'phone_ext_parsed_out'] <- ext_part
      
      dat[,'phone_ext_parsed_out'] <- ifelse(dat[,phoneColumnName] == '', '', 
                                             dat[,'phone_ext_parsed_out'])
      
      dat[,'phone_ext_parsed_out'] <- strtrim(dat[,'phone_ext_parsed_out'], 5)
      
      pos <- which(names(dat) == phoneColumnName)
      pos2 <- which(names(dat) == 'phone_ext_parsed_out')
      rest <- seq(pos+1, ncol(dat))
      rest <- rest[rest!=pos2]
      
      dat <- dat[,c(1:pos, pos2, rest)]
    }
    
    return(dat)
  }
}

I strongly encourage using the Google Style Guide or Hadley’s Style Guide. Between the two of these guides I believe the two most important rules are:

Use snake case
Name your functions with verbs and variables with nouns

Company Constants

Again, if you’re looking to enforce standards, it is a great practice to include company-wide constants in your R package. Create an R file called constants.R and define these constants as follows:

#' @export
default_price <- 9.99 # code that defines a default price constant value
lockBinding("default_price", environment())

#' @export
min_price <- 4.50 # also define a minimum price of item
lockBinding("min_price", environment())

#' @export
price_cutoffs <- c(0, 5, 10, 15, Inf) # define a vector to consistently bin data
lockBinding("price_cutoffs", environment())

Templating/Formatting

A little bit of formatting goes a long way if your team routinely presents figures internally and/or externally. Consider adapting a ggplot2 theme to your company’s colors, fonts, and other styling. Here are two links to get you started in making custom theme:

Roger Peng’s Mastering Software Development in R
https://bookdown.org/rdpeng/RProgDA/building-a-new-theme.html
Joey Stanley’s Blog - Custom Themes in ggplot2
http://joeystanley.com/blog/custom-themes-in-ggplot2

Also, you might find that a majority of stakeholders in your company prefer receiving data in Excel. You can export data from R to Excel with bolded, proper case column names, format columns as percentages and dollar signs so that your stakeholders are happy getting data from you. I highly recommend working with the openxlsx package, which does not require Java. Below is a snippet of code that will transform column names, set column widths based on the data, and format them based on some key words in the column names like formatting columns ending in “pct” as a percentage. All of this before exporting to an Excel file so that your stakeholders are happy.

#' Add Data and Format it in an Excel Workbook
#'
#' This function is a convenience function for standardizing one-off reports
#' in a certain Excel format that is pretty and well-organized for consumers
#'
#' @usage add_data_to_excel(wb, sheet, dat, startRow=1, startCol=1)
#' @concept writes data to Excel workbook in a pretty format
#' @keywords excel microsoft data export formatting
#' @seealso \code{\link[openxlsx]{createWorkbook}}
#' @param wb a Workbook object containing a worksheet. 
#' See \code{\link[openxlsx]{createWorkbook}}.
#' @param sheet the worksheet to write to. Can be the worksheet index or name.
#' @param dat data.frame to be written in the sheet
#' @param startRow a vector specifiying the starting column to write to
#' @param startCol a vector specifiying the starting row to write to
#' @return a logical indicating the success of adding the data to 
#' the workbook and formatting
#' @examples 
#' \dontrun{
#' my_wb <- createWorkbook()
#' addWorksheet(my_wb, "Iris")
#' # returns invisble TRUE
#' add_data_to_excel(my_wb, "Iris", iris)
#' saveWorkbook(my_wb, file='~/iris.xlsx', overwrite = TRUE)
#' }
#' @export
add_data_to_excel <- function(wb, sheet, dat, startCol=1, startRow=1){
  
  # convert column names to more human readable format (with spaces and proper case)
  names(dat) <- gsub("_", " ", tolower(names(dat)))
  names(dat) <- str_to_title(names(dat))
  
  writeData(wb=wb, sheet=sheet, x=dat, startCol=startCol, startRow=startRow, 
            headerStyle=createStyle(textDecoration='bold', 
                                    wrapText=TRUE, 
                                    halign='center', 
                                    valign='center'))
  for(i in 1:ncol(dat)){
    
    setColWidths(wb, 
                 sheet, 
                 cols = i + (startCol - 1), 
                 widths = pmax(12, 
                               pmin(
                                 max(nchar(sapply(dat[,i], 
                                                  as.character)), na.rm=TRUE), 
                                 40)))
    
    if(grepl('Percent|Pct|%', names(dat)[i])){
      
      addStyle(wb,
               sheet, 
               style = createStyle(numFmt='0.0%'), 
               rows = (startRow + 1):(nrow(dat) + 100), 
               cols = i + (startCol - 1), 
               gridExpand = TRUE)
      
    } else if(grepl('Dollars', names(dat)[i])){
      
      addStyle(wb,
               sheet, 
               style = createStyle(numFmt='$#,##0'),
               rows = (startRow + 1):(nrow(dat) + 100), 
               cols = i + (startCol - 1), 
               gridExpand = TRUE)
      
    } else {
      # dont add any formatting
    }
  }
  invisible(return(TRUE))
}

Data Source Connections

On the job you’re probably accessing the same data sources over and over. Help out yourself and fellow teammates by standardizing how to connect and pull data. This could be database connections, API connections, or other server connections. Below are two examples, one for connecting to MS SQL Server database and one for connecting to the Salesforce API.

DB Connection

#' Establish a connection to Our DB and execute a query
#' 
#' This function connects to the DBMS using the shared_user, executes query, 
#' closes the connection, and returns the recordset if possible.
#'
#' @usage queryOurDB(query, db_environment=c('prod','dev'), username=NULL, password=NULL)
#' @importFrom RODBC odbcDriverConnect odbcConnect sqlQuery odbcClose
#' @param query a character string defining a query to be executed against the DB
#' @param db_environment a character string either 'prod' or 'dev' upon 
#' which environment to execute against
#' @param username a character string defining your database username credential
#' @param password a character string defining your database password credential
#' @return A \code{data.frame} object of the query's resultset
#' @seealso
#'  \code{\link[RODBC]{odbcConnect}},
#'  \code{\link[DBI]{sqlQuery}}
#' @examples
#' \dontrun{
#' # This query pulls the top 10 records from CustomerTable
#' my_query <- "SELECT TOP 10 * FROM DB.dbo.CustomerTable"
#' query_result <- queryOurDB(my_query)
#' }
#' @export
queryOurDB <- function(query, db_environment=c('prod', 'dev'), 
                      username=NULL, password=NULL){
  
  # try to prevent anything really bad resulting from dropping tables by accident
  stopifnot(!grepl("drop table", query, ignore.case=TRUE))
  
  # set the username and password
  if(is.null(username) | is.null(password)){
    username <- "shared_user"
    password <- "shared_password"
  }
  
  # determine which database to query
  which_db <- match.arg(db_environment)
  if(which_db=='prod'){
    host <- "10.245.96.30" #Production DB
  } else {
    host <- "10.245.96.40" #Development DB 
  }
  
  # run query
  query_result <- tryCatch({
    con <- odbcDriverConnect(
      sprintf('driver={SQL Server};server=%s,1433;database=DUMMY;uid=%s;pwd=%s',
              host, username, password)
      )
    
    message("running query")
    sqlQuery(con, query)
  }, error = function(e){
    message("There was an error with your query. See details below.")
    message(e)
  }, finally = {
    tryCatch({
      if(exists("con")){
        odbcClose(con)
        message("closed database connection")
      }
    })
  })
  
  return(query_result)
}

API Connection

#' Establish a session with the Salesforce API
#' 
#' This function connects to a Salesforce instance via the API
#'
#' @usage createSFAPISession(environment=c('prod', 'qa'), 
#'                           user=NULL, password=NULL, apiVersion="38.0")
#' @importFrom RForcecom rforcecom.login
#' @param username a character string defining the Salesforce API username credential
#' @param password a character string defining the Salesforce API password credential.
#' Note: that this should be the concatenation of your password and security token. See the
#' instructions in \code{\link[RForcecom]{rforcecom.login}} for an example format.
#' @param apiVersion (optional) Version of the REST API and SOAP API that you want to use. 
#' (e.g. "38.0"). Supported versions from v20.0 and up.
#' @return A session object as returned by \code{\link[RForcecom]{rforcecom.login}}
#' that can be used for subsequent calls to the Salesforce API.
#' @seealso
#'  \code{\link[RForcecom]{rforcecom.login}},
#' @examples
#' \dontrun{
#' # establish a connection
#' sf_session <- createSFAPISession()
#' 
#' # logout of that session
#' rforcecom.logout(sf_session)
#' }
#' @export
createSFAPISession <- function(environment=c('prod', 'qa'), 
                              user=NULL, password=NULL, apiVersion="38.0"){
  
  which_env <- match.arg(environment)
 
  if(is.null(user) | is.null(password)){
    if(which_env == 'qa'){
      username <- "shared_user@gmail.com"
      password <- "shared_password"
      url <- 'https://test.salesforce.com/'
    } else if(which_env == 'prod'){
      username <- "shared_user@gmail.com"
      password <- "shared_password"
      url <- 'https://login.salesforce.com/'
    } else {
      stop(paste("Cannot find the specified environment:", which_env))
    }
  }

  sf_session <- rforcecom.login(username=username, password=password, 
                                loginURL=url, apiVersion=apiVersion)
  return(sf_session)
}

Note: If you do not feel comfortable centralizing credentials for security reasons or your team does not have shared credentials, then have those stored locally in a profile and called by the package in a standardized way.

Tutorials

Finally, but maybe most importantly, R packages help teammembers grow and on-board faster. You can include sample datasets and plenty of examples within the R package so that new members of your team don’t have to ask around how to do certain things. They can read the documentation and try the examples for themselves. Consider the R package a legacy asset of your team that will stand the test of time well after you’ve left the company.

Automating R Scripts with Cron

Steven M. Mortimer — Tue, 15 Aug 2017 00:00:00 +0000

Scheduling a Script to Execute

If you would like to automate R scripts, one method is to use the cron daemon already packaged on Linux servers. Let’s say you have script in your home directory called random.R and you would like to schedule it to run at 5:30pm every day. On the server you will need to edit the crontab and add the following line:

30 17 * * * Rscript /home/my-user/random.R

Note: You can add this line to the the crontab by typing

crontab -e
Press i to enter insert mode
Paste your new entry in the file
Press CTRL+C to exit insert mode
Save and quit by typing :wq

For reference, it should look like this while adding:

Sending Cron Results to your Email Inbox

It is exciting to have a routine part of your work automated, but you probably want to monitor your script and the results its creating. Cron has a built-in feature that can send an email whenever an scheduled script errors out. Just add your email address to the MAILTO option. If you want to include multiple emails, then separate them with a semicolon, like so:

NOTE: New servers may not be setup as an email server. You will not receive any emails or even errors that it didn’t work if you set the MAILTO parameter but haven’t configured the server to send email. I recommend using Postfix, a free and open-source mail transfer agent, which is fairly easy to setup. Below are abbreviated instructions for an Ubuntu server. See the additional resources or run a Google search if you need more detail on setting up a custom hostname, domain, SASL authentication, etc.

From the command prompt of the server use the following steps:

Step 1 - Update package information: sudo apt-get update
Step 2 - Insall Postfix: sudo apt-get install postfix
Step 3 - Complete two prompts to install Postfix:
- Step 3a - Select Internet Site
- Step 3b - Click OK to approve the “System mail name” (usually the server’s assigned hostname, like ip-172-88-99-0.ec2.internal for an AWS EC2 instance)

Additional resources:

Setting Things Up to Only Email on Error

Now you may be tempted to simply add your email address and call it a day, but you might notice that R starts flooding your inbox with cron emails even for successful runs of your script. If you prefer to only receive emails when your script errors out, then you have to short circuit how cron identifies R script errors. Below I’ll outline how to setup cron to only email on actual R script errors.

Cron determines whether to send an email based on the “exit code” of the script that runs. An exit code is a number emitted at the end of the script indicating a status of whether not an error occurred during the execution of the script. However, R emits exit codes a little differently than most scripting languages. In R, any messages are converted to STDERR (signaling an error), so it is possible to have a script that ran successfully, but signals an error occurred in running the script. Upon realizing that R treats messages as errors in disguise, it makes more sense to me why they are displayed in red in the R console:

Fortunately, a clever use of I/O redirection (similar to dplyr piping), can roll message-induced error codes into less benign forms (STDOUT) so that an email is not triggered when messages were printed and your script really didn’t error out. The way to do this is modify the command listed in your crontab file. Instead of just including Rscript /home/my-user/random.R, you should include:

Rscript /home/my-user/random.R > temp.log 2>&1 || cat temp.log

The > symbol will redirect the error feed(STDERR - captured by “2”) and roll it into the standard output feed (STDOUT - captured by “1”) and push them into temp.log. The || symbol checks whether the script had a non-zero exit status. If so, then it will run the cat command and print everything to the console and trigger an error email from cron.

Emailing Errors, but Logging All Runs

Finally, you might want to log the script output regardless of whether the script errored out. To do tha you just need to continue redirecting the output that you want to appear in the log file. The command below method will still roll error messages into the standard feed and only trigger the email if the script didn’t finish, but it will also push everything into a persistent log that will keep a record of every run that you scripts do. This way you can get the error emails, but still keep a complete log of everything your script did if you want to have a complete log of all runs (successful or not).

Rscript /home/my-user/random.R > temp.log 2>&1 || cat temp.log && cat temp.log >> persistent.log && rm temp.log

That’s it. Now that you’re aware of this command you can swap out the R script name and log names for every other script that you want to schedule. You really don’t need to understand all of the details, just the last snippet of code in this post, but I would recommend browsing online to understand more about cron and I/O redirection. Note: I’ve outlined this logic in a Stack Overflow response at https://stackoverflow.com/a/34442846/5258043.

A Scalable Predictive API with PFA and AWS Lambda

Steven M. Mortimer — Fri, 28 Jul 2017 00:00:00 +0000

Building machine learning models is fun and challenging, but the real challenge can be figuring out how to deploy your models in a production environment. This includes answering questions like: Does the model need to be translated to another programming language? Will it scale? Does it need to be dumbed down? Usually, the person who built the model is not an expert on how to deploy the model in infrastructure, which has led to a proliferation of turn-key solutions for converting machine learning models into web APIs: Azure ML, YHat ScienceOps, Domino Data Lab, and many others. I’m not a big fan of spending money on these types of platforms or systems because they are usually costly and require learning quirks of a specific platform that has limited features and support, so I’ve tinkered with finding alternatives.

A Scalable Infrastructure

AWS Lambda is serverless computing, in other words, you only pay for the computation of a function you’ve uploaded to AWS. AWS will manage the orchestration of servers to handle requests. AWS Lambda functions can easily be turned into a webservice using AWS API Gateway, so this accomplishes your accessibility and scalability requirements. AWS will scale to however many requests come in and you only pay for what you use. The question is: How to execute machine learning models from an AWS Lambda function?

Model Building and Execution using PFA

Models can be built in many languages like, Python, R, or Java and if you work on a large, cross-functional team it might not be possible to agree on using just one language. The Portable Format for Analytics (PFA) is an interesting solution because it is a language-agnostic, plain-text standard for representing analytical workflows. PFA can be executed from within an AWS Lambda function in either a Java or Python runtime using the Hadrian or Titus libraries, respectively (an example is provided below). At a high-level data will be sent via HTTP to API Gateway, which provides scaffolding to route the request to your AWS Lambda function. Once the data reaches the function, the PFA logic will be applied and the result passed back to complete the roundtrip API call. All of this is completely scalable, can be run at low cost, and automatically gives you infrastructure for tracking response times, assigning per-user access keys, creating API documentation, applying request validation and much more. As a bonus, using AWS may also be more likely to get more support from your IT staff should you need it.

Actually Doing This

Step 1 - Creating the Lambda Function

The absolute simplest test would be to clone the repository where I’ve created a Lambda deploy package for running Titus (Python runtime), then zip up that package and upload to the AWS Lambda console.

git clone git@github.com:StevenMMortimer/pfa-lambda.git
zip -r my-lambda-deploy.zip .

Note, you’ll want to select the Runtime for your lambda as Python 2.7 since that is the Python version that this lambda deploy package was created. Also, the deploy package has a function file named lambda_function.py so that it matches the handler specification in the AWS console by default (i.e. The filename.handler-method value in your function.). You could rename your .py file as anything, but remember to update the handler specification to match.

Finally, the Lambda function runs under an IAM service role. If you’d like the Lambda function to access other AWS resources, like reading your model file from S3, then you’ll want to make sure that the service role as permissions to access S3. For example, the image below shows the service role has access to write to CloudFront logs, but also full access to S3.

Proceed through the create Lambda function dialog to create your function, then give it a test. The function lambda_function.py has hand-coded PFA logic to take 3.14 and add 100 to it, which returns 103.14. In the test summary, you can see the output as 103.14, but you can also see that the function executed in .25 milliseconds. A machine learning model executing in .25 milliseconds is pretty darn fast and exciting to see, but this is just a toy example.

Step 2 - Build Your Model

A more realistic example, might be building a Naive Bayes classifier on a dataset of with many predictors. The mlbench package has a dataset called “Sonar” that consists of 208 data points collected on 60 predictors. Our goal will be to predict two classes (M for metal cylinder or R for rock). In the code below, I create a model and export it to PFA using the aurelius package.

library(mlbench)
library(e1071)
data(Sonar)
set.seed(1)
sonar_model <- naiveBayes(Class ~ ., data=Sonar)

library(aurelius) # library for converting to PFA
sonar_model_as_pfa <- pfa(sonar_model)
write_pfa(sonar_model_as_pfa, '~/model.pfa')

Note that the model could be created in scikit-learn, MLlib or any number of R packages for maching learning and translated using a PFA converter library. Support exists for the JVM, Python, and R, just check here.

Step 3 - Upload Your Model to S3

After you’ve created a PFA model upload it to an S3 bucket. In the Lambda deploy package there is a Lambda function called model-from-s3.py that has logic to read a PFA model from an S3 bucket and use it for scoring.

Note, you may have to adjust the bucket name and model file name in the Python script to read the file you’ve uploaded. These parameters are specified in Lines 10 and 11 in the script.

Step 4 - Test Your Lambda Makes Predictions

After configuring the script model-from-s3.py and re-uploading your Lambda deploy package, you will need to update the handler specification in the AWS console. Change the value to be model-from-s3.lambda_handler. This model requires the data to be provided via the event JSON for the function. You can use the AWS Hello World testing template which provides 3 dummy key-value pairs. Our model had 60 variables as inputs so instead of writing those out by hand, it’s easiest to pull out one record from the sample data and copy/paste into the testing template.

jsonlite::toJSON(jsonlite::unbox(Sonar[1,grepl('V[0-9]+', names(Sonar))]))

## {"V1":0.02,"V2":0.0371,"V3":0.0428,"V4":0.0207,"V5":0.0954,"V6":0.0986,"V7":0.1539,"V8":0.1601,"V9":0.3109,"V10":0.2111,"V11":0.1609,"V12":0.1582,"V13":0.2238,"V14":0.0645,"V15":0.066,"V16":0.2273,"V17":0.31,"V18":0.2999,"V19":0.5078,"V20":0.4797,"V21":0.5783,"V22":0.5071,"V23":0.4328,"V24":0.555,"V25":0.6711,"V26":0.6415,"V27":0.7104,"V28":0.808,"V29":0.6791,"V30":0.3857,"V31":0.1307,"V32":0.2604,"V33":0.5121,"V34":0.7547,"V35":0.8537,"V36":0.8507,"V37":0.6692,"V38":0.6097,"V39":0.4943,"V40":0.2744,"V41":0.051,"V42":0.2834,"V43":0.2825,"V44":0.4256,"V45":0.2641,"V46":0.1386,"V47":0.1051,"V48":0.1343,"V49":0.0383,"V50":0.0324,"V51":0.0232,"V52":0.0027,"V53":0.0065,"V54":0.0159,"V55":0.0072,"V56":0.0167,"V57":0.018,"V58":0.0084,"V59":0.009,"V60":0.0032}

The results of a test on the real model are also encouraging. The prediction was “R”, which matches the prediction of “R” using predict(sonar_model, Sonar[1,]). Also, it only took 52.46 milliseconds to execute against a Naive Bayes model model with 60 input variables. Note, different models will have different latencies, so your performance may vary.

Step 5 - Hook Lambda Up to API Gateway

The last step in making your prediction service is to open it up as a webservice so that other applications can take advantage of your hard work. I won’t go into detail about API Gateway, but it’s very powerful in terms of supporting a full-fledged API (access keys, logging, documentation, etc.). I would encourage you to take advantage of these after getting the bare bones of your API working.

First, create an API and add a POST method. You’ll want to use a POST method because new data for your prediction will be POSTed in the request body as JSON. While setting up the method you can associate it to your Lambda function. You’ll be able to test that your API works by supplying some sample JSON in a format that your model expects.

Finally, API Gateway requires you to “Deploy” your API so you’ll need to do that step before being able to access it via the web. APIs are deployed in stages, so you can create a stage called “dev” and deploy to it.

Now that everything is setup you can test calls against the invoke url by submitting a curl command, like so:

curl -X POST -H "Content-Type: application/json" -d '{"V1":0.02,"V2":0.0371,"V3":0.0428,"V4":0.0207,"V5":0.0954,"V6":0.0986,"V7":0.1539,"V8":0.1601,"V9":0.3109,"V10":0.2111,"V11":0.1609,"V12":0.1582,"V13":0.2238,"V14":0.0645,"V15":0.066,"V16":0.2273,"V17":0.31,"V18":0.2999,"V19":0.5078,"V20":0.4797,"V21":0.5783,"V22":0.5071,"V23":0.4328,"V24":0.555,"V25":0.6711,"V26":0.6415,"V27":0.7104,"V28":0.808,"V29":0.6791,"V30":0.3857,"V31":0.1307,"V32":0.2604,"V33":0.5121,"V34":0.7547,"V35":0.8537,"V36":0.8507,"V37":0.6692,"V38":0.6097,"V39":0.4943,"V40":0.2744,"V41":0.051,"V42":0.2834,"V43":0.2825,"V44":0.4256,"V45":0.2641,"V46":0.1386,"V47":0.1051,"V48":0.1343,"V49":0.0383,"V50":0.0324,"V51":0.0232,"V52":0.0027,"V53":0.0065,"V54":0.0159,"V55":0.0072,"V56":0.0167,"V57":0.018,"V58":0.0084,"V59":0.009,"V60":0.0032}' https://7uc1muh9pd.execute-api.us-east-1.amazonaws.com/dev

Load Testing

For those of you who are interested in the scalability. I used artillery.io to load test and got the following results for our classifier. I created a test where the load simulated 10 requests per second over 60 seconds. The 99th percentile latency was 0.30 seconds and the median was 0.15 seconds. The complete results were:

Load Testing Results

all scenarios completed
Complete report @ 2017-07-28T16:53:05.387Z
  Scenarios launched:  600
  Scenarios completed: 600
  Requests completed:  600
  RPS sent: 9.93
  Request latency:
    min: 115.6
    max: 806.8
    median: 149.1
    p95: 217.3
    p99: 300.1
  Scenario duration:
    min: 116.6
    max: 807.8
    median: 150.5
    p95: 219.6
    p99: 301
  Scenario counts:
    0: 600 (100%)
  Codes:
    200: 600

Load Testing Config

Here is my testing config as artillery YAML file:

config:
  target: 'https://7uc1muh9pd.execute-api.us-east-1.amazonaws.com'
  phases:
    - duration: 60
      arrivalRate: 10
  defaults:
    headers:
      Content-Type: 'application/json'
  payload:
    path: "./test-data.csv"
    fields:
      - "V1"
      - "V2"
      - "V3"
      - "V4"
      - "V5"
      - "V6"
      - "V7"
      - "V8"
      - "V9"
      - "V10"
      - "V11"
      - "V12"
      - "V13"
      - "V14"
      - "V15"
      - "V16"
      - "V17"
      - "V18"
      - "V19"
      - "V20"
      - "V21"
      - "V22"
      - "V23"
      - "V24"
      - "V25"
      - "V26"
      - "V27"
      - "V28"
      - "V29"
      - "V30"
      - "V31"
      - "V32"
      - "V33"
      - "V34"
      - "V35"
      - "V36"
      - "V37"
      - "V38"
      - "V39"
      - "V40"
      - "V41"
      - "V42"
      - "V43"
      - "V44"
      - "V45"
      - "V46"
      - "V47"
      - "V48"
      - "V49"
      - "V50"
      - "V51"
      - "V52"
      - "V53"
      - "V54"
      - "V55"
      - "V56"
      - "V57"
      - "V58"
      - "V59"
      - "V60"     

scenarios:
  - flow:
    - post:
        url: "/dev"

Google Sites for Documentation

Steven M. Mortimer — Fri, 02 Jun 2017 00:00:00 +0000

The title of this post is probably off-putting to a large number of people. Why on earth would I use Google Sites for blogging, wiki, and technical documentation? Isn’t that old, outdated tech even though Google did release a newer version? To be honest, I agree with those objections. This blog isn’t written with Google Sites. It’s written using the R package blogdown and Hugo and they are fantastic. However, there are some features of Google Sites that might make it a nice tool for you and your work colleagues.

The WYSIWYG

Never underestimate the power of the WYSIWYG editor. This magical tool allows anybody, at any technical skill level, to contribute content. Creating and maintaing a good website requires constant updates and tinkering that a single person might not be able to keep up with. The graphical interface allows you to share work with non-technical contributors as they shore up and support your efforts.

Access Control Permissions

Static website generators are nice, but lack some more robust and intutive tools for granting permissions and access to your site. With Google Sites this comes standard. If your orgnization uses the Google Suite, this is one of the main reasons for sticking to Google Sites. You can contol view and edit permissions to any level of detail for individuals inside and outside of your organization. Again, this can all be with the click of a button or API calls, just take your pick.

Embedded Google Docs, Sheets, etc.

Not sure if this needs to be said, but adding it here for completeness. Google Sites plays well with embedding Docs, Sheets, Slides, and other Google products. If you already have a significant amount of content in these formats, you’ll find it easy to integrate into a single source.

Built In Google Search and Analytics

Your Google Site automatically comes with the powerful and handy Google Search technology. A good turn-key search solution on your website is critical, especially for wikis and technical documents. Users typically won’t frequent your site unless they are looking for something. The search bar is their window into the trove of information you’ve got. The search makes finding the content they need easy and encourages them to come back for their next quesion. User engagement fosters the virtuous circle where the contributors add more content and users consume more and more of it. And, of course, Google Sites makes it easy to plug in your Google Analytics code and track that user engagement.

Final Thoughts

Google Sites are alright in my book. I’ve used them and it’s worked out well, but each time I had a strong reason for wanting to use it and was committed from the beginning. Given my experience with R and willingness to use the Sites API I went ahead and created a collection of functions that make it easier to create, update, and delete content on your Google Site without ever leaving R. I’ve started bundling together the functions into an R packge called googlesites that’s currently on my GitHub. Please take a look to see the sorts of things you can do with the package at https://github.com/StevenMMortimer/googlesites

An R User Chooses a Python IDE

Steven M. Mortimer — Fri, 07 Apr 2017 00:00:00 +0000

UPDATE - July 2020: As of last summer (June 2019), I have switched to VS Code as a Python IDE and never looked back. The reason being that it feels a little bit more lightweight than PyCharm, but still has all the functionality one would need to write Python code and can be more easily configured to act like RStudio. Check out my more recent post Setting up VS Code for Python Development like RStudio for my settings.json (user settings) and keybindings.json (shortcuts) configuration files that you can download and paste into your own VS Code installation to get up and running quickly.

Yes, this is a post about IDEs. I don’t intend to insult or incite any ill-will towards any person or project. The purpose of this post is to give other people who are familiar working with R, some insight into how a fellow R user has evaluated and experienced picking an IDE for programming Python. My criteria was to have an interactive experience, like RStudio, with support for debugging, version control, and code completion. That’s not too much to ask for, right?!

Jupyter Notebooks

If you have been interested in data science, and R and Python programming, then you’ve probably seen IPython or the newer Jupyter Notebooks. You might have also seen something similar with Kaggle Kernels or Zeppelin Notebooks. I agree that the notebook format is great for explaining and teaching, which is why so many tutorials display their work in notebook format. The problem with Jupyter notebooks is that the format is plain-text, but not easily readable when checking code diffs on a version control system like GitHub. For example, a simple code change can make massive diffs:

Second, notebooks force you to work in chunks. Chunks are nice for helping you separate and structure your workflow, but you can’t run half a chunk. It’s an all-or-nothing game. The problems occur when you need to debug something in the middle of a chunk. At that point you need to deconstruct the chunk, then test it, then put it back together. It’s a mess. Finally, when you move your code into production it likely won’t be in a notebook format. Why write code in a notebook when you know it will probably be rewritten again in another format? This is inviting silly errors and wasted time.

Rodeo

If you’ve used any other tools created by YHat or just want the RStudio exprience with Python, then you might be tempted to use Rodeo. Rodeo does a fantastic job at replicating the look and feel of RStudio, but if you take a closer look it is missing a lot of features you use in RStudio, which is frustrating. You can tell even from screenshots that it’s seriously lagging RStudio in features.

Rodeo is open-source, but development is mainly driven by the folks at YHat, so support and longevitiy is a concern. Finally, it does not have native integration with Git, which is a deal-breaker. If you can’t version control your code easily, then you’re more likely to make mistakes.

Spyder

I can’t comment much on my experience with Spyder. I had an issue with converting a notebook to a script, which was a feature I hoped would work effortlessly. Also, it doesn’t support Git, which as mentioned with Rodeo, is an immediate deal-breaker. Spyder seems good for a specific data science niche, but it’s not quite there, and with so many other full-fledged, battle-tested Python IDEs, why not expand your horizons to have an IDE that will allow you to utilize the entire spectrum of general purpose programming that Python is?

Text Editor (e.g. Sublime)

If you ask a hardcore programmer about picking an IDE, they might tell you that you don’t need one at all. It’s true that you can effectively and efficiently write Python code from a text editor; however, I wouldn’t recommend an R user take this approach. First, you’ll probably be frustrated with the lack of transparency for defined variables, folder structure, etc. that RStudio neatly provides in its “Environment” and “Files” panes. Second, if you’re just starting to learn Python after learning R, you’ll want some more guardrails to prevent against syntax errors, code completion, formatting, etc. This isn’t to say that text editors don’t provide it, because they do, but you’ll spend time configuring these features inside of getting them out of the box from an IDE.

PyCharm

PyCharm definitely has the feel of a heavier IDE. There are a lot of settings and features that would be intimidating at first, but it’s got everything right there. A console, version control support, debugging, code completion and more. The one thing I did change was the keyboard shortcuts to executing code interactively especially when highlighting code sections to get that same feeling as you get in RStudio where you can select pretty much anything and get it to run down in the console interactively. After tweaking the keyboard shortcuts, it was pretty much the same flow as using RStudio and I felt good about using a full-featured and well supported project. Ultimately, PyCharm is what I’m sticking with to continue writing Python code.

Final Thoughts

I have read quite a bit online about the IDEs mentioned above, but other peoples’ opinions didn’t do much for me. I literally downloaded each IDE and spent a couple hours trying to work on a project. If it was easy to get going I kept trying more things until I was able to test out my desired features. You can take my word for going with PyCharm to continue your Pythonic data science, but I’d recommend to anyone to try things out, see what you like, and go with it. In the end, you should be spending the majority of your efforts writing code, not fiddling with the IDE, so make the choice that makes you most happy and productive.

PFA with the R Package aurelius

Steven M. Mortimer — Mon, 13 Feb 2017 00:00:00 +0000

I first heard about the Portable Format for Analytics (PFA) through KDnuggets in 2016. There was a post that in vague way talked about the virtues and benefits of PFA, but not to completely supplant PMML, which I was already familiar with. PMML was pretty cool, but I couldn’t find good support and felt that the XML-based approach was clumsy and outdated so when I heard about PFA, I was pretty excited. A new JSON-based standard for encoding any type of analytics, not just certain supported models like PMML. I mainly work in the R programming language, so I was disappointed to find resources lacking that would convert models made in R into the PFA standard. The great folks at the Data Mining Group (DMG) had started on an R package called aurelius that contained the bare bones functions to construct the syntax. After reading through the Project pages on GitHub, I saw everything was there, but it wasn’t as convenient as I would like so I started writing “model producers”. Model producers are functions that convert model objects in R into their PFA equivalent to be exported and used in other systems. In this process I learned a great deal. I thought I knew how models worked, but actually reading through the mechanics of scoring has made me appreciate them so much more. Below is an outline of working with the aurelius package, but note that this work has yet to be integrated into the project and exists on forked branch in my GitHub (installation command shown below).

# you only need to run this once because it 
# installs the package
devtools::install_github('StevenMMortimer/hadrian', 
                         ref='feature/add-r-package-structure', 
                         subdir='aurelius', 
                         quick=T)

In this example we’ll load the randomForest package to demonstrate how to convert randomForest models into PFA. Currently, the package converts models of many types: lm, glm, glmnet, gbm, randomForest, naivebayes, lda, kcca, and kmeans.

options(stringsAsFactors = FALSE)
library(aurelius)
library(randomForest)

PFA Structure

Before we get started it’s helpful to understand the basic components of a valid PFA document. Every PFA document has an input spec, output spec, and an action. The reason for the specifications on the input and output is so that the machine executing your PFA can safely and reliably excute. PFA relies on the Avro serialization format for its type system, whichsupports your typical primitive data types like boolean, integer, and string, but also, arrays, maps, and more. You can define these types in R by using any of the functions like avro_*.

avro_double
#> [1] "double"

avro_enum(list("one", "two"))
#> $type
#> [1] "enum"
#> 
#> $symbols
#> $symbols[[1]]
#> [1] "one"
#> 
#> $symbols[[2]]
#> [1] "two"
#> 
#> 
#> $name
#> [1] "Enum_1"

After specifying inputs and outputs, you must specify an action. With aurelius an action can be specified from an R expression. A simple example is 2+2. This can be expanded to manipulate the input and/or derivatives of it. Here is a simple analytics spec that takes a double and adds 10 to it.

pfa_document(input = avro_double, 
             output = avro_double, 
             action = expression(input + 10))
#> $input
#> [1] "double"
#> 
#> $output
#> [1] "double"
#> 
#> $action
#> $action[[1]]
#> $action[[1]]$`+`
#> $action[[1]]$`+`[[1]]
#> [1] "input"
#> 
#> $action[[1]]$`+`[[2]]
#> [1] 10

You’ll notice that the resulting output is a list. PFA is based on JSON, so it’s natural for the document, when it’s in R, to be a list that can always later be converted to JSON for export.

Converting a Tree Model

Let’s step through a more realistic example that predicts the species in the iris dataset, based on the other variables using a random forest model. First, format the columns to meet Avro guidelines: [A-Za-z_][A-Za-z0-9_]*. Having periods in the variable names (e.g. Sepal.Length) is not encouraged.

# make sure column names dont have periods in them
iris2 <- iris
colnames(iris2) <- gsub('\\.', '_', colnames(iris2))

Creating the PFA is very easy. Simply fit your model, then call pfa().

# fit a random forest model
rf_model <- randomForest(Species ~ ., data=iris2) 
 
# convert that model to PFA
# pred_type='prob' means that the output is the 
# probability of all classes. If you want 
# the majority vote, then specify pred_type='response'
# check all options by running ?pfa.randomForest
rf_model_as_pfa <- pfa(rf_model, pred_type='prob')

If you have the rPython package installed and Titus (PFA for Python), then you can create an engine in R and see how the PFA behaves. In this case it predicts the same as the predict() function in R. This way we can test that the behavior of the PFA is what we expect.

# you can check the predictions from this model
# by first converting to a pfa_engine
pfa_model <- pfa_engine(rf_model_as_pfa) 
 
# confirm that the pfa engine produces the 
# same prediction as the "predict" method in R
test_dat <- iris2[73,1:4]
round(pfa_model$action(as.list(test_dat)), 6)
#>     setosa versicolor  virginica 
#>       0.00       0.83       0.17
round(unclass(predict(object=rf_model, newdata=test_dat, type='prob')), 6) 
#>    setosa versicolor virginica
#> 73      0       0.83      0.17

Once you’re happy with your model, just export to a flat file using write_pfa(). This format can be read and used by any PFA interpreter. With R you can read the file back in using read_pfa().

# you can export your model for use in other systems
write_pfa(rf_model_as_pfa, file = '~/my_rf_model.pfa')

my_model <- read_pfa(file("~/my_rf_model.pfa"))

# see that all PFA components are there
names(my_model)
#> [1] "input"  "output" "action" "cells"

Final Thoughts

PFA is a measurable step in open, portable, and reprodcible analytics. I hope to see the standard more widely adopted and tools developed to help others implement it. One final thing: Check out the unit tests. I’ve recently added quite a bit of unit test coverage and these tests are an excellent source of examples because they cover most all cases of utilizing the package functions.

Creating an RStudio Addin

Steven M. Mortimer — Sat, 10 Dec 2016 00:00:00 +0000

I never thought too much about RStudio addins, but I saw a few cool examples ( jadd, colorpicker, and addinslist) and decided to take a closer look. After a little research, it seemed easier than I thought. It was easier because RStudio provided some excellent documentation on addins and it turns out that if you can write R code, then you can write an RStudio addin.

My Idea

Around the time I was looking at addins, I started taking advantage of a little RStudio trick to navigate long scripts. If you create comments that are a pound sign and end with at least 4 dashes (# Comment ----), then they will automatically be organized and appear in an organizational perspective at the bottom left of the Source pane.

I’ll take any extra organizational help I can get so I started annotating my code with comments like these to break it up. Then the idea hit me to create an Addin and link a keyboard shortcut to it so I can format comments quickly to this standard.

The Implementation

The logic is written in R code and the rstudioapi package is used so that your code can interact with RStudio. The R code can be complex or really simple (see the RStudio example on Addin basics). However, you must structure your project with certain metadata, like an R package, that will allow RStudio to automatically discover and register these addins when the addin is installed.

The R code should be documented using the roxygen2 style. Here is the small bit of regex that powers most of my sectioncomment addin.

#' Function to create the padded comment
#' 
#' This function is the logic that runs to reformat
#' the comment as a section break
#' 
#' @usage comment_styler(x, doc_mode=FALSE, l = 80)
#' @importFrom stringr str_pad
#' @param x a string to format
#' @param doc_mode a logical indicating only to convert 
#' lines starting with at least a double hash sign
#' @param l an integer indicating length to pad
#' @return a string that is formatted
comment_styler <- function(x, doc_mode=FALSE, l=80) {
  
  starter_regex <- '(\\s*#+\\s*)(.*)'
  
  if(doc_mode)
    starter_regex <- '(\\s*#{2,}\\s*)(.*)'
  
  if(grepl(starter_regex, x)){
    clean_x <- trimws(gsub('-*$', '', gsub('-+\\s+-+', '', trimws(x))))
    x <- str_pad(gsub(starter_regex, '# \\2 ', clean_x), 
                 width=l, side='right', pad='-')  
  }
  
  return(x)
}

As you can see, it’s not very sophisticated. I just look for text in the Source pane that appear to be a section comment and pad the line up.

How to Use this Addin

If you like the idea of my Addin you can install from RStudio with the following command:

devtools::install_github("StevenMMortimer/sectioncomment")

After installing it should apper in your Addins dropdown and you can use it, like so:

My favorite reason to use is that I can write an entire outline of the steps I want to perform before writing a single line of code, then turn those steps into sections.

This a great way to force yourself to think about what you are going to code before diving in head first and getting lost in a jumbled mess of your own thoughts. I hope you enjoy using it as much as I do and you consider creating your own addin.

The Unfinished duplicated Function

Steven M. Mortimer — Sun, 20 Nov 2016 00:00:00 +0000

If you’re a regular R user, then you’ve probably used or have seen the function duplicated(). If you’ve only grown up in the world of dplyr and other tidyverse packages with its cool function distinct(), then I highly envy you. In any case, you might find yourself using the duplicated() function when you need some more control. The function has a handy argument incomparables that allows you to ignore certain values when doing the comparison for duplicates. One day I wanted to use this to remove duplicate customer entries from our CRM system based on name and phone number. I went through the typical pre-processing stages of removing unnecessary punctionation, converting to lowercase, removing spaces, etc. After I got familiar with the data I realized that some of the data points were missing or not complete enough to make any meaningful determination of a duplicate. I needed a solution to ignore certain missing patterns, but it needed to encompass two or more columns of data. One hack would be to paste0() the columns together and work on a single pasted column, but this doesn’t handle multiple incomparable values. As I found out, the duplicated function can handle multiple columns, but the incomparables argument does not work for multiple columns, let alone different incomparable values for each column. Here is the error I get:

duplicated(mtcars[,c('am', 'gear', 'cyl')], incomparables=6)

## Error: argument 'incomparables != FALSE' is not used (yet)

This is really a bummer because it is nice to have this explicit functionality of what should really be considered when evaluating for duplicates. The error message sure is hopeful at the end there: “is not used (yet)”! After some digging I concluded that the function existed, as-is, at least since R version 1.4, which was released in 2001! I wouldn’t hold my breath for anyone to actually fix this issue. Contributing changes to base R has become much for difficult. I submitted a patch to R core to fix this very issue, but they weren’t too responsive. I guess it will stay as a hidden gem in base R, waiting for the next data miner to unearth. If you run across this issue and really want a fix, then feel free to use the code below. I’ve implemented a function to support incomparables across multiple columns where a list of vectors will specify the incomparable values for each column. The list elements are recycled if it is shorter than the columns being compared.

new_duplicated <- function(x, incomparables = FALSE, fromLast = FALSE, ...) {
  
 if(!identical(incomparables, FALSE)) {
    n <- ncol(x)
    nmx <- names(x)
    nmincomparables <- names(incomparables)
    lincomparables <- length(incomparables)
    if(is.null(nmincomparables)) {
      if(lincomparables < n) {
        # pad any incomparables lists with the default value if list is shorter than the number columns in the supplied data.frame
        tmp <- c(incomparables, as.list(rep_len(FALSE, n - lincomparables)))
        names(tmp) <- nmx
        incomparables <- tmp 
      }
      if(lincomparables > n) {
        # if the list is unnamed and there are more elements in the list than there are columns, then only first n elements
        warning(paste("more columns in 'incomparables' than x, only using the first", n, "elements"))
        incomparables <- incomparables[1:n]
      }
    } else {
      # for named lists, find match, else default value
      tmp <- as.list(rep_len(FALSE, n))
      names(tmp) <- nmx
      i <- match(nmincomparables, nmx, 0L)
      if(any(i <= 0L))
        warning("not all columns named in 'incomparables' exist")
      tmp[ i[i > 0L] ] <- incomparables[i > 0L]
      incomparables <- tmp[nmx]
    }
    
    # first determine duplicates, then override when an incomparable value is found in a row since the existence of even 1 incomparable value in a row means it cannot be a duplicate
    res <- duplicated(do.call("paste", c(x, sep="\r")), fromLast = fromLast)
    
    #for better performance only bother with the columns that have incomparable values not set to the default: !identical(x, FALSE)
    run_incomp_check <- sapply(incomparables, FUN=function(x){!identical(x, FALSE)})
    if (sum(run_incomp_check) > 0L){
      incomp_check <- mapply(FUN=function(column,incomparables){match(column, incomparables)}, x[run_incomp_check], incomparables[run_incomp_check])
      # any rows with an incomparable match means, TRUE, it can override the duplicated result
      overwrite <- apply(data.frame(incomp_check), 1, function(x){any(!is.na(x))})
      res[overwrite] <- FALSE
    }
    
    return(res)
  } else if(length(x) != 1L) {
    duplicated(do.call("paste", c(x, sep="\r")), fromLast = fromLast)
  } else {
    duplicated(x[[1L]], fromLast = fromLast, ...)
  }
}

Here is the function in action:

mtcars2 <- head(mtcars[,c('am', 'gear', 'cyl')])
mtcars2$dup <- new_duplicated(mtcars2, incomparables=1)
mtcars2

##                   am gear cyl   dup
## Mazda RX4          1    4   6 FALSE
## Mazda RX4 Wag      1    4   6 FALSE
## Datsun 710         1    4   4 FALSE
## Hornet 4 Drive     0    3   6 FALSE
## Hornet Sportabout  0    3   8 FALSE
## Valiant            0    3   6  TRUE

Typically, the second row would be considered a duplicate of the first row, but since 1 is an incomparable value it is not considered a duplicate, in fact, any row containing a 1 in the column will be labeled as false since it renders the row incomparable.

stevenmortimer.com

salesforcer 0.2.2 - Relationship Queries and the Reports API

Experimental Functions for Reports and Dashboards REST API

Support for Bulk 2.0 API Queries

Support for Relationship Queries

Additional Information

Running GitHub Actions Sequentially

The issue

An Example

Step 1 - Create a Personal Access Token (PAT)

Step 2 - Add the PAT as an actions secret in the repository

Step 3 - Add the repository_dispatch event to Workflow 1

Step 4 - Add the repository_dispatch event as trigger in Workflow 2 YAML

Considerations

References

5 steps to change GitHub default branch from master to main

Steps

Step 1 - Move the ‘master’ branch to ‘main’

Step 2 - Push ‘main’ to remote repo

Step 3 - Point HEAD to ‘main’ branch

Step 4 - Change default branch to ‘main’ on GitHub site

Step 5 - Delete ‘master’ branch on the remote repo

All commands

Setting up VS Code for Python Development like RStudio

How to Configure VS Code to run like RStudio

Aesthetics (Textmate theme and margin column ruler)

Running Code

Variable Explorer (“Environment” tab in R)

Data Viewer (Like running View() in R)

Plot Viewer (“Plots” tab in R)

Version Control (“Git” tab in R)

What’s missing or different in VS Code?

R’s Document Outline (aka Minimap in VS Code)

Debugging

Setup Files and Scripts

Settings JSON File

Keyboard Shortcuts JSON File

Extensions Installation Script

salesforcer 0.1.3 - Features for Better CRM Data Management

Pulling Lists of Delete and Updated Records

Searching for Duplicates and Merging Them

Controlling API Call Behavior

Links to GitHub

One R Package a Day

Credits

Full Script

Scraping Responsibly with R

How to Check if Scraping is Permitted

Most Starred R Packages on GitHub

Pulling List of Packages & their Details

Analyzing the Data

Appendix

Top 10 Most Starred Packages

Top 10 Most Downloaded Packages with Stars

Top 10 Packages by Stars per Download (frequently starred)

Bottom 10 Packages by Stars per Download (infrequently starred)

Full Script

Analyzing Customer Data from Square

The Square Data Model

Authenticating

Pulling Transaction Data

Spend by Customer Group

Issues with the APIs

Cannot Request Specific Fields

Cannot Update Customer Groups Programmatically

Customer ID Not On Transactions

Tips for Making Professional Shiny Apps with R

Add a Progress Wheel

Create a www Folder

Style with CSS

Use a global.R file

Encourage Data Downloading

Building a Corporate R Package

Analytical Functions

Company Constants

Templating/Formatting

Data Source Connections

DB Connection

API Connection

Tutorials

Step 3 - Add the `repository_dispatch` event to Workflow 1

Step 4 - Add the `repository_dispatch` event as trigger in Workflow 2 YAML

Data Viewer (Like running `View()` in R)