Tryptophantastic

Benchmarking and profiling data: two takes

Mon, 13 Jun 2016 10:54:46 +0000

After working on a personal project that took in a lot of data, to the tune of a fraction of the human genome, I inherited a Turing brown field project that has been worked on by various groups of students in an iterative fashion. It had accumulated several thousand database records linking various jobs with their respective companies and locations and we realized we were trying to answer questions about how much chronological data we could query at a time without knowing how slow those queries actually were.

Enter benchmarking and profiling performance data. While benchmarking could be described as simple speed (or memory) measurements, profiling compares these measurements in the context of a typical user flow, pointing to where bottlenecks might be and thus allows a developer to prioritize rather than micro-optimize.

Should our API serve one month of data? Two months? We were asking questions in the wrong order. So one of my first tasks on this team became researching how we might profile our web app. We did have New Relic set up and I spent some time that and Skylight, other people have written about the distinction, and I’d rather dig into some code! I’ll show you a variant on Rails performance testing that I tried, as well as what I eventually opted to use, a rake task in combination with logging to generate data in development.

With the release of Rails 4, performance testing was moved to the gem perftest. By default, perftest gives you access to tests which can both benchmark and profile and tweaks a few config settings from the usual test environment, including caching. It simulates a production environment running against your test database.

However, my orders from high, i.e., my instructor, were to run perf tests against a seeded database of ~20000 records. How could I possibly load so much data in a test environment, only to have it wiped each time I ran tests?

Solution 1: Rails perf tests in a custom environment

The answer was, no, I couldn’t fully simulate a production database in my normal test suite. My initial solution was to implement a custom testing environment, as detailed in this blog post.

This strategy goes even further than perftest in reimplementing a production-styled environment, but instantiates a new one (call it the benchmark environment, if you will). This allows one to seed a database with massive amounts of data, use custom production-like environment variables, and keep all this contained.

The steps of my implementation were:

Config database.yml with a new setup. I pointed it at my development database.
Add a new config file in /config/environments for your new environment. While this would depend on your production config, you would try to mirror that config closely.
Set up the perftest config in a helper file
Start writing performance tests! This should and will look almost like a feature test, other than you won’t be asserting what’s on the page, but you will be hitting endpoints of note and following your users’ likely workflows through your app.
Create a rake task. As the original blog post points out, this removes the db:test:prepare step that would otherwise reset your database.
Run the tests and enjoy the data provided to you by perftest’s logging.

While this was a decent solution and logical, I got my tests written and all my config set up, and as I attempted to debug why I seemingly was in the correct environment and database both, but didn’t have my database data, I realized this solution was overly complex. It was taking a solution that was supposed to be standalone, and added another layer.

Solution 2: Use the tools we already have, but let’s make more data

While I realized there were numerous solutions to this problem, all probably useful and feasible, I turned to an instructor who had solved similar problems while at LivingSocial. He suggested I implement a rake task that automates hitting endpoints of my own definition, similar to how a performance test would be written.

While I chose to seed a development database and hit that, you can conceive of other configs that would more closely approximate a production environment. I chose my solution primarily to allow teammates to test new queries before pushing to master.

Use Capybara to describe a common user story, then log the data with Rails logger. Capybara can be used to just create a session. I then described common user stories:

session.all("h4").sample.click_link_or_button("View Listing")

Include benchmarking methods to log data automatically, and a rake task to randomly hit all my methods as much as I wanted it too.
Create a whole lot of realistic data (or use real data) in a development database. Having now seeded two large databases, I have ideas on how to do this well, but that can be another blog post!
Turn back to your profiling gems like New Relic. As much as I like their dashboard for deployed websites, installing the gem also gives access to an endpoint on your local app at /newrelic. While the visualizations are a little different, generating a lot of hits with my rake task also provided a lot of New Relic analytics while in development.

Unfortunately, as our web app is still young, our crazier database queries remain to be written, so I don’t have fun profiling I can do yet. However, having a means to monitor these queries ahead of time means that future developers can stay one step ahead and always know where their biggest performance expenses exist.

Tomorrow I’ll be headed to a local Ruby meetup where the speaker will be addressing this very topic, so I look forward to seeing what assumptions I got wrong… stay tuned.

9 Principles of Successful API Documentation In Snippet and Screenshot

Sun, 01 May 2016 14:04:27 +0000

The third module of the Turing program focuses heavily on consuming and creating APIs, and the first thing that these masses of junior developers encounter with any API is its documentation. How to authenticate. What format requests are made in. What information to expect in a response. An API inherently exists to facilitate sharing a great resource and thus its documentation is its gateway: it either convinces developers to turn elsewhere or welcomes them in for a successful encounter. Parse Server writes that the documentation is the “most important piece of UX for a developer”.

Fortunately, there’s been written a lot of good documentation on writing good documentation. Brad Fults wrote a comprehensive list of best practices. The master repository of all APIs available on the web, Programmable Web, also offers its own list of tips.

Using Fults’ principles, I hunted down the best representations of each idea. What follows are both examples to learn from, as well as a curated list for other developers looking for a successful place to start consuming.

1. Provide a good overview

Fults points out that APIs have multiple audiences: beginners, experts, and decision-makers. Thus, an API should have a good overview that addresses each of those groups.

The Digital Public Library API Codex does just that. The first paragraph of their API homepage presents different jumping off points, from someone who has never used an API and will need the fundamentals, to someone ready to “dive right into the requests.” It also addresses someone who might be making decisions by providing their policies and philosophy and also calls out a section for troubleshooting and a glossary.

A good overview might also link to an explanation of your ontology, taxonomy, or naming conventions for more complex APIs that rely on a lot of nested industry-specific information. DPLA does this on another page.

2. Document each and every call to your API separately

The Slack API is enormous. Yet each call that can be made is referenced and linked to an even more detailed breakdowns.

As an example, the Slack Methods page lists several methods for channels.

Clicking one of those methods brings up a new page with more explicit detail.

While this pattern of broad overview funneling to explicit detail generates a lot of content when you have a lot of requests, it’s necessary for good documentation, and leads to the next principle…

3. Provide all of the documentation in a single page

Audiosear.ch’s API provides a recommendation engine for podcast series. Their method of implementing full-page documentation involves being able to show and collapse sections on each of their endpoints, as so:

While the documentation itself is actually pretty good, providing well-formatted information about all the parameters and response details, the page loads automatically collapsed. Developers searching out the documentation for debugging purposes would not be able to use the browser’s find on page to locate the term “show_title”, in this example, without expanding all the categories and manually searching first.

(Note: I later discovered the same styled format on another API, so this format seems to be the implementation of a library, but still, could be considered a flaw in the library.)

That it’s a small API makes this less of a problem, but the larger the API, the more important to have at least one page where a developer can find everything at-a-glance, even if content is then linked elsewhere.

4. Provide complete details about both the request and the response

Birdsong website Xeno-canto has a mission of “sharing bird sounds from around the world”, which makes having an API a pretty obvious solution. After detailing a request and a response structure, their documentation also breaks down all the fields, a sample shown below. The more nuanced and technical your response, the more important to detail what a field might mean in the context of your API.

id: the catalogue number of the recording on xeno-canto

gen: the generic name of the species

sp: the specific name of the species

ssp: the subspecies name

en: the English name of the species

…

5. Use plain English in the beginning…

This hearkens back to the principle that you’re creating documentation for multiple audiences. Clear, nontechnical languages helps everyone find their resources appropriately, especially those who haven’t used your docs before.

Our class this year did our final exam using the Best Buy API. While I’m not a fan of its split-column table-like docs, it does have an Overview section that’s easy to find and clearly explains the structure, progressively getting more specific in detail as it moves from Response Format to Errors and Postman. Anyone working with their endpoints would benefit first from reading the Overview section before heading elsewhere.

6. …But provide cues and organization for the expert who is debugging

The Spotify doc use navigational cues like a table of contents, links, and organized headers to provide hints to experts who are debugging. A navigational column provides keywords for using find-in-page and the top of each section has a search bar. Each element in each endpoint reference further links to an appropriate resource.

And there’s nuance. In the example below, the “endpoint” and “usage” urls lead to a section detailing the request and response, while the “returns” links to a page detailing the complete object model for that resource.

7. Provide a sample API key for demonstration purposes

23 and Me’s API documentation is not the most cookie cutter for this idea, but still demonstrates a couple of interesting ideas. It was my API baby for the last several weeks and consequently I learned its documentation pretty well. It’s an interesting API to work with: few people might have a 23 and Me account to map their genome as much as they might have a, say, Spotify account. Only being able to rely on actual users for sample data might prove tricky.

However, it does provide a demo version of all of its endpoints which returns hypothetical genetic data. The tricky part is that even this demo version still requires an access token (not to be confused with a developer’s client token), but even developer accounts have that token. Some developers don’t realize they still have this token to access demo data and thus the documentation could be more clear on this account. Some developers would prefer a demo access token be provided in the documentation for purposes of trying endpoints before they have an authentication method set up.

What decision an API makes about how to provide demo access to data leads into the next point…

8. Make it simple to copy by providing sample code

Slack responds similarly in the way it offers test access to API responses. While you need to have the correct privileges on a Slack account to try out their API within the webpage, they provide a tester feature for all of their endpoints. In this fashion, any question about an argument becomes clear by the ability to receive real data right next to the documentation on that method.

Not all APIs want to or can provide this level of functionality, but the simple solution is to provide curl snippets for every request.

9. Provide context and examples for the languages you expect to see using your API

Usually all a developer might need is curl examples. But more complex APIs with authentication schemes often (should) have sample implementations in the languages they expect to see used with their product. These are often hosted in Github repos, as 23 and Me does, or as a separate tutorials/quickstart section, as with Google Analytics, which offers the fundamentals in Java, PHP, Javascript, and Python.

Look, Ma, it’s my first API

To further work with documentation, I’ll be writing my own documentation for an API I created on my last project. Looking for more about API documentation? Make sure to read The Best API Documentation. Wanting another resource to test APIs interactively? Mashape offers a nice collection and lets you test an endpoint as well as offers snippets for each API as a curl request or one of 7 different languages. Whether you’re serving or consuming, remember, good technical docs are indispensable.

Dynamic technical demos, or how I tried to show that not all hackers need wear hoodies

Thu, 10 Mar 2016 08:24:00 +0000

As one of the requirements of Turing is that on three occasions you present a lightning talk on a technical topic, I had put a lot of thought into potential topics. I knew I wanted to try something that was more of a demonstration in nature rather than just informative. After creating and presenting a demo of cross-site scripting that was successful for several of my own criteria, including audience interaction, I offer the lessons I learned on implementing your own technical demonstration.

Early on I had jokingly tweeted about wanting to use stock photography in a technical talk, since the general expectation in the infosec community is the ridiculous public perception of hackers, or at least or at least as portrayed by the media. A topic for another blog post would be how the image-driven format in technology-focused media outlets works well for gadget reporting, but lends to stereotypical presentations for more abstract science or IT topics.

My first Turing lightning talk will lead w/ numerous bad stock photos of infosec, in quick succession. Maybe set to 80s movie montage music.
— Heidi Hoopes (@HeidiHoopes) January 16, 2016

Lesson 1: Use appropriate tools

While our module is currently focused around learning Rails to serve web apps, we started with Sinatra, a lighter weight framework than Rails with fewer “helpers” to get things done. So why did I choose to demonstrate XSS attacks on a website I built with Sinatra?

Simple. I had done research and learned that Rails already contains a lot of helper methods to validate, sanitize, and encode data. Because one of the larger points I was hoping my demonstration would drive home was that hacking code can lead to learning, I wanted a simpler tool, not something with built-in methods that might lead to a developer feeling "safe" or "complacent" about needing to understand the root problem. While these solutions proved simple, more complex demos will ultimately require numerous demos of the demo, so to speak. Don't let your lack of tool preparation distract or break your demo when it's almost always going to be the simplest piece to address.

Lesson 2: Control the interaction

I hoped that the audience would load up the URL I offered for my demo web app, but wasn't quite sure if they would engaged. I planned as if they would and created a script to quickly drop and reset my database should my classmates choose to try out their own XSS attacks on my server.

Indeed, by the second time I flipped back to the website from my slides, I was greeted by numerous pop-up alerts that had been embedded in my database. Amusingly I had lost my terminal screen in my maze of Mac mirrored screens and was only able to reset the database once, so pressed on with multiple alerts.

While in some ways this effect was entirely within the expectations for the talk, it did add a little to my inner chaos. Perhaps, also, I might argue that I should've dropped the database when I wasn't interacting directly with the website, potentially to keep my audience from getting too distracted.

Lesson 3: Balance demonstration with instruction

The flow of my presentation had three examples: a simple XSS mistake or accident, another demonstrating what a real attack might look like and be countered by, and then finally an attack from a plant in the audience that used more sophisticated misuse of HTML tags. This correspondingly let me engage the audience in a demonstration, but then provide context and instruction. The first example introduced the idea of XSS, that database rows can contain any data the user submits, even that with unintended effects when propagated to the view The second example allowed me to lead into introducing (OWASP)[http://www.owasp.org] as a resource and explaining potential attacks and defenses.

Finally, the last example let me pivot on my primary point that hacking teaches us about code. It let me prove in real-time that coding with an assumption of trust about a user blinds us, in this case, to how a simple image tag can be misused, rendering our intentions and perceptions meaningless.

Final score:

I got a lot of laughs for my final XSS demo(I embedded the school favorite Kazoo Kid remix in the page), and a lot of pleased feedback. If there's something you can teach even better by using a demo the audience can engage with, I suggest you take that extra time to drive the point home even better.

Using stock photography of hackers in a lightning talk, off the bucket list pic.twitter.com/kbwCo2DJDM
— Heidi Hoopes (@HeidiHoopes) February 28, 2016

Learning about ERB...and actually using ERB

Fri, 29 Jan 2016 11:50:46 +0000

I ran into an issue trying to get Jekyll variables for previous and next pages to work, and assumed there was some Jekyll magic I was missing. Though I was misunderstanding part of the way Jekyll variables can be chained together, what I was really missing was realizing I could apply knowledge I already had to make Jekyll better. Jekyll has variables that hold the URL for the previous page relative to the file you’re in, or page.previous.url. However, I wanted to evaluate if there was actually a previous post, or if I was at the final one, chronologically speaking.

What I failed to realize was that I already had learned the tools necessary to allow Jekyll to evaluate that, in the form of the Embedded Ruby that is already all throughout a Jekyll layout. So to add one more tiny piece of code, not {{page.previous.url}}, but {% if page.previous.url %} .

I am finding I still have a disconnect as I’m thrown into learning things as quickly as possible. There’s feeling like a beginner, but also testing the boundaries of what I know or could do. I still Google all the time and hope for code snippets. But sometimes I don’t know that I already know how to figure out what I want to know.

In other words, in a somewhat cheesy koan, the magic isn’t in Jekyll, it’s in me. ;)

Picture included because I’m still testing formatting tweaks. Enjoy the meerkat.

First Commit

Thu, 28 Jan 2016 14:04:27 +0000

As cliche as the ubiquitious first post is, it’s also a fun reminder to disconnect yourself from obligations or mental hangups.

As part of beginning the second instructional module at Turing School I was to learn (refresh) my knowledge of HTML and CSS and create a blog to contribute technical posts to during the course of the year. It was recommended we try Jekyll, described as a “blog-aware” static page tool. I had utterly forgotten that my husband Mark began using Jekyll years ago, while I was fumbling with WordPress themes.

The calendar at Turing is so crowded, and my workflow currently so centered around a text editor and git, that it was frankly refreshing to admit that I just wanted to try something different.

I’m using Jekyll through the Ruby gem. Take a look. Eventually I might move it over to my husband’s VPN, but again, simplicity is sometimes nice!

My personal blog is still online, though quiet, at tryptophantastic.com.

def print_greeting(name)
  puts "Hello, #{name}"
end
print_greeting('World')
#=> prints 'Hello, World' to STDOUT.