Contact Us
Testing Webhooks with the FullContact API

Asynchronous Delivery with Webhooks (and testing locally)

Written By:

One of our primary goals at FullContact is to make our service fast, easy, and responsive. With that in mind we designed a HTTP callback mechanism to automatically POST a JSON document containing query results to a given consumer endpoint. We (and many others) call these ‘webhooks‘.

While initially a very minor feature, the HTTP callbacks have become one of the more highly used pieces of our system. The first iteration of this feature was underdesigned and fell over quite spectacularly. I ended up spending Christmas Eve rewriting this part of our infrastructure. In my defense, I found it a rather fun and relaxing break from the rigors of family. ;)

The way it works is your request first comes in to one of our web heads. From there we evaluate your request, check formats and ensure your API key quota has not exceeded its limit. If everything checks out, we toss it on a SQS queue for one of our backend processors to reindex and refresh. As a consequence, the freshest, most up to date data can be found from our webhook interface.

At the end of reprocessing, another message is sent to the outbound webhook queue. A separate service consumes these messages and executes high-performance HTTP requests to your webhook endpoints. In the case of an error, we retry delivery 2 additional times with exponential backoff to give your postback endpoints a chance to breathe.

After delivery or error we notify our core API to bill your request accordingly (200 or 404). Throughout this process, we audit the path and deliverability of a webhook in a database. We’ll be exposing a status interface built on this data to our customers soon.

Webhook Infrastructure

So how about testing?

Testing using an external API can be difficult, especially if the remote API uses webhooks as a means of asynchronous response delivery. If you’re anything like us, you probably spend a lot of time at your development box hacking, running, and iterating (with some unit testing in-between ;)). You shouldn’t need to push to staging to test an exchange with our API.

ProxyLocal is a service which allows you to install a rubygem locally and have all requests proxied to you easily and quickly using the hosted ProxyLocal service. I learned about this service from reviewing exceptions from our webhook processor. I was able to immediately use ProxyLocal to proxy requests to my local development machine. Using my extraordinarily basic webhook testing Sinatra app (https://github.com/fullcontact/webhook-sinatra), I was able to view live production webhooks for experimental verification of results.

Let’s see it in action:

$ gem install proxylocal

After that, just run:

$ proxylocal 3000
Local server on port 3000 is now publicly available via:
http://fp9k.t.proxylocal.com/

Which will give you back a random URL to proxy through. Alternatively, using the —-host switch,

$ proxylocal 4567 --host fc_test
Local server on port 4567 is now publicly available via:
http://fc_test.t.proxylocal.com/

Make a brief call to the FullContactAPI:

https://api.fullcontact.com/v2/person.json?email=michael@fullcontact.com&apiKey=<key>&webhookUrl=http://fc_test.t.proxylocal.com/webhook&webhookId=hello_webhook
WebhookSinatraTester$ ruby myapp.rb
 == Sinatra/1.3.1 has taken the stage on 4567 for development with backup from Thin
 [..]
 >> Listening on 0.0.0.0:4567, CTRL+C to stop
 "Receiving Webhook hello_webhook"
 {"organizations"=>
 [{"name"=>"FullContact, Inc.",
 "isPrimary"=>true,
 "title"=>"Senior Backend Engineer"},
[...]

Given that proxylocal is free and hosted, you really don’t want to run sensitive data through but for the FullContact API it should be more than sufficient for a little testing.

Happy hacking!

Like this post? Share it:

Recent Posts