Content Negotiation with the Pyramid Web Framework

Ever since college, I've always enjoyed working with Python. Even though I mainly work in .NET, every now and again I take a look over to see what's happening in the Python world. Lately, I've been digging into web frameworks to see what it's like to create Web APIs using Python.

A friend of mine, Michael Kennedy, has some awesome Python courses that I've been going through. In one of his courses, he introduces the Pyramid web framework. Even though it's only been about a week, I've thoroughly enjoyed working with Pyramid. It's easy to get started with, works with Python 2.7+ and 3.4+, and there is even support for it in Pycharm.

Now since I really like building APIs, one of the first things I wanted to do was see how content negotiation works in Pyramid.

What is Content Negotiation?

If you're unfamiliar with content negotiation, it is essentially a mechanism that allows clients to state the format they expect the response from the server to be returned as. In the world of HTTP and Web APIs, it allows clients to specify whether to be given JSON or XML, what the language is, and even the content encoding.

Check out the RFC 2616 for a more detailed definition of content negotiation.

Setting up the environment

The first thing I'm going to do is create my work space. I'll open up the command terminal, create new folder on my machine, and navigate into it. Next, I'll create and activate a new Python virtual environment using the following commands.

On OSX and Linux:

$ python3 -m venv pyramid-env $ . pyramid-env/bin/activate (pyramid-env) $ pip install --upgrade pip setuptools

On Windows:

C:\folder> python -m venv pyramid-env C:\folder> pyramid-env\Scripts\activate.bat (pyramid-env) C:\folder> pip install --upgrade pip setuptools

You'll know that activation of the environment was successful if you see the name of the environment to the left of the command terminal.

Next, we're going to need to install pyramid and some other packages into the virtual environment. We can do that easily with pip.

(pyramid-env) $ pip install pyramid factory-boy Faker vobject requests

Creating a simple Pyramid API

Let's start off with setting up the routes and instantiating the application. Here's what my initial app.py file looks like:

from wsgiref.simple_server import make_server from pyramid.config import Configurator

def configure_renderers(config): json_renderer = JSON() json_renderer.addadapter(Customer, lambda p, : p.dict) config.add_renderer('json', json_renderer)

if name == 'main': config = Configurator() config.add_route('customers', '/api/customers') config.add_route('customer', '/api/customers/{name}') config.scan('views') configure_renderers(config)

app = config.make_wsgi_app() server = make_server('0.0.0.0', 6363, app) server.serve_forever()

Using the Configurator, I'm creating two simple API routes and also creating a WSGI application. However, this application still doesn't do anything. The routes have been defined, but there isn't anything setup to handle requests to matching routes. This is the job of views in Pyramid. I created a views.py file and added some methods to handle requests.

from pyramid.httpexceptions import HTTPNotFound from pyramid.view import view_config from customer import CustomerFactory

__CUSTOMTERS = CustomerFactory.create_batch(20)

@view_config(route_name='customers', renderer='json', request_method='GET', accept="application/json") def retrieve_customers(request): return __CUSTOMTERS

@view_config(route_name='customer', renderer='json', request_method='GET', accept="application/json") def retrieve_customer(request): name = request.matchdict['name'] customer = [cus for cus in __CUSTOMTERS if cus.first_name.lower() == name.lower()] return customer[0] if any(customer) else HTTPNotFound()

I created one view method for each of the previously defined routes. The retrieve_customers view method returns a list of customers, while the retrieve_customer view method returns a single customer based on their first name.

From the pyramid.view package, I imported view_config decorator to associate my routes with the methods that will handle them. In addition to the route name, notice I can use view_config to specify the HTTP method and accept header values that should be matched against the request. What's even more interesting is that I can define what render to use. The job of the render is to serialize data returned from Pyramid view method and turn it into string. In the example above, I'm configuring the views to return the customer data in a JSON format.

Generating fake data

Where exactly is this customer data coming from? Well, I didn't feel the need to setup a database or copy/paste values into a file. Instead, I used the factory boy package to generate that fake data for me.

import factory import json

class Customer: def init(self, first_name, last_name, email): self.first_name = first_name self.last_name = last_name self.email = email

def str(self): return json.dumps(self.dict)

class CustomerFactory(factory.Factory): class Meta: model = Customer

first_name = factory.Faker('first_name') last_name = factory.Faker('last_name') email = factory.Faker('email')

Inside of the views.py file, I importCustomerFactory to generate 20 customer records.

Making your first requests

Everything should be in place now to make requests to the API. You can run the application by just issuing the following command in the command terminal.

(pyramid-env) $ python app.py

Based on the configuration, the application should be running at http://localhost:6363. Open up Postman, cURL, or whatever your HTTP tool of choice is and make a request to the /api/customers endpoint with and accept header of application/json.

Here's an example of making a request with cURL.

(pyramid-env) $ curl http://localhost:6363/api/customers -X "GET" -H "Accept: application/json"

Creating custom renderers

Setting up Pyramid to return JSON was pretty simple, but what if I wanted to support different formats? What we can do in this case is create custom renderers.

To do this, you will need to create a renderer factory that conforms to the IRendererFactory interface. When invoked, the factory will need to return a render that conforms to the IRenderer interface.

Here are two examples of render factories. One takes a single customer and returns their Gravatar image, while the second converts the customer information into a vCard.

import vobject from faker import Factory as FakeFactory from hashlib import md5 import requests from io import BytesIO from pyramid import renderers

class GravatarRendererFactory: def call(self, info): def _renderer(value, system): _GRAVATAR_URL_TEMPLATE = 'http://www.gravatar.com/avatar/{}'

request = system.get('request') request.response.content_type = 'image/png'

email = value.email gravatar_hash = md5(email.encode('utf-8')).hexdigest() gravatar_url = _GRAVATAR_URL_TEMPLATE.format(gravatar_hash) gravatar_response = requests.get(gravatar_url) request.response.content_type = 'image/png' return BytesIO(gravatar_response.content) return _renderer

class VCardRendererFactory: def call(self, info): def _renderer(value, system): request = system.get('request') request.response.content_type = 'text/vcard' vCard = VCardRenderer._customer_to_vcard(value) return vCard.serialize() return _renderer

@staticmethod def _customer_to_vcard(customer): fakeFactory = FakeFactory.create()

vCard = vobject.vCard() vCard.add('n') vCard.n.value = vobject.vcard.Name(family=customer.last_name, given=customer.first_name) vCard.add('fn',) vCard.fn.value = '{} {}'.format(customer.first_name, customer.last_name) vCard.add('email') vCard.email.value = customer.email vCard.email.type_param = 'WORK' tel = vCard.add('TEL') tel.value = fakeFactory.phone_number() tel.type_param = 'WORK' tel = vCard.add('TEL') tel.value = fakeFactory.phone_number() tel.type_param = 'HOME' vCard.add('title') vCard.title.value = fakeFactory.job()

Now that the renderers are created, they need to get added to the Configurator. Here's what the updated app.py looks like.

from wsgiref.simple_server import make_server from pyramid.config import Configurator from pyramid.renderers import JSON from customer import Customer from renderers import VCardRendererFactory, GravatarRendererFactory

def configure_renderers(config): json_renderer = JSON() json_renderer.addadapter(Customer, lambda p, : p.dict) config.add_renderer('json', json_renderer) config.add_renderer('vcard', VCardRendererFactory()) config.add_renderer('img', GravatarRendererFactory())

if name == 'main': config = Configurator() config.add_route('customers', '/api/customers') config.add_route('customer', '/api/customers/{name}') config.scan('views')

configure_renderers(config) app = config.make_wsgi_app() server = make_server('0.0.0.0', 6363, app) server.serve_forever()

configure_renderers has been updated to map the new render factories.

Using the new renderers

What I want to be able to do with this API is to make requests for an individual customer, but be able to control what that data looks like. Sometimes I might want JSON data, or a vCard or maybe just a profile picture from Gravatar.

To do this, we have to update the view_config setup in views.py to be aware of the new renderers.

from pyramid.httpexceptions import HTTPNotFound from pyramid.view import view_config from customer import CustomerFactory

__CUSTOMTERS = CustomerFactory.create_batch(20)

@view_config(route_name='customers', renderer='json', request_method='GET', accept="application/json") def retrieve_customers(request): return __CUSTOMTERS

@view_config(route_name='customer', renderer='json', request_method='GET', accept="application/json") @view_config(route_name='customer', renderer='vcard', request_method='GET', accept="text/vcard") @view_config(route_name='customer', renderer='img', request_method='GET', accept="image/png") def retrieve_customer(request): name = request.matchdict['name'] customer = [cus for cus in __CUSTOMTERS if cus.first_name.lower() == name.lower()] return customer[0] if any(customer) else HTTPNotFound()

On the retrieve_customer method, I've added multiple view_config decorators. They all use the same route but the renderer and accept parameters are different. So if I issue a request for a customer using a text/vcard value in the Accept header of the HTTP request, I expect to get back the customer data as in vcard format. Let's see what this looks like in Postman.

Here's a request with the Accept header set to application/json.

Here's a request with the Accept header set to text/vcard.

Here's a request with the Accept header set to image/png.

Since I'm generating fake emails, Gravatar is going to return the default image. You could easily swap in your own email address to try it out.

Cleaning things up

Everything is running the way I wanted, but there's one last thing that's bugging me. Having to place multiple view_config decorators on top of a view method is bothering me. What I would like to do is place one decorator on a view method and have it select the correct renderer from a mapping of supported media types.

What I decided to do was create another custom renderer; a NegotiatingRender. Here's what it looks like.

class NegotiatingRendererFactory: _CONNEG_MAPPINGS = { 'text/plain': renderers.string_renderer_factory }

def init(self, mappings=None, **kw): if mappings is None: mappings = {}

NegotiatingRendererFactory._CONNEG_MAPPINGS.update(mappings) NegotiatingRendererFactory._CONNEG_MAPPINGS.update(kw)

def call(self, info): def _render(value, system): request = system.get('request') accept_header = request.headers['accept']

for key in NegotiatingRendererFactory._CONNEG_MAPPINGS: if key in accept_header: negotiated_render = NegotiatingRendererFactory._CONNEG_MAPPINGS[key] result = negotiated_render(info)(value, system) return result

return _render

Essentially what this renderer does is match the accept header in the request to a renderer that's been mapped to respond to that header value. In the app.py file, the configure_renderers method needs to be updated to register the NegotiatingRendererFactory.

def configure_renderers(config): json_renderer = JSON() json_renderer.addadapter(Customer, lambda p, : p.dict) config.add_renderer('json', json_renderer) config.add_renderer('vcard', VCardRendererFactory()) config.add_renderer('img', GravatarRendererFactory())

mappings = { 'application/json' : json_renderer, 'text/vcard': VCardRendererFactory(), 'image/png': GravatarRendererFactory() } negotiator = NegotiatingRendererFactory(mappings) config.add_renderer('negotiate', negotiator)

With that in place, we should be able to reduce the number of view_config decorators to one that will use the negotiate renderer instead.

from pyramid.httpexceptions import HTTPNotFound from pyramid.view import view_config from customer import CustomerFactory

__CUSTOMTERS = CustomerFactory.create_batch(20)

@view_config(route_name='customers', renderer='json', request_method='GET', accept="application/json") def retrieve_customers(request): return __CUSTOMTERS

@view_config(route_name='customer', renderer='negotiate', request_method='GET') def retrieve_customer(request): name = request.matchdict['name'] customer = [cus for cus in __CUSTOMTERS if cus.first_name.lower() == name.lower()] return customer[0] if any(customer) else HTTPNotFound()

Trying out the same requests as before, you'll notice the results should be almost identical. The code also feels much cleaner in my opinion, and it's pretty easy to plug in another renderer.

Conclusion

I would love some feedback on this approach though. Considering I've only been using Pyramid for a few weeks, there might be a better way to do this.

I shared the code in a Github repo, so check it out and let me know what you think.