This article series goes way beyond the surface-level “Create a Phoenix API” articles that are so typical. We walk through, and learn enough to understand, the gory details.

In this first installment, we explore the “why an API?” question, our motivation for spending time on what could be argued is a distraction at this early stage in our app development learning curve, and we stumble through the process of adding a REST API to our Phoenix app.

Why an API? Why now?

That’s a reasonable question. The TL;DR: We decided to start building native Anroid & iOS apps for Mindery.

There are lots of “best practices” reasons for investing attention in a SaaS application’s API as a “first class” effort. We will elaborate on those later; in the meantime, you can Google “API-first” to see for yourself.

In our case, we decided the other day to build an Android Mindery client and an iOS Mindery client (i.e., native apps to interact with Mindery functionality). If Mindery was a moble-only application, it would make sense to look at some of the “back-end as a service” options like Firebase and its competitors.

Since we are intentionally focused on Elixir, our back-end is already using the Elixir/Phoenix stack, so that is not an option we are considering. Given that, we need to expose an API to these mobile apps, so that made thinking about a robust (externally-facing) API a must-have-now.

Don’t Phoenix Apps Already use an API?

Your may be thinking:

Wait! Isn’t the Phoenix front-end already talking to the Elixir back-end via an API?

Technically, yes. The “context” modules of every Phoenix app make up that (internal) API. Our Phoenix views are using the API defined by Phoenix and LiveView to interact with the back-end, and this works great.

In fact, maybe we could expose this functionality to the outside world somehow … by inventing some new way of defining an API, and we could probably even argue that we could somehow do that securely. But let’s not. Let’s create an industry-standard JSON API that the real world expects to see.

We could argue that exposing the Phoenix-internal APIs to the outside world can be done safely. We could also sell you screensavers showing flying pigs, but that doesn’t make real pigs fly.

Getting on with It

Enough background; let’s get on with it. Some parts of this next section can be found in the many “How to create an API in your Phoenix App” articles–but then they stop. We dig in way deeper.

1. Add a Rest API to our Phoenix App

Our app is a simple CRUD app with just a couple of tables; in fact, the most complex functionality is the user authentication logic created by running mix phx.gen.auth (which we have already done). To add an API, we wil take advantage of the Phoenix scaffolding for JSON API functionality via the phx.gen.json mix command.

mix phx.gen.json Context Schema db_table is the general syntax, where Context is the name of our context, Schema and db_table are the singular and plural names for our DB schema and table, respectively.

So, if I wanted to create an API to access our Accounts context and its Users functionality, the command would be mix phx.gen.json Accounts User users. This would happily do its Phoenix scaffolding magic by trying to create a bunch of things like this:

* creating lib/mindery_web/controllers/user_controller.ex
* creating lib/mindery_web/controllers/user_json.ex
* creating lib/mindery_web/controllers/changeset_json.ex
* creating test/mindery_web/controllers/user_controller_test.exs
* creating lib/mindery_web/controllers/fallback_controller.ex
* creating lib/mindery/accounts/user.ex
* creating priv/repo/migrations/20221129120234_create_users.exs
* creating lib/mindery/accounts.ex
* injecting lib/mindery/accounts.ex
* creating test/mindery/users_test.exs
* injecting test/mindery/users_test.exs
* creating test/support/fixtures/users_fixtures.ex
* injecting test/support/fixtures/users_fixtures.ex

But wait! We already created the Accounts context and the Users schema and table when we ran phx.gen.auth.

That’s exactly why this won’t work. Instead, we need to exclude those schema and context parts from our command.

mix phx.gen.json Accounts User users --no-context --no-schema

These --no-context and --no-schema flags tell the phx.gen.json mix task to skip trying to create (and colliding with) the context and schema definitions we already have, so it produces this:

* creating lib/mindery_web/controllers/user_controller.ex
* creating lib/mindery_web/controllers/user_json.ex
* creating lib/mindery_web/controllers/changeset_json.ex
* creating test/mindery_web/controllers/user_controller_test.exs
* creating lib/mindery_web/controllers/fallback_controller.ex

Add the resource to your :api scope in lib/mindery_web/router.ex:

    resources "/users", UserController, except: [:new, :edit]

So far, so good. (-ish)

I added the “-ish“ because following this recipe of steps (including that note about updating the router entry for api: with resources "/users", UserController, except: [:new, :edit]) does give us an externally-callable JSON API, but it’s not yet very useful.

There is no security (yet); I don’t yet know if we can use the authentication functionality we got from running mix phx.gen.auth earlier to create authentication/authorization mojo for our Phoenix/Liveview app.

But that’s OK; we’ll add authentication to our API later. Happily ignoring security for now, here is what we can do:

From our terminal: curl localhost:4942/api/users/1 results in: {"data":{"id":1}}.

In case you don’t know, curl is an ancient but powerful Unix command used for sending HTTP messages (and a bunch of other protocols). Type man curl from your favorite WSL, MacOS or other flavor of linux terminal session to get an overview.

{"data":{"id":1}} is a valid response to our API call, but it’s not very useful. I only get the value of the id echoed back to me. What about the other user attributes?

The JSON and APIs hexdocs (v 1.7.1) were not very helpful initially here, so after digging through the generated files, I stumbled across this in our user_json.ex

  @doc """
  Renders a single user.
  """
  def show(%{user: user}) do
    %{data: data(user)}
  end

That maps (mentally) to that API response: {"data":{"id":1}}. That data(user) call in the show() function is returning just the id value. Looking further down in user_json.ex, I found that data() function:

  defp data(%User{} = user) do
    %{
      id: user.id 
    }
  end

Aha! So it looks like I’ll need to manually update this to include the rest of the attributes I want to see returned in the API response. But wait!! Looking at that JSON and APIs hexdocs again, I see that in their example, the scaffolding DID include all the fields in their data() definition.

  defp data(%Url{} = url) do
    %{
      id: url.id,
      link: url.link,
      title: url.title
    }
  end

Educated guess: mix phx.gen.json does NOT read existing schemas when building JSON-handling scaffolding, so it cannot know what fields should be rendered in JSON output.

The example in Hexdocs is creating the schema at the same time as running the JSON generator: mix phx.gen.json Urls Url urls link:string title:string

The resulting data() function in url_json.ex renders those fields (plus the always-created id) specified in that mix phx.gen.json command.

2. Make the Rest API Actually Useful

After making updates to build out the rendering logic in user_json.ex, I get a more useful response to our API call: {"data":{"id":1,"email":"my-super-secret-email@something.com","confirmed_at":null}}

Our JSON rendering logic is now:

  defp data(%User{} = user) do
    %{
      id: user.id,
      email: user.email,
      confirmed_at: user.confirmed_at
    }
  end

There is almost certainly more work to do here to make sure user_json.ex does all the rendering we will need for our API calls, but a quick side-trip I took to exercise our nascent API using Postman as a way to start sharing its design with our team took a frustrating turn.

3. Mapping `curl` commands to Postman Conventions

My joy at finding and plugging gaps in our user_json.ex logic was almost spoiled by a newbie rabbit hole I fell into when I started to build out the Postman collection for our API.

In the Hexdocs “URL” api example, they use curl commands like this:

curl -iX POST http://localhost:4000/api/urls \
   -H 'Content-Type: application/json' \
   -d '{"url": {"link":"https://elixir-lang.org", "title":"Elixir"}}'

The above curl syntax sends an HTTP POST to the specified URL/port number and the api/urls endpoint. The HTTP POST command is what we send to create a new resource, so this command will try to invoke the create function in the url controller. The -H part tells the server to expect JSON, and the -d part is the arguments (AKA attrs or params) formatted as a JSON structure.

Applying this to our app, the curl syntax is:

curl -iX POST http://localhost:4942/api/users \
-H 'Content-Type: application/json' \
-d '{"user": {"email":"super@secret.org", "password":"Sup3rS3cr#tP@ssw0rd"}}'

And it works great, responding with:

HTTP/1.1 201 Created
cache-control: max-age=0, private, must-revalidate
content-length: 64
content-type: application/json; charset=utf-8
date: Fri, 08 Sep 2023 22:09:20 GMT
location: /api/users/4
server: Cowboy
x-request-id: F4ML5KiH6iSVgRoAAAEh

{"data":{"id":4,"email":"super@secret.org","confirmed_at":null}}

Mapping that to Postman

In my app terminal window (app log), I can see the paramaters that get passed in that API call: Parameters: %{"user" => %{"email" => "super@secret.org", "password" => "[FILTERED]"}}

The question is: How does that translate into what we should put into Postman for that POST to http://localhost:4942/api/users? It would be logical to think that a Postman parameter be populated with %{"user" => %{"email" => "super@secret.org", "password" => "Sup3rS3cr#tP@ssw0rd"}} or perhaps even more logical, one could guess that we populate a parameter in Postman with that exact string from the curl command: {"user": {"email":"super@secret.org", "password":"Sup3rS3cr#tP@ssw0rd"}}.

One of these choices must work, right?

I flailed around with this for way longer than I will admit; I even created another Phoenix app and repeated the steps to create a JSON API in it because I thought maybe my first app was broken. Good news: Both apps failed in the same ways with my incorrect guesses. Bad news: My guesses were incorrect.

After letting my brain not consciously think about this for a while, it dawned on me that translating curl commands to Postman’s conventions must be a solved problem, so I did some Googling.

Sure enough, the simple path to finding the answer was to simply import the curl command into Postman. Doing that revealed that the -d part of the curl command becomes the body of the Postman command, and it is in the form of “raw” body content, like this:

Isn’t it amazing how easy things seem after spending hours and hours flailing around to find the light switch?

Installment 1 Recap

In this installment, we have learned:

We can add an API to an existing Phoenix app via the mix phx.gen.json syntax detailed above.
- To add a JSON API to interact with an app’s existing context and its schemas, we will want to use this syntax:

mix phx.gen.json <ContextName> <SchemaName> <tableNamePlural> --no-context --no-schema

The resulting scaffolding is incomplete. (That’s why it’s called “scaffolding.”) We need to be programmers and fill in the details in the <schema_name>.json that it creates, and we have additional work to do. (That’s why we’re called “programmers.”)
We need to map our existing CRUD functionality to the HTTP post, put/patch, show, delete verbs. In our app, we had already created the authentication logic provided by mix phx.gen.auth, so our API calls to POST which would normally map to a create() function needed to be mapped (by programmer-me) to the Accounts.register_user() function created by the phx.gen.auth process.
We need to document our API using industry-standard best practices. Postman is the leading API design, mocking and documentation tool, so understanding how to document our API using Postman’s conventions is a good start.

We will dig deeper into finishing mapping our controller(s) and context(s) to those HTTP verbs and adding any remaining JSON formatting logic, along with making solid progress on API documentation, in the next installment. Stay tuned.

Deep Dive: API-first with Phoenix

by JSH

Why an API? Why now?

Don’t Phoenix Apps Already use an API?

Getting on with It

1. Add a Rest API to our Phoenix App

So far, so good. (-ish)

2. Make the Rest API Actually Useful

3. Mapping `curl` commands to Postman Conventions

Mapping that to Postman

Installment 1 Recap

Deep Dive: API-first with Phoenix

by JSH

Why an API? Why now?

Don’t Phoenix Apps Already use an API?

Getting on with It

1. Add a Rest API to our Phoenix App

So far, so good. (-ish)

2. Make the Rest API Actually Useful

3. Mapping curl commands to Postman Conventions

Mapping that to Postman

Installment 1 Recap

3. Mapping `curl` commands to Postman Conventions