While writing my post about working with JSON columns, I got into a rabbit hole of how Rails actually parses form variables. Most of the functionality exists in Rack, which is a web interface that allows Rails to mount onto popular application servers like Puma, Unicorn, Thin, and Passenger. Let’s dig into how Rack goes about parsing the form variables you send to your application.

Wait, what are form variables?

In many cases, when you submit a form on the web, the form will send data to a server as form variables. As opposed to JSON, which have a content type of application/json, form variables can be one of three things:

  1. application/x-www-form-urlencoded: This is most likely what your Rails form is passing through
  2. multipart/form-data: Used for uploading files
  3. text/plain: Less common for Rails forms, but it’s just straight text

We’re going to focus on the first one, application/x-www-form-urlencoded, since that’s the most common one you’ll find. Let’s say you submit a form that looks like this:

captionless image

If you peek into the network tab of the Chrome developer tools, you’ll see the raw value that’s passed in:

captionless image

That looks super overwhelming. Let’s unescape and clean it up a bit so it makes a little more sense:

form_variables = "utf8=%E2%9C%93&_method=patch&authenticity_token=%2BgjNPFCovlvOIhaHPnPKIHEEG8q3ZH%2F8t9mQwh0EKaMSlx4W04SVEmmiyXixKb8JNdmKAmiG7G%2FZeU4aeGVxpA%3D%3D&article%5Btitle%5D=Title&article%5Bauthor%5D=author+name&article%5Bbody%5D=body+text&article%5Bcomments%5D%5B%5D=comment+1&article%5Bcomments%5D%5B%5D=comment+2&commit="
URI.decode_www_form_component(form_variables).split("&")
=> ["utf8=✓",
 "_method=patch",
 "authenticity_token=+gjNPFCovlvOIhaHPnPKIHEEG8q3ZH/8t9mQwh0EKaMSlx4W04SVEmmiyXixKb8JNdmKAmiG7G/ZeU4aeGVxpA==",
 "article[title]=Title",
 "article[author]=author name",
 "article[body]=body text",
 "article[comments][]=comment 1",
 "article[comments][]=comment 2",
 "commit="]

As you can see, all of the values are separated by an ampersand, similar to how you’d see them in query parameters.

Jumping into the Rack query parser

Now that we know what form variables look like, let’s take a look at how they get parsed out. Remember that for these form variables, we eventually want to end up with a Hash of the parameters and values like so:

{"utf8"=>"✓",
 "_method"=>"patch",
 "authenticity_token"=>"+gjNPFCovlvOIhaHPnPKIHEEG8q3ZH/8t9mQwh0EKaMSlx4W04SVEmmiyXixKb8JNdmKAmiG7G/ZeU4aeGVxpA==",
 "article"=>{"title"=>"Title", "author"=>"author name", "body"=>"body text", "comments"=>["comment 1", "comment 2"]},
 "commit"=>""}

This Hash is what you’ll end up seeing in your Rails controller, for instance. The function in Rack that’s doing all the work is [parse_nested_query](https://github.com/rack/rack/blob/ab008307cbb805585449145966989d5274fbe1e4/lib/rack/query_parser.rb#L60). Here’s what it looks like:

def parse_nested_query(qs, d = nil)
  return {} if qs.nil? || qs.empty?
  params = make_params

  (qs || '').split(d ? (COMMON_SEP[d] || /[#{d}] */n) : DEFAULT_SEP).each do |p|
    k, v = p.split('='.freeze, 2).map! { |s| unescape(s) }

    normalize_params(params, k, v, param_depth_limit)
  end

  return params.to_params_hash
rescue ArgumentError => e
  raise InvalidParameterError, e.message
end

The first part is straight-forward — we’re splitting up that huge string into individual key/value pairs (by splitting on & and then =), and then unescaping them so they look sane. Finally, we pass that key and value into normalize_params. And at the end, we magically have our parameters!

Not so fast. Let’s take a look at what normalize_params is doing:

Well…. that looks a little more daunting, doesn’t it? We’ll dive into what this function is doing next.

Recursion, Regex, and You

On lines 4 and 24 you’ll see that normalize_params calls itself, indicating that it’s a recursive function. If you haven’t run into recursion yet, here’s a helpful primer on recursion in Ruby.

Recursion is nice here because we can have nested parameters (hashes in hashes in arrays, and so forth). This is a good use case for recursion, especially if you have tests to cover it. While this could have been iterative (with the depth parameter), recursion here is a little cleaner as to what’s going on.

The next big hurdle (at least for me) was understanding what these regex conditionals were doing. They’re a little confusing at first, but I’ll summarize them here:

  1. \A[[]]*([^[]]+)]* on line 4 Gets the first word in the key, inside or outside of brackets. For “foo[bar]”, we’d match “foo”, and for “[][bar]”, we’d match “bar”. You can play with the regex here
  2. ^[][([^[]]+)]$ on line 24 Finds the first word after a “[]”. For “[][foo]”, we’d match “foo”. You can play with the regex here
  3. ^[nameless link](.+)$ on line 24 Finds the first word and closing brackets after a “[]”. For “[][foo][]”, we’d match “[foo][]”. You can play with the regex here

Now that we know what these regexes are doing, you might start to see how the function builds a Hash of parameters. For example, a string like this makes sense in Ruby: foo[bar][baz]. But forms don’t have a convenient way to indicate an Array of values, so they use [] to indicate there’s an Array.

The function is basically walking through a parameter, left to right, and inserting Hash keys or appending Array values. Let’s see what that looks like with a few of the parameters:

k = article[body]
v = "body text"

1. article[body] --> found "article"
     params = {article:}
     normalize_params(params, "[body]", "body text", depth)
2. [body] --> found "body", nothing else so set the value
   params = {article: body: "body text"}

# let's try one of the Array parameters

k = article[comments][]
v = "comment 1"

# 1. article[comments][] --> found "article", it's already there
  params = {article: body: "body text"}
  normalize_params(params, "[comments][]", "comment 1", depth)
# 2. [comments][] --> found "comments"
  params = {article: body: "body text", comments:}
  normalize_params(params, "[]", "comment 1", depth)
   
# 3. "[]" --> found an Array, append the value
  params = {article: body: "body text", comments: ["comment 1"]}

Summary

Well that was a fun adventure! We got to look at some of Rack’s internals that powers many, many Rails apps out there, and we got to experience some recursion in the wild. I definitely recommend perusing through Rack’s codebase to see how it actually works under the hood. Or better yet, check out ActionDispatch in Rails and see what happens before a request hits your controller.