Testing Oddities with Amnesia

Background

I'm starting to write the tests for what I'm hoping will become a very simple set of tests to validate my Amnesia-based database.  The first test looks like this:

  PasteDB.Paste.wait
  test "Starting with an empty database" do
    assert(Amnesia.transaction! do PasteDB.Paste.keys end == [])
  end

Which makes sure my test database is empty before I start.

The second test is a little longer and more complex and looks like this:

  test "Direct insertion into Amnesia, sample data" do
    id = UUID.uuid1
    date = Timex.Date.now(:secs)
    expires = Timex.Date.now(:secs) + 2000
    title = "This is sample paste number one"
    public = true
    ircchannels = [:foo, :bar]
    previous_version =  nil
    content = "This is the first public paste content"
    assert(Amnesia.transaction! do %PasteDB.Paste{id: id, title: title, public: public, ircchannels: ircchannels, date: date, expires: expires, previous_version: previous_version, content: content} |> PasteDB.Paste.write end)

    Amnesia.transaction! do
      assert [id] == PasteDB.Paste.keys
    end

It inserts a single record into the (assured?) empty database above which means that when we pull back the list of keys, our list of keys should be an array with a single key which matches [id]

However when you run the test, you get this:

red@nukefromorbit:pastenix$ mix test
Running Pastenix.Endpoint with Cowboy on port 4001 (http)
...

Finished in 0.07 seconds (0.06s on load, 0.01s on tests)
3 tests, 0 failures

or this:

Running Pastenix.Endpoint with Cowboy on port 4001 (http)
..

  1) test Starting with an empty database (PasteDBTest)
     test/pastedb_test.exs:6
     Assertion with == failed
     code: Amnesia.transaction!() do
             PasteDB.Paste.keys()
           end == []
     lhs:  ["1efcefac-86c2-11e4-a9c8-600308a99134"]
     rhs:  []
     stacktrace:
       test/pastedb_test.exs:7

Randomly.

So what's going on here?

If I enable tracing in ExUnit by modifying the ExUnit.start function in test_helper.exs:

ExUnit.start(trace: true)

I get the gift of clarity.  Here's my two cases, spot the difference:

red@nukefromorbit:pastenix$ mix test
Running Pastenix.Endpoint with Cowboy on port 4001 (http)

PasteDBTest
  * Starting with an empty database (0.07ms)
  * Direct insertion into Amnesia, sample data (16.9ms)

PastenixTest
  * the truth (0.00ms)


Finished in 0.07 seconds (0.06s on load, 0.01s on tests)
3 tests, 0 failures

and

red@nukefromorbit:pastenix$ mix test
Running Pastenix.Endpoint with Cowboy on port 4001 (http)

PastenixTest
  * the truth (0.00ms)

PasteDBTest
  * Direct insertion into Amnesia, sample data (18.3ms)
  * Starting with an empty database (0.1ms)

  1) test Starting with an empty database (PasteDBTest)
     test/pastedb_test.exs:6
     Assertion with == failed
     code: Amnesia.transaction!() do
             PasteDB.Paste.keys()
           end == []
     lhs:  ["9327d4f0-86c2-11e4-aba2-600308a99134"]
     rhs:  []
     stacktrace:
       test/pastedb_test.exs:7


Finished in 0.07 seconds (0.05s on load, 0.02s on tests)
3 tests, 1 failures

Two differences immediately jump out at me.  The first is the ordering in which the two test suites PastenixTest and PasteDBTest execute.  The second is that the insertion test happens before the test for an empty set, the latter is obviously the cause of the test fail but is the former what causes the issue?

Removing all the other tests makes no difference, the tests still randomly fail.

So my tests all run in parallel?

Yup - pretty cool huh?  Mostly, well yes - can this be overridden?

The docs show an option you can pass to ExUnit.start which allows you to set the number of concurrent jobs to run so I can just set that to 1 and my tests will all run, one at a time in a nice predictable way right?

The assumptions of an imperative programmer...

Nope.

So I went digging into the ExUnit source-code to find out why and this is what I found in runner.ex:

{run_us, _} =

:timer.tc fn ->

EM.suite_started(config.manager, opts)

loop %{config | sync_cases: shuffle(config, sync),

async_cases: shuffle(config, async)}

end

EM.suite_finished(config.manager, run_us, load_us)

EM.call(config.manager, ExUnit.RunnerStats, :stop, @stop_timeout)

Wait, what?

The test cases are shuffled(!) before execution!!?!

Let's look at the shuffle function:

defp shuffle(%{seed: 0}, list) do

Enum.reverse(list)

end

defp shuffle(%{seed: seed}, list) do

_ = :random.seed(3172, 9814, seed)

Enum.shuffle(list)

end

So, if I pass seed: 0 in my ExUnix.start function then I avoid the shuffle. Success?!!??!?

So what did I learn?

If I use seed: 0 then what have I really achieved?  I've achieved subversion of a test that I didn't think of but the developer of ExUnit did.  In a concurrent world you have no guarantees as to the order in which things happen.

My mistake was in the first first paragraph of this post.  "The first test...".  First implies serialization which is the opposite of concurrency.  Serialization is the opposite of what I want.

Like a good imperative programmer my tests had side-effects by the very nature of my approach to the problem.  My insertion test had the side-effect of inserting a record changing state outside of itself.

This broke everything.

I'm clearly still not thinking in a functional way.  If something as simple as a shuffle of my tests (which is what concurrency does in the real world) can break my application's test suite then I'm doing it wrong.  My instinctive response was to try and serialize it to make it fit in my serial world.

So Thank You ExUnit developer for having my back when I didn't realize I left it exposed.

Mindset change required.  The battle continues...

It starts with the data...

The Plan

I just posted the initial TODO list for pastenix into the README.md file.  It currently reads as follows:

• General
  • Use Amnesia for Storage
  • Avoid Javascript as much as is possible
• Pastebin Functionality
  • Ability to Post Paste
  • Ability to View Paste
  • Ability to Edit Paste
    • Edits produce new pastes with reference to old paste (version contro)
  • Ability to Post Private Pastes
• IRC Functionality
  • Ability to publish Public Pastes on specified IRC channels
  • Query Bot for Latest X paste subjects
  • Ability to have Bot take notes from Channel and publish in a paste
• Commandline Functionality
  • Pipe to Paste

With an rough finger-in-air approximation we can start to put finger to keyboard as it were.  We have some decisions to make based around data-structures.

Why Mnesia?

This is a learning experience for me.  I've done databases ten thousand times, I want to try something different.  The biggest disadvantages to using mnesia for this project are:

1. The 2Gb Database limit on disk (unless you shard - may look at that later after my first million in ad revenue) ;-)
2. Difficulty in doing things like efficient plaintext searches of the database.

Mnesia (or Amnesia in Elixir)

mnesia is the native name in Erlang for an abstraction over ets and dets.  Way too much information to try and communicate here but the tl;dr is:

1. ETS - in-memory, ridiculously fast key/value store. Data lost on process death.
2. DETS - on disk very slow but persistent key/value store with a 2Gb database limit.
3. mnesia - Abstraction over the above giving you the ability to create tables with:
1. Arbitrary distribution of multiple nodes.
2. Arbitrary copies of tables in both memory and disk.
3. Custom sharding.
4. A Pony.

Elixir allows you to directly call any erlang function from any module by using the syntax :erlangmodulename.function(args).  As such, you can call mnesia directly using that format.

On github however there is a nice abstraction which I have chosen to use: https://github.com/meh/amnesia

What I hate about RDBMSs

I hate doing data serialization and de-serialization.  I loathe and detest the concept of having to take the data native to my application and format it the right way to go into the database and vice-versa.  This data "impedance mismatch" is a pain.

mnesia (and consequently Amnesia) store native Erlang terms which means you can store anything any way you want.  Less code, more awesome.

Creating our table

Here's the table definition that I've chosen:

defdatabase PasteDB do
  deftable Paste, [ :id, :title, :public, :ircchannels, :date, :expires, :previous_version, :content], type: :ordered_set do end
end

Proposed Datatypes:

1. :id - String representation of a UUID.  Randomly generated (Hello UUID Module)
2. :title - String representation of the Paste Title
3. :public - Boolean value as to whether something is public or not.  Guess that's :true or :false.
4. :ircchannels - Array of :atoms which represent pre-programmed irc channel / server combinations.
5. :date - Integer representation a 'la UNIX epoch time.  Considered using the record found in Timex.Date but we will need a very fast way to do date sorting later so keeping it simple.
6. :expires - Integer representation of UNIX epoch time when the Paste should be removed from the system.
7. :previous_version - String representation of the UUID of the previous version of the Paste.  Should this be nil on first Paste or set to something else?
8. :content - Arbitary binary data.

This is where I'm going to start, time to build the table and attempt to build the test cases to exercise it.

Travis and Inch - Continuous Integration and Documentation Coverage in Elixir

The why...

Sure, it may compile locally for me but after working with a friend to install my previous version and realizing that so much of what I do is dependent upon my local mnesia schema and all that jazz it became clear to be I needed an "External View" of what my project would look like as a fresh install every time.

The way you do that is with "Continuous Integration" - a build environment which automatically builds and tests on every push you do to your repository.

Documentation?  I hate writing it but I service which bugs me when I don't document something is useful for drilling good habits.

Travis and Inch

Travis, travis-ci.org provides free Continuous Integration services for Open Source projects.  So, to start with a completely blank slate I pulled down Phoenix v0.7.2 and built a brand spanky new project called "pastenix" and uploaded it to github at https://github.com/redvers/pastenix

To add Travis support I stole^H^H^H^H^Hcopied the .travis.yml file from the Phoenix framework directory, registered myself on travis-ci.org and waited for the test result...

BOOM

red@nukefromorbit:pastenix$ MIX_ENV=docs mix inch.report
** (Mix.Config.LoadError) could not load config config/docs.exs
    ** (Code.LoadError) could not load /Users/red/projects/elixir/pastenix/config/docs.exs
    (mix) lib/mix/config.ex:141: Mix.Config.read!/1
    (mix) lib/mix/config.ex:153: anonymous fn/2 in Mix.Config.read_wildcard!/2
    (elixir) lib/enum.ex:1261: Enum."-reduce/3-lists^foldl/2-0-"/3
    (stdlib) erl_eval.erl:657: :erl_eval.do_apply/6
    (stdlib) erl_eval.erl:441: :erl_eval.expr/5
    (elixir) lib/code.ex:140: Code.eval_string/3

So, what's going on here?

The default config/config.exs file in a new phoenix application allows you to modify various settings at compile-time depending on which "Mix Environment" you're running in.  For example, your DEV environment will do automatic code-loading when it detects changes in the source-code - not behavior you want reflected in production!

The simplest fix certainly is to remove the include directive at the end of the file like so:

#import_config "#{Mix.env}.exs"

The biggest issue with doing this is that you've now lost that valuable functionality.

You can't just create an empty docs.exs as the import_config does an eval and it can't eval nil:

red@nukefromorbit:pastenix$ MIX_ENV=docs mix inch.report
** (Mix.Config.LoadError) could not load config config/docs.exs
    ** (ArgumentError) expected config file to return keyword list, got: nil
    (mix) lib/mix/config.ex:141: Mix.Config.read!/1
    (mix) lib/mix/config.ex:153: anonymous fn/2 in Mix.Config.read_wildcard!/2
    (elixir) lib/enum.ex:1261: Enum."-reduce/3-lists^foldl/2-0-"/3
    (stdlib) erl_eval.erl:657: :erl_eval.do_apply/6
    (stdlib) erl_eval.erl:441: :erl_eval.expr/5
    (elixir) lib/code.ex:140: Code.eval_string/3

The way that I chose to solve it was to copy the dev.exs file to docs.exs.  As this is only for the documentation build one can argue that there's no useable settings to be overridden.  If anyone has a suggestion for a cleaner solution, please comment below.

Lastly, you want to add:

{:ex_doc, "~> 0.6", only: :docs},  {:inch_ex, "~> 0.2", only: :docs},

to your deps to make sure the code that does the actual documentation publication and coverage testing is included.

Pushing this change, results in a clean build!

Getting the Perty Buttons on your github page.

Github renders your README.md (markdown) file on the front-page of your project's github page so to add the two buttons to your page add them to the top of your README.md

For pastenix, they looked like this:

[](https://travis-ci.org/redvers/pastenix)
[](http://inch-ci.org/github/redvers/pastenix)

Elixir - I'm starting again and this time, doing it "right"...

Background

About three weeks ago I discovered Elixir, a language which compiles down to Erlang Virtual Machine bytecode.  Much awesome has been said about the language so I won't repeat it here but instead talk about why I'm starting again.

After reading "the books"[tm] I started blundering into writing an application I lovingly referred to as "pastenix".  Three things I've always found useful:

1. Pastebin.
2. IRC bot which does note-taking / link sharing.
3. Command-line note-taker ("Pastebin with a pipe[tm]")

I blundered through for a few days and achieved the grand status of being the only person on Earth capable of taking an entire ecosystem designed around resiliency and availability and producing something that would crash randomly.

So I'm starting again...

To try and instill discipline required to keep this puppy on track I'm going to commit to do all the right things and document appropriately.  I'll write unit tests, documentation, and actually use spec even though it's not compulsory.  I'll use Github for hosting, Travis for building and Inch for keeping me honest in my documentation.

As I hit snags I'll document here.