Since the announcement of Sihl, we've been working towards the first release.
Version 1.0.0 will stabilize the API, so we can write documentation and encourage others to use Sihl. In this post, we summarize the work so far and what is next on our to-do list.
What has been done
We've been working on three major topics: persistence, app configuration management and user management.
Persistence
Sihl doesn't come with an ORM, but still aims to be database-agnostic. The database-specific implementation details need to be hidden behind the Persistence API. Two questions quickly arise:
"How does that API look like?" and "Which language construct should we use to separate interface and implementation?"
Separating interface and implementation
Sihl was initially developed for a project that uses MariaDB. There was quite some MySQL/MariaDB specific code in @sihl/core. By extracting it into a package @sihl/mysql, we started to get an idea how the Persistence API could look like. The interface is defined in @sihl/core while @sihl/mysql provides an implementation.
Let's compare two possible approaches to achieve this.
The "types and functions" approach
To keep it simple and stupid, we began to define the API as record types and function signatures.
Conceptually, the API comprises of things like Database, Connection, Transaction and QueryResult. You can retrieve a Connection from a Database handle for instance. So we have a type database for the thing, and a type containing function signatures that work withdatabase.
type databaseType;
type database = {
database: databaseType,
setup: Common_Config.Db.Url.t => database,
end_: database => unit,
withConnection: (database, connection => Async.t('a)) => Async.t('a),
clean: t => Async.t(unit),
};
The type databaseType is abstract so the implementation is hidden.
Let's have a look at the usage:
let getUser = (database, ~userId) => {
database.withConnection(database.database, connection => {
...
})
We can define getUser without caring about the implementation of database. On the other hand, the definition of two separate types for the thing, (databaseType) and for the set of functions for that thing (database) is verbose.
An even bigger concern is breaking expectations. This approach might surprise an experienced Reason developers, and thus making it harder for them to understand the code. The idiomatic way requires us to use the F-word.
Module Functors
Database-agnosticism could be a textbook example for Functors. We don't want to explain here what they are and how to use them, since others already have done a great job. Instead, we'd like to share our experience using them for this particular problem.
Initially, this blog post was titled "To Functor Or Not To Functor" and we planned to discuss the up and downsides of using them in general. Instead, we discuss the points that helped our decision to use Functors.
We implemented the "types and functions" approach first. We think that this approach should be chosen first and Functors should be considered only if they provide additional value. This episode of the wonderful ReasonTown podcast briefly mentions the mistake of using Functors by default. A nice example where Functors were deliberately avoided for the sake of simplicity is Serbet (which Sihl uses internally, because of its simple API).
Why did we end up using them anyway?
Explicit compile-time dependencies between apps
When we compose a Sihl project, we have to configure it statically so we can use services across apps. This is done in the Sihl.re file, which might look like the following:
module Common = SihlCore.Core.Common;
module Persistence = SihlMysql.Mysql.Persistence;
module Authz = SihlAuthz.Rbac.Authz;
module App = SihlCore.Core.MakeApp(Persistence);
module Users = SihlUsers.Users.Make(Persistence, Authz);
It's a nice visualization of Sihl apps and their dependencies.
Information hiding
With module signatures we can precisely control what is visible to the consumer. This feature is important, because it helps to keep the API as small as possible.
(We don't discuss BuckleScript specific features that allow information hiding without module signatures, because we plan to target native as well.)
Idiomatic way
We don't want to break expectations of Reason programmers, therefore we use idioms, so it's easier to contribute to Sihl.
Framework users don't care
One of the main design goals of Sihl was friendliness towards developers, without experience in functional programming and Reason. We don't want anyone to encounter the F-word within the first days of using Sihl.
Unfortunately, framework users have to use Functors in the Sihl.re file shown above. We think it's possible to use Functors in one place when setting up the project, without caring what they actually are.
The API
Let's have a brief look at the actual Persistence API that powers Sihl.
module type CONNECTION = {
type t;
let raw:
(t, ~stmt: string, ~parameters: option(Js.Json.t)) => Async.t(Js.Json.t);
let getMany:
(t, ~stmt: string, ~parameters: option(Js.Json.t)) =>
Async.t(Belt.Result.t(Result.Query.t(Js.Json.t), string));
let getOne:
(t, ~stmt: string, ~parameters: option(Js.Json.t)) =>
Async.t(Belt.Result.t(Js.Json.t, string));
let execute:
(t, ~stmt: string, ~parameters: option(Js.Json.t)) =>
Async.t(Belt.Result.t(Result.Execution.t, string));
let withTransaction: (t, t => Async.t('a)) => Async.t('a);
};
module type DATABASE = {
type t;
let setup: Common_Config.Db.Url.t => t;
let end_: t => unit;
let withConnection: (t, Connection.t => Async.t('a)) => Async.t('a);
let clean: t => Async.t(unit);
};
module type MIGRATIONSTATUS = {
type t;
let version: t => int;
let namespace: t => string;
let dirty: t => bool;
let setVersion: (t, ~newVersion: int) => t;
let make: (~namespace: string) => t;
let t_decode: Js.Json.t => Belt.Result.t(t, string);
};
module type Migration = {
module Status: MIGRATIONSTATUS;
let setup:
Connection.t => Async.t(Belt.Result.t(Result.Execution.t, string));
let has: (Connection.t, ~namespace: string) => Async.t(bool);
let get:
(Connection.t, ~namespace: string) =>
Async.t(Belt.Result.t(Status.t, string));
let upsert:
(Connection.t, ~status: Status.t) =>
Async.t(Belt.Result.t(Result.Execution.t, string));
}
module type PERSISTENCE = {
module Connection: CONNECTION;
module Database: DATABASE;
module Migration: MIGRATION;
};
There are two things to note here.
Sihl takes care of migrations and it doesn't use an ORM. Therefore, it has no way of doing migrations on its own. This is why the persistence module has to implement the MIGRATION interface.
The API was designed to work with SQL and document databases. If other databases are requested for persistence storage that can't implement this API, we will happily extend it.
Configuration
This feature is about read-only service configuration as defined by The Twelve-Factor App.
Similarly to Rails, Sihl projects can be configured for the environments development, test and production.
A Sihl project is nothing more than a list of apps and an environment configuration.
Providing configurations
let environment =
Sihl.Common.Config.Environment.make(
~development=[
("BASE_URL", "http://localhost:3000"),
("EMAIL_SENDER", "[email protected]"),
("DATABASE_URL", "mysql://root:[email protected]:3306/dev"),
("EMAIL_BACKEND", "console"),
],
~test=[
("BASE_URL", "http://localhost:3000"),
("EMAIL_SENDER", "[email protected]"),
("DATABASE_URL", "mysql://root:[email protected]:3306/dev"),
("EMAIL_BACKEND", "memory"),
],
~production=[
("EMAIL_BACKEND", "smtp"),
("BASE_URL", "https://sihl-example-issues.oxidizing.io"),
("SMTP_SECURE", "false"),
("SMTP_HOST", "smtp.sendgrid.net"),
("SMTP_PORT", "587"),
("SMTP_AUTH_USERNAME", "apikey"),
],
);
let project = Sihl.App.Main.Project.make(~environment, [App.app([])]);
The secret configuration SMTP_AUTH_PASSWORD is provided as an environment variable. Sihl merges them with the environment configuration in Project.re. Sihl knows which environment to pick based on the SIHL_ENV value.
Configuration validation
What if web apps refused to start without proper configuration? This is exactly what Sihl apps do. Each app provides a configuration schema which defines the configuration that it needs in order to run.
let configurationSchema =
Sihl.Common.Config.Schema.[
string_(
~default="console",
~choices=["smtp", "console", "memory"],
"EMAIL_BACKEND",
),
string_("SMTP_HOST"),
int_("SMTP_PORT"),
string_("SMTP_AUTH_USERNAME"),
string_("SMTP_AUTH_PASSWORD"),
bool_("SMTP_SECURE", ~default=false),
bool_("SMTP_POOL", ~default=false),
];
Using a little configuration language we can describe strings, ints, bools, whether anything is optional by providing a default and a list of valid choices.
When starting a project, Sihl merges all configuration schemas together, and checks whether the loaded configuration is valid. In case it's not, it won't start the project and it will log a comprehensive error message.
Reading configurations
Let's have a look at some code which reads configuration:
let transport =
Nodemailer.Transport.make(
~host=Common_Config.get("SMTP_HOST"),
~port=Common_Config.getInt("SMTP_PORT"),
~auth={
"user": Common_Config.get("SMTP_AUTH_USERNAME"),
"pass": Common_Config.get("SMTP_AUTH_PASSWORD"),
},
~secure=Common_Config.getBool("SMTP_SECURE"),
~pool=Common_Config.getBool(~default=false, "SMTP_POOL"),
(),
);
"Where on earth is the type-safety?!", I hear you yell. There is none. You could have a typo in your configuration key that makes your app crash.
Simple typos could be detected by comparing the provided string key with all the expected keys.
We give up some type-safety to express more complex configuration schemas.
Dependencies in configurations
Let's look at the email backend configuration EMAIL_BACKEND. Valid choices are smtp, console and memory, all used in different environments. Say we set EMAIL_BACKEND to console and we don't provide any SMTP configuration.
According to the schema, the SMTP configurations is not optional - they don't have default values. Sihl wouldn't start the project and would ask for some SMTP configuration.
We extend the schema language to express simple dependencies:
let configurationSchema =
Sihl.Common.Config.Schema.[
string_(
~default="console",
~choices=["smtp", "console", "memory"],
"EMAIL_BACKEND",
),
string_(~requiredIf=("EMAIL_BACKEND", "smtp"), "SMTP_HOST"),
int_(~requiredIf=("EMAIL_BACKEND", "smtp"), "SMTP_PORT"),
string_(~requiredIf=("EMAIL_BACKEND", "smtp"), "SMTP_AUTH_USERNAME"),
string_(~requiredIf=("EMAIL_BACKEND", "smtp"), "SMTP_AUTH_PASSWORD"),
bool_("SMTP_SECURE", ~default=false),
bool_("SMTP_POOL", ~default=false),
];
We require SMTP configurations only if EMAIL_BACKEND equals smtp. This allows us to express various configuration "configurations".
It's just data!
Since the configuration schema is just data, we can ask an app "How can I configure you properly?". At some point, it will be possible to do that with yarn sihl apps:config <appname> and have Sihl apps document themselves.
User Management
The user management app @sihl/user got fully working password reset and email confirmation workflows. This can be tested in our deployed example app.
Road to 1.0.0
There is one major productivity-enhancing feature missing.
CLI Scaffolding
Sihl doesn't generate code for you at the moment, except for decoders and encoders. One of the main design goals is explicitness. We try to achieve that by minimizing the amount of compile-time and run-time "magic". This leaves us with more code to write, read and understand.
Nevertheless, we accept this trade-off in hopes for simplicity. We believe Reason's type system will help us to juggle that increased amount of code.
The scope of the scaffolding feature is not entirely clear yet. At minimum, it should allow developers to create an empty Sihl project and to add Sihl apps. Generating routes, models and repositories might follow in the future.
Scaffolding in Version 1.0.0
In order to create a Sihl project, we are looking into Spin, a project scaffolding tool for Reason and OCaml. It makes sense to integrate it, because the initial project setup is mostly just pulling a template. Once the Sihl project is set up however, further scaffolding will most likely be done using the Sihl CLI.
Initially, this could look like yarn sihl app:create <appname>. The created app is correctly namespaced, and contains a "Hello World" example which can be adjusted immediately.
Conclusion
As the API stabilizes, we feel more confident to start documenting Sihl. We plan to release 1.0.0 with proper code-level documentation of the module signatures that can be use to generate docs. After the release, we will write a proper documentation with examples and recipes.
If this project sounds interesting to you, please star the repo.
(Psst, this blog also maintains an RSS feed.)
If you would like to hire us, don’t hesitate to give us a shout!