TL;DR

Always explicitly set encoding for Base64 decoded input as it defaults to ASCII.

Base64.decode64(input).force_encoding('utf-8')

Base64 encoding problem

Base64 comes handy when we need to represent binary data in ASCII. For example when transferring a file or storing encryption key.

The other use case when Base64 is useful is when we can handle only ASCII characters but user may provide non-ascii strings. In such case we encode string on one end transfer encoded input and decode on the other end. You may however be surprised that this may not work straight in Ruby.

encoded = Base64.encode64('Zażółć gęślą jaźń.')
=> "WmHFvMOzxYLEhyBnxJnFm2zEhSBqYcW6xYQu\n"
Base64.decode64(encoded)
=> "Za\xC5\xBC\xC3\xB3\xC5\x82\xC4\x87 g\xC4\x99\xC5\x9Bl\xC4\x85 ja\xC5\xBA\xC5\x84."

Decoding what was previously encoded with standard Base64 class doesn’t return the same value which was encoded. It is because Base64 can’t determine input’s encoding and decodes in 8-bit ASCII:

Base64.decode64(encoded).encoding
=> #<Encoding:ASCII-8BIT>

It is very difficult task to guess input’s encoding but surprisingly decode64 doesn’t accept any parameter allowing to set the desired encoding and just defaults to ASCII.

To make above example work string encoding needs to be forced manually:

Base64.decode64(encoded).force_encoding('utf-8')
=> "Zażółć gęślą jaźń."

With the pdfinfo tool you can easily check the size of every single PDF document page by providing the -l switch:

$ pdfinfo -l 5 demo.pdf
...
Page    1 size: 592.56 x 792 pts
Page    1 rot:  0
Page    2 size: 592.56 x 792 pts
Page    2 rot:  270
Page    3 size: 592.56 x 792 pts
Page    3 rot:  0
Page    4 size: 592.56 x 792 pts
Page    4 rot:  270
Page    5 size: 592.56 x 792 pts
Page    5 rot:  0
...

The problem is that you need to know the document pages count as a -l switch accepts the number of last page to examine. That would require to execute pdfinfo twice: first to get the pages count, second to get page sizes. However there’s a trick to do everything in single call providing -1 as the last page number:

$ pdfinfo -l -1 demo.pdf
...
Page    1 size: 592.56 x 792 pts
Page    1 rot:  0
Page    2 size: 592.56 x 792 pts
Page    2 rot:  270
Page    3 size: 592.56 x 792 pts
Page    3 rot:  0
Page    4 size: 592.56 x 792 pts
Page    4 rot:  270
Page    5 size: 592.56 x 792 pts
Page    5 rot:  0
...

I didn’t found any official documentation specifying that just discovered it by trial and error.

Rails is great because it’s so easy. Rails is awful because it’s so easy. The same thing which makes that framework so good for developers, especially for beginners, makes it so bad for real life programming. The point is that a framework is not all. Tens of years of software development didn’t came up with just learning a framework to develop great products. Lot of decisions which Rails made for us is fine in most cases, but makes lot of developers unaware that there is more than just a framework their are using now and eventually they will work with another stack.

Below I’m listing skills that got lost or developers aren’t paying much attention mastering those. To be honest not all were lost due to the Rails itself, some are related to Ruby language and ecosystem.

Logging

Watching Rails logs makes me cry blood. There is a reason why events should be logged in just one line. It is because of the parsability. Repeat it: logs should be optimized for parsing not reading. Well to be exact, readability is important in development, but production logs should always be easily parsable.

I saw developers that care about indentation and line wrapping when logging some JSON/XML responses. I was wondering why they didn’t add syntax highlighting.

Logs format isn’t even the biggest problem, as changing it is quite easy (eg. with this gem). The biggest problem is that developers don’t know when and what to log. Many times I saw logging of completely meaningless information whereas important things like 3rd party requests, parameters, response (codes / message) where completely ignored. Logging all those events is invaluable when debugging production running applicationa and in any post-mortem crash analysis. In most cases however, I’m missing those information in app logs and need to rely on my intuition.

Obiect Oriented Programming

Ruby is object oriented language. It’s a pitty that so few developers really spent time learning principles of object oriented programming. Developers don’t care about proper abstractions, encapsulation, coupling. They just hack out the solution. In most cases the excuse is that the code is doing what it should.

Good code not only does what it should. Good code is readable, testable, maintainable and extendable. Note that I’ve put readability first. Code is read more often that it is written. Source code serves as one and only documentation (comments or word documents lie!).

There is a trend in Rails and Ruby to develop with the most sophisticated techniques developer can use. Convenience is used before the readability. I believe it should be the opposite. Convenience is good whey you’re writing a framework or a library. In real life coding readability is more important as at some point you or someone else will come back to things that have been implemented and would need to refactor it.

Lately we see huge conversion towards good object oriented programming. We see programmers implementing lightweight, single-purpose, stateless services. That’s good, but single responsibility is not the only SOLID principle. Now it’s time to learn Rails developers to write code that is open for extension, but closed for modification.

SQL and DB modeling

ORM did lot of good for software development, and about the same amount of damage. The damage is increased by the solutions like Rails migrations where database schema is build automagically with bunch of some method calls and rake tasks.

So whats the problem if ORM and migrations are so convenient? The problem is the convenience and laziness of software developers which do not verify that generated schema is actually correct (or efficient). In most cases, especially in Rails schema is missing indices. Indices are missing for obvious places like foreign keys, but also for columns which are searched or ordered by, though those are less obvious places.

Similar thing with querying the database. Developers do not verify the SQL queries leaving the ones generated by the framework. In most cases it is fine, but there are so typical cases where developer should verify the query for things like n+1 problem. So much time I see typical places which screams for includes statement.

Problem with database modeling is especially visible when using NoSQL databases, where developers tend to put lot of garbage and just do not care. At some point however this mess hits back and hits very hard.

Design and Architecture

Every Rails developer knows what MVC acronym means. Less will know the origins and what was the original purpose of this pattern. Even less will know anything other than MVC. With DHH advocating for removing layers and indirections the Rails community came with flat applications, which are fine for small to mid size. For anything more complex it isn’t enough though. Flat architecture Rails is promoting just doesn’t work well for complex apps. We end with god classes, callback hell, duplicated logic and complicated tests (if any as such cluttered code is untestable in most cases).

The point is, that framework should be an implementation detail, not the application itself. In Rails it is the opposite. It works well at the beginning, but in complex scenarios you need to fight against the decisions that framework developers made for you. Despite what DHH is saying adding layers and indirections is not bad idea. It’s neither bad nor good. It’s contextual and developers should decide given the context of their application.

Rails developers tend to not see any other way of implementing app that they were teached while learning the framework. When you talk about design patterns or DDD they will call you some Java zealot. However the Rails Way is only one of many approaches and the problem should determine the solution not the opposite.

Testing

That point may be controversial, as everybody nowadays is TDD-ing, BDD-ing, CI-ing. The problem is that despite code has tests it is poorly tested. Tests, if testing anything at all, completely skip testing edge/corner cases. Those are the real evaluation whether the piece of code is working correctly or not. In most cases tests are focused to sanity check some success paths and nothing else.

This problem especially escalated after DHH was advocating against the unit testing in respect of integration tests with full stack involved. Integraion tests are time consuming and requires lot of code to be written in order to be able to execute, therefore developers start testing in more general levels and in least scenarios. Also when you’re doing integration test it is harder to simulate edge case than it is in unit test.

Another problem with tests is that developers do not listen to them (even when TDD-ing). The setup pain screams for refactor, but developers tends to leave the code as is because it is doing what it should. Lot of time I see developers do not even start with red test not understanding why it is important to ensure that test is failing for a right reason.

I/O

The most typical issue with I/O I’m encountering is HTTP file downloading. For some reason developers tend to download the whole file, read its whole body into memory and than save it as an output files. All of that without any reflection. Developers tend to not care about the memory consumption. Streams and pipes are complete abstraction and so low level that not even worth thinking about.

There are even libraries to popular cloud storage services which provide no other way of downloading a file than to read its whole content into the memory, unless one implement raw client to that service.

With all the high-level abstractions over the I/O developers tend to not understand nor care what is happening at low level. The concepts of streams, pipes or non-blocking I/O seems to be complete magic, not even speaking about system processes, threading or concurrency (which probably should be a 9th lost skill in this post). The virtualization made developer careless about the resources, but not always you have the option to deploy on Heroku or Amazon or infinitely scale-out. Virtualization will help in most cases but there will be few where careless resource management will hit back.

UNIX Shell

Ruby is a scripting language that works well with the UNIX shell. Developers tend to forgot about that. Also lot of things are solved by the shell tools yet developers tend to be reinventing the wheel because their coding Ruby!

For some reason developers don’t know how to embrace the power of UNIX shell. Commands are badly executed, developers don’t care whether messages are logged to STDIN or STDERR yet lot of important debug information is just thrown away. Commands are assumed to be run successful despite of the returned status code.

UNIX shell is developed for more years then most of developers live. Lot of common tasks can be solved just by using what shell is providing. Learning the shell should be one of the most important skills to acquire, but it is not as impressive as showcasing asynchronous bidirectional JavaScript communication.

It is a common belief that shell is for devops. I believe this belief is not true.

Algorithmics

Last but not least, though maybe least important from the above set of skills. It is also least related to Rails, more to the richness of Ruby standard library and gems.

The thing is that developers given with all typical algorithmic problems already solved and provided as easy to use plugins, do not know or care about how they are solved. Also they find difficult solving certain class of problems in an efficient way (efficient from the CPU and memory usage point of view). They just hack the solution with bunch of if's and while's. I know, I know. Currently we have wells of RAM and almost unlimited CPU power, especially in web development due to the virtualization, yet there are cases (like mobile devices) where resources matter (and battery consumption).

I’m not saying that developers should implement all the lists or sorting on they own, but they should stop claiming that it is not their concern which should be solved by the platform / framework they are using. From time to time they will find a problem which they will need to solve with kind of algorithm or data structure, or maybe some implementation will misbehave in given context and they should be able to identify such case and know the alternate implementations.

Currently developers tend to don’t care at all in learning algorithms and data structures. It sounds like something for long-beard geeks.

Conclusion

I understand that nowadays the code must be developed rapidly. Thats how the state of development is in current market. But rapid development combined with frameworks embracing magic and undereducated developers is a ticking bomb. The urgent need for software developers forced us to cut corners in their education. Problem is that those education gaps aren’t filled over time. Frameworks like Rails, that abstracted away lot of concerns, are helping novice developers in being productive quickly, but for a cost of damaging their knowledge. Every Rails developer should be aware of that and gradually learn things he weren’t told in his Rails course.

Software development is not just solving all problems with the same hammer developer was given. Improvement is not only learning new frameworks. At some point every developer should learn the raw basics. It’s even more important as that basics spans across every platform or framework - just implemented using different keywords.

If you ever executed a shell command from a Ruby code like this:

result = `identify photo.jpg` # => photo.jpg JPEG 128x128 128x128+0+0 8-bit DirectClass 1.51KB 0.000u 0:00.000
raise Error unless $?.success?

Well, you’re doing it wrong! Okay, okay. It will work but you are hurting yourself (or perhaps any developer that will maintain such code later).

Most of well written shell commands provides useful information whenever something goes wrong. For example:

$ identify foo.pdf
identify.im6: unable to open image `foo.pdf': No such file or directory @ error/blob.c/OpenBlob/2638.

$ identify sample.doc
identify.im6: no decode delegate for this image format `sample.doc' @ error/constitute.c/ReadImage/544.

You can immediately tell what’s the problem when executing those commands. You may be thinking that this message will be stored in a result variable from the example above, but this is not true:

$ identify sample.doc 2>/dev/null
$

No output at all! The message instead of standard output was written to standard error. Executing shell command using backticks in Ruby returns whatever was written into a STDOUT thus all the helpful error messages are gone. That leads you to hard to find problems.

You can easily redirect standard error into a standard output in bash by appending 2>&1 to the executed command. So proper way of wrapping bash commands in Ruby should be:

result = `identify photo.jpg 2>&1`
raise Error, result unless $?.success?

That way the exception raised will contain the error message which command put into to the STDERR. So make yourself (and other developers) a favor and as a rule of thumb redirect STDERR to STDOUT in wrapped shell commands.

TL;DR

It is even more important when the command you’ve executing seems to be ignoring UNIX practices and returns always 0 as a exist status code. That happened to me lately and because of lack of redirection the command was always assumed to be executing successfully (despite of the fact that device was unplugged and tool was logging errors to STDERR).

Suppose you need to write a system daemon running in the context of Rails application, so that you have access to your models or services. You might be tempted to use gems like daemons but using such gems you’d need to load Rails environment manually as those are not aware of Rails application context. While such gems provides advanced functionality - like monitoring - they are additional dependency that must be maintained. There is a simpler approach though that will work fine in most cases.

The easiest way to load Rails environment is to execute rake environment task. Combining it with any other rake task we can easily implement a daemon running in the context of Rails application. In fact Resque (in version 1.x) embraces this approach. By reading its code you can came up with the following template for simple daemons running in the context of Rails:

task start: :environment do
  Rails.logger       = Logger.new(Rails.root.join('log', 'daemon.log'))
  Rails.logger.level = Logger.const_get((ENV['LOG_LEVEL'] || 'info').upcase)

  if ENV['BACKGROUND']
    Process.daemon(true, true)
  end

  if ENV['PIDFILE']
    File.open(ENV['PIDFILE'], 'w') { |f| f << Process.pid }
  end

  Signal.trap('TERM') { abort }

  Rails.logger.info "Start daemon..."

  loop do
    # Daemon code goes here...

    sleep ENV['INTERVAL'] || 1
  end
end

Above daemon may work in background as well as in foreground. We can specify the pid file, interval and log level. It can be started with following command:

BACKGROUND=y PIDFILE=daemon.pid LOG_LEVEL=info bundle exec rake start

To kill daemon:

kill `cat deamon.pid`

Monitoring should be delegated to tools like Monit which are far more powerful than any gem providing daemon functionality. Finally we are’t introducing any additional dependencies which is always a good thing.

After a few lost years in software architecture, during which developers were rushing towards so called “productivity” using word “pragmatic” as a justification for their poor design, finally we are experiencing conversion towards the good old OOP practices (or abandoning OO programming at all). As hacked out codebases growth developers realized that maintainability, testability and extendability is an important factor. It’s a perfect opportunity to remind some basics.

One of less known principles is command-query separation (CQS). As introduced CQS applied to methods, but over time it was also applied for application architecture where query and command interfaces were clearly separated. In this post I’d cover the former case.

The idea behind the CQS is to divide methods into following categories:

• Queries, which examine system’s state, returns value and do not cause side effects.
• Commands, which change the state of a system (world) but do not return a value.

We use queries to test object’s or system’s state while commands are used to mutate that state. Personally I like to extend this list with Factory methods, which are somewhat a combination of command and query. The purpose of a factory method is to create and return an object. It is an abstraction over object creation process.

Queries

Query method examines the state of a system. I’m using a word system instead of object by purpose, as tests are not limited to a single object. For example testing file existence is communicating via file system not just checking whether object’s file attribute is set.

Query methods are those which look like this:

process.pending?
File.exists?(path)
transaction.state

Executing above methods should never change the state of any part of a system. However it is allowed that the result of a query method is evaluated lazily, eg.:

s3object.exists?('/some/s3/key')

The S3 connection can be established whenever it is needed (exists? method called).

This type of methods should rather return primitive values which are used for control execution flow (it means that values returned by query methods are used in condition statements like if). Why you should prefer returning primitive values instead of regular object? To avoid train wrecks (and do not violate the Law of Demeter).

Train wreck is something like this:

user.profile.address.street
entry.where(type: :draft).joins(:attachmenets).all

Train wrecks are considered as a code smell. You might be asking why you should avoid train wrecks? They’re so convenient! Well if convenience is everything you need from the code then go ahead and write them, but if you are serious about testability and maintainability then embrace delegation! Delegation encapsulates implementation details, as the caller shouldn’t care how to get a street out of a user object. Train wrecks are fine for simple and small apps. But if you expect that application will grow (and which nowadays isn’t?) it’s better to avoid them earlier than later.

Train wrecks are why many developers find so difficult writing tests. The amount of stubbing they need to do in order to test simple things is overwhelming. We’re calling that setup hell or pain. Delegation basically boils down mocking to just one method call drastically reducing the code required to setup a test.

We could argue the ActiveRecord example as by some those kind of implementation is called Fluent Interface. However it is very hard to unit test it as tests requires bazillion of stubs and mocks of objects returned along to the all method invocation. That chain exposes lot of implementation details like used interface - ActiveRecord, database type - Relational, supporting join statements, column name - type together with its allowed value - draft relation - attachments. Lot of reasons to change.

Commands

As said, commands are used to mutate the state of a system. They instruct object to perform some action. Typically they shouldn’t return value as commands are not used to flow control, however often we see violation of that rule, one of which is in my opinion acceptable and is called a factory method.

Commands are methods which looks like this:

Resque.enqueue(SendNotificationJob, user_id)
user.save
document.ready!
service.call(transaction_id)

For some reason developers often violate rule of not returning a value by commands. Most often they return true or false to indicate whether command performed successfully or not (or to be more precise they are returning values which evaluates to true or false). Well this is the fastest way to get spaghetti code with a lot of if-else statements. Commands and their return values shouldn’t be used for flow control. Commands should follow invoke and forgot approach. If anything goes wrong exception should be raised.

Factories

Factories weren’t originally specified when CQS term was crafted. However I’m defining them as a special type of commands. The purpose of a factory method is to encapsulate object creation. It creates and returns new object.

Example of factories are:

@post = Post.create(post_params)
@user = CreateUser.call(user_params)

Except the returned value factories there are no other differences between factories and commands.

Why should I care?

Knowing the method type is very useful as it indicates how to write tests. Queries are mostly stubbed whereas for commands we need to setup a mock expectation ensuring command execution with correct parameters. To learn more about stubs and mocks I suggest reading Uncle Bob’s The Little Mocker post.

Knowledge about method type makes much easier to avoid typical code smells and allows making code more maintainable and easier to test. Personally I like to identify the piece of code which can be wrapped into a command method with single responsibility. That allows me making methods which are much shorter and easier to read as well designed code passes the tests and reveals intention.

So you’ve just read The Twelve-Factor App and decided that you’re going to store your configuration in the environment. Armed with dotenv you’ve changed configuration of AWS SDK to this:

AWS.config(
  access_key_id:      ENV['S3_ACCESS_KEY'],
  secret_access_key:  ENV['S3_SECRET_KEY'],
  region:             ENV['S3_REGION']
)

Everything is working fine on your local machine, so you are making commit and pushing changes to the origin. Now you need to update server configuration so you’re adding following to .bashrc:

export S3_REGION=us-west-2
export S3_ACCESS_KEY=CE358D4DB14B5BDAF5DCDD30E2C8BD7E
export S3_SECRET_KEY=55837e48dbcdf98d8277033086d5502b

Deploy, and… not working. Environment variables are not available. If you, like me, always have problems figuring out whether you are in login/interactive shell or not, and juggling configuration between .bashrc and .bash_profile perhaps this figure will help you to understand where settings should go. But you must be aware, that if you are using solutions like Upstart/Monit to start/stop your application, you can be confused about which variables and what PATH is available for the application. From the other hand you still want same variables to be available when you log in via SSH.

But this post is not about how to properly setup environment. It is about fetching environment variables.

The problem I’d like to point out is related to the way we are reading environment variables from ENV. In fact you can find that way of obtaining env variables in Rails 4:

# test/test_helper.rb
ENV['RAILS_ENV'] ||= 'test'

# config/secrets.yml
secret_key_base: <%= ENV['SECRET_KEY_BASE'] %>

# config/environments/production.rb
config.serve_static_files = ENV['RAILS_SERVE_STATIC_FILES'].present?

The problem is that if we call [] on ENV with key that is missing nil will be returned. That will probably cause weird errors which are far away from the real reason. It might take some time figuring out that error is caused by misspelled or not configured environment variable. Reading environment variables via [] does not allow gracefully handle case when key is missing or set defaults (most probably we’d use ||= to solve that). There is a better way though.

Use the fetch Luke

ENV is respoding to fetch similiary as Hash and Array are doing. That way we can set default values or handle - providing a block - gracefully missing keys. Also using fetch with unknown key will raise KeyError that will tell us which exactly key is missing. That is in fact the behavior we are expecting from the app. Without required settings is just not working and complaining about missing setting and not about some random nil references.

So refining our previous example, the AWS configuration should look as follows:

AWS.config(
  access_key_id:      ENV.fetch('S3_ACCESS_KEY'),
  secret_access_key:  ENV.fetch('S3_SECRET_KEY'),
  region:             ENV.fetch('S3_REGION')
)

The application will tell us which exactly key is missing in case we forgot to set it. Now we can go back figuring out to which file environment variables should go to be available for our application.