Comparison of automated code review tools: Codebeat, Codacy, Code Climate and Scrutinizer

Code review makes your code stronger, reduces the risk of overlooking major bugs and supports the culture of feedback. The process constitutes an inherent part of best practices in web development, so skipping it in your projects might be a big mistake. Luckily, it can be easily and effectively conducted thanks to numerous automated code review tools that are available on the market. Don’t get overwhelmed by the number of providers! With this in-depth analysis you’ll find a solution to fit your needs. 

Code review is vital 

Code review is a crucial element in product development and it strongly supports the Test-Driven Development (TDD) process to offer the best practices available in developing Rails, CSS and JavaScript code. It helps developers make the project better, faster, and more secure through making the code bulletproof. It is also an opportunity for people, especially newbies, to learn good code quickly. 

There are many code review tool available on the market. I’ve conducted an in-depth analysis to discover all the pros and cons of four of most popular and provide you with explanation of key features of each. Hopefully, this will help you pick the solution which is right for you and save time on doing your own research. 

Codebeat 

Codebeat is a dynamically growing tool that covers major technologies and programming languages. It has evolved substantially within the last few months and it’s worth noting that the team is open for feedback and implementation of new features suggested by their users. However, there are still some things missing. The tool does not allow performing any security checks (perhaps a “simple” integration with Brakeman would do the job). It also does not support any open-source tools or linters (in fact, we are still using Hound) and does not support analysing CSS/SCSS. 

Pros:

  • Support for most of the languages we use (Ruby, JS, Swift, Objective-C),
  • Metrics customisation
  • Measuring tools with own algorithms nicely described in the docs (not just a bunch of open-source projects combined together),
  • Very good support from their team
  • Small but well-documented API, which facilitates management e.g. it provides accesses to users working on selected projects (via teams),
  • Unique quick wins tab,
  • 'Ccodeclimate-test-reporter' gem that integrates Codebeat with Simplecov coverage reports,
  • Customer suggestions considered and implemented in the product,
  • It is a dynamically growing tool,
  • Support for Kotlin and Elixir (beta).

Cons:

  • Still missing some things in the documentation (e.g. explanation of code duplication detection ​here​),
  • No possible security issues check,
  • No CSS/SCSS analysis.

Possible issues:

  • I don’t see an option to list all issues found (with the option to search by category) - there’s only the option to check every single file or a “Quick Wins” tab
  • Performance is sometimes surprisingly poor.

Codacy

Codacy has an awesome UI, lots of features and it’s very flexible thanks to dozens of options. There is also a tool (in beta) which allows you to define your own patterns and implement it to be checked automatically. Frankly, it wasn’t easy to find disadvantages of this tool or any clients’ complaints. However, there is still some area for improvement and potential to growth for Codacy. 

Pros: 

  • Used by big players like Paypal or Adobe,
  • Great and intuitive UI
  • The possibility to define issue-based goals to improve the codebase
  • Checking lots of security issues (like assigning strange values to private APIs which may lead to unexpected app behaviour),
  • A nice feature of browsing commits and monitoring related issues
  • Docker analysis
  • Huge flexibility thanks to disabling/enabling patterns or whole packages and even ignoring certain patterns in selected files
  • Time to fix estimation for each issue
  • Small company but growing fast, delivering fresh features frequently
  • Well-described issues with examples right below each case (no need to browse the documentation to find out why the issue actually occurred).

Cons:

  • Incomplete documentation in some parts (some images are hard to read and the amount of information is insufficient sometimes),
  • Unintuitive one-page charts to track code quality changes over time (it’s a new feature, maybe not fully implemented yet),
  • No `hotspots` or `quick wins`,
  • No issues searching, only a few dropdown filters.

Possible issues:

  • We don’t know how the support works there
  • We also don’t know how flexible are they in terms of implementing clients’ suggestions into their product

(Image: © Image Credit: Netguru)

Code Climate

Code Climate is a well-developed and very stable solution with a great number of features. It has many advantages over its competitors and many big players recommend it as the best option.  

Pros:

  • A great number of supported languages, technologies and frameworks
  • Used by the biggest players, including Pivotal, New Relic for enterprise and Rails, jQuery for open-source
  • Very stable
  • Nice new UI
  • Well-maintained test coverage feature gem
  • Browser extensions
  • Trend charts
  • Test coverage out of the box
  • Hotspots - a 'quick wins' list

Cons:

  • Seemingly an integrated bunch of open-source project
  • Pricing - it seems to be the most expensive tool in this comparison
  • Still unpredictable API (in beta)
  • No support for Objective-C
  • No distinct types for total issues number
  • No detailed description of the issue, only a header with source code
  • No issue searching/filtering, just a paginated list with all of them

Possible issues:

  • No interest from the Code Climate team to extend the tool the way customer may suggest

(Image: © Image Credit: Netguru)

Scrutinizer

Scrutinizer doesn’t stack up well against other solutions. Actually, it seems to lack 

basic attributes that are necessary in code review. Analysing the code using different a Ruby version than defined in the repo is actually what disqualifies it. It shows `tests failed` even when the tests passed on CircleCI and locally. You can not reliably test the repository if you’re using a Ruby version other than the ones available through the dashboard. It shows lots of issues and errors with “parser could not interpret the code” message. Really strange for a tool that you’re supposed to pay for. 

Pros:

  • Seems to have very good API
  • It’s actually one of the cheapest solutions if we ignore the performance
  • Automatically detects code changes in the dashboard
  • Well written documentation
  • A feature of filtering issues by users
  • Dedicated site with current status of the services
  • Extended team management, possibility to create organisations.

Cons:

  • Available only as predefined version (e.g. no option of running tests in 2.3.0 version)
  • Dead ​test coverage gem​ (only 2 commits 3 years ago),
  • API available only when subscribed to the most expensive package,
  • Unknown warnings on Regexps (e.g. treating `/` like a division instead of the beginning of the regex when no space after it)
  • Not a predictable tool

Possible issues:

  • Performance: you pay per container and a container is simply ​one task that can be run at once, fewer containers - less performance and longer waiting time for your code to be analysed
  • It’s possible to configure checker (Settings -> Configurations -> Checks), but there’s no way to restore default configuration or even set a default for an entire organisation
  • It doesn’t read class names properly (it should include namespace - module), it’s harder to find it this way,
  • Files/issues are paginated in the way that enforces you to click `next` every time you want to move to the next e.g. 10 classes
  • There’s no option to see all the rules (checks), there are two groups for enabled or disabled rules, but there’s no “all” option.

Code review tools comparison summary 

Supported languages and technologies: 

Code Climate​: Ruby/Rails, JavaScript, Python, PHP, Swift, SCSS/CSS, Go, CoffeeScript, Apex, Ember, ESLint, Haskell, Haxe, RubyMotion, Vim Script 

Codebeat​: Ruby, Javascript, Python, Java, Swift, Go, Typescript, Objective-C, Kotlin, Elixir 

Codacy​: Ruby, JavaScript, Python, PHP, Java, Swift, CSS, TypeScript, CoffeeScript, Scala, C/C++, Dockerfile, SASS, Shell Script 

Scrutinizer​: Ruby/Rails, Javascript, Python, PHP 

Measuring tools

Code Climate: many existing, open-source tools like Rubocop, Brakeman, CSS/SCSS Lint, ESLint, Flog etc. full list here 

Codebeat: their own algorithms and implementation written from scratch see how it works 

Codacy: many existing, open-source tools like Rubocop, Brakeman, CSS/SCSS Lint, ESLint, Flog etc. full list here 

Scrutinizer: closed-source codebase with a possibility to use open source tools. OS tools list here 

Pricing

Code Climate: $16,67/user/month (when billed yearly - otherwise 20$) 

Codebeat: $20/user/month, but you can probably negotiate 

Codacy: $15/user/month 

Scrutinizer: 200EUR/month for unlimited number of users. 2 containers in the plan, where 1 is just 1 task at a time, extra container costs 50 EUR.

Documentation

Code Climate: very good and comprehensive 

Codebeat: still some things missing 

Codacy: not bad, not too much text and some images are not clickable so the readability is limited 

Scrutinizer: good

API

Code Climate: yes, still in beta version 

Codebeat: yes, simple but usable 

Codacy: yes, not described perfectly in their docs 

Scrutinizer: seems to be really good. ​read more 

Wrap up 

After a deeper look at all the tools described above, I can disqualify Scrutinizer as it doesn’t support all Ruby versions, has many issues which don’t allow to do the metrics reliably and sometimes makes it actually impossible. On the other hand, Codebeat seems to be a comparable tool to Code Climate or Codacy. In case of these three solutions, pros seem to overweigh the cons and a particular feature might clinch the final choice, which would stem from individual needs.  

Łukasz Ozimek, Web developer, Netguru
Image source: Shutterstock/McIek