Rails, PostgreSQL Performance Audit and Tuning Consultant for Hire

How to Improve Rails Caching with Brotli Compression

Tue, 12 Sep 2023 08:36:01 +0200

Caching is an effective way to speed up the performance of Rails applications. However, the costs of an in-memory cache database could become significant for larger-scale projects. In this blog post, I’ll describe optimizing the Rails caching mechanism using the Brotli compression algorithm instead of the default Gzip. I’ll also discuss a more advanced technique of using in-memory cache for extreme performance bottlenecks.

Brotli vs Gzip 101

In one of my previous posts, I’ve covered how Rails leverages compression algorithms for the HTTP transport layer. Please check it out for more details on how Gzip and Brotli differ in compression ratios and performance. Here, let’s quickly recap the basics.

You can use Gzip and Brotli using Ruby API wrapping underlying C libraries. To Gzip a sample JSON file and measure the compression rate, run the following code:

require 'json'
require 'zlib'

json = File.read("sample.json")
puts json.size # => 1127380 bytes ~ 1127kb
gzipped = Zlib::Deflate.deflate(json)
puts gzipped.size # => 167155 bytes ~ 167kb

As you can see, a standard Gzip compression on a JSON reduced the size by ~85%.

Let’s now see how to use Brotli. You have to start by installing a Brotli gem:

gem install brotli

Now you can run:

require 'json'
require 'brotli'
json = File.read("sample.json")
puts json.size # => 1127380 bytes ~ 1127kb
brotlied = ::Brotli.deflate(json, quality: 6)
puts brotlied.size # => 145056 bytes ~ 145kb

Brotli is slow with the default quality settings, so 6 is recommended for on-the-fly compression.

Brotli compression is ~13% better than Gzip for this sample JSON. But, according to the sources, improvement can be as high as 25%.

Another advantage of Brotli is the speed of the compression and decompression process. Based on my benchmarks, Brotli can be ~20% faster than Gzip when used for Rails cache. The overhead of the caching layer will probably not be the bottleneck of your Rails app. But, if you’re using caching extensively, then a 20% improvement might translate to a measurable global speed-up.

But better compression could be even more impactful than read/write performance. I’ve discussed how to reduce the costs of in-memory cache databases when caching ActiveRecord queries in my other blog post. Combined with the tips on reducing the size of cache payloads, Brotli compression could lower caching infrastructure costs. Also, more space for cache means less frequent rotation of older entries, translating to better performance.

How to use Brotli for Rails cache?

By default, Rails cache uses the Gzip algorithm for payloads larger than 1kb. But, as we’ve discussed, Brotli offers better compression ratios and speed. You can start using Brotli in your Rails app with the help of the rails-brotli-cache gem that I’ve recently released.

It works as a proxy wrapper for standard cache stores, precompressing the payloads with Brotli instead of Gzip. I’ve measured a 20%-40% performance improvement in Rails.cache depending on the underlying data store. You can check out the benchmarks.

After adding it to your Gemfile, you have to change a single line of config to hook it up:

config/environments/production.rb

config.cache_store = RailsBrotliCache::Store.new(
  ActiveSupport::Cache::RedisCacheStore.new(url: ENV['REDIS_URL'])
)

Check out the gem readme for information on using it with different cache store types.

Also, remember to run Rails.cache.clear after after first enabling it to remove previously gzipped cache entries. But watch out not to wipe your Sidekiq in the process!

Optionally, you can hook up a custom compression algorithm. For example, Google Snappy offers even better performance than Brotli for the cost of worse compression ratios. ZSTD by Facebook is also worth checking out.

Gem uses an API compatibility spec to ensure that it behaves exactly as the underlying data store. I’m currently using it in a few production projects, but please submit a GH issue if you notice any inconsistencies.

The great news is that support for custom cache compression algorithms has recently been merged into the Rails main branch. A new :compressor config option should be available in the upcoming Rails 7.1.

Using and misusing in-memory cache store

I’ve mentioned that rails-brotli-cache offers ~20% speed improvement compared to the other cache stores. But there’s a catch. Let’s consider the following benchmark:

require 'active_support'
require 'active_support/core_ext/hash'
require 'net/http'
require 'rails-brotli-cache'
require 'benchmark'

json_uri = URI("https://raw.githubusercontent.com/pawurb/rails-brotli-cache/main/spec/fixtures/sample.json")
json = Net::HTTP.get(json_uri)

redis_cache = ActiveSupport::Cache::RedisCacheStore.new
brotli_redis_cache = RailsBrotliCache::Store.new(redis_cache)
memcached_cache = ActiveSupport::Cache::MemCacheStore.new
brotli_memcached_cache = RailsBrotliCache::Store.new(memcached_cache)
file_cache = ActiveSupport::Cache::FileStore.new('/tmp')
brotli_file_cache = RailsBrotliCache::Store.new(file_cache)
memory_cache = ActiveSupport::Cache::MemoryStore.new
brotli_memory_cache = RailsBrotliCache::Store.new(memory_cache)

iterations = 100

Benchmark.bm do |x|
  x.report("redis_cache") do
    iterations.times do
      redis_cache.write("test", json)
      redis_cache.read("test")
    end
  end

  x.report("brotli_redis_cache") do
    iterations.times do
      brotli_redis_cache.write("test", json)
      brotli_redis_cache.read("test")
    end
  end

  x.report("memcached_cache") do
    iterations.times do
      memcached_cache.write("test", json)
      memcached_cache.read("test")
    end
  end

  x.report("brotli_memcached_cache") do
    iterations.times do
      brotli_memcached_cache.write("test", json)
      brotli_memcached_cache.read("test")
    end
  end

  x.report("file_cache") do
    iterations.times do
      file_cache.write("test", json)
      file_cache.read("test")
    end
  end

  x.report("brotli_file_cache") do
    iterations.times do
      brotli_file_cache.write("test", json)
      brotli_file_cache.read("test")
    end
  end

  x.report("memory_cache") do
    iterations.times do
      memory_cache.write("test", json)
      memory_cache.read("test")
    end
  end

  x.report("brotli_memory_cache") do
    iterations.times do
      brotli_memory_cache.write("test", json)
      brotli_memory_cache.read("test")
    end
  end
end

Check out the gem repo for a complete benchmark code

You should get similar results:

                       user     system   total    real
redis_cache            1.770976 0.040232 1.811208 (2.587884)
brotli_redis_cache     1.387118 0.077257 1.464375 (2.137552)
memcached_cache        1.787665 0.058871 1.846536 (2.534051)
brotli_memcached_cache 1.368171 0.088934 1.457105 (2.043203)
file_cache             1.716816 0.055140 1.771956 (1.772132)
brotli_file_cache      1.319149 0.068127 1.387276 (1.387309)
memory_cache           0.001370 0.000117 0.001487 (0.001480)
brotli_memory_cache    1.330853 0.043904 1.374757 (1.374784)

Here’s an ASCII chart for easier comprehension:

redis_cache            |████████████████████
brotli_redis_cache     |████████████████

memcached_cache        |████████████████████
brotli_memcached_cache |████████████████

file_cache             |████████████████
brotli_file_cache      |███████████████

memory_cache           |
brotli_memory_cache    |███████████

As you can see, rails-brotli-cache proxy improves the performance for all the cache storage types apart from ActiveSupport::Cache::MemoryStore. And this storage type is ~99% faster than all the other types. What’s going on?

Nested cache layers

ActiveSupport::Cache::MemoryStore is unique because compared to other cache store types, it does not compress cached entries. Also, it keeps cached entries directly in the process RAM, so there’s no serialization, networking, or filesystem IO overhead. If we run the benchmark with in-memory compression enabled, we’ll get the following results:

memory_cache = ActiveSupport::Cache::MemoryStore.new(compress: true)

memory_cache           |█████████████████
brotli_memory_cache    |███████████

As you can see, compression causes significant overhead for all the cache store types. For some extreme cases, you can use this order of magnitude better performance of uncompressed in-memory cache-store. The tradeoff will be higher RAM usage of your Ruby processes, but 99% performance improvement could be worth it.

To do it, you can wrap your standard Rails cache into a custom in-memory store like this:

config/application.rb

$memory_cache = ActiveSupport::Cache::MemoryStore.new

and now in your bottleneck endpoint:

app/controller/users_controller.rb

class ProductsController < ApplicationController

# ...

def index
  cache_key = "cached_products-#{params.hash}"
  cached_products = $memory_cache.fetch(
    cache_key,
    expires_in: 30.seconds, race_condition_ttl: 5.seconds
  ) do
    Rails.cache.fetch(cache_key, 5.minutes) do
      Product::FetchIndex.call(params)
    end
  end

  render json: cached_products
end

In the above example, we’re using two layers of caching. The outer layer using $memory_store in-memory cache is set to expire every 30 seconds. It means that our underlying standard cache store will be called significantly less often. As discussed, the performance of the caching layer for most of the requests will be ~99% better. For endpoints under heavy load, configuring race_condition_ttl will also reduce the number of redundant cache refresh calls.

This example might seem a bit convoluted. But for bottleneck endpoints that serve identical data to multiple clients, similar hacks could measurably reduce the load on your Redis/Memcache database and improve performance. But remember that it’s easy to misuse this technique and bloat the app’s RAM usage.

Summary

Replacing a standard Gzip compression with rails-brotli-cache gem is a relatively simple change that might result in a global speed-up for your Rails app. Another benefit is that it could help you reduce the costs of the in-memory cache database thanks to better compression. I invite you to give it a try. Feedback and PRs on how to improve the gem are welcome.

Five Easy to Miss Performance Fixes for Rails Apps

Tue, 09 May 2023 05:36:01 +0200

Improving the performance of a Rails application can be a challenging and time-consuming task. However, there are some config tweaks that are often overlooked but can make a significant difference in response times. In this tutorial, I will focus on a few “quick & easy” fixes that can have an immediate impact on the speed of your Rails app.

Eliminate request queue time

Request queue time means that your web servers don’t have enough capacity to process the incoming traffic. Unless you’re using a PAAS platform like Heroku, you have to add an extra NGINX header to enable request queuing data in the logs:

proxy_set_header X-Request-Start "t=${msec}";

This config will enable any popular APM monitoring tools to report the current queue time. Without it, it’s impossible to know the real backend response time as experienced by your users.

Correctly configured NewRelic APM should display Request Queuing metric

The presence of queue time is both good and bad news. Your consumers experience longer response times, but resolving it by spinning up more server instances is usually possible. Before throwing money at a dozen new Heroku dynos, double-check if you cannot extract more throughput without increasing the cost.

These Heroku docs recommend values of WEB_CONCURRENCY and RAILS_MAX_THREADS for different dyno types. It’s worth noting the Performance-M dynos offer the worst cost/metrics and should usually be replaced by a single Performance-L or a fleet of Standard-2X.

I usually try to squeeze enough Puma workers so that RAM usage never spikes above 80%. A relatively simple way to decrease memory usage and prevent leaks is to use Jemalloc instead of the default memory allocator. I’ve seen memory usage drop by 15-20% after adding Jemalloc. It usually allowed us to safely add one more Puma worker for the same server/dyno type and increase the app’s throughput. You can check out my other blog post for more info on how to use Jemalloc in Rails apps and other tips on reducing memory usage.

Enable cache and Redis connection pool

Unless you’re on the newest Rails, your app’s caching layer may be misconfigured. Production Rails apps often use Redis or Memcache as their in-memory cache backend. But, the default config of the cache client is single-threaded. A default support for connection pool was added around a year ago. So, if you’re using Rails < 6.1 with a Puma web server, multiple Puma threads compete for a single cache client connection.

You can enable a connection pool for your caching client with similar config options:

config/production.rb

config.cache_store = :mem_cache_store, "cache.example.com",
                     pool: { size: 5, timeout: 5 }

If your app uses Redis directly, it could have the same problem. It’s a popular pattern to expose Redis client via a global variable:

$redis = Redis.new(url: ENV.fetch("REDIS_URL"))

value = $redis.get("key-name")

But it means any interaction with it will be throttled in multithreaded processes like Puma or Sidekiq. Redis usually responds in ~1ms, but if you’re using it extensively, dozens of blocked calls could add up to a measurable overhead.

You can resolve it by leveraging a connection-pool gem:

$redis_pool = ConnectionPool.new(size: ENV.fetch("RAILS_MAX_THREADS", 5)) do
  Redis.new(url: ENV.fetch("REDIS_URL"))
end

value = $redis_pool.with do |conn|
  conn.get("key-name")
end

Fine-tune database connections pool

Balancing the number of web servers and background worker threads with the database pool and max connections can be challenging. pool config from config/database.yml file determines how many database connections can be opened by a single Ruby process.

If you’re using a multithreaded process like Puma or Sidekiq, you need to make sure that the value of pool is at least equal to the maximum number of threads. Otherwise, your app could start raising ActiveRecord::ConnectionTimeoutError if it cannot establish a database connection for the specified timeout. The sum of possible connections should not be larger than the max_connections PostgreSQL setting. But you cannot set max_connection to any arbitrary value. Multiple concurrent connections will increase the load on a database, so its specs must be adjusted accordingly.

AWS RDS offers powerful insights and customizable CloudWatch alerts for PostgreSQL database.

It’s possible to oversee pool misconfiguration. For example, threads could be waiting for database access for a period shorter than timeout, adding performance overhead but never raising the error.

You must be extra careful when using a load_async API introduced in Rails 7. This is because it schedules threads within threads and exhausts more database connections. You can read about it in more detail in my other blog post.

Customize PostgreSQL config

Depending on what PostgreSQL provider your app uses, you might be able to tweak the database config. More extensive control is another reason why I recommend migrating the Heroku database to RDS.

If your PostgreSQL is handling larger datasets, tweaking the default config will likely increase SQL performance. A few examples:

random_page_cost is set to 4.0 by default. It determines how likely a query planner is to use an index scan instead of a sequential scan. The default value of 4.0 originates from IO read speed of HDD disks. It is no longer relevant when databases use much faster SSDs for storage. Setting this value between 1.1-2.0 will make better use of database indexes and reduce CPU-intensive sequential scans.

work_mem determines how much RAM is available per active connection. The default value is 4096kb. Depending on your app’s usage patterns, this setting might throttle SQL queries doing more complex in-memory sorting and hash table operations. This value has to be tweaked carefully because it could cause out-of-memory issues for a database. But, ensuring that each connection has enough RAM will likely increase the performance of more complex queries.

shared_buffers and effective_cache_size settings determine how much database RAM is delegated for caching. Performant databases should read ~99% of data from the in-memory cache. You can use rails-pg-extras cache_hit to inspect your app’s database cache usage. For some cases increasing the default value is likely to result in a speedup for the database layer.

Optimize HTTP layer config

All the previous tips are related to backend layer performance. But, a typical web app issues dozens of requests on initial load. And only a few are backend-related, i.e., website HTML, API calls, etc. The majority of requests are static assets, JavaScript libraries, and images. Therefore, optimizing the frontend-related requests can have a much greater effect on the perceivable performance than fine-tuning the backend.

Check out my other blog post for more detailed tips on optimizing frontend performance. But, since I’ve promised “quick and easy” fixes, let’s cover two potentially most impactful frontend config tweaks.

Client-side caching

Correctly configuring client-side caching could be the most critical frontend optimization. There’s no better way to speed up the HTTP request than not to trigger it. But I’ve seen it misconfigured in multiple production apps. Rails have a great mechanism to leverage client-side caching, i.e., MD5 digest. In the production environment, the application.js file becomes application-523bf7...95c125.js. The random suffix is generated based on the file contents, so it is guaranteed to change if the file changes. You must add the correct cache-control header to make sure that once downloaded, the file will persist in the browser cache:

cache-control: public, max-age=31536000, immutable

The immutable parameter ensures that the cache is not cleared when the user explicitly refreshes the website on the Chrome browser.

You have to enable it in your Rails production config:

config/production.rb

config.static_cache_control = 'public, max-age=31536000, immutable'
config.public_file_server.headers = {
  'Cache-Control' => 'public, max-age=31536000, immutable',
  'Expires' => 1.year.from_now.to_formatted_s(:rfc822)
}

Or if you’re using NGINX as a reverse proxy, you can use the following directive:

location ~* \.(?:ico|css|js|gif|jpe?g|png|woff2)$ {
  add_header Cache-Control "public, max-age=31536000, immutable";
  try_files $uri =404;
}

I’ve seen apps using Etag and Last-Modified headers instead of Cache-Control. Etag is also generated based on the file contents, but the client has to talk to the server to confirm that the cached version is still correct. It means that on every page visit, the browser has to issue a request to validate its cache contents and wait for 304 Not Modified response. This completely unnecessary network roundtrip can be avoided by adding a Cache-Control header.

Use Cloudflare CDN

Using CloudFlare for your DNS together with proxying enabled could be the simplest way to get most of your frontend-related config in order. It will be especially impactful if you’re using Heroku, which still does not support HTTP2 or even native Gzip compression… (╯°□°)╯︵ ┻━┻

Client-side caching with Cloudflare proxy will result in your static assets getting cached in CDN edge locations close to your end users. Subsequent requests for the same assets will not be triggered because data is now cached in the browser.

Please remember to use Respect existing headers for Browse Cache TLL setting to regain more fine-tuned control and avoid stale cache issues.

In addition to global CDN CloudFlare will enable to following benefits for your app:

automatic gzip/brotli compression
HTTP2 support
DDOS protection
Signed Exchanges SXGs support that can benefit Largest Contentful Paint (LCP) metric and SEO (available only for paid plans)

And many features more.

You can use Page Speed insights and WebPageTest to get a quick overview of your app’s fronted performance.

Summary

Optimizing the mentioned configs is a great starting point for in-house performance tuning. They are likely to globally impact the performance metrics, so the development time spent tweaking them will probably yield measurable benefits.

How to Freeze ETH and ERC20 Tokens using Smart Contract

Wed, 19 Apr 2023 00:36:01 +0200

I’m still the kind of person that buys $500 worth of Dogecoin, and one day later, panic sells with a 30% loss. That’s why, I’ve developed Ethereum smart contracts that make speculating on cryptocurrencies less stressful. Locker smart contracts allow me to freeze crypto assets and only release them when enough time has passed or the price has reached a predefined value. This approach sacrifices liquidity for peace of mind, and for me, that’s a perfect tradeoff! In this article, I’ll describe this project and the risks associated with using it or basically any Smart Contract.

Disclaimer: The information provided in this blog post is for educational purposes only and should not be treated as investment advice.

Decentralized trust

The crypto scene is ripe with scams and exploits siphoning billions in user funds. Despite that, I kind of see it as the only tech that can be “trusted”. Ethereum Smart Contracts are literally the only existing technology guaranteed to work as programmed.

Of all the popular smart contract technologies, Ethereum is the only one that enables a standard user to run a full node. On the other hand, all the “Ethereum killers” network nodes are either centralized or require processing power and dev ops expertise accessible only to well-funded entities. As a result, the network consensus is governed by a few centralized actors rather than the decentralized community. You can check this tutorial for more info on how to run your own Ethereum full node.

Obviously, “I’m in it for the technology”. I’ve become attracted to crypto as a tech rather then casino after getting all my bank funds repeatedly locked. It is enough to send one transfer to Revolut, too many to trigger a complete lockdown of all my accounts and credit cards. “Risky transaction…“ they said. Getting my funds unfrozen usually takes ~30min on a bank support line. But the idea that all my finances can be locked at a whim of a single centralized entity is disturbing…

You’ve probably heard about Canadian anti-vax trucker supporters getting their accounts frozen without any judicial order. In Poland, we’ve also had our taste of a digital prison. People protesting against anti-abortion laws got their bank accounts blocked for “breaking the pandemic sanitary regulations”. Or the case of Ukrainian refugees seeing their EUR account balances automatically converted to UAH with a special exchange rate spread “because war”.

Diversifying a part of funds in a layer that’s unreachable by a state-level 3rd party seems like a reasonable idea. Since this revelation, I’ve started seeing the crypto price ups and downs as a “tax for safety” rather than a chance for speculation.

So much for the digression. Let’s move on to the risks of using the Locker or basically any smart contract.

Can you trust Smart Contracts?

Different layers of trust could be broken and cause the loss of tokens you deposit into smart contracts.

Bugs and deliberate scams

To start with, the contract author could be a scammer or an clumsy blockchain dev. So the Locker source code could contain a sneaky loophole or bug causing the loss of your funds. The open-source nature of Ethereum Smart Contracts is the best remedy for these risks.

Before you deploy the contract or interact with an already live instance, you can always analyze the source code. But currently, it requires a significant dose of expertise. Locker smart contracts are relatively simple, maxing out at 200 LOC with minimal external dependencies. But, most Defi projects are endless blobs of code that are impossible to analyze single-handedly. A potential revolution in the safety of Smart Contracts could be the advent of AI. ChatGPT seems to be proficient in analyzing security issues of Ethereum smart contracts. An AI-based service listing potential risks of any smart contract interaction (maybe even integrated into your wallet!) could be just around the corner.

The contract’s external dependencies and logic introduce another risk factor. In the case of Locker, it “trusts” the provided ChainLink oracles to report the correct price of the target token. The user selects an oracle address himself when configuring the deposit. The only risk is that oracles could be abandoned and not update the price correctly. In that case, your tokens would be released only after the lockup period.

In the case of more complex smart contracts, external risk factors are often significant. For example, the creator of an ERC20 token could grant himself the power to print unlimited units and tank the price at your cost. Given the complexity of Defi smart contracts spotting similar risks is non-trivial. Hopefully, AI will be able to help standard users make more sense of these issues.

Frontend interface

Another layer of trust is how one can interact with the contract instance. Locker does not offer any front-end UI. You have to interact with it directly on Etherscan or use programmatic API. Interacting with any Defi project featuring custom frontend is a risk factor. Exploits targeting frontend layer of the smart contracts stack could alter where your funds are sent. Or malicious browser extension can change the frontend code of an otherwise trustworthy web3 domain.

The best way to reduce the risk of interacting with any smart contracts is to use minimum ERC20 approval limits. Metamask has recently improved its UX by allowing users to input approval limits manually. Before that, the approval limits were implicitly unlimited, allowing attackers to withdraw all your assets.

Metamask UI for custom ERC20 approval limit

Also, using a hardware wallet like Ledger or Trezor can prevent a range of crypto exploits.

Government censorship

A government could, in theory, censor any smart contract address. So far, the most famous case of government sanctioning a piece of autonomous software was Tornado Cash. Apparently, you can also go to jail for writing open-source code.

The fight against decentralized finance may intensify. For example, the EU is currently brewing new restrictions for Smart Contracts. But, despite sanctions from the US government, the Tornado Cash smart contracts are still regularly used. It shows that cryptocurrencies excel at their core value proposition of being uncensorable even by government-level actors.

All the centralized elements of Tornado Cash like frontend interface, domain, and GitHub repos were succesfully taken down. But it’s still possible to access Tornado Cash immutable smart contracts using proprietary Ethereum full node or censorship-resistant relay providers.

User error

Any smart contract interaction is risky. There’s no edit/undo button for incorrect transactions. Ask the guy who burned 500k USD by incorrectly unwrapping WETH. If you misuse Smart Contract and lose funds, even Metamask Twitter Support will be unable to help you.

Now that we’ve briefly covered the risks, let’s move on to the Locker smart contracts.

Locker smart contracts

There are four different flavours: Locker, LockerETH, LockerPriv, and LockerETHPriv.

Locker

Locker can be used to freeze any ERC20 token and is accessible by any account. Your funds will be deposited into the same contract address as other users. But, each user can only withdraw his tokens. The benefit is that you don’t have to deploy the contract instance yourself. ETH can be held by this contract only in its ERC20 format, i.e., as WETH. To use price conditions for release, you must provide a correct ChainLink price oracle address.

[Warning] Do not transfer ERC20 tokens directly to this contract instance or they will be lost! You have to use a dedicated deposit method. Also, do not use this contract for storing rebasing tokens like stETH or rETH. Stored token balance is determined once when depositing the token. It means that rebased reward will get stuck in the contract forever.

LockerETH

LockerETH works the same as Locker but only for standard ETH instead of ERC20 tokens. It uses a predefined ChainLink ETH/USD price oracle. It’s also publicly accessible.

LockerPriv

LockerPriv works as your private vault for ERC20 tokens. Only the deployer account can interact and deposit funds into it. The downside is that you have to pay the costs of releasing it. With Mainnet gas prices at 20 gwei and the price of ETH hovering around $2000 it costs ~$100. Alternatively, you can use one of the L2 layers to reduce the deployment cost. For example, on Arbitrum L2 it costs ~$5 to deploy an instance of this Smart Contract.

LockerETHPriv

As you can probably guess LockerETHPriv, is a private vault for storing a standard ETH. Just like in the case of LockerPriv you have to deploy it yourself.

Summary

I hope the post has helped highlight the risks of using Smart Contracts. Please remember that crypto and Defi is risky, and you can easily lose all your funds.

But cryptocurrencies are here to stay. UX and general appeal is only about to improve. The worst-case scenario is that hostile governments might slow adoption for a few years until there’s a shift in power. So safely stashing your coins could be a strategy to wait out turbulent times.

How to Find, Debug and Fix N+1 Queries in Rails

Tue, 07 Mar 2023 04:09:06 +0100

Fixing N+1 issues is often the lowest-hanging fruit in optimizing a Rails app performance. However, for non-trivial cases choosing a correct fix could be challenging. Incorrectly applied eager loading does not work or even worsens response times. In this blog post, I describe tools and techniques I use to simplify resolving N+1 issues.

How to simulate N+1 issues in development

I doubt the Rails blogosphere needs yet another introduction to N+1 queries. You can check out my previous post for a quick recap.

Instead, let’s discuss how to establish a repeatable routine for spotting N+1 issues before they start manifesting in production. It’s common for developers to work with a minimal local data set. Unfortunately, working with a database resembling production is usually not possible. But seeding local data for your test user to mimic production sizes could be a perfect compromise.

Hundreds of test objects will not be enough to simulate slow SQL queries. PostgreSQL usually needs tens of thousands of table rows for any SQL-layer slowdown to manifest. But N+1 query issues are not related to absolute collection sizes but to how you use and misuse ActiveRecord ORM. A dozen objects are enough to trigger N+1 issues and make them detectable during development.

A simple way to quickly populate your local data is to use a FactoryBot gem. You may already have used it in your project but did not consider leveraging it to seed development env. To do it you have to include the gem in development group of your Gemfile. Now in the Rails console:

require 'factory_bot'

100.times do
  FactoryBot.create(:post, user: your_test_user)
end

#...

Of course, the exact implementation will be unique for each project. But leveraging existing factories should simplify the process compared to writing seed scripts from scratch.

Another benefit of working with a larger local collection is that you’ll spot places likely to benefit from pagination. Before endless lists, bring your production servers to a crawl…

How to analyze and debug local N+1 issues

Now that you have a decent local dataset, you can start digging deeper. bullet gem is a popular tools for reporting N+1 issues. However, its log output is a bit noisy or even illegible for larger projects. I’ve also seen it report many false positives so extracting actionable data it could be challenging. Instead, I usually default to the good ol’ rack-mini-profiler gem:

rack-mini-profiler reporting the number of SQL queries

The popup window shows the number of SQL queries executed per request. It even works with XHR requests. So, if you’re using a SPA framework, you can check quickly how many queries are triggered by each API endpoint.

Another great feature is that you can see the exact line of code that triggered each query:

For many more straightforward cases, this data is enough to track the missing eager loading and validate the fix. But it’s normal for legacy projects to trigger thousands of queries per view. If that is the case making sense of rack-mini-profiler endless backtraces data could be overwhelming. That why I’ve recently developed a way to simplify this process.

After expanding the traces data, click CMD+A and CMD+C to copy it into the clipboard. Now create a new empty text file and paste clipboard contents. I recommend using vim because modern text editors might not handle larger clipboard sizes. Now you can run the following bash script:

sort traces.txt | uniq -c | sort -bgr > output.txt

It groups and counts the unique lines, and pipes output into a new text file. Now on top of this file, you can see which kinds of queries and method execution hits were repeated most often:

 ...
 119 Reader
 118 Executing action: home
 101 app/controllers/web/static_pages_controller.rb:7:in `home'
 101 SELECT "teams".* FROM "teams" WHERE "teams"."id" = $1 LIMIT $2;
 100 app/controllers/web/static_pages_controller.rb:7:in `map'
 ...

It’s usually enough data to spot precisely where the worse N+1 issues originate from and to validate if the fix resolved the problem.

Use rails-pg-extras to dissect N+1 queries

As you can see, the rack-mini-profiler works for many standard scenarios. However, it’s sometimes a lot of work to simulate certain edge cases in the app’s UI. Also, the workflow of manually copying, parsing, and analyzing log outputs is far from ideal. So, I’ve recently released a small tool that helps me iterate on N+1 issues faster.

My rails-pg-extras gem has been around for a while. It includes an array of helper methods for analyzing the internals of your PostgreSQL database. measure_queries method is the newest addition to the lib’s API. It leverages ActiveSupport instrumentation to display queries executed by running any provided Ruby snippet:

RailsPgExtras.measure_queries { User.limit(10).map(&:team) }

{:count=>11,
 :queries=>
  {"SELECT \"users\".* FROM \"users\" LIMIT $1"=>
    {:count=>1,
     :total_duration=>1.9,
     :min_duration=>1.9,
     :max_duration=>1.9,
     :avg_duration=>1.9},
   "SELECT \"teams\".* FROM \"teams\" WHERE \"teams\".\"id\" = $1 LIMIT $2"=>
    {:count=>10,
     :total_duration=>0.94,
     :min_duration=>0.62,
     :max_duration=>1.37,
     :avg_duration=>0.94}},
 :total_duration=>13.35,
 :sql_duration=>11.34}

In theory, you can extract the same data by piping ActiveRecord logging to STDOUT:

ActiveRecord::Base.logger = Logger.new(STDOUT)

Produces:

D, [2023-03-01T13:57:42.815016 #19045] DEBUG -- :   User Load (1.6ms)  SELECT "users".* FROM "users" ORDER BY "users"."id" DESC LIMIT $1  [["LIMIT", 20]]
D, [2023-03-01T13:57:42.817497 #19045] DEBUG -- :   Team Load (2.1ms)  SELECT "teams".* FROM "teams" WHERE "teams"."id" = $1 LIMIT $2  [["id", "[FILTERED]"], ["LIMIT", 1]]
D, [2023-03-01T13:57:42.819068 #19045] DEBUG -- :   Team Load (1.2ms)  SELECT "teams".* FROM "teams" WHERE "teams"."id" = $1 LIMIT $2  [["id", "[FILTERED]"], ["LIMIT", 1]]
(.....N+1 more lines.....)
D, [2023-03-01T13:57:42.843380 #19045] DEBUG -- :   Team Load (1.0ms)  SELECT "teams".* FROM "teams" WHERE "teams"."id" = $1 LIMIT $2  [["id", "[FILTERED]"], ["LIMIT", 1]]
D, [2023-03-01T13:57:42.844977 #19045] DEBUG -- :   Team Load (1.4ms)  SELECT "teams".* FROM "teams" WHERE "teams"."id" = $1 LIMIT $2  [["id", "[FILTERED]"], ["LIMIT", 1]]

But it’s arguably less readable. And for more complex cases, it is likely to produce an endless log stream instead of an easy-to-comprehend hash aggregating all the necessary data.

Optionally, with the help of marginalia gem, you can add backtraces info to standard ActiveRecord logs output. As an effect it will be included in the measured queries data:

Marginalia gem appends origin line numbers to measured query logs

You have to configure Marginalia to enable this feature:

config/development.rb

Marginalia::Comment.components = [:line]

I’m using the measure_queries helper method for my ongoing Rails performance audits. It has significantly improved my workflow allowing for faster iteration on N+1 related fixes. I no longer have to use a rack-mini-profiler UI. Instead, I can now apply the change, type reload! (aliased to re) straight in the Rails console and quickly measure if a refactored method is faster and triggers fewer queries. Being able to debug on the lower layer of models/services instead of HTTP endpoints is a huge productivity boost.

Since this API is relatively new I’m open to feedback on improving it.

Use specs to prevent N+1 queries

Another way to keep the N+1 problem under control is to leverage your test suite. You can use the same measure_queries to constraint how many queries any Ruby code snippet is allowed to trigger. Arguably the simplest way to use it is on a per-endpoint level with controller specs:

describe UsersController do
  # ...

  describe "index" do
    it "does not exceed queries limit" do
      queries = RailsPgExtras.measure_queries do
        get :index
      end
      expect(queries.fetch(:count)).to be < 5
    end
  end
end

The above spec will fail if the number of queries exceeds a predefined threshold. Alternatively, you can check out the n_plus_one_control gem for more complex rspec matchers related to N+1 queries.

Summary

With adequate tooling, it’s possible to spot and resolve N+1 issues before they reach production. I hope that some of the above tips will help you improve your workflow when fixing these problems.

How to Subscribe to YouTube Channels Without an Account

Thu, 16 Feb 2023 08:09:06 +0100

I’m extremely cautious with YouTube. I’ve lost countless hours because of their flawless recommendation algorithms. In one of my previous posts, I describe how to “cripple” YouTube recommendations UI and start using it more deliberately. I also prefer to use YouTube without logging in, but it makes following my favorite channels difficult. So in this tutorial, I describe how to over-engineer your way to subscribing to YouTube channels without having an account.

How to find an RSS feed of a YouTube channel?

Every YouTube channel has an undocumented RSS feed representing its content. But it’s not available in the UI. Instead, you have to dig into the HTML source code to extract it:

In the channel "Videos" web page source code, search for RSS to find the DOM element with the feed address

Or, even simpler, you can run this JS code in the console to extract an RSS address:

document.querySelector('[title="RSS"]').href

Remember to search "Videos" and not "Home" tab of a channel for the RSS feed

Once you have RSS feeds of your favorite channels, you can use any RSS app to monitor them. I like to use Slack for this purpose. An official Slack RSS app sends a channel message when there’s a new RSS item.

Alternatively, if you’re on a paid Zapier plan, you can batch your RSS notifications with their Delay Zap Action to arrive at a specific time. Or, if you prefer self-hosted solutions, you can check out a Huginn Ruby gem or ActivePieces for similar functionality.

YouTube channels are just an example. You can use this technique for any source exposed via an RSS feed. For me, RSS is a superior way to consume social media. No ads, no click baits, just hand-picked content delivered straight to your Slack channel. At a time of your and not user engagement algorithm choice.

Custom Zapier notifications in Slack

Summary

Consuming most social media via RSS feeds allowed me to reduce my addiction to garbage content. Or at least I can now deliberately choose garbage I like without going on a binge. Most social media platforms prioritize time spent scrolling instead of quality. So similar overengineered tricks could be a way to break their “optimizations” and defend your time and focus.

Easy to Overlook Way to Break Eager Loading in Rails Apps

Tue, 10 Jan 2023 06:09:06 +0100

In theory, configuring eager loading to avoid N+1 issues is straightforward. Chaining includes method is usually enough to ensure that all the relations data is fetched efficiently. However, this rule breaks for some easy-to-overlook edge cases. In this blog post, I’ll describe how to eager load ActiveRecord relations using custom SQL ORDER BY sorting.

How to use eager loading with ORDER BY in SQL?

Based on my experience conducting Rails performance audits, N+1 issues are a primary cause of sub-optimal response times in Ruby on Rails apps. ActiveRecord makes it just too easy to sneak in an N+1 bug in production. It’s easy to miss it in development because you usually work with small datasets.

Let’s analyze a sample scenario of displaying the products of a user:

app/models/user.rb

class User < ApplicationRecord
  has_many :products
end

app/models/product.rb

class Product < ApplicationRecord
  belongs_to :user

  scope :by_rating, -> { order(rating: :desc) }
end

app/controllers/users_controller.rb

class UsersController < ApplicationController
  def index
    @users = User.limit(50).includes(:products)
  end
end

In theory, this implementation should correctly eager-load all the product objects and prevent N+1 queries.

But let’s consider the following view layer implementation:

app/views/users/index.html.erb

<div>
  <% @users.each do |user| %>
    <p> <%= user.id %> products </p>
    <ul>
      <% user.products.by_rating.each do |product| %>
        <li><%= product.name %></li>
      <% end %>
    </ul>
  <% end %>
</div>

The addition of by_rating scope effectively breaks the eager loading. Adding a where call would have the same effect. It is caused by chaining methods that alter the relation SQL and invalidate the previously eager loaded collection. It’s even worse than not using eager loading at all. includes still triggers additional queries, but results are discarded.

A naive approach to avoiding N+1 issues in similar situations could be to work with methods that do not alter the relation SQL. It means that instead of adding by_rating scope, you’d need to use a method that does the sorting in Ruby:

app/views/products/index.html.erb

<% user.products.sort_by { |pr| -pr.rating }.each do |product| %>
  <li><%= product.name %></li>
<% end %>

Similarly, instead of where, you could use the select method. It filters collection in Ruby instead of an SQL WHERE clause.

Using custom relation to prevent N+1 queries

However, there are downsides to using Ruby instead of SQL to transform relation objects. For example, sorting in Ruby can be slower than SQL ORDER BY or even impossible for larger data sizes. And using select to filter the collection means that you’ll discard already initialized ActiveRecord objects, so memory is wasted.

A better way to work with relations that need their SQL customized is to use a dedicated relation method. Let’s see it in action:

class User < ApplicationRecord
  has_many :products
  has_many :products_by_rating, -> { by_rating }, class_name: 'Product'
end

app/controllers/users_controller.rb

class UsersController < ApplicationController
  def index
    @users = User.limit(50).includes(:products_by_rating)
  end
end

app/views/products/index.html.erb

<% user.products_by_rating.each do |product| %>
  <li><%= product.name %></li>
<% end %>

This approach eliminates the additional N+1 queries while preserving custom order without a need for sorting in Ruby.

For non-trivial views, it could be challenging to prevent N+1 issues from emerging accidentally. For example, even custom relations could still trigger N+1 queries if their SQL is further customized. One way to avoid it could be to explicitly call to_a on relations collections before using them. As an effect, ActiveRecord::Relation is transformed into an Array of objects, making it impossible to chain SQL calls.

user.products.to_a.by_rating

# raises => undefined method `by_rating' for Array (NoMethodError)

But it’s arguably not the cleanest solution.

Unfortunatelly, even using the recently added strict_loading on a relation would not detect similar mistakes:

app/models/user.rb

class User < ApplicationRecord
  has_many :products, strict_loading: true
end

User.last(2).map(&:products).map(&:to_a)

# raises => ActiveRecord::StrictLoadingViolationError

User.last(2).map(&:products).map(&:by_rating).map(&:to_a)

# loads the Products but triggers N+1 queries

Summary

You should continuously monitor the number of queries executed by your application. I always recommend using rack-mini-profiler and regularly inspecting SQL queries triggered by each view. This 10-year-old Railscast is still up to date, so the tool did stand the test of time. Getting rid of sneaky N+1 issues could be the lowest-hanging fruit in optimizing your Rails app’s performance.

How to Monitor and Fix PostgreSQL Database Locks in Rails

Tue, 20 Dec 2022 09:09:06 +0100

PostgreSQL database locks usually work seamlessly until they don’t. Before your Rails app dataset and traffic reach a certain scale, you’re unlikely to face any locks-related issues. But if your app suddenly slows down to a crawl, deadlocks likely are to blame. In this blog post, I’ll describe how to monitor database locks and propose best practices to prevent them from causing issues.

PostgreSQL database locks 101

Locks in PostgreSQL are a mechanism for ensuring data consistency in multithreaded environments. This blog post is not trying to be a comprehensive introduction to how locks and related MVVC (Multi-Version Concurrency Control) system works in PG databases. Instead, I’ll focus on the basics to help us understand how locks affect Ruby on Rails apps’ performance and stability.

It’s important to understand that every single write to the database acquires a lock. Well-behaved databases usually commit changes in a few ms, so observing related locks is not straightforward. As a workaround, you can simulate a long-lasting lock by running a similar code in the Rails console:

user = User.last

User.transaction do
  user.update(email: "email-#{SecureRandom.hex(2)}@example.com")
  sleep 25
end

The new column value has to differ from the current one for locks to be acquired.

Now in a separate Rails console process run the locks method from rails-pg-extras gem:

RailsPgExtras.locks(in_format: :hash)

# [
#   {
#     "pid": 64,
#     "relname": "users_pkey",
#     "transactionid": null,
#     "granted": true,
#     "mode": "RowExclusiveLock",
#     "query_snippet": "UPDATE \"users\" SET \"email\" = $1 WHERE \"users\".\"id\" = $2",
#     "age": "PT0.954817S",
#     "application": "bin/rails"
#   },
#   {
#     "pid": 64,
#     "relname": "users",
#     "transactionid": null,
#     "granted": true,
#     "mode": "RowExclusiveLock",
#     "query_snippet": "UPDATE \"users\" SET \"email\" = $1 WHERE \"users\".\"id\" = $2",
#     "age": "PT0.954817S",
#     "application": "bin/rails"
#   },
#   {
#     "pid": 64,
#     "relname": "index_users_on_email",
#     "transactionid": null,
#     "granted": true,
#     "mode": "RowExclusiveLock",
#     "query_snippet": "UPDATE \"users\" SET \"email\" = $1 WHERE \"users\".\"id\" = $2",
#     "age": "PT0.954817S",
#     "application": "bin/rails"
#   },
#   {
#     "pid": 64,
#     "relname": "index_users_on_api_token",
#     "transactionid": null,
#     "granted": true,
#     "mode": "RowExclusiveLock",
#     "query_snippet": "UPDATE \"users\" SET \"email\" = $1 WHERE \"users\".\"id\" = $2",
#     "age": "PT0.954817S",
#     "application": "bin/rails"
#   },
# ...
# ]

As you can see, even a straightforward single-column update operation acquires an array of RowExclusiveLock locks on a corresponding table and all the indexes. Usually, this process works seamlessly behind the scenes. A lock affecting a single row is not critical (apart from slowing down a single request/job) as long as it’s not blocking other queries.

But let’s see what happens when we try to update the same row concurrently.

In the first Rails console process run:

user = User.last

User.transaction do
  user.update(email: "email@example.com")
  sleep 25
end

Then in the second one:

User.last.update(email: "email-#{SecureRandom.hex(2)}@example.com")

and now, in the third Rails console process, we can investigate the collision of concurrent update operations:

RailsPgExtras.blocking(in_format: :hash)

# [
#   {
#     "blocked_pid": 64,
#     "blocking_statement": "UPDATE \"users\" SET \"email\" = $1 WHERE \"users\".\"id\" = $2",
#     "blocking_duration": "PT20.450116S",
#     "blocking_pid": 152,
#     "blocked_statement": "UPDATE \"users\" SET \"email\" = $1 WHERE \"users\".\"id\" = $2",
#     "blocked_duration": "PT8.145238S"
#   }
# ]

As you can see, the acquired lock prevents the second SQL statement from finishing. It’s reported by blocking method from rails-pg-extras. In practice, a web server request trying to execute this SQL statement would be stuck until the first lock is released. This is how misbehaving locks can kill the performance of your application or even cause downtime by exhausting available web server connections.

Monitoring PostgreSQL locks in production

You’re unlikely to sprinkle sleep all over your codebase to simplify investigating database locks. Here’s a sample implementation of a Sidekiq job that can help you monitor long-lasting locks and blocked SQL statements in production:

config/initializers/pg_monitoring.rb

LocksMonitoringJob.perform if ENV["PG_MONITOR_LOCKS"] == "true"

app/jobs/locks_monitoring_job.rb

class LocksMonitoringJob
  include Sidekiq::Worker
  sidekiq_options retry: false

  PG_MIN_LOCK_DURATION_MS = ENV.fetch("PG_MIN_LOCK_DURATION_MS") { 200 }.to_f
  PG_MONITORING_INTERVAL_SECONDS = ENV.fetch("PG_MONITORING_INTERVAL_SECONDS") { 5 }.to_i
  SLACK_WEBHOOK_URL = ENV.fetch("SLACK_WEBHOOK_URL")

  def perform
    locks = RailsPgExtras.locks(in_format: :hash).select do |lock|
      (ActiveSupport::Duration.parse((lock.fetch("age")).to_f * 1000) > PG_MIN_LOCK_DURATION_MS
    end

    if locks.present?
      Slack::Notifier.new(
        SLACK_WEBHOOK_URL,
        channel: "pg-alerts"
      ).ping locks.to_s
    end

    blocking = RailsPgExtras.blocking(in_format: :hash).select do |block|
      (ActiveSupport::Duration.parse((block.fetch("blocking_duration")).to_f * 1000) > PG_MIN_LOCK_DURATION_MS
    end

    if blocking.present?
      Slack::Notifier.new(
        SLACK_WEBHOOK_URL,
        channel: "pg-alerts"
      ).ping blocking.to_s
    end
  ensure
    LocksMonitoringJob.perform_in(PG_MONITORING_INTERVAL_SECONDS) if ENV["PG_MONITOR_LOCKS"] == "true"
  end
end

We use an initializer file to schedule the first task and can optionally disable it by removing PG_MONITOR_LOCKS variable. The job schedules itself every configurable PG_MONITORING_INTERVAL_SECONDS and reports locks and blocked queries that cross a PG_MIN_LOCK_DURATION_MS threshold using slack-notifier gem (check out Slack docs for info on how to configure incoming webhooks). You can tweak the ENV variables to adjust monitoring sensitivity based on your app’s requirements.

If your app is running heavyweight background jobs, you’ll likely have to manually exclude some queries from being raported. And if your current database performance is far-from-perfect, you should bump up the PG_MIN_LOCK_DURATION_MS and gradually lower it when you manage fix the cause of long-lasting locks. Excluding background jobs and data migrations, a healthy database should acquire exclusive locks for a maximum of ~100-200ms.

Running a similar monitoring task can help you spot many lock-related issues early on before they start affecting end users.

So your Slack alerts channel is spammed with long-lasting locks and blocked query notifications. What now?

Overusing ActiveRecord::Base.transaction scope

I’ve seen misbehaving locks with a duration of a few seconds. Surprsingly, the underlying cause wasn’t even directly caused by the database layer. ActiveRecord makes it too easy to wrap any arbitrary chunk of code into a database transaction. Have a look at the following example:

app/models/user.rb

after_create :send_welcome_email

def send_email
  EmailSender.send_welcome_email(self)
end

ActiveRecord callbacks are a usual way to trigger side effects based on the model object’s lifecycle. For example, sending a welcome email to newly registered users is a common use case. Let’s keep discussing whether similar logic belongs in a model aside for now. Instead, let’s focus on how the database layer will react to a similar implementation.

Something to watch out for is that excluding after_commit, all the AR callbacks are implicitly wrapped in a database transaction. So, the above implementation would keep a RowExclusiveLock on the user object for the whole time needed to connect to 3rd party API and send an email. So, instead of the expected max ~200ms threshold, our lock could last up to 60 seconds in case there’s an API timeout.

This example might seem contrived. But I’ve seen multiple Rails apps suffering from long-lasting locks caused by implicit 3rd party HTTP calls executed inside database transactions.

The hidden complexity of Rails ActiveRecord makes it too easy to commit similar mistakes, even for more experienced devs. That’s why proper locks monitoring can help you spot similar issues, even if they go unnoticed during the code review.

Other ways to fix long-lasting locks

Apart from limiting the scope of ActiveRecord transactions, the “simplest” way to decrease locks duration is to optimize the performance of the database layer. Check out this blog post for an in-depth guide on analyzing and fixing PostgreSQL performance with the help of rails-pg-extras gem.

But, fine-tuning SQL queries is often a more time-involved process. So, here’s a list of “quick & easy” ad hoc solutions to try before you manage to deploy proper fixes. These are “hacks”, but they might help if your app is currently unstable because of locks-related issues.

1. Throttle Sidekiq jobs

One way to limit the occurances of locks is to reduce the concurrency of database commits. It will limit the scalability and throughput of your app. But the tradeoff might be worth it if you’re experiencing locks-related downtimes.

I usually recommend implementing the Sidekiq config file in a similar way:

config/sidekiq.yml

:concurrency: <%= ENV.fetch("SIDEKIQ_CONCURRENCY") { 5 } %>
:queues:
  - urgent
  - default
  # ...

Even if you’re currently not planning to tweak the concurrency setting, using an ENV variable will allow you to easily change this value without doing a code-change release. A panic fix could be to throttle concurrency down to 1 to minimize concurrent locks’ impact before you apply the proper solution.

Alternatively, if you know which Sidekiq job is causing the issue, you could throttle it from running concurrently with the help of the sidekiq-unique-jobs gem. This is also a good solution for more heavyweight jobs to limit the max memory usage of Sidekiq.

2. Limit the app’s concurrency

Similar throttling can be done for web app processes. Limiting your app’s concurrency could reduce performance by increasing request queue time. But, it’s possible that the throttled app will be more performant because blocking SQL queries are limited.

You can implement your Puma web server config in a similar way to easily tweak settings without a need for code release:

config/puma.rb

threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }
threads threads_count, threads_count
workers ENV.fetch("WEB_CONCURRENCY") { 2 }
# ...

3. Kill stuck connections

RailsPgExtras.locks and RailsPgExtras.blocking could report that a single long-running query is blocking dozens of other database commits. Depending on the scale of the issue, a similar traffic jam could cause many of your web server connections to get stuck and your app to become unresponsive. A “panic button” solution for this issue that could potentially restore your app’s responsiveness is to kill the offending connection. rails-pg-extras blocking method provides a blocking_pid value you can use to cherry-pick and terminate the stuck SQL query. Additionally, you can use RailsPgExtras.connections to verify a process name that generated the problematic query. It could be helpful for debugging.

You can terminate an SQL query/connection based on its PID value by running the following code:

RailsPgExtras.kill_pid(args: { pid: 4657 })

Optionally, if your database is a bad enough shape that killing a single connection is not enough, you can run:

RailsPgExtras.kill_all

Remember to restart all the web and background worker processes to prevent ghost connections from hanging around.

As I’ve mentioned, these are not proper fixes, but rather “band-aids” that might help in case your app is unstable.

Summary

PostgreSQL database locks usually work seamlessly behind the scenes. But, following the mentioned best practices and configuring correct monitoring can save you a lot of headaches. A proactive approach to monitoring lock-related issues can help you resolve most of the problems before they start affecting your users.

Easy to Miss Way to Optimize ActiveRecord SQL Memory Usage in Rails

Mon, 05 Dec 2022 07:39:06 +0100

By default, Rails ActiveRecord executes most of the SQL queries in a non-optimal way. In this blog post, I’ll describe how to fix this issue to speed up bottlenecks and reduce memory usage.

Bloated ActiveRecord queries

Active Record empowers developers to write fairly complex database queries without understanding the underlying SQL. Unfortunately, the same ActiveRecord is a reason for the majority of the issues that I encounter while conducting my Rails performance audits. Next to the infamous N+1 calls, “bloated” queries are another common problem.

Let’s see it in action:

app/models/user.rb

class User < ApplicationRecord
  # == Schema Information
  #
  # Table name: users
  #
  #  id                     :bigint           not null, primary key
  #  api_auth_token         :string           not null
  #  block_uuid             :uuid
  #  blocked_from           :datetime
  #  blocked_until          :datetime
  #  messaging_active_today :boolean          default(FALSE), not null
  #  pseudonym              :string
  #  uuid                   :uuid             not null
  #  voting_active_today    :boolean          default(FALSE), not null
  #  web_auth_token         :string
  #  team_id                :integer          not null

  # ...

  def blocked?
    return false if blocked_until.nil?
    blocked_until < Time.current
  end
end

Model schema annotations can be added using the annotate_models gem. I highly recommend it!

app/controllers/users_controller.rb

class UsersController < ApplicationController
  def index
    @users = User.order(:updated_at).limit(200)
  end
end

app/views/users/index.html.erb

<div>
  <% @users.each do |user| %>
    <div>
      ID: <%= user.id %>
      Blocked: <%= user.blocked? %>
    </div>
  <% end %>
</div>

It is common for core models in legacy Rails projects to consist of dozens of columns. In the above example, we fetch 200 rows from the database and display info based on the logic implemented in a User model.

SQL query executed in the controlled looks like that:

SELECT "users".* FROM "users" ORDER BY "updated_at" ASC LIMIT 200

Can you spot the issue now? A little hint: *

By default, ActiveRecord fetches ALL the model columns. But, in the above example, we only really need the id and blocked_until columns for our view to display correctly. If you only care about raw data, a recommended practice is using the ActiveRecord pluck method to avoid the overhead of instantiating model objects.

But, in this case, we need the model logic. So to improve performance, we can use a select method:

class UsersController < ApplicationController
  def index
    @users = User.order(:updated_at).select(:id, :blocked_until).limit(200)
  end
end

The above implementation would produce the following SQL query:

SELECT "users"."id", "users"."blocked_until" FROM "users" ORDER BY "updated_at" ASC LIMIT 200

Contrary to pluck, chaining a select method would still return ActiveRecord object instances, so you’ll be able to call methods implemented in the model. One catch is that if you try to read a value from a column that was not fetched from the database, an ActiveModel::MissingAttributeError would be raised.

How to reduce ActiveRecord RAM usage

Now that we’ve discussed how to slim down bloated queries, let’s measure the potential impact on memory usage. memory_profiler gem offers a simple API for measuring in-memory object allocations.

Let’s start with checking how many objects are allocated when we fetch 1000 instances of a sample model with 12 columns:

MemoryProfiler.report do
  User.limit(1000).to_a; 0;
end.pretty_print

We append 0; to prevent IRB from printing the query output to the console and skew memory usage results.

MemoryProfiler outputs loads of data, but we’re interested in the top few lines:

Total allocated: 1723042 bytes (11577 objects)
Total retained:  1536160 bytes (10183 objects)

allocated memory by gem
-----------------------------------
   1522398  activerecord-7.0.4
    176080  activemodel-7.0.4
    ...

Total allocated: 1723042 bytes represents memory usage increase caused by running the code snippet. The absolute value is not important, but it will serve as a reference point when we compare more efficient ways to run this query. The absolute memory usage would vary based on total columns number and the model implementation.

Let’s now see how memory usage looks if we pluck two columns:

MemoryProfiler.report do
  User.limit(1000).pluck(:id, :blocked_until); 0;
end.pretty_print

Total allocated: 204350 bytes (4422 objects)
Total retained:  3252 bytes (48 objects)

allocated memory by gem
-----------------------------------
    141347  activerecord-7.0.4
     40040  local/lib
     ...

That’s a ~90% decrease! But, as previously mentioned, pluck could be limiting in some cases since it returns raw data instead of model objects. So let’s now measure the memory impact of using select to fetch only two columns:

MemoryProfiler.report do
  User.limit(1000).select(:id, :blocked_until).to_a; 0;
end.pretty_print

Total allocated: 613271 bytes (7432 objects)
Total retained:  531648 bytes (6055 objects)

allocated memory by gem
-----------------------------------
    412930  activerecord-7.0.4
    176080  activemodel-7.0.4
    ...

That’s ~65% less than the original * query. And you can still interact with full model APIs as long as non-fetched column values are not accessed.

Memory usage comparison depending on SQL query format

Another advantage of putting bloated SQL queries on a diet is that fetching less columns will decrease the number of disk-read operations. Database read IOPS are sometimes a bottleneck for apps with larger traffic. Fetching fewer columns is a relatively straightforward way to improve this metric.

Summary

Like with most SQL-related issues, the benefits of fine-tuning queries will only provide measurable effect once the app and dataset reach a certain scale. But, I’ve seen significant performance and memory usage improvements from limiting columns fetched from the database. You can check out my blog post about rails-pg-extras to learn how to find queries that are likely to benefit from this optimization technique.

Rails Quick Tip - Use Private Debugging Aliases

Tue, 08 Nov 2022 09:39:06 +0100

I don’t like to type much. Even minor improvements in your debugging workflow are likely to accumulate into huge keystrokes savings over time. In this blog post, I’ll describe a simple way to add debugging shortcuts to the project without modifying the codebase shared with other team members.

My aliases good, your aliases bad

In one of my previous posts, I described a way to improve your productivity by using Rails console aliases. The downside of the described approach is that the .irbrc file is usually committed to the repository.

If you’re working in a larger team, it might be challenging to agree on a definitive list of aliases that everyone finds useful. So instead of starting code review battles on which aliases are worth committing to the shared repo, you can keep your private aliases collection.

In theory, you could leverage an ~/.irbrc file to define them. But, the problem is that if you start an IRB session outside of the context of a Rails project, any of the calls to the custom classes or included gems would break. Also, it does not seem correct to include per-project customizations in global dotfiles.

Instead, you can customize a project without modifying its shared source code. You’ll have to configure git to parse the file with a list of entries that should never be committed to any of the local repositories:

~/.gitconfig

[core]
  excludesfile = ~/.gitignore_global

You can now define any file to be excluded from git commits:

~/.gitignore_global

config/initializers/my_aliases.rb

I use a Rails initializer file to define my custom aliases and local tweaks. Files from config/initializers/ are called when the app’s models have already been loaded so you can interact with all the app’s classes:

config/initializers/my_aliases.rb

User.filter_attributes = []

def me
  User.find_by(email: "my@email.com")
end

def pr1
  Project.find_by(slug: "my-debug-project")
end

def pr2
  Project.find_by(slug: "my-other-debug-project")
end

def refr
  User.recent.each(&:refresh_usage_data)
end

# ...

Sample project-specific aliases and tweaks

I usually keep a dozen of three max four-letters aliases per project. They represent objects and methods I often interact with when debugging locally. Smuggling such global methods past code review would probably not be possible, hence the private aliases workaround.

Summary

I regularly inspect the list of my Rails console commands for potential alias candidates using the lazyme gem. Correctly configured aliases can save you hundreds of keystrokes in just a single day of work. Your fingers will appreciate it.

The In-depth Guide to Caching ActiveRecord SQL Queries in Rails

Mon, 17 Oct 2022 10:39:06 +0200

Caching might seem a perfect solution to “speed up” slow database queries. However, caching in Rails apps can be easily misused, leading to poor maintainability or even slower performance than without it. In this blog post, I’ll discuss the common pitfalls of caching SQL queries in Rails apps. I’ll also describe my toolkit for assessing the cacheability of database queries and techniques for reducing the cost of caching infrastructure.

Rails.cache 101

Ruby on Rails offers a rich toolkit for caching different web application layers. Ranging from simple instance variables, through external in-memory cache shareable between processes, up to HTTP caching relying on headers and status codes. This blog post will focus on caching SQL-related data in separate in-memory storage.

I won’t elaborate on choosing the best in-memory database for your app’s cache. For projects that are just starting, Redis is usually a decent choice. Once the app’s scale increases, it might be a good idea to separate the cache into a Memcached with Dalli gem so that it does not clash with background worker data. Alternatively you can use two separate Redis instances.

You can check your currently configured cache storage by running:

Rails.cache

For external in-memory storage, you should see:

<ActiveSupport::Cache::RedisCacheStore...

<ActiveSupport::Cache::MemCacheStore...

Please consult the documentation of respective gems for configuration details. You can also read the official Rails guide on caching for a detailed comparison between different cache storages supported in Rails. For production applications with non-trivial traffic, the external in-memory store should be the best choice in most cases. An advantage of an external in-memory cache is that it can be shared between different Ruby processes, e.g., multiple Heroku dynos. It means that time-consuming cache write will only be performed once, and all the other dynos can reuse the result.

Rails cache under the hood

Before we discuss how and when we should cache SQL queries, let’s first take a closer look at what it means to “cache” data using Rails framework. In this blog post, we’ll focus on the Rails.cache module. It abstracts away the interface to the underlying data store and exposes simple methods for storing and retrieving cached data.

You use it like that:

important_value = 42

Rails.cache.write("important_value_key", important_value)
Rails.cache.read("important_value_key") # => 42

An often-used feature is the auto-expiry of cached values:

Rails.cache.write("important_value_key", important_value, expires_in: 5.seconds)
Rails.cache.read("important_value_key") # => 42
sleep 6
Rails.cache.read("important_value_key") # => nil

A handy shortcut for automatically refreshing expired data is the fetch method:

Rails.cache.fetch("important_value_key", expires_in: 5.seconds) do
  important_value
end

It reads the cached value if present and refreshes it with the result of executing provided block if the selected key is expired.

A useful trick is to leverage a cache_key method of an ActiveRecord query to ensure that it will be reused.

You can also cache more complex objects:

user = User.find(42)
Rails.cache.write("important_user", user)
cached_user = Rails.cache.read("important_user")
user == cached_user # => true

Here’s where things are starting to get more interesting. User is not a primitive value, but an ActiveRecord object built based on an SQL query results.

So how come that the Rails.cache can reconstruct the object without talking to the database? Introducing the Marshal module:

user = User.find(42)
dumped_user = Marshal.dump(user)
puts dumped_user
# "\x04\bo:\tUser\x11:\x10@new_recordF:\x10@attr..."

This is a serialized representation of our User object containing all the info necessary to restore it from text format back to the program memory. You can do it like that:

restored_user = Marshal.load(dumped_user)
user == restored_user # => true
user.object_id == restored_user.object_id # => false

Remember to avoid loading objects from user-provided input because it can result in a remote code execution vulnerability.

You can see that our restored object equals our original user, but they are now different objects in memory since their object_id attributes don’t match.

Let’s now dig a bit deeper by peeking into the internals of how the cached object binary blobs are stored. If you’re using Redis as your cache database, you can read raw entries using a similar code snippet:

require 'redis'

redis_client = Redis.new(url: ENV.fetch("REDIS_URL"))
puts redis_client.get("important_user")
# "\u0004\bo: ActiveSupport::Cache::Entry\n:\v@value\"\u0002\xFC\u0003x\x9C\x9DU\u007Fo..."

You can notice that the resulting entry is different from output of using Marshal.dump on the same user object. One notable difference is that Rails.cache automatically compresses larger objects using the Zlib::Deflate stdlib module, but it still uses Marshall.dump under the hood.

Now that we’ve covered the basics let’s move on to speeding up our queries.

How to find SQL queries worth caching?

In theory, you could “cache all the things” and Rails would scale…

Nope :(

When I work on speeding up a Rails application, I usually treat applying any backend caching technique as a last resort. The most significant downside of caching is the additional complexity it introduces. By adding a caching layer, you instantly lose a single source of truth trait of your SQL database. Any debugging effort now requires an analysis whether a potential stale cache issue could have affected it.

I could keep on ranting about the downsides of cache layer… My approach is to avoid caching unless the potential cost is potentially worth it. That’s why in this tutorial, I focus on caching SQL queries. I think that it is one of the least complex backend caching techniques that can significantly speed up the application without all the usual downsides.

The best part about SQL queries is that you can quickly find the ones worth caching and measure that expected gain before you deploy the change to production. One tool for finding queries that consume a significant amount of database resources is my rails-pg-extras gem with its calls and outliers methods. Check out my other blog post for more details on using rails-pg-extras to improve PostgreSQL performance.

Now that we know which SQL queries to potentially cache, let’s finally discuss how to do it.

Two Hard Things

The challenging part of leveraging caching is not finding bottleneck queries, but rather deciding which queries can be cached without breaking the business logic. There are no straightforward answers since it differs per project. The best candidates for caching are always queries that are shared between multiple users. E.g. results of a popular or default search query.

I’m not a fan of “smart” cache expiry policies since it can add a hard-to-maintain complexity. In theory, you could base your cache expiry keys on the updated_at attribute of an object. But, in Rails, it’s straightforward to modify objects bypassing all the callbacks magic, thus leading to hard-to-debug stale cache issues. And what about updated_at of the object relations? Do you have to keep all the relations structures in sync with every state update now?… Even if the initial version of the “smart” cache expiry policy might look straightforward, keeping the complexity at bay with the new features requested might not be possible.

If you’re just starting with caching, usually, the simplest way to implement it is to add a fixed expiry threshold. The longer it is, the greater will be the performance benefit, but data displayed to users will be more outdated. Unfortunately, there are also no simple answers to configuring the correct cache expiry threshold. You can check out my other blog post for tips on using dynamic config in Rails to easily tweak the values in production and observe results.

But, as an example, let’s assume you’re optimizing an endpoint with traffic of 10 RPS. If all the requests generate the same slow SQL query, caching its results for 1 second would speed up ~90% of all traffic. Depending on your app’s traffic, even a small caching threshold could translate to huge performance and scalability benefits.

It’s impossible to give universal advice on configuring the “perfect” caching policies. So let’s instead move on to describing how to store cached data correctly.

Caching SQL queries in Rails

In the following examples, we’ll be working with this sample ActiveRecord model:

model User < ApplicationRecord
  scope :slow, -> {
    where("SELECT true FROM pg_sleep(1)").order(:created_at).limit(10)
  }

  def to_json
    {
      id: id,
      email: email,
      nickname: nickname,
      created_at: created_at
    }
  end
end

The slow scope is supposed to return 10 User model objects after an artificial 1-second delay.

Let’s now consider the following code snippet:

@users = Rails.cache.fetch("my_slow_query", expires_in: 1.minute) do
  User.slow
end

The big question is whether this caching technique will improve performance?

Nope :(

The previously mentioned trick of inspecting raw cache values can help us understand why this is a bug:

puts redis_client.get("my_slow_query")
# => "ActiveSupport::Cache::Entry\t:\v@valueo: User::ActiveRecord_Relation...

If the cache entry is unreadable you can save it with compress: false to simplify debugging.

We’ve accidentally cached an ActiveRecord::Relation object representing an SQL query that has not yet been executed. You can read my blog post about using load_async API in Rails for more in-depth info on how and when ActiveRecord queries are triggered in Rails. Query object would only be executed after instantiating it from the cache, meaning that each request would suffer from the additional 1-second slowdown. It would not be possible to debug this issue without inspecting raw cache entry because any attempt to display ActiveRecord::Relation executes it obfuscating the original stored format.

A better way to cache this query would look like that:

@users = Rails.cache.fetch("my_slow_query", expires_in: 1.minute) do
  User.slow.to_a
end

Appending to_a forces query execution, and the resulting User objects are serialized and stored in the cache database. Now reading the contents of the cache would not execute any SQL queries.

How to spend less money on an in-memory cache?

But, here comes the tricky part. Do you need to store full-blown ActiveRecord objects? Let’s compare the size of cached ten full User ActiveRecord objects with their JSON representation:

Rails.cache.write("user_ar_objects", User.slow.to_a)
Rails.cache.write("user_json_objects", User.slow.map(&:to_json))

redis_client.get("user_ar_objects").size # => 1721
redis_client.get("user_json_objects").size # => 1064

As you can see, storing JSON representation takes almost half the space. Another benefit is the smaller memory usage required to instantiate JSON data compared to full-blown AR objects.

Optimizing the size of your cached data is critical. Compared to the SSD disk space, memory storage is expensive. If you’re using Heroku, 1GB of Memcachier storage costs $70/month, and 100GB would incur a monthly cost of $4000 (!!). You could reduce the price by spinning up a custom infrastructure, but this is just an example. The takeaway is that in-memory storage is expensive.

How to minimize cache size by storing only IDs

There’s a way to significantly limit the amount of data you have to cache while maintaining good enough performance. The slowdown in SQL queries is usually caused by complex search criteria that span multiple joined tables. But the outcome of the query are often objects from a single table. It means that in many cases, we should be able to store just the IDs of original query results and later reuse them to fetch objects without all the search logic overhead. Based on my tests saving only integer IDs takes ~10% of the in-memory space needed to store a JSON representation of an object with just a few attributes.

Let’s see it in action:

users_ids = Rails.cache.fetch("my_slow_query_ids", expires_in: 1.minute) do
  User.slow.ids
end

@users = User.where(id: users_ids)

We’re caching only user IDs and later use them to fetch objects efficiently.

Unfortunately, this implementation has a sneaky bug. Maybe you’ve also learned the hard way that PostgreSQL does not preserve the order of IDs passed as a search param… (╯°□°）╯︵ ┻━┻

It means that while returned objects would be correct, without an explicit ORDER BY, their ordering would be random. This is critical if you’re paginating results because subsequent pages could return duplicates and omit some of the rows.

One way to fix it is by reordering the objects with Ruby:

users_ids = Rails.cache.fetch("my_slow_query_ids", expires_in: 1.minute) do
  User.slow.ids
end

@users = User.where(id: users_ids).sort_by { |u| user_ids.index(u.id) }

Based on my benchmarks fetching by IDs and sorting 1000 ActiveRecord objects in Ruby has an overhead of ~50ms. That’s usually an acceptable performance for fetching SQL data compared to an unoptimized database query. If you have to work with more objects, you should probably consider adding pagination.

But, this technique forces us to execute the AR query object, which might not always be optimal. There are scenarios where you need the raw query to add eager loading or merge it with other queries. In this case, you can use an ugly way of custom sorting that’s supported by PostgreSQL (copy-paste source):

class User
  def self.find_as_sorted(ids)
    values = []
    ids.each_with_index do |id, index|
      values << "(#{id}, #{index + 1})"
    end
    relation = self.joins("JOIN (VALUES #{values.join(",")}) as x (id, ordering) ON #{table_name}.id = x.id")
    relation = relation.order('x.ordering')
    relation
  end
end

users_ids = Rails.cache.fetch("my_slow_query_ids", expires_in: 1.minute) do
  User.slow.ids
end

@users = User.where(id: users_ids).find_as_sorted(user_ids)

This method has a comparable overhead of ~50ms for 1000 objects. But, it produces a huge SQL query with the custom sorting order hardcoded. Always measure the performance impact if you’re about to merge it with other queries.

Now our cached results are correctly sorted, taking up only a fraction of the original space. This technique allows you to aggressively cache more data without bloating the costs of your in-memory database.

┳━┳ ヽ(ಠل͜ಠ)ﾉ

That’s another advantage of caching just the critical data, i.e., bottleneck SQL results, instead of entire HTML views. Storage space needed is an order of magnitude smaller.

Summary

The best way to implement caching is to avoid it. So, please always double-check if adding a database index cannot save you from developing a complex cache expiration strategy. But, if you have to do it, I hope some of the above tips will prove helpful.

Rails, PostgreSQL Performance Audit and Tuning Consultant for Hire

How to Improve Rails Caching with Brotli Compression

Brotli vs Gzip 101

How to use Brotli for Rails cache?

Using and misusing in-memory cache store

Nested cache layers

Summary

Five Easy to Miss Performance Fixes for Rails Apps

Eliminate request queue time

Enable cache and Redis connection pool

Fine-tune database connections pool

Customize PostgreSQL config

Optimize HTTP layer config

Client-side caching

Use Cloudflare CDN

Summary

How to Freeze ETH and ERC20 Tokens using Smart Contract

Decentralized trust

Can you trust Smart Contracts?

Bugs and deliberate scams

Frontend interface

Government censorship

User error

Locker smart contracts

Locker

LockerETH

LockerPriv

LockerETHPriv

Summary

How to Find, Debug and Fix N+1 Queries in Rails

How to simulate N+1 issues in development

How to analyze and debug local N+1 issues

Use rails-pg-extras to dissect N+1 queries

Use specs to prevent N+1 queries

Summary

How to Subscribe to YouTube Channels Without an Account

How to find an RSS feed of a YouTube channel?

Summary

Easy to Overlook Way to Break Eager Loading in Rails Apps

How to use eager loading with ORDER BY in SQL?

Using custom relation to prevent N+1 queries

Summary

How to Monitor and Fix PostgreSQL Database Locks in Rails

PostgreSQL database locks 101

Monitoring PostgreSQL locks in production

How to resolve lock-related issues in Rails apps

Overusing ActiveRecord::Base.transaction scope

Other ways to fix long-lasting locks

1. Throttle Sidekiq jobs

2. Limit the app’s concurrency

3. Kill stuck connections

Summary

Easy to Miss Way to Optimize ActiveRecord SQL Memory Usage in Rails

Bloated ActiveRecord queries

How to reduce ActiveRecord RAM usage

Summary

Rails Quick Tip - Use Private Debugging Aliases

My aliases good, your aliases bad

Summary

The In-depth Guide to Caching ActiveRecord SQL Queries in Rails

Rails.cache 101

Rails cache under the hood

How to find SQL queries worth caching?

Two Hard Things

Caching SQL queries in Rails

How to spend less money on an in-memory cache?

How to minimize cache size by storing only IDs

Summary