External USB storage is preferred, as internal router memory is limited to a few hundred MB and Tailscale binaries are about 50 MB.

Prepare USB drive

The first step is to prepare a USB drive with an ext4 file system. It can be a partition or the entire USB can be formatted as an ext4 file system. It needs to be done on a separate Linux system. (See the documentation on how to format a USB drive on different systems)

1
2
3
4
5
6

# check which devices are the USB drives (match TRAN=usb)
lsblk -o NAME,MODEL,TRAN,TYPE,SIZE,MOUNTPOINT

# format the entire USB as an ext4 file system and assign label `USB128GB`
# replace sdX with the device name from `lsblk` output
mkfs.ext4 -L 'USB128GB' /dev/sdX

The next step is to download the Entware installer and save it to the USB drive. The installer should match the router architecture (mipsel, mips or aarch64). See the documentation section 3 to identify which router model requires a specific architecture. The example below uses mipsel architecture.

1
2
3
4
5
6
7
8
9
10
11
12
13

# mount the file system
mkdir /mnt/usb # create a mount point
mount -t ext4 /dev/sdX /mnt/usb # replace sdX with the device name

# create install directory
mkdir /mnt/usb/install

# download the Entware installer for mipsel architecture
curl -O --output-dir /mnt/usb/install \
  https://bin.entware.net/mipselsf-k3.4/installer/EN_mipsel-installer.tar.gz

# unmount the USB drive
umount /mnt/usb

Install Tailscale on the router

After umount /mnt/usb, it is safe to remove the USB drive and plug it into the router USB port. The drive should appear on the router System Dashboard section USB Drives and printers and in the Applications page under the USB Devices section. Make sure that it shows the correct label set before, in our example, USB128GB. See the documentation for example screenshots.

Go to the OPKG page and select appropriate device in the Drive dropdown. Click the Save button.

Go to the Diagnostics page and check System Log for the "Entware" installed! message and the default ssh login and password.

1
2

Dec 7 11:50:02 ndm Opkg::Manager: /opt/etc/init.d/doinstall: Log on to start an SSH session using login - root, password - keenetic.
Dec 7 11:50:02 ndm Opkg::Manager: /opt/etc/init.d/doinstall: [5/5] "Entware" installed!

Add a section to .ssh/config

1
2
3
4
5
6
7

cat >> ~/.ssh/config <<'EOF'

Host keenetic
  Hostname 192.168.1.1
  User root
  Port 222
EOF

Optionally, put your ssh public keys into /opt/etc/dropbear/authorized_keys for passwordless login.

1

scp -O ~/.ssh/id_ed25519.pub keenetic:/opt/etc/dropbear/authorized_keys

Log in with ssh using the default password (see the System log above):

1

ssh keenetic

Change the default password:

1

passwd

Update the OPKG packages:

1

opkg update

Install Tailscale packages:

1

opkg install iptables tailscale

Bring the Tailscale node up

• The tailscaled won’t be able to modify the system resolv.conf, so use --accept-dns=false.
• The system iptables rules are periodically reset by the router software, so we can’t rely on tailscaled to maintain them and need to turn them off: --netfilter-mode=off.
• We want to be able to ssh into the router through the Tailnet, so use --ssh (it works only through the Tailscale web admin panel).
• We want to access the subnet connected to the router through the Tailnet, so use --advertise-routes 192.168.1.0/24 (replace with your own subnet address).
• We want to use the route as an exit node, so use --advertise-exit-node.

1
2

tailscale up --accept-dns=false --netfilter-mode=off --ssh \
  --advertise-routes 192.168.1.0/24 --advertise-exit-node

Configure netfilter rules

Without netfilter rules, the exit node and advertised routes won’t work. We need to set up custom hooks for Keentic to configure netfilter rules. We need to create two files in the hook subdirectory /opt/etc/ndm/netfilter.d (See the documentation).

/opt/etc/ndm/netfilter.d/tailscale-filter.sh to configure the default filter table:

Set the variable ROUTER_TAILSCALE_IP to the IP returned by the tailscale ip -4 command on the router or look up in tailscale status output on another Tailscale host.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

#!/bin/sh

[ "$type" = "ip6tables" ] && exit 0   # check the protocol type in backward-compatible way
[ "$table" != "filter" ] && exit 0   # check the table name

ROUTER_TAILSCALE_IP=100.X.X.X/32 # Tailscale IP address assigned to this router

# Create chain only once; skip if it already exists
if iptables -w -N ts-forward >/dev/null 2>&1; then
  iptables -w -A ts-forward -i tailscale0 -j MARK --set-xmark 0x40000/0xff0000
  iptables -w -A ts-forward -m mark --mark 0x40000/0xff0000 -j ACCEPT
  iptables -w -A ts-forward -s 100.64.0.0/10 -o tailscale0 -j DROP
  iptables -w -A ts-forward -o tailscale0 -m conntrack ! --ctstate RELATED,ESTABLISHED -j DROP
  iptables -w -A ts-forward -o tailscale0 -j ACCEPT
  iptables -w -I FORWARD 1 -j ts-forward
fi

if iptables -w -N ts-input >/dev/null 2>&1; then
  iptables -w -A ts-input -s $ROUTER_TAILSCALE_IP -i lo -j ACCEPT
  iptables -w -A ts-input -s 100.115.92.0/23 ! -i tailscale0 -j RETURN
  iptables -w -A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP
  iptables -w -A ts-input -i tailscale0 -j ACCEPT
  iptables -w -A ts-input -p udp -m udp --dport 41641 -j ACCEPT
  iptables -w -I INPUT 1 -j ts-input
fi

/opt/etc/ndm/netfilter.d/tailscale-nat.sh to configure nat table:

1
2
3
4
5
6
7
8
9

#!/bin/sh

[ "$type" = "ip6tables" ] && exit 0   # check the protocol type in backward-compatible way
[ "$table" != "nat" ] && exit 0   # check the table name

if iptables -w -t nat -N ts-postrouting >/dev/null 2>&1; then
  iptables -w -t nat -A ts-postrouting -m mark --mark 0x40000/0xff0000 -j MASQUERADE
  iptables -w -t nat -I POSTROUTING 1 -j ts-postrouting
fi

The router software resets iptables rules frequently, and it can do this even while these hooks are running, so it is normal to see iptables errors in the router logs. Verify that the rules are applied properly with ssh keenetic iptables-save |grep ts-.

The rules are created by running tailscale with --netfilter-mode=on and saving the rules with iptables-save.

If the kernel is unable to reclaim enough pages when memory.current > memory.max, then the OOM killer is invoked, and by default, the largest process in the cgroup is killed. The kernel does 16 attempts to reclaim the pages before invoking the OOM killer.

If the kernel is unable to reclaim enough pages when memory.current > memory.high, then the OOM killer is never invoked, but the allocating process (not the entire cgroup) is throttled proportionally to the number of pages above memory.high limit.

With the classic split-LRU, the reclaim process checks pages at the tails of the inactive file LRU and, if swap is enabled, of the inactive anon LRU.

• If a page doesn’t have its “accessed” bit set by the CPU, then it is considered inactive and can be reclaimed.
• If a page in the inactive file LRU isn’t dirty and doesn’t need to be written back to disk, then it is discarded and reclaimed. If it is dirty, then it is scheduled for writeback but can’t be reclaimed yet in this reclaim pass.
• If a page in the inactive anon LRU already has a copy in swap (it was swapped in but wasn’t modified by the process), then it is discarded; otherwise, it is scheduled for swap out.

The kernel scans the inactive file and the inactive anon LRUs proportionally to vm.swappiness settings. If the kernel needs to reclaim 200 pages and vm.swappiness has the default 60 value, the kernel tries to reclaim 60 pages from the inactive anon LRU and 140 pages from the inactive file LRU. If the OOM is about to be invoked, then this proportion is ignored and the kernel reclaims whatever it can get.

The swap usage for a cgroup is controlled with memory.swap.max and memory.swap.high limits. memory.swap.max is the hard limit for swap usage and exceeding memory.swap.high causes throttling. Default values are max, so the limits are disabled.

When inactive LRUs are reduced by the reclaim process below a specific proportion between active and inactive LRU size, then the kernel shrinks an active LRU by moving pages from its tail almost unconditionally to the corresponding inactive LRU to restore the minimal proportion.

Originally published as an answer on serverfault.com.

Digging through the web, mailing lists and documentation, I discovered that it is a known and documented problem with the default system malloc. The solution is to replace it with jemalloc or tcmalloc, which is also documented. In my case, memory consumption dropped by 40%.

Curiously, it survived without OOMs for months after the last InnoDB size change. As I discovered, MariaDB can monitor for memory pressure events through /proc/pressure/memory and, on detecting memory pressure, it drops all non-dirty pages from the InnoDB buffer pool and marks them with MADV_FREE. That hovering just a few MB from the memory limit was the result of this behaviour, as the kernel reclaimed only the small amount of pages. After the OOM, I added the MemoryHigh limit to the MariaDB systemd unit to give MariaDB a bit of early memory pressure, and was surprised that it wasn’t able to fill its InnoDB buffer. The logs indicated that the InnoDB buffer was reset multiple times per hour. That behaviour to drop almost the entire InnoDB buffer is a bit of an overreaction, and it was disabled by default in MariaDB 10.11.12 (not yet in the Bookworm).

Then I found an old GitHub issue for the Terraform AWS provider that suggested increasing default timeouts to 15 minutes. Given that the DMS replication instances use the same classes as EC2, I expected them to start within a few minutes and waited for 10 minutes before assuming that it hung. It actually took 17 minutes to start an instance in my case. The current default timeout for the aws_dms_replication_instance terraform resource is 40 minutes.

When I got the instance running, the fun didn’t end. First, it didn’t like the database password due to some special characters. The database accepted that password. The AWS Secrets Manager also accepted it. DMS - nope, there are some characters I can’t handle. Ok, no problem, just generated a long alphanumeric password. Then I tried to run ‘Pre-migration assessments’. One of the tests failed to run at all, but no error logs again. Try to run it again - the assessment fails to run at all. Ok, not important, move on. Try to run the migration task itself. It failed. CloudWatch logs contain a lot of output, but don’t tell why the task failed. Ok, when you click on the task status in the console, it shows that it ran out of memory. With a database that can fit entirely into the memory even on t3.micro. Ok, another 20 minutes of waiting to change the instance class. Try again, run out of memory again. Didn’t log the memory failure into CloudWatch logs. Another 20 minutes of waiting.

When I finally got the tables dumped into S3 in Parquet format and moved to the final goal of this adventure: run AWS Macie to detect sensitive information. Another disappointment - it detected only two entries, and one was a false positive. In the end, it was more efficient to review the SQL dump manually - I spent half the time I wasted on DMS and Macie and found more than 30 columns in different tables with sensitive information.

The official CrowdSec documentation doesn’t provide instructions on data migration from SQLite to MySQL or MariaDB, and expects the user to re-register all machines and bouncers, which is inconvenient.

Let’s try a straightforward approach: dump the SQLite database into plain SQL and import it into MariaDB (we are running Ubuntu 24.04 Noble).

We need to create a database for CrowdSec first. We can follow the instructions from the documentation:

1
2
3
4

mysql> CREATE DATABASE crowdsec;
mysql> CREATE USER 'crowdsec'@'%' IDENTIFIED BY '<password>';
mysql> GRANT ALL PRIVILEGES ON crowdsec.* TO 'crowdsec'@'%';
mysql> FLUSH PRIVILEGES;

We will need mysql and sqlite3 on the host where we have CrowdSec installed:

1

sudo apt install mariadb-client sqlite3

Check the location of the CrowdSec SQLite database file:

1

cscli config show --key "Config.DbConfig.DbPath"

Let’s try the dump-pipe-import:

1
2
3
4
5
6
7
8

# sqlite3 `cscli config show --key "Config.DbConfig.DbPath"` '.dump' |\
    mysql -h mysqlhost -u crowdsec -p crowdsec
Enter password:
--------------
PRAGMA foreign_keys=OFF
--------------

ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'PRAGMA foreign_keys=OFF' at line 1

Unfortunately, SQLite and MySQL/MariaDB have incompatible syntax.

Let’s look for a solution. There is a Python tool, sqlite3-to-mysql, which does exactly what we need: transfer data from SQLite3 to MySQL.

1
2
3
4

apt install python3-venv
python3 -m venv .venv
source .venv/bin/activate
pip install sqlite3-to-mysql

Now we can convert the data, but stop the CrowdSec first, so we don’t get inconsistent data.

1

systemctl stop crowdsec

Now convert the data:

1

sqlite3mysql -f `cscli config show --key "Config.DbConfig.DbPath"` -h mysqlhost -d crowdsec  -u crowdsec -p

The data conversion runs without errors, and we get the data into MariaDB.

Now we can update the /etc/crowdsec/config.yaml to use MariaDB (See the documentation):

1
2
3
4
5
6
7
8
9
10

db_config:
  log_level: info
  # type: sqlite
  # db_path: /var/lib/crowdsec/data/crowdsec.db
  type: mysql
  db_name: crowdsec
  user: crowdsec
  password: <password>
  host: mysqlhost
  port: 3306

But when we try to start CrowdSec, it fails with a fatal error:

1
2

level=fatal msg="unable to create database client: failed creating schema resources: sql/schema: modify \"machines\" table: Error 1833: Cannot
 change column 'id': used in a foreign key constraint 'alerts_FK_0_0' of table 'crowdsec.alerts'"

There seems to be a difference in CrowdSec database schemas between SQLite and MySQL/MariaDB. Let’s try to work around that.

First, recreate the database to clear all the imported data.

1
2

DROP DATABASE crowdsec;
CREATE DATABASE crowdsec;

Start CrowdSec with systemctl start crowdsec. It is going to fail, but it will create the correct database schema. (Backup your local_api_credentials.yaml - it overwrote it with new credentials, but I can’t repeat this in a clean environment.)

Now we can try to convert the data again, but without creating the schema (option -K) this time (it also uses a hardcoded path to the SQLite database):

1

sqlite3mysql -K -f /var/lib/crowdsec/data/crowdsec.db -h mysqlhost -d crowdsec  -u crowdsec -p

Now we can start CrowdSec with systemctl start crowdsec, and it works without any issues.

False, even if a system has a lot of memory, swap allows for better memory utilisation by swapping out allocated but rarely used anonymous memory pages.

Swap is going to slow down your system by its mere presence

False, as long as the system has enough memory, there would be very little or no swap-related I/O, so there is no slowdown.

It is really bad if you have some memory swapped out ¹

False, the kernel swapped out some unused pages, and the memory can be allocated for something more useful, like the file cache.

swap is going to wear out your SSD ²

False, as long as there is no swap-related I/O, there is no wearing out of the SSD. And modern SSDs have enough resources to handle swap-related I/O anyway.

Swap is an emergency solution for out-of-memory conditions

False, once your working set exceeds actual physical memory, swap makes things worse, causing swap thrashing and a slower OOM trigger.

Swap allows running workloads that exceed the system’s physical memory

False, once your active working set exceeds actual physical memory, the performance degradation is exponential. If we assume a 10³ difference in latency between RAM and SSD with random 4K page access, then we can calculate that a 0.1% excess causes a 2x degradation, 1% - 10x degradation, 10% - 100x degradation.

Swap causes gradual performance degradation

Under stable workloads, until the active working set exceeds physical memory, swap improves performance by freeing unused memory. Once the working set exceeds physical memory, performance degradation is exponential and the system gets unresponsive very quickly.

On a desktop system, when an inactive application gets swapped out, switching back to it would feel slow.

Kernel evicts program executable pages, making the system unresponsive.

Active executable pages are explicitly excluded from reclaim. The kernel reclaims inactive file cache and inactive anonymous pages first, proportionally to vm.swappiness setting. Then it starts cannibalizing active file cache and anonymous pages, but executable pages are explicitly excluded from that. The system becomes unresponsive due to I/O starvation, because all file cache is dropped. Also, when system memory is below the low watermark, any task that needs a free memory page causes the kernel to go into direct reclaim and the task gets blocked until free pages are found.

Swap size should be double the amount of physical memory.³

False. Unless the system has megabytes of memory instead of gigabytes. If you allocate more than a few GB of swap size, you are going for a long swap thrashing session when you run out of memory and before OOM gets triggered.

The proper rule of thumb is to make the swap large enough to keep all inactive anonymous pages after the workload has stabilized, but not too large to cause swap thrashing and a delayed OOM kill if a fast memory leak happens. For an average system with 2 GB - 128 GB RAM, you start by adding a 256 MB swap file, and if it fills up entirely, increase it by another 256 MB. The step can be increased to 512 MB - 1 GB on larger systems with faster storage.

Swap use begins based on the vm.swappiness threshold, e.g. when 40% of RAM remains for vm.swappiness=40. ⁴

False. Before the introduction of the split-LRU design in kernel version 2.6.28 in 2008, there used to be a different algorithm that used the percentage of allocated memory, but it was more complicated and with the vm.swappiness=40, it wouldn’t start swapping even if all memory was allocated from processes and with the default vm.swappiness=60 it would start swapping at 80% memory allocation. This algorithm is no longer in use.

Swap aggressiveness is configured using vm.swappiness and it is linear between 0 and 100 ⁵

False. vm.swappiness was first described in the kernel documentation in 2009 with the following text:

This control is used to define how aggressive the kernel will swap memory pages. Higher values will increase aggressiveness, lower values decrease the amount of swap. A value of 0 instructs the kernel not to initiate swap until the amount of free and file-backed pages is less than the high water mark in a zone.

It doesn’t say that the relation between vm.swappiness and aggressiveness is linear, but people made assumptions.

This description is still present in some texts on kernel.org (this file isn’t present in the kernel tree anymore, and it wasn’t updated since 2019).

The documentation was updated in 2020 to a more appropriate description and the values up to 200 were allowed.

With vm.swappiness=0 kernel won’t swap

False, if the kernel hits the low water mark in any zone, then it is going to swap anyway.

With vm.swappiness=100 kernel is going to swap out everything from memory right away

False, if there is no memory pressure, the kernel isn’t going to swap anything.

vm.swappiness=60 is too agressive ⁶

False, the vm.swappiness value 60 means that anon_prio is assigned the value of 60 and file_prio the value of 200 - 60 = 140. The resulting ratio 140/60 means that the kernel would evict 2.33 times more pages from the page cache than swap out anonymous pages.

The default value of 60 was chosen with the assumption that the file I/O operations, which tend to be sequential, are more effective than random swap I/O, but this applies to rotating media like HDDs only. For SSDs, vm.swappiness=100 is more appropriate.

As the documentation states:

For in-memory swap, like zram or zswap, as well as hybrid setups that have swap on faster devices than the filesystem, values beyond 100 can be considered

vm.swappiness=10 is just the right setting and makes your system fast

This value gives a ratio of 19 times preference for discarding page cache over swapping out. Your system is going to have a lot of unused anon pages sitting around while churning through file cache pages, making it less effective.

Swap won’t happen if there is some free RAM.

False. If a process runs within a cgroup with defined memory limits, it can be swapped out, even though the system still has a lot of free memory. Swap and OOM can also be triggered due to memory fragmentation when high-order allocations fail, even though there are a lot of free low-order pages.

Swap happens just randomly, when the kernel has nothing to do

False. Swap happens when memory allocation brings the number of free memory pages below the low watermark specified for a memory zone. See /proc/zoneinfo and this question on Unix.StackExchange.

Swapping over NFS is a good idea. ⁷

False. It is very slow, and any packet lost/delayed on the network would cause the system to hang.

OOM won’t trigger if there is swap enabled. ⁸

False. OOM is triggered regardless of swap being enabled or disabled, full or empty.

OOM won’t trigger if there is some free RAM.

False. Swap and OOM can be triggered due to memory fragmentation when high-order allocation fails, even though there are a lot of free low-order pages. ⁹

OOM kills a random process.

The current Linux kernel just kills a process with the largest RSS+swap usage (with per-process OOM score adjustable through /proc). In v5.1 (2019), it dropped the heuristic to prefer to sacrifice a child instead of the parent. In v4.17 (2018), CAP_SYS_ADMIN processes lost their 3% bonus. Before v2.6.36 (2010), it used to be much more complicated and involved factors like forking, process runtime, nice values, but at least this is described in the current man 5 proc. But enabling vm.oom_kill_allocating_task sysctl can cause killing a random process because the random process can be the last one trying to allocate memory and failing.

You can predict OOM. ¹⁰

People sometimes assume that you can get metrics from /proc/meminfo and elsewhere, make some calculations, and predict OOM. Even Kubernetes does some naive calculations trying to determine the working set size.

But you can’t predict OOM. The kernel itself can’t predict OOM. There is no precise information readily available to make this prediction. The kernel doesn’t know how much memory is reclaimable. It doesn’t know the exact working set size and how much memory is active and inactive, despite having appropriate fields in /proc/meminfo (See another blog post for details). The hardware doesn’t provide this information, and it is too expensive to track in the kernel from a performance point of view. The kernel has to go through the reclaim process and check each memory page if it has the Accessed flag set before reclaiming it. Only after failing the reclaim process multiple times does the kernel invoke OOM.

• Active anon pages
• Active file pages
• Inactive anon pages
• Inactive file pages
• (unevictabled pages)

The total sizes of these pages are reported in vmstat and /proc/meminfo

1
2
3
4
5
6
7
8
9
10
11
12
13
14

$ vmstat -s | head -5
    131669088 K total memory
     69020576 K used memory
     56007124 K active memory
     29691936 K inactive memory
      6140772 K free memory

$ cat /proc/meminfo | grep -i active
Active:         56007676 kB
Inactive:       29691936 kB
Active(anon):   34381776 kB
Inactive(anon): 16134108 kB
Active(file):   21625900 kB
Inactive(file): 13557828 kB

These lists are maintained for the purpose of the page reclaim. When the kernel needs to allocate some memory pages and there are not enough free pages, the kernel goes through the LRU lists, trying to find pages that can be reclaimed. The pages are organised in LRU lists so the kernel can reclaim these pages that weren’t used recently and are supposedly least likely to be used soon.

The kernel goes through both inactive anon and inactive file LRU lists, starting from their tails while maintaining the proportion specified by vm.swappiness between the reclaimed anon and file pages. With the default value vm.swappiness=60, when the kernel needs 200 pages, it is going to reclaim 60 pages from inactive anon LRU and 140 pages from inactive file LRU, but if there are not enough pages available, it won’t keep the balance and is going to reclaim whatever it finds.

When a file-backed page is reclaimed, it can be discarded right away if it isn’t modified, or it needs to be written back into the file. An anon page always needs to be written into a swap when reclaimed, so it is more expensive to reclaim anon pages.

So far, so good - we established the purpose of the separation of the memory into active/inactive, so the kernel knows which pages to reclaim.

Now the main question: how exactly does the kernel decide which page is active and which is not?

The problem is that the kernel can’t really track all memory access, and the hardware provides only one single bit of information - if the page was ever accessed. It doesn’t know when it was accessed and how many times. The kernel works around this by periodically scanning the pages and clearing the ‘Accessed’ bit and looking at this bit again on the next scan. But it doesn’t scan all the pages all the time. Only when the kernel needs some memory pages, it scans the tail of the inactive LRU, and it stops the scan when it finds enough pages. So, most of the pages don’t get activity information updated.

To illustrate this: a newly allocated page starts its life on the head of the inactive LRU with the ‘Accessed’ bit reset to 0. As the other new pages are allocated, our page is getting pushed back to the tail of the inactive LRU. If the page gets accessed by the user space program, the CPU sets the ‘Accessed’ bit to 1. The page can be accessed many times, but it is still considered inactive until the reclaim scan reaches this page position in the inactive LRU from the tail. Pages get promoted to the active LRU on the second (some are on the first) scan if they have the ‘Accessed’ bit set. If a page doesn’t get promoted to the active LRU, it gets reclaimed if it is at the tail end of the inactive LRU and the kernel needs some free pages. Once a page gets to the active LRU, it is considered active even if it isn’t accessed there at all. Until the active list grows too large or the inactive list gets too small by the reclaim, there is no change to the active list and pages in the active list are not scanned at all and more active pages are never moved to the head within the active LRU.

The kernel maintains a specific ratio between the active and inactive LRU, depending on the memory size (since v4.10):

1
2
3
4
5
6
7
8
9
10

 * total     target    max
 * memory    ratio     inactive
 * -------------------------------------
 *   10MB       1         5MB
 *  100MB       1        50MB
 *    1GB       3       250MB
 *   10GB      10       0.9GB
 *  100GB      31         3GB
 *    1TB     101        10GB
 *   10TB     320        32GB

When the kernel needs to grow the inactive LRU, it moves pages from the tail of the active LRU regardless of their activity status, except for file-backed executable pages, which get promoted back to the head of the active LRU.

Even though the design of active/inactive LRUs is to reference active and inactive pages, the kernel intentionally spends minimal effort to maintain these lists (for performance reasons), so the information about the pages in these lists is outdated most of the time for most of the pages.

What can we actually tell about the pages in the LRUs?

• The page at the head of the active list was accessed just recently. It was accessed once or twice, or a million times.
• The page at the tail of the active list was accessed some time ago when it was added to the head of the list. It could have been accessed a million times or zero times while being in the active list.
• The page at the head of the inactive list was just allocated or it was just pushed out of the active list where it spent unknown time and has been accessed zero or million times.
• The page in the middle of the inactive list was allocated some time ago, and we don’t know if it has been accessed yet. Until it is scanned, it can be accessed zero times or a million times.
• The pages at the tail of the inactive list are only pages that have accurate information, but only when they are scanned. We also still don’t know if they were accessed more than once. We also don’t know if they are going to be accessed again, but at least this is expected.

So the more appropriate name for the ‘Inactive’ LRU would be ‘candidates for the reclaim’, and for ‘active’ - ‘not considered for the reclaim yet’. The numbers derived from the lengths of these LRUs and reported as Active/Inactive memory in /proc/meminfo have little relation to the actual working set size.

The initial question is simple: “How to add rDNS for AWS EIP with Terraform”. All models answered that Terraform doesn’t support it natively and offered a workaround with “local-exec” and call to “aws ec2 modify-address-attribute”. Claude gave the correct parameter “—domain-name”, both 4o and o3-mini-high hallucinated parameter names “—reverse-dns-name” and “—reverse-dns”.

Given a correction that Terraform does support this natively, the models started hallucinating by inventing or repurposing “aws_eip” resource attributes. 4o suggested using the “domain” attribute, which is not related to DNS. o3-mini-high invented “reverse_dns” block for “aws_eip” resource. Claude suggested assigning the “reverse_dns” attribute, which doesn’t exist. Interestingly, with web search enabled, 4o was able to find the correct “aws_eip_domain_name” resource. Both Claude and o3 went back to suggesting using “local-exec” and inventing random resource names like “aws_ec2_address”, “aws_ec2_address_attribute”, “aws_eip_reverse_dns”.

I have noticed that o3 is much more stubborn, and if it goes a wrong way, it is almost impossible to correct - a few weeks ago, it tried to correct me that MySQL 9 doesn’t exist. Not sure if the new Claude works the same way, but at least it is much more cheerful. Sill gave that subscription option a pass as there is no improvement, and these tasks are still too challenging for AIs.

While all the services are internal and not customer-facing, there are some critical for the team and infrastructure. To avoid much service disruption and downtime, I decided to move services one at a time so I could deploy and test the service on the new infrastructure, shut down the old one, re-sync the data and switch the DNS (almost forgot to lower DNS TTL before the move) with the option to roll-back to the old service if anything goes wrong. Beforehand, I created a table in Notion with all service resource requirements, SLAs, inter-dependencies, priorities and nice color labels. Had this table open in front of me for almost two months.

The initial infrastructure preparation was a clean greenfield - made it all modern, fresh and shiny. While I had time, I started with the most complex and important services, refreshed them, threw away a lot of organic-growth stuff, and re-wrote some Chef cookbooks to Ansible. When the deadline started approaching, it became a kind of a slog - move the service, search all the configuration spread for IPs to update², get the service working, repeat, preferably early in the morning. The team also dropped on me two other projects(including a major OS upgrade) with the same deadline to keep the pressure. Had a lot of overtime and even broke my anti-burnout rule of not working full-time on weekends once. Ironically, I managed to follow Parkinson’s law and finished by retiring the old infrastructure exactly at the deadline, the last day of the month.

But there are still a lot of small broken parts to fix and old dust and junk to clear. Still sorting out and prioritising my ToDo list as it added more than a hundred items during the migration. Two major things to re-work - remove as much as possible from the Chef attributes and re-organize packet filter rules management.

Also, had some interesting and surprising moments:

The first one was when the hosting provider moved an IP from one server to another, and I started seeing incoming traffic for the IP on both the old and the new servers simultaneously. I was monitoring the traffic with tcpdump on both servers to catch the moment when the IP switch was about to happen. At first, I suspected their network engineer somehow managed to mirror the traffic and even sent them a message, as I was clearly getting ‘host unreachable’ from the old server public IP, but I didn’t stop investigating. The IP wasn’t responding even locally, though I double-checked that it was assigned to a VM and that a route was configured properly. When I got a ‘host unreachable’ message from an internal IP from the old server, it all became clear. There was an IPSec tunnel between the old and the new servers, and the IP was still routed to the old server through the tunnel. The confusing part was that the packets coming in through the IPSec tunnel on the old server were seen as coming through normal ethernet. The irony is that I had already prepared IPSec reconfiguration, and it was the next step in my checklist.

The second was a chicken-and-egg problem. To move the Chef server, I needed to reconfigure load balancers, but the load balancers’ configuration was managed with Chef. Ended up changing the configuration manually.

The third was a little adventure with a new secondary server at the old hosting provider. They delayed its delivery for a month. When I got access to its console, it wasn’t even re-imaged with a fresh Debian; it was a used node from the provider’s cloud solution. Their netboot image was running Alpine version from 2019, and their netboot installer didn’t have anything newer than Debian Buster. Their netboot didn’t support UEFI boot with GPT, and I found it the hard way. It also explained why they failed to re-image the server. Re-partitioned the disks with MBR, installed Debian 10, and followed with two immediate upgrades to Bullseye and Bookworm. Almost entire day wasted doing someone else work. And then I had to wait 24h for them to re-route an IP to this server. Oh, and the disks had 50K power-on hours on a supposedly new server. My expectations were adjusted lower with every interaction with these guys. But in the end, most services were moved from this provider and we got twice the capacity for a lower price.

Notes:

But “Premature optimisation is the root of all evil”, so first things first: enable detailed monitoring and collect query statistics for a day. The next morning, there is some data: the code is hitting mostly a single table with a simple query but with WHERE on a column without an index. Add an index, maybe it will help improve the performance before I have to dive into the code.

The next morning, I check the DB graphs, and there is no load at all. Did the job run? Did someone disable it? Did I change something, causing the job to crash?

Checking the application logs. The job did run. And completed successfully. In 10 minutes. One single index reduced run time from more than 3 hours to 10 minutes. The classic “one hit with the hammer but you need to know where to hit”.

On the other hand, I’m trying to imagine what someone who “doesn’t know where to hit” and has no visibility into the database and application performance would do to solve this issue. Just crank up the instance size? I suspect they would end up paying ten times more.

The problem which Vagrant solved was that both applications are too complex to setup manually. Even when working with WordPress developers didn’t work locally but instead used to edit files directly on live server and even when we imported all code into git they started using integration server for day-to-day development and their workflow looked terrible — change a line, commit, push, wait for deploy script to run, check integration server for results, repeat. Moreover, as a result of this workflow git history looked ugly — myriad of one-line commits with no commit messages, which are painful to merge. For Rails app we needed some CSS/HTML tweaks and there is just no way average front-end developer can setup Rails development environment on Windows.

At first I thought about distributing binary Vagrant box but I still needed to distribute application source code as git repository and Vagrantfile to configure sharing and I was too lazy to setup password protected directory on web server to download binary box and hand out all credentials to individual developers (Vagrant Cloud for organizations hadn’t been available yet). So I decided to make one single git repo with Vagrant configuration and cookbooks and source code repo included as submodule.

It was a while since I did chef cookbook development so I googled a lot at first trying to find what is the best current approach and which tools to use. Cookbook development completely changed over last year or two — there are now test-kitchen, berkshelf, serverspec etc and all these tools are changing very fast — almost any tutorial older than a few months is obsolete.

So far I have found following blog posts as the most current:

• Getting Started Writing Chef Cookbooks the Berkshelf Way (ignore parts 1 and 2)

• Automating Cookbook Testing with Test-Kitchen, Berkshelf, Vagrant and Guard

In my setup I have followed the second one and cross-checked with the first article. I chose to include in my toolbox test-kitchen, berkshelf, serverspec, chefspec, foodcritic, rubocop and wrap everything with guard (but later disabled test-kitchen run from guard as it was failing). In the beginning I started preparing custom Vagrant base box with veewee but dropped it as I didn’t really need anything custom and standard chef/debian box from vagrantcloud.com worked well.

My main repo has very simple structure — Gemfile with berkshelf, Berksfile with all necessary cookbooks, Vagrantfile and INSTALL file with step-by-step instructions for developers. In www sub-directory I have site source code as git submodule and in cookbooks sub-directory all depended cookbooks vendored using berks vendor cookbooks. At first I used to include my own cookbooks as git submodule too into site-cookbooks but as berks vendor retrieves them anyway I dropped this. Also I decided not to use vagrant-berkshelf plugin to maintain cookbooks as it is deprecated.

For each application I created individual cookbook and one cookbook for common configuration. Each cookbook has own git repo and follows standard layout created by berks cookbook. I have also decided to rely on community cookbooks for all dependencies like MySQL, PHP etc, even though I didn’t do much customization but this decision caused a bit of pain — I had to fork cookbooks for MySQL and monit to support Debian squeeze and had to use alternative cookbook for PHP as phpmyadmin cookbook depends on it. Each cookbook has multiple recipes: for Vagrant setup, for integration server setup and for live server setup as there is some differences between them — SSL support and while integration server runs php-fpm live server still uses mod_php.

At first I followed quite strict TDD/BDD loop — create serverspec tests, then chefspec and then write recipe but after a while dropped chefspec tests as I find writing expect(chef_run).to include_recipe('apache2') and then include_recipe 'apache2' a bit boring. Also running kitchen converge && kitchen verify is quite slow even with a lot of RAM and on SSD disk. I tried to speed up things by switching to LXC but kitchen-lxc seems to be broken and unsupported and using vagrant-lxc with test-kitchen isn’t documented very well and requires building LXC base boxes manually using outdated instructions — some links to configuration templates are 404 and after you build base boxes recent Vagrant complains about outdated box format. My attempts to use more up to date scripts to build base box failed as these scripts just segfaulted on me and I didn’t have time to fix them as manually built base boxes already working. Another issue is that my Linux Mint box had sudo configuration setting which caused vagrant-lxc to fail when used with test-kitchen and a couple weeks passed before I found time to find a solution so all cookbooks were developed slowly using VirtualBox.

But overall development went quite smoothly except for few PHP/WordPress surprises in the end — e.g. PHP with short_open_tag switched fails with syntax error pointing to the end of huge 5K LOC .php file without any hint of real error cause or WordPress shows just blank front page without any error messages in error logs if some plugin fails or missing. But real adventure was still ahead. When all cookbooks were ready and fully tested locally on Linux and Mac OS X it was time to deploy to Windows boxes where everything failed just at the very beginning — Vagrant was launching VirtualBox VMs but unable to ssh into them. Few days of remote debugging using email and I had found that even vagrant init hashicorp/precise failed to work on Windows so I got idea and tried to switch to 32-bit OS image which worked. Later I got RDP access to Window 8 box and launched VirtualBox directly which complained that VT-x is disabled (it needs to be enabled in BIOS and this feature is unavailable on Celeron processors) and it can’t launch 64-bit image. Once I switched images to 32-bit all Windows users were able to use them without much problems, except occasional cases when developers didn’t read documentation and forgot to use git clone --recursive and similar issues.

Another quite problematic issue with Windows was that it is impossible to create symbolic links on shared file system with default settings and Rails app were deployed capistrano style and relied on symbolic links heavily. I had to revamp whole recipe for Rails app and remove all symbolic links to get it working on Vagrant under Windows. Another Rails specific issue is that rvm cookbooks needs special recipe rvm::vagrant to be included before any other recipe if it is run in Vagrant VM.

Before making any action I started by measuring actual performance and gathered metrics using New Relic and Chrome Developer Tools Audit Tab. New Relic showed a few of critical insights into performance troubles.

First one was two widgets in the footer of every page each making requests to external API services taking on average ~600ms one and ~1500 ms another. As the second service was our own custom service I quickly optimized it by adding counter cache field to table instead of making select on depended records on each request and request time went down to around ~150 ms. But these requests were still made on every page load so I patched both widgets to cache responses from external API in memcached for 5 minutes. Average page generation time went down to around ~800 ms.

Another thing to optimize was W3 Total Cache. First I turned off its minify option as it was sometimes taking up 2-3 seconds according to New Relic. Next I switched cache storage from APC to memcached as APC was constantly reset every minute by some rogue code somewhere on this server: grep -r apc_clear_cache showed about hundred of matches. This issue also affected PHP opcode caching so I decided to switch to Zend OPcache for opcode caching. For page cache I choose to switch from memcached storage to extended disk so if page was cached then Apache would serve it directly without hitting any PHP code. As database queries were taking insignificant percentage of all page generation time I switched DB cache completely off to avoid PHP overhead and instead cranked up MySQL own query cache memory limits.

With all these optimizations page generation time stabilized around ~600ms against ~2500 ms week ago and I called it a day - I don’t think I can squeeze more performance out of WordPress without going through analyzing performance impact of each plugin.

The next step in the site optimization was tuning of mod_pagespeed settings. At first it looked like it was not working at all. After checking mod_pagespeed logs I have found that it doesn’t work with SSL by default and we have https-only site. Another obstacle with which I have spent good half hour is that W3 Total Cache page cache interferes with mod_pagespeed, looks like later ignores static HTML files. After that I have enabled most of mod_pagespeed CoreFilters, focusing on CSS and JS files optimization, as we had about 30-40 assets of each kind, and was able to reduce number of external CSS/JS down to 9-10 per page. I have also tried to optimize page loading by using filter defer_javascript which moves JS files down to page footer but had to turn it off later as it broke some JS navigation menus. Overall page load speed went down from around 10 seconds to 5 seconds on average.

I set it up pretty quickly but the first problem was that I wanted to keep old content but I didn’t have source code for it anymore. Converting by hand seemed tedious so I thought about hiring someone on oDesk but then, after a few Google searches I found a tool to convert HTML back to markdown – reverse_markdown ruby gem. At first attempt it did no conversion but after stripping all HTML code around actual post content (the most important is to remove article tags around) it produced nice markdown which I put back into Octopress.

After initial import I did some cleanup – removed unnecessary /blog prefix from post permalinks, fixed links in old content pointing to my old-old MovableType blog and imported static files into Octopress. To check all links I installed link-checker ruby gem – it works pretty fine but seems to be having problems with some https:// links.

Once all content was in good shape I tweaked CSS colors back to my old palette, added Stack Exchange badge, enabled Disqus comments, updated Google Analytics JavaScript to latest universal code.

After comparing generated HTML with old blog using diff I have found a bug in Octopress: canonical link for categories pages is broken by default – has missing /, see Octopress issue #949 for fix. Once I was satisfied with content I deployed it to server using rsync.

There is currently Octopress version 3.0 in development and it is close to final release but it seems to be quite different from version 2.0 in its concept and as its author says:

For those currently using Octopress, it will be a while before the new releases can compete with the features of the old version.

At first I ignored this issue – I had just added two new factories, which coincidentally used class parameter in their definition to specify generated class explicitly, and I needed these factories to test code I was working on and I though that I’ll fix or just remove these factories later.

But as it usually happens it wasn’t simple as I thought. Suddenly I discovered that tests started failing with the same symptoms on common develop branch and not on only topic branch. And I broke tests already in two other places, so clean up was really needed.

First two days I spent trying to find out what happening in factory_girl internals using old-school print logging and later pry-debugger but without much success except that I was able to locate single spec file in spec/workers/ which caused failure of all consecutive factory calls. Then I started looking at git history trying to find commit which introduced this issue. Luckily, in spite of heavy rebasing and few backported commits, my master branch didn’t have this issue and I was able to pinpoint this to single commit. At first glance this commit looked almost innocent – it just extracted code from model and moved it to app/workers/. But there were two tests added to failing spec file and they were the tests which caused cascading failure of all remaining tests in suite. After reviewing the code under test I had found that real culprit was memory leak debugging code I quickly slapped in without running tests:

It seems that FactoryGirl::DefinitionsProxy undefines all methods including class and method_missing in this class adds any calls as traits to factory so walking through ObjectSpace and calling class on every object wreaks havoc on factories.

Следующим шагом вносим в исходники все необходимые изменения для поддержки русского языка:

• patch-rudate41.gz — добавляет русский формат дат
• patch-rudirify41.gz — добавляет русские символы в таблицы преобразования заголовков статей в имена файлов (в отличии от патча Алексея Тутубалина изменяется также и Javascript код, а преобразование русских символов в латиницу сделано в соответствии с ГОСТ 7.79-2000)
• patch-nofollow.gz — добавляет поддержку тэга необходимого для российских поисковиков (автор Алексей Тутубалин)
• patch-monday-mt41.gz — делает первым днем недели в календаре понедельник (автор Алексей Тутубалин)

После внесения изменений в код и шаблоны осталось добавить собственно файл перевода lib/MT/L10N/ru.pm ( перевод проекта movable-type.ru, либо мой, менее полный перевод), файл стилей mt-static/styles_ru.css и два HTML-файла — index.html.ru и readme.html.ru (они есть в репозитарии русской версии Movable Type на Google Code). Файл перевода необходимо обработать в соответствии с инструкциями в предыдущей статье. В завершении можно изменить некоторые настройки по умолчанию (часовой пояс, кодировки, ссылки на новости, портал и техподдержку) для собираемого пакета, отредактировав файл build/mt-dists/ru.mk.

Сделав все необходимые изменения выполняем сборку пакета MTOS:

env LANG=C ./build/exportmt.pl --local --pack=MTOS --lang=ru --prod

В результате выполнения данной команды получим два архива с русской версией Movable Type — MTOS-4.1-ru.tar.gz и MTOS-4.1-ru.zip.

Для того, чтобы синхронизировать файл перевода с текущим релизом (а также будущими релизами) и добавить в него отсутствующие строки, нужен полный комплект исходников Movable Type из svn-репозитария — в них пристутствует набор скриптов для манипуляций с файлами переводов (для работы скриптов нужен perl и набор unix-утилит - shell, awk, find и т.д.). Сначала делаем чекаут исходников из репозитария:

svn co https://code.sixapart.com/svn/movabletype/latest mtos-latest

После выполнения данной команды в каталоге mtos-latest будет три подкаталога: dev, stable и release. Нам нужен подкаталог release, в котором находится последний релиз Movable Type с которым мы будем работать. Первым делом кладем наш файл перевода ru.pm в подкаталог lib/MT/L10N/. Следующим шагом запускаем собственно генерацию обновленного файла перевода из каталога release:

sh build/l10n/make-l10n ru

В результате работы данного скрипта в каталоге /tmp будет сгенерировано четыре файла:

• ru-base.pm — файл со всеми строками трансляции (в том числе дублирующиеся строки)
• ru-nodupe.pm — файл с удаленными дубликатами строк
• ru-old.pm —строки трансляции которые более не используются
• ru.pm —новый файл перевода

В новом файле перевода /tmp/ru.pm будут добавлены отсутствующие строки перевода (отмечены комментарием «Translate - New») и будут отчемены строки с регистром перевода отличающимся от оригинала — например, переводы следующего вида:

'video' => 'Видео'

Этот файл уже готов к использованию, но для полноценной поддержки русского языка я сделал дополнительный фильтр ru-filter.pl, через который надо пропустить файл перевода:

cat /tmp/ru.pm | perl ru-filter.pl > lib/MT/L10N/ru.pm

Данный фильтр делает следующее:

• отмечает комментарием «# Translate - No translation» непереведенные строки, в которых оригинал и перевод совпадают
• отмечает комментарием «# Translate - No russian char» строки перевода в которых не встречаются русские буквы
• отключает строки с пустым переводом
• добавляет perl’овый код для правильной поддержки множественного числа

Поддержка множественного числа сделана через переопределение функцииquant из Locale::Maketext. Теперь второй и третий параметры этой функции принимают формы слова для значений числительного от 2 до 4 и от 5 до 0 соответственно. Таким образом можно писать «[quant, _1,минута,минуты,минут]» и соответствующий текст будет генерироваться как «1 минута, 2 минуты, 5 минут» вместо «1 минута, 2 минут., 3 минут.»

• приложен патч patch-monday-mt41.gz для того чтобы в календаре неделя начиналась с понедельника
• приложен патч patch-dirify.gz для конвертации русских символов в заголовках статей в латиницу при генерации имени файла
• приложен патч patch-nofollow.gz для поддержки тэга для Яндекса

Патч patch-rudate.gz не прикладывал, но посмотрев в него, сделал свой. Вместо того, чтобы использовать итальянский формат, добавил русский в список форматов, изменив lib/MT/Util.pm и tmpl/cms/cfg_entry.tmpl. Изменен также и формат отображения — вместо 24.03.2008 будет Март, 24 2008. Патч patch-rudate41.gz прилагается.

Далее последовал наиболее нудный этап — перевод шаблонов. Разбор со скриптом Алексея Тутубалина для экспорта/импорта шаблонов я оставил на потом (этот скрипт был сделан до выхода версии 4.1 и неизвестно как он с ней работает, к тому же он требует установки плугина TemplateInstaller), поэтому шаблоны перевел вручную через админку Movable Type. Хотя шаблонов около полусотни, но сами они немногословны и переводить можно не все, а только те, которые реально используются. Поэтому первые результаты появился сравнительно быстро, но в процессе перевода увлекся файн-тюнингом оформления и внес слишком много изменений в шаблоны, так что собравшись создавать блог для своей SO, обнаружил что мне надо либо вычищать свое оформление, либо возвращаться к стандартным шаблонам и переводить их опять с нуля.

Немного поразмышляв, решил выбрать другой подход. В Movable Type есть в наличии полная инфраструктура для локализации базирующаяся на perl’овом Locale::Maketext и документация рекомендуют именно этот подход. Проект movable-type.ru после выхода версии 4.0 в августе 2007 вроде бы огранизовал коллективную попытку сделать русский перевод и месяц назад даже заявлена 90% готовность, но реальных результатов пока не наблюдается. Конечно объем текста в более чем 4 тысячи строк на первый взгляд выглядит ужасающе, но ничто не мешает переводить текст по частям, например, для начала только те строки, которые используются в самих блогах, а не в админке.

За пару вечеров у меня был готов перевод следующих компонентов используемых в опубликованном блоге:

• шаблоны default_templates/*.mtml которые используются при генерации шаблонов нового блога
• шаблоны plugins/WidgetManager/default_widgets/*.mtml для widget’ов
• php-скрипты динамической публикации

Все это занимает где-то чуть более 600 строк ( ru-published-only.pm). Данный файл необходимо переименовать в ru.pm и положить в lib/MT/L10N/, при этом русский язык автоматически станет доступен для выбора в профиле пользователя. Стоит иметь в виду, что перевод фраз в шаблонах на русский производится в момент генерации шаблонов и только если у пользователя выбран русский язык в профиле. Если блог уже создан, то можно перегенерировать шаблоны заново, зайдя в своем блоге через меню «Design» → «Manage Templates» → «Refresh Blog Templates». При перегенерации большинство новых руссифицированных шаблонов будут иметь русские имена и у меня лично слетели Archive Maps и наборы widget’ов. Нужно будет их пересоздать, предварительно удалив все старые шаблоны и widget’ы.

Все остальное пока находится в процессе перевода, хотя наиболее часто используемые части админки уже переведены практически полностью (можно скачать Файл перевода). В качестве основы для этого файла взят es.pm, поэтому непереведенные строки будут по испански. Также добавлен скрипт mt_ru.js который содержит локализованные строки для JavaScript и локализованную таблицу для динамической dirtif’икации заголовков в имена файлов. Файл стилей styles_ru.css содержит одну строчку — модификация ширины колонок в виджете календаря.

По мере дальнейшего перевода я планирую выкладывать обновленную версию файла ru.pm, следите за обновлениями.

Первой мыслью было попробовать запустить его под mod_perl, который у меня уже есть, но как оказалось Movable Type поддерживает только mod_perl 1.x, а я уже третий год как переехал на Apache 2.x. Поэтому пришлось обратиться к альтернативном варианту — FastCGI, отличие которого от классического CGI именно в том, что скрипт обслуживает не один, а множество HTTP-запросов, чем минимизируются временные затраты на запуск и инициализацию скрипта.

Сначала обеспечиваем поддержку FastCGI в Apache — это делается с помощь модуля mod_fastcgi (или альтернативная реализация mod_fcgid). Ставим модуль и подключаем его в httpd.conf.настройки для mod_fastcgi:

LoadModule fastcgi_module libexec/apache22/mod_fastcgi.so
<IfModule mod_fastcgi.c>
  AddHandler fastcgi-script .fcgi
  FastCgiIpcDir /var/run/fcgidsock/
</IfModule>

настройки для mod_fcgid:

LoadModule fcgid_module libexec/apache22/mod_fcgid.so
<IfModule mod_fcgid.c>
  AddHandler fcgid-script .fcgi 
  SocketPath /var/run/fcgidsock/
</IfModule>

Сам Movable Type начиная с версии 4 поддерживает работу через FastCGI из коробки, поэтому для его настойки достаточно указать для каталога где находятся скрипты mt*.cgi следующую директиву:

AddHandler fcgid-script .cgi

или

AddHandler fastcgi-script .cgi

После включения данной опции тестирование с помощью ApacheBench показало на моей домашней машинке ускорение более чем на порядок — с 0.75 запросов в секунду до 15. Но особого ускорения при работе с админкой не прозошло. Налицо классический случай «premature optimization», которая «root of all evil» © C.A.R. Hoare.

Поэтому пришлось все-таки более внимательно посмотреть в YSlow и последовательно применить предлагаемые рекомендации. Основная проблема оказалась в большом количестве объектов на странице и как следствие, большом количестве HTTP запросов (около 50 в данном случае), причем большинство запросов были If-Modified-Since на файлы из mt-static, в осовном изображения. Лечиться это добавлением заголовков Expires для каталога mt-static с помощью mod_expires:

ExpiresActive On
ExpiresDefault "access plus 1 month"

С включением данной опции броузер перестал обращаться к каждой картинке при каждой загрузке страницы, что сократило количество запросов с полусотни до ровно одного — обращение собственно к mt.cgi. Общий объем передаваемых данных также сократился с 100Kb (уже с учетом включеной компрессии для text/*) до 10Kb. Общее же время загрузки страницы сократилось с 15 секунд до 5-6 секунд по таймеру YSlow и до 3-4 секунд по секундомеру с отключеным FireBug‘ом и YSlow.

Достигнутый результат меня вполне устроит и тему оптимизации можно отложить. Дальше нужно будет заниматься уже переводом на русский и переносом на боевой сервер в тесный jail. А пока в процессе первых дней эксплуатации выяснилось пара нюансов. Во-первых, не все скрипты MT работают в режиме FastCGI, поэтому лучше использовать следующие настройки подсмотренные в документации Movable Type:

<FilesMatch "^mt(?:-(?:comments|search|tb|view))?\.cgi$">
     AddHandler fcgid-script .cgi
</FilesMatch>

Во-вторых, процесс mt.cgi со временем может разрастаться до достаточно неприличиных размеров (более 100 Mb) и поэтому лучше настроить mod_fastcgi так, чтобы он переодические его перезапускал:

FastCgiConfig -maxProcesses 10 -killInterval 3600

и для mod_fcgid

ProcessLifeTime 3600
   MaxProcessCount 10
   MaxRequestsPerProcess 500

И в третьих, при работе через FastCGI возникают неудобства с изменением кода MT на ходу — необходимо постоянно перезапускать mt.cgi чтобы он перечитал измененные модули.

Продолжение следует …