We’ve been upgrading our Lambda functions to use the Ruby 2.7 runtime on AWS Lambda and some of the functions in our projects ran into unexpected issues. In this article we’re going to take a brief step back to give context to the issue we experienced and walk you through our investigation and resolution.
Ruby Gems with Native Extensions in AWS Lambda
Previously we had to jump through some hoops to get gems with native extensions working in Lambda as it requires the included binaries to be built in the same environment. Two typical use cases in our projects that required us to jump through these hoops are the excellent Pg and Nokogiri gems.
If you’re reading this article we’re going to assume you are familiar with the native extensions issue in AWS Lambda with Ruby. If you are not familiar; the short version is that you need to emulate the AWS Lambda environment with LambCI and Docker on a development machine or build server. This allows you to compile your native dependencies in a docker container that will be compatible when deployed to a Ruby Lambda function.
The LibLDAP Pg Gem Error in our Lambda Ruby Functions
After upgrading our Serverless project to Ruby 2.7 and switching to the ruby2.7 runtime we were greeted with this unexpected error message when executing a function that was leveraging the postgres gem:
libldap_r-2.4.so.2: cannot open shared object file: No such file or directory - /var/task/vendor/bundle/ruby/2.7.0/gems/pg-1.2.3/lib/pg_ext.so
This left us scratching our heads for a while as we were bundling libpq.so.5 with our Lambda Ruby function, which is all we previously needed using the ruby2.5 runtime. Why was our function complaining about a library for LDAP anyway?
As it turns out this is a shared library that is indeed needed by the native pg dependencies. We confirmed this by using the ldd utility to list dynamic dependencies within our LambCI build container:
bash-4.2# ldd pg_ext.so
linux-vdso.so.1 (0x00007ffe34ff1000)
libruby.so.2.7 => /var/lang/lib/libruby.so.2.7 (0x00007f6805888000)
libpq.so.5 => /usr/lib64/libpq.so.5 (0x00007f680565a000)
libm.so.6 => /usr/lib64/libm.so.6 (0x00007f680531a000)
libc.so.6 => /usr/lib64/libc.so.6 (0x00007f6804f6f000)
libz.so.1 => /usr/lib64/libz.so.1 (0x00007f6804d5a000)
libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007f6804b3c000)
librt.so.1 => /usr/lib64/librt.so.1 (0x00007f6804934000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f6804730000)
libcrypt.so.1 => /usr/lib64/libcrypt.so.1 (0x00007f68044f9000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6806068000)
libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f680428a000)
libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007f6803e35000)
libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007f6803b51000)
libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x00007f680394d000)
libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007f6803701000)
libldap_r-2.4.so.2 => /usr/lib64/libldap_r-2.4.so.2 (0x00007f68034a3000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007f6803272000)
libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007f6803063000)
libkeyutils.so.1 => /usr/lib64/libkeyutils.so.1 (0x00007f6802e5f000)
libresolv.so.2 => /usr/lib64/libresolv.so.2 (0x00007f6802c49000)
liblber-2.4.so.2 => /usr/lib64/liblber-2.4.so.2 (0x00007f6802a3a000)
libsasl2.so.3 => /usr/lib64/libsasl2.so.3 (0x00007f680281d000)
libssl3.so => /usr/lib64/libssl3.so (0x00007f68025c7000)
libsmime3.so => /usr/lib64/libsmime3.so (0x00007f68023a1000)
libnss3.so => /usr/lib64/libnss3.so (0x00007f680207f000)
libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007f6801e50000)
libplds4.so => /usr/lib64/libplds4.so (0x00007f6801c4c000)
libplc4.so => /usr/lib64/libplc4.so (0x00007f6801a47000)
libnspr4.so => /usr/lib64/libnspr4.so (0x00007f680180b000)
libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x00007f68015e4000)
libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f6801380000)
But why was the library missing in our Lambda, this should be included in /usr/lib64?
We started to dig into the Lambda runtimes and discovered a much larger change we were not initially aware of moving from Ruby 2.5 to Ruby 2.7 as seen below:
AWS Lambda Ruby Runtimes
| Name | Identifier | AWS SDK for Ruby | Operating System |
|---|---|---|---|
| Ruby 2.7 | ruby2.7 | 3.0.1 | Amazon Linux 2 |
| Ruby 2.5 | ruby2.5 | 3.0.1 | Amazon Linux |
The ruby2.7 runtime is actually running on Amazon Linux 2! With the move to Amazon Linux 2 there are significantly less bundled libraries available by default as can be seen in these two gists:
Cross referencing our dynamic dependency results from above, we determined the following shared libs needed to be bundled with libpq.so.5 in our function:
- libldap_r-2.4.so.2
- liblber-2.4.so.2
- libsasl2.so.3
- libssl3.so
- libnss3.so
- libnssutil3.so
After updating our Makefile to copy these shared libs to our task lib directory, our world is no longer on fire and we are back to working as expected with the pg gem in our AWS Lambda Ruby functions.
Getting the Ruby Postgres Pg Gem working again in AWS Lambda
We have included our Dockerfile and Makefile below that builds our LambCI docker container and copies over our required Postgres dependencies before deploying. We are assuming for brevity that you have an established process in place to deploy your Ruby Lambda function. We leverage Serverless in our all AWS Lambda projects, which handles the heavy lifting of deployments for us.
Dockerfile
FROM lambci/lambda:build-ruby2.7
RUN yum install -y postgresql postgresql-devel
RUN gem update bundler
CMD "/bin/bash"
Makefile
image:
docker build -t northsail/sls-ruby2.7-builder:latest .
install:
make image
docker run --rm -it -v $$PWD:/var/task -w /var/task northsail/sls-ruby2.7-builder:latest make _install
_install:
bundle config --local silence_root_warning true
bundle install --path vendor/bundle --clean
mkdir -p /var/task/lib
cp -a /usr/lib64/libpq.so.5.5 /var/task/lib/libpq.so.5
cp -a /usr/lib64/libldap_r-2.4.so.2.10.7 /var/task/lib/libldap_r-2.4.so.2
cp -a /usr/lib64/liblber-2.4.so.2.10.7 /var/task/lib/liblber-2.4.so.2
cp -a /usr/lib64/libsasl2.so.3.0.0 /var/task/lib/libsasl2.so.3
cp -a /usr/lib64/libssl3.so /var/task/lib/
cp -a /usr/lib64/libsmime3.so /var/task/lib/
cp -a /usr/lib64/libnss3.so /var/task/lib/
cp -a /usr/lib64/libnssutil3.so /var/task/lib/
Could you benefit from a dependable Ruby development partner?
Learn about our ruby on rails development services.