Articles

AWS Lambda Ruby 2.7 Pg Gem LibLDAP Error

We’ve been upgrading our Lambda functions to use the Ruby 2.7 runtime on AWS Lambda and some of the functions in our projects ran into unexpected issues. In this article we’re going to take a brief step back to give context to the issue we experienced and walk you through our investigation and resolution.

Ruby Gems with Native Extensions in AWS Lambda

Previously we had to jump through some hoops to get gems with native extensions working in Lambda as it requires the included binaries to be built in the same environment. Two typical use cases in our projects that required us to jump through these hoops are the excellent Pg and Nokogiri gems.

If you’re reading this article we’re going to assume you are familiar with the native extensions issue in AWS Lambda with Ruby. If you are not familiar; the short version is that you need to emulate the AWS Lambda environment with LambCI and Docker on a development machine or build server. This allows you to compile your native dependencies in a docker container that will be compatible when deployed to a Ruby Lambda function.

The LibLDAP Pg Gem Error in our Lambda Ruby Functions

After upgrading our Serverless project to Ruby 2.7 and switching to the ruby2.7 runtime we were greeted with this unexpected error message when executing a function that was leveraging the postgres gem:

libldap_r-2.4.so.2: cannot open shared object file: No such file or directory - /var/task/vendor/bundle/ruby/2.7.0/gems/pg-1.2.3/lib/pg_ext.so

This left us scratching our heads for a while as we were bundling libpq.so.5 with our Lambda Ruby function, which is all we previously needed using the ruby2.5 runtime. Why was our function complaining about a library for LDAP anyway?

As it turns out this is a shared library that is indeed needed by the native pg dependencies. We confirmed this by using the ldd utility to list dynamic dependencies within our LambCI build container:

bash-4.2# ldd pg_ext.so

linux-vdso.so.1 (0x00007ffe34ff1000)
libruby.so.2.7 => /var/lang/lib/libruby.so.2.7 (0x00007f6805888000)
libpq.so.5 => /usr/lib64/libpq.so.5 (0x00007f680565a000)
libm.so.6 => /usr/lib64/libm.so.6 (0x00007f680531a000)
libc.so.6 => /usr/lib64/libc.so.6 (0x00007f6804f6f000)
libz.so.1 => /usr/lib64/libz.so.1 (0x00007f6804d5a000)
libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007f6804b3c000)
librt.so.1 => /usr/lib64/librt.so.1 (0x00007f6804934000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f6804730000)
libcrypt.so.1 => /usr/lib64/libcrypt.so.1 (0x00007f68044f9000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6806068000)
libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f680428a000)
libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007f6803e35000)
libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007f6803b51000)
libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x00007f680394d000)
libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007f6803701000)
libldap_r-2.4.so.2 => /usr/lib64/libldap_r-2.4.so.2 (0x00007f68034a3000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007f6803272000)
libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007f6803063000)
libkeyutils.so.1 => /usr/lib64/libkeyutils.so.1 (0x00007f6802e5f000)
libresolv.so.2 => /usr/lib64/libresolv.so.2 (0x00007f6802c49000)
liblber-2.4.so.2 => /usr/lib64/liblber-2.4.so.2 (0x00007f6802a3a000)
libsasl2.so.3 => /usr/lib64/libsasl2.so.3 (0x00007f680281d000)
libssl3.so => /usr/lib64/libssl3.so (0x00007f68025c7000)
libsmime3.so => /usr/lib64/libsmime3.so (0x00007f68023a1000)
libnss3.so => /usr/lib64/libnss3.so (0x00007f680207f000)
libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007f6801e50000)
libplds4.so => /usr/lib64/libplds4.so (0x00007f6801c4c000)
libplc4.so => /usr/lib64/libplc4.so (0x00007f6801a47000)
libnspr4.so => /usr/lib64/libnspr4.so (0x00007f680180b000)
libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x00007f68015e4000)
libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f6801380000)

But why was the library missing in our Lambda, this should be included in /usr/lib64?

We started to dig into the Lambda runtimes and discovered a much larger change we were not initially aware of moving from Ruby 2.5 to Ruby 2.7 as seen below:

AWS Lambda Ruby Runtimes

Name Identifier AWS SDK for Ruby Operating System
Ruby 2.7 ruby2.7 3.0.1 Amazon Linux 2
Ruby 2.5 ruby2.5 3.0.1 Amazon Linux

The ruby2.7 runtime is actually running on Amazon Linux 2! With the move to Amazon Linux 2 there are significantly less bundled libraries available by default as can be seen in these two gists:

Cross referencing our dynamic dependency results from above, we determined the following shared libs needed to be bundled with libpq.so.5 in our function:

After updating our Makefile to copy these shared libs to our task lib directory, our world is no longer on fire and we are back to working as expected with the pg gem in our AWS Lambda Ruby functions.

Getting the Ruby Postgres Pg Gem working again in AWS Lambda

We have included our Dockerfile and Makefile below that builds our LambCI docker container and copies over our required Postgres dependencies before deploying. We are assuming for brevity that you have an established process in place to deploy your Ruby Lambda function. We leverage Serverless in our all AWS Lambda projects, which handles the heavy lifting of deployments for us.

Dockerfile

FROM lambci/lambda:build-ruby2.7

RUN yum install -y postgresql postgresql-devel
RUN gem update bundler

CMD "/bin/bash"

Makefile

image:
        docker build -t northsail/sls-ruby2.7-builder:latest .

install:
        make image
        docker run --rm -it -v $$PWD:/var/task -w /var/task northsail/sls-ruby2.7-builder:latest make _install

_install:
        bundle config --local silence_root_warning true
        bundle install --path vendor/bundle --clean
        mkdir -p /var/task/lib
        cp -a /usr/lib64/libpq.so.5.5 /var/task/lib/libpq.so.5
        cp -a /usr/lib64/libldap_r-2.4.so.2.10.7 /var/task/lib/libldap_r-2.4.so.2
        cp -a /usr/lib64/liblber-2.4.so.2.10.7 /var/task/lib/liblber-2.4.so.2
        cp -a /usr/lib64/libsasl2.so.3.0.0 /var/task/lib/libsasl2.so.3
        cp -a /usr/lib64/libssl3.so /var/task/lib/
        cp -a /usr/lib64/libsmime3.so /var/task/lib/
        cp -a /usr/lib64/libnss3.so /var/task/lib/
        cp -a /usr/lib64/libnssutil3.so /var/task/lib/
Back to articles
Back to top