[Pdns-users] Authoritative server (v4.1.3) crashes on zone creation via API when too many rrsets included

Thu Jul 19 18:33:40 UTC 2018

Greetings,

Since upgrading our authoritative name servers from 4.0.x to 4.1.3, we've been experiencing strange server crashes when creating zones from scratch. After some trial and error, it seems that the trigger is not the content of the zones being created, but rather their size. Below, I've crafted a zone creation command with some meaningless hosts and IPs that will always cause the server to crash. The symptom of the crash from the API client's perspective is usually an abrupt socket disconnection with no response, but sometimes I will see a HTTP 500 error with no details. If I remove any one of the A rrsets from my crafted zone - it doesn't really matter which one - it succeeds.

Example zone that causes crash (3 NS records, 1 SOA, and 13 A):

POST to http://10.9.9.64:8081/api/v1/servers/localhost/zones:

{

  "id": null,

  "name": "wntest201807191044jr.com.",

  "type": "Zone",

  "url": null,

  "kind": "Master",

  "serial": 0,

  "notified_serial": 0,

  "masters": [],

  "dnssec": false,

  "nsec3param": null,

  "nsec3narrow": false,

  "presigned": false,

  "soa_edit": "",

  "soa_edit_api": "",

  "nameservers": [],

  "servers": null,

  "last_check": 0,

  "rrsets": [

    {

      "name": "wntest201807191044jr.com.",

      "type": "NS",

      "ttl": 21600,

      "records": [

        {

          "content": "ns1.webnames.ca.",

          "disabled": false,

          "set-ptr": false

        },

        {

          "content": "ns2.webnames.ca.",

          "disabled": false,

          "set-ptr": false

        },

        {

          "content": "ns3.webnames.ca.",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "wntest201807191044jr.com.",

      "type": "SOA",

      "ttl": 21600,

      "records": [

        {

          "content": "ns1.webnames.ca. postmaster.webnames.ca. 1531937515 21600 180 1209600 3600",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "yuioyuio.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "uiopuiop.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "asdfasd.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "sdfgsdfg.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "dfghdfgh.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "fghjfgjh.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "ghjkghjk.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "hjklhjkl.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "jkljkl.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "zxczxc.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "xcvxcv.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "cvbcvb.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "vbnvbn.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    }

  ]

}

Response (no response received, we just get an exception in our .NET client):

System.Net.WebException: The underlying connection was closed: The connection was closed unexpectedly.

   at System.Net.HttpWebRequest.GetResponse()

Digging through the PDNS logs, we find that a signal 11 is generated, causing a hard crash and auto-restart of the server:

Jul 19 10:44:54 devpdns1 pdns_server: HTTP: Handling request "/api/v1/servers/localhost/zones"

Jul 19 10:44:54 devpdns1 pdns_server: gmysql Connection successful. Connected to database 'powerdns' on '127.0.0.1'.

Jul 19 10:44:54 devpdns1 pdns_server: Query: select id,name,master,last_check,notified_serial,type,account from domains where name=?

Jul 19 10:44:54 devpdns1 pdns_server: gmysql Connection successful. Connected to database 'powerdns' on '127.0.0.1'.

Jul 19 10:44:54 devpdns1 pdns_server: Got a signal 11, attempting to print trace:

Jul 19 10:44:54 devpdns1 pdns_server: /usr/sbin/pdns_server(+0x2288fb) [0x7fd03005d8fb]

Jul 19 10:44:54 devpdns1 pdns_server: /lib64/libc.so.6(+0x35250) [0x7fd02de72250]

Jul 19 10:44:54 devpdns1 pdns_server: /usr/sbin/pdns_server(_ZNK7DNSName11countLabelsEv+0x29) [0x7fd02ff4f299]

Jul 19 10:44:54 devpdns1 pdns_server: /usr/sbin/pdns_server(_ZNK7DNSName12getRawLabelsEv+0x43) [0x7fd02ff52683]

Jul 19 10:44:54 devpdns1 pdns_server: /usr/sbin/pdns_server(_ZNK7DNSName8toStringERKSsb+0x7c) [0x7fd02ff537fc]

Jul 19 10:44:54 devpdns1 pdns_server: /usr/sbin/pdns_server(+0x29a8c5) [0x7fd0300cf8c5]

Jul 19 10:44:54 devpdns1 pdns_server: /usr/sbin/pdns_server(+0x2a044e) [0x7fd0300d544e]

Jul 19 10:44:54 devpdns1 pdns_server: /usr/sbin/pdns_server(+0x2aee24) [0x7fd0300e3e24]

Jul 19 10:44:54 devpdns1 pdns_server: /usr/sbin/pdns_server(+0x2bae9e) [0x7fd0300efe9e]

Jul 19 10:44:54 devpdns1 pdns_server: /usr/sbin/pdns_server(+0x28d135) [0x7fd0300c2135]

Jul 19 10:44:54 devpdns1 pdns_server: /usr/sbin/pdns_server(_ZN5boost6detail8function26void_function_obj_invoker2INS_3_bi6bind_tIvPFvNS_8functionIFvP11HttpRequestP12HttpResponseEEES7_S9_ENS3_5list3INS3_5valueISB_EENS_3argILi1EEENSH_ILi2EEEEEEEvS7_S9_E6invokeERNS1_15function_bufferES7_S9_+0x49) [0x7fd0300c7e79]

Jul 19 10:44:54 devpdns1 pdns_server: /usr/sbin/pdns_server(+0x28adb9) [0x7fd0300bfdb9]

Jul 19 10:44:54 devpdns1 pdns_server: /usr/sbin/pdns_server(_ZN5boost6detail8function26void_function_obj_invoker2INS_3_bi6bind_tIvPFvNS_8functionIFvP11HttpRequestP12HttpResponseEEEPN6YaHTTP7RequestEPNSC_8ResponseEENS3_5list3INS3_5valueISB_EENS_3argILi1EEENSM_ILi2EEEEEEEvSE_SG_E6invokeERNS1_15function_bufferESE_SG_+0x49) [0x7fd0300c7de9]

Jul 19 10:44:54 devpdns1 pdns_server: /usr/sbin/pdns_server(_ZNK9WebServer13handleRequestER11HttpRequestR12HttpResponse+0x1e5) [0x7fd0300c2925]

Jul 19 10:44:54 devpdns1 pdns_server: /usr/sbin/pdns_server(_ZNK9WebServer15serveConnectionESt10shared_ptrI6SocketE+0x27f) [0x7fd0300c38cf]

Jul 19 10:44:54 devpdns1 pdns_server: /usr/sbin/pdns_server(+0x28eff4) [0x7fd0300c3ff4]

Jul 19 10:44:54 devpdns1 pdns_server: /usr/sbin/pdns_server(_ZNSt6thread5_ImplISt12_Bind_simpleIFPFPvPK9WebServerSt10shared_ptrI6SocketEEPS3_S8_EEE6_M_runEv+0x43) [0x7fd0300c6e03]

Jul 19 10:44:54 devpdns1 pdns_server: /lib64/libstdc++.so.6(+0xb52a0) [0x7fd02e9fa2a0]

Jul 19 10:44:54 devpdns1 pdns_server: /lib64/libpthread.so.0(+0x7dc5) [0x7fd02e20cdc5]

Jul 19 10:44:54 devpdns1 pdns_server: /lib64/libc.so.6(clone+0x6d) [0x7fd02df3473d]

Jul 19 10:44:54 devpdns1 systemd: pdns.service: main process exited, code=killed, status=6/ABRT

Jul 19 10:44:54 devpdns1 systemd: Unit pdns.service entered failed state.

Jul 19 10:44:54 devpdns1 systemd: pdns.service failed.

Jul 19 10:44:56 devpdns1 systemd: pdns.service holdoff time over, scheduling restart.

Jul 19 10:44:56 devpdns1 systemd: Cannot add dependency job for unit firewalld.service, ignoring: Unit is masked.

Jul 19 10:44:56 devpdns1 systemd: Starting PowerDNS Authoritative Server...

Same example zone with one A record removed - does not cause crash (3 NS records, 1 SOA, and 12 A):

{

  "id": null,

  "name": "wntest201807191044jr.com.",

  "type": "Zone",

  "url": null,

  "kind": "Master",

  "serial": 0,

  "notified_serial": 0,

  "masters": [],

  "dnssec": false,

  "nsec3param": null,

  "nsec3narrow": false,

  "presigned": false,

  "soa_edit": "",

  "soa_edit_api": "",

  "nameservers": [],

  "servers": null,

  "last_check": 0,

  "rrsets": [

    {

      "name": "wntest201807191044jr.com.",

      "type": "NS",

      "ttl": 21600,

      "records": [

        {

          "content": "ns1.webnames.ca.",

          "disabled": false,

          "set-ptr": false

        },

        {

          "content": "ns2.webnames.ca.",

          "disabled": false,

          "set-ptr": false

        },

        {

          "content": "ns3.webnames.ca.",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "wntest201807191044jr.com.",

      "type": "SOA",

      "ttl": 21600,

      "records": [

        {

          "content": "ns1.webnames.ca. postmaster.webnames.ca. 1531937515 21600 180 1209600 3600",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "yuioyuio.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "uiopuiop.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "asdfasd.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

       }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "sdfgsdfg.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "dfghdfgh.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "fghjfgjh.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "ghjkghjk.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "hjklhjkl.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "jkljkl.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "zxczxc.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "xcvxcv.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    },

    {

      "name": "cvbcvb.wntest201807191044jr.com.",

      "type": "A",

      "ttl": 21600,

      "records": [

        {

          "content": "77.77.77.200",

          "disabled": false,

          "set-ptr": false

        }

      ],

      "comments": null,

      "changetype": "REPLACE"

    }

  ]

}

Response:

HTTP 200 with full zone JSON. (For brevity, I won't paste it here, but it's a normal successful response to zone creation.)

Any ideas what this could be? It feels like some sort of resource/buffer size exhaustion leading to an overflow somewhere deep in the code. Note that once the zone size is cut down and it is created successfully, PATCH updates will succeed and leave the zone as large as requested. It seems to only affect the initial zone creation.

Server: RHEL 7.3 VM with 2GB RAM

Backend: MySQL 5.6.36

Thanks,

Jordan Rieger

Software Development Manager

Webnames.ca Inc.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20180719/a50958cd/attachment-0001.html>