MongoDB rs.initiate error: replSetInitiate quorum check failed because not all proposed set members responded affirmatively

I have to start my own replica set with internal authentication enabled using X.509 certificates, but I failed. Any advice is welcome.

MongoDB 3.2 x64 on Debian 8.2 x64.

It is a problem from MongoDB University course “M310: MongoDB Security”.
I missed a deadline.

A tutor:

You can bring up your server members with authentication options then
rs.initiate, then create your user.

I have this folder structure:

~
`-- shared
    `-- certs
        |-- ca.pem
        |-- client.pem
        `-- server.pem

I created this bash script that set ups my replica set:

#!/bin/bash

course="M310"
exercise="HW-1.3"
workingDir="$HOME/${course}-${exercise}"
dbDir="$workingDir/db"
logName="mongodb.log"

ports=(31130 31131 31132)
replSetName="rs1"

host=`hostname -f`
initiateStr="rs.initiate({
                 _id: '$replSetName',
                 version: 1,
                 members: [
                  { _id: 0, host: '$host:${ports[0]}' },
                  { _id: 1, host: '$host:${ports[1]}' },
                  { _id: 2, host: '$host:${ports[2]}' }
                 ]
                })"

# create working folder
mkdir -p "$workingDir/"{r0,r1,r2}

# launch mongod's
for ((i=0; i < ${#ports[@]}; i++))
do
    mongod --dbpath "$workingDir/r$i" --logpath "$workingDir/r$i/$logName" --port ${ports[$i]} --replSet $replSetName --fork --sslMode requireSSL --sslPEMKeyFile ~/shared/certs/server.pem --sslCAFile ~/shared/certs/ca.pem --auth
    sleep 3
done

echo "Initiate replSet"
mongo --port ${ports[0]} --ssl --sslPEMKeyFile ~/shared/certs/client.pem --sslCAFile ~/shared/certs/ca.pem --host $host --eval "$initiateStr"

The last command returns this error:

MongoDB shell version: 3.2.11
connecting to: database.m310.mongodb.university:31130/test
{
        "ok" : 0,
        "errmsg" : "replSetInitiate quorum check failed because not all proposed set members responded affirmatively: database.m310.mongodb.university:31131 failed with not authorized on admin to execute command { replSetHeartbeat: \"rs1\", pv: 1, v: 1, from: \"database.m310.mongodb.university:31130\", fromId: 0, checkEmpty: true }, database.m310.mongodb.university:31132 failed with not authorized on admin to execute command { replSetHeartbeat: \"rs1\", pv: 1, v: 1, from: \"database.m310.mongodb.university:31130\", fromId: 0, checkEmpty: true }",
        "code" : 74
}

This is mongodb.log file of r0 instance:

2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten] MongoDB starting : pid=6091 port=31130 dbpath=/home/enabokov/M310-HW-1.3/r0 64-bit host=database.m310.mongodb.university
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten] db version v3.2.11
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten] git version: 009580ad490190ba33d1c6253ebd8d91808923e4
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1k 8 Jan 2015
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten] allocator: tcmalloc
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten] modules: none
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten] build environment:
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten]     distmod: debian71
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten]     distarch: x86_64
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten]     target_arch: x86_64
2017-01-18T15:48:08.498+0300 I CONTROL  [initandlisten] options: { net: { port: 31130, ssl: { CAFile: "/home/enabokov/shared/certs/ca.pem", PEMKeyFile: "/home/enabokov/shared/certs/server.pem", mode: "requireSSL" } }, processManagement: { fork: true }, replication: { replSet: "rs1" }, security: { authorization: "enabled" }, storage: { dbPath: "/home/enabokov/M310-HW-1.3/r0" }, systemLog: { destination: "file", path: "/home/enabokov/M310-HW-1.3/r0/mongodb.log" } }
2017-01-18T15:48:08.536+0300 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-01-18T15:48:09.958+0300 I CONTROL  [initandlisten] 
2017-01-18T15:48:09.958+0300 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2017-01-18T15:48:09.958+0300 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2017-01-18T15:48:09.958+0300 I CONTROL  [initandlisten] 
2017-01-18T15:48:09.958+0300 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 7948 processes, 65536 files. Number of processes should be at least 32768 : 0.5 times number of files.
2017-01-18T15:48:09.958+0300 I CONTROL  [initandlisten] 
2017-01-18T15:48:10.833+0300 I REPL     [initandlisten] Did not find local voted for document at startup;  NoMatchingDocument: Did not find replica set lastVote document in local.replset.election
2017-01-18T15:48:10.833+0300 I REPL     [initandlisten] Did not find local replica set configuration document at startup;  NoMatchingDocument: Did not find replica set configuration document in local.system.replset
2017-01-18T15:48:10.834+0300 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/home/enabokov/M310-HW-1.3/r0/diagnostic.data'
2017-01-18T15:48:10.834+0300 I NETWORK  [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2017-01-18T15:48:11.188+0300 I NETWORK  [initandlisten] waiting for connections on port 31130 ssl
2017-01-18T15:48:21.914+0300 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:56026 #1 (1 connection now open)
2017-01-18T15:48:21.922+0300 I ACCESS   [conn1] note: no users configured in admin.system.users, allowing localhost access
2017-01-18T15:48:21.927+0300 I REPL     [conn1] replSetInitiate admin command received from client
2017-01-18T15:48:21.947+0300 I REPL     [conn1] replSetInitiate config object with 3 members parses ok
2017-01-18T15:48:21.947+0300 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to database.m310.mongodb.university:31131
2017-01-18T15:48:21.948+0300 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to database.m310.mongodb.university:31132
2017-01-18T15:48:21.965+0300 I ASIO     [NetworkInterfaceASIO-Replication-0] Successfully connected to database.m310.mongodb.university:31131
2017-01-18T15:48:21.966+0300 W REPL     [ReplicationExecutor] Got error (Unauthorized: not authorized on admin to execute command { replSetHeartbeat: "rs1", pv: 1, v: 1, from: "database.m310.mongodb.university:31130", fromId: 0, checkEmpty: true }) response on heartbeat request to database.m310.mongodb.university:31131; { ok: 1.0, hbmsg: "" }
2017-01-18T15:48:21.966+0300 I ASIO     [NetworkInterfaceASIO-Replication-0] Successfully connected to database.m310.mongodb.university:31132
2017-01-18T15:48:21.967+0300 W REPL     [ReplicationExecutor] Got error (Unauthorized: not authorized on admin to execute command { replSetHeartbeat: "rs1", pv: 1, v: 1, from: "database.m310.mongodb.university:31130", fromId: 0, checkEmpty: true }) response on heartbeat request to database.m310.mongodb.university:31132; { ok: 1.0, hbmsg: "" }
2017-01-18T15:48:21.967+0300 E REPL     [conn1] replSetInitiate failed; NodeNotFound: replSetInitiate quorum check failed because not all proposed set members responded affirmatively: database.m310.mongodb.university:31131 failed with not authorized on admin to execute command { replSetHeartbeat: "rs1", pv: 1, v: 1, from: "database.m310.mongodb.university:31130", fromId: 0, checkEmpty: true }, database.m310.mongodb.university:31132 failed with not authorized on admin to execute command { replSetHeartbeat: "rs1", pv: 1, v: 1, from: "database.m310.mongodb.university:31130", fromId: 0, checkEmpty: true }
2017-01-18T15:48:21.969+0300 I NETWORK  [conn1] end connection 127.0.0.1:56026 (0 connections now open)

Without –auth parameter my replica set initiates successfully, but I need x509 authenticaion turned on for the problem purpose.

Related:


Leave a Reply